Recovering the Firmware on a Supermicro BPN-SAS3-846EL1 Backplane

In a previous adventure I replaced an Adaptec HBA with a LSI SAS3 HBA, and the chassis drive bay LED’s stopped working. I suspect the LSI card does not play nice with the SGPIO sideband controller, and I decided to replace the chassis with one similar to my SC846 chassis, where the LSI card and drive bay LED’s do work fine.

What should have been a simple replacement turned into quite a recovery operation.

Since I now had SAS3 HBA’s in both my servers, I really wanted to get SAS3 backplanes, but I did not want to pay SAS3 chassis prices. I found a refurbished SuperChassis 846E16-R1200B chassis and two refurbished Supermicro BPN-SAS3-846EL1 backplanes on eBay. The one SAS3 backplane would replace the SAS2 backplane in my existing SC846, and the second SAS3 backplane would replace the SAS2 backplane in the newly purchased SC846. The combination of the 24 bay 4U chassis with a SAS2 backplane and a replacement SAS3 backplane is much cheaper compared to any native SAS3 chassis I could find.

I do understand that using SATA3 drives on a SAS3 backplane will not perform like SAS3 drives, but with multipath the aggregate throughput can still outperform SAS2.

I received my chassis and my backplanes. The chassis was clean but a bit dinged in one corner, and the expanders were clean, but the metal frames showing a little rust. I had no idea how old the firmware on the backplanes were, so I contacted Supermicro support to ask for the current firmware. After asking for my serial numbers, Supermicro sent me the latest firmware for my hardware. The firmware update instructions were included in the “ReleaseNote.txt” file that came with the firmware.

I removed the motherboard from the old chassis and installed it in the new SC846. I removed the SAS2 backplane, and installed the SAS3 backplane in its place. The power cable layout on the SAS3 backplane is a bit different, and I had to use a few molex power splitters to extend the power cables to reach the power plugs on the SAS3 backplane. The standard rails are too long to fit in my rack, and the short rails are too short for the chassis, so as I again used short outer rails and standard inner rails.

I powered the machine up through remote IPMI KVM, all looked good, and I booted into my Ubuntu Server USB stick so I could SSH into the box, and update the firmware.

The instructions from “ReleaseNote.txt” say:

How to Flash Firmware
-------------------------
Under Linux/Windows Environment to use CLIXTL (ver.6.00)
1. use "CLIXTL -l" to show SAS addresses
2. use "CLIXTL -f all  -d " to update firmware
3. use "CLIXTL -f 3  -d  -r" to update MFG and reset expander

example:
CLIXTL -l
CLIXTL -f all -t 500304800000007F -d SAS3-EXPFW_66.16.11.00.fw
CLIXTL -f 3 -t 500304800000007F -d BPN-SAS3-846EL-PRI_16_11.bin -r

The existing firmware on the expanders were v66.06.01.02 and MFG v06.02, while the new firmware was v66.16.11.00 and MFG v16.11.

Firmware Update History
-------------------------
01. migrate expander firmware to phase 16 
02. enhance TMP, VOL, FAN, and PWS status in SES pages
03. present version information of current running firmware
04. Dynamic SES page element presentation
05. move BMC IP to SCSI network inquiry 
06. support I2C R/W as slave to let BMC to identify platforms
07. redundant side has sensor information 
08. firmware rewrite and optimization

I did the following:

$ sudo ./CLIXTL -i -t 5003048001B24DBF
================================================================================
COMMAND-LINE INTERFACE XTOOL
version 6.10.C
Supermicro Computer ,Inc.
================================================================================
UNIT SPECIFIC INFORMATION:
SAS ADDRESS - 5003048001B24DBF
ENCLOSURE ID - 5003048001B24DBF
ENCLOSURE INFORMATION:
PLATFORM NAME - SMC846ELSAS3P
SERIAL NUMBER -
VENDOR ID - LSI
PRODUCT ID - SAS3x40
VERSION INFORMATION:
FLASH REGION 0 - 66.06.01.02
FLASH REGION 1 - 66.06.01.02
FLASH REGION 2 - 66.06.01.02
FLASH REGION 3 - 06.02
DEVICE INFORMATION:
DEVICE NAME - /dev/sg0
BMC IP - NULL

$ sudo ./CLIXTL -f all -t 5003048001B24DBF -d SAS3-EXPFW_66.16.11.00.fw
================================================================================
COMMAND-LINE INTERFACE XTOOL
version 6.10.C
Supermicro Computer ,Inc.
================================================================================
Firmware Region 0 - Finished
Firmware Region 1 - Finished
Firmware Region 2 - Finished
Please reset expander to activate

pieter@ubuntuusb:~/SAS3$ sudo ./CLIXTL -f 3 -t 5003048001B24DBF -d BPN-SAS3-846EL-PRI_16_11.bin -r
================================================================================
COMMAND-LINE INTERFACE XTOOL
version 6.10.C
Supermicro Computer ,Inc.
================================================================================
Error, incompatible file type or directory

And the expander dropped and never came back up.

$ sudo ./CLIXTL -l
================================================================================
COMMAND-LINE INTERFACE XTOOL
version 6.10.C
Supermicro Computer ,Inc.
================================================================================
Error, no enclosure has been found

I can hear the comments now, why risk updating the firmware if it is not broken, true, but I didn’t know if it would work, and I’d rather start fresh. Do also note that I did the update on my secondary server, my primary server is still running unmodified, so no interruption of home service or work experiments.

I still had the second SAS3 backplane, so I replace the “bricked” one, leaving the firmware as is, and brought my Unraid server back up, all appeared fine. At least I had two working servers, giving me time to try and recover the backplane.

I sent Supermicro support an email asking for help, but since it was weekend I had to wait, so I did my own research. I found a forum post of a user that recovered a “bricked” SAS2 expander via the factory serial port, and I decided to give it a try.

I used my Arduino programming FTDI USB RS232 adapter, and the pin connections for PRI-SDB / 8 are:

PRI-SDB : 1 : TX  -> RS232 : RX
PRI-SDB : 2 : GND -> RS232 : GND
PRI-SDB : 3 : RX  -> RS232 : TX

The current XTools v6.10.C CLI does not include COM support, at least none that I could find in the documentation or CLI help, so I used the older v1.4 version.

>g3Xflash.exe -s com4 get avail
********************************************************************************
g3Xflash
LSI SAS Expander Flash Utility
Version: 2.0.0.0
Copyright (c) 2013 LSI Corporation. All rights reserved.
********************************************************************************
Initializing Interface.....
Expander: Unknown (SAS_3X_40)
1) Unknown (SAS_3X_40) (00000000:00000000)

Good sign, the COM port worked, and the expander hardware was detected, but did not have an address.

I flashed the firmware and the MFG data:

>g3Xflash.exe -s com4 down fw SAS3-EXPFW_66.16.11.00.fw 0
********************************************************************************
g3Xflash
LSI SAS Expander Flash Utility
Version: 2.0.0.0
Copyright (c) 2013 LSI Corporation. All rights reserved.
********************************************************************************
Initializing Interface..
Expander: Unknown (SAS_3X_40)
Expander Validation: Passed
Checksum: Passed
Target Firmware Region: 00
Current Version: 255.255.255.255
Replacement Version: 66.16.11.00
Image Validation: Passed
Pre-Validation of image is successful.
Are you sure to download file to expander?(y/n):y
Downloading File...........................................................................Download Complete.
Post-validating........................................................Post-Validation of image is successful.
Download Successful.

>g3Xflash.exe -s com4 down mfg BPN-SAS3-846EL-PRI_16_11.bin 3
********************************************************************************
g3Xflash
LSI SAS Expander Flash Utility
Version: 2.0.0.0
Copyright (c) 2013 LSI Corporation. All rights reserved.
********************************************************************************
Initializing Interface.....
Expander: Unknown (SAS_3X_40)
Image Validation: Passed
Checksum: Passed
Reading MFG version from flash...Unable to retrieve version.
Replacement Version: 10.0b
Pre-Validation of image is successful.
Are you sure to download file to expander?(y/n):y
Downloading File............................Download Complete.
Post-validating.........Post-Validation of image is successful.
Download Successful.

I reset the expander, the LED’s now did a test pattern that they did not do before, and things looked good:

>g3Xflash.exe -s com4 reset exp
********************************************************************************
g3Xflash
LSI SAS Expander Flash Utility
Version: 2.0.0.0
Copyright (c) 2013 LSI Corporation. All rights reserved.
********************************************************************************
Initializing Interface....................
Expander: SC846-P (SAS_3X_40)
Are you sure you want to reset Expander?(y/n):y
Expander reset successful.

>g3Xflash.exe -s com4 get avail
********************************************************************************
g3Xflash
LSI SAS Expander Flash Utility
Version: 2.0.0.0
Copyright (c) 2013 LSI Corporation. All rights reserved.
********************************************************************************
Initializing Interface..
INFO: Bootstrap is not present on board.
Downloading the Bootstrap
................................................................Download Bootstrap Complete.
..................
Expander: SC846-P (SAS_3X_40)
1) SC846-P (SAS_3X_40) (50030480:0000007F)

>g3Xflash.exe -s com4 get exp

********************************************************************************
g3Xflash
LSI SAS Expander Flash Utility
Version: 2.0.0.0
Copyright (c) 2013 LSI Corporation. All rights reserved.
********************************************************************************
Initializing Interface....................
Expander: SC846-P (SAS_3X_40)
Reading the expander information................
Expander: SC846-P (SAS_3X_40) C1
SAS Address: 50030480:0000007F
Enclosure Logical Id: 50030480:0000007F
Component Identifier: 0x0232
Component Revision: 0x03

>g3Xflash.exe -s com4 get ver 0
********************************************************************************
g3Xflash
LSI SAS Expander Flash Utility
Version: 2.0.0.0
Copyright (c) 2013 LSI Corporation. All rights reserved.
********************************************************************************
Initializing Interface....................
Expander: SC846-P (SAS_3X_40)
Firmware Region Version: 66.16.11.00

Everything looked good, except the SAS address defaulted to 50030480:0000007F.

The firmware “ReleaseNote.txt” file states that the v6.00 CLIXTL tool can change the SAS address, but the only version on the Supermicro site is the  v6.10.C version, that does not support changing the SAS address.

How to modify SAS address
-------------------------
Under Linux/Windows Environment to use CLIXTL (ver.6.00)
1. use "CLIXTL -l" to show SAS addresses
2. use "CLIXTL -s  -t  -r" to change SAS address and reset expander

example:
CLIXTL -l
CLIXTL -s 500304801234567F -t 500304800000007F -r

The v1.4 GUI does support changing the SAS address. It appears that the GUI dynamically creates a MFG image (I could see a BIN file get created in the directory), but after it changed the address, the backplane was back to a borked state, and I had to repeat the recovery process.

By the next week I heard back from Supermicro, and they confirmed the instructions from the “ReleaseNote.txt” file were wrong, and I should use the instructions from the “Command-line Xtool 6.10.C.pdf” file.

Wrong, will bork the MFG data:
CLIXTL -f 3 -t 5003048001B24DBF -d BPN-SAS3-846EL-PRI_16_11.bin -r

Right:
CLIXTL -c -t 5003048001B24DBF -d BPN-SAS3-846EL-PRI_16_11.bin

Better, update firmware and MFG and retain settings:
CLIXTL -a usc -t 5003048001B24DBF -d ~/

I used the all in one update method on the server that was running the original firmware backplane, and it updated without issue:

# ./CLIXTL -a usc -t 500304800914683F -d ~/CLIXTL6.10.C_Linux/
================================================================================
COMMAND-LINE INTERFACE XTOOL
version 6.10.C
Supermicro Computer ,Inc.
================================================================================
Firmware Region 0 - Finished
Firmware Region 1 - Finished
Firmware Region 2 - Finished
MFG page Region 3 - Finished
New configuration is uploaded successfully
Please reset expander to activate

[Reboot]

# ./CLIXTL -i -t 500304800914683F
================================================================================
COMMAND-LINE INTERFACE XTOOL
version 6.10.C
Supermicro Computer ,Inc.
================================================================================
UNIT SPECIFIC INFORMATION:
SAS ADDRESS - 500304800914683F
ENCLOSURE ID - 500304800914683F
ENCLOSURE INFORMATION:
PLATFORM NAME - SMC846ELSAS3P
SERIAL NUMBER -
VENDOR ID - LSI
PRODUCT ID - SAS3x40
VERSION INFORMATION:
FLASH REGION 0 - 66.16.11.00
FLASH REGION 1 - 66.16.11.00
FLASH REGION 2 - 66.16.11.00
FLASH REGION 3 - 16.11
DEVICE INFORMATION:
DEVICE NAME - /dev/sg11
BMC IP - NULL

I now had one perfectly updated backplane preserving all the original MFG data, and one backplane with default MFG data. I wanted to apply the MFG data from the good backplane to the default values backplane.

I downloaded the firmware and manufacturing data from the good backplane using the v1.4 tools (not supported in current CLIXTL):

./g3Xflash -i get avail
./g3Xflash -y -i 500304800914683F up fw up_fw_loader_0.fw 0
./g3Xflash -y -i 500304800914683F up fw up_fw_loader_1.fw 1
./g3Xflash -y -i 500304800914683F up fw up_fw_loader_2.fw 2
./g3Xflash -y -i 500304800914683F up mfg up_mfg_loader.bin 3

The downloaded files are larger and are padded with 0xFF or 0x00. I trimmed the MFG file to the right size, and modified the SAS address in two places:

2020-01-22 (3)

I tried to uploaded the modified MFG data:

>g3Xflash.exe -y -s com4 down mfg BPN-SAS3-846EL-PRI_16_11.bin 3
********************************************************************************
g3Xflash
LSI SAS Expander Flash Utility
Version: 2.0.0.0
Copyright (c) 2013 LSI Corporation. All rights reserved.
********************************************************************************
Initializing Interface.....
Expander: Unknown (SAS_3X_40)
Image Validation: Passed
Checksum: Failed

But the tool complains that the checksum failed. From the file diff we can see that there is more than just the SAS address that change, I assume some sort of checksum calculation that goes with the data.

The v1.4 g3xFlash CLI help does reference XML options for converting between binary and XML MFG formats, but no instructions on how to use it. Like the serial recovery procedure these tools are probably for internal use only, and I could find no public references.

>g3Xflash.exe -h
********************************************************************************
    g3Xflash
    LSI SAS Expander Flash Utility
    Version: 2.0.0.0
    Copyright (c) 2013 LSI Corporation.  All rights reserved.
********************************************************************************
SYNTAX:
    g3Xflash OPTIONS INTERFACE COMMAND
OPTIONS:
...
COMMAND:
...
        up mfg  
...
            In case of mfg upload using XML file, command syntax changes
            as below. XML file needs to be specified at two places.
            e.g.
            "g3Xflash -x   up mfg  3"

Supermicro support confirmed the current tools cannot change the SAS address, they would not supply the older version of the tools, and recommended I send the backplane in for service, or allow them to remotely SSH to the machine and they will change it for me. A bit disappointing that something so simple is made so complicated, and really way too much trouble.

Since I only had one expander in the chassis, there would be no issues using a default SAS address, and I decided to leave it as is. I replaced the other SAS2 backplane with the recovered SAS3 backplane, and the expander and all drives were back online.

If anybody knows how to update the SAS address, or has a copy of the v6.00 CLIXTL tools that supposedly can change the address, please do let me know.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.