Supermicro BIOS Shenanigans

Let me take you on a small journey. You receive a large number of Supermicro workstations with X9SRG-F boards. You get them connected and racked up, try to install something and the keyboard doesn't work. You try another, same thing.

OK, it's a USB keyboard, maybe there is a problem with the BIOS reading USB, I'll just plug in a PS/2 keyboard. Oh there isn't a PS/2 port on this board. Right lets try IPMI to get into the BIOS using the console. Great, this works, but you cannot mount any USB devices at all. I'll just plug in a SATA drive and get the BIOS flashed to see if it's something up with the BIOS.




I download the latest (X9SRG3_801) and only version available on the Supermicro website which doesn't come with any release notes whatsoever. I'm sure this version will fix it, it's the latest right? I boot a DOS boot CD, loaded with the update and flash the BIOS (and IPMI). Reboot.....F9 BIOS error code. This means the main BIOS block is corrupt. In this instance you need to recover the BIOS by renaming the BIOS file to SUPER.ROM, place it on the root drive of a FAT32/16/12 formatted USB flash drive and hold down Ctrl+Home on the keyboard when the machine turns on.

After recovering via the recovery option screen, this would still not get past the above error once rebooted, so I can only surmise that the latest BIOS version is to blame. Unfortunately Supermicro doesn't release older versions on their site, you have to email support to get these. Although, you can use Wayback machine to view the cached version of the BIOS download page, to obtain older versions (click the date you think is right, go through the agreement and then modify the link when it cannot find the file). I downloaded the previous version (X9SRG3_306) and recovered with this instead.

It works! The BIOS "recovered"....but USB devices nor SATA devices would detect after the update.

So now this machine doesn't boot from any device and it just goes straight into the BIOS. Triple checking through and trying different BIOS options didn't help. But there is a "dxe bs driver unrecognised" error in the BIOS log (BS indeed).

By the looks of things this board is well and truly toast. The only thing I can sum up is that the BIOS startup block and the main block are mismatched and causing issues (the recovery mode only flashes the main block). I spoke to our distributor and they said just RMA it. But we don't have time to do this as these machines need to go into service and I cannot trust any updates at this time to work on the other machines.

Let's check what type of chip this is:

It's a Windbond 25Q64CV. Hmm, lets see if flashrom supports this:

Oh wonderful! Well, I do still have my buspirate from my previous flashing endeavors. I get the soldering iron out and whip the chip out. Using an SOIC ZIF socket I got on ebay a few years back I checked the pinout:

Here is the buspirate to 25Q64CV pinout (notice it's exactly the same as the MX25L8005PC from the previous blog post):

Bus Pirate     25Q64CV - Pin#
CS             CS   - 1
MISO           DO   - 2
V+ 3.3v        WP   - 3
GND            GND  - 4
MOSI           DI   - 5
CLK            SCK  - 6
V+ 3.3v        HOLD - 7
V+ 3.3v        VCC  - 8
Vpullup        VCC  - 8

I updated the buspirate firmware to the latest just to be sure:

Grabbed the lasted svn release of flashrom (0.9.7-r1763), connected it up and read the chip:

Success!

Note that this flashing process did take a little time and I created a script to read, flash and verify (run this script as root/sudo):
#!/bin/bash
flashrom -p buspirate_spi:dev=/dev/ttyUSB0,spispeed=8M -r ./OLD.ROM  -V | tee ./read
flashrom -p buspirate_spi:dev=/dev/ttyUSB0,spispeed=8M -w ./X9SRG3.306  -V | tee ./write
This will create two text files once complete with the output for you to check (the write also verifies), plus another with the data originally on the chip.

There is a more recent patch that might not have made it's way into the buspirate firmware which makes the reading and flashing much faster too:

http://dangerousprototypes.com/forum/viewtopic.php?f=40&t=3864&start=15#p41505

Erase/write done.
Verifying flash... VERIFIED.
Raw bitbang mode version 1
Bus Pirate shutdown completed.

I solder the chip back into the machine.

Even more success! The machine boots, can once again see devices AND USB NOW WORKS!

Thank you so much Supermicro for not releasing different BIOS revisions and not even having the decency to have release notes for the different revisions. Really bloody helpful!