Routine Update gone Wrong
One of my servers, an ASUS server from 2017 had some issues booting. During some commands that ran a bit longer, I decided in my ultimate wisdom to update the IPMI interface of the server. Usually, that isn’t that big of a deal. I’ve done that on multiple occasion using HPE, Dell or super micro servers. So i figured, the process should be rather similar on this Asus server.
IPMI update is supported using two ways as stated in the support page. The first way is to flash the image via FreeDOS, the second to just upload the new image via the IPMI interface directly. As I did not want to physically go to the server, reboot it and flash a USB drive right now, I decided to go with the more comfortable seeming option.
The images on the server were quite old, both the IPMI and UEFI images were from early 2018, so more than 6 years old at this point. Multiple new versions were available, also a mayor version jump from version 1 to 2. Here my first small error occurred, I did not check on the first release with a new major version to check for any migration steps. So I simply downloaded the latest release and tried flashing it.
Uploading the new image did not result in any errors and went well. However, as I now know, you still have to confirm the actual flashing. Instead, the IPMI interface went dark and was not reachable anymore. It did not respond to ping messages, and ipmitool on debian error’d out and stated, no IPMI interface was found. Rebooting did not help. Quite in contrast, the boot issues was not resolved, so it did not boot correctly and stayed completely offline.
ohno.jpg
Trying to Recover
Of course, I did that all remotely. So as I seemingly bricked the IPMI interface, I had no way to remotely debug the machine anymore. Now I also started to read the release notes more closely and looked at all releases of the firmware images. My server was at IPMI version 1.11.5 and I tried to move to the newest version which was 2.2.9. Version 2.2.4, the earliest version starting with 2, had a note attached, stating, that starting with 2.2.3 update via WebUI is not available. This is due to the bump in mayor version, so only FreeDOS is possible. That was an error on my side, not checking the earlier release notes. Frustrated I packed my bag, grabbed some USB drives and biked to the data center.
On the way there, I hatched some plan to get everything back going. The boot issue is not super important, once IPMI is back up, I can work on that. However, the not working update now had me worked up and I wanted to finish what I started.
As access via IPMI was not possible anymore, I had to try the other way to update it. I downloaded all IPMI images needed for the incremental upgrade and flashed FreeDOS on a USB stick. That last part was a bit more complicated than anticipated as the official documentation on their wiki was down, apparently for a few months already. At the end, it was just a simple dd command to copy the image onto an USB stick. FreeDOS only supports BIOS booting, so I rebooted the machine and switched the global boot mode to CSM.
This however did not result in the machine booting cleanly into FreeDOS. It always got stuck on a black screen. The seven-segment indicator on the back of the server (nicest feature ever btw) indicated a successful POST and handover to the booted operating system. So it appeared to have booted successfully, however, it did not work as intended.
The awesome seven-segment showing POST codes on the back of this ASUS server.
I tried four different USB sticks, none of which worked, all resulted in the same black screen. Tried cold booting the server, so fully removing power waiting a bit and cleanly starting the machine. The cold boot recovered the IPMI interface, so that was good news. I still wanted to try the update via FreeDOS, as I was not trusting the IPMI update process via webpage anymore. At the end, I did not manage to get FreeDOS to boot. It always hanged at a black screen without displaying anything. Frustrated I conceded and went home for the day.
On the search for the missing piece of the puzzle
As this problem puzzled me for the wrong reasons, I used any free minute to read up on this particular issue. The FreeDOS wiki was down, so we got no real documentation to go from. Any other post were mostly other, unrelated issues, none included a black screen. Finally, on one particular deep duckduckgo search I discovered a post on the servethehome forum. fmatthew5876 mentioned, FreeDOS also needs option ROM for the video card to be set to legacy. He was using a Super Micro X11 board, but this difference should not be an issue. I flashed FreeDOS again, this time more organized. I further created a second partition on the stick, to put all update images on it.
On the next day in the datacenter, I rebooted the server and changed all option ROMs to legacy. This time the server boots instantly into FreeDOS without issues. I have to admit, I really started to doubt my own abilities and did a small victory dance on the successful boot. Flashing worked as expected, each firmware upgrade took a good while, around 30min per flash. For IPMI, a reboot was not needed. I went from version 1.11.5 (Oct 2017) to 1.15.4, to 2.2.3 and finally to 2.2.9 (Jan 2023).
CSM Settings within the UEFI, Option ROMS are not yet correctly set for FreeDOS.
All is good now, IPMI worked as intended. Already in the mood for updating the server I further tried updating BIOS via the IPMI webinterface. This time I looked through all release notes and found one note to upgrade to an intermediate version. My system was already above that version, so I could just install the latest one, which worked without any issues. The new IPMI version feels way more responsive. It also added a BIOS button on the top that should bring up a remote control for the UEFI menu. However, the new window doesn’t show anything, so not everything appears to work correctly. Still, I learned a lot during this update process and was happy that the machine was again working as planned.
Key Lessons learned
As a long-time reader of servethehome, I really liked the introduction of the Key Lessons learned section as a conclusion. So let’s try to replicate that as I learned quite some lessons here.
Reading the release notes of all new releases since last update
This one should have been obvious, as you never know when a breaking change was released. Further, nobody will highlight breaking changes for older releases on new releases. So in the future, I will look at all the releases published between the currently flashed version and the newest version.
However, I would have loved a small notification somewhere to highlight the correct way from V1 to V2 instead of finding it in a PDF of release 2.2.4. A small text file included with all zip archives could have tripped me. Still, this was my error and I cannot trust anyone blindly to put up such notices.
Legacy BIOS / CSM Option ROMS
I learned quite some things about the settings available for CSM - or Compatibility Support Module for UEFI. For booting FreeDOS it is not enough to only enable CSM legacy support for booting. You also have to set the Option ROMs for the different devices it wants to access. In this case, I think, only Video ROM was necessary, however, I did not explicitly check that.
Schedule some time for these kind of things
I tried to do too many things at once here. Updating the IPMI interface seems like an easy task, that can be done next to some other tasks. However, both tasks involved the same machine, interrupting both tasks to get no work done at all. Further, IPMI is a crucial part of the machine. Without it, remote debugging of failed OS or other things is simply not possible. Maybe only update it when you have physical access to the machine in case any unforeseen problems occur. Also, I should have tried a cold (re-)start right at the beginning, that might have eased my nerves knowing I did not brick anything.
Flashing FreeDOS
Here a small write-up how I prepared my USB stick in the end.
- Grab the Full USB version of FreeDOS from their official website.
- Unpack zip archive
- Identify your USB stick using lsblk or similar
- Copy content using dd, THE DRIVE WILL BE FULLY WIPED!
- Optional: The drive will not be using the entire space, so create a second partition
- Optional: Format new partition to FAT32
wget https://www.ibiblio.org/pub/micro/pc-stuff/freedos/files/distributions/1.3/official/FD13-FullUSB.zip
unzip FD13-FullUSB.zip
sudo dd if=./FD13FULL.img of=/dev/your_disk_here bs=4M status=progress
sudo fdisk /dev/your_disk_here
# create new partition in fdisk using all available space left
sudo mkfs.fat -F32 /dev/your_partition_here
Last modified on 2024-12-16