I seem to be cursed when it comes to Ubuntu updates. I just tried upgrading Ubuntu 19.10 to 20.04 (ie: eoan to focal) on a ordinary home server. Once again do-release-upgrade failed right as systemd was being updated. In the past that’s because the systemd upgrade broke my network configuration, because of course I’m doing the upgrade via ssh. (In a screen session though!) This time around it was worse, because the system wouldn’t boot: `- - `-`[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]- - -`z Well that’s exciting. Turns out this is a well known problem with an easy solution. Basically if something interrupts an apt upgrade of the kernel at the wrong time, the kernel will be installed and chosen as the default but the matching initrd ramdisk to make it bootable won’t be properly installed. (ramdisk creation seems to be deferred to the very end of the apt process). The result is the system won’t boot with the new kernel. UEFI may be part of what makes this a challenge. Fortunately there are usually other older kernels also installed that still work on the hard drive, so all you have to do is convince GRUB or whatever boot loader you’re using to boot an older kernel. Then you can go in and fix things. The command that seems most clearly to fix things is `dpkg --triggers-only linux-image-5.4.0-29-generic` Of course you need to substitute the right package name there, for the latest kernel that’s not booting. You can find it this way by listing all packages installed with “triggers pending”: `dpkg -l | grep ^it` Another bit of advice on that Stack Overflow page is to run `dpkg --configure -a` This will force any packages that were installed but not configured to configure themselves. My aborted Ubuntu upgrade seems to have left me with ~40 unconfigured package, this fixed them all in a jiffy. I sure wish I knew why upgrading systemd sometimes breaks networking on Ubuntu. It’s happened on two different systems of mine. And it’s an absolute disaster if you only have local access to the machine. Hilariously, do-release-upgrade still advises against running it if you’re ssh into the machine. How am I supposed to update the machine, fly to the datacenter and type at the keyboard?! I also wish that apt or do-release-upgrade were a little more robust in leaving you with a bootable kernel. The good thing here is that apt and dpkg are very robust themselves. This is the third time I’ve had do-release-upgrade fail on me now, but each time I can pretty much rely on apt and dpkg to fix whatever mess was created. **Bonus round: USB keyboards** This whole ordeal got a lot more complicated because I couldn’t find a keyboard to plug into the Intel NUC hardware to access the boot console. I use an old USB wired Apple Mac 101 key keyboard on my regular computer. For some bizarre reason that fails to work when plugged in to the NUC for accessing the BIOS. I ended up having to wait two days for Amazon to ship me a cheap $10 Chinese keyboard. That’ll teach me to not have a bunch of old computer parts lying around the house. Like Loading... ### _Related_

I updated my main Proxmox server today from 8.4 to 9.1. The big change is an upgrade from Debian bookworm to trixie. And of course lots of Proxmox changes on top of that. This server is the one doing important work. And to make things spicy, it’s a little customized: I have it running an NFS server on the PVE host itself, with a special loop subnet for communication between the Proxmox guest systems (including NFS). I also have Tailscale installed, Proxmox has no first class support for that. The upgrade went fine. A little scary, as always, but no real problems. They have good docs on it and a nice `pve8to9` tool for testing as you go. It’s a little bit of a manual process, I don’t love running `sed` commands via copy-and-paste and being told “just look at the config file changes and decide for yourself what to keep”. But at its core it’s basically just a regular Debian update and that process is well tested. I did make a rookie mistake of upgrading via `ssh` without a `screen` session but I got away with it. (The upgrade process is smart enough not to kill your active ssh session!) Little hiccups I encountered: * I had to hand-edit one apt source to get tailscale on trixie instead of bookworm. * I got a very confusing message from the upgrade about `/etc/initramfs/post-update.d/systemd-boot` being changed and which version did I want to keep. This issue is not in the Wiki (it should be) but it is discussed here. I went with “N”, keeping the old Proxmox version which just got stubbed out with `This hookfile has been disabled by proxmox-boot-tool`. I’m guessing its contents don’t matter as Proxmox overrides Debian’s boot setup. * After the install `pve8to9` told me I should remove the `systemd-boot` package to prevent future upgrade problems. This issue is well documented on the wiki and removing it was fine in my case. The boot hiccups are scary. Booting in Linux is confusing, Proxmox is doing something even trickier with its own bootstrap and booting from ZFS. And if this screws up your whole system is broken in a hard-to-repair way. But I think the underlying tools are pretty robust. The `pve8to9` script also seems to have enough checks and safeguards in place it works as a safety net. I think. Like Loading... ### _Related_

Shared via ChatGPT