KB Article Updates

June 2, 2021

• Fixed typos stating that the issue exists for coreboot branch 4.12.0.X, when in fact the issue has been identified on the 4.9.0.X branch.
• New coreboot version 4.9.0.2 was released for the FW2B that fixes the HPET issue at the BIOS level. No update for FW4B at this time.
• Added new article section on how to update to 4.9.0.2 on the FW2B.

TL;DR

Some FW2B and FW4B CPUs have issues booting the Linux kernel from coreboot. This is due to how Linux tries to setup a high-precision internal timer and how coreboot exposes said timer. By passing a kernel parameter of hpet=disable, you can instruct the Linux kernel to not try to use an HPET timer and default to an alternative.

Synopsis

Attempting to use some Linux kernels with an FW2B or FW4B device with coreboot 4.9.0.1 will result in the inability to boot due to a kernel panic when attempting to set up clock timing and IO-APIC. This is due to some inconsistencies seen in the Intel Braswell CPU platform employed in some Protectli devices.

Targeted Usecases

This bug may affect devices that meet all of the following criteria

• The device is an FW2B or FW4B
• The device BIOS is coreboot version 4.9.0.1 or lower
• The kernel used is Linux-based (Debian, Ubuntu, etc.)

Symptoms

An affected unit is difficult to identify, as the origin of the issue is not entirely known. Protectli have seen this issue only with some Intel Braswell CPUs, and less than 10% of our tested devices suffer from this issue. To identify the issue, simply attempting to boot an OS using the Linux kernel will produce immediate failure like the following:

[ 1.042672] ACPI: Core revision 20180810
[ 1.046907] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 133484882848 ns
[ 1.056149] APIC: Switch to symmetric I/O mode setup
[ 1.062662] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[ 1.168761] ..MP-BIOS bug: 8254 timer not connected to IO-APIC
[ 1.174670] ...trying to set up timer (IRQ0) through the 8259A ...
[ 1.180923] ..... (found apic 0 pin 2) ...
[ 1.285109] ....... failed.
[ 1.287978] ...trying to set up timer as Virtual Wire IRQ...
[ 1.393720] ..... failed.
[ 1.396416] ...trying to set up timer as ExtINT IRQ...
[ 1.629867] ..... failed :(.
[ 1.632825] Kernel panic - not syncing: IO-APIC + timer doesn't work! Boot with apic=debug and send a report. Then try booting with the 'noapic' option.
[ 1.632825]
[ 1.648228] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.19.0-16-amd64 #1 Debian 4.19.181-1
[ 1.656581] Hardware name: Protectli FW2B/FW2B, BIOS v4.9.0.1 03/25/2019
[ 1.663360] Call Trace:
[ 1.665897] dump_stack+0x66/0x81
[ 1.669290] panic+0xe7/0x24a
[ 1.672335] setup_IO_APIC+0x761/0x846
[ 1.676161] ? clear_IO_APIC_pin+0xb8/0x110
[ 1.680419] x86_late_time_init+0x17/0x1c
[ 1.684503] start_kernel+0x460/0x52a
[ 1.688244] secondary_startup_64+0xa4/0xb0
[ 1.692514] ---[ end Kernel panic - not syncing: IO-APIC + timer doesn't work! Boot with apic=debug and send a report. Then try booting with the 'noapic' option.
[ 1.692514] ]---

Resolutions

If you have an FW2B you may simply upgrade to coreboot 4.9.0.2 which fixes the HPET issue at the BIOS level. If you do not wish to upgrade to newer coreboot BIOS, then you may follow the below instructions to edit your Vault's bootloader to instruct the Linux kernel to avoid using HPET for timing.

Option 1: Upgrade to coreboot 4.9.0.2 on the FW2B

This resolution was added on June 2, 2021.

Follow the instructions listed on our article How to Use Flashli and flash the coreboot 4.9.0.2 BIOS.

Option 2: Edit Kernel Command Line Options on the Bootloader

Resolving this issue involves editing your bootloader configuration to pass a parameter to the Linux kernel advising to not use hpet for timing. This is achieved by passing the following parameter to the command line: hpet=disable. Some examples follow for common bootloaders.

GRUB from the affected system

1. At the boot screen, press e to edit the console commands
2. Add the following parameter to the end of the command line:
   hpet=disable
3. Save and boot.

GRUB from a different system

If you had a stable experience with your Vault device, but then flashed an affected coreboot BIOS, you can remove the hard disk from an affected machine and mount it elsewhere to apply the fix.

1. Unscrew the bottom plate of your Vault and remove the mSATA device.
2. Place the mSATA device in a known, working system that also has Linux. For example, you can boot into a Live Ubuntu USB OS and attach the mSATA by a USB interface.
3. Launch a terminal on the Live OS.
4. Mount the disk if it is not already mounted. In this example we use sdX1 as the affected mSATA, but be sure to replace it with the proper device and partition.
   sudo mkdir /mnt/msata
sudo mount /dev/sdX1 /mnt/msata
5. Use chroot to change the apparent root directory to /mnt/msata
   cd /mnt/msata && sudo chroot .
6. Add the parameter to the boot file. This step may vary based on your operating system. If this fix does not work, it's most likely because grub is managed differently for your system. This example is assuming Debian.
   echo 'hpet=disable' | sudo tee -a /etc/default/grub
7. Update the GRUB bootloader with your new configuration
   sudo update-grub
8. Exit chroot and unmount the device
   exit
   sync
   sudo umount /mnt/msata
   sudo rmdir /mnt/msata
9. Remove the mSATA from the system and reassemble your Vault. Test your results by booting the machine.

Conclusion

We are working to have this issue fixed in a future coreboot release. When the new coreboot version is available, you may refer to our Knowledgebase again for instructions on how to flash the new BIOS. We do not anticipate needing to "undo" the fix explained in this article after flashing the new coreboot version.