Broadwell, Instability, and Microcode

September 25, 2015

This issue has since been resolved by the updated microcode packages available in nearly all Linux distributions, and by the updated UEFI packages provided by most motherboard vendors.

I recently built a new computer with the i5-5675C “Broadwell” processor. It’s one of the only two desktop Broadwell processors Intel has released. Due to issues with the 14nm process, it seems that Broadwell’s release was pushed too close to Skylake, which is already shipping.

This processor is a rather odd choice, especially since it’s currently retailing well above the MSRP, but I picked it for the Iris Pro 6200 graphics, which should be fast enough to let me avoid a discrete GPU, and all the driver-related headaches that go along with that on Linux. Unfortunately, I didn’t realize until I started trying to use it that there’s been a whole slew of Linux stability issues with it and its sibling, the i7-5775C.

Personally, I was a bit suspicious about the microcode, since the last microcode update package Intel released for Linux is from January, while Windows last got an update in June.

If you don’t know what microcode is, it’s essentially just a binary firmware update for your processor. Modern CPUs are incredibly complex, and often ship with bugs, to the point that Intel has had to recall processors. With microcode updates, the manufacturer (Intel, AMD, etc) can turn off broken functionality, or improve performance by tweaking the hardware’s configuration or how instructions are processed.

Naturally, I wanted to see if I could extract the Windows microcode and use it on Linux. I got an ISO of Windows 7 for free though my University with DreamSpark, and proceeded to install it in KVM. I then installed the June 2015 microcode update while running procmon to figure out that the update overwrites mcupdate_GenuineIntel.dll.

Armed with the source code to iucode-tool, I managed to quickly hack together a script in Python that can extract anything that looks like microcode from a binary file:

#!/usr/bin/env python2
# License: GPLv3+

import struct
import sys
import itertools

# struct intel_ucode_v1_hdr { /* 48 bytes */
#     uint32_t        hdrver; /* must be 0x1 */
#     int32_t         rev;    /* yes, it IS signed */
#     uint32_t        date;   /* packed BCD, MMDDYYYY */
#     uint32_t        sig;
#     uint32_t        cksum;
#     uint32_t        ldrver;
#     uint32_t        pf_mask;
#     uint32_t        datasize;  /* 0 means 2000 */
#     uint32_t        totalsize; /* 0 means 2048 */
#     uint32_t        reserved[3];
# } __attribute__((packed));

STRUCT_FMT = '<IiIIIIIIIIII'
STRUCT_SIZE = struct.calcsize(STRUCT_FMT)

with open(sys.argv[1], 'rb') as f:
    for i in itertools.count():
        f.seek(i)
        header_bytes = f.read(STRUCT_SIZE)
        if len(header_bytes) < STRUCT_SIZE: # EOF
            break
        hdrvr, rev, date, sig, \
            cksum, ldrvr, pf_mask, \
            datasize, totalsize, \
            reserved0, reserved1, reserved2 = \
            struct.unpack(STRUCT_FMT, header_bytes)

        # validate the header
        if hdrvr != 1:
            continue
        # newer microcode updates include a size field, whereas older containers
        # set it at 0 and are exactly 2048 bytes long
        totalsize = totalsize or 2048
        if not 1024 * 2 <= totalsize < 1024 * 50:
            # microcode updates should probably be 5KB to 50KB
            continue
        if datasize + STRUCT_SIZE > totalsize:
            continue

        body_nbytes = totalsize - len(header_bytes)
        body_bytes = f.read(body_nbytes)
        if len(body_bytes) < body_nbytes:
            continue # partial read
        with open('microcode/%s.bin' % i, 'wb') as out:
            out.write(header_bytes)
            out.write(body_bytes)

Of course, it’s possible to write a cleaner solution, or one with less false-positives than this, but when run against mcupdate_GenuineIntel.dll, I managed to extract a bunch of blobs, some of which had matching checksums with known Linux microcode updates!

While they’re undocumented, it’s known that the microcode updates are validated before being applied by the processor, and iucode-tool does a bunch of its own sanity checks, so it’s pretty safe to blindly apply all of them, including the false positives:

$ for i in *.bin; do echo $i; sudo iucode-tool $i; done

However, checking the microcode version before and after via /proc/cpuinfo, I saw no changes — still version 0x10. I might already have the latest microcode if Intel hasn’t shipped any updates for this processor, if the January microcode package already included them, or if my motherboard’s UEFI installed it for me.

At least for now, I guess I’m stuck with running with the kernel flags:

processor.max_cstate=0 intel_idle.max_cstate=0 idle=poll

and dealing with the occasional kernel panic.

Update

Since originally writing this article, I’ve improved my script slightly (updated in-place) to generate less false positives. I’ve also discovered that certain UEFI updates (but not any Windows updates) include a microcode update that does solve this problem!

I’ve published an installer for these updates on GitHub.