Inherited a Vendor BSP? Measure These Five Things First

When you inherit a vendor BSP, the useful first move is not to read the code but to measure the distance between what the vendor shipped and what mainline contains, using tools the kernel already gives you. Five measurements — running configuration, mainline delta, externally built modules, the device tree the board actually booted, and the Yocto layer map — take about two days and produce an Inherited BSP Baseline Report, the handover document nobody left you.

At some point in an embedded Linux career, somebody hands you a board and a git repository and leaves the company. The repository builds. The board boots. You now own an inherited vendor BSP you did not write, and within a week somebody will ask how long it would take to move it to a newer kernel.

You cannot answer that yet, but producing a number you can defend is a mechanical exercise: five measurements, run in order, before any other work on the platform. One boundary up front: this maps the Linux side of the platform. U-Boot, TF-A and binary firmware carry their own deltas and are not covered here.

What an unmeasured vendor BSP costs

Here is what happens when nobody runs those five measurements. An engineer takes over an inherited vendor BSP on a 6.6 kernel, reads release notes promising "board support and stability fixes", glances at the driver directories, and estimates six weeks to reach a newer longterm kernel. The estimate goes on a roadmap. Eight months later the work is not finished — not through incompetence, but because the tree held around four hundred non-merge commits, a few dozen touching core memory management rather than board files, and eleven modules built from a source tree nobody could locate. None of that was in the release notes. All of it was discoverable in two days.

The damage is worse than one bad estimate, because the gap repeats. Each vendor release rebases and the delta is never measured, so the debt stays invisible until somebody tries to pay it. Then the SoC vendor ends support, and the number nobody measured becomes the number the schedule depends on.

The five measurements:

1. Recover the running kernel configuration

Be precise about what you are collecting: not the defconfig in the repository, but the configuration of the kernel binary on the board. In an inherited vendor BSP those disagree more often than they agree, because fragments, recipe appends and local builds all mutate the config on its way to the image.

The kernel can carry its own configuration. CONFIG_IKCONFIG embeds the .config into the image; CONFIG_IKCONFIG_PROC exposes it at /proc/config.gz. The distinction catches people: the first can be a module, the second cannot. Seeing CONFIG_IKCONFIG=m does not mean /proc/config.gz will be there.

root@imx8mp-lpddr4-evk:~# zcat /proc/config.gz > /tmp/running.config
root@imx8mp-lpddr4-evk:~# wc -l /tmp/running.config
6314 /tmp/running.config

If it is missing, the configuration may still be embedded. Point the kernel's own extraction script at the binary the board boots:

raghu@techveda.org:~$ scripts/extract-ikconfig /path/to/vmlinux > running.config

Reduce it before comparing. A raw .config holds thousands of symbols nobody chose — they were implied by others — so comparing two raw configs produces a diff nobody reads. make savedefconfig writes a minimised defconfig stripped of everything implied elsewhere, leaving the decisions a human actually made.

raghu@techveda.org:~$ cp running.config .config
raghu@techveda.org:~$ make savedefconfig
raghu@techveda.org:~$ scripts/diffconfig arch/arm64/configs/vendor_defconfig defconfig

The output is a short list rather than a wall of text:

-DEBUG_FS y
 PREEMPT n -> y
+VENDOR_THERMAL_HACK y
+MODULE_SIG_FORCE y

Four lines of real information. Somebody turned off debugfs, switched the preemption model, and added a symbol mainline does not have. Treat that last line as a strong investigation signal rather than proof: it may be a vendor Kconfig addition, a stale fragment no longer consumed by the active build path, or an option renamed upstream. Trace it through Kconfig and the Yocto configuration path before calling it live out-of-tree functionality.

Commit the minimised defconfig into your own repository with today's date in the filename. It is the first written record of what the board runs, and every future argument about "did we change that?" is settled by diffing against it.

2. Measure the distance from mainline

Now size the delta. Add a mainline remote and count commits added on top of the vendor's base:

raghu@techveda.org:~$ git remote add mainline https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
raghu@techveda.org:~$ git fetch mainline --tags
raghu@techveda.org:~$ git log --oneline --no-merges $(git merge-base v6.6 vendor/release)..vendor/release | wc -l

That count is the headline number and on its own close to meaningless. What decides the difficulty of a kernel bump is not how many patches the vendor wrote, but which directories they wrote them in. Ask for that directly:

raghu@techveda.org:~$ git log --format= --name-only $(git merge-base v6.6 vendor/release)..vendor/release | sort | uniq -c | sort -rn | head -20

Read the result as a shape, not a score. Weight sitting in arch/arm64/boot/dts/, drivers/soc/ and a few drivers means an ordinary vendor BSP: large but predictable, because device tree files usually rebase cleanly. Commits in mm/, kernel/sched/ or the DMA core change the character of the job, because those subsystems move internally between releases. As a heuristic rather than a rule, a large device tree delta is often more predictable to rebase than a much smaller delta in core code. Location and dependency depth matter more than commit count.

One limit. This assumes the vendor tree shares history with mainline. Some vendors ship a squashed import with no common ancestor, and git merge-base fails. Then compare trees rather than history — git diff --stat v6.6 vendor/release against the closest tag. Treat that candidate tag as a hypothesis, not a fact: validate it against version metadata, a representative subsystem diff and the vendor's own release context before any estimate rests on it.

3. Inventory the externally built modules

Commits are the visible part of an inherited vendor BSP. The expensive part is usually code that was never in a tree at all, and the kernel gives one signal for it: loading an externally built module sets a taint flag. Bit 12, shown as O, has raw value 4096. Read it for exactly what it is — at least one externally built module has been loaded. It does not enumerate them, say where the code came from, or assign ownership.

root@imx8mp-lpddr4-evk:~# cat /proc/sys/kernel/tainted
4096

To find them, check each loaded module for the intree marker kbuild stamps onto modules built inside the tree. In-tree modules report intree: Y; externally built ones do not carry the field at all.

root@imx8mp-lpddr4-evk:~# for m in $(lsmod | tail -n +2 | cut -d' ' -f1); do echo "$m $(modinfo -F intree $m)"; done

A module without a Y is externally built relative to the running kernel tree. That is all the flag establishes. It does not follow that the vendor wrote it — it may belong to your own product team, a supplier, or an integration nobody documented. Record five things for each: source repository, build path, version, licence and maintainer. However ownership falls, no kernel upgrade carries these modules for you, and because they are invisible in the git history you just counted, they are the part of the estimate engineers consistently forget.

If you cannot say where a module's source lives and who last touched it, you have found the largest risk in the platform — in your first week.

4. Read the device tree the board actually booted

The device tree source in the repository is what the build produced, not necessarily what the kernel saw. The bootloader modifies the blob in transit, commonly inserting memory node sizes, MAC addresses and overlays. Read the real one back out of the running kernel:

root@imx8mp-lpddr4-evk:~# dtc -I dtb -O dts -o /tmp/booted.dts /sys/firmware/fdt

Not every platform exposes /sys/firmware/fdt. Where it is absent, inspect the live tree under /sys/firmware/devicetree/base or /proc/device-tree, and compare semantics rather than expecting decompiled output to match source formatting.

Diff the result against the source. The differences are the bootloader's fixups, and knowing them saves you from a class of confusing bugs where the property you see in the repository is not the property the driver read.

Then ask which devices are still waiting. Deferred probe retries a device whose dependency was not ready at that point in boot, and the driver core exposes the queue:

root@imx8mp-lpddr4-evk:~# cat /sys/kernel/debug/devices_deferred
30a20000.i2c:pmic@25
32c00000.bus:mipi_dsi@32e60000

Each line names a device whose probe was deferred because a dependency — commonly a regulator or clock provider — was not ready when the driver core tried. Some bind later once the supplier appears; others never do. Read the list as dependency chains to investigate, then confirm final binding state separately. A non-empty list is normal, and often the first clear statement of which hardware is described but not working.

5. Map the Yocto layers

If the vendor BSP ships as Yocto metadata, three commands give you the build-side map:

raghu@techveda.org:~$ bitbake-layers show-layers
raghu@techveda.org:~$ bitbake-layers show-recipes
raghu@techveda.org:~$ bitbake-layers show-appends

show-layers lists layers with priorities, which decides who wins when two provide the same recipe. show-appends repays the effort fastest: it lists every .bbappend and the recipe it modifies, answering "who edited this, and where does that edit live". An append in a layer nobody remembers adding is the most common source of behaviour that appears to come from nowhere.

Then pin the kernel. bitbake -e virtual/kernel prints the recipe's expanded variables; you want SRC_URI, SRCREV and KERNEL_DEVICETREE. Those name the tree, commit and device tree files the image is really built from, frequently not what the documentation claims.

What this method does not tell you

A map that overstates its coverage is worse than no map.

• Count is not difficulty. The directory breakdown gives the shape; only reading a sample tells you whether patches are mechanical or deep. Read twenty at random before committing to a number.
• The config comparison assumes one kernel version. Across versions, renamed symbols produce noise — use make listnewconfig against the newer tree instead.
• Deferred probe output needs debugfs, which production vendor configurations often disable.
• These commands are verified against documentation, not against every vendor tree. Vendors modify kbuild and strip instrumentation. Where a step fails, the failure itself tells you how far this tree has drifted.

The deliverable: an Inherited BSP Baseline Report

Run the five measurements in order, then write the result up as one document. Name it, because a named deliverable survives handovers in a way scattered notes do not: the Inherited BSP Baseline Report. One page, not a wiki tree. It opens with the board's identity, carries the five measurements in the order you ran them, and closes with what you could not determine:

1. Board and SoC identity, with bootloader, firmware and kernel versions.
2. The minimised runtime config snapshot, committed with today's date.
3. The mainline base hypothesis and the delta by directory, with one sentence on the shape.
4. External module inventory: repository, build path, version, licence and maintainer for each.
5. Booted FDT versus source DTS differences, and the deferred-device list with final binding state.
6. The Yocto layer, append and kernel recipe map, with SRC_URI, SRCREV and KERNEL_DEVICETREE.
7. Risks, unknowns and next actions — including what you could not determine.

That last line matters most. A baseline that admits what it could not establish is trustworthy; one that quietly omits the gaps will be used to justify an estimate it cannot support.

Do this before you are asked for an estimate, not after. An engineer who answers "how far are we from mainline" with a measured number in week two is treated differently for the rest of the project.

Key takeaways

• The first job on an inherited vendor BSP is measurement, not reading code.
• Compare minimised defconfigs; patch location beats patch count.
• Externally built modules are invisible in git history and no upgrade carries them.
• The booted device tree and the Yocto layer map both differ from what the repository shows.
• Record what you could not establish. A baseline hiding its gaps will be misused.

Frequently asked questions

Why compare minimised defconfigs instead of the raw .config files?
A raw .config contains thousands of symbols that were implied by other symbols rather than chosen by anyone. make savedefconfig strips those, so the resulting comparison shows only the decisions a person actually made.

What if git merge-base fails on the vendor tree?
That usually means a squashed import with no shared history. Compare trees directly with git diff --stat against the closest matching mainline tag, treating that tag as a hypothesis to validate rather than a fact.

How do I tell which loaded modules are out-of-tree?
Modules built inside the kernel tree carry an intree marker that modinfo -F intree reports as Y. Modules built outside the tree do not carry the field, and loading one sets kernel taint bit 12, which appears as raw value 4096 in /proc/sys/kernel/tainted.

Further reading

• Configuration targets and editors — Linux kernel documentation (covers listnewconfig and scripts/diffconfig)
• Tainted kernels — Linux kernel documentation (taint bit 12, externally-built modules)
• Kbuild — Linux kernel documentation
• Understanding and Creating Layers — Yocto Project Development Tasks Manual
• Yocto Project Board Support Package Developer's Guide
• proc_config.gz(5) — Linux manual page

I teach Linux kernel and embedded Linux engineering at TECH VEDA. If your team is taking over a vendor BSP and wants to build this baseline properly, our Embedded Linux and Yocto programme covers the layer and kernel-recipe side in depth.