UBI on Raw NAND: The Layer That Hides Bad Blocks and Wear

UBI is the volume-management layer between a raw NAND partition and a filesystem such as UBIFS. It hands the layer above it eraseblocks that never go bad and never wear unevenly, and charges for that: one or two NAND pages of every eraseblock for headers, about 2% of the chip reserved for bad blocks, and four further eraseblocks. This article traces those costs, and shows the raw NAND flashing mistake that becomes read errors months after a board ships.

Most engineers meet UBI as three commands in a flashing script somebody else wrote. The script works, the board boots, and nobody looks again until a unit returns from the field with read errors, or the rootfs turns out to be megabytes smaller than the datasheet suggested. Both follow from how UBI behaves on raw NAND.

Why raw NAND cannot be treated as a disk

The kernel's UBIFS documentation states the difference plainly. A block device supports read-a-block and write-a-block; blocks are rewritten in place, do not wear out and do not go bad, because the drive substitutes spares itself. An MTD device supports three operations instead: read at an offset in an eraseblock, write at an offset in an eraseblock, and erase a whole eraseblock. Nothing is rewritten until the eraseblock holding it has been erased, eraseblocks wear out after a bounded number of cycles, and on raw NAND they also go bad.

One mistake follows immediately: putting ext4 on /dev/mtdblock2. It appears to work, because mtdblock emulates a block device by reading an eraseblock, modifying it in memory and writing it back. What it does not emulate is wear levelling or bad-block handling, so every metadata update lands on the same few eraseblocks. The failure is a device that passes every lab test and dies in the field.

UBI removes both problems. The MTD project describes it as a volume management system for raw flash, comparable to LVM except that it maps logical eraseblocks onto physical ones and adds wear levelling and error handling. It is not a filesystem, and explicitly not a Flash Translation Layer.

UBIFS  (or squashfs on ubiblock, read-only)
  |
UBI    volumes: rootfs, data, ...      drivers/mtd/ubi/
  |
MTD    /dev/mtd2, eraseblocks + pages  drivers/mtd/
  |
raw NAND chip

The two layers are configured separately and can disagree. UBI is told the physical eraseblock size and minimum I/O size of the raw NAND; UBIFS is told the logical eraseblock size UBI presents. Get the second wrong and the filesystem will not mount, reporting a size mismatch rather than the mistake behind it.

The two headers, and why your eraseblocks shrank

UBI writes two 64-byte headers at the start of every physical eraseblock (PEB) that is not marked bad. The erase counter header, or EC header, records how many times that eraseblock has been erased. The volume identifier header, or VID header, records which volume it belongs to and which logical eraseblock (LEB) of that volume it holds. Both carry a CRC-32.

Those headers are the whole basis of the design. The EC header lets UBI compare wear across the chip; the VID header means the map need not live anywhere else, because UBI rebuilds it by reading headers. That keeps the on-flash format simple, and it is also why attach time grows with flash size.

The headers occupy space, so a logical eraseblock is smaller than a physical one. The EC header sits at offset 0; the VID header goes at the next minimum I/O unit or sub-page. On raw NAND with sub-page write support, UBI puts both in different sub-pages of the same page, costing one page. Without it the VID header goes in the second page, costing two. On 128 KiB eraseblocks with 2048-byte pages that gives the numbers you will actually see:

Flash	Header cost per PEB	LEB size
NAND, 512-byte sub-pages	2048 bytes (one page)	129024 bytes (126 KiB)
NAND, no sub-page writes	4096 bytes (two pages)	126976 bytes (124 KiB)
NOR, 1-byte minimum I/O	128 bytes	130944 bytes

If the value passed to mkfs.ubifs -e does not match one of these, the image will not mount on the volume it was built for. The gap between 126 KiB and 124 KiB is not a rounding choice; it is whether the controller can write sub-pages. Some controllers read sub-pages but cannot write them, and UBI must then use the two-page layout even though the datasheet advertises sub-pages.

The eraseblocks you never get to use

UBI also withholds whole eraseblocks. Four are fixed: two hold the copies of the volume table, one is reserved for wear levelling, one for the atomic LEB change operation. The rest covers bad blocks, and its size is a Kconfig option. Current mainline drivers/mtd/ubi/Kconfig reads:

config MTD_UBI_BEB_LIMIT
        int "Maximum expected bad eraseblock count per 1024 eraseblocks"
        default 20
        range 0 768

The help text derives 20 from the minimum and maximum valid block counts a NAND datasheet gives for the device's lifetime, which works out at 20 per 1024 for most parts — about 1.9% of the chip. The word chip is doing important work there, and it catches people. The reserve is counted against the whole device, not the partition being attached, because bad blocks are not distributed evenly and one partition could inherit more than its share. So two UBI devices on one chip each reserve the full amount.

Work it through on a 512 MiB SLC part with 128 KiB eraseblocks, 2048-byte pages and 512-byte sub-pages. The chip has 4096 eraseblocks, so the reserve is 20 times 4096 divided by 1024, which is 80 PEBs. Attach one 256 MiB partition of 2048 PEBs and the usable count is 2048 minus 80 minus 4, giving 1964 LEBs. At an LEB size of 129024 bytes that is 253,403,136 bytes — about 241.7 MiB from a nominal 256 MiB, a loss near 14.3 MiB or 5.6%.

Split that chip into two UBI devices and each reserves 80 PEBs, so 160 go to bad-block handling instead of 80: 10 MiB for no benefit. The MTD documentation states the conclusion directly. On one raw NAND chip use one UBI device with several volumes. Volumes are free; devices are not. The same arithmetic prices a controller that cannot write sub-pages, at a further 3.8 MiB here, or roughly 16 MiB per gigabyte.

Wear levelling and scrubbing on raw NAND

Wear levelling works because every PEB carries its erase count in the EC header. UBI rebuilds the erase counter table in RAM at attach time and uses it to decide when data should move. The trigger is a configured threshold:

config MTD_UBI_WL_THRESHOLD
        int "UBI wear-leveling threshold"
        default 4096
        range 2 65536

The parameter is the maximum permitted difference between the highest and lowest erase counter on the device. When the gap is exceeded, UBI moves data out of eraseblocks with low erase counters into ones with high counters. That looks backwards at first. A low erase count means the block holds data that never changes, so moving that cold data into a worn block frees the fresh block for write traffic and parks the worn block under data that will not erase it again.

The default of 4096 suits SLC raw NAND and NOR, with eraseblock lifetimes of 100,000 cycles or more. The help text is explicit that MLC NAND, typically under 10,000 cycles, needs a much smaller threshold, suggesting 128 or 256. If your product ships MLC and nobody revisited this symbol, it is running a policy meant for a different class of flash.

Scrubbing is the other half of the job. Raw NAND accumulates bit flips, which ECC corrects on read. A corrected flip is not an error but it does signal that the eraseblock is degrading, and UBI treats it as a reason to copy that PEB elsewhere and recycle the original. This happens in the background, hidden from the layers above, which is why UBIFS needs no bit-rot handling of its own. That is as far as the documentation goes: I did not read drivers/mtd/ubi/io.c for this article, so I am not quoting internal return codes for the scrub path.

Bad blocks use the same idea with a different trigger. If a write fails, UBI moves the data away and tortures the eraseblock: erase it, confirm it reads back as all 0xFF, then write and verify a series of test patterns. A block that survives returns to service; one that fails, or shows a bit flip during the test, is marked bad. If an erase fails with EIO there is no test — the block is marked bad immediately.

Why attach takes as long as it does

Because the erase counters and the map live only in the headers, UBI must read those headers from every eraseblock to rebuild its tables in RAM. Attaching costs one read per PEB plus a CRC-32 per header, so attach time scales linearly with flash size. The MTD documentation gives measured figures: a 256 MiB OneNAND attaches in under a second, a 1 GiB NAND in about two, and a full scan of a 4 GiB NAND takes several.

Several seconds is a large share of a boot budget, which is why fastmap exists. It stores a checkpoint so attach reads the map instead of rebuilding it, and in mainline it is still experimental and off by default. Its anchor eraseblock must sit within the first 64 PEBs, it keeps a free-PEB pool of 5% of the total, and if the fastmap is corrupt UBI falls back to a full scan — so a failure costs boot time, not data. On a 128 MiB part it solves no problem you have.

The flashing mistake that appears months later

This failure is worth understanding, because it passes every test at the factory. The MTD documentation sets it out step by step. You erase the flash, so every byte is 0xFF. You write a UBI image with a general-purpose tool such as nandwrite. Some eraseblocks in that image end with empty pages, and the tool writes those too, filling them with 0xFF. That makes the controller calculate an ECC code for a page of 0xFF bytes and store it in the out-of-band area — and on many hardware ECC engines that code is not itself all 0xFF. Later, UBIFS uses one of those pages. It writes real data, producing a new ECC code written on top of the old one. Because the page was never erased between the two writes, the out-of-band area now holds a corrupted combination. Nothing fails then. It fails when the page is next read, and the read returns -EBADMSG, which is -74.

The fix is to flash with a tool that understands the format. As the documentation puts it, ubiformat ensures every page is written once and only once after erasure, whereas nandwrite writes some pages twice, once by the flasher and once by UBIFS. It also preserves the existing erase counters, so reflashing a returned device does not discard its wear history.

When the flashing tool cannot be changed — a fixed factory programmer, for instance — the fallback is to set the free-space fixup flag when building the image, with mkfs.ubifs -F. On the first mount the filesystem finds the affected free space and re-erases it before use. It works, and it makes that first boot noticeably slower.

Which image you flash depends on the layer. mkfs.ubifs makes a UBIFS image, which goes into an existing volume with ubiupdatevol. ubinize wraps those into a UBI image, which is what ubiformat writes to the MTD device. Confuse the two and the device will not attach.

Checking raw NAND and UBI on your own board

All of this is observable. On the host, the minimum I/O size and the LEB size are arguments you must get right:

raghu@techveda.org:~$ mkfs.ubifs -q -r rootfs-dir -m 2048 -e 129024 -c 2047 -o ubifs.img
raghu@techveda.org:~$ ubinize -o ubi.img -m 2048 -p 128KiB -s 512 ubinize.cfg

The numbers for -m, -p and -s come from the board, so read the geometry rather than trusting a script that hardcodes them:

root@imx6ull-evk:~# mtdinfo /dev/mtd0 -u
mtd0
Name:                           nand
Type:                           nand
Eraseblock size:                131072 bytes, 128.0 KiB
Amount of eraseblocks:          4096 (536870912 bytes, 512.0 MiB)
Minimum input/output unit size: 2048 bytes
Sub-page size:                  512 bytes
Bad blocks are allowed:         true

Then attach and read back what UBI made of it. The logical eraseblock size printed here is the value mkfs.ubifs -e needs:

root@imx6ull-evk:~# ubiattach --mtdn 0
root@imx6ull-evk:~# ubinfo -a
ubi0
Volumes count:                           1
Logical eraseblock size:                 129024 bytes, 126.0 KiB
Total amount of logical eraseblocks:     4012
Amount of available logical eraseblocks: 0 (0 bytes)
Maximum count of volumes                 128
Count of bad physical eraseblocks:       6
Count of reserved physical eraseblocks:  80

Read those last two lines together. Six eraseblocks are bad; eighty are reserved. The reservation is not a measurement of damage, it is a budget for damage that has not happened yet, and a fleet whose bad count is climbing toward it is worth investigating.

root@imx6ull-evk:~# zcat /proc/config.gz | grep MTD_UBI
CONFIG_MTD_UBI=y
CONFIG_MTD_UBI_WL_THRESHOLD=4096
CONFIG_MTD_UBI_BEB_LIMIT=20
CONFIG_MTD_UBI_BLOCK=y
# CONFIG_MTD_UBI_FASTMAP is not set
# CONFIG_MTD_UBI_GLUEBI is not set

Three checks are worth running on any product that uses raw NAND. If the flash is MLC and the threshold is still 4096, that is a real problem, not a cosmetic one. If the chip carries more than one UBI device, each pays the full bad-block reserve. And if the factory process uses anything other than ubiformat, change it or build images with mkfs.ubifs -F.

Reading flash geometry off the board and matching the image to it is part of the bring-up work covered in TECH VEDA's Embedded Linux with Yocto training.

Where this does not apply

The MTD project puts the following at the top of its UBI page. UBI works with bare flash. It does not work with managed consumer flash: eMMC, SD, microSD, CompactFlash or USB drives. Those carry a Flash Translation Layer in the controller that already performs wear levelling and bad-block substitution and presents a block device. Stacking UBI on one would be two remapping layers competing for the same job. On managed flash, use a block filesystem and choose it on its own merits — a decision covered in choosing a root filesystem format for embedded Linux.

In the other direction, CONFIG_MTD_UBI_BLOCK exposes a read-only block device over a UBI volume — how a squashfs rootfs runs on raw NAND with UBI still handling bit flips and bad blocks underneath.

Key takeaways

UBI stores the logical-to-physical map, and a per-eraseblock erase count, in two 64-byte headers in every good eraseblock.
Those headers cost one page per eraseblock with sub-page writes and two without: a 126 KiB or 124 KiB LEB on a 128 KiB part.
UBI reserves 20 eraseblocks per 1024 for bad blocks, counted over the whole chip, so use one UBI device with several volumes.
CONFIG_MTD_UBI_WL_THRESHOLD defaults to 4096, which suits SLC raw NAND and NOR. MLC needs a far lower value and nobody adjusts it for you.
Flash UBI images with ubiformat. A general-purpose flasher writes empty pages twice and produces ECC errors long after the device ships.
None of this applies to eMMC or SD, which have their own translation layer.

Frequently asked questions

Why is my UBI volume smaller than the MTD partition?
Three costs are subtracted. Each good eraseblock loses one or two NAND pages to the UBI headers, four eraseblocks are reserved for the volume table, wear levelling and the atomic LEB change operation, and 20 eraseblocks per 1024 of the whole chip are held back for bad blocks. On a 256 MiB partition of a 512 MiB chip that is roughly 14 MiB.

Can I use UBI or UBIFS on eMMC or an SD card?
No. UBI works with raw flash only. Managed flash such as eMMC, SD and USB drives contains its own Flash Translation Layer that already handles wear levelling and bad blocks, and does not expose the eraseblock geometry UBI needs.

Why must I use ubiformat instead of nandwrite?
A general-purpose flasher writes the empty NAND pages at the end of eraseblocks, causing an ECC code to be stored for a page of 0xFF bytes. When the filesystem later writes real data there, a second ECC code is written over the first, and the page returns -EBADMSG when read. Using ubiformat ensures every page is written once after erasure, and it preserves the erase counters.