I wanted to see what would happen if an AI coding agent had to work on a real embedded device, not only on the source code.
On a normal project, the loop is easy. I describe a change, the agent edits the code, runs a test, reads the result, and tries again. The first version is usually wrong in some way, but that is fine because the next iteration is cheap.
Embedded development is different. The code may compile correctly and still fail on the board. The device may not boot, the network may not come up, or the container may start with a different environment than the one used during the build. To find out, somebody has to flash the device, power it on, and read the serial console.
That somebody was usually me.
So I built a small lab around a Raspberry Pi Compute Module 5. The purpose was not to make a better flashing station. The purpose was to give the agent a feedback loop that included the real hardware.
The loop
The setup has a CM5 on its IO board and an Orange Pi 3 LTS connected to it. The Orange Pi controls the power button and the boot mode pin through two small transistor circuits. It also connects to the CM5 USB port and the serial console.
This is what the setup looked like while I was putting it together. It is not a pretty lab, but it is enough to let the agent work with the real board:
The software side is cm5lab. It turns the physical actions into commands. For example, the agent can ask the lab to power cycle the board, put it in USB boot mode, or flash a new image. The important command is the flash command because it does all the steps together: boot the CM5 through USB, expose the eMMC, write the image, and restart the board from the new image.
The board is still a board on my desk. There are wires and a separate power supply. But the agent does not need to know that. It sees commands and output.
For a change to the operating system, the agent runs the meta-pantavisor build and the flash-cm5lab script. For a change to an application, it builds a container export and posts a new Pantavisor revision. In both cases, it gets a running device at the end instead of only a successful build.
The loop is roughly this:
change code
↓
build an image or application export
↓
flash or post a new revision
↓
boot the device
↓
read serial output, logs, and device state
↓
change code again
The output from the board is what makes this useful. The agent can use the device progress endpoint, inspect the running containers, and read the serial console with commands such as:
curl -s http://$DEVICE_IP:12368/cgi-bin/steps/current/progress | jq
pvr device tty ps
pvr device tty log myapp 50
pvr device tty run "pvcontrol steps get current"
I did not want the agent to receive a screenshot or a vague description from me. I wanted it to receive the same kind of output it gets from a test process.
The first sessions were not very impressive
At the beginning, the agent did what I expected. It made a reasonable change, built the application, and posted a revision. The revision did not work.
The source code looked fine. The container also ran fine on the development machine. On the CM5, the application started and then stopped because one of the assumptions made by the init script was not true on the device.
This is where the hardware loop changed the session. The agent could read the container log, see the error, modify the init script, build another export, and post it again. It did not need me to translate “the board failed” into a new debugging task.
Sometimes the change required a cold boot. Sometimes it required a full image reflash because the state left by the previous test was not trustworthy. The agent could make that decision from the serial output and the device status. A reflash was slower, but it was still just another command.
I have seen it go through this process several times in one session. It did not get every fix right on the first attempt. That was not the interesting part. The useful part was that every failed attempt produced information for the next one.
Without the lab, the session would have ended with a message like this:
The change is ready. Please flash the board and tell me what happens.
That is code generation, not a development loop.
A clean board found a WiFi bug
The BLE WiFi provisioning service was one of the first real features we developed with this setup.
The bug was not visible on the development machine. It was also not visible on a board that had connected to WiFi before. The CLI was asking D-Bus for the wrong ConnMan bus name, the request failed, and the error was swallowed. A device with a saved WiFi profile connected anyway, which made the provisioning command look correct.
A freshly flashed board did not have that profile. The feedback loop made the actual test simple:
flash a clean image
boot the CM5
provision WiFi over BLE
read the serial log
check whether wlan0 has an address
The agent could repeat this after each change. The clean flash was important because it turned a rare precondition into a normal test step.
The same process found more problems in the provisioning flow. ConnMan did not scan while the access point was running, so the network list was empty until the scan cache was initialized. Saved connections also used a randomized MAC address and did not always reconnect after a reboot.
These are the kind of bugs that are difficult to find by looking at the code. They depend on the state of the device and on the order in which services start. The board had to be part of the test.
The hub agent showed another kind of failure
The pvsm hub agent was a different case. Here the problem was not only between the application and the operating system. It was between the device and the cloud.
The agent communicates with the hub using MQTT and REST. It handles OTA updates, metadata synchronization, and log forwarding. To debug it, we needed to observe both sides: the device through serial and local HTTP endpoints, and the hub through its API.
The first real hub-managed device found several problems in one session. The agent was parsing the Pantavisor configuration incorrectly and registering the device against the production host instead of the configured host. The baseline upload kept retrying when the hub returned 409 for objects it already had. User metadata never reached the device because the applier had not been connected to the rest of the system.
A later field failure was harder to understand. The device stayed in a read-timeout loop, but the log did not say which part of the connection had failed. We changed the failure path to report the connection stage and to fall back to REST polling when the normal connection was unavailable.
Again, the agent did not magically solve the problem. It needed a useful observation. Once the logs showed where the timeout happened, the next code change was much less speculative.
What changed for development
The main difference is not that the agent can now flash a CM5. The main difference is that the agent can test assumptions against the environment where the feature will actually run.
For example, an agent can look at a Dockerfile and decide that a command should work. It cannot know from the Dockerfile that the container will receive a different device filesystem, that a service will start later than expected, or that the board will reboot before the network is ready. The feedback loop exposes those differences quickly.
It also makes failure less expensive. A bad revision can be rolled back. A bad image can be flashed again. A broken network connection does not necessarily stop the investigation because the serial console is still available.
The loop is not fast compared with a unit test. Flashing an image can take several minutes. But it is much faster than waiting for a person to notice the failure, walk to the bench, connect a cable, flash the device, and describe the result back to the agent.
This is also why the hardware control and the observation are equally important. Automating the flash without automating the logs only gives the agent a remote button. Automating the logs without being able to reset the board leaves it stuck when the test fails. Both sides are needed.
What I learned
The first lesson was that the agent does not need a perfect plan. It needs a short feedback loop.
The second lesson was that embedded work has more state than the source code shows. A clean image, a warm reboot, a missing WiFi profile, a changed serial device, and a failed revision are all different starting points. If the agent cannot see that state, it will make confident guesses.
The third lesson was that the useful documentation is often a list of small operational facts. The skill file is not a large framework. It is a few commands and the mistakes we have already made.
This setup is useful for normal hardware CI too. The same properties are required: the board must be controllable, its state must be observable, and a failed update must be recoverable. The AI agent simply makes the need more obvious because it cannot walk over to the bench when the test fails.
I still have a CM5, an Orange Pi, a breadboard, and a handful of cables on the desk. But the board is now part of the development loop instead of being the place where the loop stops.
The agent changes the code, runs it on the device, reads what happened, and tries again. That is the part that makes developing embedded features with an AI agent useful.
