If you're trying to break into physical AI or robotics, you've probably already noticed that the field has a specific problem: everyone's resume looks the same.
ROS 2. Python. Computer vision. Familiar with deep learning.
None of that tells anyone anything useful. What actually matters in this field is whether you can take a pile of hardware, write the software to make it work, and ship something that runs reliably in the real world. That's a very different skill from knowing the right keywords.
This post is a list of projects I think are worth building if you're serious about this space. Not because completing a checklist will land you a job, but because these projects cover the gaps I see most often between what people say they can do and what they can actually demonstrate.
Pick one. Build it properly. Document it honestly. That's worth more than ten resume lines.
If any of these overlap with what you're already working on, drop a comment or reach out — always happy to compare notes or collaborate.
The Hardware Stack These Projects Assume
You don't need an expensive lab. A serious but accessible setup looks something like this:
- An edge compute module with a capable GPU (8GB+ recommended)
- Differential drive rover chassis with encoder motors
- 2D LiDAR sensor
- 6-DOF robotic arm
- Two matched cameras for stereo vision + one wide-angle camera for scene coverage
- An ARM-based microcontroller
Total cost: under $1,500. Very doable for a serious side project.
Project 1 — Autonomous Navigation Robot
Skills covered: ROS 2, Nav2, LiDAR SLAM, EKF, UKF, Sensor Fusion, IMU, Wheel Odometry
Build a fully autonomous differential drive rover that maps an unknown environment and navigates to goals without human intervention.
Fuse your LiDAR with IMU and wheel odometry using an Extended Kalman Filter for robust localization. Mount a wide-angle camera as a forward obstacle detection feed into your Nav2 costmap. Deploy the full Nav2 stack on your edge compute module and get it running in real time.
This is table stakes for most robotics roles but surprisingly few people have done it end-to-end on real hardware rather than in simulation. If you've only ever run Nav2 in Gazebo, this is the project to start with.
Project 2 — Stereo Vision Pick and Place
Skills covered: MoveIt 2, Trajectory Optimization, Stereo Vision, Depth Estimation, 3D Object Detection, PCL, OpenCV
Mount two matched cameras as a calibrated stereo pair on your robotic arm. Calibrate using OpenCV's stereo calibration pipeline, compute a disparity map, extract metric depth, and use that to localize target objects in 3D space. Plan and execute pick-and-place trajectories using MoveIt 2 with CHOMP.
The README writeup matters as much as the code here. Document your stereo baseline selection and calibration tradeoffs. The reasoning behind your decisions is what separates a real portfolio project from a tutorial followthrough.
Project 3 — Edge LLM Robot Assistant
Skills covered: Edge LLM Deployment, Llama.cpp, VLA, VLM, TensorRT, ONNX, ONNX Runtime, PyTorch
Deploy an open-source LLM or VLA model on your edge compute module using a quantized inference runtime. Use a wide-angle scene camera feeding a vision-language model for environment understanding. Export your perception model from PyTorch to ONNX, optimize with TensorRT, and run inference via ONNX Runtime.
Build a simple voice or text interface that translates natural language commands into robot actions and benchmark latency at each stage. The latency numbers are the interesting part — document them honestly including the stages where you couldn't hit real time and what you tried.
Project 4 — Imitation Learning on a Robotic Arm
Skills covered: Imitation Learning, Reinforcement Learning, VLA Fine-tuning, LeRobot, PyTorch, Hugging Face
Mount one camera as a wrist-mounted eye-in-hand camera and a second wide-angle camera as an overhead scene camera. Record 50-100 teleoperated demonstrations using an open-source imitation learning framework with both camera streams as input. Fine-tune a VLA model in PyTorch on your dataset, evaluate on held-out tasks, and publish your model publicly.
Compare imitation learning versus RL policy performance and document the gap honestly. The failure modes are as interesting as the successes and most people never write them down.
Project 5 — Sim-to-Real Transfer
Skills covered: Gazebo, Isaac Sim, MuJoCo, URDF, Sim-to-Real
Model your rover and robotic arm in URDF with all cameras as sensor plugins. Simulate across at least two environments — a physics-based simulator and a GPU-accelerated sim. Train navigation and manipulation policies in sim and transfer to real hardware.
The sim-to-real gap is where most projects fall apart. Documenting how you dealt with it is more valuable than pretending it didn't exist. Anyone who has done real sim-to-real work will immediately recognize whether you have too.
Project 6 — Embedded Firmware Controller
Skills covered: C, C++, Zephyr, FreeRTOS, Low-Level Firmware, PID, SPI, I2C, UART, CAN Bus
Write bare-metal firmware for motor control and sensor reading on an ARM-based microcontroller. Implement SPI for IMU, I2C for encoders, UART for debug output, and CAN Bus for motor commands. Tune PID controllers for motor velocity and position control. Port the same implementation to both a real-time OS and a lightweight embedded OS and benchmark the latency difference. Bridge to ROS 2 via micro-ROS.
A lot of robotics engineers never go below the ROS 2 abstraction layer. This project is worth doing just to understand what's actually happening underneath — and it shows up immediately in how you talk about systems in technical conversations.
Project 7 — Latency Optimized Inference Pipeline
Skills covered: Latency Optimization, Linux Systems Programming, CUDA, TensorRT, ONNX Runtime
Build an end-to-end inference pipeline that processes stereo camera input and wide-angle scene input simultaneously on your edge compute module. Profile and optimize preprocessing, stereo matching, model inference, and postprocessing using Linux perf tools. Use CUDA kernels where bottlenecks require it. Document before-and-after benchmarks at each stage.
Real-time performance on constrained hardware is a fundamentally different problem than getting a model to run at all. This project is specifically about that difference and it's one of the most underrepresented skills in robotics portfolios.
Project 8 — Robot CI/CD Pipeline
Skills covered: CI/CD, GitHub Actions, Hardware-in-the-Loop, Regression Testing, Docker, BehaviorTree.CPP
Containerize your full ROS 2 stack including stereo vision and Nav2 with Docker. Set up a CI/CD pipeline for automated build and test on every commit. Implement Hardware-in-the-Loop regression tests that run on real hardware. Build a behavior tree task executor covering navigation, manipulation, and vision tasks.
Most robotics projects on GitHub are one-off demos that stopped working six months after they were posted. This project is about building something you can actually maintain, iterate on, and hand to someone else.
Project 9 — PCB and Power System Design
Skills covered: KiCad, Mobile Robot Power Systems, FreeCAD, OpenSCAD
Design a custom power distribution board for your rover using open-source EDA tools. Handle battery management and voltage regulation for your compute module, motors, servos, and cameras. Design a stereo camera mount with a fixed baseline and a wide-angle mount for scene coverage using open-source CAD tools.
Most software-focused robotics engineers never touch this layer. Even a basic custom board demonstrates a level of full-stack hardware ownership that is genuinely uncommon and immediately visible to anyone reviewing your work.
A Few Thoughts on How to Approach These
- Prioritize real hardware over simulation — photos and videos of things actually running matter more than clean code on its own
- Document your failures and tradeoffs, not just the wins — the reasoning behind decisions is what demonstrates real understanding
- A clean reproducible README is worth as much as the code itself — if someone can't clone and run it, it may as well not exist
- You don't need all nine — projects 1 through 4 cover the core of what most physical AI roles care about
Where to Start
If you're just getting into this space, start with Project 1. It's the foundation everything else builds on. Once your rover is navigating autonomously on real hardware, the path to the rest of these becomes much clearer.
If you're already comfortable with navigation and want to push into embodied AI, jump straight to Projects 3 and 4. That's where the field is moving and where the interesting problems are right now.
If you're working on any of these or something in the same space, drop a comment or reach out directly. Always happy to compare notes, give feedback, or collaborate on something interesting.
