Hey DEV Community! 👋
This is my first post here, and I'm excited to share a project I've recently started working on.
I'm a Computer Science undergraduate interested in GPU programming, high-performance computing, and the systems behind large language models.
Over the past few months, I've been learning CUDA and exploring how GPUs execute parallel workloads. Eventually, that curiosity led me to a bigger question:
What would it take to build an LLM inference engine from scratch?
Not just load a model through an existing inference framework, but actually understand and implement the components responsible for running it.
That's how Project EREN started.
What is Project EREN?
EREN is my attempt to build a CUDA-accelerated LLM inference engine from the ground up, using C++ and CUDA.
The initial development target is my NVIDIA RTX 4060 Laptop GPU with 8 GB of VRAM.
I'll start with a small decoder-only transformer such as GPT-2, establish correctness, and gradually work toward more capable models and efficient inference.
My long-term goal is to support multiple concurrent inference requests with high throughput, while understanding the underlying GPU kernels, memory management, scheduling, and execution strategies.
Eventually, I want EREN to serve as the inference backend for a RAG-based AI agent.
This isn't about competing with mature inference engines right away. It's about learning how these systems work by building one.
Why build an inference engine from scratch?
Libraries like PyTorch and established inference runtimes make running language models surprisingly accessible.
But I wanted to understand what happens underneath those abstractions.
For example:
How are transformer operations mapped onto GPU kernels?
What happens to model weights and intermediate activations in GPU memory?
How does autoregressive token generation work internally?
Why do memory bandwidth and kernel launch overhead matter during inference?
How can an engine efficiently serve multiple requests on a resource-constrained GPU?
Instead of only reading about these concepts, I want to implement, test, benchmark, and optimize them myself.
Technology stack
My current development environment includes:
Language: C++17 and CUDA C++
GPU: NVIDIA GeForce RTX 4060 Laptop GPU (8 GB)
Environment: Ubuntu on WSL2
Build system: CMake
Testing: CTest
Reference implementation (upcoming): Python, PyTorch, and Hugging Face Transformers
The core inference runtime will be implemented in C++ and CUDA. Python and PyTorch will initially help validate correctness against a trusted reference.
Progress so far: Phase 0
My first milestone was establishing a reliable C++/CUDA development foundation.
So far, I've worked on the following components.
- Project structure and build system
Organized the repository into components for kernels, source code, headers, tests, benchmarks, and documentation.
Configured CMake to build the C++ and CUDA targets.
Here's a look at the current EREN project structure in VS Code:
- CUDA vector addition
Implemented a basic vector addition kernel to validate device memory allocation, host-to-device transfers, kernel launches, and result verification.
- GPU reduction
Implemented a shared-memory, multi-stage reduction baseline.
This was an important exercise in understanding parallel reduction and coordinating work across thread blocks.
- CUDA error handling
Created a reusable CUDA error-checking helper to make runtime failures easier to identify and debug.
- Testing and benchmark foundation
Added smoke tests using CTest and an initial reduction benchmark target.
A clean build succeeded, and all four configured tests passed.
Here's the actual Ubuntu terminal output from the EREN development environment, showing the build and test results:
The inference engine itself isn't operational yet. This milestone establishes the foundation for the next stage.
What's next?
I'm developing EREN in phases rather than jumping straight into optimized kernels.
Phase 1 — Python/PyTorch reference implementation
I'll begin with GPT-2, study its forward pass, inspect its weights and tensor shapes, and generate reference outputs for correctness testing.
Phase 2 — First CUDA inference implementation
Implement the initial CUDA operators and connect them into a complete GPT-2 forward pass.
Later phases — Optimization and serving
Explore optimized matrix multiplication, quantization, GPU profiling, memory management, batching, concurrent requests, and inference scheduling.
The aim is to progress from a correct implementation toward a measurable, efficient inference runtime.
Why I'm documenting this publicly
I want this series to be more than a collection of project announcements.
As I progress, I'll share:
What I'm implementing and why
CUDA concepts and engineering decisions
Bugs, failed experiments, and lessons learned
Benchmarks and performance comparisons
Design tradeoffs and improvements
I'll also try to distinguish between what works, what I've measured, and what I'm still experimenting with.
Final thoughts
EREN is still at an early stage, and there's a lot left to learn.
But that's exactly why I want to document the process from the beginning.
If you're interested in CUDA, GPU computing, LLM inference, or systems programming, I'd love to exchange ideas and learn from your experiences.
Next up: Building a GPT-2 reference implementation in PyTorch before moving into CUDA inference.
Thanks for reading!
GitHub Repository: Project EREN
