My new book is now available on Amazon! 😀
This book is written for AI application developers who are not satisfied with simply calling an LLM endpoint and want to understand model architectures and the internal workings of inference engines. It uses the open-source TensorSharp project and Google’s Gemma 4 E4B GGUF model as practical examples.
TensorSharp has achieved performance parity with llama.cpp across the main benchmarks, while outperforming it in several scenarios. The book explains some of the key performance optimizations and their implementations, including paged and prefix KV caching, continuous batching, GPU kernel fusion, and more.
I chose Gemma 4 E4B, a dense model, because it is a compact multimodal model that supports images, audio, and video, making it suitable for a wide range of devices. TensorSharp also supports and is optimized for MoE and diffusion architectures, as well as model families such as Qwen and GPT-OSS. However, due to limitations in time and book length, these topics are not covered in this edition. Those interested can explore the project directly on GitHub or contact me for further discussion.
I selected GGUF because it is an inference- and edge-device-friendly model format. This is particularly relevant to the .NET ecosystem, where local applications, mobile applications, and game development are important use cases. TensorSharp also supports the Safetensors format, which it currently uses for VAE and LoRA models.
For clarity and ease of understanding, the book primarily presents the CPU code path. In practice, however, TensorSharp supports and is extensively optimized for multiple GPU backends, including NVIDIA CUDA, Apple Metal/MLX, and Vulkan for AMD, Intel, and other devices. More implementation details are available in the GitHub repository.
TensorSharp and this book focus exclusively on model inference. For model training, readers can refer to Seq2SeqSharp, one of my earlier open-source projects. Adding a trainer to TensorSharp itself would not be particularly difficult, but building a modern, efficient architecture that unifies training and inference is a much larger undertaking. I do not currently have sufficient resources to explore that topic properly, so it remains outside the project’s scope for now.
One more point: TensorSharp is a project built with the assistance of a native coding agent. Throughout the book, I have also incorporated my own views on how code-agent-driven projects should be developed and managed—perhaps a little personal perspective woven into the technical content.
The philosophy can be summarized as:
Contracts as the source of truth.
Test-driven development.
Evaluation before optimization.
Here is the book link on Amazon:From Tensors to Tokens: Building a Multimodal LLM Inference Engine from Scratch with TensorSharp and Gemma 4 E4B
Here is TensorSharp Github Repo: https://github.com/zhongkaifu/TensorSharp
