The below is a continuation of the series on the history of Expanso. Today, we're talking about the tenets of Expanso. Read the whole series starting from Part 1.
The Four Tenets of Sane Data Pipelines
In the last post, I explained the origin of Expanso: hitting the physical limits of the network makes you realize you can't move exabytes of data. You have to move the compute.
But an idea is not a platform. Building a system that doesn't just work in a lab but survives in the wild requires a clear philosophy. We weren't just building another distributed system for the sake of it—we were trying to build something real teams could use without burning out. After years of watching brilliant engineers get bogged down by data logistics, we built Expanso on four core tenets.
Tenet One: Velocity is the Only Thing That Matters
The most elegant architecture in the world is useless if your developers can't ship. Before we talked about cost, security, or scale, we started with one question: how do we make our users faster?
At both Google and Microsoft, I saw many customers spend more time writing boilerplate for data movement and failure handling than on actual business logic. A simple pipeline to filter logs could take weeks to production-harden. Every new on-prem cluster or cloud region added a unique layer of complexity that brought progress to a crawl. The work was repetitive, error-prone, and demoralizing.
We decided Expanso had to solve this aggressively. That meant letting developers use the tools they already knew—Bash, JavaScript, containers—without forcing them to learn a new DSL. It meant the platform, not the developer, would be responsible for the genuinely hard parts of distributed systems: consensus, scheduling, retries, and observability.
Anyone can log into a machine and execute a script; cron has been around for 40 years! The real work—the work that separates a prototype from a production system—is everything else. It's enforcing data models, managing rollouts, ensuring correct execution when disconnected, versioning, and debugging a black box when a failure inevitably happens at 2 AM. This is the operational friction that kills velocity.
We built those primitives into the platform so you have full transparency without having to write it yourself. A developer should be able to write a script, define its relationship to the data, and deploy. The system handles the rest. Velocity isn't a feature; it's the foundation.
Tenet Two: Your Data, Your Rules
For too long, data infrastructure has presented a series of false choices. Want fast insights? Pay a fortune for high-bandwidth links. Need tight security? Lock the data down until it's useless. This leads to shadow IT, shocking egress bills, and security audits that end in disaster.
Our second tenet was to give users uncompromising control over their data. This isn't an abstract concept; it provides concrete advantages:
- Control over Cost: The most unpredictable cost in a distributed environment is data movement. By processing data at the source, you significantly reduce egress fees, slash intermediary storage costs, and subvert the need to spin up extra compute for boring actions. You control the budget because you control the bytes.
- Control over Speed: Latency is a function of physics. When compute runs where data is generated, you get results in milliseconds, not hours. Real-time analytics becomes a practical reality.
- Control over Security: Every copy of your data expands its attack surface. Keeping data within its original security perimeter is the simplest, most effective way to protect it. Our model enforces a least-privilege approach by default.
- Control over Provenance: In a tangled pipeline, it's easy to lose the thread of where data came from and what's been done to it. A central control plane that logs every job—what ran, where, on which data—provides an automatic, auditable record. You can trust your outputs because you can trace their lineage.
Control isn't about restriction. It's about having the clarity and tools to manage your data ecosystem with intention.
Tenet Three: Build for an AI-Native Future
The first two tenets solve today's problems. The third prepares you for tomorrow's. The next decade will be defined by AI, and AI is a data problem. A model is only as good as the data it's trained on and the freshness of the data it uses for inference.
The old "train once, deploy statically" model is already breaking. The future of AI is dynamic and embedded in the real world—on factory floors, in retail stores, in hospital networks. That requires a new data architecture. By building a platform that excels at running compute at the source, we are also creating the ideal foundation for the next generation of AI:
- A Foundation of Trustworthy Data: You can't have good AI without good data. By automatically tracking the origin and transformation of every dataset, the platform builds a foundation of lineage and provenance that your downstream models can rely on.
- Feature Engineering at the Edge: Instead of shipping petabytes of raw sensor data to the cloud, you can generate clean, structured features right where the data is created.
- Real-Time Inference: You can run models locally for immediate decisions, like flagging a fraudulent transaction or identifying a defective part on an assembly line.
- Federated and Continuous Learning: You can preprocess—or even train—models across distributed datasets without ever pooling sensitive information, paving the way for more powerful, privacy-preserving AI.
Tenet Four: Design for Humility
The first three tenets—velocity, control, and future-readiness—aren't a checklist. They are a single philosophy, held together by a final, crucial principle: humility.
The graveyard of failed platforms is filled with brilliant, opinionated systems that told users how they should work. The history of infrastructure is a story of creators assuming they could anticipate every need, only to be broken by the reality of production. Every data system is different. Every company has its own unique constraints. The use cases in finance look nothing like those in manufacturing, and a single organization might have wildly different needs from one team to the next.
We knew we couldn't possibly anticipate every scenario our users would face. Acknowledging that isn't a weakness; it's a design principle.
So, instead of building a rigid, prescriptive framework, we focused on creating a flexible and extensible one. This meant building a system that provides strong opinions on the hard parts—reliability, security, auditability—while remaining unopinionated about your business logic. Our job is to give you a reliable engine, not tell you how to drive.
This principle manifests in concrete ways:
- No Lock-in: You bring your own code, in your own languages and containers. We run it. We don't force you into a proprietary DSL or a new way of writing your logic.
- Composable Primitives: We provide a powerful set of tools for scheduling, data placement, and observability, but we don't dictate how you must piece them together. The platform is designed to enable, not prescribe.
- Extensibility by Default: We assume that you will need to integrate with tools and systems we've never heard of. The architecture is built to be extended, allowing you to adapt the platform to your world, not the other way around.
Building with humility means accepting that the most innovative uses for our platform will be invented by our users. Our job is to give them a powerful, reliable set of tools and then get out of their way.
Next Up
Each of these tenets deserves its own post, and I'll explore them in the coming weeks. But first, I'm interested in your experience.
What's the one principle you feel is most often sacrificed in your data infrastructure, and what has the impact been?
Originally published at The History of Expanso (Part 2): Our Core Tenets.
