This is a submission for the Gemma 4 Challenge: Write About Gemma 4
There is a comforting story going around at the moment, and it goes something like this: The technology has finally arrived; it is open now and free now, and so the parts of the world that got left behind in every previous wave of computing can at last begin to catch up. There is no more waiting for someone else's infrastructure to reach you, no more begging for an API key with a credit card you do not have, because the intelligence fits on a phone now and the gap that once felt permanent starts to close on its own.
I want to say plainly that I believe that story, because most of what follows is going to complicate it, and I would hate for the complication to be mistaken for cynicism. The story is genuinely true. It is only true up to a point, and about a week ago, when I stopped reading about Gemma and actually started building with it, I found out where that point sits and what it feels like to be standing on the edge of it.
The question of who actually gets to use these tools is not a new one for me. Four years ago I gave a TEDx talk about why young people in Sierra Leone should get into tech, and earlier this year that talk resurfaced as a memory on my phone. Watching it back, I realised I no longer wanted to ask the question I had asked then. The question that felt honest now was a harder one: how young people should orient themselves when technology is moving faster than the institutions, the curricula and the job markets that are meant to prepare them for it? That reframed question is where the name Orient comes from. Somewhere in the middle of sitting with it, I wrote down a line I have not been able to shake since: geography still shapes access, even as the internet promises openness. I believed it then as an idea, and building with Gemma is how I came to believe it as a fact. I will come back to the exact moment that turn happened.
But Orient did not begin as a thesis or a product. It began as something I wanted for myself. I read a great deal, articles and essays and the things people forward to me, and I forget most of it almost as quickly as I take it in. So I wanted something that could sit beside me while I read and help me actually understand and hold on to what was in front of me, because I have come to believe that the real skill of this era is not any particular tool or fact but learning how to learn, and the sense of agency that grows out of that. I also argue with my friends constantly, about more or less everything, and I wanted something I could genuinely own, something personal and local that lived on my own device and that I could think against and argue with whenever I felt like it, without having to ask anyone's server for permission first. But, there is a detail about me that is easy to miss in all of this, which is that I have decent wifi on most days and I am not the hard case at all. And yet even I get caught out by the environment when the network quietly dies or the power goes out, and in that moment the tool that lived on someone else's server is simply gone. If that can happen to me on a good day, you'd be amazed to know it is the ordinary daily reality for a lot of people where I live.
Then one evening it stopped being about me at all. I was working in a café here in Freetown when I noticed a couple of university students at the next table, clearly cramming in the way where you can almost watch someone trying to pour an entire textbook into their head in a single night. We got talking, and I tried to show them NotebookLM, which I use all the time. The idea is that you drop your notes in and it gives you back a podcast or a mind map or a short video, something you can actually engage with rather than rereading the same page until the words stop meaning anything. One of them had a decent phone and the other did not. I gave them the URL, but the café connection was a mess, so I ran fast.com to see what we were actually working with, and it came back at 80 Kbps kilobits per second, kilobits rather than megabits, which at that speed made the tool effectively unreachable. In the end I tethered them to my own mobile wifi; we generated an audio version of their notes right there at the table, and then they said they'd listen to it on their way home.
I sat with that afterwards and saw two separate gaps stacked one on top of the other. The first was that they had not known a tool like that even existed, something that could genuinely change the way they studied. The second, and the heavier of the two, was that the moment they did know, the connectivity simply slammed the door in their faces anyway, so that knowing turned out not to be enough and the network got the final say. That was the afternoon I understood properly who I should be building for, and why. What I should build was not a clever thing for me to argue with over good café wifi, but a study companion for students like those two, built on an open model that runs on the device itself, with no server in the loop and nothing leaving the phone, working the same on a Sunday with no signal as it does on a Tuesday with full bars. The thing that is missing here was never access to the tools, because the tools are free and a download away. What is missing is exposure and use, and "use" only happens when the thing actually works inside the conditions people are living in.
This is not some niche habit that only anxious students reach for, either. Andrej Karpathy, who has been about as close to how these models are built as anyone alive, recently described a habit he has fallen into of reading almost everything, blogs and articles and whole book chapters, with a language model alongside him. He does a first pass on his own, a second pass where he asks it to explain and summarise, and a third pass of going back and forth in questions and answers. He said he comes away understanding the material more deeply than if he had simply read it once and moved on, and that this is becoming one of the top uses of the whole technology. When I read that I thought, yes, that is exactly it, because it is precisely the thing I want to put into the hands of a student in Freetown on the Sunday night before a Friday exam. The most powerful everyday use of all of this turns out to be one of the simplest, which is to sit with your own material and have something intelligent help you genuinely understand it.
So I committed fully to the on-device version of that idea.
Why I reached for the smallest model
A quick word on why Gemma 4 E2B in particular, because Gemma comes in a range of sizes and this was a deliberate pick rather than an accident. E2B is the smallest of them, the edge model, two billion parameters trimmed down to run on a phone rather than in a data centre, with larger and more capable versions sitting above it. I did not reach for those, and the reason is really the whole argument of this piece in miniature. The work Orient actually does, helping a student quiz themselves against their own notes, pulling a summary out of a long document, sitting beside them while they study, does not need a frontier-sized model. It needs one that is good enough at those things and as light as it can possibly be, because on the hardware my users carry, model size is not really a question of capability, it is a question of who gets left out.
Every step up in size raises the floor a little higher, asking for more memory, a longer download and more battery, and quietly pushing a few more of the cheapest phones out of the room. So choosing E2B meant choosing the lightest thing that could still do the job, on the understanding that going bigger would have bought me capability I did not need at the price of excluding the very people I was building for. The catch, which is what the next section is about, is that even the lightest model in the family still has a floor.
Where the catch-up theory meets the floor
Here is the thing the comforting story tends to leave out, which is that the open weights are free but the silicon you need to run them is not.
E2B is a genuinely impressive piece of engineering, but it is still a file of roughly 2.59 gigabytes, and to actually run it the phone needs somewhere between 1.3 and 1.7 gigabytes of free memory simply to hold the model and its working state. And that is free memory, the memory left over after the operating system has already taken the 1.5 or 2 gigabytes it needs to keep the phone running at all.
Now picture the phone that the students I am building for actually carry, which is not a flagship but a mid-range Android with three or four gigabytes of memory in total, the kind you can buy for a hundred and something dollars because that is what the market here is made of. When you do the arithmetic it simply does not close, because the operating system takes its share and the model asks for its share and there is not enough to go round. So the phone does the only sensible thing it can, and either refuses to load the model or loads it and then crashes, in order to save itself.
Open models preach inclusion, and I do not doubt for a second that they mean it. But on the ground here there is a quiet caveat sitting underneath the word, a subtle exclusion that is easy to miss from somewhere with better hardware, because the phones that can comfortably run Gemma on their own, the ones with eight gigabytes of memory and a dedicated chip for this kind of work, tend to belong to the people who already had access to almost everything. The student who would get the most out of a free, private, offline study tool is very often the same student whose phone cannot run it.
That is not the work of a villain but simply physics, and it is real all the same, so pretending otherwise would have meant quietly building a product for the wealthiest sliver of the people I had set out to serve while telling myself I was being inclusive. I was not willing to do that, so I went and tried to understand the machine properly.
What it actually means to run a model on your phone
This is the part I had to learn the hard way over the course of that week, so let me try to save you the trouble of repeating it.
I started where a lot of people start, which was inside Google's own AI Edge Gallery, the open-source app they publish to show what on-device generative AI can do. I was not just playing with it; I was reading it, going through the repository with an agent to understand how it is genuinely wired underneath, how it loads a model and how it handles the skills that sit on top, so that my agent and I could learn the architecture and adapt the parts that fit what I was building. What I found and wrote down in my own reference notes from that week, is that the path it uses to load Gemma 4 runs through something called AICore by way of Google's ML Kit GenAI Prompt API, which is the system-level, Google-managed way of getting at the model.
The appeal of that path is real, and it is mostly about distribution. Instead of you bundling a two-and-a-half-gigabyte model inside your own app, which would turn your app into a three-gigabyte download that nobody on a metered connection is ever going to accept, AICore lets the operating system download and hold the model once at the system level, where any app on the phone can share it, so that your own app stays small. For a market like mine that is a genuinely valuable thing, and when I first understood it I was excited.
But I had quietly assumed something that turned out to be wrong, and it took a clear correction for me to see it. I had imagined that going through Google's GenAI service meant the model was not really computing on the phone, that the heavy lifting was happening somewhere else and the little phone in the student's hand was just a window onto it. That is not how any of it works. Whether the model reaches the phone through AICore or through any other route, the inference still runs on the phone's own silicon, on its processor or its graphics chip or its dedicated AI core, drawing the phone's power, warming the phone up and consuming its memory while it thinks. The GenAI service solves storage and distribution, which is to say it solves how the model gets onto the device and where it is kept. But it does nothing at all about compute and battery, because wrapping a model and routing it through a service only changes which engine runs it and how it arrives, never where the actual thinking happens. The thinking happens in the student's hand, every single time.
And because Google knows perfectly well that asking a weak processor to run a two-billion-parameter model will freeze or crash the phone, that managed path is gatekept. During the current developer preview it is restricted to an allowlist of recent, capable devices, and this is where the abstract argument turned concrete for me. I am on an iPhone and Orient is an Android app, I could not test any of this on my own device at all, so I borrowed a friend's Samsung Galaxy S22, which is a flagship phone, precisely so that I would have capable hardware to build against. And even that borrowed flagship was not on Google's AICore allowlist, which meant the managed path was closed to me, not on some cheap handset but on a current, expensive, genuinely powerful device sitting right there on my desk. That was the moment the line I had written down earlier in the year, about geography still shaping access even as the internet promises openness, stopped being an idea I found clever and became a fact I could feel, because the door was being held shut on a flagship phone in Freetown.
The Gallery actually offers two paths behind a single switch, the AICore one I had been reading and a second that runs the model through LiteRT-LM, and it was to that second path that I moved, Google's on-device runtime and, as it happens, the route they now recommend for this anyway. With LiteRT I bundle and run the model myself rather than waiting for the system to hand it to me, which takes me out from behind the allowlist and lets the app actually run on the borrowed S22. The trade is that the 2.5 gigabyte model comes back into the picture as something the user has to download, which drags the whole connectivity problem back in with it, the same problem I had watched defeat those students in the café. There is a cleaner way to hand that download to people, through Google Play's own asset delivery, which needs a Play Console developer account and the small one-time fee that comes with it. But that too is purely a question of distribution, of how the file reaches the phone, and it leaves the compute exactly where it always was. I have built the model access in Orient behind a single interface, a seam that lets the app route to whichever engine is available without the rest of the code needing to know or care, so that LiteRT can run things today and the managed AICore path can take over later, on the devices and at the moment that allow it. One quietly important consequence of designing it that way is that the context window, the amount of material the model can hold at once, is not a fixed constant but a property of whichever provider the app is routed through, so the budget the model has to work with can widen or narrow depending on the path rather than being carved permanently into the app.
Battery is the part of all this that stayed with me the most. For a student here who is already living around rolling power cuts, battery life is a kind of currency, and if a single study session drains a noticeable slice of it and leaves the phone too warm to hold comfortably, they will stop using the tool, not because it failed to work but because it cost them something they could not afford to spend.
The budget you cannot see
There was a second limit waiting for me underneath the hardware, and it was one I had misunderstood just as badly, which is the context window.
I had been thinking of the context window as a kind of memory, but it is not that at all; it is a per-call budget. For Gemma 4 E2B that budget is around 128,000 tokens, and that single number has to hold everything for one run of the model at once, the instructions, the source material you have fed in, whatever conversation has happened so far, and crucially the room the model needs to write its answer, all of it drawn from the same pool and capped at the same ceiling. The model itself is stateless, which is to say it remembers nothing from one call to the next, and the only thing it knows on any given run is exactly what you managed to pack into that budget.
That stays abstract right up until it bites. Imagine a student dropping their entire lecture-notes PDF into the app so they can quiz themselves on it, and imagine those notes are large enough to fill nearly the whole window on their own. There is now almost no budget left for the actual back and forth, for the questions and the answers and the follow-ups, which is the very thing they opened the app to do, because the source has eaten the room the conversation needed to live in.
And the sources do not all cost the same, which was the part that genuinely surprised me. Text is featherweight, roughly 4 characters to a token. But a photographed page of a textbook is not measured in pixels; it is run through a vision encoder that turns it into something like a 1000/2000 tokens. And audio is heavier still, because Gemma's on-device audio works out to somewhere around 375 tokens for every minute, which means a 47 minutes lecture recording is already roughly 18,000 tokens, and a 3 hour recording can swallow half the entire window from a single file. Three ordinary text PDFs barely make a dent in the budget, while one long voice note can blow straight past the ceiling.
How the constraint shows up in what I had to build
I am laying all of this out in detail because the shape of Orient, the actual decisions sitting underneath it, is the clearest evidence I can offer that the bottleneck did not vanish when the models opened up but simply moved down a layer, and you can see exactly where it landed by looking at what I was forced to build around it.
Take the window limit. Because the model carries nothing between runs and can only hold so much at once, the natural way to deal with material that is too large to fit is to distil it down into something small and reusable, and then carry that forward instead of the original. You do not feed the four-hundred-page textbook back into the model every time, because you cannot; it is far too large. So instead you summarise the chapter you care about once into a compact note, and from then on everything else you do, the quizzing and the further questions and the rest, works off that small note rather than the enormous book behind it. Large source in, small reusable piece out, and you build forward from the small piece. Orient is built around exactly that, with everything you produce becoming something you can study against later, all of it chained through a personal library that lives on the phone. I had originally chosen that design for reasons to do with how people actually learn, and the window limit handed me a second and completely independent reason for the very same architecture, which is that on a 128,000 token device it is simply the only way large material becomes usable at all. When the constraint you were dreading and the design you already believed in turn out to point in the same direction, that is about the strongest signal you can get that you are building the right thing.
The compute limit shows up just as plainly. Since no way of wrapping or delivering the model can move the thinking off the phone, the only thing that genuinely can is sending the work out to the cloud, and so the principle Orient is built on is local-first rather than local-only. The app tries to do everything on the device first, because that is where the privacy and the zero running cost and the works-offline promises all actually live, but when a task is genuinely too heavy for the phone in hand, the app does not simply fail in the student's face. It hands the work off gracefully, including a guided handoff to the richer cloud tools when the connection is good enough to earn it, the same NotebookLM I could not get to load in that café, prepared properly and pointed to rather than offered as a watered-down imitation inside my own app.
Underneath all of it sits a truth that most writing about building for emerging markets quietly skips over, which is that there is no neutral choice available here. Every architecture you could pick serves some people and abandons others. If you insist on staying purely offline, you serve the privacy purist with the flagship phone and you abandon the majority on mid-range hardware. If you go purely to the cloud, you reach the cheap phone, but you abandon the student with no data budget, and you give away the privacy entirely. If you go hybrid, the way I have, you reach the most people, but you blur the clean promise that nothing ever leaves the phone, the very promise you might have planted your flag on at the start. You do not get to escape the tradeoff. You only get to make it consciously, choosing the architecture that leaves the fewest people outside while keeping your eyes open about exactly who that still is.
At the edge, product strategy is the thing that changes
If there is one lesson I would pull out of all of this, it is that at the edge the model is not the thing that decides whether you succeed. The model matters, and Gemma being open and small enough to run on a phone at all is genuinely what makes any of this possible in the first place, but on its own it is nowhere near enough. What actually decides things is the product strategy wrapped around it, and looking back at everything this one week taught me, that strategy rests on three understandings that have to be held together at the same time.
The first is understanding the model itself, which is what tells you where you can lean on it and where you have to push against its edge or route around it altogether. You only learn that by working with its real shape rather than the shape the demo implies, the way I had to learn that the window is a budget and not a memory, that audio is heavy where text is light, that the managed path is gatekept behind an allowlist, and that no amount of clever wrapping moves the compute off the phone. Once you know the real edges, you know which jobs the model can carry on its own and which ones you have to hand somewhere else.
The second is understanding your users, which is what tells you which problems are actually worth solving in the first place, so that you build the thing that matters to them rather than the thing that looks impressive on a stage. I did not really understand this until the café, watching two students who needed exactly what these tools can do and could not reach any of it, and learning in that moment that for someone living around power cuts, battery is not a number on a spec sheet but a currency they have to spend carefully. You cannot design the tradeoffs well until you know whose day they actually land in.
The third is understanding the environment and the infrastructure you are building on, which is what tells you what will survive once the thing leaves your hands. That is the 80 Kbps in the café, the power that comes and goes, the storage and the data that cost real money to the people spending them, and the fact that academic life here already moves through WhatsApp rather than through apps that people are being asked to adopt. A design can be perfectly sound on a whiteboard and still die the first time it meets any of that.
None of the three is optional, and none of them stands alone. The product strategy is the act of holding all three at once and making the tradeoffs they force on you, deciding what runs on the phone and what goes to the cloud, what you build yourself and what you hand off, who you serve first and who you reach later. That is the deeper meaning of something I have believed since before I had the words for it, which is that the edge cases are not a distraction from the product; they are the product. At the edge, the strategy is the thing that changes who this actually works for.
The one thing I have not decided
And since the strategy is the part that matters most, I want to be honest about the one piece of mine I have not finished working out, because it would be easy to tie a bow on this that I have not actually earned. I know the principle, local-first with a graceful handoff to the cloud, and I know why the principle is right. What I have not finished deciding is exactly where the line should fall in practice, how much should stay on the phone and how much should route outward, especially at the very bottom of the device range where even LiteRT struggles and the cloud starts to look less like a luxury and more like the only thing that actually works. That is exactly the kind of tradeoff I described a moment ago, the kind you make with measurements taken from real hardware rather than from a diagram, and I am still in the middle of making it. Writing as though I had already settled it would be the dishonest part, so I will leave it where it genuinely is, which is open.
For the next builder who hits this floor
If you are reading this because you are about to build something real with an open model for a place that looks like the one I am building for, and you have just walked into the same floor I did, here is what I would want to tell you.
The first thing is that this gets better, and not slowly. The whole reason on-device AI is worth caring about is that the phones already in our pockets are doing things this year they could not do last year, with nobody having to buy new hardware. The context windows are widening while the compute these models demand is falling, and the line where running this locally requires a flagship is drifting down toward the mid-range a little more with every few months that pass. The exclusion I have described is real, but I genuinely believe it is temporary, so build for where the floor is heading rather than only for where it sits today, because by the time you ship, it will already have moved.
The second thing is that if this bottleneck is blocking you right now, today, and you cannot afford to wait for the floor to fall, you do have options inside the open world. There are other open models with larger context windows and lighter footprints than Gemma carries at this moment, and some of the labs coming out of China in particular are pushing well past it on exactly these dimensions, so if your whole product depends on running locally on weak phones this quarter, it is genuinely worth your while to look around properly before you commit.
I did look, and I chose to stay where I am, and that was a deliberate decision rather than an oversight. I am betting on Gemma and the wider Google ecosystem on purpose, not because it is flawless on day one for my exact hardware, but because of the other things I want this app to do further down the road and the way the whole stack fits together for where I am taking Orient next, and that is a bet about my own roadmap rather than a claim that it is the only sensible answer. Yours may well point somewhere else, and that is completely fine, because the point was never which model you land on. The point is that you choose it with the floor in full view, and that you design honestly around the people who are standing below it.
The catch-up story is true. The technology really has arrived, it really is open, and it really does fit on a phone. You simply have to build for the phone that people actually own rather than the one the demo was filmed on, and you have to stay honest that for now, geography still shapes access, even as the internet promises openness.
