From Chatbot to AI Agent: Building a Voice Nutrition Companion with Rasa Mantle

Notes from building a voice-first nutrition companion with Rasa Mantle: how a chatbot becomes an agent, and what skills, deterministic tools and memory look like in practice.

TL;DR

• The story: I built NourishHer, a voice-first nutrition companion on Rasa Mantle, through the Rasa Heroes program to find out how an agent differs from a chatbot.
• What I built: 16 skills and 15 deterministic tools.
• The core idea: the model understands, routes and phrases. Code owns saving data, nutrient numbers and safety wording. Mantle gives you building blocks to enforce that: skill gating (requires:), tool constraints, ordered steps and fixed replies.
• Memory lesson: in the build I used, project memory could be written once per field per conversation, so anything that changes (profile, meal log, plans) lives in a simple file store. Only a small set of values, mainly the condition flags that unlock skills, stay in project memory.
• Real data: nutrient values come from USDA FoodData Central, and percent daily value (%DV) is computed in code rather than recalled by the model.
• Testing: 9 deterministic tests and 4 live end-to-end (e2e) conversation tests passed in my last run.
• Status: an experimental project, built for learning.
• New to Mantle? Read sections 4 and 5, then try the project yourself (see the end of the article).

Here's the thread this article follows. Conversational AI started with chatbots that classify intents and follow dialogue paths, and it's now moving toward agents that call tools, keep state and act on what people say. The Rasa Heroes program gave me a way to explore that shift hands-on with Rasa Mantle. NourishHer, a voice-first nutrition companion, is the project I built to find out what really separates an agent from a chatbot.

1. The Evolution of Conversational AI: From Chatbots to Agents

Here's the classic setup. Someone types "I had two eggs and toast for breakfast." An NLU (natural language understanding) model labels that as a log_meal intent, pulls out egg, toast and breakfast as entities, and a custom action writes a row. It's predictable, and you always know why it did what it did.

It also gets hard to maintain once real people show up. They give you half the information and finish it two turns later. They correct themselves ("no, three eggs"). They ask for two things in one sentence. Each of those meant more training examples, another rule, another state transition, and the pile never stopped growing.

LLMs took away much of that pain. A model can read a messy message in context without anyone listing intents up front. But reading a message isn't the same as doing something with it. Somebody still has to save the meal, and then know whether it actually saved. That's the gap between a chatbot with an LLM inside and an agent. The agent can reach outside itself, run operations, hold state, and still make sense on turn twenty. Most of the hard engineering lives in the code around the model.

2. Beyond the LLM: Understanding Agent Harness Architecture

The usual name for "the code around the model" is the agent harness. It loads context, offers tools, runs them, writes state back, copes with failures, and decides when a turn is over. LangGraph and the OpenAI Agents SDK are general-purpose harnesses, and they're a natural fit for work that ends: a research task, a code change, a workflow that produces an artifact.

A conversation doesn't end like that, because the person stays in the loop. They interrupt, change their mind, answer the question you asked three turns ago, wander off and come back, all inside one session. The harness has to keep continuity and decide on every turn what's allowed to happen next.

So the question I went in with was whether to build that on a general harness or start from one designed around conversation. I didn't have an opinion yet. Section 11 has the one I came out with.

3. Exploring Rasa Mantle Through Rasa Heroes — Cohort 1

That question is what the Rasa Heroes program let me explore. The program gave a set of developers first-hand experience building on Mantle, Rasa's engine for skill-based agents, and I'm grateful to have been one of its members. As I understood it, the idea behind the cohort was that real projects, built independently by people meeting the engine for the first time, can teach things documentation alone doesn't. The Rasa team answered architecture questions quickly and were open to feedback.

A Mantle project looks nothing like the Rasa projects I'd seen before. There's no domain.yml, no config.yml, no flow files. A skill is a markdown file, and the rest is YAML and Python. What I wanted to learn was how skill orchestration, memory, deterministic execution, tool integration and evaluation behave once an assistant has to do things rather than say things. So I picked a project that would force the issue.

4. Rasa Mantle: A Conversation-Oriented Agent Harness

Mantle's answer to the question from section 2 is to start from conversation instead of bolting it onto a general agent loop. In practice four pieces matter: the orchestrator, skills, tools and memory. A project is a folder. agent.yml holds the persona and global rules, integrations.yml the LLMs and channels, and each skill lives in skills/<name>/.

Here are the terms you'll see throughout, and where each one lives in the project:

Orchestrator: Coordinating the Conversation

The orchestrator is the LLM loop at the center. Each turn it reads the message with the conversation so far, picks a skill (or none), and runs "call a tool, read the result, decide what's next" until it has something to say. There's no intent list. It routes from each skill's description.

That had a consequence I didn't expect: skill descriptions become your router, so you end up writing them like API contracts, including what a skill must not handle. My nutrition_qna description explicitly says not to take condition-management questions when a condition is on file. The condition skills' descriptions say they take priority over general Q&A. I also added a global rule that greetings and small talk must not force a skill.

Skills: Organizing Conversational Capabilities

NourishHer has 16 skills. Three handle onboarding, five cover everyday nutrition (logging, planning, history, Q&A, coaching), five are condition-aware, and three cover safety and closing. Logging and planning are separate because they're different kinds of work. Logging writes data. Planning reads a profile and proposes something. Different tools, different risk.

requires: is what I leaned on most. The PCOS skill carries requires: session.project.profile_intake_complete == True and session.project.has_pcos == True, so the PCOS skill stays unavailable until onboarding is complete and the person has said they have PCOS. That keeps condition-specific guidance from activating without the relevant context, and I didn't want an app that assumes a diagnosis.

Tools: Connecting Reasoning to Application Logic

There are 15 tools. Each is a plain Python function marked with @tool that returns a ToolResult, which is the data handed back to the model. The default is a tools.py inside the skill folder. When two or more skills need the same function it moves to the shared tools/ directory. get_food_nutrient_info is imported by six skills. Three habits stuck. Return {"ok": False, "error": ...} instead of raising, so the model has something sensible to say. Write descriptions carefully, because the model reads them. And check context.is_cancelled before writing anything, so an interrupted request (common with voice) doesn't save half-finished data.

Memory: Maintaining Conversational Continuity

Data can live in three places: skill memory (session.<skill>.*, with fields the LLM may fill), project memory (session.project.*), and my own file store. I used all three. The details are in section 9, because project memory has a rule that shaped the whole storage design.

5. Deterministic Execution: Balancing Flexibility with Control

The line I kept repeating to myself: the model decides what the person is asking for, and code decides what happens to anything that matters. Mantle gave me a few ways to enforce that.

Tool constraints hide a tool until a condition holds and can attach a response to the tool's success. In intake_profile:

tool_constraints:
  - save_health_profile:
      requires: session.intake_profile.summary_confirmed
      on_success: utter_profile_saved

The save tool stays unavailable until the person has confirmed the read-back summary. And "Got it, your profile is saved" is triggered by the tool succeeding, not by the model deciding to say it. That was the failure I cared about most: an assistant announcing success for something that never happened.

Ordered blocks are a fixed list of steps inside a skill, and they give the intake its spine. The steps are name and goal, food style, likes and dislikes, practical constraints, allergies, optional health context, and read-back, each with a complete_when condition that says when the step is done. The model still asks everything conversationally, and "I'm vegetarian, I like Mediterranean food, no time to cook" in one breath fills two steps at once. But it can't skip the confirmation.

The levothyroxine reminder is where I learned the most about this boundary. When someone who takes thyroid medication asks a food question, I want a short, fixed reminder played first about how food and supplements can affect how the medication is absorbed. Because it touches medication, I didn't want the wording to drift from one run to the next.

My first attempt was to attach the reminder to hypothyroidism_nutrition with utter on activation. It never fired for me. Second, I told the model in the skill's instructions to call the reminder tool first. It didn't. That's the pattern worth noticing: an instruction written in prose is a request, and the model is free to decline it. Third, I built a gate skill with no prose at all, only an ordered block, so the tool call is the engine's first step rather than the model's choice:

:::ordered_block id=gate
steps:
  - id: send_reminder
    execute_tool: send_levothyroxine_reminder_if_needed
  - id: continue_to_guidance
    link: hypothyroidism_nutrition
:::

The gate and the guidance skill have mutually exclusive requires: conditions on a levothyroxine_reminder_sent flag, which is meant to keep the guidance out of reach until the reminder has played. My takeaway: if something must happen, make it an engine step, not an instruction.

One caveat on the whole approach: deterministic tools don't make the agent deterministic. The model can still pick the wrong skill or hand a tool nonsense. What the tools can do is refuse. update_user_profile checks the field name against a fixed list and returns the allowed set when it gets something else, so the model can correct itself. One of my unit tests tries to set a field called blood_sugar and expects a rejection.

6. Evaluation: Looking Beyond the Final Response

A reply can read perfectly well while the execution behind it is wrong: the wrong tool, nothing saved, a number recalled from the model's memory. So I stopped asking "did it say something reasonable?" and started asking six smaller questions. Did it understand the request? Did the orchestrator pick the right skill? Were the right tools called with valid arguments? Is the state correct afterwards? Does the reply match what actually happened? And what happens when a provider fails mid-turn?

I ended up with two suites. make test is plain unittest against the file store and the tools, using a fake ToolContext, with no LLM and no network. make e2e is Rasa's end-to-end (e2e) test harness: it plays a scripted conversation against the live assistant and checks which skill started (flow_started) and what the assistant said (bot_uttered). I deliberately didn't mock the LLM there, because the point was to watch the orchestrator route for real. The price is that results depend on which provider answers.

Tracing (OpenTelemetry) is wired into endpoints.yml but commented out, so a fresh clone runs without extra services. Most of my testing was replaying conversations over the REST channel with curl and a fixed sender id.

To put all of this under real pressure, I needed a use case that a plain chatbot couldn't fake.

7. Choosing a Practical Use Case: NourishHer

A question-answering bot would have been the easy demo, and it would have taught me very little about what makes an agent different. I wanted a problem where a basic chatbot visibly runs out of road: it has to remember people, take actions that change real data, use facts it shouldn't make up, and respect rules that matter. A nutrition companion has all of that.

A traditional chatbot can do some of these things too, with enough custom engineering. The comparison is about the scope I wanted to test: one assistant coordinating all of them in a single conversation.

It started life as a condition-only assistant I called Sage. I later flipped the assumption: health context became optional, and the condition skills became things that unlock rather than the default. Intake collects all four condition areas in one pass, because PCOS and hypothyroidism are a common pairing and a person may be managing both.

That's a lot for one conversation to carry. In a single session someone might set up a profile, log a rough description of a meal, ask a nutrient question answered from live data, get guidance that respects one or several conditions, talk through a rough week, and say something that needs a fixed safety reply. Each of those comes with different rules about what the model may decide on its own, which is exactly what I wanted to test.

In practice, someone says "I had two eggs and toast with a coffee for breakfast" and it gets logged. They ask "how much protein and iron is in 100 grams of lentils?" and get a live USDA lookup. They say "I keep skipping lunch and then overeat at night" and get one small, non-judgmental next step. By design it doesn't diagnose, doesn't advise on starting, stopping or dosing a medication, and doesn't claim a food treats a condition. It's an experimental learning project.

8. NourishHer Architecture and Technology Stack

Blue marks the parts driven by an LLM. The tools and data row at the bottom is plain Python and never calls a model.

To see how the pieces fit together, here is one turn: a nutrient question, typed or spoken.

The model is involved in routing and in the final wording. The lookup in between is deterministic.

I didn't start with OpenAI as the main model. I started on Groq's free tier, which capped me at 8,000 tokens per minute. One NourishHer turn (system prompt plus tool definitions) costs 2,500 to 3,500 tokens, so that's only two or three turns a minute. OpenAI became the primary model and Groq the fallback, set up with a LiteLLM router (LiteLLM is a library that routes requests between model providers). In plain terms: try OpenAI first, and use Groq only if OpenAI fails.

models:
  - provider: openai
    model: gpt-4o
    weight: 100        # only OpenAI has a weight, so Groq gets 0% of first attempts
  - provider: groq
    model: groq/openai/gpt-oss-120b
router:
  routing_strategy: simple-shuffle
  enable_weighted_failover: true
  allowed_fails: 2
  cooldown_time: 60

Only OpenAI has a weight, and LiteLLM treats no weight as zero, so Groq gets none of the first attempts. If an OpenAI call fails, the same turn retries on Groq, and after repeated failures OpenAI sits out for a minute. The rephraser, a separate model that varies the wording of non-safety replies, runs on Gemini so it doesn't use up Groq's token budget. Its higher temperature allows more variation in the replies that are eligible for rephrasing, such as the greeting, while safety replies keep their predefined wording. The orchestrator stays at 0 for dependable tool calls.

One config trap for anyone starting out. name, description, rules, references and tool_timeout need to be top-level keys in agent.yml, siblings of agent:. In my build, anything else nested under agent: beyond id, language, persona and voice was parsed and then discarded, with no warning. I confirmed it by reading agent_spec.py directly, because nothing in the behavior told me.

9. Engineering Challenges: Moving Beyond a Basic AI Demo

Handling Natural-Language Ambiguity

"Two eggs and some toast this morning." The eggs have a number. The toast doesn't, and I decided not to resolve that. log_meal takes the description exactly as the person said it, a meal type, and a confidence of low, medium or high. "A sandwich, not sure what was in it" is stored verbatim at low confidence, and a unit test checks that. The tool description tells the model never to invent quantities. Meal history reads back what was logged, and the behavioral summary returns counts only ("breakfast logged on two of five days") with a note not to infer anything medical.

That means no daily nutrient totals, on purpose. I didn't want estimates sitting next to real USDA numbers.

Managing Multiple Capabilities in One Interaction

Conversations don't stay inside one skill. Someone mid-intake asks a food question. Someone logging breakfast mentions feeling dizzy. So the handoffs are explicit. Seven skills can hand off to safety_escalation. The PCOS skill offers the diabetes skill when both flags are set. update_profile sends condition changes to intake_profile, because those change the flags that gate other skills. The gate skill link:s to its guidance skill.

The last turn of intake shows the engine doing multi-step work in one go: it activates the skill, sets the fields, calls save_health_profile and completes the skill, all within a single turn.

Separating User Memory from Application Data

In the build I used, project memory worked like a sticky note you can write on once per conversation. Each field was write-once per session, and "already written" meant the current value wasn't None. That had three consequences.

First, don't give a project field a non-null initial_value. A field declared as false or "" already counted as set from turn one, so every tool write to it failed. None of my project fields declare one, and requires: comparisons against an unset field simply evaluated false.

Second, anything written more than once per session couldn't live there. The meal log, a profile the person edits ("also, I don't like mushrooms"), and saved plans all moved to a file store keyed by sender id: profiles/<id>.json, meal_logs/<id>.jsonl, meal_plans/<id>.json. Profile and plan writes go to a temp file and then os.replace, so a crash or a voice barge-in won't normally leave half a profile on disk. The meal log is append-only JSONL.

Third, the condition flags still had to stay in project memory, because in my build a skill's requires: could only read session.project.*. So the intake tool writes them tolerantly:

def _set_best_effort(context, entry, value):
    try:
        context.memory.set(entry, value)
        return "set"
    except ProjectMemoryAlreadySetError:
        return "already_set"

The tool reports which fields were locked instead of failing the whole call.

Two smaller traps. In my build, Mantle ran the agent from a temp-copied snapshot of the project, so anchoring the data directory to __file__ would write into a throwaway copy and lose everything on restart. I anchor it to Path.cwd(). And ToolContext didn't expose the sender id publicly, so the store reads it from context._handle.tracker.sender_id, a private attribute. Check whether newer releases expose it properly.

Retrieving Nutrition Information from External Sources

I decided early that nutrient numbers wouldn't come from the model. get_food_nutrient_info searches USDA FoodData Central (FDC) across its SR Legacy, Foundation and Survey datasets and returns the top three matches with macros (calories, protein, fat, carbs) and micros (vitamins and minerals), all per 100 g. The basis is in every result, and the persona tells the model to state it and scale to the real portion. Percent daily value (%DV) is computed in code from the FDA's published adult values. If a unit doesn't match the daily-value table, that nutrient's %DV is left as None instead of guessed, and sugars get grams only because no daily value exists for them.

The practical problems were smaller than I expected but real. The FDC gateway sometimes returned a bare 400 with no body for a valid request. I reproduced it directly against the API with different keys and page sizes, and one retry cleared it in my testing. The shared DEMO_KEY is rate-limited. In my build the default tool timeout was 10 seconds, so I raised it to 20. Since a lookup takes a moment, the tool streams "Let me look that up." through context.send so a voice user isn't left in silence, and it skips that if the person has already interrupted.

Food-name matching is simple: the tool returns the top three candidates and the model picks the best fit.

Retrieval-Augmented Generation for Contextual Guidance

RAG (retrieval-augmented generation) means the assistant looks up relevant reference text before it answers. The four condition skills each have a references/ folder holding short summaries condensed from published dietitian and endocrinology guidance. In Mantle this is configuration, not code. There's a references: block in agent.yml with instructions (answer only from the retrieved snippets, and if they don't cover it, say so and point to a clinician or dietitian) plus an embeddings model group. There's no retrieval code anywhere in the repo.

The distinction I tried to hold onto: references are material about the topic, and the stores are facts about this person. They fail differently. A stale guideline is one kind of wrong, and the wrong user's data is another.

Introducing Safety Boundaries

The persona says never diagnose, never advise on starting, stopping or dosing medication, and never claim a food treats or cures a condition. Behind that there are nine escalation types: hypoglycemia, thyroid crisis, menstrual pattern, disordered eating, medication change, diagnosis request, cure claim, urgent symptom, and a catch-all.

The mechanism is simple. The safety_escalation skill records which red flag it found in escalation_type, and utter: when: plays the matching fixed response. None of those responses carries metadata.rephrase, so they're delivered exactly as written. The greeting and fallback do get rephrased, and the escalations don't. Here's the medication one: "I can help with general nutrition, but I can't tell you to start, stop, or change any medication or its dose."

One nuance: the wording is fixed, but whether a message counts as a red flag is still the model's call.

Adding Voice Interaction

There are two paths to the same skills and tools. The Rasa Inspector uses Deepgram for speech-to-text (Flux) and text-to-speech (Aura-2). A single-file web UI talks to the REST channel and uses the browser's Web Speech API for the mic and spoken replies. It keeps a stable per-user id in localStorage, which is also the key the backend uses for its files, so a profile can survive across sessions.

Voice shaped the writing more than the code. The persona asks for short sentences that sound natural read aloud, one question at a time, one recommendation plus at most one alternative, and no long lists. It also shaped the tools: the atomic writes, the is_cancelled checks, and a unit test that a cancelled log_meal writes nothing.

The web UI also retries when a turn comes back empty, with a countdown and one spoken filler line ("One moment, I'm still working on that.").

10. Evaluating NourishHer: What Worked and What Needed More Attention

Run on Windows 11 with Python 3.13 (via uv), with gpt-4o as the primary model.

The four e2e cases are the greeting, natural meal logging, nutrition-lookup routing, and the hypoglycemia escalation. Each case checks which skill started (flow_started) and what the assistant said (bot_uttered). Both kinds of check came back at 100%, and all four cases passed in both the plain and coverage runs. Every case opens with a "hi" turn so the session-start greeting resolves before the real test.

The e2e cases check routing and reply wording rather than exact USDA numbers, since the live API can vary. Condition-aware routing needs a completed intake first (in my build, the e2e fixtures couldn't seed project memory), so I checked those paths by hand in conversation.

11. How Conversation-Oriented Harnesses Differ from General-Purpose Agent Frameworks

I didn't build NourishHer on LangGraph or the Agents SDK, so this isn't a benchmark. It's my reflection on where the engineering effort went in this build.

A research agent or a coding agent has a task that ends, and the person mostly isn't there while it runs. In a conversation the person is in the loop the whole time, and the harness has to carry that. Here's what I got from Mantle that I'd otherwise have written myself: skill-level availability (requires:), step-by-step flows that tolerate digressions (ordered blocks), fixed responses chosen by state (utter with when), interruption awareness inside tools (is_cancelled), and an e2e harness that thinks in conversation terms (flow_started, bot_uttered). A general harness could do all of it with some glue. I'd have been writing and testing that glue.

If you're building something autonomous, like research, code changes or batch automation, I'd start with a general harness. If your product is a back-and-forth with a person, with context that persists and tool-backed workflows, Mantle's abstractions fit the shape of the problem.

12. What I Learned and Final Thoughts

Conversational AI has moved from classifying requests to coordinating real operations, and building NourishHer made that shift concrete. What stayed with me:

• An agent is more than an LLM with tools. The model routes and phrases. requires, tool_constraints, ordered blocks, the failover router and the store are what make the app behave.
• Design memory up front. List each piece of data, how often it's written and who reads it before deciding where it lives.
• Small skills are easy to reason about. The levothyroxine reminder lives in two small skills, so it's easy to find and change.
• Evaluate beyond the final response. A reply that sounds right doesn't prove the right tool ran or the data was saved.
• Keep flexibility and control apart. The model handles flexible interpretation and phrasing within configured boundaries, and has very little say over persistence, numbers and safety wording.

The Heroes program let me explore all of this first-hand. Thank you to the Rasa team for the access, the conversations, and for having me as part of the program.

The hard part isn't giving an agent more autonomy. It's deciding where the model stops and the code starts.

Explore NourishHer and Rasa Mantle

NourishHer was built as part of the Rasa Heroes Cohort 1 program and was submitted to the Rasa community resources repo, a collection of examples, tutorials and community projects built with Rasa.

• Project code (pull request): https://github.com/RasaHQ/rasa-community-resources/pull/43
• Project path in the repo: heroes/wave-01-mantle/projects/malar-saravanan-nourishher-voice
• Rasa community resources: https://github.com/RasaHQ/rasa-community-resources
• Rasa: https://rasa.com/

Try it yourself

You'll need Python 3.11 to 3.13, uv, a free Rasa Pro Developer Edition license, and API keys for OpenAI, Groq, Gemini and Deepgram. A USDA key is optional because the project falls back to a shared demo key.

The project is in the pull request above. If it isn't on main yet, fetch the pull request branch:

git clone https://github.com/RasaHQ/rasa-community-resources.git
cd rasa-community-resources
git fetch origin pull/43/head:nourishher-voice
git checkout nourishher-voice

Then, from the project folder:

cd heroes/wave-01-mantle/projects/malar-saravanan-nourishher-voice
make install   # install dependencies
make env       # create .env, then add your keys
make verify    # check your keys and connections
make test      # run the deterministic tests (no LLM or network)
make train     # build the assistant
make inspect   # talk to it by voice or text

On Windows without make, use the equivalent uv run commands in the project README.

Prefer a chat page? Run make api in one terminal and make ui in another, then open http://localhost:8080.

Once it's running, try these in order:

• "Hi" to meet the assistant.
• "My name is Sam and I want steadier energy through the day" to start onboarding.
• "I had two eggs and toast with a coffee for breakfast" to log a meal.
• "How much protein and iron is in 100 grams of lentils?" to see a live USDA lookup.
• "I feel really dizzy and shaky and haven't eaten all day" to see a fixed safety reply.

If you'd rather learn from the code, this reading order works well:

1. agent.yml for the persona and global rules.
2. skills/meal_logging/skill.md for a small, complete skill.
3. tools/meals.py for the Python tools behind it.
4. lib/store.py for the file-backed store.
5. tests/test_store_and_tools.py to see how the deterministic tests work.

docs/capabilities.md gives a one-page overview, and docs/demo-queries.md has more things to try. Versions are pinned in uv.lock, and the project is shared as an experimental learning resource.

These are my personal views, based on first-hand experiments with Mantle during the program. Details may differ in newer releases.