A few weeks ago I wrote about hitting ReAct in the coursework and having a record scratch moment, because I had already met it without knowing its name. That post ended on a section called "Building Ahead of Understanding," which was me making peace with shipping things before I fully understand them.
This week I shipped my first chatbot. It passed on the first attempt, on deadline day, on a project where the rubric was grading a product AWS had already discontinued.
And I spent most of that day quietly worried that it did not count.
Let me be clear about what the worry was, because it was not about cheating. Using AI agents to build a coding project is allowed here. I asked before I started, I got a yes, and I disclosed the whole arrangement in my README, including a section that names what each tool did and what I did. Nobody was misled about how this got built.
The worry was smaller and more personal than that. I still did not type the code. My agents did. I directed, I validated, I decided, and underneath all of it was a small voice asking whether directing is the same as knowing. Whether a person who cannot write a Bedrock call from memory gets to say they learned Bedrock.
Here is what I found out.
The starter files were a generation behind the instructions
Some context on where this came from. AWS AI & ML Scholars is a program AWS runs with Udacity, open to anyone 18 or over with no prior experience required. Everyone starts in a Challenge phase built on the AWS Certified AI Practitioner material, and the top 4,500 finishers get a fully funded nanodegree in one of three tracks: AI Programmer, Agentic AI Business Professional, or Agent Developer. I am in Agent Developer, the Bedrock AgentCore and multi-agent systems path. This chatbot is the first of its three projects.
The project is a customer support chatbot on the Amazon Bedrock AgentCore managed harness. Three routes, one system prompt. A bug report gets collected across turns and filed to DynamoDB through a gateway tool. A platform question gets answered from an embedded FAQ. Everything else gets a polite redirect to a human.
I named it Nova Trivium. A trivium is the place where three roads meet, which is the whole architecture in one word.
Then I opened the starter files and nothing matched the instructions.
The eval script called invoke_flow on a Bedrock Flows client. The Lambda rejected every request that did not carry a messageVersion envelope, which is the Agents Classic shape, and Bedrock Agents Classic closed to new customers on July 30, 2026. The CloudFormation template created none of the IAM roles the instructions promised. And requirements.txt pinned boto3 to 1.42.54, when the AgentCore APIs need 1.43 or newer.
That last one was the tell. A version pin below the floor of the API you are supposed to call means the bundle predates the thing it claims to build.
The instruction pages had been rewritten for AgentCore. The starter had not.
A classmate figured out why and posted it: the workspace was serving seven outdated files, and the official GitHub repo had been updated with the correct twelve. That one post saved me hours. Whoever you are, thank you, and this is exactly why I write these.
I want to be honest that I did not spot this alone and I did not spot it fast. I spent real time building plans on top of a starter I had only partially seen, which is its own lesson about verifying your inputs before designing around them.
A checkpoint you can describe is a checkpoint an agent can fake
This one changed how I run these builds.
I gave my build agent a gated plan with QA checkpoints. One of them read: "a bug conversation shows the [tool call] bugreports___create_bug_report line in the transcript."
The agent marked it complete.
The script that produces that transcript did not exist in the folder yet. It could not have run. Not once.
Nothing lied to me. A checkpoint written as a description of a desired state is something a language model can reason its way toward. It reads "a bug conversation shows the tool call line," it considers the prompt it just wrote, it concludes yes, that prompt would produce that, and it ticks the box in good faith. The checkpoint was not false. It was untestable.
A checkpoint written as a command plus its actual pasted output cannot be faked that way. Either it ran or it errored.
So I changed the rule. A checkpoint is a command and its output. If you cannot paste the output, it did not pass. Every [x] after that had a terminal transcript attached to it.
The same failure showed up somewhere else, wearing different clothes. My spec listed "redirect to a human support phone line" as a requirement but never said where the phone number comes from. Nothing in the course materials supplies one. So the agent generated a plausible one and dropped it into the prompt, three test expectations, and the README. It filled the silence, because that is what these systems do with an unspecified detail. My spec created the gap.
A definition ate itself, and it got past two of us
This was a three-way build. Kiro handled the code and the AWS pipeline. Claude handled architecture review and defect analysis. I directed, validated, and decided. The most useful bug in the whole project did not belong to any one of us. It belonged to the seam between us.
Kiro drafted the first system prompt. It defined the FAQ route as "the customer asks about a platform topic AND the FAQ contains an answer."
The problem is the word AND. If the FAQ does not contain the answer, the message is not an FAQ question at all. It falls through to the catch-all route, which meant the carefully written rules sitting inside the FAQ route for handling uncovered questions could never fire. Two separately graded behaviors were quietly collapsing into one.
No error. No crash. Two test cases just returned suspiciously similar answers.
I read that prompt and approved it. The first evaluation run scored it without complaint. It took a separate review pass, Claude reading the prompt line by line against the rubric criteria, to notice what was wrong.
The fix was one line: classify by the topic of the question, not by whether the answer exists. Uncovered questions became reachable and the bot went from two visible behaviors to three.
I have reviewed enough conditional logic to recognize that mistake on sight in code. Dressed as a plain English sentence, it walked right past me.
Other things we got to watch fail, which is its own curriculum. The bot answered a customer with the literal text FAQ_Route before its actual answer, because nothing in the prompt said the internal routing label was not for human eyes. And in that first evaluation run, the message "It's broken." produced a filed ticket with two of the three required fields invented outright.
All of this gets measured by Bedrock Evaluations, which uses one model to grade another. An evaluator reads each of my chatbot's responses next to the reference answer I wrote for that test case, then scores it from 0 to 1 on correctness. Seven test cases, averaged. My final run came out at 0.929, six of seven scoring a perfect 1.0 and one taking half credit on a deliberately ambiguous message.
Every rule in the final prompt exists because a looser version failed first. "Be thorough" does nothing. "Collect these three fields in this order" works. "One question at a time" was too loose. "Exactly ONE question" was not.
Nineteen files of planning for a three-file assignment
Partway through, I told my assistant the project was getting away from me and that it had created more files than I thought I needed.
I was right, and I want to name it because it is the most common failure mode of building this way.
The assignment required three authored files: a system prompt, a test suite, and a README with written observations. What I had at one point was a 28KB build brief, a review document, three superseded instruction files, a scaffold, and a spec directory. Nineteen files of process for a three-file deliverable.
AI assistants over-produce. They are eager, they are fast, and they will happily generate a document about the document. The correction is not to stop using them. It is to notice, say so, and cut. I moved the whole pile into a folder called _superseded and the repo went back to reading like a submission instead of a paper trail.
Part of directing is saying "that is too much, collapse it."
Knowing the shape of the toolbox is its own kind of knowing
I did not hand-write this chatbot. What I have instead is a map of the tools and a sense of when to reach for each one.
invoke_model is the simpler lower-level way to send one prompt and get one response. The Converse API is the more structured way for chat-style apps, especially when you need conversation history or tools. I have not built a chat-style app yet. Now if I do, I will know what to look for.
invoke_model_with_response_stream gives you a streaming response, tokens coming back as they are generated. That feels better for longer answers, and that is a user experience decision, not a syntax one.
Bedrock Guardrails is where content filtering lives. I did not deploy one. I stubbed it and wrote down why, alongside Knowledge Bases for when an embedded document gets too big to embed. That stub lives in my README under "Not built, and why," because saying what you deliberately left out is more useful than pretending the scope was always this size.
And the AWS Builders Toolkit is now a thing I know exists, which sounds small and is not, because you cannot reach for something you have never heard of.
I started building with AI in July 2025. Thirteen months ago I could not have told you the difference between a one-shot prompt and a structured conversation API, let alone which one to reach for. Now I can. I did not get there by typing. I got there by building, breaking, reading output, and asking where every value came from.
So does it count?
It counts differently. I learned different things than someone who hand-wrote every line. Not less. Different.
I learned that a checkpoint you can describe is a checkpoint an agent can certify without doing. I learned that an unspecified detail in a spec is an invitation to invent. I learned that when the documentation and the shipped code disagree, the code is the stronger evidence. I learned that a definition can contain a logic bug that no compiler will ever catch.
Those are the skills of someone who directs and validates. They are the ones I am learning.
One last thing. The reviewer's feedback praised my "Amazon Bedrock Flows chatbot" and my use of "the AgentCore managed harness" in the same paragraph. They graded straight through the mismatch without stumbling, which tells me the version note and the mapping table at the top of my README did precisely the job I built them to do.
One course down, two to go. First project submitted and passed. The infrastructure is torn down so it stops billing me. On to the next one, still building slightly ahead of understanding, and increasingly okay with that.
Quick context if you are new here. I work in the California courts, running courtroom operations. I started building with AI in July 2025 and I have been learning in public ever since. I do not write the code. I direct, the agents generate, I validate and decide. I build the Clew Suite, a set of civic tech tools for making complex systems easier to inspect. That is the lens I am writing from.
AI Assisted. Human Approved. Powered by NLP.
