I experiment with skills often, constantly tweaking the ones I create and trying out skills from other people. I also run more than one agent setup, and some of my skills are tailored to specific projects. Keeping track of which skill lives where, and which copy is the latest, got messy fast.
So I figured I'd build a CLI tool to manage my skills. The idea is simple: a single store holds the current copy of a skill, and the tool tracks where each of them goes. I decided I'd call it grimoire, and I estimated that it would take me a weekend or two to build with Claude.
Twelve weekends later, it has ten commands, more than a thousand tests, and a pipeline that publishes it to npm. No project is as small as you initially think it will be, and an agent doesn't change that.
With the first version of the project out, here are five lessons I learned from building it, starting with the one I didn't see coming.
1. Tests and CI are the biggest win, not shipping speed
Most people celebrate the shipping speed of agentic coding. I agree that it's a huge advantage, but I think cheap tests and DevOps are the bigger win.
Tests and a solid CI/CD pipeline are the cornerstone of maintaining any project. They're also the most tedious parts. On solo projects, they often get deferred to the "later" that never comes. Which is understandable, since time is a limited resource, and features are the part we actually enjoy building. Tests, on the other hand, just take precious time away from the next feature.
Now, we get tests and CI/CD almost for free.
Setting up the testing framework and the CI pipeline for grimoire took about half a day combined. All it needed was a handful of ground rules in CLAUDE.md.
The pipeline tests the project on two operating systems and three Node versions, and it builds and publishes to npm. Additionally, thorough tests get written with every feature, because AI agents are phenomenal at spotting edge cases and tracking every code branch that needs testing.
With a foundation like this, I can take my time with features and still ship much faster than I used to. I can step away for weeks and still come back to a project that tells me if I broke something.
2. Remain in the loop, and hold the reins early
Despite free tests, I found that vibe coding doesn't work for a project you plan to maintain. I tried it in the beginning and committed the agent's work without real reviews. Within a couple of weeks, even navigating the file structure became a challenge, and obnoxiously long comments and entangled modules made the code nearly impossible to follow.
A project with spaghetti code only agents can follow is not your project anymore.
The way out was a complete refactor that took several days and touched 85 files. I restructured the whole thing with Claude's help, but my input shaped the structure and set the rules moving forward, including two global rules for myself:
- The agent is never allowed to commit anything without my explicit say-so.
- I never commit what I can't understand, if only at a high level.
Early on, there was a lot of back-and-forth. However, I found that remaining in the loop was an investment in a more streamlined workflow. As the project grew, patterns of my coding style and design preferences emerged, and LLMs love patterns. The more I worked on the project, the shorter the reviews became.
3. Avoid the Waterfall trap
AI agents make the Waterfall model tempting. It's easy to sit down one afternoon with your agent and design everything: features, architecture, system requirements, infrastructure, etc., and defer all implementation. The agent writes fast, and it gives you the illusion that it can see all the angles regardless of how large your design document becomes.
That's the trap I fell into. I spent the first day only discussing design decisions at every level with Opus until I ended up with a 600-line markdown file. I was confident this would work because this document meant that I had all the rules set up for the model, and the path clear. The problem was that the document also crippled my ability to lead the agent. The size of the document meant that keeping track of the decisions in it, and making any edit, required complete reliance on the model. Subtle contradictions kept creeping into the document, and every prompt had a bloated context window early on. What started as a means to ground the agent ended up as the main source of my problems.
The first thing I did during the refactor was mark the document deprecated. I restarted the work by defining the rules for the layers only. Then I designed and implemented vertical slices across the layers, feature by feature, until I had a working version. As I mentioned in lesson 2, the more I implemented, the more consistent the agent's output became.
Agents take away the friction of the Waterfall model, but never its pitfalls. Plan the first slice, ship it, and let what you learn shape the next one.
4. Plan with high reasoning, execute with low
Avoiding the Waterfall method doesn't mean planning less. Planning shifts to a per-feature basis. I found that implementing features gets more effective and more efficient when planning happens with higher reasoning, and implementation is run with lower reasoning on established plans.
My workflow now runs through three skills I wrote:
-
architectgrounds a feature's requirements into a design document, and breaks the work into high-level parts. -
pathfinderreads the design document and turns one part into a detailed, step-by-step implementation plan in structured JSON. -
roadrunnersimply executes the JSON plan one step at a time without attempting to work around unexpected blockers.
At the heart of my planning skills is Matt Pocock's grill-me skill. It catches misinterpretations of the initial prompt early, and helps me account for all the different angles while planning. By the time roadrunner starts, there are very few judgment calls left for it to get wrong.
Note: The efficiency claim is based on my experience, not on any tests I ran. My guess is that a plan needs fewer output tokens at high reasoning than the code would, but that's just a guess. What I can say for certain is that this is how I work every day, on Claude's $20 plan.
5. Your workflow is only as good as your skills
Ad-hoc prompting breeds inconsistency. It's better to establish a structured workflow, and define skills that you constantly tweak and optimize based on your usage. Other people's skills are incredibly useful too, and can be built upon when creating your skills.
A good rule of thumb: whenever you have to correct the agent the same way twice, that's a rule or a skill waiting to be written.
That's also why grimoire exists. Once you have skills tailored to your workflows, you want them everywhere you work, without copying folders around by hand.
My engineering skills are public in my skills repo. Take a look if you're interested, but keep in mind that none of them were built to be universally useful. Finally, if you juggle skills the way I do, give grimoire a try.
