For about five years or so, I've reverse-engineered my way through building Vault secrets engines (AKA a type of plugin). While Vault has a number of them already built, I often want Vault to manage all of my secrets and it doesn't support a few of them. As a result, I end up building my own, such as ones to manage HCP Terraform API tokens or Vault as a session token service.
Unlike plugin development for Terraform, Vault does not have any frameworks to expedite plugin development. After forgetting multiple times how to build a Vault secrets engine, I ended up creating a tutorial with the steps and considerations.
This problem got me thinking: why can't I create a skill for a coding agent to help me build a Vault secrets engine? After all, I wrote the tutorial so it should not be so difficult to convince a coding agent to build it and not generate slop. I am not an expert in creating Vault secrets engines, but I do have experience reading through lots of Vault plugins and finding patterns in them!
Building designer skills
In my limited experience, I found that Vault secrets engine development involves 60% debating the expected workflow of the plugin, 30% writing code, and 10% testing and hoping it doesn't break anything. This breakdown reflects that service APIs for credentials issuance and revocation have different behavior. Furthermore, secrets take the form of certificates or other mechanisms that don't quite fit CRUD actions. Even the identity or group domain models using the credentials differ.
Three years ago, I had to review an existing secrets engine, find something similar to what I wanted, and reverse engineer it for the secrets engine I wanted. I hoped that a coding agent, like IBM Bob, would help with the mental load of scanning through API specifications and designing the plugin. Furthermore, I needed Bob to remember common patterns and "gotchas" for designing secrets engines.
Ideally, I would summarize all the collective wisdom of the engineers who built various Vault plugins over the years and put that into a skill. Turns out I could actually do that by exporting all of the RFCs (Request for Comments) designing various secrets engines into files and asking Bob to summarize and identify patterns for me. (Note: I had to extract the RFCs into PDFS and then use Docling to make it LLM-friendly. Thank you, Docling.)
It generated an essay for me, which I will not include here, but the key points at the end did help.
## Common Pitfalls
- Unintended revocation when generating new credentials
- ID confusion (multiple ID types in same system)
- Incomplete cleanup during revocation
- Scope creep in initial implementation
- Testing gaps with hosted-only services
How refreshing. I did recall scope creep and incomplete cleanup in my small sample size of secrets engines that I created. With this promising start, I drafted a vault-secrets-engine-designer skill to help design a secrets engine.
I decided to test the skill on Cloudflare API tokens, which do not have an official secrets engine. It started quite promisingly, in that Bob activated my skill and proceeded to research the Cloudflare API.
Things took an interesting turn when Bob opened a browser to navigate the API documentation.
RIP my tokens. Bob found the information it needed and went right into designing the interface.
The design phase turned out to be fairly quick, although I caught a few mistakes. A major one had to do with how Cloudflare API tokens have an expires_in field. Since Vault has a leasing system as well, these fields must match. During design, Bob concluded that if you renew the lease, the token will just have to expire with expires_in. I deemed this unacceptable behavior, so we went back to refining the design, such that the secrets engine updates the expires_in field any time the lease renews. I had Bob write the design to a file before moving on to the next task.
Trying to code
For the most part, I had to assume that a number of LLMs probably had some inkling of a tutorial for building a custom Vault secrets engine. Awesome. However, I wanted to make sure that I was clear on the steps I wanted Bob to take for building the secrets engine. Bring on yet another skill, the vault-secrets-engine-developer skill
Bob went through and wrote each file in the order that I found best to implement secrets engines. First, define the backend and configuration. Then, define the role that maps to identities. Finally, implement the logic for credentials rotation.
Again, a few mistakes during this phase. I was not careful enough in my initial design review and discovered that Bob added permission_groups to the role schema. Cloudflare's API has a set of policies you can define for the token but no top-level permission_groups. Oops. This was quite costly as a refactor (again, RIP my tokens).
After resolving this problem, Bob proceeded to generate and run unit tests. I appreciated this since I really did not enjoy generating test cases for secrets engines. I never remembered how to mock the plugin backends and various API calls, so this went much faster than expected.
Conclusion
Overall, the secrets engine works. However, what about token usage? I used about 2.8 million in designing and 4.1 million in developing. I asked Bob to provide me a breakdown. During the design phase, most of the tokens went toward browser research of the API and iterative feedback. During the development phase, most of the tokens went toward refactoring the schema based on the incorrect attributes. It turns out that if you do some work upfront, you could optimize your tokens. If I gave Bob a downloaded specification for Cloudflare's API token and reviewed the architecture more carefully, I could save myself a half million to 2 million tokens.
Did it make developing a secrets engine plugin faster? It did, albeit for a more straightforward use case with a fairly well-automated rotation workflow. The API to revoke and generate Cloudflare's API tokens follows a standard pattern for creating and deleting, which helped Bob and me plan and generate the secrets engine quickly. I have yet to try this with a more complicated use case, so the skills may need more refining. I'll give it a try again the next time I write a secrets engine, pending I still have tokens to use in my budget.
