TL;DR: When I let an agent investigate a codebase, I make it cite the output of deterministic tools (ripgrep, ast-grep, Serena, CodeQL) as evidence, and I keep the LLM's own analysis clearly separate in the report.
"If an external interface of this system changes, where do I need to fix things?"
I wanted to be able to answer questions like this when I face a codebase I don't know.
I plan to let an agent do the investigation. If I just make it read everything, the context overflows (I guess most of you have seen this already?). I also want the results to be reproducible.
So I looked for tools that let me make the agent cite the output of static analysis tools as its evidence. I sorted out the field of static analysis and its tools, and narrowed them down by how easy they are to adopt and how much they cover.
Initial direction
After a little research, my main policy was:
- Decide the investigation steps, based on investigations I have done before.
- Prepare tools that do deterministic computation for those steps, and base the investigation on their output as much as possible.
The static analysis tools I chose were ripgrep, ast-grep, Serena, and CodeQL.
The investigation flow and the tools I chose
First, I thought about how I would investigate this myself. There seemed to be a few cases: going from the big picture to the details, finding every string that matches, and listing the code that refers to something.
After thinking about it, I decided this combination could cover a fairly wide range.
- ripgrep: A fast and handy version of grep, the programmer's best friend. It is good at finding names and strings exhaustively, and at double-checking the results of other tools. On the other hand, it is a little tricky when you want to tell comments and string literals apart from code. When I want to narrow things down by syntax, I use ast-grep. When I need types, I use Serena.
- ast-grep: It searches the nodes of an AST (abstract syntax tree). Search queries can be built flexibly. grep may be enough, but structural matching might make it easy to list things like external communication, so I included it. When you hear "AST", you may worry about information lost in the conversion (comments, macros, preprocessor). But this tool actually uses a CST (concrete syntax tree) inside, so it can search comments without trouble. It can also extract macro and preprocessor definitions, although it does not expand them. It looks like it could support other languages by building a parser with Tree-sitter, the parser library it uses underneath (not verified).
- Serena: It offers, through MCP, the language server (LSP) features that run behind an IDE's Go to Definition and Find Usages. It hands symbol-level operations to the agent. I think of it as a tool for digging deeper from a specific, small piece of code. Its strength is following accurate references with types resolved, but building the index takes a little time.
- CodeQL: It turns code into a database and offers many kinds of analysis. It seems to be used mainly to find vulnerabilities and errors, but it can also do data flow analysis and compute key metrics of a codebase. I included it to get the big picture. On the other hand, it needs the most preparation of the four.
However, each tool supports different languages, and old languages are hard.
In my case, some systems could use all the tools, and some could use only ripgrep. For the latter, as an experiment, I tried to see how far ripgrep alone could go.
What I learned by trying
This is only a small trial (I have not been able to use CodeQL yet), but here is what I learned.
Even without tools, the AI built its own tool. In an investigation of an old language, I told the AI to build the report on deterministic computation as much as possible. It wrote and ran its own analysis scripts, and used the results as evidence. Thinking about it, of course it can do that, but I was surprised. Note that a tool the AI builds also needs to be reviewed.
The difference between ripgrep and ast-grep was what I expected. I used both to detect external communication, and the results were different. When I looked into it, ripgrep had counted comments. I learned the strength of ast-grep, and it also made me think about how important it is to verify with more than one tool.
Serena stayed in a supporting role. In this simple trial, I only used it to cross-check against ast-grep and to complement the verification of results. I ran into a few problems, such as needing to convert character encodings and missing type information for external libraries. Still, I have some idea where it should help, in places I have not tried yet.
- A strict check of things ast-grep flagged, such as methods that are probably never called.
- Checking whether something is really used or not, using type information, for example resolving interface implementations in Java.
- Impact analysis for the original question at the top: what happens when this code changes?
I think Serena will be quite effective in these areas.
The LLM is worth using for reading, too. At first, I only had the LLM interpret results that were produced deterministically. But then the report becomes less useful. Using the LLM to interpret the real meaning of the code, the unwritten rules, and the flow of the whole program is valuable. The output needs review, but I decided the value is much bigger than that cost.
Choosing tools again for each investigation target looks better than fixing them. The source code and data differ from system to system. If I read Java, I want Java tools. If I read JSON, I want JSON tools. The best combination changes. A fixed set of tools is not enough, so I feel I need to survey a wider range of tools. What I am not sure about is how dynamic to make this, including the tool survey. Still, from my experience with LLMs so far, my intuition is that the stage of "finding tools" and the stage of "using tools" should be separate. How many tools to give the agent is also something I need to look into from now on.
With these findings, my current policy is as follows.
- Let the LLM write the report, but have it cite the results of tools that do deterministic computation as much as possible.
- In addition, include the LLM's own agentic code analysis in the report, but clearly separate it from the above.
- For tools the AI built, include in the report what they do, with their source code.
- Choose tools again for each investigation target. Keep tool survey and tool use separate, and define "which tool for what purpose" before starting the investigation.
- When one of the AI's own tools is used, look for other tools that do the same thing, and think about updating the tool definitions.
Thoughts
I had built RAG systems before, so I thought I understood how important evidence and citations are. But the idea of giving the LLM deterministic computation as tools, to narrow down the vagueness of its answers, is something I became aware of only now.
Until now, when a discussion went in this direction, I often thought "a workflow with LLM steps is the way to go". In this case, though, I really want the agent to choose tools and dig deeper on its own. But there are steps I never want it to skip. So I would need a process where the required steps run first, their results are handed to the agent, and then the agent investigates on its own. That is a workflow with an agent inside.
On the other hand, I also feel this might be over-engineering. My impression is that recent, reasonably smart models follow instructions better than humans do. As more people feel this, I think the concern about probabilistic answers will change little by little. As I wrote in my previous post, I think industry standards for how much to tolerate will appear in the next two to three years. Still, during this transition period, leaning to the safe side is the best practice.
From now on, how to make AI output convincing will probably be the key to its usefulness. How this will evolve is something I will keep thinking about through experiments, while watching the cost.
On top of that, I still think the first step to trust is to state your sources clearly.
[Source] ChatGPT
Originally published on my blog.
