CLI-Anything: Turning Desktop Software into Tools Your AI Agent Can Actually Use

If you have used an AI coding agent like Claude Code, Codex, or Cursor for more than a week, you have probably noticed something: agents are great in a terminal and pretty bad everywhere else.

Ask an agent to refactor a Python module and it does fine. Ask it to cut a video in Shotcut, add a layer in GIMP, or export a diagram from Draw.io, and things fall apart. The usual workaround is GUI automation: take a screenshot, guess where the button is, click, take another screenshot, repeat. It is slow, expensive in tokens, and breaks the moment a dialog moves.

CLI-Anything from the Data Intelligence Lab at the University of Hong Kong (HKUDS), takes a different route. Instead of teaching agents to use GUIs, it gives the software a proper command-line interface that agents can call directly.

This article covers what it is, how to try it quickly, and whether it is worth your time.

The core idea in one paragraph

Most "GUI" applications are really a user interface sitting on top of an engine. Blender has bpy. LibreOffice can run headless. Shotcut and Kdenlive render through MLT and melt. Inkscape works on SVG. CLI-Anything analyzes an application's codebase, finds that underlying engine, and generates a Python CLI (built on Click) that exposes the real functionality as commands. Every generated CLI supports --json output for agents, a human-readable mode for you, an interactive REPL, and project state with undo/redo.

So instead of an agent clicking through menus, it runs something like:

cli-anything-gimp project new --width 1920 --height 1080 -o poster.json
cli-anything-gimp --json layer add -n "Background" --type solid --color "#1a1a2e"

Structured input, structured output, no screenshots.

Two ways to use it

The project has two distinct entry points, and it helps to know which one you want before you start.

Most developers should start with the first one.

Option 1: Install ready-made CLIs with CLI-Hub

You need Python 3.10 or newer.

pip install cli-anything-hub

# See what is available
cli-hub list
cli-hub search image

# Install one and inspect it
cli-hub install gimp
cli-hub info gimp
cli-hub launch gimp

One thing that trips people up: these CLIs wrap the real software. If you install the Blender CLI, you still need Blender installed on your machine. The CLI is the interface, not a replacement for the application.

Let your agent pick the tool

If your agent supports skills (Claude Code, Codex, OpenClaw, and others), you can install a "meta-skill" that teaches it to search CLI-Hub, install what it needs, and read the usage docs on its own:

npx skills add HKUDS/CLI-Anything --skill cli-hub-meta-skill -g -y

Then just describe the task:

Find appropriate CLI software in CLI-Hub and complete the task: convert
all .docx files in ./reports to PDF and merge them into one file.

Each generated CLI also ships with a SKILL.md file describing its commands, so the agent gets documentation in a format it understands without you writing any prompts.

Option 2: Generate a CLI for something new

This is the more ambitious part of the project. You point the plugin at a codebase (a local folder or a GitHub URL), and your coding agent runs a seven-phase pipeline:

1. Analyze the source and map GUI actions to backend APIs
2. Design command groups, state model, and output formats
3. Implement the Click CLI with REPL, JSON output, and undo/redo
4. Plan tests in a TEST.md file
5. Write tests, both unit and end-to-end against the real software
6. Document the results
7. Publish a setup.py so the CLI installs onto your PATH

In Claude Code, setup looks like this:

/plugin marketplace add HKUDS/CLI-Anything
/plugin install cli-anything

/cli-anything ./gimp

You can also pass a repo URL, for example /cli-anything https://github.com/blender/blender.

When the first pass is done, the refine command looks for gaps between what the software can do and what the CLI covers, then adds commands and tests for them:

/cli-anything:refine ./gimp "batch processing and filters"

Then install and use the result like any Python package:

cd gimp/agent-harness && pip install -e .
cli-anything-gimp --help
cli-anything-gimp          # no arguments opens the REPL

The plugin is available for Claude Code, Cursor, Codex, OpenCode, GitHub Copilot CLI, Pi, Qodercli, OpenClaw, Hermes, and Reasonix. Several of these are marked experimental or community-maintained, so expect the Claude Code path to be the most polished.

A note for Windows users: Claude Code runs shell commands through bash, so install Git for Windows or use WSL, otherwise you will hit cygpath: command not found.

Where it is genuinely useful

A few use cases where this approach makes a lot of sense:

• Media and document pipelines. Batch converting documents with LibreOffice, rendering scenes in Blender, cutting and captioning video. These are tasks agents previously could not do reliably at all.
• Wrapping messy APIs. You can feed API docs or an SDK to the generator and get one stateful CLI instead of dozens of raw HTTP calls. Fewer tool calls usually means fewer tokens and fewer mistakes.
• Internal tools. If your team has an internal app with a decent codebase but no API, this is a fast way to make it scriptable for both agents and humans.
• Testing and benchmarks. Once software has a CLI, you can generate agent tasks and evaluators in code rather than through screen recordings.

The repo's demos are worth a look. One shows an agent building a full HTTPS handshake diagram in Draw.io purely through commands. Another shows an agent assembling a rover model in FreeCAD with a live preview updating at each step.

The honest caveats

The project is popular (over 51k stars at the time of writing) and moves fast, but go in with realistic expectations.

Generation is not free. The seven-phase pipeline is a long agent run: reading a large codebase, writing an implementation, then writing and running tests. On a big project like Blender, that is a lot of tokens and time. Treat the output as a solid first draft that you review, not a finished product.

Coverage depends on what was exposed. A CLI is only as good as the commands it has. When a task needs something the harness does not expose, the agent cannot fall back to "just clicking it" the way a GUI agent could. This is a real trade-off: structured interfaces are more reliable when they cover the task and more brittle when they do not. The refine command helps, but it is not magic.

Quality varies across the registry. The core harnesses (GIMP, Blender, LibreOffice, Inkscape, Audacity, and others) have substantial test suites, and the repo reports more than 2,000 passing tests overall. Newer community contributions are thinner. Check the test count and last update before relying on a specific one.

You still need the real software. This is a feature, since it means you get real rendering and not a toy reimplementation, but it also means setup involves installing the upstream app and sometimes configuring it (for example, an API key or a local server).

Mind what you install. Like any community registry, review what a CLI does before giving an agent permission to run it, especially ones that touch credentials, file systems, or remote services. The maintainers have been fixing issues like path injection and unsafe XML parsing, which is a good sign, but it also tells you the attack surface is real.

So, is it worth a try?

Yes, with a clear scope.

If your agent work touches anything outside the code editor, such as documents, media, diagrams, or a desktop app your team relies on, CLI-Hub is a low-effort win. A pip install, one cli-hub install, and your agent can do things it simply could not do before. That takes about ten minutes to evaluate.

The generator is worth trying if you have a specific tool you want agents to control and you are willing to review and refine the output. It is less appealing if you just want to experiment casually, because the token and time cost of a full generation run is real.

The bigger idea behind the project is also worth paying attention to: rather than making agents better at pretending to be humans in front of a screen, give software an interface designed for agents in the first place. Whether or not CLI-Anything becomes the standard way to do that, it is a practical, working example of the approach today.

A good first experiment:

pip install cli-anything-hub
cli-hub search office
cli-hub install libreoffice

Then ask your agent to convert and merge a folder of documents, and compare how that feels against doing it through GUI automation.

Links

• Repository: github.com/HKUDS/CLI-Anything