Arid 2.0: From Fast Python Duplicate Detection to CI-Ready Tooling

Arid started because Pylint R0801 was too slow. Version 2.0 keeps the fast detector and builds the workflows around it.

A few months ago, I had a fairly simple problem: Pylint was too slow.

I use Pylint's R0801 duplicate-code detection on Polaris, a fairly large Python project I've been building. I also use Ruff, which handles most of the Python linting I care about and handles it very quickly.

Unfortunately, Ruff doesn't detect duplicate code. So every time I wanted that one check, I was back to waiting for Pylint.

Eventually I got tired of waiting and built Arid, a focused Python duplicate-code checker written in Rust. The idea was deliberately simple: do the job I was using Pylint R0801 for, do it accurately, and do it fast enough that I wouldn't mind running it all the time.

The first versions of Arid proved that idea worked. Arid 2.0 is about something different: what does a fast detector need around it before it becomes a tool you can comfortably build into real development workflows? That question ended up defining the release.

The Detector Didn't Need Reinventing

Major versions have a way of encouraging major rewrites: new architecture, new algorithm, new semantics. Everything is better because everything is new.

I didn't do that.

Arid 2.0 uses the same basic detection model as 1.2. It still detects exact duplicated Python source after configurable Python-aware normalization. It still reports DUP001. Comments, docstrings, imports, and function signatures can be excluded from duplicate identity. The detector isn't suddenly trying to find semantically equivalent code or fuzzy AST clones.

That's intentional. The detector was already solving the problem I wanted it to solve, so instead of redesigning the part that worked, I concentrated v2 on the things surrounding it: stable machine contracts, CI integration, baseline management, focused workflows, incomplete-analysis handling, project control, and better support for external tooling.

Arid 2.0 isn't really a new detector. It's the same detector with a much more useful development workflow around it.

Fast Still Matters

None of that would matter much if Arid stopped being fast.

The v2 performance campaign used the same pinned benchmark corpora and Hyperfine methodology used to qualify Arid 1.2. Against Pylint 4.0.6, running serially, Arid 2.0 measured:

Those aren't comparisons against an entire Pylint run. The benchmark isolates Pylint's duplicate-code functionality so the comparison is actually about the job Arid replaces.

I also compared v2 directly against the qualified Arid 1.2 implementation. The additional v2 functionality introduced only low-single-digit overhead across the canonical corpora.

Could I have spent more time trying to recover that few percent? Sure. Would anybody using Arid notice the difference between roughly 219x faster than Pylint and slightly more than 219x faster than Pylint? Probably not. At some point optimization becomes an excellent way to avoid working on things users actually need.

Stable Identity for a Finding

One of those things is identity.

Suppose Arid finds the same duplicated code today and tomorrow, but somebody inserts 20 lines near the top of the file. The physical line numbers changed.

The duplicate didn't.

Or perhaps a file moves to another directory. Maybe the order of occurrences changes, or the same duplicate appears in another file. If external tooling identifies findings using locations, those findings become surprisingly unstable.

Arid 2.0 gives every finding a versioned fingerprint:

arid-finding-v1:sha256:...

That fingerprint identifies the normalized duplicate content independently of path, physical line number, occurrence ordering and multiplicity, structural metadata, output format, and worker mode. The same identity is exposed in SARIF through a versioned partial fingerprint.

This isn't particularly exciting when you're looking at a CLI report. It becomes considerably more useful when a CI system, reporting service, coding agent, or other tool needs to reason about the same finding across multiple runs.

Focus the Report, Not the Analysis

Large projects create another problem. Sometimes I don't care about every duplicate in the repository. I'm working on one package, directory, or file and want to know what's relevant to the thing I'm changing.

The obvious implementation is to scan only that path, but that's wrong for duplicate detection.

Suppose I'm working in:

src/payments/

and some code there duplicates code in:

src/customers/

If I analyze only src/payments/, I've removed half of the evidence.

Arid 2.0 therefore separates what gets analyzed from what gets reported:

arid . --focus src/payments

Arid still performs whole-corpus duplicate detection. Baseline enforcement still happens against the complete result, and only afterward does focus filtering determine which groups are reported. If a focused finding also occurs outside the focused path, those occurrences remain part of the finding.

In other words, focus changes what you ask Arid to show you without changing the corpus Arid uses to determine whether the code is duplicated. That's an important distinction for CI jobs and coding agents operating on a specific part of a larger repository.

Existing Duplicate Debt Is a Lifecycle

Baselines were already part of Arid before v2. The idea is straightforward: perhaps you're introducing duplicate-code enforcement into a mature project that already has 300 duplicate groups.

You could fix all 300 before adopting the tool.

Or don't adopt the tool.

Neither is especially compelling.

A baseline gives you a third option: accept the existing debt temporarily while preventing new duplicate debt from being introduced. Arid 2.0 extends that into an actual lifecycle.

arid . --baseline-status arid-baseline.json

can distinguish accepted duplicate debt, active/new findings, and stale baseline entries. Then:

arid . --prune-baseline arid-baseline.json

removes stale acceptance when the corresponding duplication no longer exists. It never silently accepts new debt.

That gives a team a useful progression:

existing duplication
        ↓
baseline it
        ↓
prevent new duplication
        ↓
refactor existing duplication over time
        ↓
prune stale baseline entries
        ↓
smaller baseline

You don't have to make an old codebase perfect before you're allowed to stop making it worse. I suspect that principle applies to considerably more than duplicate code.

Failure Doesn't Have to Mean "Tell Me Nothing"

Source analysis has another annoying edge case. Imagine scanning 3,000 Python files and one cannot be read, parsed, or normalized. Should the entire analysis disappear?

Sometimes yes. If you're enforcing a complete quality gate, an incomplete analysis cannot be treated as success. But that doesn't mean the useful results from the other 2,999 files need to vanish.

Arid 2.0 adds:

arid . --keep-going --json

Independent source failures are collected while valid files continue through detection. The important part is that Arid doesn't pretend partial analysis is complete analysis.

A report-v4 result explicitly says:

{
  "complete": false
}

and includes structured source errors. The process still exits with operational status 2, and incomplete reports cannot be emitted as SARIF.

The intent is simple: produce as much useful information as you safely can, but be explicit about how complete that information is. A human can inspect the partial result, and an automated consumer can make its own decision. Neither has to guess whether the analysis silently skipped something.

One Scan, Several Consumers

A CI pipeline often wants more than one representation of the same result. Maybe developers want readable console output, the build system wants JSON, GitHub code scanning wants SARIF, and the job summary wants Markdown.

The inefficient answer is to run the analyzer four times.

Arid 2.0 can instead produce multiple outputs from one in-memory report:

arid . \
  --format text \
  --report json=artifacts/arid.json \
  --report markdown=artifacts/arid.md \
  --report sarif=artifacts/arid.sarif

The source isn't reparsed four times and duplicate detection isn't repeated four times. The analysis happens once, and the result is rendered for the consumers that need it.

Obvious in retrospect? Probably. Still worth doing.

Machine-Readable Means Having a Contract

Once other software starts consuming CLI output, "it happens to be JSON" isn't enough.

Arid 2.0 introduces report schema v4 with explicit fields for things like the schema version, tool version, analysis metadata, completion state, structured errors, and finding fingerprints. The schema itself is published:

schemas/report-v4.schema.json

Arid also publishes contracts for capabilities and fatal JSON-mode operational errors. And:

arid --capabilities

allows tooling to discover deterministic build capabilities without first discovering or analyzing a project.

This is partly about ordinary CI integration, but there's another consumer I care about more now than I would have a few years ago: coding agents.

I use AI heavily in my own development workflow. An agent interacting with a tool shouldn't have to scrape human-readable console output and hope a sentence doesn't change in the next release. If we're increasingly going to have software using software on our behalf, the interfaces between those tools need to become more explicit, not less.

JSON gives us a machine-readable format. Publishing the schema tells the consumer what that format actually promises.

Arid Now Has an Official GitHub Action

Of course, the easiest integration is the one you don't have to assemble yourself.

Arid 2.0 ships an official composite GitHub Action:

- uses: sponge-b0b/arid@v2.0.0
  with:
    paths: .

The Action installs the exact Arid release associated with its tag and performs one scan. It can expose core metrics as outputs, write a job summary, and produce SARIF when configured.

Before v2, Arid could certainly be used in CI. Now there's a supported integration that makes doing it considerably simpler.

Sometimes You Want to Analyze Code That Isn't on Disk

Arid normally discovers and analyzes Python files in a project, but editors, coding agents, and other tools frequently have source that doesn't exist on disk yet—or source that differs from what's currently there.

V2 adds virtual Python source through standard input:

cat src/example.py | arid . --stdin-path src/example.py

The virtual source goes through the same Python parser and normalizer as disk-backed source. If an equivalent disk path exists, the virtual version replaces it for that scan. Otherwise it can be added to the corpus when the resolved project context permits it. Arid never writes that source to disk.

That means another tool can effectively ask, "If this were the contents of src/example.py, what duplicates would exist?" without first modifying the working tree. That's useful for editors and automation, and yes, it's particularly useful for coding agents.

Explicit When You Need It

Convention is great until automation needs certainty.

Arid still supports its existing nearest-config behavior, but v2 adds explicit control over project and configuration context:

arid . --config path/to/pyproject.toml
arid . --no-config
arid . --project-root path/to/project
arid . --show-config
arid . --list-files

For somebody running Arid manually in a normal repository, most of this can stay invisible. For CI, monorepos, editor integrations, and agents, being able to ask exactly which project and configuration are being used becomes considerably more important.

The common case can still rely on convenient defaults. The less common cases now have a way to be explicit.

Not Everything Became Public

There was one place where v2 deliberately became less extensible.

Arid is primarily a CLI application, but its Rust crate naturally exposes Rust code too. In v2, I narrowed the semver-supported Rust surface to a small crate-root application API. Implementation modules, detector internals, and reporting internals are no longer promises to downstream Rust consumers.

That decision fits something I've been thinking about a lot lately: every public interface creates an obligation. Once implementation details become supported API, changing your own internals becomes somebody else's breaking change.

If Arid were intended to be a general duplicate-detection framework, that would be a different conversation. It isn't, and not everything another developer could call needs to become something they should depend on.

What Didn't Change May Matter More

For a major release, the compatibility list is almost as important as the feature list.

Arid 2.0 preserves the things ordinary users depend on:

• exact normalized duplicate semantics;
• DUP001;
• normal CLI invocation;
• [tool.arid] configuration;
• normalization behavior;
• source suppression;
• existing baseline-v1 files;
• serial execution by default;
• worker controls;
• the 0 / 1 / 2 exit meanings;
• pre-commit integration;
• supported release platforms.

For CLI-only users who don't consume Arid's machine contracts or Rust internals, upgrading from 1.2 may require no migration work at all.

The intentional breaking changes are concentrated where a major version gives us room to make contracts cleaner: report JSON, SARIF finding identity, and the supported Rust API. That's the kind of major version I prefer: break what you have a good reason to break and leave everything else alone.

I Tried It on Real Projects

I don't want Arid's correctness story to be based entirely on unit tests and a repository containing six carefully selected Python files.

The v2 validation campaign exercised Arid against Black, Django, mypy, Rich, Unicode and space-containing paths, and combinations of the new workflow features.

For equivalent settings, canonical duplicate groups from Arid 2.0 were compared directly with qualified Arid 1.2 results across Black, Django, mypy, and Rich. No detector-semantic regression was found.

Validation also covered focus behavior, baseline-before-focus ordering, virtual-source replacement without disk mutation, controlled malformed source with --keep-going, multi-output on Django, worker determinism, and the published GitHub Action.

Performance matters, but so does knowing that the fast answer is still the right answer.

Where Arid Fits

Arid isn't trying to replace Ruff. Quite the opposite.

My normal mental model is:

ruff check .
arid .

Ruff handles the broad Python linting problem extraordinarily well. Arid handles one problem Ruff currently doesn't: duplicate code.

Because Arid is deliberately focused on that problem, I can make decisions around its detection model, reporting, baselines, and automation without turning it into another general-purpose linter. That's still the philosophy behind v2:

Cleaner contracts. Better automation. Same focused detector.

Try It

With uv:

uv tool install "arid==2.0.0"
arid .

Or with pip:

python -m pip install "arid==2.0.0"
arid .

If you're already using Ruff and want duplicate-code detection, try Arid on a real project. If you're still running Pylint primarily because you need R0801, I'm particularly interested in what you think.

I'm also interested in people integrating static-analysis tools into CI, editors, or coding-agent workflows. A large part of Arid 2.0 exists because once a command-line tool starts participating in larger development systems, speed isn't the only thing that matters anymore.

Arid started because I didn't want to wait for Pylint.

Version 2.0 is what happened after the detector became fast enough that speed stopped being the most interesting problem.

Arid is an open-source Python duplicate-code checker written in Rust. Version 2.0 is available now.

Documentation: Arid 2.0 release notes

About the Author

Bob Taylor is a software engineer and architect who builds developer tools and AI systems. He is currently developing Arid and Polaris.

GitHub