[Day 17] I analyzed 33,469 of my own AI conversations to audit how I actually use AI

Intro

Day 17!

Today I collected everything I've ever said to an AI and had a local model read it back to me. Seventeen months of history — then a self-audit of how I actually use these tools, and where the easy wins are.

What I used: DGX Spark (my home AI machine) / qwen2.5 (the local model doing the analysis).

Usage drifted from browser to terminal over time.

Where the history lives

ChatGPT (browser)

1. Open Settings from the account icon (top right)
2. Go to Data controls
3. Hit Export data → confirm
4. A link arrives by email; download the zip from there

Claude (browser)

1. Open Settings from your name (bottom left)
2. Go to Privacy
3. Hit Export data
4. A link arrives by email; download the zip from there

Neither one is instant — you wait a bit for the email.

Terminal (Claude Code, Codex CLI)

No request needed. The logs are already on your machine — just copy them.

# Claude Code
~/.claude/projects/<per-project>/*.jsonl

# Codex CLI
~/.codex/sessions/<year>/<month>/<day>/*.jsonl

What you get from each

Note that the browser export does not contain your terminal history, and vice versa. If you work across two machines, collect from both.

Audit 1: what I use it for

First, what was I actually using AI for? I had qwen2.5 read each conversation and tag it one by one ("this is research," "this is coding").

The big ones were research & learning (291), writing & editing (236), coding (193), and setup & troubleshooting (151) — two-thirds of everything.

Split by tool, it separates cleanly:

Terminal is 70% code-related; browser is 40% research and writing. The topics themselves split neatly between the two tools.

One caveat: the terminal side is only 75 conversations, so read it as a rough tendency, not a precise number. (Why so few? See "Not keeping work data in the first place" below.)

Audit 2: how I ask

Next, how I phrase requests. I counted phrasing patterns as a proxy for care. The yardstick is Anthropic's AI Fluency Framework.

In the terminal, my instructions are much more detailed.

(A fair reading: the terminal is where I do code work, and code work naturally invites step-by-step instructions and verification — so this may be a difference in task, not in skill.)

Audit 3: which features I use

The terminal logs record what the AI actually did. Here's the operation history:

So the reality is: read, edit, run a command. Nothing fancy.

And here's how often I reached for the fancier stuff. Turns out I barely write my own skills...!

Agents and MCP I already lean on. Skills and plan mode? Almost untouched.

The easy wins

Three obvious places to improve:

1. Turn routine work into a Skill.

Zero of my own. My article pre-publish checklist runs the same way every time — I'd rather call it by name than re-ask by hand each time.

2. Use plan mode before big jobs.

Once in 17 months. I keep course-correcting mid-run; I'd rather see the plan first.

3. Move all coding into the terminal.

12.0% of it still happens in the browser, where the AI can't touch files directly — so I waste round-trips copy-pasting.

The details

What I counted, and what I left out

I counted only human↔AI text exchanges. I dropped:

• agent-*.jsonl / journal.jsonl — internal work logs the AI keeps
• Sub-agent (AI helper) messages, 5,542 of them — not typed by a human
• Auto-inserted text like <system-reminder> — same reason
• Image/attachment contents (563 .dat files) — text only this time

One thing I missed: Claude's design_chats (3 conversations) got skipped because my loader only reads conversations.json. It's just one message, so the numbers don't move — but I can't claim I read the entire export.

What counts as a "phrasing"

I picked these up with regular expressions:

It's counting surface phrasing, not actual competence — treat it as a rough proxy. The comparison used conversations from May 2026 onward, where I use both tools: 348 browser vs 102 terminal.

Not keeping work data in the first place

Nearly half the history was work. That isn't even really my data — it belongs to the other party.

At load time I classify by folder name; for work, I throw away the body and keep only a character count.

cat = classify(project)
# keep the body only when it's confirmed personal
body = (text or None) if cat in ("personal", "web") else None

The classification keywords are literally client/partner names, so they don't live in the code — they're read from a config file that never enters Git. The classifier itself is more sensitive than the data it classifies — an obvious point I only noticed during the pre-publish check.

Because of this design, I could only read 75 of the 12,530 terminal conversations. A design that protects me also trimmed my own analysis.

The measurement flipped three times

My first pass showed every metric dropping. Read naively: "I got sloppy." But it was the denominator's fault.

I couldn't compare past-me to present-me directly: the tools swapped underneath me, so a skill difference and a place difference get tangled together.

A junked experiment: blind-judging old vs new requests

I hid the dates and had qwen2.5 compare an old request against a new one. Result: old won 10, new won 3. But it was browser-vs-browser, so I was really comparing "browser when I used it seriously" against "browser after it became an afterthought." It says nothing about whether I improved. (I judged each pair twice with the order flipped, counting a win only when both agreed — a fix for the position bias I found in the Day 14 cat-meow quiz.)

Speed notes, since I got stuck:

Switching to 32B (19 GB) fit entirely on the GPU → a few seconds each. Fitting or not fitting changes the order of magnitude. And the scoring isn't very reliable: of 25 browser-vs-browser pairs only 16 were usable, and for browser-old vs terminal-new, 17 of 25 timed out. No cloud API was used, so this analysis cost $0 (electricity aside).

Sources

• The AI Fluency Framework — Rick Dakan, Joseph Feller, and Anthropic (CC BY-NC-SA 4.0): https://aifluencyframework.org/

Outro

Line up 17 months and you can see exactly what you were doing each month. Re-uploading all of that somewhere to analyze it feels like a bit much. Keeping it on my own machine is well suited to moments like this.

Soon, I'd like to build a Skill of my own.

Thanks for reading!