Most Microsoft 365 tenants we are asked to look at have the same shape. A few thousand active documents that people use every week, and somewhere behind them a much larger pile that nobody has opened since the project closed. Old tender responses. Three versions of the same policy in three team sites. A OneDrive belonging to someone who left in 2022, still holding a CSV export of the customer database.
None of that is a crisis on any given day. It becomes one when a mailbox is compromised and you have to work out what the attacker could reach, or when someone asks you to prove that personal information has actually been deleted rather than merely forgotten. The honest answer in most organisations is that nobody knows what is in there.
This post is about how we build a scheduled audit that answers the question, and how we roll it out without deleting something a court later asks for.
Three signals worth checking on a schedule
Rather than trying to classify every file, we look for three patterns that reliably indicate a problem.
The first is subtree staleness. A single old file means nothing. A folder where nothing at any depth has been modified or opened in eighteen months, with no retention label and no legal hold, is a different animal. It is usually a finished project, and the whole subtree can be dealt with as one decision instead of four hundred.
The second is sharing links that outlived their purpose. Anonymous links created for a tender that closed two years ago. Guest accounts from a consultancy engagement that ended. Links with edit rights where view rights would have done. These are cheap to find because the permission graph is right there in the API, and they are the findings that most often make an IT manager sit up.
The third is personal-drive data that should not exist. System exports, copies of finance workbooks, HR documents in an individual's OneDrive rather than the site that governs them. This is the least popular part of the audit and usually the most valuable.
The architecture
The system is deliberately boring. Most of the work is enumeration and diffing, and only a thin slice of it needs a model.
- A scheduled job walks every site and drive through the Microsoft Graph, using delta queries so subsequent runs only pull what changed. Metadata only at this stage: path, size, owner, timestamps, sensitivity and retention labels, version count, permissions and sharing links.
- Each run is written as a snapshot. An embedded store with JSON support is enough for this. SQLite will accept the raw Graph JSON and let you index fields out of it with generated columns, so you can add a new attribute to the audit without rebuilding the schema.
- Deterministic rules run over the snapshot first, because they are free. Exact duplicates by content hash, stale subtrees, links with no expiry, external sharing on sites marked internal, orphaned drives.
- Near-duplicate detection uses embeddings rather than hashes. Two copies of the same contract that differ by a filename and a date field will never match on a hash, but they sit next to each other in vector space. That is the same property behind dense retrieval work such as HyDE: documents get compared by meaning, not by bytes.
- Content classification runs last and only over the ambiguous remainder. Files that need parsing before anything can be said about them go through a document parsing model, then a classifier that answers narrow questions: does this contain personal information, is it a record or a working copy, which business function does it belong to.
- Everything ends in a per-owner report and a proposed-actions table, which is where the rollout begins.
Diffing snapshots is the part people underestimate. A single run tells you the tenant is messy. Two runs a month apart tell you which parts of the mess are growing, and that is what gets budget.
-- folders where nothing in the subtree has changed in 18 months
SELECT folder_path,
COUNT(*) AS files,
SUM(size_bytes) / 1024 / 1024 AS mb,
MAX(modified) AS last_touched
FROM items
WHERE retention_label IS NULL
AND legal_hold = 0
GROUP BY folder_path
HAVING MAX(modified) < date('now', '-18 months')
ORDER BY mb DESC;
Where the model earns its place, and where it does not
Most findings come out of rules and joins. The model is worth paying for on the slice where a human would otherwise have to open the file: is this scanned PDF an executed contract or a draft, does this spreadsheet hold customer records, are these two documents the same thing.
Cost is manageable if you keep that slice small. Cohere prices its Parse model at USD $1.50 per 1,000 pages through its API, which is a useful anchor for what tenant-scale parsing costs; vendors are now pitching document intelligence at whole organisational corpora rather than single files. At that order of magnitude, parsing a hundred thousand pages is a few hundred dollars, and re-parsing them every night is a waste. Parse once, cache by content hash, and only reprocess when the hash changes.
Where the model does not belong is the delete decision. Classification is an input to a rule, not a substitute for one. If the pipeline cannot explain a proposed action in terms a records officer can read, it does not get to propose it.
Rollout: audit, dry run, then enforce
We have never had a good outcome from turning deletion on early. The sequence that works:
Audit only, for at least a month. The system reports and does nothing else. This is where you discover that the "stale" site is the one Finance opens each June, and that half the external links belong to an auditor who is still engaged.
Dry-run deletes, for several weeks. Every action the system would have taken is written to a ledger with its rule, its evidence and its owner. Owners get a weekly digest of what would have happened to their content. The rules get tuned against real objections rather than imagined ones. This phase is deliberately long, because seasonal work is invisible in a two-week window.
Enforcement, scoped narrowly and reversibly at first. Start with the actions nobody defends: expiring sharing links, removing guest access for closed engagements, archiving stale subtrees to cool storage rather than deleting them. Keep the recycle bin window generous and keep the ledger. Actual destruction should come last, run against retention labels, and generate a record of what was destroyed and under which rule.
Privacy and retention, in Australian terms
Two things matter here for Australian organisations.
The first is that the audit is the evidence. Retention policies are easy to write and hard to prove. A dated snapshot, a rule, a dry-run ledger and a destruction record together demonstrate that the policy is operating, which is a far better position than an assertion. If your organisation holds personal information it no longer needs for a permitted purpose, the audit is what tells you where it is, and the ledger is what shows you dealt with it. Take the specifics of retention periods from your privacy officer or legal counsel, not from the pipeline.
The second is that the audit is itself a privacy surface. Paths and filenames leak content, sometimes including names and case identifiers. The index is a concentrated map of everywhere sensitive material lives, so it needs the same controls as the data it describes: Australian region, tenant-controlled keys, restricted access, its own retention period. If you send content to a hosted model for classification, know which jurisdiction it lands in and what the vendor retains. Private or on-premise deployment is a real option for regulated workloads.
For health, legal and financial records, keep the pipeline administrative. Inventory, duplication, sharing exposure and retention status are safe ground. Any action touching clinical or case records goes to a named human for sign-off before it executes, and that sign-off is designed into the workflow, not bolted on as a warning.
Limitations worth stating up front
Access timestamps in Microsoft 365 are less reliable than they look, so treat staleness as a prompt for a human, not proof of abandonment. Near-duplicate detection has false positives on templated documents, where twenty tender responses are legitimately ninety per cent identical. Legal holds and matter-based retention must be resolved before any rule runs, because a wrong deletion there is far more expensive than a thousand stale files. And no audit fixes the behaviour that created the mess. Expiring sharing links by default and giving teams somewhere sensible to put finished work does more for hygiene over a year than any cleanup run.
This post is part of our Practical AI in Business Operations series, which covers the unglamorous internal systems where this kind of work tends to pay off first. A document hygiene audit is a good first project precisely because it is measurable: you can count the links you closed and the gigabytes you retired.
PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.
Originally published at picnet.com.au.
