Every site eventually faces the same small chore: ship a machine-readable list of the pages you actually want crawled. The output format is fixed — a strict XML schema defined in the original sitemaps protocol on sitemaps.org — but the way you get there is not. Engineers usually land on one of three paths: editing XML by hand, assembling URLs in a spreadsheet and exporting, or delegating the heavy lifting to a purpose-built online utility. Each path has honest trade-offs around correctness, repeatability, and how much it costs the team every quarter.
This guide walks through all three so you can pick the option that matches your site's size, your release cadence, and the people who will touch the file next.
Path 1: Writing the File by Hand
For a personal blog with twenty posts, hand-rolling a flat file is perfectly reasonable. You open your editor, type out a <urlset> wrapper, drop in <url> blocks with <loc> and <lastmod> children, and save the result with an .xml extension. The schema is small enough to keep in your head, and the validation surface is equally small.
Where this approach breaks down is the second time you need to update it. Hand-edited files drift: an editor renames a slug in the CMS but forgets the listing file, a staging URL escapes into production, an old campaign page keeps showing up months after the campaign ends. Once drift sets in, the file becomes a liability rather than an asset.
A hand-edited file also becomes a coordination problem the moment more than one person owns it. Two engineers editing the same file in parallel will overwrite each other's work unless someone introduces a merge step, and at that point you have built a worse version of Path 3.
Treat hand editing as a learning exercise or a one-time export, not as the steady-state production source for anything beyond a static site with a handful of pages.
Path 2: The Spreadsheet-and-Export Workflow
This is the path most small teams adopt without naming it. Someone owns a shared sheet with one row per public URL, plus columns for the canonical path, the last meaningful update date, and an optional priority or change-frequency hint. At release time, the sheet is exported to CSV, then transformed with a short script into the XML wrapper the protocol expects.
The spreadsheet pattern earns its keep because the data layer is now human-shaped. A content editor who has never opened a terminal can sort, filter, and curate the list. Engineers stop being a bottleneck for routine changes. Auditing which URLs are listed becomes a review of a sheet, which most teams already know how to review.
The cost lives in the export step. CSV is lossy around XML escaping, dates, and CDATA sections, so the transformation script must be careful with characters like & and < in URL parameters. Date columns tend to drift across timezones, and the canonical URL column tends to drift across editors who paste in different cases. None of these problems are fatal, but each one needs a rule, and the rules need to live somewhere a new team member can find them.
If your site has between fifty and a few thousand pages and ships on a regular cadence, this workflow scales further than people expect, provided you write the export script once and resist the temptation to "just fix it manually this once."
Path 3: A Purpose-Built Online Utility
When the URL count climbs past the point where a sheet is comfortable, or when the team does not want to own another script, a focused generator is the path of least resistance. You point it at a root URL, it walks the site, dedupes, and emits the file in the shape crawlers expect. The trade-off is that you are trusting an external service to read your site, so the evaluation criteria shift from "how clever is my script" to "is this service honest about its limits."
A useful evaluator covers three things before you commit:
- Scope control. Can you exclude query strings, tag pages, search results, or staging paths? A generator that emits every URL it finds will quietly double your listing with junk.
- Output shape. Does it produce a single file, split files, and an index file when the count crosses the protocol's 50,000-URL threshold? Does it let you inspect the result before download?
- Refresh story. Can you re-run the crawl without recreating the configuration from scratch?
For teams who want the concrete steps without writing their own walker, the practical XML sitemap guide on lizecheng.net walks through the workflow in detail and is a useful companion if you go this route.
The honest weakness of this path is portability. The crawl configuration lives in someone else's database. When the service changes pricing, shuts down, or simply returns nonsense one morning, your team needs a fallback. Keep a copy of the last good output in version control, and keep the schema knowledge inside the team rather than inside the vendor.
Picking the Right Path for Your Situation
Rather than treating the three options as a ladder, treat them as tools for different shapes of problem. The decision hinges on four concrete signals.
- How many URLs do you actually want listed? Under fifty favors hand editing. Fifty to a few thousand favors the spreadsheet pipeline. Above that, a generator starts to pay for itself.
- How often does the URL set change? A quarterly changelog can survive hand editing. A weekly release cadence will punish it.
- Who owns the decision of what gets listed? If the answer is "engineering," a script is fine. If the answer is "the content team," the spreadsheet pattern usually wins.
- What happens when the file is wrong? If a bad listing means a support ticket, you need a path with a quick audit trail. If a bad listing means a slightly stale index, the cheapest path is fine.
A simple rule of thumb: pick the cheapest path whose failure mode you have actually rehearsed.
Validation: The Step Everyone Skips
Whichever path you choose, the file must be validated before it ships. Crawlers are unforgiving about malformed markup, and the failure mode is silent — your pages just stop appearing in the index without an error message you would naturally find.
A practical validation checklist:
- The root element is
<urlset>and the namespace declaration matches the protocol. - Every
<loc>resolves to an HTTP 200, not a redirect. -
<lastmod>values are real dates in ISO 8601 form, not placeholders. - No URLs contain characters that need XML escaping without being escaped.
- The file size stays under the protocol's 50 MB uncompressed limit.
- If the file is split, an index file lists every part, and every part exists.
Tools for this step are well-trodden. The XML specification on W3C gives the formal grammar, and most editors with an XML mode will flag structural mistakes the moment you save. Treat validation as part of the production job, not as a separate concern.
Operational Habits That Save You Later
A sitemap that is correct on day one is not the same as a sitemap that stays correct. Three habits make the difference between a one-time project and an asset the team can rely on.
First, generate at release time, not on demand. If your listing file is regenerated as part of the deploy pipeline, it cannot drift away from the deployed site. If it lives outside the pipeline, drift is a question of when, not whether.
Second, log the inputs. Whether the input is a sheet, a database query, or a crawl configuration, the file's contents should be reproducible from those inputs alone. If someone asks "why is this URL listed," the answer should be a query, not a memory.
Third, watch the failure modes. After every regeneration, spot-check a handful of listed URLs against what the site actually serves. Catching a redirect loop or a stray staging host early is the difference between a ten-minute fix and a quiet quarter of degraded indexing.
Frequently asked questions
How often should the file be regenerated?
Treat the regeneration cadence as a function of your release cadence. If you ship weekly, regenerate weekly. If you ship continuously, regenerate on every deploy. The cost of regeneration should be near zero, which means the threshold for "is it worth running again" should also be near zero.
Do I need a sitemap if my site is small?
Yes, in the sense that the cost of producing one is so low that the only reason to skip it is laziness. Search engines do not penalize small sites for omitting one, but they do discover new content faster when the file is present. For a site under fifty pages, hand editing is fine and the maintenance burden is trivial.
What belongs in the listing and what should be excluded?
List canonical, indexable pages. Exclude paginated archives, internal search results, admin or login routes, thank-you pages from form submissions, and any URL that returns a redirect or a non-200 status. A listing full of low-value URLs dilutes the signal crawlers use to prioritize, so curation matters as much as completeness.
Where should the file live?
The conventional location is the site root, named sitemap.xml. If you split the listing into multiple files, host them at the same root and reference each part from a sitemapindex file. The location should be declared in your robots.txt so crawlers can find it without guesswork.
This article was drafted with AI assistance and reviewed for technical accuracy before publishing.
