freehire: An Open-Source Job Search Engine That Crawls Directly From Company Career Pages

Disclosure: I was offered credits on the freehire platform in exchange for writing this article. The assessment below is honest — including the limitations section — and the credits do not change what I checked or how I reported it.

What freehire actually is

freehire is an open-source job aggregator that crawls company career pages directly — no recruiter reposts, no aggregator middlemen, no dead links. Every listing links to the original posting on the company's own ATS. At the time of writing it indexes 3.1M+ live postings from 294,000+ companies across 92 ATS platforms.

It is MIT-licensed, self-hostable, built in Go with PostgreSQL + Meilisearch, and the public API needs no key.

curl 'https://freehire.me/api/v1/jobs?limit=1'

That returns a single job posting with its full wire shape — title, company, location, description, work mode, enrichment data, and a direct URL to the company's own posting page. No auth, no rate limit header needed for basic use.

The architecture: run-once workers, not daemons

The most interesting design decision in freehire is how it handles background work. It is not a monolith with a job queue daemon. The HTTP server (cmd/server) and the mail listener (cmd/mail-ingest) are the only two long-lived processes. Everything else — crawlers, enrichment, search index rebuilds, backfills — is a run-once worker that takes a DATABASE_URL, does one pass, and exits.

This means the system has no long-lived background runtime to manage. Coordination between workers happens entirely through Postgres via transactional outboxes: a worker writes its result and queues a follow-up action in the same transaction, so nothing gets lost if the worker crashes mid-run.

The cmd/ directory holds roughly sixty entry points. Each ATS provider (Greenhouse, Lever, Ashby, Workday, iCIMS, etc.) gets its own board file under sources/ — one YAML file per provider, one entry per company. A company on a platform freehire already crawls is a one-line addition. cmd/ingest takes one board file per run, so each provider crawls on its own schedule and a slow platform never blocks a fast one.

The search path: Meilisearch stores the full wire shape

The search path is the most-travelled code in the system, and its defining property is that a search result page is served without touching Postgres for the payload. The Meilisearch document is the public wire shape of a job — not a pointer to one. This means a search response needs no database round trip to render.

// internal/search/searchdrain — search index document embeds the full job view
type JobDocument struct {
    jobview.Job  // the same JSON shape the API returns
    // ...index-specific fields
}

The internal/job/jobview package owns the single JSON representation used by the list endpoint, the detail endpoint, and the search index. This means the API, the SPA, and the search index cannot drift apart — they all use the same struct.

One trade-off worth noting: the description field is capped at 1000 runes in the index to keep rebuild sizes manageable. The detail endpoint serves the full text. A separate GET /api/v1/agent/jobs/search endpoint rehydrates full descriptions from Postgres for programmatic consumers.

Deep pagination is refused rather than slowed — the window guard caps offset + limit at 10,000. The reported total can count higher, but you cannot reach beyond that offset.

Deduplication: one schema, one key

Every posting is normalized into a single schema regardless of source. The dedup key is jobs.UNIQUE (source, external_id), so re-running a crawl is free — an unchanged re-crawl only refreshes a last_seen timestamp. The system does not re-index the whole catalogue every few hours; it only pushes to Meilisearch when content actually changed.

This matters because Meilisearch re-merges its inverted index across the whole live index on every push — measured at 90–180 seconds per push on a ~2.7M-document index. The earlier design had ~169 independent per-board processes each pushing directly, which saturated host disk IO. The outbox pattern collapses many small pushes into few fat ones.

The ghost-job signal

Some postings stay open without being filled. freehire flags observable behavior — repost patterns, age, how a posting moves — and carries the evidence alongside the flag. When there is nothing to say, it says nothing.

The ghost-job signal combines structural evidence (about a posting's shape) with outcome evidence (from people who applied). Structural evidence alone can never produce the stronger claim — the system observes facts about a posting, never an employer's intent, and that constraint is enforced in code rather than in wording.

The ghost signal lookup is best-effort: a failure leaves the badge off the page rather than failing the search. Postgres is queried after the Meilisearch response, only for the ghost stamps on that page.

The CV workspace: an evidence-gated agent

The CV workspace is built on a rule that shapes every part of it: the agent may not write a claim the candidate has not made. This is not a prompt instruction — it is an evidence gate in the write path.

The experience bank is a durable store of the candidate's achievements. Every banked achievement records whether the candidate asserted it or the model inferred it. Only candidate-asserted achievements may be written into a CV. Unknown provenance fails closed.

There is exactly one writer — internal/candidate/cvedit. Nothing outside it writes a stored CV. Each edit records both what it did and what would undo it, written with the document in one transaction against a locked row. This also prevents two agent turns from interleaving on one CV.

CV rendering uses Typst templates embedded in the binary. The renderer shells out to the Typst CLI in a temporary root with system fonts disabled and bundled fonts staged in. Candidate data reaches it through a JSON file, never through command arguments.

The assistant: in-process, no shell, no outbound channel

The in-app assistant runs in the same process as the HTTP server. There is no external agent runtime, no shell access, and no credential minted for it. A tool receives the session owner's user ID and calls the same Go service the HTTP handler calls.

A turn is bounded twice — by tool-calling rounds and by the model client's per-call timeout. Both bounds are chosen server-side. Zero or negative values fall back to defaults rather than meaning "unbounded".

The mail tools are instructive about the boundary: no tool opens a message by ID (because that marks it read and an agent sweeping the backlog would zero the owner's unread count), and no tool sends mail (because message bodies are attacker-controlled text and the surest answer to prompt injection is no outbound channel).

The stack

The generated TypeScript contracts (web/src/lib/generated/contracts.ts) are produced by cmd/gen-contracts from the Go wire structs, so a value added in Go and missing from the SPA's maps is a TypeScript error rather than a blank cell that ships green.

Getting started locally

# Clone and run the whole stack
git clone https://github.com/strelov1/freehire.git
cd freehire
make up   # builds + starts: api, web, postgres, meilisearch, redis, minio

# Verify it is up
curl localhost:8080/health
curl localhost:8080/api/v1/jobs

Adding a company on a supported ATS is one line in the provider's board file under sources/. For example, to add a Greenhouse-hosted company:

# sources/greenhouse.yml
- name: Some Company
  board_id: 1234

Retiring a board means moving its line to sources/retired/, never deleting it — ingest takes one file by path, so the retirement is expressed by where the line lives.

Honest assessment: limitations and trade-offs

No semantic/hybrid search. A previous Meilisearch-backed jobs_semantic index was removed once its only two real consumers stopped needing a live index. Similar-job suggestions now read a precomputed nearest-neighbour lookup filled offline by cmd/similar-backfill. CV-based recommendations were dropped outright. If you want semantic job matching, you will need to build it yourself.

Facets are curated, not guessed. Every facet comes from a curated dictionary — skills, roles, locations. An unrecognised value produces no tag at all. The trade is deliberate: what a filter returns is right, at the cost of a posting phrasing something unusually falling outside it. This means the facet coverage is only as good as the dictionaries, which are community-maintained.

The API is keyless and public. This is a feature, not a limitation, but it means there is no per-user rate limiting on the public catalogue. The hosted instance at freehire.me handles this at the infrastructure level.

Self-hosting requires real infrastructure. You need Postgres, Meilisearch, Redis, and optionally an S3-compatible store. The Docker Compose setup works, but running it in production means managing those services. This is not a SQLite-single-binary project.

AI features need an LLM endpoint. CV tailoring, fit analysis, and the in-app assistant all require an OpenAI-compatible endpoint configured. Without it, those features stay disabled — like OAuth sign-in without provider credentials.

Why I think this project matters

Most job boards are black boxes. They scrape each other, resell postings, and insert themselves between you and the employer. freehire's approach — crawl the source, normalize, deduplicate, and link directly — is transparent by design. The code is open, the data is open, and adding a company is a one-line YAML change.

The architecture choices are also worth studying if you build systems that crawl and index large datasets. The transactional outbox pattern, the run-once worker model, and the decision to store the full wire shape in the search index are all ideas you can steal regardless of whether you are building a job board.

freehire is MIT-licensed and lives at github.com/strelov1/freehire. Try it live at freehire.me. The public API needs no key.