Automated Greenwashing Detection for Oil & Gas Sustainability Reporting

Stack: Python · React · Vercel | Live: claimify-esg.vercel.app

Abstract

Corporate sustainability reports are structurally difficult to audit. Claims range from specific quantified commitments ("reduced Scope 1 emissions by 37% against our 2019 baseline") to vague aspiration ("we support the energy transition"). Claimify is a RAG (Retrieval-Augmented Generation) pipeline that scores each claim against a curated NGO evidence corpus rather than relying on the LLM's training weights alone. Retrieval gives the model citable, up-to-date source material for each claim; generation produces a structured verdict grounded in that material, not in generalities about the sector.

The pipeline has five stages: PDF ingestion, NLP filtering (ClimateBERT + GPT-4o structured extraction), two-stage retrieval (SBERT shortlisting + cross-encoder reranking), LLM scoring (a structured two-step prompt), and materiality adjustment (category-specific multipliers). A separate Commitment Tracker layer reuses the same retrieval stack to score 2021 pledges against 2023–2025 NGO evidence, measuring how much follow-through actually happened.

The pipeline runs entirely offline. No API calls happen at read time. The frontend reads two pre-generated JSON files, served as static files on Vercel.

Pipeline stages:

PDF download → pdfminer parse → ClimateBERT filter → GPT-4o extract → SBERT + rerank → GPT-4o-mini score → Materiality weight → React / Vercel

§01 — Background and Design Decisions

Oil and gas sustainability reports are written for multiple audiences at once: shareholder, regulator, and general public. The same company can, on the same page, disclose a precise Scope 1 reduction figure and an unfalsifiable aspiration in the next paragraph. Distinguishing between these is not a keyword problem or a sentiment problem. It requires knowing whether a claim makes a falsifiable factual assertion and, separately, whether available evidence refutes it.

Existing approaches had a shared gap: they either flagged anything containing "net-zero" or "carbon neutral" (keyword matching) or relied on journalists manually cross-referencing company statements with NGO reports. Neither approach scales, and neither produces a traceable reasoning chain. This pipeline is the automation of that intermediate layer.

The choice to use RAG rather than a prompt-only classifier is deliberate. LLM training weights are frozen at a cutoff date and do not contain the specific Carbon Tracker reports, Reclaim Finance assessments, or InfluenceMap briefings that contradict individual company claims. Giving the model that material at inference time means every verdict can be traced back to specific retrieved documents, not to generalised knowledge about the sector.

Architectural decisions and their reasons

Platform note: The Commitment Tracker extracts historical claims in a background subprocess. On Windows, PyTorch DLL loading fails in that context. The Tracker substitutes a keyword regex filter for ClimateBERT. The main ingestion pipeline, which runs in the foreground, keeps ClimateBERT. This is a platform limitation, not a design choice.

§02 — System Architecture

Two input streams meet at the retrieval stage.

Claims stream: Sustainability PDFs → pdfminer.six → ClimateBERT filter → GPT-4o extract → claims.jsonl + SBERT embeddings

Evidence stream: NGO sources + Guardian API → HTTP/PDF scraping with applies_to tagging → 350-word chunks / 50-word overlap → SBERT corpus embeddings → corpus_vectors.npy

Both streams converge at:

1. SBERT ANN search (company-scoped, k=20) — cosine similarity over applies_to-filtered corpus subset
2. Cross-encoder rerank (ms-marco-MiniLM-L-6-v2) — pairwise (claim, evidence) attention scoring → top-5
3. GPT-4o-mini scorer — CLAIM + RETRIEVED EVIDENCE in prompt → verdict grounded in docs
4. Materiality weight → rationales.json → React / Vite / Vercel static

Commitment Tracker branch (separate): 2021 PDFs → keyword filter → 710 pledges → same SBERT + cross-encoder retrieval → GPT-4o-mini gap scoring

§03 — NLP Pipeline: Filtering and Extraction

Before retrieval can happen, the pipeline needs claims. A 300-page corporate sustainability report typically contains thousands of sentences. Most of them are financial tables, legal disclaimers, headers, and general corporate narrative that has nothing to do with climate commitments. Running GPT-4o over all of them would be slow, expensive, and would pull in a lot of noise. Three steps handle this: sentence splitting, ClimateBERT relevance filtering, and structured claim extraction.

ClimateBERT is the gating step. It runs locally, no API call required, and processes sentences in batches of 32. In practice it drops 60–70% of a typical report. Only what passes goes to GPT-4o for extraction.

ClimateBERT relevance filter

python
# nlp/relevance_filter.py

MODEL_NAME = (
    "climatebert/"
    "distilroberta-base-climate-detector"
)
BATCH_SIZE = 32

def predict_batch(sentences: list[str]) -> list[bool]:
    tokenizer, model = get_model()
    inputs = tokenizer(
        sentences,
        padding=True,
        truncation=True,
        max_length=256,
        return_tensors="pt"
    )
    with torch.no_grad():
        outputs = model(**inputs)
    preds = torch.argmax(outputs.logits, dim=-1).tolist()
    # label 1 = climate-relevant
    return [p == 1 for p in preds]