📝 Growing pytest into an “Incident Prevention Log” — Stage 5: Questioning GREEN with Mutation Testing

Hello from Japan 🇯🇵

I'm a truck driver in Japan, teaching myself web application development mainly with Python while continuing to work full-time.

In my personal Flask application, I've been gradually growing pytest into more than just a tool for checking whether the application works.

I think of it as an "incident prevention log":

a regression suite that helps stop previously discovered failures and near misses from silently returning.

This is Stage 5, the final stage of that series.

This article was originally published in Japanese on Qiita and has been translated and adapted for DEV Community.

https://github.com/tosane932/sales_data_app

87 GREEN Tests Still Had Five Blind Spots

The pytest suite originally contained only:

3 passed

From there, I strengthened it step by step:

Stage 1:  3 → 9
Stage 2:  9 → 51
Stage 3: 51 → 69
Additional demo seed tests: 69 → 71
Stage 4: 71 → 87

Stage 5 took a slightly different direction.

Up through Stages 1–4, the process had mainly been:

There might be a problem
↓
Write a pytest test
↓
RED
↓
Fix the problem
↓
GREEN

But once all 87 tests were GREEN, one question came to mind:

Can I really trust that GREEN?

If the tests themselves have blind spots, broken code may still remain GREEN.

So in Stage 5, I decided to do the opposite:

temporarily break working code on purpose and check whether pytest actually turns RED.

This time, I borrowed the idea of falsification: instead of assuming that GREEN meant the tests were strong, I tried to disprove that assumption with manual mutation testing.

The result was:

Starting point: 87 passed

Mutations performed: 11

Initially KILLED:   6
Initially SURVIVED: 5

Strengthen pytest for the 5 survivors
↓
Apply the same mutations again
↓
Confirm all 5 now turn RED

Final result:
91 passed, 2 known warnings

So even with 87 GREEN tests, five kinds of failures could still slip through.

This article focuses mainly on those five survivors.

pytest Improvement Series

• Stage 1
  
  https://qiita.com/tosane932/items/f3de1e190873a90de39f

• Stage 2
  
  https://qiita.com/tosane932/items/b91261e7103df5792f7d

• Stage 3
  
  https://qiita.com/tosane932/items/6d1ca5490979c8cf9d62

• Stage 4
  
  https://qiita.com/tosane932/items/372270330e73583a227f

• Stage 5
  
  https://qiita.com/tosane932/items/85fd24c7baa6fe7c76a7

In Stage 5, I Started Questioning GREEN

When pytest is GREEN, it means:

the current implementation satisfied the conditions expressed by the tests that currently exist.

It does not mean:

GREEN = completely safe

For example, imagine this safe JavaScript code:

productName.textContent = item[0];

Suppose pytest only checks that:

textContent is used

Then someone adds:

productName.innerHTML = item[0];

directly after it.

The original textContent is still present.

So the test may still pass even though a dangerous HTML sink has been introduced.

In other words:

Safe code exists

and:

Dangerous code does not exist

are two different things.

Instead of only imagining this possibility, I decided to:

actually break the code and see what happened.

What Is Mutation Testing?

Mutation testing intentionally introduces small defects into otherwise working code.

For example, change:

if not product.is_active:

to:

if False:

Or take this query:

DailySales.query.filter_by(
    product_id=product.id,
    date=sale_date,
).first()

and remove:

date=sale_date

Then run pytest against the mutated code.

The result generally falls into two categories.

KILLED

A mutation is KILLED when introducing the defect causes pytest to turn RED.

Normal code
↓
GREEN

Mutation
↓
RED

That means the test suite can detect that failure.

SURVIVED

A mutation SURVIVES when pytest remains GREEN even after the defect is introduced.

Normal code
↓
GREEN

Mutation
↓
GREEN

That suggests the tests may not be able to distinguish:

the correct implementation from the broken implementation.

I did not introduce a dedicated mutation testing framework in this stage.

Instead, Codex and I selected representative defects and manually applied them:

one mutation at a time.

Because I Was Intentionally Breaking Working Code, I Defined Safety Rules First

Mutation testing means intentionally putting production code into a broken state.

So before starting, I defined strict boundaries:

Do not mutate main

Work only on feature/pytest-stage5

Only one mutation at a time

Do not connect to the production database

Do not connect to the development database

Do not connect to Render

Do not call the real Gemini API

Do not access external networks

Use only isolated pytest SQLite / temporary databases

Do not commit mutations

Do not push mutations

Always restore the original code after verification

If a mutation SURVIVED, I also did not start modifying pytest while the broken production code was still in place.

Every survivor followed this process:

Apply mutation
↓
Run pytest
↓
Confirm SURVIVED
↓
Restore normal code
↓
Add or strengthen pytest formally
↓
Confirm GREEN with normal code
↓
Apply the same mutation again
↓
Confirm RED
↓
Restore again
↓
Confirm GREEN with normal code

All five surviving mutations followed this same sequence.

The 11 Mutations

These are the 11 mutations I selected.

The first run produced:

KILLED   6
SURVIVED 5

The six mutations that were KILLED immediately were already detected by existing pytest coverage, so I did not add new test functions for them.

The rest of this article focuses mainly on:

the five mutations that survived.

Six Mutations Were KILLED Immediately

The six mutations detected by the existing suite were:

• #2 Disable CSRF protection
• #3 Remove session fingerprint mismatch rejection
• #4 Break year/month validation
• #5 Disable discontinued-product validation
• #6 Break the model UniqueConstraint
• #7 Break the migration UniqueConstraint

For example, in Mutation #2, I temporarily disabled CSRF protection.

Requests that should have been rejected with HTTP 400 were then able to reach application logic, and the existing tests turned RED.

For Mutation #5, I changed:

if not product.is_active:

to:

if False:

This allowed discontinued products to pass through sales processing.

The existing pytest suite detected that too.

Mutation #7 was especially interesting.

The migration originally had a unique constraint on:

product_id + date

I temporarily changed that to:

product_id + quantity

The migration itself still completed successfully.

But the pytest test that inspected the resulting schema detected that:

the required UniqueConstraint does not exist

and turned RED.

That confirmed another distinction:

"The migration completed successfully" and "the correct schema was created" are not the same thing.

SURVIVED #1: A Safe textContent Check Missed a Dangerous innerHTML

The first surviving mutation involved XSS protection.

The dynamic ranking displays product names using:

productName.textContent = item[0];

I temporarily added:

productName.innerHTML = item[0];

immediately after it.

So the mutated code looked like this:

productName.textContent = item[0];
productName.innerHTML = item[0];

The safe textContent line still existed.

But now the unsafe innerHTML assignment existed too.

I ran the existing XSS regression tests.

The result was:

GREEN.

The mutation SURVIVED.

Why Did It Survive?

The existing pytest test checked that:

textContent is used

But it did not check that:

innerHTML is not used

The test only proved:

Safe behavior exists

It did not prove:

Dangerous behavior does not also exist

So I strengthened the regression tests to also detect dangerous HTML sinks that could become XSS vectors when used with untrusted data, including:

• innerHTML
• outerHTML
• insertAdjacentHTML

After confirming GREEN with the normal implementation, I applied the exact same mutation again.

This time, pytest turned RED.

SURVIVED
↓
Strengthen assertion
↓
Apply same mutation again
↓
KILLED

The test count became:

87 → 88 tests

The first lesson was:

sometimes it's not enough to confirm that something safe exists. You also need to confirm that something dangerous does not exist.

The Second Wave Focused Only on Failures Existing pytest Might Miss

You can create almost unlimited mutations if you want to.

Change a condition.

Remove a return.

Disable exception handling.

Remove a database filter.

Add unsafe template behavior.

But the purpose of Stage 5 was not to maximize the number of mutations.

After the first wave, I inspected the code again without changing production code.

I selected four additional mutations where I thought:

The current fixture might not distinguish
correct code from broken code

The current assertion might miss this failure

This mutation might reveal something different
from the existing tests

Those became Mutations #8 through #11.

I decided these would be the final group for Stage 5.

The result:

all four SURVIVED.

This became the most important part of Stage 5 for me.

SURVIVED #8: Removing the year Filter Changed Nothing Because the Fixture Was Too Simple

The AI business-advice feature retrieves sales data for a selected year and month.

The normal code calls:

_get_sales_from_db(target_year, target_month)

I temporarily removed the year condition:

_get_sales_from_db(None, target_month)

So instead of asking for:

August 2026

the mutated code effectively asked for:

August from any year

I ran the existing pytest test.

The result:

GREEN.

The mutation SURVIVED.

The Problem Was the Fixture, Not the Production Code

At the time, the fixture only contained data for:

August 2026
July 2026

So even after removing the year condition:

month = August

still returned only the August 2026 data.

The correct code and the mutated code produced:

the same result with the existing fixture.

So I added:

August 2025

data to the fixture.

The product name was:

Previous-year August product

with quantity:

77

Then, when requesting August 2026, I strengthened the existing test with an assertion equivalent to:

assert "Previous-year August product" not in contents

Normal code:

GREEN.

Same mutation applied again:

RED.

The mutation was now KILLED.

Because I strengthened an existing test function instead of adding a new one, the total remained:

88 tests

What I Learned Here

Test data does not merely need to:

exist.

It needs enough discriminating power to produce different results between:

correct code

and:

broken code

A test can execute a line of code without proving:

that the condition on that line is actually necessary.

SURVIVED #9: Existing XSS Tests Missed Jinja |safe

The next mutation involved the dashboard's initial server-rendered ranking.

Normally, Jinja autoescapes:

{{ name }}

I temporarily changed it to:

{{ name | safe }}

Then I ran the existing XSS tests.

The result:

6 passed

The mutation SURVIVED.

Why Did It Survive Even Though I Already Had XSS Tests?

The existing tests focused mostly on:

• rankings generated later by JavaScript
• AI responses
• unsafe HTML sinks

But they did not directly verify:

the product name rendered by the server in the initial ranking.

So I inserted a product name like:

<em>HTML-like product name</em>

into the database.

If normal Jinja autoescaping is working, <em> should appear as text rather than becoming an actual HTML element.

The new pytest test verifies that:

• escaped text exists in the raw HTML
• .prod-name displays the original product-name text
• no em element appears inside .prod-name

Normal implementation:

GREEN.

Reapply the |safe mutation:

RED.

SURVIVED
↓
Add server-rendered XSS coverage
↓
KILLED

The test count became:

88 → 89 tests

This taught me that:

having something called an "XSS test" does not mean every rendering path is protected by that test.

SURVIVED #10: A Normal Login Fixture Could Never Produce a Missing-Fingerprint Session

The existing session tests verified whether an old session was rejected after administrator credentials changed.

But what happens if:

the fingerprint itself disappears from the session?

The normal implementation rejects a session if the fingerprint is missing.

So I introduced a mutation that treated a missing fingerprint as though it matched the current fingerprint.

Conceptually:

Fingerprint missing
↓
Pretend it matches the current fingerprint

The existing pytest tests returned:

2 passed

The mutation SURVIVED.

The Problem Was the "Normal Login" Fixture

The existing tests logged in normally.

A successful login naturally stores the fingerprint in the session.

So those tests never created:

a session where only the fingerprint is missing

For the new test, I performed a normal login and then used:

session_transaction()

to remove only the fingerprint from the session.

The Flask-Login _user_id remained.

Now the behavior differed:

Normal implementation:

redirect to login

Mutated implementation:

dashboard → 200

I confirmed GREEN with the normal code.

Then I applied the mutation again.

pytest turned RED.

KILLED.

89 → 90 tests

This showed me that:

fixtures created only through normal user flows may be unable to reproduce abnormal states.

SURVIVED #11: Removing the date Filter Changed Nothing Because There Was No Same-Product, Different-Date Data

This was the final mutation.

DailySales searches for the row to update with:

DailySales.query.filter_by(
    product_id=product.id,
    date=sale_date,
).first()

So both:

product
+
date

must match.

I temporarily removed the date condition:

DailySales.query.filter_by(
    product_id=product.id,
).first()

Then I ran the existing normal sales POST tests.

PASS.

I also ran the database UniqueConstraint tests.

PASS.

The mutation SURVIVED.

Again, the Fixture Was the Problem

The existing fixture did not contain:

sales for the same product on another date.

If a product only has:

August 2 sale

then removing the date filter may still return that exact same row by coincidence.

So I created an existing sales record:

2026-08-01
quantity=5

Then I POSTed:

2026-08-02
quantity=9

With the normal implementation, the expected result is:

August 1 → quantity=5
August 2 → quantity=9

With the mutated implementation:

August 1 → quantity=9

The bug causes:

• the historical August 1 sale to be incorrectly changed from 5 to 9
• the new August 2 sale to never be created

So the new pytest test verifies both:

The August 1 record remains unchanged

and

The August 2 record is created

Normal implementation:

GREEN.

Same mutation reapplied:

RED.

KILLED.

The total became:

90 → 91 tests

All Four Mutations in the Second Wave Initially SURVIVED

The second-wave results were:

This really stayed with me.

By the end of Stage 4, the suite had already grown to:

87 passed

Yet all four mutations I deliberately selected because:

"This might still be weak"

passed straight through the suite.

More tests can create a greater sense of confidence.

But:

There are 87 tests

and:

Those 87 tests can distinguish important failures

are not the same statement.

In the End, All 11 Selected Mutations Became Killable

I performed 11 mutations.

The first run produced:

KILLED   6
SURVIVED 5

The five survivors were:

#1
#8
#9
#10
#11

For each of them, I followed the full process:

Restore mutation
↓
Formally add or strengthen pytest
↓
Confirm GREEN with normal code
↓
Apply the same mutation again
↓
Confirm RED
↓
Restore again
↓
Confirm final GREEN

So for the 11 representative mutations I selected, pytest was ultimately able to KILL all of them.

But I want to be careful about what that means.

It does not mean:

Mutation Score for the whole application = 100%

I only mutated:

11 representative cases that I selected manually.

pytest Grew from 87 to 91 Tests

At the beginning of Stage 5:

87 tests

At the end:

91 tests

Only four new test functions were added.

Since I performed 11 mutations, I initially thought the suite might grow more.

But:

• six mutations were already KILLED by existing tests
• #8 only required strengthening the fixture in an existing test function
• new test functions were added for #1, #9, #10, and #11

So:

87 + 4 = 91

I actually think that was a good result.

If my goal had been:

Reach 100 tests

I might have added tests simply to make the number larger.

But the purpose of Stage 5 was not:

to increase the test count.

It was:

to evaluate the failure-detection ability of the GREEN tests I already had.

From the Original Three Tests to About 30 Times as Many

Before I started strengthening pytest:

3 tests

By the end:

Stage 1    9
Stage 2   51
Stage 3   69
Demo seed 71
Stage 4   87
Stage 5   91

Numerically, that is roughly 30 times the original count.

But after Stage 5, what matters more to me is not:

that I have 91 tests.

It is:

that I can explain which failures cause those tests to turn RED.

Stage 5 Made No Formal Production-Code Changes

The only files with formal changes from Stage 5 were:

test_ai_integration.py
test_auth.py
test_sales.py
test_xss_regressions.py

There were no formal production-code changes:

app.py        no formal diff
templates     no formal diff
models        no formal diff
migrations    no formal diff

Every temporary mutation was restored.

So Stage 5 did not strengthen the application by permanently changing production code.

Instead:

it strengthened pytest's ability to detect defects.

Five Things I Learned from Stage 5

1. GREEN Can Still Miss Broken Code

Five mutations actually passed through the suite while it remained GREEN.

So:

pytest is GREEN

does not guarantee:

No broken code exists

GREEN means:

the current tests did not detect a violation of the conditions they express.

2. Fixtures Need Discriminating Power

Mutation #8 could not be detected without data from the same month in another year.

Mutation #11 could not be detected without sales for the same product on another date.

So a fixture should not merely be:

data that lets the test run

It may need to be:

data that distinguishes correct behavior
from broken behavior

3. Assertions Sometimes Need to Check Both "Exists" and "Does Not Exist"

Mutation #1 showed this clearly.

I already verified that:

textContent is used

But that assertion still passed after dangerous innerHTML was added next to it.

So I needed to test both directions:

Safe behavior exists

and:

Dangerous behavior does not exist

4. Normal Fixtures Cannot Always Represent Abnormal States

Mutation #10 involved a session where only the fingerprint was missing.

A normal login always creates that fingerprint.

So I had to explicitly construct:

an abnormal session state

Normal user-flow fixtures alone do not always reach the failure condition you actually need to test.

5. RED Does Not Automatically Mean the Mutation Was KILLED

During mutation testing, I did not classify something as:

KILLED

just because pytest became RED.

I checked whether:

the intended mutation caused the intended assertion to fail.

If the test failed for an unrelated reason, that would not prove that the mutation itself had been detected.

The Scariest Part Wasn't Breaking the Code — It Was Accidentally Leaving It Broken

The part I was most careful about was not the mutation itself.

It was:

accidentally leaving intentionally broken code behind.

So every mutation followed the rule:

Do not commit
Do not push

And every cycle included:

Mutation
↓
pytest
↓
restore
↓
git diff
↓
pytest with normal code

Mutation testing is often described as:

"break it and see if the tests catch it."

But for me, reliably restoring the code is part of the same operation.

Final Result

The final pytest result was:

91 passed, 2 warnings

The two warnings are the previously known:

Flask-SQLAlchemy get_engine() DeprecationWarning

from:

migrations/env.py

They were not introduced by Stage 5.

I'm treating them as separate maintenance work, so I did not fix them as part of this stage.

This Still Does Not Mean the Application Is Completely Safe

I performed 11 mutations and eventually strengthened the suite so that all 11 selected failures could be detected.

But I still cannot say:

This application is completely safe.

There are many areas outside the scope of this work, including:

• PostgreSQL-specific behavior
• locking
• concurrency
• race conditions
• simultaneous POST requests
• duplicate submissions
• differences from the Render environment
• production environment variables
• the real Gemini API
• API quotas
• timeouts
• SDK changes
• real-browser DOM behavior
• CSP
• specifications I have not decided yet
• future code
• failures I have not imagined yet

And I did not mutate the entire application exhaustively.

What I verified was narrower:

Can pytest actually detect these 11 representative types of failure that I selected?

It Feels a Little Like Testing a Truck's Safety Equipment

I work as a truck driver.

A truck has many safety systems.

But simply saying:

The safety equipment exists

does not mean accidents can never happen.

When a dangerous situation occurs:

does that equipment actually work the way it is supposed to?

And:

am I becoming overconfident simply because the equipment exists?

I started seeing something similar with pytest.

It is not enough to say:

pytest exists
CI exists
authentication exists
CSRF protection exists
database constraints exist

I also need to ask:

if one of those things breaks, can the system actually detect that something is wrong?

And even excellent safety equipment does not automatically make every human decision safe.

Years of experience do not guarantee perfect judgment either.

Familiarity can sometimes lead to:

This is probably fine.

Software may have a similar trap:

All pytest tests are GREEN,
so it's probably fine.

If I stop thinking there, I may miss something important.

Not "It's Probably Fine," but "What If?"

In Stage 5, I used a mindset from my day job almost directly.

Instead of:

"It's probably fine."

I try to think:

"What if?"

For example:

pytest is GREEN,
but it might still miss a failure

Dangerous behavior might be added
next to safe behavior

The fixture might be too simple
to distinguish broken code
from correct code

A migration might succeed
while producing the wrong schema

I might only be testing normal login
and never checking an abnormal session

I thought through those possibilities and then:

introduced small defects to test them for real.

That was Stage 5.

Summary

Stage 5 moved the suite from:

87 passed
↓
91 passed

But the important part was not the additional four tests.

I introduced 11 representative defects.

The first run produced:

Initially KILLED   6
Initially SURVIVED 5

Investigating the five survivors showed that the problem was not always simply:

Not enough tests

The blind spots included:

Assertions that only checked one direction

Fixtures without enough comparison data

Server-rendered output not being tested

No abnormal-session fixture

No different-date sales data for the same product

For all five survivors, I completed this sequence:

Restore normal code
↓
Add or strengthen pytest
↓
GREEN with normal code
↓
Apply the same mutation again
↓
RED
↓
Restore again
↓
Final GREEN

So in the end:

pytest could KILL all 11 mutations selected for this stage.

Stage 1 began with only three pytest tests and the idea of turning them into a regression suite.

Stage 2 expanded into validation, database updates, and rollback behavior.

Stage 3 expanded into authentication, CSRF protection, and access control.

Stage 4 looked ahead to failures that had not happened yet, including migrations, AI-service failures, and session problems.

And Stage 5:

questioned the GREEN result itself.

The biggest lesson for me was:

GREEN ≠ proof of safety

GREEN means:

the current implementation satisfied the conditions expressed by the current tests.

That is why I think there is value in occasionally turning the question around:

Can this test actually detect the failure
it is supposed to detect?

pytest started with three tests.

It ended this series with:

91 tests

But now I care more about:

being able to explain what I can break to make those tests turn RED

than simply saying:

"I have 91 tests."

For me, this marks the end of Stage 5.

What began as a small list of normal behavior checks gradually became an:

"incident prevention log" that I also crash-tested to see whether it could actually detect failures.

pytest Improvement Series

• Stage 1 — From 3 Simple Tests to 9

• Stage 2 — Strengthening Sales and Product Registration

• Stage 3 — Authentication, CSRF, and Access Control

• Stage 4 — Testing Failure Scenarios

• Stage 5 — Questioning GREEN with Mutation Testing

https://github.com/tosane932/sales_data_app

https://qiita.com/tosane932