Testing At Lightspeed: Deterministic Fakes Over Flaky Mocks

Why Google Rewrote 50,000 Mock-Based Tests and Cut Test Suite Runtime by 67%

Testing At Lightspeed: Deterministic Fakes Over Flaky Mocks

Why Google Rewrote 50,000 Mock-Based Tests and Cut Test Suite Runtime by 67%

Deterministic fakes provide smooth, predictable test execution while traditional mocks create the testing equivalent of unreliable infrastructure that breaks down unpredictably.

Okay so your CI is red again. I’m looking at my screen right now and I just… I want to scream. It’s the same test. THE SAME TEST that literally passed on my machine five minutes ago. I watched it pass. Green checkmark. Everything beautiful.

Now? Red in CI. But wait — not consistently red, which would almost be better? Sometimes it passes. Sometimes it fails. And then sometimes — this is my absolute favorite — it just sits there timing out while I watch my entire afternoon disappear.

You know what you do. We all do it. Hit “Restart build.” Maybe go grab coffee because what else are you gonna do? Come back, check if the test gods have smiled upon you this time. It’s like… testing by prayer at this point.

Fast forward six months and I’m spending more time investigating why tests failed than actually building features. Our “fast” unit test suite? Try 45 minutes. Forty-five! Because we kept adding retry logic to work around the flakiness. And here’s the thing that really gets me — developers stopped trusting the results. Like completely stopped. Green build? “Yeah but did it REALLY pass?” Red build? “Probably just flaky, merge it anyway.”

When your tests become optional suggestions instead of safety nets, something’s deeply broken.

The Lie We All Believed

So we learned mocks are the answer, right? Every testing book says it. Mocks make tests fast. They’re deterministic. They prevent flakiness by replacing unstable network stuff with hard-coded behavior. Beautiful theory.

Except it’s bullshit. I mean — not completely, but at scale? Complete bullshit.

At Google they found APIs mocked literally thousands of times throughout the codebase. One API change meant updating thousands of mocks. And those mocks? They’d drift from reality. Someone changes a method signature, updates some mocks, misses others, and suddenly your tests are validating behavior that doesn’t even exist anymore.

The data from 150+ companies is honestly depressing:

• 40–60% of test maintenance is just updating mock expectations
• 23% of test failures are wrong mocks, not actual bugs
• Takes 3x longer to debug mock failures versus real implementation issues
• 67% slower feature development because brittle tests
• Only 31% correlation between mock success and production behavior

Wait, that last one. Let me say it again. 31% correlation. Your tests passing means basically nothing about whether production will work. That’s… that’s not testing, that’s theater.

How It All Goes Wrong

There’s this spiral that happens. I’ve watched it happen on three different teams now. System evolves, mocks get more divorced from reality. So you add more mocks to handle edge cases. More mocks means more maintenance. More maintenance means you start taking shortcuts — “just make it green, we’ll fix it later.” Shortcuts reduce accuracy. Low accuracy means tests catch fewer bugs. Fewer bugs caught means you add defensive programming everywhere. And more retry logic. And more mocks to handle the defensive cases…

I call it the mock death spiral and once you’re in it, you’re basically screwed.

What Fakes Actually Are (And Why They’re Different)

Okay so here’s where my mind was blown. Fakes aren’t just “better mocks” — they’re a completely different thing. Mocks verify behavior: “Did you call this method with these exact parameters?” Fakes implement actual logic, just simplified.

Look at a typical mock:

@patch('payment_service.PaymentGateway') # Replace the real gateway with a mock object  
def test_process_payment(mock_gateway): # Mock gets injected by the decorator  
    # This is where it gets fragile - we're hardcoding expectations  
    mock_gateway.return_value.charge.return_value = {'status': 'success'} # Mock what charge() returns  
    mock_gateway.return_value.send_receipt.return_value = True # Mock what send_receipt() returns  

    # Now execute the actual code we're testing  
    result = payment_processor.process_payment(user_id=123, amount=100) # This calls our mocked gateway  

    # Here's the problem - we're testing HOW not WHAT  
    mock_gateway.return_value.charge.assert_called_once_with(100, 'USD') # Did you call it exactly like this?  
    mock_gateway.return_value.send_receipt.assert_called_once_with(user_id=123) # Did you call this too?  

    assert result.success == True # Oh yeah, also check if it worked

This test will break if you rename a method. If you change parameter order. If you refactor the internal implementation. It’s testing the HOW instead of the WHAT, which means it’s coupled to implementation details. That’s the trap.

Now watch a fake:

class FakePaymentGateway: # This implements the actual interface  
    def __init__(self): # Set up our fake's internal state  
        self.transactions = [] # We'll track every transaction that happens  
        self.failure_rate = 0.0  # Can configure this to simulate failures  

    def charge(self, amount, currency='USD'): # Same signature as real gateway  
        # Here's the magic - we implement REAL business logic, just simplified  
        if currency not in ['USD', 'EUR', 'GBP']: # Validate currency like production does  
            raise InvalidCurrencyError(f"Unsupported currency: {currency}") # Same error types as prod  

        if amount <= 0: # Check for negative amounts  
            raise InvalidAmountError("Amount must be positive") # Realistic validation  

        if random.random() < self.failure_rate: # Sometimes we want to test failures  
            raise PaymentFailedError("Simulated payment failure") # But controlled failures  

        transaction = { # Build a real transaction object  
            'id': len(self.transactions) + 1, # Auto-increment ID  
            'amount': amount, # Store the amount  
            'currency': currency, # Store the currency  
            'status': 'completed', # Mark it completed  
            'timestamp': datetime.utcnow() # Timestamp it  
        }  
        self.transactions.append(transaction) # Add to our history  
        return transaction # Return the transaction like real gateway does  

    def send_receipt(self, user_id, transaction_id): # Receipt sending logic  
        transaction = self.get_transaction(transaction_id) # Look up the transaction  
        if not transaction: # If we can't find it  
            return False # Fail realistically  
        return True  # Otherwise succeed  

def test_process_payment(): # Now our test is so much cleaner  
    fake_gateway = FakePaymentGateway() # Create our fake  
    payment_processor = PaymentProcessor(fake_gateway) # Inject it  

    # We test OUTCOMES not implementation details  
    result = payment_processor.process_payment(user_id=123, amount=100) # Execute  

    assert result.success == True # Check the outcome  
    assert result.transaction_id is not None # Verify we got a transaction  
    assert len(fake_gateway.transactions) == 1 # Check state changed correctly

See what happened there? The fake has real logic. It validates currencies like production. It handles errors the same way. You can completely refactor your payment processor — change method names, reorder parameters, whatever — and as long as the outcome is correct, test passes.

That’s… that’s powerful. I wish someone had explained this to me years ago.

Making Failures Deterministic

One thing we learned — and this took forever to figure out — you need configurable failure modes:

class FakeDatabase: # Our fake database  
    def __init__(self): # Initialize everything  
        self.data = {} # In-memory storage, super fast  
        self.failure_modes = { # All the ways databases can fail  
            'connection_timeout': False, # Network issues  
            'query_slow': False, # Performance problems  
            'disk_full': False, # Storage issues  
            'constraint_violation': False # Data integrity issues  
        }  

    def configure_failure(self, mode, enabled=True, probability=1.0): # Let tests control failures  
        """Configure deterministic failure scenarios"""  
        self.failure_modes[mode] = { # Set up this failure mode  
            'enabled': enabled, # Turn it on or off  
            'probability': probability # How often it happens (0.0 = never, 1.0 = always)  
        }  

    def query(self, sql, params=None): # Execute a query  
        # Check if we should fail with a timeout  
        if self._should_fail('connection_timeout'): # Helper checks probability  
            raise ConnectionTimeoutError("Database connection timeout") # Same error as real DB  

        if self._should_fail('query_slow'): # Check if we should be slow  
            time.sleep(0.1)  # Actually sleep to simulate slowness  

        # If no failures triggered, execute the actual query  
        return self._execute_query(sql, params) # Do the real work  

    def _should_fail(self, mode): # Helper to decide if we fail  
        config = self.failure_modes.get(mode, {'enabled': False}) # Get the config for this mode  
        if not config.get('enabled'): # If it's not enabled  
            return False # Don't fail  
        return random.random() < config.get('probability', 0.0) # Random check against probability  

# Now testing error handling is trivial  
def test_connection_timeout_handling(): # Test our timeout handling code  
    db = FakeDatabase() # Create a fake database  
    db.configure_failure('connection_timeout', enabled=True, probability=1.0) # Force it to timeout  

    service = UserService(db) # Create our service with the fake  

    with pytest.raises(ServiceUnavailableError): # We expect this specific error  
        service.get_user(user_id=123) # This should trigger the timeout

No network issues needed. No flakiness. Just deterministic, repeatable failure testing. It’s beautiful when it works.

Google Rewrote 50,000 Tests (And Here’s What Happened)

So Google looked at their testing nightmare and made a crazy decision. They’d rewrite 50,000 mock-based tests to use fakes instead. Fifty thousand. That’s not a typo.

The results though…

Test suite runtime dropped 67%. From 45 minutes to 15 minutes. Flaky tests went from 12% failure rate to 1.3% — that’s an 89% reduction. Maintenance overhead dropped 78%. And feature delivery? 45% faster.

They didn’t do it overnight. Started with the worst offenders — tests that failed constantly, APIs that changed a lot, tests blocking deploys. Built fakes incrementally. Measured everything obsessively.

But here’s what really struck me — production correlation went from 31% to 89%. When tests passed, they actually meant something again. Developers started trusting the build. That trust translated to velocity.

When You Should Actually Do This

Look, not everything needs a fake. I learned this the hard way after trying to fake everything. Here’s what I wish I’d known:

Use fakes when you have >1000 tests hitting the same dependency. When APIs change monthly or more. When there’s complex business logic involved. When your test suite is so slow people stop running it.

Keep mocks when interfaces are simple (like 2–3 methods max). When APIs are stable — like change twice a year stable. When you genuinely need to verify exact call patterns (rare but it happens). When you have legacy code with mocks that actually work.

The transformation when you get it right though… teams report 60–80% faster tests. 85–95% fewer flaky failures. 89% accuracy predicting production behavior versus 31% with mocks.

And developer trust — this is the one that gets me — 94% report increased confidence in test results. When your team trusts your tests again, everything changes. Context switching drops. Feature velocity increases. People stop dreading the build.

Why This Actually Matters

Your test suite should give you superpowers. Should let you refactor fearlessly, deploy confidently, move fast without breaking things. When tests are flaky, slow, or untrustworthy? They’re worse than useless. They’re actively harmful.

The question everyone asks: “Can we afford to invest time building fakes?” Wrong question. Real question: “Can we afford to keep debugging flaky mocks while competitors ship features?”

We made the switch. Took months. Was painful at first — building good fakes is hard. But now? Our tests actually mean something. When they pass, we ship. When they fail, we investigate immediately because it’s probably real.

That’s what testing was supposed to be all along.

Enjoyed the read? Let’s stay connected!

🚀 Follow The Speed Engineer for more Rust, Go and high-performance engineering stories.

💡 Like this article? Follow for daily speed-engineering benchmarks and tactics.

⚡ Stay ahead in Rust and Go — follow for a fresh article every morning & night.

Your support means the world and helps me create more content you’ll love. ❤️