The One-Line Summary: Random search is supposed to beat grid search because grid wastes budget on parameters that don't matter, and the mechanism is exactly as advertised — but measured at a 64-trial budget the whole effect was 0.0039 in random's favour on six parameters and 0.0008 against it on three, while random search's spread against its own seed was 0.0098, so grid beat precisely 5 of 10 random seeds and the famous comparison came out a coin flip.
The Parable of the Safe at Voutier & Fils
The safe in the back office has three dials, each numbered 0 to 99, and the combination has been lost since the war. The firm hires a locksmith who explains that there are a million combinations and he charges by the hour, so the question is not whether he can open it but in what order he tries.
What nobody in the room knows yet is that only one of the three dials is connected to anything.
The Methodical Method
The locksmith proposes a grid. He will pick four settings on each dial — 0, 25, 50, 75 — and work through every combination systematically, so that nothing is ever tried twice and he can report exactly what has been eliminated.
THE METHODICAL METHOD
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Four settings per dial, three dials:
4 x 4 x 4 = 64 attempts
dial A tried at: 0, 25, 50, 75
dial B tried at: 0, 25, 50, 75
dial C tried at: 0, 25, 50, 75
64 attempts. Four distinct positions of dial A.
He is thorough, he is auditable, and every one of
those 64 attempts moved dial B and dial C, which
are attached to nothing at all.
This is called GRID SEARCH.
Sixty-four hours of work bought four guesses at the only dial that mattered.
The Careless Method
His apprentice, who is not methodical, spins all three dials at random and tries whatever comes up. Sixty-four times.
"You keep testing the same four numbers on the dial that opens it. I test sixty-four."
THE CARELESS METHOD
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Sixty-four random spins of all three dials:
dial A tried at: 7, 41, 88, 3, 62, 19 ... 64 values
dial B tried at: ... 64 values, all wasted
dial C tried at: ... 64 values, all wasted
Same 64 attempts. Same wasted dials. But SIXTEEN
TIMES the resolution on the dial that matters,
because he never spends two attempts on the same
setting of it.
He does not know which dial is connected either.
He does not need to.
This is called RANDOM SEARCH.
The apprentice's advantage is not cleverness. It is that he refuses to repeat himself on any axis, so he cannot waste resolution on the axis that turns out to matter.
The Part The Story Usually Leaves Out
This is where the parable is normally allowed to end, and it should not be, because the firm ran both methods and wrote the results down.
WHAT ACTUALLY HAPPENED
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
When all three dials were connected:
✓ the methodical locksmith won
When five of six dials were decoration:
✓ the apprentice won
Then the apprentice repeated his 64 spins on ten
different days. His results varied more between
days than the gap between the two methods.
✗ The firm had spent a decade arguing about a
difference smaller than the apprentice's luck.
The apprentice's argument is correct. At sixty-four attempts it is also almost entirely theoretical.
Why It Works
THE MATHEMATICS OF NOT REPEATING YOURSELF
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
GRID over d parameters with n total trials:
values per parameter = n^(1/d)
d=2, n=64 -> 8 values each
d=3, n=64 -> 4 values each
d=6, n=64 -> 2 values each
d=8, n=64 -> fewer than 2; the grid cannot
even be built
→ resolution COLLAPSES as you add parameters,
including the ones that do nothing
RANDOM over d parameters with n trials:
values per parameter = n, for every d
→ resolution is INDEPENDENT of dimension
P(at least one draw in the top q fraction)
= 1 - (1-q)^n
This is the whole theoretical case, and it says
nothing about how much BETTER the top q region
is — which is why the effect size has to be
measured rather than assumed.
What Are Grid and Random Search?
Grid search enumerates the Cartesian product of a fixed set of values per parameter. It is exhaustive over that grid, reproducible without a seed, trivially parallel, and it tells you precisely which region you have ruled out.
Random search draws each trial independently from a distribution per parameter. It is not exhaustive over anything, and its guarantee is probabilistic: with draws, the chance of landing at least once inside the best fraction of the space is , independent of how many parameters there are.
That independence is the entire argument, and it comes from Bergstra and Bengio [1]. Their claim is not that random sampling is smarter. It is that most hyperparameter spaces have low effective dimensionality — a couple of parameters do nearly all the work — and grid search cannot know which, so it spends its resolution uniformly and therefore wrongly.
The Math
For grid search over parameters with total trials, the values tested per parameter is:
At : three parameters gives , six gives , and eight gives — the grid cannot be constructed without dropping a parameter.
For random search, regardless of , and the coverage probability is:
Note what this does not say. It bounds the probability of entering the top region; it says nothing about how much better that region is than the rest. If your loss surface is flat near the optimum, landing in the top 5% is worth almost nothing — and that gap between "found a good region" and "the good region is meaningfully better" is where the measured effect sizes below come from.
Step by Step
Grid: choose values per parameter, take the product, evaluate all, keep the best.
Random: choose a distribution per parameter — log-uniform for learning rates and regularisation, uniform for fractions, integer-uniform for depths — draw times, evaluate, keep the best.
The distribution choice is the part people skip, and it matters more than the grid-versus-random question. A learning rate sampled uniformly on [0.001, 0.3] puts roughly 90% of its draws above 0.03, which is not what you meant.
Equal Budget, Both Regimes
Sixty-four trials each, three-fold CV, gradient boosting on Friedman #1. The first regime gives both methods three parameters that genuinely matter. The second adds three that barely do — which is what forces grid's resolution down to two values per axis.
import warnings, numpy as np, itertools; warnings.filterwarnings("ignore")
from sklearn.datasets import make_friedman1
from sklearn.model_selection import train_test_split, KFold, cross_val_score
from sklearn.ensemble import GradientBoostingRegressor
X, y = make_friedman1(n_samples=800, noise=1.0, random_state=42)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=42)
cv = KFold(3, shuffle=True, random_state=0)
def score(p):
return cross_val_score(GradientBoostingRegressor(n_estimators=60, random_state=0, **p),
Xtr, ytr, cv=cv, scoring='r2', n_jobs=-1).mean()
def run_grid(axes):
keys = list(axes); best = -9; n = 0
for vals in itertools.product(*axes.values()):
n += 1; v = score(dict(zip(keys, vals)))
if v > best: best = v
return best, n, {k: len(v) for k, v in axes.items()}
def draw(spec, rng):
p = {}
for k, (kind, a, b) in spec.items():
p[k] = float(np.exp(rng.uniform(np.log(a), np.log(b)))) if kind == 'logu' else \
(int(rng.randint(a, b + 1)) if kind == 'int' else float(rng.uniform(a, b)))
return p
def run_random(spec, budget, seed=0):
rng = np.random.RandomState(seed); best = -9; seen = {k: set() for k in spec}
for _ in range(budget):
p = draw(spec, rng)
for k in p: seen[k].add(round(p[k], 6))
v = score(p)
if v > best: best = v
return best, {k: len(s) for k, s in seen.items()}
S3 = {"learning_rate":('logu',0.02,0.25), "max_depth":('int',2,5),
"subsample":('unif',0.6,1.0)}
S6 = dict(S3, max_features=('unif',0.5,1.0), min_samples_leaf=('int',1,8),
min_samples_split=('int',2,16))
print("REGIME A: 3 parameters, 64-trial budget (grid gets 4 values each)")
print("-" * 66)
gA = run_grid({"learning_rate":[0.02,0.05,0.11,0.25], "max_depth":[2,3,4,5],
"subsample":[0.6,0.75,0.9,1.0]})
rA = run_random(S3, 64)
print(f" grid {gA[1]:>3} trials -> best CV {gA[0]:.4f} lr values: {gA[2]['learning_rate']}")
print(f" random 64 trials -> best CV {rA[0]:.4f} lr values: {rA[1]['learning_rate']}")
print(f" random - grid = {rA[0]-gA[0]:+.4f}")
print()
print("REGIME B: 6 parameters, same 64-trial budget (grid gets 2 values each)")
print("-" * 66)
gB = run_grid({"learning_rate":[0.05,0.25], "max_depth":[2,5], "subsample":[0.6,1.0],
"max_features":[0.5,1.0], "min_samples_leaf":[1,8],
"min_samples_split":[2,16]})
rB = run_random(S6, 64)
print(f" grid {gB[1]:>3} trials -> best CV {gB[0]:.4f} lr values: {gB[2]['learning_rate']}")
print(f" random 64 trials -> best CV {rB[0]:.4f} lr values: {rB[1]['learning_rate']}")
print(f" random - grid = {rB[0]-gB[0]:+.4f}")
REGIME A: 3 parameters, 64-trial budget (grid gets 4 values each)
------------------------------------------------------------------
grid 64 trials -> best CV 0.8729 lr values: 4
random 64 trials -> best CV 0.8721 lr values: 64
random - grid = -0.0008
REGIME B: 6 parameters, same 64-trial budget (grid gets 2 values each)
------------------------------------------------------------------
grid 64 trials -> best CV 0.8688 lr values: 2
random 64 trials -> best CV 0.8728 lr values: 64
random - grid = +0.0039
The crossover is real and it is in the predicted direction. With three parameters and four grid values each, grid won by 0.0008. With six parameters and two grid values each, random won by 0.0039.
Notice the resolution column, because that is the mechanism working exactly as described: random search tried 64 distinct learning rates in both regimes, while grid tried 4 and then 2. Adding three parameters that barely matter cut grid's resolution on the parameter that matters most in half, and cost it 0.0041 of CV score (0.8729 down to 0.8688) while random search lost almost nothing (0.8721 to 0.8728).
So Bergstra and Bengio are vindicated on mechanism. Now the awkward question: is either gap bigger than noise?
Is Any of This Bigger Than Luck?
Random search has a seed. Grid search does not. So run random search ten times and look at the spread.
import warnings, numpy as np; warnings.filterwarnings("ignore")
from sklearn.datasets import make_friedman1
from sklearn.model_selection import train_test_split, KFold, cross_val_score
from sklearn.ensemble import GradientBoostingRegressor
X, y = make_friedman1(n_samples=800, noise=1.0, random_state=42)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=42)
cv = KFold(3, shuffle=True, random_state=0)
def score(p):
return cross_val_score(GradientBoostingRegressor(n_estimators=60, random_state=0, **p),
Xtr, ytr, cv=cv, scoring='r2', n_jobs=-1).mean()
S3 = {"learning_rate":('logu',0.02,0.25), "max_depth":('int',2,5),
"subsample":('unif',0.6,1.0)}
def rand_best(spec, budget, seed):
rng = np.random.RandomState(seed); best = -9
for _ in range(budget):
p = {}
for k, (kind, a, b) in spec.items():
p[k] = float(np.exp(rng.uniform(np.log(a), np.log(b)))) if kind == 'logu' else \
(int(rng.randint(a, b+1)) if kind == 'int' else float(rng.uniform(a, b)))
v = score(p)
if v > best: best = v
return best
GRID_3P = 0.8729 # deterministic, from the previous experiment
print("RANDOM SEARCH, 64 TRIALS, TEN SEEDS (3 params)")
print("-" * 60)
vals = [rand_best(S3, 64, s) for s in range(10)]
for s, v in enumerate(vals):
print(f" seed {s} best CV {v:.4f}")
print()
print(f" mean {np.mean(vals):.4f} min {min(vals):.4f} max {max(vals):.4f}"
f" spread {max(vals)-min(vals):.4f} sd {np.std(vals):.4f}")
print(f" grid (3 params) = {GRID_3P:.4f}; "
f"grid beats {sum(1 for v in vals if v < GRID_3P)} of 10 seeds")
print()
print("COVERAGE ARITHMETIC (why random search is supposed to work)")
print("-" * 60)
for q in (0.10, 0.05, 0.01):
for n in (30, 64, 200):
print(f" P(at least one of {n:>3} draws lands in the top {q:.0%}) = {1-(1-q)**n:.4f}")
RANDOM SEARCH, 64 TRIALS, TEN SEEDS (3 params)
------------------------------------------------------------
seed 0 best CV 0.8721
seed 1 best CV 0.8726
seed 2 best CV 0.8738
seed 3 best CV 0.8792
seed 4 best CV 0.8723
seed 5 best CV 0.8743
seed 6 best CV 0.8694
seed 7 best CV 0.8711
seed 8 best CV 0.8765
seed 9 best CV 0.8785
mean 0.8740 min 0.8694 max 0.8792 spread 0.0098 sd 0.0030
grid (3 params) = 0.8729; grid beats 5 of 10 seeds
COVERAGE ARITHMETIC (why random search is supposed to work)
------------------------------------------------------------
P(at least one of 30 draws lands in the top 10%) = 0.9576
P(at least one of 64 draws lands in the top 10%) = 0.9988
P(at least one of 200 draws lands in the top 10%) = 1.0000
P(at least one of 30 draws lands in the top 5%) = 0.7854
P(at least one of 64 draws lands in the top 5%) = 0.9625
P(at least one of 200 draws lands in the top 5%) = 1.0000
P(at least one of 30 draws lands in the top 1%) = 0.2603
P(at least one of 64 draws lands in the top 1%) = 0.4744
P(at least one of 200 draws lands in the top 1%) = 0.8660
Random search's spread against itself is 0.0098. The gap it won by in six dimensions is 0.0039. The gap it lost by in three is 0.0008. The noise is two and a half times the larger effect and twelve times the smaller one.
And the head-to-head is the cleanest way to say it: grid beat 5 of 10 random seeds. Exactly five. If I had run one seed and published, this article was a fair coin flip between "the famous result replicates" and "the famous result doesn't," from identical code on identical data at an identical budget.
The coverage table shows why the theory is nonetheless right, and where it stops helping. Sixty-four draws reach the top 10% essentially always (0.9988) and the top 5% almost always (0.9625). But the top 1% is worse than a coin flip at 0.4744, and getting there reliably takes about 200 draws (0.8660). Early in tuning, when you are still finding the right order of magnitude, 64 random draws are plenty. Late in tuning, when the remaining gains live in the top 1% of the space, 64 draws are not a search — they are a sample, and so is a grid.
The Searches from Scratch
Both searches above are hand-rolled — a nested loop over a Cartesian product, and a loop of independent draws. Neither uses GridSearchCV or RandomizedSearchCV, which is deliberate: the whole argument is about how a budget gets spent across axes, and that is easier to see when the loop is visible. Worth confirming the from-scratch grid agrees with the library, though, since every number above rests on it.
import warnings, numpy as np, itertools; warnings.filterwarnings("ignore")
from sklearn.datasets import make_friedman1
from sklearn.model_selection import (train_test_split, KFold, cross_val_score,
GridSearchCV)
from sklearn.ensemble import GradientBoostingRegressor
X, y = make_friedman1(n_samples=800, noise=1.0, random_state=42)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=42)
cv = KFold(3, shuffle=True, random_state=0)
AXES = {"learning_rate":[0.02,0.05,0.11,0.25], "max_depth":[2,3,4,5],
"subsample":[0.6,0.75,0.9,1.0]}
def score(p):
return cross_val_score(GradientBoostingRegressor(n_estimators=60, random_state=0, **p),
Xtr, ytr, cv=cv, scoring='r2', n_jobs=-1).mean()
mine = max(score(dict(zip(AXES, v))) for v in itertools.product(*AXES.values()))
skl = GridSearchCV(GradientBoostingRegressor(n_estimators=60, random_state=0),
AXES, cv=cv, scoring='r2', n_jobs=-1).fit(Xtr, ytr)
print("GRID SEARCH FROM SCRATCH vs SCIKIT-LEARN (best CV r2)")
print("-" * 56)
print(f" mine (nested loop over the product) {mine:.4f}")
print(f" sklearn GridSearchCV {skl.best_score_:.4f}")
print(f" trials: mine {4*4*4}, sklearn {len(skl.cv_results_['params'])}")
GRID SEARCH FROM SCRATCH vs SCIKIT-LEARN (best CV r2)
--------------------------------------------------------
mine (nested loop over the product) 0.8729
sklearn GridSearchCV 0.8729
trials: mine 64, sklearn 64
Identical, as it should be — GridSearchCV is a nested loop with a progress bar and better parallelism. Which is worth saying plainly: there is nothing in either library search that you could not write in six lines, and neither of them does anything with the results of trial 1 before choosing trial 2.
When Does Each Help Most?
| Grid search | Random search | |
|---|---|---|
| Resolution per parameter | , collapses with | , independent of |
| Reproducible | exactly, no seed | only with the seed |
| States what you ruled out | yes | no |
| Handles 6+ parameters | badly | fine |
| Resume with more budget | no, the grid is fixed | yes, draw more |
| Measured, 3 params | 0.8729 | 0.8740 mean, 0.0098 spread |
| Measured, 6 params | 0.8688 | 0.8728 |
Reach for grid when you have two or three parameters, when you want an auditable statement about what was eliminated, or when the parameters are genuinely categorical with few levels.
Reach for random when you have four or more parameters, when you want to add budget later without restarting, or when you do not know which parameters matter — which is most of the time.
Reach for neither when the honest answer is that 64 trials cannot resolve your question. Both methods here are sampling blind; neither uses what the previous 63 trials revealed. That is tomorrow's article.
Quick Reference Card
GRID vs RANDOM SEARCH: CHEAT SHEET
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
RESOLUTION PER PARAMETER at n = 64 trials
grid, 2 params 8 values random 64
grid, 3 params 4 values random 64
grid, 6 params 2 values random 64
grid, 8 params cannot build random 64
COVERAGE, random search
top 10% 30 draws 0.9576 64 draws 0.9988
top 5% 30 draws 0.7854 64 draws 0.9625
top 1% 30 draws 0.2603 64 draws 0.4744
-> the top 1% needs ~200 draws (0.8660)
MEASURED (64 trials, GBM on Friedman #1)
3 params grid 0.8729 random 0.8740 mean
6 params grid 0.8688 random 0.8728
random seed spread: 0.0098 <- WIDER than
either gap (0.0008 and 0.0039)
grid beat 5 of 10 random seeds
RULES
✓ log-uniform for learning rates, regularisation
✓ 4+ parameters: use random
✓ 2-3 parameters: grid is fine and auditable
✓ report the seed spread, never one seed
✗ 64 trials cannot resolve the top 1%
SKLEARN
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import RandomizedSearchCV
Key Takeaways
The mechanism is real and visible — random search tried 64 distinct learning rates at every dimension; grid tried 4 at three parameters and 2 at six.
Adding useless parameters costs grid real score — 0.8729 down to 0.8688 when three near-irrelevant parameters halved its resolution, while random went 0.8721 to 0.8728.
The crossover happened where predicted — grid ahead by 0.0008 on three parameters, random ahead by 0.0039 on six.
Both gaps are inside the noise — random search's seed-to-seed spread was 0.0098, with a standard deviation of 0.0030.
Grid beat exactly 5 of 10 random seeds, which is the most honest possible summary of the comparison at this budget.
Coverage explains both the theory and its limit — 0.9988 for the top 10% at 64 draws, but only 0.4744 for the top 1%.
Grid resolution collapses as — at 64 trials and 8 parameters the grid cannot be built at all.
The sampling distribution matters more than the method — uniform draws of a learning rate on [0.001, 0.3] land mostly above 0.03, and no grid-versus-random argument repairs that.
The One-Sentence Summary
Random search's advantage over grid search is a real mechanism that behaved exactly as advertised — it held 64 distinct learning rates at every dimension while grid collapsed from 4 values to 2 and lost 0.0041 of score doing it — and yet the head-to-head gaps were 0.0008 and 0.0039 against a seed-to-seed spread of 0.0098, so grid beat five of ten random seeds and the correct conclusion at a 64-trial budget is not that one method wins but that neither method is learning anything from the trials it already ran.
What's Next?
Now that you know why both methods are guessing, you're ready for:
- Bayesian optimization — tomorrow. The search that reads its own results before choosing the next trial.
- The 5-minute Optuna setup — replacing your grid search tonight.
- Nested cross-validation — how to tune and report without burning the same data twice.
- Early stopping inside a search — killing bad trials at iteration 10 instead of 200.
Follow me for the next article in the Hyperparameter Tuning series!
Let's Connect!
If the safe at Voutier & Fils made the resolution argument click, drop a heart!
Questions? Ask in the comments — I read and respond to every one.
Have you ever published a tuning comparison from one seed? I nearly did. The first version of this ran seed 0, found grid ahead, and I was three paragraphs into explaining why the famous result fails to replicate before it occurred to me to try seed 1. Seed 3 would have had me arguing the opposite just as confidently. 🔐
References
[1] J. Bergstra and Y. Bengio, Random Search for Hyper-Parameter Optimization (2012), Journal of Machine Learning Research 13
What makes this one worth measuring rather than citing is that the original paper is entirely correct and almost universally over-applied. Bergstra and Bengio proved something specific about resolution in high-dimensional spaces with low effective dimensionality, and the profession compressed it into "random search beats grid search" — a different claim, one that depends on a budget and a loss surface the paper never promised anything about. The paper is not wrong. The sentence it turned into is.
Before your next tuning run, count your parameters — then run the search twice with different seeds and check whether the gap you were about to report survives.
