Descriptive stats, probability distributions, confidence intervals, hypothesis testing, and the mistakes that fool experienced engineers.
The Alert That Cried Wolf
A team at a mid-size SaaS company set up monitoring for their API response times. They configured an alert: "Trigger if average response time exceeds 500 ms over a 5-minute window." The alert fired constantly. Engineers sprinted to investigate, found nothing wrong, and started ignoring it. Two weeks later, a genuine outage went undetected for 20 minutes because everyone had learned to dismiss the alert.
The problem was the word "average." The average response time was 350 ms, but the distribution was heavily skewed: 95% of requests completed in under 200 ms, while 5% took over 2 seconds (timeouts and retries). The average hid the tail — the 5% of users having a terrible experience. When the team switched to alerting on the 95th percentile (p95) of response time instead of the mean, the false alarms stopped. The p95 was 450 ms — still healthy — and the tail latency that had been dragging up the average became visible in a separate dashboard.
This is the fundamental lesson of statistics for developers: the summary statistic you choose determines the story you see. Mean, median, and percentile each reveal different truths. Standard deviation tells you whether your system is consistent or erratic. Confidence intervals tell you whether a change is real or noise. Hypothesis testing tells you whether your "improvement" actually improved anything or whether you're just seeing random variation.
This guide explains the statistics that matter for developers who work with data — whether that data is response times, conversion rates, error rates, or business metrics. It covers descriptive statistics (how to summarize data), probability (how to model uncertainty), and inferential statistics (how to draw conclusions from samples) — with the goal of making better decisions and avoiding the traps that fool even experienced engineers.
Descriptive Statistics: Summarizing Without Lying
Descriptive statistics compress a dataset into a few numbers. The art is choosing numbers that tell the truth — and not choosing numbers that hide it.
Central Tendency: Mean, Median, Mode
The mean (average) is the most common summary — and the most misleading when data is skewed. If 9 users pay $10/month and 1 user pays $1,000/month, the mean revenue per user is $109, which describes no one's actual experience.
The median (middle value) is robust to outliers. In the same dataset, the median is $10 — a more honest summary of the typical user. Use median when data is skewed (revenue, response times, house prices) and mean when data is symmetric (heights, test scores, measurement errors).
The mode (most frequent value) matters for categorical data: the most common error code, the most popular feature, the most frequent user country.
Spread: Variance and Standard Deviation
Central tendency tells you where the middle is. Spread tells you how far the data ranges around that middle. Two APIs can have the same mean response time (350 ms) but wildly different consistency: one with σ = 50 ms (predictable) and one with σ = 400 ms (erratic).
Variance is the average squared deviation from the mean:
Standard deviation is the square root of variance, bringing the units back to the original data:
For the normal distribution, roughly 68% of values fall within ±1 standard deviation of the mean, 95% within ±2, and 99.7% within ±3 — the empirical rule. The Standard Deviation Calculator and Variance Calculator compute these from your raw data, including the crucial sample-vs-population distinction (divide by n-1 for samples, n for populations).
Percentiles: The Tail Tells the Story
For skewed data (response times, revenue, file sizes), percentiles are more honest than mean + standard deviation. The p50 is the median. The p95 is the value below which 95% of observations fall. The p99 captures the worst 1%.
Why percentiles matter for monitoring:
- Mean hides the tail (the alert that cried wolf)
- p95 captures the experience of the slowest 5% without being dominated by outliers
- p99 captures the worst 1% — the users who are most likely to churn
Percentiles are not additive. The p95 of a sum is not the sum of the p95s. If API A has p95 = 200 ms and API B has p95 = 300 ms, the end-to-end p95 of calling both is not 500 ms — it's typically less, because the slowest 5% of A's calls don't always coincide with the slowest 5% of B's calls. This is why distributed tracing matters: you need to measure the end-to-end percentile, not sum individual ones.
Response time percentiles for a typical API. The median (180ms) looks healthy, but p99 reveals 1% of users wait over 1 second. The mean (350ms) is pulled up by the tail.
Probability: Modeling Uncertainty
Probability is the language of uncertainty. For developers, it appears in reliability engineering (what's the chance of outage?), capacity planning (what's the chance we exceed capacity?), and A/B testing (what's the chance this result is real?).
Distributions: The Shapes of Data
Every dataset has a shape — a distribution. The most important distributions for developers:
Normal (Gaussian) distribution is the bell curve: symmetric, mean = median = mode. Heights, measurement errors, and sums of many independent variables tend to be normal (thanks to the Central Limit Theorem, discussed below).
Log-normal distribution is skewed right: the log of the values is normal. Response times, file sizes, and revenue per user tend to be log-normal. The mean is much higher than the median. Use geometric mean or percentiles, not arithmetic mean.
Exponential distribution models time between events: time between requests, time between failures. It has a "memoryless" property: the probability of a failure in the next minute is the same regardless of how long it's been since the last failure.
Poisson distribution models counts of events in a fixed interval: requests per second, errors per hour, support tickets per day. If events arrive independently at rate λ, the count in any interval is Poisson-distributed with mean λ.
The Central Limit Theorem
The Central Limit Theorem (CLT) is one of the most powerful results in statistics: if you take sufficiently large random samples from ANY distribution (normal, skewed, it doesn't matter), the distribution of the sample means will be approximately normal.
This is why the normal distribution appears everywhere: any metric that's the sum or average of many independent contributions (response times averaged over a window, conversion rates averaged over days) tends toward normal, even if the underlying data is wildly skewed. The CLT is also why you can use normal-based methods (z-tests, t-tests) even when your data isn't normal — as long as your sample size is large enough (typically n > 30).
Correlation: The Number That Gets Misused
Correlation measures the linear relationship between two variables, ranging from -1 (perfect inverse) to +1 (perfect direct). A correlation of 0 means no linear relationship.
The Pearson correlation coefficient:
Correlation ≠ causation is the most violated principle in data analysis. Ice cream sales and drowning deaths are correlated — not because ice cream causes drowning, but because both increase in summer (a confounding variable ). In software: server load and error rates are correlated, but the cause is a third factor (a deployment that introduced both a memory leak and increased CPU usage).
Spurious correlations appear by chance when you test enough pairs. If you check 100 unrelated metrics, about 5 will show "significant" correlation at p < 0.05 purely by chance. This is why pre-specifying hypotheses matters — and why data mining without theory produces nonsense.
Inferential Statistics: Drawing Conclusions from Samples
Descriptive statistics summarize what you have. Inferential statistics let you draw conclusions about a population from a sample — which is essential when you can't measure everything.
Confidence Intervals: The Range of Plausible Values
A confidence interval is a range that likely contains the true population parameter. A 95% confidence interval of [2.1%, 2.5%] for a conversion rate means: "if we repeated this experiment many times, 95% of the intervals we'd compute would contain the true conversion rate".
The formula for a confidence interval on a mean:
The key insight: the width shrinks with — to halve the margin of error, you need 4× more data. The Confidence Interval Calculator computes this from your sample mean, standard deviation, and sample size, for any confidence level.
Why confidence intervals beat point estimates. Reporting "conversion rate is 2.3%" hides the uncertainty. Reporting "conversion rate is 2.3% ± 0.2% (95% CI)" tells you the precision of your estimate. If the CI for the difference between two variants includes zero, the difference is not statistically significant.
Hypothesis Testing: Is This Real or Noise?
Hypothesis testing is the formal framework for deciding whether an observed effect is real or due to chance. The logic: assume the boring explanation (null hypothesis), compute the probability of seeing your data under that assumption (p-value), and reject the null if that probability is low enough.
The framework:
- Null hypothesis (H₀): the default assumption (e.g., "the new feature does nothing")
- Alternative hypothesis (H₁): what you want to prove (e.g., "the new feature increases conversion")
- Test statistic: a number computed from your data (t-statistic, chi-square statistic, z-score)
- p-value: the probability of seeing a test statistic at least as extreme as yours, assuming H₀ is true
- Decision: if p < α (typically 0.05), reject H₀
The p-value is not the probability that H₀ is true. It is the probability of the data given H₀ — not the probability of H₀ given the data. This confusion is so pervasive that the American Statistical Association published a formal statement clarifying it.
The t-Test Calculator and Chi-Square Calculator compute test statistics and p-values for the two most common tests: t-test for comparing means (revenue, time, score) and chi-square for comparing proportions (conversion rates, click/no-click).
Statistical Power: The Probability You'll Detect a Real Effect
Statistical power is the probability that your test will detect an effect if one truly exists. A test with 80% power has a 20% chance of missing a real effect (Type II error).
Power depends on three things:
- Effect size: larger effects are easier to detect
- Sample size: more data → more power
- Significance level: lower α → less power (stricter threshold)
The Sample Size Calculator computes how many observations you need for a desired power, given your expected effect size and α. Use it before running any experiment — an underpowered test is worse than no test at all, because it produces false confidence in a "no difference" result.
Practical Applications for Developers
Monitoring and Alerting
The most common statistical mistake in monitoring is alerting on the mean. For skewed distributions (response times, queue lengths, file sizes), the mean hides the tail. Alert on percentiles instead:
- p95 for "most users are having a good experience"
- p99 for "the worst 1% are having a bad experience"
- Mean only for symmetric metrics (CPU utilization, memory usage).
Set thresholds using baselines, not absolutes. A response time of 500 ms might be normal for a database query but catastrophic for a cache lookup. Establish per-endpoint baselines using historical percentiles, and alert on deviations from the baseline (e.g., "p95 is 3× higher than the 7-day rolling average").
A/B Testing and Experimentation
A/B testing is hypothesis testing applied to product decisions. The same principles apply: pre-specify your primary metric, compute sample size in advance, run the test to completion, and report confidence intervals — not just p-values.
Common mistakes:
- Peeking: checking results daily and stopping when p < 0.05 inflates false positives
- Multiple comparisons: testing 20 metrics and reporting the one that's significant is p-hacking
- Underpowered tests: running on too few users produces inconclusive results
- Ignoring practical significance: a 0.1% lift may be statistically significant with 1M users but not worth shipping
Data-Driven Development
"Data-driven" means letting evidence guide decisions — but evidence can be misleading if you don't understand the statistics. Before trusting any metric:
- Check the distribution. Is it normal or skewed? Use mean for normal, median/percentiles for skewed.
- Look at the spread. A mean of 350 ms with σ = 50 ms is very different from a mean of 350 ms with σ = 400 ms.
- Compute confidence intervals. A conversion rate of "2.3%" is less useful than "2.3% ± 0.2%."
- Beware of confounding. Correlation is not causation. A metric may move because of a third factor you're not measuring.
- Pre-specify hypotheses. Decide what you're testing before you look at the data. Post-hoc rationalization produces false discoveries.
Capacity Planning and Forecasting
Statistics also powers capacity planning: predicting how much infrastructure you'll need. If your traffic grows 10% per month, you can extrapolate when you'll need to scale. But extrapolation requires understanding variance — if traffic is highly variable (high standard deviation), you need more headroom than the average suggests.
The rule of thumb: plan for the p95 of expected traffic, not the mean. If your mean projected traffic is 10,000 requests per second, but the 95th percentile is 15,000, provision for 15,000. Otherwise, you'll be overloaded 5% of the time — which is roughly 3.6 hours per day.
Regression for forecasting. Simple linear regression fits a line to historical data (traffic over time) and extrapolates. The regression calculator on this site fits a line to your data and reports the slope (growth rate) and R² (how well the line fits). Use it to answer: "At current growth rate, when will we hit our limit?" But be cautious — regression assumes the future resembles the past. A viral post, a product launch, or a pandemic breaks that assumption.
Error Budgets and Reliability
Site Reliability Engineering (SRE) uses statistics to define error budgets. If your SLO (Service Level Objective) is 99.9% uptime, your error budget is 0.1% — roughly 43 minutes of downtime per month. Statistics tells you whether you're on track to meet that budget or whether you've already blown it.
The math: if you've had 30 minutes of downtime in the first 15 days of a 30-day month, you're on pace for 60 minutes — double your budget. But if your downtime events are random (Poisson-distributed), the variance matters. A month with 30 minutes of downtime could be normal variation or the start of a trend. Statistical process control charts (which plot metrics with control limits at ±3σ) distinguish between normal variation and a genuine shift.
Common Mistakes That Fool Experienced Engineers
- Confusing correlation with causation. Two metrics moving together doesn't mean one causes the other. Look for confounding variables.
- Ignoring sample size. A 5% conversion rate from 20 users (1 conversion) is not a reliable estimate. Compute confidence intervals.
- Using mean for skewed data. Response times, revenue, and file sizes are almost always skewed. Use percentiles.
- Not checking assumptions. t-tests assume approximately normal data (or large samples). Chi-square tests assume sufficient expected counts. Violating assumptions produces unreliable p-values.
- Treating p < 0.05 as truth. A p-value of 0.04 means there's a 4% chance of this result under the null — not that the alternative is true. Replicate findings before acting on them.
- Forgetting about practical significance. With enough data, any tiny effect becomes statistically significant. Ask: "Is this effect large enough to matter?"
Frequently Asked Questions
Q: What is the difference between mean and median?
A: Mean is the average (sum divided by count). Median is the middle value when data is sorted. Mean is sensitive to outliers; median is robust. Use mean for symmetric data, median for skewed data (response times, revenue).
Q: What is standard deviation?
A: A measure of spread: how far data typically sits from the mean. A small standard deviation means data is tightly clustered; a large one means data is widely scattered. For normal data, 68% falls within ±1σ, 95% within ±2σ.
Q: What is a percentile?
A: The value below which a given percentage of observations fall. p50 is the median. p95 is the value below which 95% of observations fall. Percentiles are robust to outliers and ideal for skewed data like response times.
Q: What is a confidence interval?
A: A range that likely contains the true population parameter. A 95% CI of [2.1%, 2.5%] means: if we repeated this experiment many times, 95% of the intervals would contain the true value. If the CI for a difference includes zero, the result is not statistically significant.
Q: What is a p-value?
A: The probability of observing a result at least as extreme as yours, assuming the null hypothesis (no effect) is true. A p-value of 0.05 means: if there were no real effect, you'd see a result this large 5% of the time by chance. It is NOT the probability that your hypothesis is true.
Q: What is statistical power?
A: The probability that your test will detect a real effect if one exists. Typically set to 80% (β=0.20). An underpowered test — one with too few users — will often conclude 'no significant difference' even when a real effect exists.
Q: What is the Central Limit Theorem?
A: The principle that the distribution of sample means approaches a normal distribution as sample size increases, regardless of the underlying data distribution. This is why normal-based methods work even with non-normal data — as long as your sample is large enough (typically n > 30).
Q: What is the difference between correlation and causation?
A: Correlation means two variables move together. Causation means one causes the other. Correlation does not imply causation — both variables may be influenced by a third confounding factor. Ice cream sales and drowning are correlated (both increase in summer) but not causally related.
notAcalculator provides free online calculators and educational guides covering finance, fitness, mathematics, and everyday calculations.
