Classic Machine Learning Through the Eyes of an SRE — Part 7
Picture a client health metric that has been flat at 2 out of 10 for six months. Ask PCA to compress your client-health data and that metric will contribute almost nothing to the directions PCA decides to keep.
Not because PCA is broken. Because PCA treats variance as importance, and a signal that barely moves contributes almost no variance. Reduce the data far enough and the independent information it carried is simply not there anymore.
But a CSAT frozen at 2/10 is not noise. It is a crisis nobody is escalating. And after compression, it may no longer be available to anything downstream.
That is the bet, and in ops data it is frequently wrong. The critical signals are often the quiet ones.
There is a cheaper version of the same failure that catches most people first. PCA measures variance in whatever units your features happen to be in, so a metric ranging from 0 to 10,000 can dominate one ranging from 1 to 5 purely because it is bigger. Standardize before you compress, or your first principal component may just be an elaborate way of saying "ticket count." Same class of bug as unscaled features in K-Means and SVM, and it fails just as quietly.
What PCA actually is
Third answer-finding strategy in the unsupervised set, using the same shorthand as the last two articles.
K-Means SEARCHES: iterate and hope.
DBSCAN DEFINES: declare a rule and traverse.
PCA SOLVES: an eigendecomposition or SVD gives a direct solution rather than an iterative local search. No convergence to babysit, no restarts, no local optima to escape.
Two caveats on the word "direct," both worth knowing. Many libraries will use randomized SVD on large matrices, which is approximate and stochastic. And even with an exact solver, eigenvectors are only defined up to sign, so a component can come back inverted between runs or across implementations. The variance explained is identical either way, which is precisely why nobody notices.
Hold onto that second one. It comes back later and it is worse than it sounds.
PCA is also the first algorithm in this set whose bet is about columns rather than rows. Everything before it asked which points belong together. PCA asks which features move together. If ticket volume, escalation count and response-time degradation all rise and fall as one, PCA blends them into a single composite axis. One dial where there appeared to be three.
Note the verb. It blends. My most persistent mistake while learning this was writing that PCA "removes similar features," which it does not, and which is a description of feature selection rather than of anything PCA does. PCA constructs new axes as weighted combinations of the original features. The output is a re-description of your data, not a verdict on which columns deserve to exist.
Layer 1.5
The thing that reframed PCA for me is that it is not analysis. It is infrastructure.
A clustering sits at the top of a pipeline. If it is wrong it embarrasses you in one meeting, someone pushes back, and you fix it. PCA sits underneath, at what I have started calling Layer 1.5, and it decides what every layer above it can ever see.
A bad compression blinds every downstream consumer, quietly, for quarters. And the post-mortem may never find it, because the missing signal is not anywhere anymore. The 2/10 CSAT crisis was not mislabeled. It was not misclustered. It is absent, and absence leaves no trace to investigate.
Failures that leave no evidence are the ones ops people fear most. PCA can manufacture them at scale, politely, while reporting an excellent variance-explained figure.
Treat loadings like a schema
The engineering conclusion I keep coming back to: treat a fitted PCA like a database schema, not a preprocessing step.
The loadings, meaning the recipe for each composite axis, are an API contract with every downstream consumer. So version them, diff them between refits, assign them an owner, monitor how their meaning drifts, and roll back the model and the loadings together rather than separately. If PC2 used to mean "escalation pressure" and gradually starts looking more like "account size," every dashboard built on PC2 changed meaning without anyone editing a dashboard.
And now the sign problem gets its moment, because this is the part I did not know and would have walked straight into.
Refit the model, get a component back with its sign flipped, and every consumer reading "high PC2 means high escalation pressure" may now be reading the exact opposite. The variance explained is unchanged. The dashboard still renders. Nothing errors, nothing alerts, and somebody acts on an inverted signal for a quarter before anyone thinks to check the one thing that has no reason to be checked.
The fix is almost annoyingly simple. Pin the sign. Define a convention, such as requiring a chosen reference loading to be positive, assert it after every refit, and fail the pipeline when the convention is violated.
Nobody would push a schema migration without versioning and review. Refitting PCA is a schema migration for your feature space. It deserves the same ceremony.
What I'd tell my ops team
Before compressing anything, inventory the quiet-but-critical signals. The flat CSATs. The metrics whose silence is the message. Either protect them outside the compression step, or make sure the representation you keep still preserves what they carry.
Then check how much variance your retained components actually explain, and read the loadings out loud in business language. If PC1 cannot be named in a sentence, nobody downstream will know what they are looking at, and they will use it anyway.
One more limit is worth stating plainly, because it is easy to forget once the math starts working. PCA finds linear combinations. Structure that is real but strongly nonlinear may not survive compression well, and variance explained will not tell you that you lost it. If your features relate to each other in ways a straight line cannot describe, PCA is probably the wrong instrument, and it will not mention this to you.
PCA answers one question precisely and repeatably: what varies most together. Whether that was your question is still your problem.
Production takeaway
Standardize before you compress, or your first component may just be your largest-unit feature wearing a disguise.
Treat fitted loadings like a schema migration: versioned, diffed between refits, owned, and rolled back together with the model. Pin the component signs and assert them after every refit, because a silent flip can invert meaning for everything downstream.
And before compressing anything, inventory the quiet-but-critical signals, the flat metrics whose silence is the message, and make sure variance-based compression does not erase them.
Common interview mistake
Describing PCA as removing correlated features. It removes nothing column-wise. It builds new axes that are weighted combinations of the original features, so the output is a re-description rather than a selection.
Two more that come up. Forgetting that PCA is scale-sensitive, when standardization is usually necessary if features sit on different scales. And calling PCA fully deterministic, when randomized SVD can be approximate and stochastic, and even exact solvers leave eigenvector signs ambiguous.
Where I'd use this in a real production system
Compressing wide, correlated telemetry before clustering. Building a composite client-health or skills index. Reducing dimensionality before visualization. And anywhere downstream consumers can tolerate a re-described feature space, with the loadings under version control.
Classic Machine Learning Through the Eyes of an SRE — Part 7. What ML algorithms bet about your world, through a production and ops lens.
Previous: DBSCAN.
Next: hierarchical clustering, the algorithm that fails beautifully.
