Most learning-to-rank examples end when the model has been trained. A production search system still has several harder questions to answer:
- Which user events are safe to treat as training examples?
- How should result position affect a click?
- How do we build features across catalogs with different schemas?
- How do we prevent document leakage between training and evaluation?
- What happens when there is too little usable data?
- Does the model need to run inside every search request?
I encountered these questions while building the popularity-training pipeline for an Intelligent Search Platform backed by Python, PostgreSQL, scikit-learn, and Typesense.
The resulting design trains a small model offline, converts its predictions into bounded popularity scores, and writes those scores back to the search index. The search path reads a number; it never invokes scikit-learn.
This article explains the architecture, the decisions behind it, and several edge cases that only became obvious after testing the full training path.
Why replace a fixed popularity formula?
A simple popularity calculation might assign every event a contribution such as:
event contribution = position weight Ă— event value
A click could be worth 1.0, an impression could be worth 0.1, and results farther down the list could receive a logarithmic discount.
This approach is understandable, cheap, and often a reasonable starting point. Its limitation is that every catalog receives the same assumptions. A fixed formula cannot learn that ratings matter in one catalog, recency matters in another, or a particular categorical attribute is associated with user engagement in a third.
The trainer changes the role of the formula. Rather than directly adding position-discounted events into the final score, it uses position weighting while fitting a model from document features and click labels.
That distinction matters:
Fixed formula:
events → hand-written aggregation → popularity
Offline trainer:
events + document features → fitted model → predictions → popularity
This is not a claim that a small logistic-regression model automatically improves relevance. It is an implementation that makes the ranking signal learnable and independently testable.
The complete offline architecture
The pipeline looks like this:
Browse impressions and clicks in PostgreSQL
│
â–Ľ
Join events to current documents
│
â–Ľ
Build schema-independent features
│
â–Ľ
DictVectorizer → sparse matrix
│
â–Ľ
Position-discounted logistic regression
│
â–Ľ
Predict a score for every document
│
â–Ľ
Max-normalize scores to 0–1,000,000
│
â–Ľ
Partial updates sent to Typesense
Training is exposed through an administrative endpoint:
POST /v1/training/popularity?index_id=my-catalog&days=30
The endpoint is deliberately on demand. A deployment can invoke it manually or from cron or a Kubernetes CronJob, but scheduling is not hidden inside the search service.
Step 1: restrict the first model to browse behavior
The first version uses events where the query is empty:
SELECT doc_id, rank, clicked
FROM search_events
WHERE index_id = $1
AND query = ''
AND created_at >= $2
This models browse popularity rather than query-specific relevance.
That narrower scope avoids mixing two different questions:
- Which documents are generally attractive during browsing?
- Which documents are relevant to a particular query?
Query-conditional ranking requires query features, query-document interactions, and a different evaluation design. Treating all traffic as interchangeable would make the model simpler on paper but less precise about what it actually learns.
There is an important mismatch in the current implementation: this browse-only boundary is enforced during training, but not during serving. The resulting popularity field is also added to explicit-query results. The later section, How the learned score enters ranking today, quantifies why that unscoped use is not yet a relevance-safe deployment policy.
Step 2: build features without catalog-specific branches
The platform can index unrelated domains. Hardcoding fields such as brand, plant_type, or screen_size into the trainer would make every new catalog an engineering task.
Instead, the trainer consumes affinity dimensions already discovered for each index and converts the current document into a flat feature dictionary:
def document_features(document, affinity_dimensions):
features = {}
for dimension in affinity_dimensions:
if not dimension.get("enabled", True):
continue
field = dimension["field"]
derived_from = dimension.get("derived_from")
if derived_from:
raw_value = document.get("metadata", {}).get(derived_from)
if raw_value is not None:
features[f"num__{derived_from}"] = float(raw_value)
continue
value = document.get(field) or document.get("metadata", {}).get(field)
if isinstance(value, str) and value.strip():
features[f"cat__{field}"] = value.lower().strip()
return features
The current implementation also includes these optional signals when available:
is_featured- a rating signal:
averageStarRating Ă— log(1 + ratingsCount) - bounded recency:
1 / (1 + days_since_indexing) - the raw numeric source behind discovered tiers such as
price_tier
is_featured deserves special caution. Featured placement can itself cause additional exposure and clicks, and ISP applies a separate serving-time featured boost as well. Including it in the learned model can encode an editorial intervention into popularity and then apply that intervention again during serving. I retained it in this description because it is present in the current implementation, but I would exclude it from a subsequent version unless an experiment explicitly estimates its effect.
Missing fields are omitted. There is no requirement for every catalog to expose an identical schema.
DictVectorizer is a good fit for this representation:
from sklearn.feature_extraction import DictVectorizer
vectorizer = DictVectorizer(sparse=True)
X = vectorizer.fit_transform(feature_dicts)
It one-hot encodes categorical strings, retains numeric values, and produces a sparse matrix without building a DataFrame or maintaining a manual vocabulary.
Step 3: account for result position—carefully
Clicks are not direct relevance labels. A result near the top is more likely to be examined than the same result near the bottom.
The current trainer uses a logarithmic weight:
import math
def position_weight(rank: int) -> float:
return 1.0 / math.log2(rank + 2)
Each event becomes:
X = features of the displayed document
y = 1 for click, 0 for impression without click
w = 1 / log2(rank + 2)
The model receives w through sample_weight:
from sklearn.linear_model import LogisticRegression
model = LogisticRegression(max_iter=1000)
model.fit(X, labels, sample_weight=weights)
This logarithmic weight is an exposure-confidence discount inherited from the formula-based pipeline. It reduces the influence of events at deeper positions, including deep clicks. It therefore does not correct position bias and may suppress informative clicks on poorly ranked documents.
Proper inverse-propensity scoring requires estimated examination probabilities, usually obtained through randomized exposure or a validated click model. Click contributions are then weighted by inverse propensity, often with clipping to control variance. Non-clicks also require careful treatment because absence of a click does not establish irrelevance. The counterfactual basis for this approach is described in Unbiased Learning-to-Rank with Biased Feedback.
Calling this version “position-discounted” is accurate. Calling it “position-debiased” would not be. Propensity estimation, IPS or SNIPS objectives, randomized exposure data, and counterfactual evaluation remain separate follow-up work.
Step 4: join against the current index
Interaction logs can outlive documents. Products may be removed between an impression and a later training run.
The trainer therefore fetches the current documents from Typesense and joins each event by doc_id. Events for documents that no longer exist are skipped.
This creates an easily missed edge case. Suppose the database returns six events, clearing a five-event minimum, but four events reference deleted documents. Only two usable examples remain.
The minimum must be checked twice:
if len(raw_events) < MIN_EVENTS:
return empty_result()
usable_examples = join_events_to_current_documents(raw_events, documents)
if len(usable_examples) < MIN_EVENTS:
return empty_result()
Checking only the SQL result count can send an unexpectedly tiny matrix into model fitting.
The five-event threshold is only a mechanical guard against invalid model fitting. It is not evidence that five events are statistically sufficient for a useful ranking model. A production promotion policy should require a separately validated traffic threshold and ranking-oriented evaluation.
Step 5: handle data that cannot train a classifier
Production data is often incomplete in uninteresting ways. The trainer treats these situations as valid empty runs instead of server failures:
- fewer than the minimum number of browse events
- Typesense unavailable or the collection missing
- too few events remaining after the document join
- all labels belonging to one class
- every document producing an empty feature dictionary
The last case exposed a real failure mode. DictVectorizer can produce a matrix with rows but zero columns when every feature dictionary is empty:
X = vectorizer.fit_transform(feature_dicts)
if X.shape[1] == 0:
return empty_result()
Without the guard, the downstream estimator receives no features and raises instead of reporting that there is nothing to learn.
An empty training run is still recorded in the training_runs table. “No model was produced” is operational information, not an event that should disappear from history.
Step 6: prevent document leakage during evaluation
The same document can generate many events, and every one of those events shares the same document feature vector.
A random per-event split can therefore place impressions for one document in training and clicks for the same document in testing. The model has effectively already seen the test document's features, producing an overly optimistic metric.
The implementation groups by document ID:
from sklearn.model_selection import GroupShuffleSplit
splitter = GroupShuffleSplit(
n_splits=1,
test_size=0.2,
random_state=0,
)
train_indices, test_indices = next(
splitter.split(X, labels, groups=example_document_ids)
)
All events for a document remain on one side of the split.
Small grouped datasets introduce another edge case: one side may contain only clicks or only non-clicks. In that situation, evaluation AUC is unavailable. The trainer returns auc: null but still fits the production model on the complete usable dataset.
Evaluation failure and training failure are not necessarily the same event.
Step 7: score the entire catalog
The model is trained from documents associated with observed events, but it scores every current document using its features:
all_document_matrix = vectorizer.transform(all_document_features)
probabilities = model.predict_proba(all_document_matrix)[:, 1]
This is useful for documents that have features but little or no direct interaction history. It does not solve cold start completely—the model still depends on relationships learned from other documents—but it avoids limiting scores only to documents that already received clicks.
The predictions are max-normalized into the existing popularity range. This is a serving-scale transformation, not statistical probability calibration:
maximum = max(probabilities, default=0.0)
if maximum > 0:
popularity_scores = [
round(probability / maximum * 1_000_000)
for probability in probabilities
]
else:
popularity_scores = []
The trainer then performs a partial update:
typesense.update_document_fields(
collection_name,
document_id,
{"popularity": popularity},
)
Structurally, the existing search and merchandising paths can continue reading the same field, so replacing the score generator does not require model inference inside every query. That is an operational compatibility property—not a claim that the current serving-time blend is relevance-safe. The next section describes the unresolved ranking risk.
How the learned score enters ranking today
The training and serving boundaries are decoupled, but the scale used to combine their outputs still matters.
ISP currently retrieves lexical and semantic candidates separately and merges their ranks using:
rrf_score =
0.7 / (60 + lexical_rank + 1)
+ 0.3 / (60 + semantic_rank + 1)
It then adds the stored popularity value to every candidate's score:
popularity_boost = min(popularity, 1_000_000) / 2_000_000
final_score = rrf_score + popularity_boost + other_configured_boosts
A document ranked first in both retrieval channels receives an RRF score of approximately 0.0164, while the largest popularity boost is 0.5. Popularity can therefore dominate the relative order of the retrieved candidates rather than act as a small tie-breaker.
That behavior requires particular caution because this model is trained only on empty-query browse events, while the stored field is currently consumed during both browsing and explicit search. The safer target design is:
- during browsing, allow popularity to be a primary signal;
- during explicit search, disable it, bound it relative to relevance, or use it only within relevance groups;
- measure how often popularity displaces a more relevant result before promoting a new blending policy.
Typesense documents a related relevance-bucketing approach in which a custom popularity score reorders results only within text-relevance groups. That is not identical to ISP's application-side RRF, but it illustrates the serving constraint. See Ranking Based on Relevance and Popularity.
This serving refinement is not implemented yet. The present article documents the current pipeline rather than claiming that its blending coefficient is already validated.
Why keep inference outside the request path?
Offline scoring has several practical advantages:
- no scikit-learn model needs to be loaded by every API worker
- no feature vector is constructed for every candidate during search
- the search latency budget remains controlled by the search engine
- rollback can restore or recompute one numeric field
- the existing formula-based pipeline remains available as an independent fallback
The trade-off is freshness. Scores change only when the training job runs. That is acceptable for a browse-popularity signal whose update interval can be measured in hours or days, but it would be inappropriate for a feature requiring immediate adaptation.
Testing the path that matters
Router tests can confirm authentication, index resolution, and graceful API responses while still missing the core ML path.
The domain-level suite therefore drives the complete trainer with a real test database and mocked Typesense reads and writes. Its regression cases include:
- Zero-feature matrix: mixed click labels but no usable document features must return zero scored documents rather than crash.
- Post-join minimum: raw events may clear the threshold while usable events do not.
- Grouped evaluation: events for the same document must never cross the evaluation boundary.
- Single-class grouped split: AUC can become unavailable without preventing the final full-data model from scoring documents.
These tests caught assumptions that ordinary endpoint tests could not exercise.
What this system does not prove
An operational model pipeline and an improved ranking policy are not the same achievement.
This implementation does not yet provide:
- unbiased propensity correction
- query-specific learning-to-rank
- counterfactual policy evaluation
- nDCG or MRR validation against relevance judgments
- automated retraining schedules
- pooled learning across low-traffic indexes
- evidence of conversion or revenue improvement
- a validated guarantee that popularity cannot overpower explicit-query relevance
- freedom from editorial-exposure confounding or double-counting through
is_featured - statistically calibrated probabilities across separate training runs
A held-out AUC is a sanity check for the classifier, not proof that users receive better rankings. Demonstrating ranking improvement requires a stronger offline evaluation protocol with temporal splits and ranking metrics such as nDCG or MRR and, when appropriate and authorized, a controlled online experiment.
Lessons learned
The model was the shortest part of the feature. The important engineering decisions were around it:
- Define exactly which behavior the model represents.
- Reuse schema-derived features instead of hardcoding a vertical.
- Describe position discounting accurately; do not present it as debiasing.
- Recheck data sufficiency after every destructive transformation.
- Group repeated entities when splitting interaction data.
- Record empty and failed-to-evaluate runs explicitly.
- Keep offline learning separate from query-time serving when the signal allows it.
- Test vectorization, fitting, scoring, normalization, and write-back—not just the endpoint.
- Keep learned browse signals from silently overpowering explicit-query relevance.
The result is intentionally modest: a small, auditable model that learns a browse-popularity signal and hands the search engine a number it already knows how to use.
That modest boundary is also what makes the system practical.
