Stop Giving Channel Bios a Vote in Every Video Search

Embedding every exported field gives a creator's biography a vote in every video search.

Example: you search for “sourdough starter,” and a bicycle-repair Short becomes a plausible match because its uploader's biography mentions baking. Your ingestion code used json.dumps(row). The biography came along, then aboutChannelInfo brought another copy.

Lab note 001: can we keep creator context without letting it speak for every video? An early catch: subtitles indicates caption availability, and captionLanguages lists available tracks. Neither supplies transcript text, so this corpus can support metadata retrieval, not claims about spoken content.

What belongs in a video embedding?

Start with the text you could reasonably label as describing that individual Short: its title and description.

Channel-mode rows carry the uploader's profile on every row, and aboutChannelInfo repeats the profile fields inside a nested object. Serializing the whole row turns a useful export into a very opinionated embedding document.

The biography gets extra representation simply because you collected more videos.

Here's the document boundary we'd use. Example text makes the potential false match visible before an embedding model enters the picture:

Query: sourdough starter

Whole-row embedding
  Bicycle brake adjustment
  Fixing a rubbing brake
  Creator bio: sourdough experiments and workshop projects
  Nested creator bio: sourdough experiments and workshop projects

Video-text embedding
  Bicycle brake adjustment
  Fixing a rubbing brake

Creator bio stays behind channelId for an optional later join.

That doesn't predict a ranking. It shows why the first document can match the query for a reason unrelated to the video, and which evidence the second document removes.

The cleaner below strips email addresses and web links by default, then normalizes whitespace. You can retain either when your retrieval task needs it.

import hashlib
import json
import os
import re
from collections import Counter, defaultdict
from datetime import datetime, timezone

import tiktoken
from apify_client import ApifyClient

def clean_text(value, *, strip_emails=True, strip_links=True):
    if not isinstance(value, str):
        return ""
    if strip_emails:
        value = re.sub(
            r"\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}\b",
            " ", value, flags=re.IGNORECASE,
        )
    if strip_links:
        value = re.sub(
            r"https?://\S+|www\.\S+",
            " ", value, flags=re.IGNORECASE,
        )
    return " ".join(value.split())

def canonical(value):
    return json.dumps(value, sort_keys=True, ensure_ascii=False)

This is a baseline scrubber. Phone numbers, names, addresses, and disguised contact details can survive it.

Removing links also changes the task: a description whose useful information lives in a destination URL may become unhelpful. Keep the video's source URL separately for attribution, and review the scrubbed text before sharing it.

Give the biography its own address

Treat the channel as a parent entity you can join after retrieving a video.

That makes the distinction useful during debugging. When a result looks odd, inspect the exact video text that matched before opening its creator profile.

For labeling, the same boundary matters. A biography mentioning baking doesn't justify assigning a baking label to every upload.

There are no conversation threads or speaker turns to reconstruct here. channelId preserves the uploader relationship; it doesn't identify a reply parent or a speaker in the video.

Keep the join deliberate, too. Automatically attaching the biography to every retrieved passage would bring the repetition straight back into the answer context. The biography is persistent. We can be persistent about leaving it out.

Keep collection failures out of the text

A successful run can still emit error or status rows. Embedding an error message would give your corpus a different sort of confident nonsense.

This Example input requests channel-mode details; the code checks run status and accepts only rows with video and creator identities.

run_input = {
    "inputMode": "channel",
    "channelUrls": ["UCX6OQ3DkcsbYNE6H8uQQuVA"],
    "sortShortsBy": "newest",
    "maxShorts": 50,
    "includeDetails": True,
}
client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor(
    "vexaloom-labs/youtube-shorts-metadata-extractor-for-channels"
).call(run_input=run_input)
if run["status"] != "SUCCEEDED":
    raise RuntimeError(f"Collection ended with {run['status']}")

audit = Counter()
rows_by_video = defaultdict(list)
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    if row.get("errorCode") or row.get("error"):
        audit["excluded_error"] += 1
        continue
    if not all(
        isinstance(row.get(key), str) and row[key].strip()
        for key in ("videoId", "channelId")
    ):
        audit["excluded_non_video_or_missing_identity"] += 1
        continue
    rows_by_video[row["videoId"]].append(row)

The identity gate excludes status rows without interpreting their messages as content. Its audit category also includes incomplete records, so don't read that category as a collection-failure count.

Requiring creator identity is a corpus choice: the unseen-creator split later needs a defensible grouping key.

The input requests a bounded slice of the newest material. Keep that selection policy with your evaluation configuration, because a retrieval benchmark sampled from recent uploads asks a different question from one sampled across a creator's history.

Save profile disagreements instead of hiding them

A creator can change their biography. An observation can also be partial, or disagree with another representation in the same export.

We'd rather keep the disagreement visible than let the last row win.

This cell retains distinct, scrubbed profile variants and records where each appeared. It reads the nested representation only when it's an object.

channel_profiles = defaultdict(dict)
for versions in rows_by_video.values():
    for row in versions:
        representations = [("top_level", row)]
        if isinstance(row.get("aboutChannelInfo"), dict):
            representations.append(
                ("aboutChannelInfo", row["aboutChannelInfo"])
            )
        for representation, payload in representations:
            profile = {
                field: text
                for field in ("channelName", "channelDescription")
                if (text := clean_text(payload.get(field)))
            }
            if not profile:
                continue
            snapshot = channel_profiles[row["channelId"]].setdefault(
                canonical(profile),
                {"profile": profile, "observations": set()},
            )
            snapshot["observations"].add(canonical({
                "videoId": row["videoId"],
                "scrapedAt": row.get("scrapedAt"),
                "representation": representation,
            }))

These are observations of selected fields, not a complete change history. A variant doesn't prove that a creator edited their profile, and changes removed by the scrubber won't create separate variants.

channelDescriptionLinks, counters, banners, and other profile details stay outside this corpus because this join only needs creator text.

For a temporal benchmark, restrict the join to profile observations available at the evaluation cutoff. A later biography can reveal information your earlier retrieval system couldn't have known.

Collapse reruns without losing their sources

Repeated observations of a video and repeated text across different videos need different treatment.

First, collapse by videoId. Select the newest parseable scrape timestamp, with a deterministic tie-break when timestamps can't distinguish observations.

Then group identical cleaned text while retaining each video's source reference. The records below introduce local corpus fields such as text, sources, and provenance.

def observation_key(row):
    try:
        stamp = datetime.fromisoformat(
            row.get("scrapedAt", "").replace("Z", "+00:00")
        )
        if stamp.tzinfo is None:
            stamp = stamp.replace(tzinfo=timezone.utc)
    except (AttributeError, TypeError, ValueError):
        stamp = datetime.min.replace(tzinfo=timezone.utc)
    return stamp, canonical(row)

encoding = tiktoken.get_encoding("cl100k_base")
text_groups = defaultdict(list)
for video_id, versions in sorted(rows_by_video.items()):
    if len({row["channelId"] for row in versions}) != 1:
        audit["conflicting_video_parent"] += 1
        continue
    selected = max(versions, key=observation_key)
    text = "\n".join(
        part for field in ("title", "description")
        if (part := clean_text(selected.get(field)))
    )
    if not text:
        audit["empty_video_text"] += 1
        continue
    provenance = {
        canonical({
            key: row[key]
            for key in ("sourceType", "source", "scrapedAt")
            if key in row
        })
        for row in versions
    }
    source = {
        "videoId": video_id,
        "channelId": selected["channelId"],
        "provenance": [
            json.loads(value) for value in sorted(provenance)
        ],
        **{
            key: selected[key]
            for key in ("url", "captionLanguages", "subtitles")
            if key in selected
        },
    }
    text_groups[text].append(source)
    audit["collapsed_video_observations"] += len(versions) - 1

Keeping title and description from the same observation avoids manufacturing a document from different versions.

The tie-break makes the build repeatable; it can't establish chronology. Conflicting creator identities go into the audit because that's a relationship worth investigating before an evaluation depends on it.

Caption availability remains attached to the selected source. It never enters the embedding text disguised as something someone said.

Split creators before trusting the score

For generalization to unseen creators, group by channelId. For retrieval within a known creator's catalog, creator overlap may be intentional.

Creator separation alone still allows copied descriptions to cross the boundary. The code sends exact-text groups spanning different splits to review rather than placing those documents in the evaluation corpus.

It also counts tokens and holds oversized documents for review. The budget is a local setting, and the chosen tokenizer needs to match your embedding model before you treat its counts as the model's actual input budget.

def creator_split(channel_id):
    bucket = hashlib.sha256(
        channel_id.encode("utf-8")
    ).hexdigest()[0]
    return {"0": "test", "1": "validation"}.get(bucket, "train")

token_budget = 512
corpus, review = [], []
token_totals = Counter()
for text, sources in sorted(text_groups.items()):
    splits = {
        creator_split(source["channelId"]) for source in sources
    }
    tokens = len(encoding.encode(text, disallowed_special=()))
    reasons = []
    if len(splits) != 1:
        reasons.append("identical_text_crosses_splits")
    if tokens > token_budget:
        reasons.append("over_token_budget")
    document = {
        "textHash": hashlib.sha256(text.encode("utf-8")).hexdigest(),
        "text": text,
        "tokenCount": tokens,
        "sources": sources,
        "split": next(iter(splits)) if len(splits) == 1 else None,
    }
    if reasons:
        review.append({**document, "reviewReasons": reasons})
        audit.update(reasons)
    else:
        corpus.append(document)
        token_totals[document["split"]] += tokens

channels = [
    {
        "channelId": channel_id,
        "snapshots": [
            {
                "profile": snapshot["profile"],
                "observations": [
                    json.loads(value)
                    for value in sorted(snapshot["observations"])
                ],
            }
            for _, snapshot in sorted(variants.items())
        ],
    }
    for channel_id, variants in sorted(channel_profiles.items())
]

def write_jsonl(filename, records):
    with open(filename, "w", encoding="utf-8") as handle:
        for record in records:
            handle.write(canonical(record) + "\n")

write_jsonl("videos.jsonl", corpus)
write_jsonl("channels.jsonl", channels)
write_jsonl("review.jsonl", review)
with open("audit.json", "w", encoding="utf-8") as handle:
    json.dump({
        "counts": dict(audit),
        "videoDocuments": len(corpus),
        "channelRecords": len(channels),
        "reviewDocuments": len(review),
        "embeddingTokensBySplit": dict(token_totals),
    }, handle, ensure_ascii=False, indent=2)

Every contributing video reference survives in an accepted text group. A shared document can therefore lead back to all its sources.

The report gives you corpus sizes, exclusions, and token totals when you execute it. Those are quantities to inspect, with no token savings or retrieval improvement claimed here.

Hashing keeps assignments stable, but it doesn't balance topics or guarantee useful split sizes. With a single creator, this policy can't produce an unseen-creator comparison.

The fix has a boundary

Our verdict: embed video metadata and fetch creator context when the task needs it.

The worked source is YouTube Shorts Metadata Extractor for Channels & Hashtags. The same corpus boundary helps whenever parent-level prose follows every child record into an export.

What doesn't work is treating exact deduplication as a complete leakage check. Lightly edited promotions, paraphrases, and recurring passages inside otherwise different descriptions survive it.

Before trusting a score, inspect near-duplicates and compare representative queries against both document constructions. In the sourdough example, check whether relevant video text earns the match after the profile prose is removed. A cleaner corpus is a hypothesis about better retrieval until that evaluation supports it.

Distinct channels can also share an owner. Channel-disjoint doesn't necessarily mean owner-disjoint.

For oversized documents, decide how to chunk before admitting them to the corpus. Every chunk must inherit the video's sources and split, so neighboring chunks can't drift into different evaluation partitions.

Privacy follows the files, too. Source identifiers remain identifying metadata, and review.jsonl still contains text. Apply your retention and access policy to the profiles, review records, and raw exports as well as the embedding input.

Reproduce the corpus

• Install apify-client and tiktoken, and set APIFY_TOKEN in your environment.
• Save the Python cells in order and replace the channel identifier in run_input with your target.
• Execute the program in a dedicated output directory. Inspect audit.json, split coverage, and review.jsonl before indexing.
• Embed only the text field from the appropriate split in videos.jsonl. Keep sources as metadata and channels.jsonl available for deliberate joins.

Example input, matching the acquisition cell exactly:

{
  "inputMode": "channel",
  "channelUrls": ["UCX6OQ3DkcsbYNE6H8uQQuVA"],
  "sortShortsBy": "newest",
  "maxShorts": 50,
  "includeDetails": true
}

Let the video earn the match. Then open the biography.