Python ctypes: Calling C Libraries from Python

tags: python, programming, tutorial, advanced

tags: python, programming, tutorial, advanced

tags: python, programming, tutorial, advanced

tags: python, performance, tutorial, programming

tags: python, performance, tutorial, programming

tags: python, performance, tutorial, programming

tags: python, search, tutorial, beginners

tags: python, search, tutorial, beginners

You’ve probably typed a query into Google and wondered, “How does it actually find that exact article in a blink?” Spoiler: it’s not magic—it’s math, data structures, and a little bit of Python. And guess what? You can build your own search engine today, from scratch, with just a few hundred lines of code. No cloud APIs, no paid subscriptions—just pure, educational Python.

Let’s ditch the complexity and build something simple, functional, and surprisingly powerful: a text-based search engine that indexes local documents and lets you query them instantly.

Why Build Your Own Search Engine?

Before we dive into code, let’s be clear: you don’t need to build a search engine to get results. But if you’re a developer, data scientist, or curious learner, building one teaches you:

• How inverted indexes work (the backbone of real search engines)
• The power of tokenization and text preprocessing
• How ranking and scoring drive relevance
• The fundamentals behind tools like Elasticsearch, Solr, and even Google

This isn’t just theory. You’ll walk away with a working tool you can use to search your own notes, code snippets, or documentation folder.

The 5-Step Blueprint

Every search engine, from the simplest script to massive distributed systems, follows the same core pipeline:

1. Collect documents – Gather the text files you want to search.
2. Preprocess text – Clean, normalize, and tokenize the content.
3. Create an inverted index – Map words to the documents they appear in.
4. Implement search – Match queries against the index.
5. Test and rank – Validate results and optionally score them by relevance.

We’ll implement all five steps in Python, using only built-in libraries. No external dependencies.

Step 1: Collect Your Documents

Start by creating a folder called docs and dropping in a few .txt files. For example:

docs/
  python_basics.txt
  machine_learning.txt
  web_development.txt

Each file should contain plain text you’d like to search. In our case, we’ll use sample content about programming topics.

Step 2: Preprocess and Tokenize

Raw text is messy. Words have punctuation, mixed casing, and stop words like “the” or “and” that add noise. We’ll clean it up:

• Convert to lowercase
• Remove punctuation
• Split into tokens (words)

Here’s a helper function to do that:

import re
from collections import defaultdict

def tokenize(text):
    # Convert to lowercase and remove non-alphanumeric characters
    text = text.lower()
    text = re.sub(r'[^\w\s]', '', text)
    # Split into words
    return text.split()

This function turns "Hello, world!" into ['hello', 'world']. Clean, consistent, and ready for indexing.

Step 3: Build the Inverted Index

The inverted index is the heart of our search engine. Instead of storing “which words are in document A,” we store “which documents contain word X.”

We’ll use a defaultdict where each key is a word and the value is a list of document filenames.

def build_index(doc_folder):
    import os
    index = defaultdict(list)

    for filename in os.listdir(doc_folder):
        if filename.endswith('.txt'):
            filepath = os.path.join(doc_folder, filename)
            with open(filepath, 'r', encoding='utf-8') as f:
                text = f.read()
                tokens = tokenize(text)
                for token in tokens:
                    index[token].append(filename)

    return index

Now, if you search for "python", the index instantly tells you all files containing that word.

Step 4: Implement the Search Function

With the index built, searching is straightforward:

1. Tokenize the query.
2. Look up each token in the index.
3. Return documents that match all tokens (AND logic) or any token (OR logic).

We’ll use AND logic for stricter matches:

def search(query, index):
    tokens = tokenize(query)
    results = set(index[tokens[0]])

    for token in tokens[1:]:
        if token in index:
            results &= set(index[token])
        else:
            return []  # No document contains this word

    return list(results)

Try it out:

if __name__ == "__main__":
    index = build_index("docs")
    query = input("What do you want to search? ")
    matches = search(query, index)

    if matches:
        print(f"Found {len(matches)} document(s):")
        for doc in matches:
            print(f"  - {doc}")
    else:
        print("No matches found. Try a different query.")

Run this script, type a query like python basics, and watch it return matching files. You’ve just built a working search engine.

Step 5: Test and Rank (Optional Upgrade)

Right now, every matching document is returned equally. But in real search engines, ranking matters. You can score documents by:

• Term frequency: How often the query word appears?
• Document length: Shorter docs with the term might be more relevant.
• Position: Did the term appear early in the document?

For a quick upgrade, add term frequency scoring:

def search_with_ranking(query, index, doc_folder):
    tokens = tokenize(query)
    scores = defaultdict(int)

    # Get candidate documents
    candidates = set()
    for token in tokens:
        if token in index:
            candidates |= set(index[token])

    # Score each candidate
    for doc in candidates:
        filepath = os.path.join(doc_folder, doc)
        with open(filepath, 'r', encoding='utf-8') as f:
            text = f.read()
            tokens_in_doc = tokenize(text)
            for token in tokens:
                scores[doc] += tokens_in_doc.count(token)

    # Sort by score
    ranked = sorted(scores.items(), key=lambda x: x[1], reverse=True)
    return [doc for doc, _ in ranked]

Now, documents with more occurrences of your query terms appear first. That’s how relevance is born.

What You Can Do TODAY

You don’t need to wait for a perfect setup. Here’s how to use this today:

• Search your code notes: Drop all your .txt or .md learning notes into docs and query them.
• Build a personal FAQ engine: Index your team’s documentation and let anyone search it.
• Extend it: Add a web interface with Flask, or plug in a vector store for semantic search.

The code is modular. Swap out the tokenizer, add stemming, or integrate with SQLite for persistence. You’ve got the foundation.

The Bigger Picture

This tiny engine mirrors the architecture of giants like Google. They just scale it: distributed indexes, real-time updates, machine-learned ranking, and billions of documents. But the core idea—map words to documents, then match queries—is the same.

By building this, you’ve touched the same principles that power modern search. And that’s worth more than just a working script.

Ready to Go Further?

Try these next steps:

• Add fuzzy matching to handle typos.
• Integrate Flask to make it web-accessible.
• Replace term frequency with BM25 or TF-IDF for better ranking.
• Experiment with vector search using embeddings (e.g., sentence-transformers).

Drop your version in the comments, or fork the code and share what you built. The best way to learn is to build, break, and rebuild.

You didn’t just read about search engines—you built one. Now go search something.

If you found this helpful, consider buying me a coffee ☕ — it keeps these articles coming!

Also check out my AI tools collection: AI 次元世界 — free AI tools for developers.