Vibe-Memory: AI Semantic Memory That Fixes ChatGPT's Amnesia

Honestly, I've been burned by AI memory enough times to know something's broken.

Let me ask you this: Have you ever told ChatGPT about a personal project, shared your ideas, your struggles, your design decisions... and then a week later when you come back, it's completely forgotten? It asks you the same basic questions again, it doesn't remember the context you painstakingly laid out, it feels like talking to a goldfish with a PhD.

I got fed up. So I built Vibe-Memory — an AI-powered semantic memory system that actually remembers who you are, what you've done, and the "vibe" behind your experiences. And after three months of building, I want to share what I learned, what broke, and why this might be the missing piece in your personal AI workflow.

What Problem Does This Actually Solve?

So here's the thing: Large language models are incredible, but they don't remember. Every conversation is a fresh start. Every time you have to re-explain everything. This is fine for one-off questions, but when you're working on long-term projects, journaling, building a personal knowledge base, or just want your AI assistant to actually know you — it's exhausting.

Other solutions exist:

• ChatGPT Custom GPTs: Great for predefined instructions, but they don't dynamically grow with your experiences
• Vector databases: Too complex for most people, requires technical setup to get working
• Note-taking apps with AI: They're still focused on you searching, not AI automatically using your memory

I wanted something simpler:

1. You add a memory (text, note, experience)
2. It automatically embeds it semantically
3. When you chat with AI, it automatically pulls relevant memories
4. The AI knows your context without you having to re-explain everything

That's it. No complex setup. No fancy UI that gets in your way. Just semantic memory that works.

The Architecture: It's Simpler Than You Think

Honestly, I overcomplicated it at first. I tried building a fancy vector search pipeline with multiple stages, different embedding models, caching layers... it was a mess. After a month of fighting complexity, I stripped it all back down to something really simple:

// Core architecture in Go — yes, I went with Go for this because I wanted it lightweight and fast
type VibeMemory struct {
    store     VectorStore
    embedder  Embedder
    config    Config
}

// Add a new memory to the store
func (vm *VibeMemory) AddMemory(ctx context.Context, content string, metadata Metadata) error {
    embedding, err := vm.embedder.Embed(ctx, content)
    if err != nil {
        return err
    }

    return vm.store.Upsert(ctx, &Memory{
        Content:    content,
        Metadata:  metadata,
        Embedding: embedding,
        CreatedAt: time.Now(),
    })
}

// Search for relevant memories given a query
func (vm *VibeMemory) Search(ctx context.Context, query string, limit int) ([]*MemoryResult, error) {
    embedding, err := vm.embedder.Embed(ctx, query)
    if err != nil {
        return nil, err
    }

    return vm.store.Search(ctx, embedding, limit)
}

That's literally the core. Everything else is just wrapping around this to make it easy to use.

For the vector store, I ended up going with pgvector on PostgreSQL. Hear me out — I know everyone reaches for specialized vector databases these days, but:

• Most people already have PostgreSQL running somewhere
• pgvector is mature, stable, and honestly fast enough for personal use (I have ~10,000 memories and it's still sub-100ms response)
• You get ACID compliance and all the goodness of a real relational database for free
• No extra services to maintain for a personal project

Here's what the table definition looks like:

CREATE TABLE memories (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    content TEXT NOT NULL,
    metadata JSONB NOT NULL DEFAULT '{}',
    embedding vector(1536) NOT NULL, -- using OpenAI text-embedding-3-small
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW()
);

CREATE INDEX ON memories USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);

Super straightforward. The 1536 dimension is for OpenAI's embedding model, which is cheap ($0.025 per million tokens) and good enough for this use case.

For the embedding step, I started with local models (like all-MiniLM) but honestly — for personal use, OpenAI embeddings are cheap enough and better quality. The API call adds a tiny bit of latency, but it's negligible for this use case.

The Surprising Challenges I Didn't Expect

I learned the hard way that semantic memory isn't just "embed everything and search". There are some tricky problems that don't get talked about much.

1. The "Vibe" Problem: Relevance Isn't Just Semantic

Here's what caught me off guard: Sometimes two memories are semantically similar but contextually completely different. Or you want to remember recent things more than old things.

For example, if you're talking about your current side project "Vibe-Memory", you don't want the search pulling up an old blog post you wrote about "memory management in C" from 10 years ago just because it contains the word "memory".

I ended up adding two simple adjustments that helped a lot:

Time weighting: I boost the score of more recent memories. Nothing fancy:

// Boost score based on recency — newer memories get a small boost
const recencyBoostHalfLife = 30 * 24 * time.Hour // 30 days
age := time.Since(memory.CreatedAt)
recencyScore := math.Exp(-float64(age) / float64(recencyBoostHalfLife))
finalScore = score * (1 + 0.2 * recencyScore) // 20% max boost for very new

This helps push recent, relevant memories to the top without completely ignoring older ones.

Metadata filtering: You can tag memories with categories, projects, tags, whatever. When searching, you can filter to just certain tags. Super simple but super effective.

2. The "Too Much Context" Problem

Once you start adding a lot of memories, you can easily end up with 10 relevant memories that total 10k tokens — which eats up your entire context window.

I tried a few approaches:

• Just take top N: Works okay, but sometimes you get several long memories that still bloat context
• Summarize before injecting: This works better, but adds extra embedding cost and latency
• Truncate long memories: I ended up going with this for now — keep the first ~500 tokens of each memory, which is usually enough to understand what it's about

The tradeoff: Sometimes you lose the full detail, but in practice, when the AI needs more context, it can ask you to elaborate. And keeping the context window lean makes the conversation faster and cheaper.

I'm still experimenting with this, honestly. It's one of those problems that doesn't have a perfect answer — it's all tradeoffs.

3. The "Duplicate Memory" Problem

If you're like me, you end up writing the same thought multiple times in different places. Or you update a memory, and now you have the old version and the new version. Duplicates clutter up the search and add irrelevant context.

My solution: When adding a new memory, first search for semantically similar existing memories. If something is over a certain similarity threshold (I use 0.95 cosine similarity), ask if you want to replace it or keep both.

func (vm *VibeMemory) CheckDuplicate(ctx context.Context, newEmbedding []float32) (*Memory, float32, error) {
    results, err := vm.store.Search(ctx, newEmbedding, 1)
    if err != nil || len(results) == 0 {
        return nil, 0, err
    }
    return results[0].Memory, results[0].Score, nil
}

Simple, effective. You'd be surprised how often this catches accidental duplicates when you're journaling regularly.

Pros & Cons: Let's Be Honest

Okay, I've been building this for three months. I use it every day. Here's what I think after all that:

✅ Pros

1. It just works — Once you set it up, you don't have to think about it. It just automatically injects relevant context into your AI conversations.

2. Your data stays in your database — Unlike sending all your personal memories to some third-party app, everything stays in your PostgreSQL database. You control your data. You can query it directly if you want. No vendor lock-in.

3. Surprisingly cheap — For ~10,000 memories totaling ~1 million tokens, that's $0.025 for embeddings per full re-embed. Even if you add one memory a day, that's like $0.000025 per day. Nothing. Hosting PostgreSQL is pennies on platforms like Fly.io or Supabase.

4. Simple architecture — No complex distributed systems, no multiple services that need to be coordinated. Just Go + PostgreSQL. Easy to deploy, easy to maintain.

5. Extensible — Because it's just a simple Go library with a clean API, you can wrap it into whatever you want: CLI, REST API, Slack bot, VS Code extension, whatever.

❌ Cons

1. Still requires some technical setup — You need to have a PostgreSQL database with pgvector, you need an OpenAI API key. It's not a one-click app yet. I'm working on that, but it's not there.

2. No fancy UI right now — Currently it's API/CLI first. I'm building a simple web UI, but it's not ready yet. Power users will be fine, but if you want a polished consumer app, this isn't it (yet).

3. Embeddings aren't perfect — Sometimes it still pulls irrelevant memories. It's better than nothing, but it's not magic. Semantic search isn't 100% accurate.

4. No collaboration features — It's designed for single user personal use. No sharing, no team workspaces. That's by design for now, but it's a limitation if you need that.

My Real-World Usage: How Actually Use This

I use Vibe-Memory every day for a few things:

1. Long-term project context: When I'm working on a side project that I pick up every few weeks, I just add notes as I go, and when I come back, the AI already remembers all the decisions I made, what the current problems are, what I've already tried. No more re-reading all my old notes trying to get back up to speed.

2. Personal journaling with AI: I journal every day, and Vibe-Memory remembers patterns in my mood, what activities make me happy, what problems keep coming up. When I'm thinking through an issue, the AI can pull up similar situations from the past and how I handled them. It's like having a therapist that actually remembers your history.

3. Knowledge base for writing: When I'm writing articles like this one, I can search my old notes, ideas, drafts, and the AI helps me connect dots I would have forgotten.

The ROI for me has been huge. I waste so much less time re-explaining context and searching through old notes. The AI actually feels like it knows me now, instead of starting from zero every conversation.

Getting Started: If You Want to Try This

If you want to try Vibe-Memory, it's open source on GitHub:

https://github.com/kevinten10/vibe-memory

Here's the quickstart to get going:

# Clone it
git clone https://github.com/kevinten10/vibe-memory.git
cd vibe-memory

# Configure
cp config.example.yaml config.yaml
# Edit config.yaml with your PostgreSQL connection string and OpenAI API key

# Run migrations (makes sure the tables are created)
go run cmd/migrate/main.go

# Add a memory
go run cmd/cli/main.go add "Today I built the core memory system and it actually works!" --tags "dev, building-vibe-memory"

# Search for memories
go run cmd/cli/main.go search "how did I build the memory system"

That's it. That's literally all you need to get started. The API is simple if you want to embed it into your own projects:

import "github.com/kevinten10/vibe-memory/pkg/vibememory"

func main() {
    vm, err := vibememory.New(vibememory.Config{
        ConnectionString: "postgresql://...",
        OpenAIAPIKey:    "sk-...",
    })
    if err != nil {
        log.Fatal(err)
    }

    // Add a memory
    err = vm.AddMemory(context.Background(), "My first memory", nil)

    // Search
    results, err := vm.Search(context.Background(), "my first memory", 5)
}

What's Next for Vibe-Memory

I'm still actively building this. Here's what's on the roadmap:

1. Simple web UI — So you don't have to use the CLI if you don't want to
2. One-click deployment — Docker compose, Fly.io template, etc.
3. Multiple embedding model support — Local models (Ollama), Cohere, Anthropic, etc.
4. Better memory management UI — Browse, edit, delete memories easily
5. Integration with popular tools — Obsidian, VS Code, Raycast, etc.

Final Thoughts: Why This Matters

AI is amazing, but right now it's like having a genius friend who has amnesia every time you talk to them. That's fine for quick questions, but for anything long-term, it's broken.

We shouldn't have to constantly re-teach our AI who we are and what we've been doing. Our AI should remember us. It should remember our projects, our ideas, our experiences, our "vibe".

That's what Vibe-Memory is about. It's not going to replace ChatGPT or Claude or whatever you use. It's the missing layer that gives your AI some actual long-term memory.

I've been using it daily for three months now, and honestly, I can't imagine going back to working without it. It's one of those things that once you have it, you wonder how you ever lived without it.

What About You?

Have you built anything like this? Do you struggle with AI forgetting your context? What's your solution for personal semantic memory? I'd love to hear your experiences in the comments — I'm always looking for ideas to improve this.

If you try Vibe-Memory, let me know what you think! PRs and issues are welcome on GitHub — it's open source and I'm actively maintaining it.