A New Way to Build Aggregation Pipelines in Go

This article was written by Lin Borland

Aggregation pipelines are one of the most powerful tools in MongoDB. They let you filter, reshape, compute, and group documents in a single query.

In practice, the aggregation framework feels almost like a language of its own. With its combination of stages, expressions, and operators, you can describe everything from straightforward filtering to sophisticated transformation logic. This expressive power is what makes aggregation pipelines so useful, and is also why they have a learning curve associated with them.

If you’ve worked with MongoDB in Go, you may know that the existing syntax for writing pipelines in Go can be cumbersome to work with. This is especially true when a pipeline includes several stages, repeated computed logic, or deeply nested expressions. In these cases, both readability and writability may begin to suffer.

There’s a need for a more Go-native way to build aggregation pipelines. This is why we’re introducing a new approach: an experimental aggregation builder in Go.

In this article, we’ll compare the traditional and new approaches, then go through an example.

The traditional BSON-based approach

Today, if you want to build an aggregation pipeline with the Go driver, you typically do it with bson.D, bson.A, and mongo.Pipeline. While this approach is flexible, it can be hard to spot small mistakes.

Let’s use a simple example from the sample_mflix.movies collection.

Suppose we want to find movies released after the year 2000. Here’s a pipeline that demonstrates how easy it can be to get the shape wrong:

mongo.Pipeline{bson.D{{Key: "$match", Value: bson.E{Key: "$gte", Value: bson.E{Key: "$year", Value: 2000}}}}}

At a glance, the mistake might not be obvious. The document is valid BSON, but the pipeline uses “bson.E” instead of “bson.D” for some values, resulting in a pipeline that returns zero results.

If we try to fix the nesting, we can still end up with a pipeline that is structurally valid BSON but invalid aggregation syntax:

mongo.Pipeline{bson.D{{Key: "$match", Value: bson.D{{Key: "$gte", Value: bson.D{{Key: "$year", Value: 2000}}}}}}}

This fails at runtime with an error like:

(BadValue) unknown top-level operator: $gte. If you have a field name that starts with a '$' symbol, consider using $getField or $setField.

Here’s the corrected version:

mongo.Pipeline{bson.D{{Key: "$match", Value: bson.D{{Key: "year", Value: bson.D{{Key: "$gte", Value: 2000}}}}}}}

It would be helpful if the compiler could guide us toward this version more easily. This is why we built a new aggregation builder API in Go.

The new approach

Instead of expressing every stage as raw BSON, the new API lets you build pipelines out of typed helpers for stages, expressions, and accumulators.

The underlying aggregation logic stays the same, but the code reads differently. Stage boundaries are more explicit, nested expressions are easier to follow, and repeated logic can be extracted into helper functions more naturally.

As pipelines become more complex, developer tooling becomes increasingly valuable. The builder API allows for autocomplete and compile-time checks, guiding you toward what fits next and catching mistakes earlier. This also lessens the need to constantly reference documentation.

To see what that looks like in practice, let’s compare the same $match stage in the new API. Here’s the same mistake of using $gte at the top level:

agg.Pipeline{agg.MatchStage(agg.Gte("$year", 2000))}

This produces a compiler error:
cannot use agg.Gte("$year", 2000) (value of struct type agg.BoolExpr) as query.Filter value in argument to agg.MatchStage

The correct version is:

agg.Pipeline{agg.MatchStage(query.Field("year", query.Gte(2000)))}

Before running the program, the compiler has already pointed us toward the right shape.

Now that we’ve seen the basic difference, let’s move into a more complex example.

The example: Since 2000, which genres have the most highly-rated movies

For this example, we’ll be using MongoDB’s sample_mflix.movies collection.

The question we want to answer is:
Since 2000, which genres have the most highly-rated movies?

We will define a simple blendedScore for each movie based on its IMDb rating, Rotten Tomatoes viewer rating, and the number of comments (capped at 20):
blendedScore = (IMDb rating × 10) + (Rotten Tomatoes rating × 20) + min(comments × 2, 20)

From there, the pipeline will:

1. Filter to movies released in 2000 or later
2. Compute the score
3. Unwind the genres array
4. Group by genre
5. Count how many strong movies each genres has
6. Sort the results

Here’s is what our example looks like in the traditional BSON form:

mongo.Pipeline{
        bson.D{{Key: "$match", Value: bson.D{
            {Key: "type", Value: "movie"},
            {Key: "year", Value: bson.D{{Key: "$gte", Value: 2000}}},
        }}},
        bson.D{{Key: "$project", Value: bson.D{
            {Key: "genres", Value: 1},
            {Key: "blendedScore", Value: bson.D{{Key: "$add", Value: bson.A{
                bson.D{{Key: "$min", Value: bson.A{
                    bson.D{{Key: "$multiply", Value: bson.A{
                        bson.D{{Key: "$cond", Value: bson.A{
                            bson.D{{Key: "$isNumber", Value: "$num_mflix_comments"}},
                            "$num_mflix_comments",
                            0,
                        }}},
                        2,
                    }}},
                    20,
                }}},
                bson.D{{Key: "$multiply", Value: bson.A{
                    bson.D{{Key: "$cond", Value: bson.A{
                        bson.D{{Key: "$isNumber", Value: "$imdb.rating"}},
                        "$imdb.rating",
                        0,
                    }}},
                    10,
                }}},
                bson.D{{Key: "$multiply", Value: bson.A{
                    bson.D{{Key: "$cond", Value: bson.A{
                        bson.D{{Key: "$isNumber", Value: "$tomatoes.viewer.rating"}},
                        "$tomatoes.viewer.rating",
                        0,
                    }}},
                    20,
                }}},
            }}}},
        }}},
        bson.D{{Key: "$unwind", Value: "$genres"}},
        bson.D{{Key: "$group", Value: bson.D{
            {Key: "_id", Value: "$genres"},
            {Key: "movieCount", Value: bson.D{{Key: "$sum", Value: 1}}},
            {Key: "totalBlendedScore", Value: bson.D{{Key: "$sum", Value: "$blendedScore"}}},
            {Key: "strongMovieCount", Value: bson.D{{Key: "$sum", Value: bson.D{{Key: "$cond", Value: bson.A{
                bson.D{{Key: "$gte", Value: bson.A{"$blendedScore", 180}}},
                1,
                0,
            }}}}}},
        }}},
        bson.D{{Key: "$sort", Value: bson.D{
            {Key: "strongMovieCount", Value: -1},
            {Key: "movieCount", Value: -1},
        }}},
}

What stands out is how quickly the structure becomes dense once the nesting starts. The blendedScore calculation is a good example: the actual scoring logic is relatively simple, but it is wrapped in multiple layers of bson.D, bson.A, and braces. By the time you get into the nested $cond, $multiply, and $min expressions, it takes more effort to visually trace where one operator ends, and the next begins.

Now here’s the same pipeline using the new API:

asNumber := func(field string) agg.AnyExpr {
        return agg.Cond(agg.IsNumber(field), field, 0)
    }

agg.Pipeline{
        agg.MatchStage(
            query.Field("type", query.Eq("movie")),
            query.Field("year", query.Gte(2000)),
        ),
        agg.ProjectStage(
            agg.Include("genres"),
            agg.Compute("blendedScore", agg.Add(
                agg.Min(agg.Multiply(asNumber("$num_mflix_comments"), 2), 20),
                agg.Multiply(asNumber("$imdb.rating"), 10),
                agg.Multiply(asNumber("$tomatoes.viewer.rating"), 20),
            )),
        ),
        agg.UnwindStage("$genres"),
        agg.GroupStage(
            "$genres",
            agg.Accumulate("movieCount", agg.SumAccumulator(1)),
            agg.Accumulate("totalBlendedScore", agg.SumAccumulator("$blendedScore")),
            agg.Accumulate(
                "strongMovieCount",
                agg.SumAccumulator(
                    agg.Cond(
                        agg.Gte("$blendedScore", 180),
                        1,
                        0,
                    ),
                ),
            ),
        ),
        agg.SortStage(
            agg.Sort("strongMovieCount", agg.Desc),
            agg.Sort("movieCount", agg.Desc),
        ),
    }

This version reads more like composed Go code than embedded document structure. In the blendedScore expression, the indentation follows the calculation itself, so it’s easier to see that the score is built from three parts: a capped comment boost, an IMDb contribution, and a Rotten Tomatoes contribution.

Also note that the repeated numeric check has been factored out into the asNumber() helper. This kind of extraction is possible with the BSON-based approach, but it feels especially natural here because the builder API is already functional in shape: each stage, expression, and accumulator is built by composing smaller expression helpers.

This functional style is incredibly useful. As aggregation logic grows, small helpers like asNumber() make it easier to name common patterns and reuse them across expressions.

Closing thoughts

Aggregation pipelines are one of the most expressive tools in MongoDB. With this expressivity comes complexity, which can make it difficult to maintain and scale pipelines over time.

This is the value of the aggregation builder API. It gives Go developers another way to write pipelines–one that can be easier to follow, easier to refactor, and better supported by the editor through autocomplete and compile-time checks.

This does not replace the existing BSON-based approach. bson.D and bson.A give you a direct and flexible way to express any pipeline, and they can still serve as an escape hatch when the builder API fails to express the logic you need.

What this adds is another option. If you are writing aggregation-heavy Go code, the builder API can improve the development experience for existing Go developers and lower the barrier to entry for newer ones.

This API is experimental, which is an important part of the story. It gives us room to explore how this approach is useful in practice, and how it should evolve from here.

We’d love for you to try it out and share your feedback with us. Click here to get started!