Spring Data Unlocked: Advanced Queries With MongoDB

Description: Explore Spring Data's capabilities by creating advanced queries with MongoRepository and MongoTemplate. This second part of the series covers derived queries, @Query, @Update, @Aggregation, pagination, and bulk operations.

I bet you're here because you've already read the first article in this series, Getting Started With Java and MongoDB. But even if you haven't, don't worry—you're in the right place. In the first part of this series, we explored key concepts related to Spring and learned how to create a project and configure it to seamlessly integrate with MongoDB.

This is the second part of the series Spring Data Unlocked, where we will continue working on our project and explore Spring Data's capabilities by creating advanced queries and seeing how seamlessly it can be done.

Pre-requisites

• Source code
• MongoDB Atlas account
  • Get started with MongoDB Atlas for free! If you don't already have an account, MongoDB offers a free-forever Atlas cluster.
• Java 21+
• Gradle 8.8+
• IDE of your choice

Refreshing our memory

If you don't remember what we discussed earlier, don't worry. Read the previous article on how to get started with Java and MongoDB or refresh your memory with the image of our data model that we are working on.

Our transaction model contains this structure, and we will proceed with it.

The MongoRepository

Still focusing on productivity and speed, we will continue using our MongoRepository interface and create a query to retrieve our transactions by type. Open the TransactionRepository class and add the following method:

List<Transaction> findByTransactionType(String type);

As you can see, we are finding all transactions by the type. Spring, through naming conventions (derived queries), allows you to create queries based on the method names.

Tip: We can monitor the log process by enabling a DEBUG setting in the application.properties file:

logging.level.org.springframework.data.mongodb=DEBUG

We will be able to see the query in the console:

2024-10-15T18:30:33.855-03:00 DEBUG 28992 --- [SpringShop] [nio-8080-exec-6] o.s.data.mongodb.core.MongoTemplate : find using query: { "transactionType" : "Transfer"} fields: Document{{}} sort: { "transactionType" : "Transfer"} for class: class com.mongodb.Transaction in collection: transactions

Now, let's say we want all transactions where the amount is greater than 3000:

List<Transaction> findByAmountGreaterThan(double amount);

It is possible to delete records in the same way. Take a look:

void deleteByTransactionType(String type);

If everything is working correctly, your code should be similar to this:

package com.mongodb;

import org.springframework.data.mongodb.repository.MongoRepository;
import org.springframework.stereotype.Repository;
import java.util.List;

@Repository
public interface TransactionRepository extends MongoRepository<Transaction, String> {
    List<Transaction> findByTransactionType(String type);
    List<Transaction> findByAmountGreaterThan(double amount);
    void deleteByTransactionType(String type);
}

@Query

Continuing our development, we can use the Query class that Spring provides instead of working with derived queries. This alternative is also interesting. Below is an example of how to fetch transactions by status, displaying (Projection) only the fields createdAt, accountDetail, and amount, and sorting by createdAt in descending order:

@Query(
    value = "{ 'status' : ?0 }",
    fields = "{ 'createdAt': 1, 'accountDetails' : 1, 'amount' : 1 }",
    sort = "{ createdAt: -1 }"
)
List<Transaction> findByStatus(String status);

@Update

We can also perform update operations on our database using this annotation. Notice in the code below that we are combining @Query and @Update. In the first part, we apply the filter, and then we proceed to update the status.

@Query("{ '_id' : ?0 }")
@Update("{ '$set' : { 'status' : ?1 } }")
void updateStatus(String id, String status);

@Aggregation

Another important annotation in Spring Data is @Aggregation. As the name suggests, it allows us to create aggregation stages. Let's imagine the following scenario:

We want to return the total amount grouped by transaction type. To do this, we'll create the following method:

@Aggregation(pipeline = {
    "{ '$match': { 'transactionType': ?0 } }",
    "{ '$group': { '_id': '$transactionType', 'amount': { '$sum': '$amount' } } }",
    "{ '$project': { 'amount': 1 } }"
})
List<Transaction> getTotalAmountByTransactionType(String transactionType);

• In the first stage, $match, we filter the transactions by the specified type.
• In the second stage, $group, we group the transactions by transactionType and sum the values from the amount field.
• Finally, in the $project stage, we display the total amount.

A new request from our product team has come in, and they want us to save all transactions with an error status in a new collection. We could set up a job to run once a day and trigger a method to handle this for us. For that, we can use MongoDB's $out stage:

@Aggregation(pipeline = {
    "{ '$match': { 'status': 'error' } }",
    "{ '$project': { '_id': 1, 'amount': 1, 'status': 1, 'description': 1, 'createdAt': 1 } }",
    "{ '$out': 'error_transactions' }"
})
void exportErrorTransactions();

After this method is executed, the aggregation will process, and any documents with an error status will be inserted into a new collection within the same database called error_transactions.

Let's break it down again:

• $match to filter transactions with an error status
• $project to specify which fields we want to include in the error collection
• $out to define the collection where we will export the error transactions

Note: While we won't go into the specifics of $out, I highly recommend reviewing its behavior before using it in a production environment.

As you can see, this annotation is very powerful and can provide features for increasingly complex queries. Let's assume we now have a new collection called customers with some additional fields:

{
  "name": "Ricardo Mello",
  "email": "ricardo.mello@mongodb.com",
  "accountNumber": "2234987651",
  "phone": "5517992384610",
  "address": {
    "street": "Street 001",
    "city": "New York"
  }
}

We can use $lookup to perform a join between the transaction and customer collections using the accountNumber field and bring in extra values like phone and address.

@Aggregation(pipeline = {
    "{ '$lookup': { " +
        "'from': 'customer', " +
        "'localField': 'accountDetails.originator.accountNumber', " +
        "'foreignField': 'accountNumber', " +
        "'as': 'originatorCustomerDetails' } }",
    "{ '$project': { " +
        "'amount': 1, " +
        "'status': 1, " +
        "'accountDetails': 1, " +
        "'originatorCustomerDetails': 1 } }"
})
List<CustomerOriginatorDetail> findTransactionsWithCustomerDetails();

In the code above, we're using the $lookup operator to join the new customer collection through the accountNumber field, and returning a new list of CustomerOriginatorDetail.

public record CustomerOriginatorDetail(
    double amount,
    String status,
    Transaction.AccountDetails accountDetails,
    List<OriginatorCustomerDetails> originatorCustomerDetails
) {
    private record OriginatorCustomerDetails(
        String name,
        String accountNumber,
        String phone,
        Address address) {
        private record Address(String city, String street) {}
    }
}

Note: Remember, we're not focusing on data modeling here, just showing that it's possible to work with $lookup in this context.

Custom aggregation

Another approach with the @Aggregation annotation is to create an interface that represents an aggregation, which simplifies its usage. Our SearchAggregate annotation enables the creation of reusable search functionality using MongoDB's aggregation framework, facilitating efficient and structured querying of text data in your database.

import org.springframework.data.mongodb.repository.Aggregation;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Retention(RetentionPolicy.RUNTIME)
@Target({ ElementType.METHOD })
@Aggregation(pipeline = {
    "{ '$search': { 'text': { 'query': ?0, 'path': ?1 } } }"
})
@interface SearchAggregate {
}

Now, to utilize our annotation in a method, we can pass the arguments for the query and path in the TransactionRepository class:

@SearchAggregate
List<Transaction> search(String query, String path);

PagingAndSortingRepository

An important aspect to note is that if we look through the interfaces extended by MongoRepository, we'll find PagingAndSortingRepository, which includes the method:

Page<T> findAll(Pageable pageable);

This method is crucial for working with pagination. Let's go back to our TransactionService and implement the following code:

public Page<Transaction> findPageableTransactions(Pageable pageable) {
    return transactionRepository.findAll(pageable);
}

Now, in the TransactionController, we can make the paginated call as follows:

@GetMapping("/pageable")
public PagedModel<Transaction> findAll(
    @RequestParam(defaultValue = "0") int page,
    @RequestParam(defaultValue = "100") int sizePerPage,
    @RequestParam(defaultValue = "ID") String sortField,
    @RequestParam(defaultValue = "DESC") Sort.Direction sortDirection) {
    Pageable pageable = PageRequest.of(page, sizePerPage, Sort.by(sortDirection, sortField));
    return new PagedModel<>(transactionService.findPageableTransactions(pageable));
}

And finally, we can call our method using the following curl command:

curl --location 'http://localhost:8080/transactions?page=0&sizePerPage=10&sortField=description&sortDirection=ASC'

The MongoTemplate

A very interesting alternative to achieve flexibility and control over MongoDB operations is MongoTemplate. This template supports operations like update, insert, and select and also provides a rich interaction with MongoDB, allowing us to map our domain objects directly to the document model in the database. Let's start by creating a new MongoConfig class:

package com.mongodb;

import com.mongodb.client.MongoClient;
import com.mongodb.client.MongoClients;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.mongodb.core.MongoOperations;
import org.springframework.data.mongodb.core.MongoTemplate;

@Configuration
public class MongoConfig {
    @Bean
    public MongoClient mongoClient() {
        MongoClientSettings settings = MongoClientSettings.builder()
            .applyConnectionString(new ConnectionString("<your connection string>"))
            .build();
        return MongoClients.create(settings);
    }

    @Bean
    MongoOperations mongoTemplate(MongoClient mongoClient) {
        return new MongoTemplate(mongoClient, "springshop");
    }
}

We can observe the @Bean annotation from Spring, which allows us to work with the injected MongoTemplate in our services. Continuing with our development, we will work with the Customer class to manipulate it. To do so, create the Customer record:

package com.mongodb;

public record Customer(
    String name,
    String email,
    String accountNumber,
    String phone,
    Address address
) {
    public record Address(
        String street,
        String city
    ) {}
}

Insert

Let's start with the basic insertion model and then evolve to other operations. To do this, we will create our CustomerService with the following method:

import org.springframework.data.mongodb.core.MongoOperations;
import org.springframework.stereotype.Service;

@Service
public class CustomerService {
    private final MongoOperations mongoOperations;

    CustomerService(MongoOperations mongoOperations) {
        this.mongoOperations = mongoOperations;
    }

    public Customer newCustomer() {
        var customer = new Customer(
            "Ricardo",
            "ricardohsmello@gmail.com",
            "123",
            "1234",
            new Customer.Address("a", "Sp")
        );
        return mongoOperations.insert(customer);
    }
}

Our service will work with mongoOperations, which will handle the insertion of our document.

Note: I am creating new Customer to illustrate the insertion. A good practice would be to receive this Customer as an argument in our function.

BulkWrite

One way to handle bulk insertions is by using bulkWrite. The MongoTemplate provides us with the bulkOps method, where we can provide a list, and it will be executed in batches:

public void bulkCustomer(List<Customer> customerList) {
    BulkWriteResult result = mongoOperations.bulkOps(
        BulkOperations.BulkMode.ORDERED, Customer.class
    ).insert(customerList)
     .execute();

    System.out.println(result.getInsertedCount());
}

Query

Moving on to queries, we have the implementation of queries in the MongoTemplate class. In the code below, we are searching for a customer by email and returning only one:

public Customer findCustomerByEmail(String email) {
    return mongoOperations.query(Customer.class)
        .matching(query(where("email").is(email)))
        .one()
        .orElseThrow(() -> new RuntimeException("Customer not found with email: " + email));
}

The query method (imported statically) expects a Criteria (where), also imported statically, where we can apply various filters such as gt(), lt(), and(), and or(). For reference, see the Spring documentation.

Aggregation

Let's imagine we need a query that returns the total number of customers by city. To do this, we need to group by city and count the number of customers. To achieve this goal, first, let's create a new record to handle it:

public record CustomersByCity(
    String id,
    int total
) {}

And then, we will create a totalCustomerByCity method:

public List<CustomersByCity> totalCustomerByCity() {
    TypedAggregation<Customer> aggregation = newAggregation(Customer.class,
        group("address.city")
            .count().as("total"),
        Aggregation.sort(Sort.Direction.ASC, "_id"),
        project(Fields.fields("total", "_id")));

    AggregationResults<CustomersByCity> result = mongoOperations.aggregate(aggregation, CustomersByCity.class);
    return result.getMappedResults();
}

The static method newAggregation offers us a good approach to manipulate our aggregation stages. We are grouping by city and using the count() function, which internally performs a sum on the total number of customers, sorting by _id in ascending order and displaying both the total and id fields.

Conclusion

In this second part of our series, Spring Data Unlocked, we explored how to create complex queries with MongoRepository and MongoTemplate. This tutorial covered concepts such as pagination, custom annotations, bulk inserts, and powerful aggregation queries.

The complete code is available in mongo-developer.

If you have any questions, feel free to leave them in the comments.