When a request takes five seconds, the obvious question is what to make faster. The more useful question is what, precisely, the system spent those five seconds doing, or waiting to do. The answer may lie in unnecessary database work, excessive data movement, an execution plan, contention, or the costs of concurrency - and different causes require fundamentally different interventions. Performance engineering begins not with optimization, but with establishing causality: understanding what the system is doing before deciding what it should do differently.
A request takes five seconds to completed. The database query looks suspicious. The endpoint contains an Include(). The response contains considerably more data than the client appears to need. Somewhere in the service layer there is a loop that makes repeated calls. The application is running asynchronously, but under sufficient concurrency the server begins to queue requests and occasionally time out.
There is no shortage of things that could be optimized.
That is precisely the problem.
Performance engineering becomes difficult not when a system has no apparent inefficiencies, but when it has too many.
A slow request can be the visible consequence of several independent mechanisms: inefficient data access, excessive data movement, contention, blocking, serialization, network latency, or resource exhaustion.
Improving one of them may have little effect on the request - or may simply expose another bottleneck further down the chain.
The first task, therefore, is not optimization. It is diagnosis.
Latency Alone Says Very Little
Suppose an API endpoint takes 4.8 seconds.
That number is useful, but only as a starting point.
It does not tell us whether the database consumed 4.7 seconds or 40 milliseconds. It does not tell us whether the application was computing, waiting for a lock, waiting for another service, or waiting for an available thread. It does not tell us whether the database processed millions of rows to return a handful of records
Two requests can have identical latency while having completely different causes.
Consider two simplified cases.
In the first, SQL Server spends several seconds performing a large scan, executing expensive joins, and retrieving rows that the application subsequently discards.
In the second, SQL Server executes the same query in a few milliseconds, but the request spends several seconds waiting for a lock held by another transaction.
From the perspective of the API consumer:
The endpoint is slow.
From the perspective of the engineer:
These are entirely different problems.
The first calls for investigation of query shape, indexes, filtering, projections, and execution plans.
The second may require understanding transactions, locking behavior, concurrent workload, and the session responsible for blocking.
Optimizing the query in the second scenario could accomplish almost nothing.
This distinction is one of the reasons performance engineering cannot be reduced to a collection of ORM tricks or database tuning recipes.
Begin with the Shape of the Work
A useful way to approach performance problems is to ask a deceptively simple question:
What work is the system actually doing?
Consider a seemingly innocent EF Core operation:
var projects = await context.Projects
.ToListAsync();
The query itself may be perfectly valid. The problem is what happens afterward.
- Perhaps the application filters the projects in memory.
- Perhaps it maps every entity into a large DTO.
- Perhaps it accesses related entities and triggers additional queries.
- Perhaps it serializes fields that the client never requested.
- Perhaps the endpoint eventually returns hundreds of thousands of rows.
The database may not be the fundamental problem. The system may simply be doing far more work than the request requires.
This is why projection, filtering, and pagination are not merely code-style preferences. They are mechanisms for controlling the quantity of work performed by the system.
The earlier unnecessary work can be eliminated, the less work every subsequent stage needs to perform.
- A row that is never selected does not need to be materialized.
- An entity that is never modified does not necessarily need to be tracked.
- A column that is never returned does not need to be transferred.
- A related collection that is never displayed does not need to be loaded.
- And data that has already been computed and remains sufficiently stable may not need to be computed again at all.
Performance optimization is often less about making operations faster than about preventing unnecessary operations from occurring in the first place.
The ORM can Conceal the Database
This becomes particularly important when working with an ORM.
LINQ is expressive enough that relatively small pieces of application code can represent surprisingly complicated SQL.
[HttpGet("nplus1-nested")]
public async Task<ActionResult> GetProjectsWithProposals_NestedNPlus1()
{
var projects = await _context.Projects.ToListAsync();
var result = new List<object>();
foreach (var project in projects)
{
// Query per project
var proposals = await _context.Proposals
.Where(p => p.ProjectId == project.Id)
.ToListAsync();
var proposalDtos = new List<object>();
foreach (var proposal in proposals)
{
// Query per proposal - multiplying the problem
var freelancer = await _context.Users
.FirstOrDefaultAsync(u => u.Id == proposal.FreelancerUserId);
proposalDtos.Add(new
{
ProposalId = proposal.Id,
Bid = proposal.Bid,
FreelancerName = freelancer?.KnownAs
});
}
result.Add(new
{
ProjectId = project.Id,
Proposals = proposalDtos
});
}
return Ok(result);
}
The endpoint above is deliberately simple. It retrieves the projects, then retrieves the proposals for each project, then retrieves the freelancer for each proposal.
Imagine a dataset containing 500 projects and 10 proposals per project. Under the assumptions of this example, the endpoint would perform approximately:
1 + 500 + 5,000 = 5,501 queries.
The code may remain readable and the application may even appear perfectly responsive with ten records. But the problem emerges as the cardinality increases.
This is one of the recurring patterns in performance engineering: an implementation can be locally reasonable and globally pathological.
In this particular example, the solution is not necessarily to make each of those queries faster. It is to change the shape of the operation.
[HttpGet("projects-with-proposals")]
public async Task<ActionResult> GetProjectsWithProposals()
{
var result = await _context.Projects
.Select(p => new
{
p.Id,
Proposals = p.Proposals.Select(pr => new
{
pr.Id,
pr.Bid,
FreelancerName = pr.Freelancer.KnownAs
}).ToList()
})
.ToListAsync();
return Ok(result);
}
Here the relationships are expressed as part of a single projection.
Rather than loading complete entities and navigating between them procedurally, the query describes the data the endpoint actually needs.
The remedy in cases like this is often projection, appropriate joins or grouping, or carefully chosen batched loading. But even here, the first apparent solution can introduce another problem.
Eliminating thousands of round trips is a substantial improvement. But it leaves us with another question:
How much data are we asking that single query to retrieve?
Bad:
1 Project Query → 500 Proposal Queries → 5,000 Freelancer QueriesBetter:
1 Projection Query → Project + Proposals + Freelancer Name
The objective is therefore not:
Make the number of SQL queries as small as possible.
The objective is:
Make the overall data access strategy appropriate for the workload.
That distinction matters.
One query can still be too much. Eliminating thousands of round trips is a significant improvement. But reducing the number of queries does not, by itself, make a data access strategy efficient.
We can replace thousands of small operations with a single operation that retrieves far more information than the request actually requires.
To make the downstream cost visible in a small test dataset, the example below deliberately amplifies several fields and collections. The amplification is artificial; the underlying pattern is not.
[HttpGet("over-fetching")]
public async Task<IActionResult> OverFetching()
{
var data = await _context.Projects
.Include(p => p.Client)
.Include(p => p.Freelancer)
.Include(p => p.Conversations)
.AsNoTracking()
.ToListAsync();
return Ok(data.Select(p => new
{
p.Id,
p.Title,
Client = p.Client == null ? null : new
{
p.Client.Id,
p.Client.KnownAs,
p.Client.FirstName,
p.Client.LastName,
p.Client.Bio,
p.Client.Website,
p.Client.LinkedIn,
p.Client.GitHub,
// Deliberately amplified to make the payload cost visible
BioCopies = Enumerable.Repeat(p.Client.Bio, 500).ToList()
},
Freelancer = p.Freelancer == null ? null : new
{
p.Freelancer.Id,
p.Freelancer.KnownAs,
p.Freelancer.Bio,
BioCopies = Enumerable.Repeat(p.Freelancer.Bio, 500).ToList()
},
Proposals = Enumerable.Range(0, 50)
.SelectMany(_ => p.Proposals)
.ToList(),
Conversations = p.Conversations.Select(c => new
{
c.Id,
Messages = Enumerable.Range(0, 500)
.SelectMany(_ => c.Messages)
.Select(m => new
{
m.Id,
m.Content
})
.ToList()
})
}));
}
The repeated biographies, proposals, and messages are there to make the cost visible in a small dataset. The underlying pattern, however, is quite ordinary.
The endpoint loads complete related entities and entire collections before constructing its response. Only afterward does it decide what information to expose.
That distinction matters.
Data does not become free simply because it arrived in a single query.
Every unnecessary piece of data has consequences beyond the database. It may become an object that EF Core has to materialize, memory that the application has to retain, work that the serializer has to perform, bytes that have to cross the network, and data that the client has to process.
A database optimization that reduces query count while dramatically increasing the amount of data transferred can therefore move the cost rather than eliminate it.
The more fundamental solution is to establish the required data shape before materialization. Filter what is unnecessary, project only the required fields, and paginate collections that do not need to be loaded in their entirety.
In other words, don't ask the database for an object graph and then decide what you need. Whenever practical, ask the database only for what you need, first.
Read the Execution Plan, not Just the LINQ
At this point, we have encountered two very different forms of inefficiency. One produced thousands of database round trips. The other produced a single operation carrying an excessive amount of data. Neither problem can be diagnosed reliably by looking only at how elegant the LINQ appears.
When database work is genuinely contributing to latency, we need to see what the database is actually doing. The database's execution strategy is what matters.
An actual execution plan can reveal behavior invisible at the application layer:
- index seeks versus scans
- expensive key lookups
- inefficient join strategies
- filters being applied later than expected
- large quantities of data being read to produce a small result
- operators responsible for disproportionate execution cost
A query can therefore look perfectly reasonable in C# and still ask SQL Server to do an unreasonable amount of work.
This is why generated SQL and execution plans are indispensable when investigating database performance.
The ORM tells us what we asked the database to do.
The execution plan tells us how the database chose to do it.
Those are not the same thing.
But execution plans have limits
An execution plan is an excellent description of how a query executes. It is not necessarily a description of why a request is slow.
A query can be efficient and still spend most of its lifetime waiting.
Suppose the execution plan looks healthy.
The query uses an appropriate index, returns a small number of rows, and consumes very little CPU.
Yet the endpoint still occasionally takes several seconds.
At this point, continuing to optimize the query may be a category error.
The missing information may exist outside the query itself.
- SQL Server could be waiting on a lock.
- Another session could be holding a transaction open.
- The same query could be executing thousands of times across concurrent requests.
- A parameter-sensitive workload could be causing different parameter values to experience radically different execution behavior.
- Or the database could be perfectly healthy while the application is exhausting its available resources elsewhere.
This is where workload-level instrumentation becomes important.
From the query to the workload
Tools such as SQL Server Extended Events allow us to observe database activity over time rather than examining a single query in isolation.
That temporal dimension changes the investigation.
Instead of asking:
Why did this query take five seconds?
Ask:
What was happening in the database during those five seconds?
Duration becomes one signal among several.
Logical reads tell us how much data was processed, CPU time helps distinguish computation from waiting, row counts provide clues about over-fetching and inefficient access, session identifiers can help establish relationships between activity, query hashes can reveal repeated execution patterns, wait behavior can indicate contention rather than computational expense.
Consider a particularly revealing combination:
high duration + low CPU + relatively modest reads.
That should make us suspicious: The query may not be expensive at all. It may simply be waiting.
And a query that is waiting for another transaction cannot be made meaningfully faster by shaving a few milliseconds from its execution plan.
The bottleneck exists somewhere else in the system.
The same principle applies above the database
The database is only one stage in the request pipeline.
Imagine that a query is optimized from 800 ms to 80 ms. But the endpoint retrieves a large object graph, maps thousands of entities, serializes an oversized JSON response, and transfers it across a relatively high-latency connection.
The database optimization may be real while the user-visible improvement remains disappointing.
The system has simply moved its center of gravity.
This is why performance investigation needs to follow the request across boundaries:
- Database work
- Application computation
- Object mapping
- Serialization
- Network transfer
- Client-side processing
Each stage can amplify the cost of the previous one.
An unnecessarily large result set is not merely a database problem. It creates more objects. More objects create more allocations. More allocations create more garbage collection pressure. More data requires more serialization. More serialization consumes CPU. More serialized bytes require more network bandwidth. More bytes require more client-side processing.
A seemingly small decision at the database layer can therefore propagate through the entire request pipeline.
Reducing data is often more powerful than optimizing its processing
The preceding examples point toward a broader principle.
Performance problems often become easier to solve when we reduce the quantity of work rather than attempting to accelerate the work itself.
Projection reduces columns.
Filtering reduces rows.
Pagination reduces result sets.
DTO shaping reduces object graphs.
Caching eliminates repeated work.
Aggregation reduces network round trips.
Batching reduces repetitive database interaction.
These techniques look unrelated when presented as isolated recommendations. Conceptually, however, they are variations of the same strategy: Reduce unnecessary work and unnecessary movement.
When Parallelism is the Right Tool
Parallel execution is not inherently a performance problem. In the right circumstances, it is one of the most effective ways to reduce elapsed time.
The strongest candidates are workloads in which the operations are genuinely independent, contain enough useful work to amortize the cost of parallelization, and have sufficient computational or I/O capacity available to execute concurrently.
For example, imagine an application that needs to process a large collection of independent documents, perform CPU-intensive transformations on each one, and combine the results afterward. There is little value in forcing one document to wait for another when the operations have no dependency between them.
The same principle applies to independent I/O operations. If several remote requests can proceed concurrently, waiting for each one sequentially can unnecessarily extend the critical path.
This is where constructs such as Task.WhenAll can be valuable:
var results = await Task.WhenAll(
GetCustomerAsync(),
GetPortfolioAsync(),
GetMarketDataAsync()
);
The operations can make progress concurrently rather than forcing the second operation to wait for the first, and the third to wait for the second.
But concurrency is not free.
The fact that operations can execute concurrently does not mean that they should execute without limit. The useful question is not simply whether work can be parallelized, but whether the system has enough available resources, and enough useful work, to make parallel execution worthwhile.
That distinction becomes clearer in a small benchmark.
When Parallelism Makes Things Worse
A system can sometimes do more work in parallel and become slower rather than faster.
Parallelism has a cost. Work must be partitioned, scheduled, synchronized, and eventually recombined. If the amount of useful work is not sufficient to offset those costs, introducing parallel execution can make the operation slower rather than faster.
I built a deliberately simple BenchmarkDotNet experiment to make that tradeoff visible.
Both implementations retrieve the same 5,000 projects asynchronously from SQL Server. They then perform the same CPU-bound operation on every project. The only difference is how that processing is performed: sequentially in one case, and through Parallel.ForEach in the other.
[Benchmark]
public async Task<int> SequentialProcessing()
{
var projects = await _context.Projects
.AsNoTracking()
.Take(5000)
.ToListAsync();
int total = 0;
foreach (var project in projects)
{
total += SimulateWork(project.Title);
}
return total;
}
[Benchmark]
public async Task<int> ParallelProcessing()
{
var projects = await _context.Projects
.AsNoTracking()
.Take(5000)
.ToListAsync();
int total = 0;
Parallel.ForEach(projects, project =>
{
Interlocked.Add(
ref total,
SimulateWork(project.Title));
});
return total;
}
The result was not what a simplistic notion of parallelism might suggest.
| Method | Mean | Allocated |
| -------------------- | -------: | --------: |
| SequentialProcessing | 206.5 μs | 31.12 KB |
| ParallelProcessing | 217.0 μs | 33.96 KB |
On this workload, the parallel implementation was approximately 5% slower than the sequential implementation, while also allocating somewhat more memory.
The important observation is not that Parallel.ForEach is "slow." Nor is the lesson that sequential execution is preferable.
The lesson is that parallelism has economics.
There is a cost associated with introducing concurrency, and that cost has to be amortized by the work being parallelized. Partitioning work, scheduling execution, coordinating workers, and synchronizing shared state all introduce overhead.
If that overhead is greater than the time saved through concurrent execution, parallelism has made the operation less efficient.
This is why performance engineering resists rules such as "parallel is faster." The answer depends on the workload, the cost of the individual operation, the available hardware, and the resources competing for that hardware.
The same principle becomes even more important at the application level.
Increasing concurrency can improve throughput when independent work can genuinely proceed simultaneously. But unbounded concurrency can also exhaust database connection pools, increase contention, overwhelm downstream services, or simply move the bottleneck somewhere else.
More parallelism does not necessarily mean more throughput.
Sometimes it simply means that the bottleneck is reached faster.
The Optimization Hierarchy
After working through these patterns, I find it useful to think about performance investigations as a progression rather than a checklist.
- First: Where is the time going?
- Then: What work is being performed?
- Then: Which of that work is actually necessary?
- Then: Is the remaining work being performed efficiently?
- And finally: What happens when many requests perform this work concurrently?
That sequence matters.
It is tempting to begin with familiar optimizations: AsNoTracking(), caching, an additional index, Task.WhenAll(), response compression, a larger connection pool, more threads.
Sometimes these are exactly the right interventions.
But applying them before establishing the bottleneck turns optimization into speculation. And speculation is expensive. It consumes engineering time, increases system complexity, and can produce changes that improve a benchmark while making the production system less predictable.
Across the examples in this article, the specific mechanisms are different, but the underlying question is the same.
- In the first case, the application performed the same category of database work thousands of times.
- In the second, it moved and materialized substantially more data than necessary.
- In the third, it introduced parallel execution whose overhead could exceed its benefit.
In each case, the important question was not simply "Can this operation be made faster?"
It was
"What work is the system doing, and is that work justified by the result?"
Performance Engineering is an Exercise in Causality
The deeper lesson is that performance work is fundamentally an exercise in establishing causality.
We observe a symptom, formulate a hypothesis, collect evidence, identify the mechanism producing the symptom, change the mechanism. Then we measure again.
That sounds obvious, but it is surprisingly easy to skip.
A slow endpoint invites a fix.
A high CPU graph invites more CPU.
A slow query invites an index.
A large response invites compression.
A concurrency problem invites more parallelism.
Sometimes the intuition is correct.
But the engineering discipline lies in refusing to confuse correlation with cause.
A performance engineer should be able to look at a five-second request and resist the urge to immediately make something faster.
The more important question is:
What, precisely, is responsible for those five seconds?
Until that question has a defensible answer, optimization has not really begun. It is only intervention. And intervention without diagnosis is often just another form of guesswork.
The code examples in this article were adapted from an original .NET freelancer marketplace application I built from scratch and subsequently simplified to demonstrate the performance issues discussed herein. The original project is available on GitHub, and the original performance optimization code is available in the copilot-sanbox branch:
tigerbluejay / Angular-.Net-Integration-Freelancer-Marketplace-App
.NET 8 API with Angular 17/18 Client - An application for freelancers and their clients to work together
💼 Freelancer Marketplace
A full-stack freelancer marketplace web application built with ASP.NET 8 Web API and Angular 18. Users can register, login, and participate with three distinct roles: Administrator, Client, and Freelancer. Freelancers can create profiles, manage portfolios, browse and apply for projects, and chat with clients in real-time. Clients can post projects, manage proposals, and collaborate with freelancers. Admins can manage users and enforce platform rules.
🚀 Features
✅ User Registration & Login
- Secure authentication using JWT tokens
- Role-based authorization (Admin / Client / Freelancer)
✅ Freelancer Features
- Profile management with editable details
- Create, edit, delete portfolio items (title, description, photo) with pagination
- Browse projects and filter by required skills (5 distinct skills available)
- Submit proposals for projects
- View submitted proposals (filter by Approved / Rejected / Pending)
- Approved proposals automatically generate active projects and enable real-time chat with clients
- Messaging system with unread counters, last message preview, timestamps…
