R&D: The Art of Managing Uncertainty in Software Development

Here are two facts for you:

R&D is the only path to real innovation - the kind that creates competitive advantages and, potentially, reshapes or even creates entirely new markets
Most managers and business stakeholders are terrified of R&D tasks, and these tasks often get solved "despite" the process, not "because of" it

Let's figure out why this happens and what to do about it.

Preamble

I've wanted to share my thoughts on this topic for years, but never got around to it. I've seen it too many times: tasks with many unknowns treated no differently from the rest of the backlog, with the same cookie-cutter formal approaches and the same production metrics applied across the board. And it ALWAYS caused problems. At best - difficulties with planning and realistic deadline estimation. At worst - failed projects and entire teams burning out. I'm guessing many of you can relate.

The brave soul who takes on an R&D task faces a whole extra set of obstacles: from their own mental traps to a complete lack of understanding of what research work actually involves - from colleagues and management alike. On the other hand, it's exactly these kinds of tasks that make our work genuinely exciting.

The first step toward solving many problems, including the ones mentioned above, is acknowledging them. Regardless of your title, you need to be able to distinguish R&D from everything else. Hiding from uncertainty behind formalism only delays the painful collision with reality.

Throughout this article, I'll be drawing on personal experience, so expect a few true stories.

What Is R&D?

Research And Development - literally what it says on the tin. Development is the part that seems relatively predictable. Research is the space of uncertainties and the previously unknown. R&D tasks vary wildly in complexity and importance. The domain of our ignorance can extend in many directions - not necessarily only the technical one.

For example, I once worked on a telemedicine service, and most of the unknowns we had to deal with came from the legal and regulatory domain. The legal part was the key challenge and critically important. From a technical standpoint, the project was relatively straightforward from the start. The project never saw the light of day - it simply couldn't outlast the wait for a telemedicine law that it was being built for. The government insiders let us down with their insider knowledge.

Another example where the uncertainty was purely technical. We were building a virtual clothing try-on service. When I joined the project, there was already a semi-functional prototype that could load clothing models and simulate fabric behavior on a virtual 3D mannequin matched to the potential user's measurements.

The unknown was whether the heavy simulation process could be scaled at all. Fabric physics is complex stuff, and the prototype could barely run on a dedicated GPU. It seemed impossible to make it work in milliseconds per request. Figuring out HOW to do that - that's R&D.

Back then, nobody had really heard of neural networks yet, so we built an interpolation engine that could produce a plausible-looking image based on several pre-computed simulations, simply shifting vertices (polygon points) toward the nearest calculated values. That project, by the way, also didn't survive - for the very reason I'm trying to describe here: the people making key business decisions didn't understand what they were dealing with or what risks they should be accounting for. By the time we had a truly working concept, it was already too late.

This is how people throw millions out the window. Sometimes it's hard to stop when the investment has already passed a critical threshold. You desperately want to recoup what you've spent, and the only way to do that, it seems, is to invest just a little more. This dangerous mental trap is a direct consequence of not having a separate, systematic approach to R&D.

So what do we do?

Managed Chaos

Surprisingly, many people dislike the word "Chaos," treating it as some synonym for disorder. However, chaos as a mathematical concept is an important characteristic of what inevitably surrounds us, whether we like it or not. Any small thing can trigger serious consequences. The states of complex systems are difficult (impossible) to reliably predict over long time horizons.

At the same time, the field of possible variants and states tends toward a certain common Attractor. To put it simply: we can be wildly wrong about tomorrow's weather, but we always know that winter is generally cold and summer is generally warm (in the corresponding hemisphere). This kind of knowledge, applied to development, comes with experience. An experienced engineer may not be able to name an exact date, but they can feel the "temperature corridor" of a project - where the gravitational center of risk lies, where the zone of relative stability is. They also recognize that the illusion of control through following rituals consumes resources beautifully while guaranteeing, essentially, nothing.

When it comes to R&D, this is especially relevant. The way forward is to learn to read signals from the real world and apply chaos engineering practices - where the primary data source for analysis is the working system itself, at any stage of its lifecycle. Here we simply apply the scientific method and get rid of "shamanic" practices. The problem is that we're surrounded by plenty of "shamans" who don't appreciate this.

The Focus Trap

Focusing on one thing rather than jumping between different aspects of a larger task sounds, at first glance, reasonable. Predictable, controllable... Your average manager likes that. The problem is that it often leads to perfecting one node or aspect of a fundamentally unviable model.

This is essentially the "local optimization" problem: you're polishing one component to a mirror finish without noticing that the entire mechanism is flawed. In conventional development, you might get away with it - at the cost of a refactor. In R&D, the price is different: you can spend months perfecting a component that turns out to be useless after its first contact with reality.

In R&D, it's crucial to periodically step back from narrow or intermediate goals, forming and refining the big picture before diving back into the details. Focusing on small tasks gives a pleasant feeling of progress, but be careful not to fall into this trap - celebrating progress on a road to nowhere.

Here's another example. I worked on a platform for finding and selling auto parts. The complexity was that searches had to run across enormous databases of part numbers, including all possible analogs. You had to find matches across car makes and models, production years, VIN numbers, manufacturer data, supplier and warehouse data - practically in real time. Making this search fast was a non-trivial task, especially considering we were assembling the data from various sources, practically from scratch. One of our microservices was a custom VIN decoding API. We spent a significant amount of time on this task - it turned out to be far less trivial than it looked at first glance. In the end, the project was cancelled and all the work went into the drawer. Our grand plans never even received a fair assessment of feasibility - we simply ran out of time. But while we were grinding away at that VIN decoder, it felt like we were making great progress.

Plan B

It's important to understand that in R&D, a negative result is still a valid result. Sometimes a task hits a previously unknown factor and turns out to be simply unsolvable within the given constraints. If your entire project depends on it, you should ideally think about a "Plan B" while still defining the initial conditions.

Think about what assets you're acquiring along the way and how they might recoup costs, fully or at least partially. Think about possible pivot points. This should be done both upfront and periodically throughout the work. Think in advance about strategies for confronting risks. Many failures can be turned into wins if you think in the right frame.

A good R&D process generates value even on the path to failure. Libraries you've written. Datasets you've collected. Expertise the team has acquired. Patents. Articles. Open-source components. All of these are side assets that can be monetized or reused if the main hypothesis doesn't hold up.

A key practice here is defining kill-criteria in advance. Before work even begins, formulate specific, measurable conditions under which you stop the project. This isn't pessimism - it's professionalism. Without predefined boundaries, it's easy to fall victim to sunk cost fallacy, which was mentioned above.

Fresh Eyes

Sometimes people who've been stewing in a big project for too long lose the ability to see it clearly from the outside. The effects of mental traps and self-compromises made for psychological comfort tend to accumulate over time. Don't assume this is a sign of someone's incompetence. We're all (still) human, and human behavior is inherent to all of us - even the most brilliant among us.

That's why a fresh perspective is so important in projects with a high R&D component. Sometimes it's useful to criticize yourself, retrospectively evaluate your decisions, and be willing to reconsider them. Sometimes it's important to hear an outside opinion. Such opinions may seem amateurish and superficial, but often they reveal exactly how things look from outside your cozy process - the one you undoubtedly understand better than anyone.

The Start-Pause-Stop System

One of the practices used for managing uncertainty. Each new step may depend on the results of the previous one. Sometimes the entire process can hang for a long time waiting for some data. Sometimes work should be stopped entirely and the team should switch to something else. Your processes need to be adapted to three basic states. Pausing the main R&D branch automatically pauses all subtasks. Stopping triggers a preservation algorithm (for further reuse of artifacts, for example in another project, or for possible revival and continuation of work at some undefined point in the future). All process participants must be notified in time about changes in global status and be ready to switch as painlessly as possible. This approach helps significantly save resources, makes the process more transparent and manageable for business, and preserves the work done.

R&D Metrics

Classic development metrics - velocity, story points, burndown charts - work poorly in R&D. They measure speed of movement, but not its direction. You can "burn" story points at cosmic speed while heading in the wrong direction.

For R&D, other indicators are more informative:

Number of hypotheses validated per unit of time - shows real research velocity
Cost of validating a single hypothesis - helps optimize the experimentation process
Time to first contact with reality - the sooner a prototype hits a real environment, the better
Ratio of reusable artifacts to total work volume - shows how much value you're generating "along the way"

These metrics don't give an illusion of control - they give real feedback.

Team Requirements

One important consideration is the elevated requirements for team members working in R&D. Unfortunately, this type of work doesn't suit those who haven't yet had the chance to learn the hard way on real projects. Juniors can be valuable in routine development with a clear spec, but R&D demands something different: the ability to work in conditions where the spec is something you must formulate yourself as you go. You can try running such a project with juniors, mid-levels, or offshore contractors, but there's a high probability the result will be painful. R&D demands a high degree of self-organization and SELF-discipline from the team, and those only come with experience. At the same time, the team needs a creative atmosphere, because the best ideas will come from people who are genuinely INTERESTED. This process is practically impossible to manage effectively through micromanagement and pre-defined formal methodologies. This creates a paradox: cutting costs on people will end up costing you more. At the same time, an R&D team should be compact, so that communication overhead doesn't consume all your resources.

AI

Neural networks are excellent for automating routine work and conducting quick intermediate micro-research. They're also extremely useful for creating side artifacts: documentation, demos, tests, etc.

But there's a nuance. AI works great in the zone of the known - it generalizes patterns from already existing solutions. R&D, by definition, operates in the zone of the unknown. AI is not (yet) capable of generating truly new ideas - it can only combine existing ones within a limited range. This makes it a powerful accelerator for an experienced R&D specialist who knows which questions to ask. But a poor substitute for such a specialist.

Fully replacing R&D specialists is not yet possible. But AI has already become an excellent tool in the hands of an experienced researcher - and the significance of this tool continues to grow.

Crisis-Mode R&D

Consider a situation where R&D becomes not the source of a crisis, but the answer to one. When the market collapses, a technology becomes obsolete, or an unexpected competitor arrives - it's exactly R&D thinking that enables rapid adaptation.

Crisis-mode R&D differs from planned R&D in three ways:

Speed over quality. You don't need a perfect product - you need a working prototype that proves the viability of a new model. MVP in the literal sense.
Resources are limited. Crisis typically means a slashed budget. Paradoxically, this sometimes helps: constraints force more inventive solutions and cut through all the noise.
High risk tolerance. When there's almost nothing left to lose, space opens up for experiments that would be unthinkable in "peacetime."

In essence, a well-established R&D process is your insurance against crises. A team accustomed to working with uncertainty doesn't freeze up when that uncertainty shows up uninvited. They already know how to ask the right questions, build hypotheses, and validate them quickly.

Summary

R&D is not an anomaly in the production process that needs to be "survived." It's a natural and inevitable part of it, especially in industries where the technological landscape changes faster than you can update your documentation. Trying to squeeze research tasks into the framework of a linear production pipeline is one of the most expensive mistakes a business can make.

Key principles to remember:

Acknowledge R&D tasks as a separate category - don't hide behind generic processes
Accept uncertainty as a working factor, not a hostile anomaly
Invest in people - R&D is done by people, not processes
Prepare a Plan B and define kill-criteria in advance
Measure real progress, not the illusion of progress
Use AI as an accelerator, not as a substitute for thinking