The Pipeline Worked. Then the Research Outgrew It.

About a year ago, I was building a terminal-based workflow manager called Glyph.Flow.

It was mostly a learning project. I wanted to understand Python better, experiment with Textual, think about commands, state, configuration, logging, and all the small architectural decisions that suddenly appear when a script stops being a script.

Somewhere between then and now, the workflows became a little more real.

For my Master's thesis, I built a data pipeline to construct and process a cross-national research database from multiple sources. It had a clear purpose: take heterogeneous input data, transform it consistently, validate important assumptions, and produce the dataset I needed for the analysis. And it worked.

But this is no longer enough. I am not rebuilding it because the original system failed. I am rebuilding it because the question changed:

My Master's thesis needed a pipeline.
My PhD will need research infrastructure.

And I am slowly discovering that these are not the same thing.

A pipeline can be finished

There is something comfortable about building software for a well-defined research project.

You know the research question. You know most of the variables you need. You know which datasets are involved. You can define the transformations, produce the outputs, validate them, run the analysis, and eventually say:

Done.

Of course, research is never really that clean. Data sources change. Weird edge cases appear. A country disappears from one dataset. Another source changes a variable name. An indicator turns out to mean something slightly different than you thought. But there is still a boundary around the problem. A PhD changes that boundary.

Now I have to think about a system that may need to survive several years of research, new questions I have not formulated yet, datasets I have not discovered yet, and methodological decisions I will probably reconsider more than once.

Suddenly, "Does it work?" becomes a surprisingly weak design criterion.

The more useful questions are becoming things like:

Can I extend it without breaking old processing steps?
Can I tell exactly where an output value came from?
Can I rerun only the parts affected by a change?
Can I replace a data source without rewriting half the project?
Can I detect when something is technically valid but scientifically suspicious?
Will I still understand my own decisions two years from now?

That's a different problem.

The dangerous success of a working system

I think working software creates an interesting trap.

Once something reliably produces the output you need, there is a strong temptation to treat its architecture as validated.

But successful execution only proves a surprisingly narrow thing:

the system can currently do what you currently ask it to do.

It says much less about what happens when the scope changes.

This is especially important in research software because the code and the research process evolve together. A new theoretical question may require a new variable and a new variable may require another source. That source may use different country identifiers, time periods, aggregation rules, missing-value conventions, or definitions. Then a seemingly small analytical change starts travelling backwards through the entire pipeline.

The problem is no longer just transformation.

It is dependency. And provenance. And validation. And maintainability. And eventually, architecture.

From a research product to a research process

This is where my thinking has shifted the most.

For the Master's thesis, I was primarily building a research product: a reproducible database suitable for a specific analysis. Now I need to think much more explicitly about the research process that keeps producing and modifying that database. That means treating things I previously considered implementation details as first-class design problems.

Where should validation happen?
What should a processing unit actually be responsible for?
How should dependencies between datasets be represented?
Which transformations should be configuration and which should be code?
What information should be stored about every intermediate dataset?
When should a pipeline stop instead of trying to recover?

And perhaps the most uncomfortable question:

How much flexibility should I design for when I do not yet know what I will need?

Too little, and every new research question becomes a refactor. Too much, and I spend six months building a beautiful general-purpose framework instead of doing real social research.

I suspect the useful architecture is somewhere in the annoying middle.

Reproducibility is necessary, but not enough

Researchers talk a lot about reproducibility, for good reason. Given the same data and the same code, we should be able to reproduce the same result. But for a long-running research system, I think that is only the starting point.

I also want to be able to understand why a result looks the way it does. Which source supplied this value? Which transformations affected it? Which version of the processing logic produced it? What assumptions were validated? What changed since the previous run?

That starts sounding less like a collection of scripts and more like infrastructure. Not infrastructure in the "let's deploy Kubernetes for a CSV file" sense. Quite the opposite. The challenge is figuring out the smallest amount of structure that makes the research safer, more transparent, and easier to extend.

So this time, I'm designing before rebuilding

My instinct as a hobby programmer has often been simple:

See problem. Write code. Refactor when it becomes ugly.

That approach taught me a lot. Glyph.Flow itself was basically one long experiment in discovering architectural problems by running directly into them.

Research data is less forgiving. So before turning the current pipeline into anything larger, I want to spend some time defining what "larger" actually means. Not choosing frameworks. Not writing orchestration code. Not drawing an impressive architecture diagram with seventeen boxes. Just identifying the problems the system needs to solve. The technical implementation will come later.

For now, I'm starting with a more fundamental question: