You ask how to handle late-arriving events in your pipeline. The answer is clean, modern and sensible. It involves Kafka, a streaming framework, and a state store.
You run batch. Everything runs batch. The batch runs on an Airflow instance somebody set up in 2021, the transformations are four hundred stored procedures nobody wants to touch, and the team spent the last quarter learning dbt and is finally productive with it.
None of that was secret. You mentioned Airflow. You just mentioned it the way you would mention the weather.
🔍 Facts and constraints are different things
Here is the sentence most people write:
We use Airflow for orchestration and mostly work in dbt.
And here is what it does not say: whether any of that can change.
Read as a fact, "we use Airflow" is background. It describes where you are, not where you have to stay. An answer that suggests moving to something else has not contradicted anything you said. You said what you use. You did not say you were stuck with it.
Read as a constraint, the same information is completely different:
Orchestration is Airflow and cannot change this year. Transformations must stay in dbt because the team learned it last quarter and is finally productive. We can add libraries but cannot add infrastructure.
That version rules things out. The first version does not rule anything out, so nothing gets ruled out.
This is a different problem from not describing your system at all. You can write a perfectly accurate description of your stack, engine versions and everything, and still receive an answer that quietly replaces half of it, because a description tells you what exists and says nothing about what is fixed.
🧠 Why this catches intermediates specifically
Beginners usually are not asked architecture questions.
Seniors state constraints reflexively, because they have had a recommendation die in a planning meeting over something nobody put in writing.
Intermediates are at the point where they get asked "how should we handle this," which is an architecture question, while still describing their situation the way they would describe it to a colleague. And to a colleague it works fine, because a colleague knows the stored procedures are not going anywhere. They have been on the same team as you for two years. The context they are filling in is not in your sentence, it is in the building.
There is a second reason, and it is about how the constraints feel from inside. The reason you cannot replace Airflow is not technical. It is that the migration would take a quarter nobody has, and the person who knows it best is leaving in March. Those do not feel like engineering facts. They feel like circumstances. So they get left out of a technical question, by the only person who knows them.
✅ Say what cannot move
Three lines, and each one has a job:
Must run on: Airflow 2.7, existing cluster, no new infrastructure this year
Cannot introduce: streaming, new languages, anything needing a dedicated operator
Team knows: SQL and dbt well, Python moderately, Spark barely
Must run on is the boundary of the answer. Anything outside it is not a recommendation, it is a project proposal.
Cannot introduce is the line people skip, and it is the one doing the most work. "No new infrastructure this year" removes about half of all modern data advice in six words.
Team knows is the one that seniors include and everyone else forgets. A recommendation your team cannot operate is not a recommendation. If nobody has run Spark in production, an answer built on Spark tuning is a plan to find out what happens when they try.
Notice these are not preferences. "We prefer dbt" invites a discussion about whether you should. "Transformations must stay in dbt" does not. If it genuinely is a preference, say so, and expect it to be argued with.
⚖️ The objection worth taking seriously
The weak version is "sometimes I want the unconstrained ideal answer." Fine, and you should ask for that on purpose rather than getting it by accident.
The strong version is more troubling, and there is research behind it. If you state a constraint, it will be accepted, including when the constraint is the actual problem.
There is a measured behaviour here called framing sycophancy: models tend to accept a user's premises rather than challenge them. Researchers at Stanford describe this as defaulting to going along with the user's assumptions without enough scepticism, and a 2026 paper in Science found the broader pattern of excessive agreement present across eleven current models.
Applied here, that means if you write "must stay on Airflow," you will get the best available answer for staying on Airflow. You will not get told that late-arriving events are painful in your setup because of the batch orchestration, and that you are solving a problem you have chosen to have. The constraint you stated becomes the ceiling on what you can learn, and nothing signals that a ceiling was applied.
That is a real cost, and over a couple of years it means these tools become very good at making your existing decisions work and never at questioning them.
Two things follow. First, ask the unconstrained version as a separate question, deliberately, and read it as information rather than as a plan. "If you were starting fresh, how would you handle this" tells you what you are paying for your existing setup, which is worth knowing even when you cannot change it.
Second, invite the challenge explicitly. The same Stanford research found that small changes to how a prompt is phrased, including something as simple as adding "wait a minute," measurably improved models' willingness to push back on a premise without producing a lot of false alarms. So a line like "before answering, tell me whether any of these constraints is what is actually causing this problem" is worth adding to anything consequential. You will usually be told nothing is wrong. Occasionally you will not, and that is the one you needed.
🎯 The takeaway
Three lines, stated as boundaries and not descriptions:
1. Must run on. What the answer has to fit inside.
2. Cannot introduce. The line that removes most generic advice, and the one people leave out.
3. Team knows. A solution nobody can operate is not a solution.
Then one more line for anything important: ask whether one of those constraints is the cause of the problem. It will usually say no. That is not the point of asking.
Describing your stack tells the answer where you are. Only a constraint tells it where you have to stay.
