Be Careful of Your Dependencies

Starting from a situation I encountered, just when I am writing this blog:

My operating system -- Valecium is currently still in the alpha stage.

It does not provide system headers.

I need to use Linux headers to build the toolchain.

The Kernel Archives broke.

They return 404 for the exact version I need -- 6.12.7.

Leaving me staring at my screen, don't know what to do.

Luckily for me they had already discovered this issue, and I only had to wait 10 hours for this to be fixed.

But what about 10 hours was 10 days? Or the problem was never discovered
and fixed because the project is inactive? Of course, my project does
not have many dependencies, but there are many that do.

Problems Will Not Stop

We are humans, or AI (if AI is reading this), can never 100% guarantee
that everything we do is correct, that is common knowledge.

But the worst part is that machines follow a set of rules, defined by
us, so even if the machine executes the rules perfectly, the rules could
be wrong, this is where the undefined behaviors come it, and still
breaks the things.

So, if we cannot remove the problems from source, then we have to
counteract them, and in this blog, I organized a few rules to save your
project in case of your dependencies break.

Different Types of "dependency problems"

Not all breakage is the same, and it helps to know which kind you're
staring at before you pick a way to resolve that problem, and
systematically to reduce the cost when your dependencies breaks..

Version compatibility: The dependency itself did not die; it just
moved on without you realizing. If you are still using the old API and
they have already deprecated it for a few months. This is the slowest,
most survivable kind of problem -- it usually gives you warning, if
you're paying attention. This is especially a thing when you are
dealing with modern technology like AI (e.g. the /chat/completions API
is already replaced with /responses still allowed though).

A bug with a vulnerability: This one is scarier because it does not
announce the problem every time you are running it. That dependency
looks fine, compiles fine, runs fine -- until someone finds out it's
been leaking memory, or worse, that it was compromised on purpose. As
the famous XZ incident.

Support for that dependency going down: This is exactly what
happened to me, as the start of this blog. If someday your npm or
crate.io is down, and you just accidentally deleted your cache to test
something, you are done. Which basically what it means, is that in
order to make your program work you are relying on a specific level of
uncertainty that is always not zero, which also applies to the
previous type -- Bugs that cause vulnerabilities.

Hardware compatibility: A famous thing that I encounter is that when
you are compiling, you used -march=native or -O3, and just luckily you
are super rich to use a supercomputer with a Xeon processor and GCC
decided to go with AVX512. And another poor person on an i5 will just
get an illegal instruction error because they don't have AVX512.

Always Have a Backup

This is the cheapest and easiest way, and probably the one you should
already be doing either because you are told to by your boss or had
learnt a lesson from not doing so like what happened to me.

Having backups is common knowledge, and not just in software, in
basically anything you do in just normal life actions, with it being the
first line of defense.

In the context of writing software, "backup" isn't one thing. It's a
few different insurance policies stacked together:

The National Archives (United Kingdom), CC BY 3.0, via Wikimedia Commons

Keep a copy of your dependency's source. If the archive 404s,
the maintainer is gone, or your companies firewall decided to block
the source. You will not be stuck waiting on someone else's
infrastructure.
Be compatible with multiple dependencies that does the same
thing. If one breaks just switch to another one and you instantly
get the program back up and running, leaving you with lots of time
to fix the problem from source. For example, you can support both
Vulkan and OpenGL when making a game.
Support more than one version of a dependency. This one mostly
answers the version-compatibility problem -- if you're not
hard-locked to exactly one version, an upstream change doesn't
automatically become your emergency.

Of course, none of this stops the dependency from breaking. It just
makes sure you don't break with it.

Also haven't you realized, this simple thing of having backups is
already fixing all 4 of the problems that I said in the previous
section, with the only downside I can think of being a few more lines of
code, not even that much.

Do Not Depend, Copy

The next tier up is deciding you don't want to depend on anyone's
uptime at all, you statically link most of what you need, besides the
truly unavoidable stuff like LLVM. Another way to say it be
self-contained.

This is the strongest answer to the "support went down" problem
specifically. If the code is already sitting inside your binary, it
doesn't matter if npm, or the Kernel Archives, or any registry is down.
Or there are any not compatible versions, you don't care.

A project that I think did this well is Blender, every time you build
it, it will download the pre-compiled dependencies only for itself, and
creates a self-contained directly on install with all dependencies and
statically links most of it.

Although this method ensures full version compatibility and removes the
need for downloading, it would be worse for the other 2. If someone
found a bug in one of your statically linked dependencies, you don't
just swap the system shared library and you are done, you have to push
the update to everyone that is using this software, which is annoying.

Nuclear: Do Everything Yourself

If you really want the last word on every single failure mode:
version drift, vulnerabilities, hardware quirks, all of it. This is the
only option that actually gets you there, because you're no longer
depending on anyone else's judgment at all.

Ludovic Courtès, GFDL 1.3, via Wikimedia Commons

But you see, you cannot really not depend on anything, because there
is always something below your software, like the OS kernel, or the
hardware. Therefore, getting the correct amount that you do yourself is
a very tricky thing, and if you do not get that right, the same problems
will still emerge.

But "ultimate control" and "ultimate workload" are the same option
here. You now have to think of everything yourself, write far more code
than the feature actually needed, and since you're human, same as every
maintainer whose dependency just broke on you. You will write your own
share of bugs into it. You have not removed the risk. You've just
moved it from "someone else's mistake" to "your mistake," which
is a trade, not a victory.

Have Fun

All of this, the backups, the self-contained builds, the nuclear option.
Is really only for when the stakes are real. When a break actually costs
you something: users, uptime, a release date, your own mental sanity at
2 o'clock staring at a 404.

If it's just a project for fun, none of this is a rule you owe anyone.
Let it break. Fix it when you feel like it, or don't. As most of these
dependency problems will get fixed within a few hours/days to fix, and
it is never a problem to do a roll back if you use git.

The entire point of the four problems and three answers above was to
never always do the maximum. It was knowing what you're risking, so
you can choose how much you're willing to pay to avoid it. For a hobby
project, the most suitable answer is just go alone with it (the
dependencies), and wait patiently if it breaks.