I'm experimenting with an alternative implementation of portage in Rust.
It is quite different from pkgcraft even if both projects have similar targets.
Main differences:
- separation of concerns: portage-cli is based on a set of layered crates, that let people reuse specific parts of it. pkgcraft on the other hand is a bit monolithic right now.
- pure-rust: pkgcraft for correctness wraps bash, portage-repo leverages brush fixing issues as they are found.
- focus on different tooling: pkgcraft offers already extensive linting support via
pkgcruft, portage-cli offers an emerge workalike (em) with a pubgrub-based dependency resolver.
Some numbers
Some comparisons with the current implementations.
metadata regen
All 3 tools are able to do that and we have a script to do that in a simple way.
GENTOO_REPO=/var/db/repos/gentoo \
EM=target/release/em \
PK=../pkgcraft/target/release/pk \
INCLUDE_EGENCACHE=1 \
ITERATIONS=1 \
./benchmarks/scripts/compare-regen.sh 8 16 20 24
| tool | j | run | real | user | sys | files |
|---|---|---|---|---|---|---|
| em | 8 | 1 | 0m18.008s | 2m12.733s | 0m9.387s | 31880 |
| egencache | 8 | 1 | 5m24.107s | 0m0.003s | 0m0.004s | 31880 |
| pk | 8 | 1 | 0m48.226s | 5m27.982s | 1m8.484s | 31880 |
| em | 16 | 1 | 0m10.115s | 2m17.674s | 0m11.647s | 31880 |
| egencache | 16 | 1 | 4m36.731s | 0m0.000s | 0m0.007s | 31880 |
| pk | 16 | 1 | 0m28.172s | 6m0.818s | 1m22.995s | 31880 |
| em | 20 | 1 | 0m9.593s | 2m27.314s | 0m13.700s | 31880 |
| egencache | 20 | 1 | 4m35.749s | 0m0.000s | 0m0.006s | 31880 |
| pk | 20 | 1 | 0m23.707s | 5m55.650s | 1m33.241s | 31880 |
| em | 24 | 1 | 0m9.229s | 2m39.258s | 0m16.244s | 31880 |
| egencache | 24 | 1 | 4m56.389s | 0m0.007s | 0m0.000s | 31880 |
| pk | 24 | 1 | 0m23.367s | 5m54.775s | 1m44.188s | 31880 |
Here we see that the rust implementation are way faster and portage-repo parallel visit via jwalk has an edge.
dependency resolution
Only emerge and em have a resolver wired in and the search support.
RUNS=2 EM=target/release/em ./benchmarks/bench-em-vs-emerge.sh
| Benchmark | em (mean ± σ) | emerge (mean ± σ) | speedup (em vs emerge) |
|---|---|---|---|
| firefox -p | 0.917 s ± 0.082 s | 3.675 s ± 0.000 s | 4.01× |
| libreoffice -p | 0.918 s ± 0.026 s | 3.953 s ± 0.013 s | 4.31× |
| multi (5 pkgs) -p | 1.041 s ± 0.025 s | 4.724 s ± 0.196 s | 4.54× |
| gcc -s | 0.154 s ± 0.011 s | 5.175 s ± 0.006 s | 33.63× |
Since it is fun to watch, here an example of em dealing with some use changes compared with emerge.
The pubgrub solver wiring covers already plenty of the niceties of emerge, but I need to test it more.
Progress so far
To make updating simpler all the crates are now in the same portage-cli monorepo and most are already released, with few on hold waiting on having all the fixes I prepared for brush landing (help reviewing and polishing very welcome).
| Crate | Version | Status | crates.io |
|---|---|---|---|
| gentoo-interner | 0.3.1 | Published | crates.io/crates/gentoo-interner |
| gentoo-core | 0.5.1 | Published | crates.io/crates/gentoo-core |
| gentoo-stages | 0.5.1 | Published | crates.io/crates/gentoo-stages |
| portage-atom | 0.10.0 | Published | crates.io/crates/portage-atom |
| portage-metadata | 0.8.0 | Published | crates.io/crates/portage-metadata |
| portage-atom-pubgrub | 0.5.0 | Published | crates.io/crates/portage-atom-pubgrub |
| portage-atom-resolvo | 0.7.0 | Published | crates.io/crates/portage-atom-resolvo |
| portage-vdb | 0.1.0 | Pending brush release | crates.io/crates/portage-vdb |
| portage-repo | 0.1.0 | Pending brush release | N/A |
| portage-distfiles | 0.1.0 | Pending brush release | N/A |
| portage-cli | 0.1.0 | ~20% complete | N/A |
Right now it is possible to enjoy a quite faster dependency resolution, cache regeneration and search. There is some essential support for building and merging packages, but it is far from being battle tested.
In the coming weeks I'll try to fill more gaps and make it more functional and hopefully blog more often.
