lang-bootstrap/06/README.md

# `bootstrap-from-tcc`

## What

Bootstrap a modern toolchain for [ZilchOS Core](https://github.com/ZilchOS/core)
starting from a random trusted statically linked seed tcc
(+ trusted kernel on trusted hardware, of course).

~320 KB binary + gigs of sources = a modern Clang + musl toolchain,
usable for building much more serious stuff.

My goal is to beeline for bootstrapping Nix package manager,
then bootstrap a usable toolchain using Nix.
But even if you don't care about Nix,
this repo might be of some interest for minimal binary seed bootstrappers.

Separate packages aren't just dumped into `/`, they're properly managed,
each one residing in its own prefix under `/store`.

`x86_64`-only for now, possibly forever.

## Why

I wanted to build a minimal distro to understand NixOS better,
so I decided to have a decent trusted binary core bootstrap as well.

Could be of use for bootstrapping other distributions.

I'm aware of https://savannah.nongnu.org/projects/stage0 which does even better,
but I'm not as hardcore as them, so, let's start small.

## How

### In brief

Compiler chain so far (`recipes`):
input TinyCC -> stable TinyCC -> GNU GCC 4 -> GNU GCC 10 -> -> Clang

`recipes/1-stage1.c` is the most fun, since we don't have libc yet.

Then I build Nix and start the entire bootstrapping chain all over again,
but now using that Nix I've built (`using-nix`).

### Outlined bootstrap order

* stage 0: seeded binary `tcc`
* stage 1 (`recipes/1-stage1.c` / `using-nix/1-stage1.nix`, no libc):
  * `libtcc1`
  * `protomusl`
  * `tcc`
  * `libtcc1`
  * `protomusl`
  * `tcc` that is gonna be the final one
  * `libtcc1`
  * `protomusl`
  * `tcc` that we build just to prove the finality of the previous one
  * `protobusybox`
* stage 2 "compiler ascension" part (`recipes/2a*.sh` / `using-nix/2a*.nix`):
  * `gnumake`
  * `binutils`
  * `gnugcc4`
  * `musl`
  * `gnugcc4`
  * `gnugcc10`
  * `linux-headers`
  * `cmake`
  * `python`
  * `clang`
* stage 2 "build with the new compiler" part (`recipes/2b*.sh`):
  * `musl`
  * `clang`
  * `busybox`
  * `gnumake`
* stage 3 "dependencies of useful stuff" (`recipes/3a*.sh`): ???
  * `sqlite`
  * `boost`
  * `mbedtls`
  * `pkg-config`
  * `curl`
  * `editline`
  * `brotli`
  * `gnugperf`
  * `seccomp`
  * `libarchive`
  * `libsodium`
  * `lowdown`
* stage 3 "useful stuff" (`recipes/3b*.sh`): ???
  * `busybox-static`
  * `tinycc-static` (1st time only)
  * `nix`

Then repeat stage 1 and most of stage 2 all over again, but under Nix.
The final exports of this flake are musl, clang toolchain and a busybox
that ZilchOS Core later bootstraps from.

* stage 4 "rebootstrap with nix" (`recipes/4-rebootstrap-using-nix.sh`):
  build the toolchain again from scratch, but using nix (`using-nix/`)

* stage 5 "go beyond using nix" (`recipes/5-go-beyond-using-nix.sh`):
  build some stale version of [ZilchOS Core](https://github.com/ZilchOS/core)
  with the nix we've built, culminating with a bootable ZilchOS ISO.

### In more detail

given:

* **statically linked target tcc (`tcc-seed`, you have to provide it)**
* host `unshare` for `chroot`ing and isolation
* host `wget`, `tar`, `gzip`, `bzip2`, `sha256sum`, `tar`
  for optional convenient fetching of source files in stage 0
* host `sed` to preprocess the sources needed for stage 1, unfortunately
* a replacement for `musl/arch/x86_64/syscall_arch.h` that works with tcc
  (`syscall.h`)
* a bunch of sources to execute along the way

`download.sh" downloads a ton of sources, scraping hashes/URLs from recipes

`seed.sh` seeds:

* unpacks sources into the stage area
* FIXME: uses host `sed`/`rm` for preprocessing stage 1 source code,
  unfortunately
* copies `seed-tcc`, the only starting binary, into the stage area

At the end of it we obtain
a ton of sources and a single externally seeded `tcc` binary.

`recipes/1-stage1.c`, executed with `tcc -run`:

* compiles a `libtcc1.a` from `tinycc` sources
* compiles a protomusl `libc.a` and others from `musl` sources
* compiles the first `tcc` that comes from our sources
* recompiles `libtcc1.a` from `tinycc` sources
* recompiles protomusl `libc.a` and others from `musl` sources
* recompiles our `tcc` with our `tcc`
* recompiles `libtcc1.a` from `tinycc` sources just in case
* recompiles protomusl `libc.a` and others from `musl` sources just in case
* compiles and links standalone applets out of busybox, notably `ash`
* copies over protomusl include files
* recompiles `tcc` once again
  and verifies that we've previously reached a stability point

At the end of stage 1 we have, all linked statically:

* a `tcc` (+ `libtcc` + `libtcc1`) that recompiles to same binary
* a protomusl `libc.a` (+ crt stuff)
* select standalone busybox applets, most notably `ash`

`stage2.sh`, executed with protobusybox `ash`:

* Performs 'compiler ascension' from tcc to GNU GCC 4:
  * `gnumake`, intermediate, built without make
  * `gnumake`, statically linked
  * `binutils`, statically linked
  * `gnugcc4`, statically linked
  * `musl`, now a shared library as well
  * `gnugcc4` with C++ support and linking to a shared musl
  * `gnugcc10`
  * `linux-headers` (clang & cmake dependency)
  * `cmake` (clang dependency)
  * `python` (clang dependency, presumably)
  * `clang`, intermediate, 2-stage

* Recompile the world with clang, free of GNU runtime libs:
  * `musl`, final
  * `clang`, final
  * `busybox`, final
  * `gnumake`, final

* Build a bunch of Nix dependencies
* Build Nix

* Start over, build toolchain, build ZilchOS Core

### Building options

There are three major ways to build it.

If you have Nix and want to skip the first half that bootstraps Nix,
you can just `nix build`, but what's the fun in taking shortcuts =).
You'll need `experimental-features = nix-command flakes ca-derivations`.

If you want to do a full bootstrap with all imaginable speedups enabled,
try something to the tune of
`make all-pkgs all-tests verify-all-pkgs-checksums -j2 NPROC=$(nproc) USE_CCACHE=1 USE_NIX_CACHE=1`.
Dependencies: host GNU Make, host zstd, basic host stuff like sed and bash,
a target TinyCC you supply or Nix to build you one.
This is the recommended way, especially shining when you
iteratively debug reproducibility-unrelated build problems.
Consider mounting `tmp/build` as tmpfs with 8G size.

Finally, the least-dependency way is `NPROC=$(nproc) ./build.sh`.
This one doesn't even need GNU Make or zstd,
but there are zero intermediate checkpoints, you always start all over.
Very impractical, this is for increased portability only.

### Reproducibility

Reproducibility is deeply cared about,
but it's a constant struggle and one cannot foresee everything.

Hashes are checked for intermediate steps for both `make` and `nix` builds.
`raw` builds only verify the resulting ZilchOS Core ISO built during stage5.
I try to build on different machines and note down the results in `git notes`.
Commits require a specific (but adjustable) amount of successful
`make`, `raw` and `nix` before getting into the main branch.
Refer to `make all-with-make`, `make all-raw` and `make all-with-nix`
to see what exactly is being tested.

For hard mode, you can try `USE_DISORDERFS`.

For ZilchOS/core, see `.maint/hashes` and `.maint/tools/hashes`.