Project homepage Mailing List  Warmcat.com  API Docs  Github Mirror 
    npro  
 Modern all-safe Rust Network Protocol library supporting h1, h2, h3, ws, wt sans-IO and with socket IO + tls
git clone https://npro.rs/repo/npro
 
root / crates / npro-test / h1 / requests / uri-query-first.http
Author[]Andy Green <andy@warmcat.com> 2026-10-03 12:04 UTC
Committer[]Andy Green <andy@warmcat.com> 2026-10-05 06:32 UTC
Tree44e8e3120ec0c064e3d3616ab8f53e8e8ef9c56a   Raw Patch
 
meta: say what a machine needs, and check for it before using it
meta: say what a machine needs, and check for it before using it

Setting up the first sai builders turned up the same thing four times: a
missing tool surfaced as a cargo error naming the symptom, not the cause.
- "no such command: fmt" came from a minimal rustup profile.
- "`resolver` setting `3` is not valid" came from Ubuntu's 1.75 cargo,
  found because rustup was not installed for the sai user.
- "toolchain '1.85' is not installed".
- "no such command: hack".

scripts/require.sh holds small checks, sourced by both scripts:
- a rustc at least the workspace's rust-version;
- a cargo subcommand;
- a rustup toolchain or target;
- a program, or any command that succeeds.

Each check records what is missing, with the command that installs it.
require_done then lists everything missing at once and exits.

ci.sh checks everything it uses before its first gate.  Each sai.sh
profile checks exactly what it uses: test needs only rustc; features
needs clippy and cargo-hack; nostd needs its three targets; miri needs
nightly with miri; oracle needs git, cmake, make, cc and zlib's headers.
The rustc check sai.sh had becomes require_rust.

docs/toolchain.md is new.  It says what a developer machine needs for
ci.sh:
- the pieces in C terms (toolchain: a compiler release; component: an
  optional tool; target: a cross sysroot);
- why each is needed;
- installing with rustup, or with a distro's rustup package rather than a
  script from the network;
- checking it, where it lives, and keeping it current;
- the errors above, with their fixes.

docs/sai.md's builder section is rewritten around what sai adds:
- install as the user jobs run as;
- with sai-virt, install into the base image while no overlay is in use,
  then let fresh overlays be made;
- a table of what each builder needs;
- `cargo miri setup` once in the base image, so jobs do not rebuild
  Miri's std;
- how to check a builder, and the network jobs use;
- a table of setup failures seen in job logs;
- why the riscv64 board stays on Ubuntu 24.04.

The README links toolchain.md.

Tested here:
- ci.sh and sai.sh test, features and nostd pass with everything
  installed;
- with an empty CARGO_HOME, features reports cargo hack missing;
- a missing toolchain, target and header are listed together;
- `cargo +nightly miri setup --target …` works as the docs give it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019kg5Eemy68ZaqDBcUJQG6J
diff --git a/README.md b/README.md index 4841152..ce9ea39 100644 --- a/README.md +++ b/README.md @@ -47,6 +47,9 @@ sansIO half is the specification, and the port is held to it by: - Big-endian + 32-bit via Miri - Every feature combination tested +[docs/toolchain.md](docs/toolchain.md) explains how to set a machine up to run +the tests. + ## Licence MIT diff --git a/docs/sai.md b/docs/sai.md index d3b3631..0838e08 100644 --- a/docs/sai.md +++ b/docs/sai.md @@ -31,84 +31,116 @@ How the C build's dimensions map here: - **Machines sai has no builder for**, big-endian and 32-bit, are interpreted by Miri rather than run on hardware. -## Installing Rust on a builder +## Setting up a builder -Install as the user the sai builder runs jobs as. sai sets `HOME` to the -builder's home for each job, and `scripts/sai.sh` puts `$HOME/.cargo/bin` -first on `PATH`, so rustup's usual per-user install is found without -changing the builder daemon's environment. +[toolchain.md](toolchain.md) explains the pieces: rustup, toolchains, +components, targets, and cargo subcommands. This section says which ones +each builder needs, and the things that are particular to sai. -### Every unix builder with rustup +### Install as the user sai runs jobs as -These are the Linux builders (x86_64, aarch64 and riscv64) and the macOS -ones. rustup provides host toolchains for all of them. A C linker must -be present: it is already, as these builders build C lws. On macOS that -means the Xcode command line tools. +sai runs each job as the builder's user (`sai`), with `HOME` set to that +user's home, and `scripts/sai.sh` puts `$HOME/.cargo/bin` first on `PATH`. +So rustup must be installed **for that user**. An install for your own +login, or for root, is invisible to the jobs. ```sh -curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile default +sudo -iu sai # a login shell as sai, with its HOME ``` -That gives stable with rustfmt and clippy, which is all the `test` -profile needs. Updating later is `rustup update`. +On a builder whose jobs run in **sai-virt overlays** of a base VM image, +install into the base image, by booting the base itself, while no job is +using an overlay of it. Anything installed inside a job's overlay goes +away with the overlay. Shut the base down cleanly afterwards, and let +sai-virt make fresh overlays: an overlay made from the old base must not +be used with the new one. -A distro's packaged cargo is usually too old: Ubuntu noble's is 1.75, and -npro needs the workspace's `rust-version`, 1.85, the first to know -edition 2024. `scripts/sai.sh` checks `rustc` before anything else, and -refuses an older one with the install command, rather than leaving cargo -to fail parsing the manifest. +### What each builder needs -### The fedora44 x86_64 builder, which runs everything else +| builder | configurations | needs | +|---|---|---| +| x86_64 and aarch64 Linux, riscv64, both macOS | `test` | rustup with stable (`--profile default`) | +| fedora44 x86_64 | `gate`, `features`, `nostd`, `miri-*`, `c-oracle` | everything below | +| freebsd/aarch64 | `test-freebsd` | `pkg install rust`, 1.85 or later; rustup has no FreeBSD aarch64 host | +| w11/x86_64 | `test-windows` | `rustup-init.exe`, as the account the sai service runs as, with its `%USERPROFILE%\.cargo\bin` on the service's `PATH` | -On top of the above: +**Every builder with rustup**, as the `sai` user: ```sh -rustup toolchain install 1.85 --profile minimal # the MSRV build -rustup toolchain install nightly --profile minimal --component miri,rust-src -rustup target add thumbv6m-none-eabi thumbv7em-none-eabihf riscv32imc-unknown-none-elf -cargo install --locked cargo-deny cargo-audit cargo-hack -dnf install cmake gcc zlib-devel git # the C oracle build +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile default ``` -The first Miri run of each target builds that target's standard library -for Miri, and takes a few minutes. Later runs reuse it. +Or install the distro's `rustup` package, then run `rustup default stable` +as `sai`; [toolchain.md](toolchain.md) covers both routes. Do not rely on +distro `cargo` / `rustc` packages: Ubuntu 24.04's 1.75 is too old to read +the workspace. On macOS, the Xcode command line tools provide the linker. -Network access during jobs: -- `cargo audit` fetches the RustSec advisory database from github.com on - each run; -- the oracle clones and fetches libwebsockets from libwebsockets.org, - keeping its checkout in `$HOME/lws-oracle` (or `$LWS_ORACLE`). +**The fedora44 builder**, on top of that, as `sai`: -Nothing else fetches. The workspace has no dependencies, and every build -is `--locked`. +```sh +rustup toolchain install 1.85 --profile minimal # gate: the MSRV build +rustup toolchain install nightly --profile minimal --component miri,rust-src # miri-* +rustup target add thumbv6m-none-eabi thumbv7em-none-eabihf riscv32imc-unknown-none-elf # nostd, gate +cargo install --locked cargo-hack cargo-deny cargo-audit # features, gate +cargo +nightly miri setup --target s390x-unknown-linux-gnu # build Miri's std once +cargo +nightly miri setup --target i686-unknown-linux-gnu +``` -### freebsd/aarch64 +and, as root, for the C build `c-oracle` does: -rustup has no host toolchain for aarch64 FreeBSD, so use the packaged one, -which must be at least the workspace's `rust-version` (1.85): +```sh +dnf install git cmake make gcc zlib-devel +``` + +Miri builds a standard library for each target it interprets. +`miri setup` does it once ahead of time; otherwise every job on a fresh +overlay spends several minutes on it. + +### Checking a builder + +As `sai`: ```sh -pkg install rust +rustc --version # 1.85 or later +rustup toolchain list # fedora44: stable, 1.85, nightly +rustup target list --installed # fedora44: the three no_std targets +cargo hack --version; cargo deny --version; cargo audit --version # fedora44 ``` -It installs under `/usr/local/bin`, which the builder's `PATH` already has. +Each profile in `scripts/sai.sh` also checks for exactly what it uses +before running. A job on a builder that lacks something fails at once, +listing what is missing and the command for each, rather than partway +through with a cargo error. -### w11/x86_64 (MSVC) +### Network access during jobs -Run `rustup-init.exe` from <https://rustup.rs> with the default host, -`x86_64-pc-windows-msvc`; the builder already has the Visual Studio build -tools it links with. Do it as the account the sai builder service runs -as, and make sure that account's `%USERPROFILE%\.cargo\bin` is on the -service's `PATH`, since the Windows step runs `cargo` directly under -`cmd`. +- `cargo audit` (in `gate`) fetches the RustSec advisory database from + github.com. +- `c-oracle` clones and fetches libwebsockets from libwebsockets.org, + keeping its checkout in `$HOME/lws-oracle` (or `$LWS_ORACLE`). -### Later +Nothing else fetches. The workspace has no dependencies, and every build +is `--locked`. -- **esp32**: building for the ESP32's xtensa cores needs Espressif's Rust - fork, installed with `espup`. That waits until there is something to - run on the device. -- **fuzzing**: cargo-fuzz needs nightly and clang. It joins the fuzz - builder once the h1 parser has fuzz targets. +### When a job fails on setup + +| what the job log shows | why | fix | +|---|---|---| +| `rustc 1.75 is older than npro's rust-version 1.85`, or cargo's `` `resolver` setting `3` is not valid `` | the job found the distro's cargo: rustup is not installed for `sai` | install rustup as `sai`, as above | +| `no such command: hack` (or `deny`, `audit`) | the cargo subcommand is not installed for `sai` | `cargo install --locked cargo-hack` (etc.) as `sai` | +| `toolchain '1.85-…' is not installed` | the MSRV toolchain is missing | `rustup toolchain install 1.85 --profile minimal` as `sai` | +| something installed earlier is missing again | it was installed inside a sai-virt overlay, not the base image | install it into the base image | +| Miri spends minutes "preparing a sysroot" every job | Miri's std is rebuilt in each fresh overlay | `cargo +nightly miri setup --target …` in the base image | + +### Notes on particular builders + +- **ubuntu-noble/riscv64** stays on Ubuntu 24.04. From 25.10 on, Ubuntu + supports only RISC-V hardware meeting a newer profile, which this + board's silicon does not. rustup supports it as it is. +- **esp32**, later: xtensa needs Espressif's Rust fork, installed with + `espup`, once there is something to run on the device. +- **fuzzing**, later: cargo-fuzz needs nightly and clang, and joins the + fuzz builder with the h1 parser's fuzz targets. ## A possible sai change diff --git a/docs/toolchain.md b/docs/toolchain.md new file mode 100644 index 0000000..3ee167c --- /dev/null +++ b/docs/toolchain.md @@ -0,0 +1,126 @@ +# Setting up a machine to build npro + +This is what a developer machine needs to run `scripts/ci.sh`, the gates +every commit must pass. A sai builder needs the same pieces, depending on +which configurations it runs: see [sai.md](sai.md). + +## The pieces, in C terms + +Rust's tools are managed by **rustup**, which installs and switches +between compiler versions per user, under your home directory. It is +the nearest thing to having several gcc versions and cross sysroots side +by side, with one tool to fetch and select them. + +| rustup word | C analogue | example | +|---|---|---| +| toolchain | a compiler release: rustc, cargo, the standard library | `stable`, `1.85`, `nightly` | +| component | an optional tool shipped with a toolchain | `rustfmt` (formatter), `clippy` (lint), `miri` (interpreter) | +| target | a prebuilt standard library for a cross target, like a sysroot | `thumbv7em-none-eabihf` (cortex-m4f, no OS) | + +cargo is the build tool. `cargo <name>` runs a subcommand, and extra ones +are programs called `cargo-<name>` that `cargo install` builds from +source into `~/.cargo/bin`. + +## What `scripts/ci.sh` needs, and why + +| piece | used by | why | +|---|---|---| +| a current stable toolchain, at least the workspace's `rust-version` (1.85) | everything | the workspace is edition 2024, which older cargo cannot even parse | +| `rustfmt`, `clippy` components | `== fmt`, `== clippy` | formatting and the lint rules AGENTS.md requires | +| the `1.85` toolchain | `== msrv` | proves the code still builds with the oldest release npro supports | +| the `thumbv7em-none-eabihf` target | `== no_std` | proves the sans-IO crates build with no operating system | +| `cargo-deny` | `== deny` | checks licences, duplicate versions and sources of dependencies | +| `cargo-audit` | `== audit` | checks dependencies against the RustSec advisory database | + +`ci.sh` checks for all of these before it starts. If any are missing, it +lists them with the command that installs each. + +## Installing + +### 1. rustup and the stable toolchain + +On Linux and macOS: + +```sh +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile default +. ~/.cargo/env # or open a new shell; it adds ~/.cargo/bin to PATH +``` + +`--profile default` includes `rustfmt` and `clippy`. The `minimal` +profile leaves them out, and `cargo fmt` then fails with "no such +command". + +If you would rather not run a script from the network, most distros +package rustup itself. On Fedora: `dnf install rustup`, then +`rustup-init`. On Debian 13 and Ubuntu 24.04: `apt install rustup`, then +`rustup default stable`. Either way, rustup then fetches toolchains from +rust-lang.org over https, and checks them against the release manifest's +checksums. + +Do not use the distro's `cargo` and `rustc` packages. They are usually +older than npro's `rust-version`: Ubuntu 24.04 ships 1.75, which cannot +read the workspace. + +### 2. The rest of what ci.sh needs + +```sh +rustup toolchain install 1.85 --profile minimal +rustup target add thumbv7em-none-eabihf +cargo install --locked cargo-deny cargo-audit +``` + +`cargo install` compiles each tool from source, which takes a few +minutes. + +### 3. Check it + +```sh +rustc --version # 1.85 or later +cargo fmt --version; cargo clippy --version +cargo deny --version; cargo audit --version +rustup toolchain list # stable and 1.85 +rustup target list --installed # includes thumbv7em-none-eabihf +scripts/ci.sh +``` + +## Where it all lives + +- `~/.rustup`: toolchains, components and targets. +- `~/.cargo/bin`: the `cargo`, `rustc` and `rustup` front ends, and + everything `cargo install` builds. +- `~/.cargo/registry`: downloaded crate sources. npro has no dependencies + yet, so this stays small. + +The `cargo` in `~/.cargo/bin` picks the toolchain per command: +- `cargo +1.85 check` uses 1.85; +- `cargo +nightly miri test` uses nightly; +- a plain `cargo` uses the default, stable. + +## Keeping it current + +```sh +rustup update # stable, nightly, and their components +cargo install --locked cargo-deny cargo-audit # rebuilds a tool if a newer release exists +``` + +The `1.85` toolchain never changes, which is the point of it. + +## When something fails + +| what you see | why | fix | +|---|---|---| +| `error: no such command: 'fmt'` (or `clippy`) | the component is missing: a `minimal` rustup profile, or a distro cargo | `rustup component add rustfmt clippy` | +| `` `resolver` setting `3` is not valid `` or "failed to parse manifest" | cargo older than 1.85, usually a distro package found first on `PATH` | install rustup as above, and remove the distro `cargo` / `rustc` | +| `toolchain '1.85-…' is not installed` | the MSRV toolchain is missing | `rustup toolchain install 1.85 --profile minimal` | +| `can't find crate for 'core'` with a `--target` | that target's library is not installed | `rustup target add <target>` | +| `error: no such command: 'deny'` (or `audit`, `hack`) | that cargo subcommand is not installed | `cargo install --locked cargo-<name>` | + +## Build output + +cargo builds into `./target` unless `CARGO_TARGET_DIR` says otherwise. +To keep two people's builds apart in one tree, as AGENTS.md asks for C's +`./build`, give each their own: + +```sh +CARGO_TARGET_DIR=$PWD/target-andy scripts/ci.sh +``` diff --git a/scripts/ci.sh b/scripts/ci.sh index dc27d3a..a066593 100755 --- a/scripts/ci.sh +++ b/scripts/ci.sh @@ -5,11 +5,9 @@ # # scripts/ci.sh # -# Set CARGO_TARGET_DIR to keep build output out of the tree. The MSRV build -# needs the toolchain named in Cargo.toml's rust-version (rustup toolchain -# install 1.85 --profile minimal); cargo-deny and cargo-audit must be -# installed, and the no_std check needs the thumbv7em-none-eabihf target -# (rustup target add thumbv7em-none-eabihf). +# Set CARGO_TARGET_DIR to keep build output out of the tree. It checks +# first for everything it uses, and lists what is missing with the commands +# that install it; docs/toolchain.md explains each piece. set -eu @@ -17,6 +15,16 @@ cd "$(dirname "$0")/.." msrv=$(sed -n 's/^rust-version *= *"\(.*\)"/\1/p' Cargo.toml) +. scripts/require.sh +require_rust +require_cargo fmt "rustup component add rustfmt" +require_cargo clippy "rustup component add clippy" +require_toolchain "$msrv" "rustup toolchain install $msrv --profile minimal" +require_target thumbv7em-none-eabihf +require_cargo deny "cargo install --locked cargo-deny" +require_cargo audit "cargo install --locked cargo-audit" +require_done + echo "== fmt" cargo fmt --all --check diff --git a/scripts/require.sh b/scripts/require.sh new file mode 100644 index 0000000..afa9ca6 --- /dev/null +++ b/scripts/require.sh @@ -0,0 +1,73 @@ +# Checks that the tools a script is about to use are installed, so it fails +# at the start with what to install, not halfway with a cargo error. +# Sourced by scripts/ci.sh and scripts/sai.sh; docs/toolchain.md and +# docs/sai.md say what each tool is for. +# +# Each require_* call notes what is missing; require_done then lists all of +# it with the command that installs each, and exits 1 if anything was. + +_missing="" + +_miss() { + _missing="$_missing + $1 + $2" +} + +# rustc at least the workspace's rust-version. A distro's packaged rust is +# the usual cause of an older one: cargo itself only says it cannot parse +# the manifest. +require_rust() { + _msrv=$(sed -n 's/^rust-version *= *"\(.*\)"/\1/p' Cargo.toml) + _have=$(rustc --version 2>/dev/null | + sed -n 's/^rustc \([0-9]*\.[0-9]*\).*/\1/p') + if [ -z "$_have" ]; then + _miss "rustc (none found; npro needs $_msrv or later)" \ + "curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile default" + return + fi + if [ "$(printf '%s\n%s\n' "$_msrv" "$_have" | + sort -t. -k1,1n -k2,2n | head -n1)" != "$_msrv" ]; then + _miss "rustc $_have is older than npro's rust-version $_msrv" \ + "install rustup and use its toolchain, not the distro's: see docs/toolchain.md" + fi +} + +# a cargo subcommand: fmt, clippy, hack, deny, audit... +require_cargo() { + cargo "$1" --version >/dev/null 2>&1 || _miss "cargo $1" "$2" +} + +# a rustup toolchain, eg the MSRV or nightly +require_toolchain() { + rustup run "$1" rustc --version >/dev/null 2>&1 || _miss "toolchain $1" "$2" +} + +# a rustup target installed for a toolchain (default: the active one) +require_target() { + _t=$(rustup target list --installed ${2:+--toolchain "$2"} 2>/dev/null) + printf '%s\n' "$_t" | grep -qx "$1" || + _miss "target $1${2:+ for $2}" \ + "rustup target add $1${2:+ --toolchain $2}" +} + +# anything else: a description, how to install it, and a command that +# succeeds when it is there +require_run() { + _what="$1" + _how="$2" + shift 2 + "$@" >/dev/null 2>&1 || _miss "$_what" "$_how" +} + +# a program on PATH +require_cmd() { + command -v "$1" >/dev/null 2>&1 || _miss "$1" "$2" +} + +require_done() { + [ -z "$_missing" ] && return 0 + echo "Missing on this machine:$_missing" >&2 + echo "See docs/toolchain.md (and docs/sai.md for a sai builder)." >&2 + exit 1 +} diff --git a/scripts/sai.sh b/scripts/sai.sh index 0c805f7..635de7c 100755 --- a/scripts/sai.sh +++ b/scripts/sai.sh @@ -14,7 +14,9 @@ # oracle C lws main-dev's transcripts, recorded afresh, compared # with the copies in crates/npro-test/transcripts # -# docs/sai.md says what each builder needs installed. +# Each profile first checks for the tools it uses and lists any that are +# missing with their install commands; docs/sai.md says what each builder +# needs. set -eu @@ -31,38 +33,39 @@ jobs="${SAI_PARALLEL:-4}" nostd_crates="npro-core" nostd_targets="thumbv6m-none-eabi thumbv7em-none-eabihf riscv32imc-unknown-none-elf" -# Refuse a toolchain older than the workspace's rust-version up front: cargo -# itself only says it cannot parse the manifest. A distro's packaged cargo -# is the usual cause, when rustup is not installed for the builder's user. -msrv=$(sed -n 's/^rust-version *= *"\(.*\)"/\1/p' Cargo.toml) -have=$(rustc --version 2>/dev/null | sed -n 's/^rustc \([0-9]*\.[0-9]*\).*/\1/p') -if [ -z "$have" ] || - [ "$(printf '%s\n%s\n' "$msrv" "$have" | sort -t. -k1,1n -k2,2n | head -n1)" != "$msrv" ]; then - echo "rustc ${have:-not found} on this builder; npro needs $msrv or later." >&2 - echo "Install rustup for the user sai runs jobs as, see docs/sai.md:" >&2 - echo " curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile default" >&2 - exit 1 -fi +. scripts/require.sh profile="${1:-}" [ $# -gt 0 ] && shift case "$profile" in gate) + # ci.sh checks its own tools exec scripts/ci.sh ;; test) + require_rust + require_done cargo --version exec cargo test --workspace --all-features --locked -j "$jobs" ;; features) + require_rust + require_cargo clippy "rustup component add clippy" + require_cargo hack "cargo install --locked cargo-hack" + require_done exec cargo hack clippy --workspace --feature-powerset --all-targets \ --locked -j "$jobs" -- -D warnings ;; nostd) + require_rust + for t in $nostd_targets; do + require_target "$t" + done + require_done for t in $nostd_targets; do for c in $nostd_crates; do echo "== $c for $t" @@ -74,6 +77,13 @@ nostd) miri) target="${1:?miri needs a target, eg s390x-unknown-linux-gnu}" + require_rust + require_toolchain nightly \ + "rustup toolchain install nightly --profile minimal --component miri,rust-src" + require_run "miri for nightly" \ + "rustup component add miri rust-src --toolchain nightly" \ + cargo +nightly miri --version + require_done # npro-test reads its transcripts from disk, which Miri's isolation # would refuse MIRIFLAGS="${MIRIFLAGS:-} -Zmiri-disable-isolation" \ @@ -86,6 +96,13 @@ oracle) # every transcript needs: fault injection for the seeded random, # extensions and zlib for the permessage-deflate ones c="${LWS_ORACLE:-$HOME/lws-oracle}" + require_cmd git "dnf install git" + require_cmd cmake "dnf install cmake" + require_cmd make "dnf install make" + require_cmd cc "dnf install gcc" + require_run "zlib headers" "dnf install zlib-devel" \ + sh -c 'echo "#include <zlib.h>" | cc -E - >/dev/null' + require_done if [ ! -d "$c/.git" ]; then git clone --depth 50 -b main-dev \ https://libwebsockets.org/repo/libwebsockets "$c"
Page fetched 0s ago, creation time: 4ms (vhost etag hits: 0%, cache hits: 0%)