Project homepage Mailing List  Warmcat.com  API Docs  Github Mirror 
    npro  
 Modern all-safe Rust Network Protocol library supporting h1, h2, h3, ws, wt sans-IO and with socket IO + tls
git clone https://npro.rs/repo/npro
 
root / fuzz / seeds / h1-request / version-2.http
Author[]Andy Green <andy@warmcat.com> 2026-10-03 09:24 UTC
Committer[]Andy Green <andy@warmcat.com> 2026-10-05 06:32 UTC
Tree4b76663b836d84e854997edbe580fd6b0c81f075   Raw Patch
 
meta: sai builds, on the C library's builders
meta: sai builds, on the C library's builders

.sai.json uses the C tree's platform names, so a builder that serves lws
serves npro once it has a Rust toolchain.  sai's step template only
substitutes ${prep}, ${cmake} and ${cpack}.  So every unix platform runs
`scripts/sai.sh ${cmake}`, and each configuration's "cmake" names a
profile there:
- test: the tests natively on every default platform: x86_64 and
  aarch64 Linux, riscv64, and macOS on intel and apple silicon.  Also on
  freebsd/aarch64 (packaged rust) and w11 (MSVC, one cargo step under
  cmd).
- gate: scripts/ci.sh, every gate a commit must pass.
- features: cargo hack's clippy over every combination of every crate's
  features.  That is the cargo form of C's cmake option dimensions, since
  code behind a feature is only compiled when it is on.
- nostd: the sans-IO crates built for cortex-m0 (no atomics), cortex-m4f
  and riscv32imc.
- miri <target>: the tests interpreted for s390x and i686, a big-endian
  and a 32-bit machine sai has no builder for.
- oracle: C lws main-dev built afresh, its api-test-sansio transcripts
  recorded, and compared with crates/npro-test/transcripts.  It fails
  when C's behaviour has moved and npro has not been synced, saying how
  to sync.

sai.sh puts $HOME/.cargo/bin on PATH, so rustup's per-user install under
the builder's home is found without changing the daemon's environment.

docs/sai.md is the setup for each builder: rustup everywhere it has a
host toolchain; the MSRV, nightly with Miri, the no_std targets,
cargo-deny / cargo-audit / cargo-hack and the C build tools on the one
that runs everything but the tests; pkg on FreeBSD; rustup-init on
Windows.  The README says what sai runs.

Every profile was run here except Windows and FreeBSD:
- gate, test, features and nostd pass;
- Miri passes for both targets in about two minutes each;
- oracle, run first against C 1ba2879, failed as it should, naming the
  two new transcripts; after the sync commit it passes.

The .sai.json was checked as JSON with its # comments removed, and its
platform names against the C tree's.  It has not been through sai's own
parser.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019kg5Eemy68ZaqDBcUJQG6J
diff --git a/.sai.json b/.sai.json new file mode 100644 index 0000000..f57071a --- /dev/null +++ b/.sai.json @@ -0,0 +1,116 @@ +{ + "schema": "sai-1", + + # + # The platform names are the C tree's, so a builder serves both. Each + # unix platform runs scripts/sai.sh with the configuration's "cmake", + # which names a profile there: sai's step template only substitutes + # ${prep}, ${cmake} and ${cpack}, so "cmake" carries it. What each + # builder needs installed is in docs/sai.md. + # + # The toolchain part of the names (gcc, llvm, msvc) is the C build's; + # rustc builds npro everywhere, and uses that platform's linker. + # + + "platforms": { + "linux-fedora44/x86_64-amd/gcc": { + "build": [ "scripts/sai.sh ${cmake}" ] + }, + "rocky9/x86_64-amd/gcc": { + "build": [ "scripts/sai.sh ${cmake}" ] + }, + "rocky9/aarch64-a72a55-rk3588/gcc": { + "build": [ "scripts/sai.sh ${cmake}" ] + }, + "linux-ubuntu-2404/aarch64-a72-bcm2711-rpi4/gcc": { + "build": [ "scripts/sai.sh ${cmake}" ] + }, + "ubuntu-noble/riscv64/gcc": { + "build": [ "scripts/sai.sh ${cmake}" ] + }, + "netbsd-OSX-bigsur/x86_64-intel-i3/llvm": { + "build": [ "scripts/sai.sh ${cmake}" ] + }, + "netbsd-OSX-tahoe/aarch64-apple-m1/llvm": { + "build": [ "scripts/sai.sh ${cmake}" ] + }, + "freebsd/aarch64/llvm": { + "default": false, + "build": [ "scripts/sai.sh ${cmake}" ] + }, + "w11/x86_64-amd/msvc": { + "default": false, + "build": [ "cargo test --workspace --all-features --locked -j %SAI_PARALLEL%" ] + } + }, + + "configurations": { + + # the tests, natively, on every default platform: x86_64 and + # aarch64 linux, riscv64, and macOS on intel and apple silicon + + "test": { + "cmake": "test" + }, + + # platforms whose rust comes from their own packages, or that + # run a fixed command: tests only + + "test-freebsd": { + "cmake": "test", + "platforms": "none, freebsd/aarch64/llvm" + }, + "test-windows": { + "cmake": "", + "platforms": "none, w11/x86_64-amd/msvc" + }, + + # every gate a commit must pass, as scripts/ci.sh runs them: fmt, + # clippy, tests, docs, the MSRV build, the no_std build, cargo deny + # and cargo audit + + "gate": { + "cmake": "gate", + "platforms": "none, linux-fedora44/x86_64-amd/gcc" + }, + + # clippy over every combination of every crate's features: the + # rust form of C's build option dimensions, since code behind a + # feature is only compiled when it is on + + "features": { + "cmake": "features", + "platforms": "none, linux-fedora44/x86_64-amd/gcc" + }, + + # the sans-IO crates for targets with no std: cortex-m0 (no + # atomics), cortex-m4f, and a 32-bit risc-v microcontroller + + "nostd": { + "cmake": "nostd", + "platforms": "none, linux-fedora44/x86_64-amd/gcc" + }, + + # the tests under Miri interpreting a machine sai has no builder + # for: big-endian 64-bit, and 32-bit little-endian, where usize and + # byte order differ from every builder above + + "miri-big-endian": { + "cmake": "miri s390x-unknown-linux-gnu", + "platforms": "none, linux-fedora44/x86_64-amd/gcc" + }, + "miri-32bit": { + "cmake": "miri i686-unknown-linux-gnu", + "platforms": "none, linux-fedora44/x86_64-amd/gcc" + }, + + # C lws main-dev's api-test-sansio transcripts, recorded afresh and + # compared with npro's copies: fails when C's behaviour moved and + # npro has not been synced with it + + "c-oracle": { + "cmake": "oracle", + "platforms": "none, linux-fedora44/x86_64-amd/gcc" + } + } +} diff --git a/README.md b/README.md index 628e315..4841152 100644 --- a/README.md +++ b/README.md @@ -43,6 +43,9 @@ sansIO half is the specification, and the port is held to it by: - the dependency tree stays close to empty (cargo deny + audit) - maximally linted via clippy to enforce code quality - fuzzing in CI on pushes and when CI idle +- CI runs tests natively on Linux, macOS, risc-v, aarach64, Windows +- Big-endian + 32-bit via Miri +- Every feature combination tested ## Licence diff --git a/docs/sai.md b/docs/sai.md new file mode 100644 index 0000000..80f4923 --- /dev/null +++ b/docs/sai.md @@ -0,0 +1,111 @@ +# npro on sai + +`.sai.json` uses the C tree's platform names, so a builder that already +serves libwebsockets serves npro too once it has a Rust toolchain. This is +what each one needs, and what each configuration checks. + +## How the config is shaped + +sai's step template substitutes three variables, `${prep}`, `${cmake}` and +`${cpack}`, so the name `cmake` is sai's and not a build system's. Every +unix platform has one step, `scripts/sai.sh ${cmake}`. Each configuration's +`cmake` names a profile in that script: + +| configuration | profile | platforms | what it shows | +|---|---|---|---| +| `test` | `test` | every default platform: x86_64 and aarch64 Linux, riscv64, macOS on intel and apple silicon | the tests pass natively there | +| `test-freebsd` | `test` | freebsd/aarch64 | the same, with FreeBSD's packaged Rust | +| `test-windows` | (its own step) | w11/x86_64 | the same, under MSVC | +| `gate` | `gate` | fedora44 x86_64 | everything `scripts/ci.sh` checks: fmt, clippy, tests, docs, the MSRV build, the no_std build, cargo deny, cargo audit | +| `features` | `features` | fedora44 x86_64 | clippy over every combination of every crate's features | +| `nostd` | `nostd` | fedora44 x86_64 | the sans-IO crates build for cortex-m0 (no atomics), cortex-m4f and riscv32imc | +| `miri-big-endian` | `miri s390x-unknown-linux-gnu` | fedora44 x86_64 | the tests pass on a big-endian machine, interpreted by Miri | +| `miri-32bit` | `miri i686-unknown-linux-gnu` | fedora44 x86_64 | the tests pass where `usize` is 32 bits | +| `c-oracle` | `oracle` | fedora44 x86_64 | C lws main-dev, built afresh, records the same transcripts npro holds | + +How the C build's dimensions map here: +- **The `cmake` option dimensions** become the `features` configuration. + cargo-hack builds every combination of features, so no configuration + per combination is written. +- **Platforms stay platforms.** +- **Machines sai has no builder for**, big-endian and 32-bit, are + interpreted by Miri rather than run on hardware. + +## Installing Rust on a builder + +Install as the user the sai builder runs jobs as. sai sets `HOME` to the +builder's home for each job, and `scripts/sai.sh` puts `$HOME/.cargo/bin` +first on `PATH`, so rustup's usual per-user install is found without +changing the builder daemon's environment. + +### Every unix builder with rustup + +These are the Linux builders (x86_64, aarch64 and riscv64) and the macOS +ones. rustup provides host toolchains for all of them. A C linker must +be present: it is already, as these builders build C lws. On macOS that +means the Xcode command line tools. + +```sh +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile default +``` + +That gives stable with rustfmt and clippy, which is all the `test` +profile needs. Updating later is `rustup update`. + +### The fedora44 x86_64 builder, which runs everything else + +On top of the above: + +```sh +rustup toolchain install 1.85 --profile minimal # the MSRV build +rustup toolchain install nightly --profile minimal --component miri,rust-src +rustup target add thumbv6m-none-eabi thumbv7em-none-eabihf riscv32imc-unknown-none-elf +cargo install --locked cargo-deny cargo-audit cargo-hack +dnf install cmake gcc zlib-devel git # the C oracle build +``` + +The first Miri run of each target builds that target's standard library +for Miri, and takes a few minutes. Later runs reuse it. + +Network access during jobs: +- `cargo audit` fetches the RustSec advisory database from github.com on + each run; +- the oracle clones and fetches libwebsockets from libwebsockets.org, + keeping its checkout in `$HOME/lws-oracle` (or `$LWS_ORACLE`). + +Nothing else fetches. The workspace has no dependencies, and every build +is `--locked`. + +### freebsd/aarch64 + +rustup has no host toolchain for aarch64 FreeBSD, so use the packaged one, +which must be at least the workspace's `rust-version` (1.85): + +```sh +pkg install rust +``` + +It installs under `/usr/local/bin`, which the builder's `PATH` already has. + +### w11/x86_64 (MSVC) + +Run `rustup-init.exe` from <https://rustup.rs> with the default host, +`x86_64-pc-windows-msvc`; the builder already has the Visual Studio build +tools it links with. Do it as the account the sai builder service runs +as, and make sure that account's `%USERPROFILE%\.cargo\bin` is on the +service's `PATH`, since the Windows step runs `cargo` directly under +`cmd`. + +### Later + +- **esp32**: building for the ESP32's xtensa cores needs Espressif's Rust + fork, installed with `espup`. That waits until there is something to + run on the device. +- **fuzzing**: cargo-fuzz needs nightly and clang. It joins the fuzz + builder once the h1 parser has fuzz targets. + +## A possible sai change + +Nothing here needs sai changed. A neutral name for the configuration +key, such as `build` beside `cmake`, would make npro's `.sai.json` read +better, but `cmake` does the job as it is. diff --git a/scripts/sai.sh b/scripts/sai.sh new file mode 100755 index 0000000..ddcbde7 --- /dev/null +++ b/scripts/sai.sh @@ -0,0 +1,109 @@ +#!/bin/sh +# +# What each sai configuration runs on a unix builder. .sai.json's platforms +# all run "scripts/sai.sh ${cmake}", and each configuration's "cmake" names +# a profile here and its arguments: sai's step template only knows the +# variable names prep, cmake and cpack, so "cmake" carries the profile. +# +# gate every gate a commit must pass (scripts/ci.sh) +# test the workspace's tests, natively on this builder +# features clippy over every combination of every crate's features +# nostd the sans-IO crates built for targets with no std +# miri <target> the tests under Miri, interpreting <target>: a 32-bit or +# big-endian machine without needing one +# oracle C lws main-dev's transcripts, recorded afresh, compared +# with the copies in crates/npro-test/transcripts +# +# docs/sai.md says what each builder needs installed. + +set -eu + +cd "$(dirname "$0")/.." + +# rustup installs for the user the builder runs as, in its $HOME, which sai +# sets for the job; its tools are not on the builder daemon's PATH +PATH="$HOME/.cargo/bin:$PATH" +export PATH + +jobs="${SAI_PARALLEL:-4}" + +# the sans-IO crates, which must build with no std at all +nostd_crates="npro-core" +nostd_targets="thumbv6m-none-eabi thumbv7em-none-eabihf riscv32imc-unknown-none-elf" + +profile="${1:-}" +[ $# -gt 0 ] && shift + +case "$profile" in +gate) + exec scripts/ci.sh + ;; + +test) + cargo --version + exec cargo test --workspace --all-features --locked -j "$jobs" + ;; + +features) + exec cargo hack clippy --workspace --feature-powerset --all-targets \ + --locked -j "$jobs" -- -D warnings + ;; + +nostd) + for t in $nostd_targets; do + for c in $nostd_crates; do + echo "== $c for $t" + cargo build -p "$c" --all-features --target "$t" --locked \ + -j "$jobs" + done + done + ;; + +miri) + target="${1:?miri needs a target, eg s390x-unknown-linux-gnu}" + # npro-test reads its transcripts from disk, which Miri's isolation + # would refuse + MIRIFLAGS="${MIRIFLAGS:-} -Zmiri-disable-isolation" \ + exec cargo +nightly miri test --workspace --all-features --locked \ + --target "$target" + ;; + +oracle) + # A C lws checkout of main-dev kept between jobs, built with what + # every transcript needs: fault injection for the seeded random, + # extensions and zlib for the permessage-deflate ones + c="${LWS_ORACLE:-$HOME/lws-oracle}" + if [ ! -d "$c/.git" ]; then + git clone --depth 50 -b main-dev \ + https://libwebsockets.org/repo/libwebsockets "$c" + fi + git -C "$c" fetch --depth 50 origin main-dev + git -C "$c" checkout -q --detach FETCH_HEAD + echo "== C lws $(git -C "$c" log -1 --format='%h %s')" + + cmake -S "$c" -B "$c/build-oracle" -DCMAKE_BUILD_TYPE=DEBUG \ + -DLWS_WITH_SSL=OFF -DLWS_WITH_MINIMAL_EXAMPLES=ON \ + -DLWS_WITH_SYS_FAULT_INJECTION=ON -DLWS_WITHOUT_EXTENSIONS=OFF \ + -DLWS_WITH_ZLIB=ON + cmake --build "$c/build-oracle" --target lws-api-test-sansio -j "$jobs" + + rec="$(mktemp -d)" + trap 'rm -rf "$rec"' EXIT + (cd "$c/minimal-examples-lowlevel/api-tests/api-test-sansio" && + "$c/build-oracle/bin/lws-api-test-sansio" -d 1 --record "$rec") + + ours=crates/npro-test/transcripts + if diff -rq "$rec" "$ours" -x 'README*' -x C-COMMIT; then + echo "== the transcripts match C $(git -C "$c" log -1 --format=%h)" + exit 0 + fi + echo "C's transcripts moved since $(cut -c1-12 "$ours/C-COMMIT"):" + echo "sync them with scripts/sync-c-oracle.sh and say why in the commit" + exit 1 + ;; + +*) + echo "usage: $0 gate | test | features | nostd | miri <target> | oracle" >&2 + exit 1 + ;; +esac
Page fetched 0s ago, creation time: 1ms (vhost etag hits: 0%, cache hits: 0%)