| Author | Andy Green <andy@warmcat.com> 2026-10-06 04:19 UTC | | Committer | Andy Green <andy@warmcat.com> 2026-10-06 05:30 UTC | | Tree | cf21d41664466bed365038030dc322693ed47708 Raw Patch | | | scripts: autobahn, in upstream's Docker image, and its builder | scripts: autobahn, in upstream's Docker image, and its builder
scripts/autobahn.sh runs Autobahn|Testsuite's wstest, which is Python 2
only, in upstream's crossbario/autobahn-testsuite image, pinned by digest
once the builder has pulled it: as fuzzingclient against an echo server
started on 9001, or as fuzzingserver against an echo client run once a
case, as C's scripts/autobahn-test-*.sh do by hand. It writes the
configuration itself, with C's exclusions and why (the 12.x ones' reason
is still to be found), and fails unless every case is OK, NON-STRICT or
INFORMATIONAL, as C judges them. It runs against C's minimal echo
examples now, for a baseline, and against npro-io's once phase 1g has
them.
docs/sai.md says how to set up the Debian 13 VM with Docker it runs on,
and no longer says the main workspace fetches nothing: since
permessage-deflate, cargo fetches miniz_oxide and adler2.
Not yet run: this was written where Docker is not available, so its first
run is the builder's.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019kg5Eemy68ZaqDBcUJQG6J
|
diff --git a/docs/sai.md b/docs/sai.md
index 5af949b..0a2b9a4 100644
--- a/docs/sai.md
+++ b/docs/sai.md
@@ -239,8 +239,9 @@ through with a cargo error.
`fuzz/Cargo.lock` pins them (`cargo fetch --locked`), once per builder.
The pool is synced by sai-builder itself, not the job.
-Nothing else fetches. The main workspace has no dependencies, and every
-build is `--locked`.
+Nothing else fetches but cargo itself: the main workspace's only
+dependencies, `miniz_oxide` and `adler2` for npro-ws's `pmd` feature, come
+from crates.io as `Cargo.lock` pins them, and every build is `--locked`.
### When a job fails on setup
@@ -261,6 +262,57 @@ build is `--locked`.
- **esp32**, later: xtensa needs Espressif's Rust fork, installed with
`espup`, once there is something to run on the device.
+## The autobahn builder
+
+Autobahn|Testsuite is the ws conformance gate (phase 1g of
+[port-plan.md](port-plan.md)). Its `wstest` is Python 2 only, so it runs
+in upstream's Docker image, `crossbario/autobahn-testsuite`, on a builder
+of its own: a Debian 13 VM with Docker, like the one qir uses.
+`scripts/autobahn.sh` runs it, both ways, against any echo server or
+client.
+
+Setting one up:
+
+1. **The VM.** Debian 13 (trixie), x86_64; 2 vCPUs, 4GiB and 20GiB of
+ disk are plenty. Make it as you make the other builders, with the
+ user sai runs jobs as. If it is a sai-virt image, everything below goes
+ into the base image, not an overlay.
+2. **Docker.** Debian's own package is enough:
+
+ apt install docker.io
+ systemctl enable --now docker
+ usermod -aG docker sai # the user jobs run as
+
+ Membership of `docker` is as good as root on that VM, which is a reason
+ for it to be a VM of its own. Rootless Podman, with the
+ `podman-docker` package giving a `docker` command, is the alternative
+ if that matters.
+3. **The image, pinned.** As that user:
+
+ docker pull crossbario/autobahn-testsuite
+ docker inspect --format '{{index .RepoDigests 0}}' crossbario/autobahn-testsuite
+
+ The second prints `crossbario/autobahn-testsuite@sha256:…`; that goes
+ in `scripts/autobahn.sh` as its `image`, in a commit, so every run uses
+ exactly that image. Once it is pulled, jobs need no network for it.
+4. **Rust,** as for every builder (above): rustup as that user.
+5. **A first run against C,** to know the builder works before npro has
+ anything to test. As that user, in a libwebsockets checkout:
+
+ apt install build-essential cmake git libssl-dev
+ mkdir build && cd build
+ cmake .. -DLWS_WITH_MINIMAL_EXAMPLES=1 && make -j4
+ /path/to/npro/scripts/autobahn.sh server bin/lws-minimal-ws-server-echo -p 9001
+ /path/to/npro/scripts/autobahn.sh client bin/lws-minimal-ws-client-echo -s 127.0.0.1 -p 9001 -u
+
+ Each says how many cases ran and how many failed, and leaves the
+ reports in `./autobahn/reports`, `index.html` among them. Whatever C
+ fails is worth knowing in itself, and is the baseline npro is held to.
+6. **Into sai,** as the other builders are, with a platform name such as
+ `linux-debian13/x86_64-amd/gcc`. The `.sai.json` configuration that
+ runs autobahn there comes with npro-io's echo server and client, in
+ phase 1g: until then there is nothing of npro's to run.
+
## A possible sai change
Nothing here needs sai changed. A neutral name for the configuration
diff --git a/scripts/autobahn.sh b/scripts/autobahn.sh
new file mode 100755
index 0000000..1d433e9
--- /dev/null
+++ b/scripts/autobahn.sh
@@ -0,0 +1,137 @@
+#!/usr/bin/env bash
+#
+# Autobahn|Testsuite against a ws echo server or client, both ways, with
+# upstream's Docker image: wstest is Python 2 only, and the image is how
+# upstream runs it. docs/sai.md says how to set up the builder it runs
+# on; docs/port-plan.md (phase 1g) why it is a gate.
+#
+# scripts/autobahn.sh server <command starting an echo server on 9001>
+# scripts/autobahn.sh client <command running one echo client connection>
+#
+# In "server" mode, the command is started in the background, and wstest's
+# fuzzingclient runs every case against ws://127.0.0.1:9001. In "client"
+# mode, wstest's fuzzingserver listens on 9001, and the command is run
+# once a case, with the path to ask for appended as its last argument
+# (/runCase?case=N&agent=AGENT), until it fails, the cases being over; then
+# once with /updateReports?agent=AGENT. The echo examples of C, for a
+# baseline, built with -DLWS_WITH_MINIMAL_EXAMPLES=1:
+#
+# scripts/autobahn.sh server bin/lws-minimal-ws-server-echo -p 9001
+# scripts/autobahn.sh client bin/lws-minimal-ws-client-echo \
+# -s 127.0.0.1 -p 9001 -u
+#
+# Reports go to ./autobahn/reports/{servers,clients}, index.html among
+# them. The exit code is 0 only if every case not excluded below is OK,
+# NON-STRICT or INFORMATIONAL, as C's scripts judge it.
+#
+# AUTOBAHN_IMAGE overrides the image; AUTOBAHN_AGENT the agent name.
+
+set -euo pipefail
+
+# Upstream's image, pinned: a digest, not a tag, so what runs is what was
+# checked. Pin it on the builder's first run: docker pull
+# crossbario/autobahn-testsuite, then docker inspect --format
+# '{{index .RepoDigests 0}}' crossbario/autobahn-testsuite, and put that
+# here.
+image="${AUTOBAHN_IMAGE:-crossbario/autobahn-testsuite@sha256:PIN-ME}"
+agent="${AUTOBAHN_AGENT:-npro}"
+port=9001
+
+# Cases excluded, and why. C's, until npro's own reasons replace them:
+# 2.10, 2.11: several pings in flight; RFC 6455 does not require it, and
+# C and npro keep one pending pong.
+# 12.3.1, 12.3.2, 12.4.*, 12.5.*: excluded by C's client script for
+# permessage-deflate with its client; the reason is not
+# written down there, and is to be found before npro keeps
+# them out.
+exclude_server='"2.10", "2.11"'
+exclude_client='"2.10", "2.11", "12.3.1", "12.3.2", "12.4.*", "12.5.*"'
+
+mode="${1:?usage: $0 server|client <command...>}"
+shift
+[ $# -gt 0 ] || { echo "$0: no command given" >&2; exit 2; }
+
+case "$image" in
+*PIN-ME*)
+ echo "$0: pin the image's digest first: see the top of $0" >&2
+ exit 2
+ ;;
+esac
+
+work="$PWD/autobahn"
+mkdir -p "$work/config" "$work/reports"
+
+# wstest in the container, on the host's network so 127.0.0.1 is shared,
+# as this user so the reports are ours
+name="npro-autobahn-$$"
+wstest() {
+ docker run --rm --name "$name" --network host --user "$(id -u):$(id -g)" \
+ -v "$work/config:/config:ro" -v "$work/reports:/reports" \
+ "$image" wstest "$@"
+}
+
+# the cases that are not OK, NON-STRICT or INFORMATIONAL, from an index;
+# "behavior": exactly, not "behaviorClose":
+failures() {
+ grep '"behavior":' "$1" | grep -v -e '"OK"' -e '"NON-STRICT"' \
+ -e '"INFORMATIONAL"' || true
+}
+
+judge() {
+ local index="$1" what="$2" total bad
+ [ -s "$index" ] || { echo "$what: no report" >&2; return 1; }
+ total=$(grep -c '"behavior":' "$index" || true)
+ bad=$(failures "$index" | wc -l)
+ echo "$what: $total cases, $bad failed"
+ [ "$bad" -eq 0 ]
+}
+
+case "$mode" in
+server)
+ cat >"$work/config/fuzzingclient.json" <<EOF
+{
+ "outdir": "/reports/servers",
+ "servers": [ { "agent": "$agent", "url": "ws://127.0.0.1:$port" } ],
+ "cases": [ "*" ],
+ "exclude-cases": [ $exclude_server ],
+ "exclude-agent-cases": {}
+}
+EOF
+ "$@" &
+ server=$!
+ trap 'kill $server 2>/dev/null || true' EXIT
+ sleep 1
+ kill -0 "$server" || { echo "$0: the server did not start" >&2; exit 3; }
+ wstest -m fuzzingclient -s /config/fuzzingclient.json
+ judge "$work/reports/servers/index.json" "autobahn's client, $agent's server"
+ ;;
+client)
+ cat >"$work/config/fuzzingserver.json" <<EOF
+{
+ "url": "ws://127.0.0.1:$port",
+ "outdir": "/reports/clients",
+ "cases": [ "*" ],
+ "exclude-cases": [ $exclude_client ],
+ "exclude-agent-cases": {}
+}
+EOF
+ wstest -m fuzzingserver -s /config/fuzzingserver.json &
+ fuzzer=$!
+ # stopping the docker client need not stop its container
+ trap 'docker rm -f "$name" >/dev/null 2>&1 || true' EXIT
+ sleep 3
+ kill -0 "$fuzzer" || { echo "$0: wstest did not start" >&2; exit 3; }
+ n=1
+ while "$@" "/runCase?case=$n&agent=$agent"; do
+ n=$((n + 1))
+ done
+ "$@" "/updateReports?agent=$agent" || true
+ sleep 2
+ docker rm -f "$name" >/dev/null 2>&1 || true
+ judge "$work/reports/clients/index.json" "autobahn's server, $agent's client"
+ ;;
+*)
+ echo "usage: $0 server|client <command...>" >&2
+ exit 2
+ ;;
+esac
|