Project homepage Mailing List  Warmcat.com  API Docs  Github Mirror 
    npro  
 Modern all-safe Rust Network Protocol library supporting h1, h2, h3, ws, wt sans-IO and with socket IO + tls
git clone https://npro.rs/repo/npro
 
root / assets / linux-ubuntu-1804.svg
Author[]Andy Green <andy@warmcat.com> 2026-09-18 19:07 UTC
Committer[]Andy Green <andy@warmcat.com> 2026-09-21 07:06 UTC
Tree49b040cea507269eac987f4c0cea7f40536e56dd   Raw Patch
 
server: authenticate the builder and power ws endpoints with a link secret, fixes F-014
server: authenticate the builder and power ws endpoints with a link secret, fixes F-014

Anyone who could reach sai-server's listener could connect to /sai/builder
or /sai/power and be instantly treated as a builder / sai-power peer:
register platforms to win real task dispatches (with the repo build
script and the artifact upload nonce), forge task results, starve real
builders, and feed false power state.  There was no secret, token or
cert anywhere on the path, and the listener is expected to be reachable
by distributed builders across the internet.

Introduce a fleet-wide link secret, "link-key", configured on all
daemons (matching sai-server's "notification-key" style, a required pvo
so sai-server refuses to start without it).  The first ws message on
any /builder or /power connection must be

  {"schema":"com.warmcat.sai.linkauth","secret":"<link-key>"}

compared in constant time; sai-server processes nothing else from the
peer until then and drops the connection on anything else.  sai-builder
(main and artifact links), sai-power and sai-virt send it ahead of
their first messages on (re)connect.

An unauthenticated connection can still sit on the endpoint and receive
the few server->client notices that are queued at establish (viewer
present flags, pending platform lists); task dispatch itself requires
platform registration, which is rx and therefore behind the gate.
diff --git a/README.md b/README.md index 36ff504..c0da922 100644 --- a/README.md +++ b/README.md @@ -95,6 +95,50 @@ the browser, with JWT-authentication for manual job control. most cases spend most of their time idle, this enables a very good optimization of average power down to nearly zero. +## Link authentication ("link-key") + +sai-server's listener is expected to be reachable by distributed builders +across the internet, so daemons connecting to it must prove a shared +fleet-wide secret before sai-server treats them as part of the fleet. +Without it, anyone who could reach the listener would count as a builder, +able to register platforms, win real task dispatches (with the repo build +script and artifact upload nonce) and forge task results. + +The first ws message on any connection to sai-server's `/sai/builder` or +`/sai/power` endpoints must be + +``` +{"schema":"com.warmcat.sai.linkauth","secret":"<link-key>"} +``` + +sai-server compares the secret in constant time and processes nothing else +from the peer until it is proven, dropping the connection on a wrong secret. +The client daemons take care of this themselves and send the auth message +ahead of their first messages on (re)connect: sai-builder (on its main and +artifact links), sai-power and sai-virt. sai-power also requires the same +secret from builders registering with it over the LAN. + +Set up the same key on both sides, generating it as 64 hex chars from 32 +random bytes the same way as `notification-key`: + +``` +$ dd if=/dev/random bs=32 count=1 | sha256sum | cut -d' ' -f1 +``` + + - on sai-server, in the `vhosts|ws-protocols|com-warmcat-sai` section of + the config JSON next to `notification-key`... sai-server refuses to start + without it + +``` + "link-key": "<link-key>", +``` + + - on every sai-builder, sai-power and sai-virt host, at the top level of + the daemon's conf, eg, `/etc/sai/builder/conf` + +``` + "link-key": "<link-key>", +``` ## sai-web <-> sai-server control link ("sockpath") sai-web is the only thing that talks to sai-server on behalf of browsers, diff --git a/etc-sai-EXAMPLE/builder/conf b/etc-sai-EXAMPLE/builder/conf index fd03488..4f9ee48 100644 --- a/etc-sai-EXAMPLE/builder/conf +++ b/etc-sai-EXAMPLE/builder/conf @@ -2,6 +2,12 @@ "perms": "sai:nobody", "home": "/home/sai", "host": "bionic-noi", + + # the fleet-wide secret sai-server's "link-key" pvo also has; + # the server refuses our connection without it + # + "link-key": "34c8e17b90d2a6f5c1e8430b76af2915e0d4cbf68a27e19dd5b0f3c6e48a71d2", + "rebuild_script_user": "cd ~/libwebsockets/build && git fetch origin +main:m && git reset --hard m && make -j12 && cd ../../sai/build && git fetch origin +main:m && git reset --hard m && make -j12", "rebuild_script_root": "cd /home/agreen/libwebsockets/build && make -j12 install && cd /home/agreen/sai/build && make -j12 install && systemctl restart sai-builder", diff --git a/etc-sai-EXAMPLE/power/conf b/etc-sai-EXAMPLE/power/conf index 5573538..bc4947c 100644 --- a/etc-sai-EXAMPLE/power/conf +++ b/etc-sai-EXAMPLE/power/conf @@ -1,6 +1,11 @@ { "perms": "sai:nobody", + # the fleet-wide secret sai-server's "link-key" pvo also has; + # the server refuses our connection without it + # + "link-key": "34c8e17b90d2a6f5c1e8430b76af2915e0d4cbf68a27e19dd5b0f3c6e48a71d2", + "servers": [ { "url" : "wss://libwebsockets.org:4444/sai/builder", diff --git a/etc-sai-EXAMPLE/server/conf.d/mydomain.com b/etc-sai-EXAMPLE/server/conf.d/mydomain.com index 7269381..1a3dee8 100644 --- a/etc-sai-EXAMPLE/server/conf.d/mydomain.com +++ b/etc-sai-EXAMPLE/server/conf.d/mydomain.com @@ -74,6 +74,15 @@ # "notification-key": "51b3ee2f06ef2a893cfe901972bd13065d7dbae4cf087b396ee38e7bf78f79a6", + # The fleet-wide secret that builders and sai-power + # daemons must present in the first ws message on + # their connection, before anything else from them is + # processed. sai-server refuses to start without it. + # Generate it the same way as notification-key and set + # the same value as "link-key" in every builder's and + # sai-power's conf. + # + "link-key": "34c8e17b90d2a6f5c1e8430b76af2915e0d4cbf68a27e19dd5b0f3c6e48a71d2", # Unix socket sai-server serves its control link for # sai-web on. The link is admin-equivalent, so give it # a filesystem path: lws binds it before dropping diff --git a/etc-sai-EXAMPLE/virt/conf b/etc-sai-EXAMPLE/virt/conf index 7fe0033..975df3f 100644 --- a/etc-sai-EXAMPLE/virt/conf +++ b/etc-sai-EXAMPLE/virt/conf @@ -4,6 +4,11 @@ "hostname": "virt-host-1", "max_vms": 4, + # the fleet-wide secret sai-server's "link-key" pvo also has; + # the server refuses our connection without it + # + "link-key": "34c8e17b90d2a6f5c1e8430b76af2915e0d4cbf68a27e19dd5b0f3c6e48a71d2", + "servers": [ { "url": "wss://libwebsockets.org:4444/sai/builder" diff --git a/src/builder/b-artifacts.c b/src/builder/b-artifacts.c index 4639a37..7636431 100644 --- a/src/builder/b-artifacts.c +++ b/src/builder/b-artifacts.c @@ -56,6 +56,23 @@ saib_artifact_tx(void *userobj, lws_ss_tx_ordinal_t ord, uint8_t *buf, *flags = 0; + if (!ap->sent_auth) { + /* + * This connection lands on sai-server's /builder endpoint + * too, so the link auth message has to go out before the + * artifact JSON + bulk data are accepted. + */ + *flags |= LWSSS_FLAG_SOM | LWSSS_FLAG_EOM; + *len = (size_t)lws_snprintf((char *)buf, *len, + "{\"schema\":\"" SAI_LINKAUTH_SCHEMA + "\",\"secret\":\"%s\"}", + builder.link_key ? builder.link_key : ""); + + ap->sent_auth = 1; + + return lws_ss_request_tx(ap->ss); + } + if (!ap->sent_json) { *flags |= LWSSS_FLAG_SOM; diff --git a/src/builder/b-conf.c b/src/builder/b-conf.c index 8e237f0..6701c28 100644 --- a/src/builder/b-conf.c +++ b/src/builder/b-conf.c @@ -37,6 +37,7 @@ static const char * const paths_global[] = { "metrics_uri", "metrics_path", "metrics_secret", + "link-key", "sai-power", "power_controller", "power-on.type", @@ -57,6 +58,7 @@ enum enum_paths_global { LEJPM_METRICS_URI, LEJPM_METRICS_PATH, LEJPM_METRICS_SECRET, + LEJPM_LINK_KEY, LEJPM_SAI_POWER, LEJPM_POWER_CONTROLLER, LEJPM_POWER_ON_TYPE, @@ -306,6 +308,10 @@ saib_conf_global_cb(struct lejp_ctx *ctx, char reason) pp = &a->builder->metrics_secret; break; + case LEJPM_LINK_KEY: + pp = &a->builder->link_key; + break; + case LEJPM_POWER_ON_TYPE: pp = &a->builder->power_on_type; break; diff --git a/src/builder/b-private.h b/src/builder/b-private.h index bf64359..993acb8 100644 --- a/src/builder/b-private.h +++ b/src/builder/b-private.h @@ -173,6 +173,9 @@ struct sai_builder { const char *metrics_path; const char *metrics_secret; + /* fleet secret shared with sai-server ("link-key" in conf) */ + const char *link_key; + const char *url_sai_power; const char *power_controller_name; diff --git a/src/builder/b-ws-server.c b/src/builder/b-ws-server.c index 9b70798..2b758c5 100644 --- a/src/builder/b-ws-server.c +++ b/src/builder/b-ws-server.c @@ -797,6 +797,33 @@ saib_m_state(void *userobj, void *sh, lws_ss_constate_t state, lws_sul_schedule(builder.context, 0, &spm->sul_load_report, saib_sul_load_report_cb, 1); + /* + * sai-server refuses to process anything from us until we + * prove the fleet link secret, so it has to be the first + * message on the (re)connection, ahead of the plats. + */ + { + uint8_t abuf[LWS_PRE + 256]; + size_t al; + + if (!builder.link_key) + lwsl_err("%s: no link-key in conf, " + "sai-server will refuse us\n", + __func__); + + al = (size_t)lws_snprintf((char *)abuf + LWS_PRE, + sizeof(abuf) - LWS_PRE, + "{\"schema\":\"" SAI_LINKAUTH_SCHEMA + "\",\"secret\":\"%s\"}", + builder.link_key ? + builder.link_key : ""); + + if (saib_srv_queue_tx(spm->ss, abuf + LWS_PRE, al, + LWSSS_FLAG_SOM | + LWSSS_FLAG_EOM)) + return -1; + } + if (saib_srv_queue_json_fragments_helper(spm->ss, lsm_schema_map_plat, LWS_ARRAY_SIZE(lsm_schema_map_plat), diff --git a/src/common/include/private.h b/src/common/include/private.h index bb11c4f..8dc7d8e 100644 --- a/src/common/include/private.h +++ b/src/common/include/private.h @@ -61,6 +61,14 @@ */ #define SAI_WEBSRV_UDS_DEFAULT "@com.warmcat.sai-websrv" +/* + * Builders and sai-power daemons prove the fleet-wide link secret (conf + * "link-key" on all three daemons) in the first ws message on their + * connection to sai-server, before sai-server processes anything else from + * them. This is the schema name of that message. + */ +#define SAI_LINKAUTH_SCHEMA "com.warmcat.sai.linkauth" + #define SAI_BUILDER_INSTANCE_LIMIT 256 struct sai_plat; @@ -488,6 +496,7 @@ typedef struct { size_t len; int uid; int fd; + char sent_auth; char sent_json; int run; } sai_artifact_t; diff --git a/src/power/p-conf.c b/src/power/p-conf.c index 95eaec5..3c8bb93 100644 --- a/src/power/p-conf.c +++ b/src/power/p-conf.c @@ -33,6 +33,7 @@ static const char * const paths_global[] = { "perms", "wol-if", "database", + "link-key", "servers[].url", "servers[]" }; @@ -41,6 +42,7 @@ enum enum_paths_global { LEJPM_PERMS, LEJPM_WOL_IF, LEJPM_DATABASE, + LEJPM_LINK_KEY, LEJPM_SERVERS_URL, LEJPM_SERVERS }; @@ -99,6 +101,10 @@ saip_conf_global_cb(struct lejp_ctx *ctx, char reason) pp = &a->power->database; break; + case LEJPM_LINK_KEY: + pp = &a->power->link_key; + break; + case LEJPM_SERVERS_URL: pp = &a->sai_server->url; lwsl_user("%s: server url %.*s\n", __func__, ctx->npos, ctx->buf); diff --git a/src/power/p-private.h b/src/power/p-private.h index fe50aeb..0f0d880 100644 --- a/src/power/p-private.h +++ b/src/power/p-private.h @@ -143,6 +143,9 @@ struct sai_power { const char *wol_if; const char *database; + + /* fleet secret shared with sai-server ("link-key" in conf) */ + const char *link_key; struct sqlite3 *pdb; diff --git a/src/power/p-ws-server.c b/src/power/p-ws-server.c index 1e88492..755f082 100644 --- a/src/power/p-ws-server.c +++ b/src/power/p-ws-server.c @@ -475,6 +475,33 @@ saip_m_state(void *userobj, void *sh, lws_ss_constate_t state, case LWSSSCS_CONNECTED: lwsl_ss_notice(sps->ss, "@@@@@@@@@@@@@@ sai-power CONNECTED to server"); + + /* + * sai-server refuses to process anything from us until we + * prove the fleet link secret, so it has to be the first + * message on the (re)connection, ahead of the stay info. + */ + { + saip_server_link_t *m = + (saip_server_link_t *)lws_ss_to_user_object(sps->ss); + uint8_t abuf[LWS_PRE + 256]; + size_t al; + + if (!power.link_key) + lwsl_err("%s: no link-key in conf, sai-server " + "will refuse us\n", __func__); + + al = (size_t)lws_snprintf((char *)abuf + LWS_PRE, + sizeof(abuf) - LWS_PRE, + "{\"schema\":\"" SAI_LINKAUTH_SCHEMA + "\",\"secret\":\"%s\"}", + power.link_key ? power.link_key : ""); + + sai_ss_queue_frag_on_buflist_REQUIRES_LWS_PRE(sps->ss, + &m->bl_pwr_to_srv, abuf + LWS_PRE, al, + LWSSS_FLAG_SOM | LWSSS_FLAG_EOM); + } + saip_queue_stay_info(sps); break; diff --git a/src/server/s-comms.c b/src/server/s-comms.c index b456bc2..a6c3fbf 100644 --- a/src/server/s-comms.c +++ b/src/server/s-comms.c @@ -60,6 +60,84 @@ static const struct { { "sai sha512=", LWS_GENHMAC_TYPE_SHA512 }, }; +/* paths of the link auth message's members we care about */ +static const char * const link_auth_paths[] = { "secret" }; + +static signed char +sais_link_auth_lejp_cb(struct lejp_ctx *ctx, char reason) +{ + struct pss *pss = (struct pss *)ctx->user; + struct vhd *vhd = pss->vhd; + + if (reason == LEJPCB_VAL_STR_END && ctx->path_match == 1) { + size_t kl = strlen(vhd->link_key); + + if (strlen(ctx->buf) != kl || + lws_timingsafe_bcmp(ctx->buf, vhd->link_key, (uint32_t)kl)) + /* wrong secret... kill the parse */ + return -1; + + pss->auth_secret_ok = 1; + } + + return 0; +} + +/* + * The first ws message on a /builder or /power connection must prove the + * fleet link secret ({"schema":"com.warmcat.sai.linkauth","secret":...}). + * Nothing else from the peer is processed until it did: without this, any + * internet peer that could reach the listener was a "builder", able to + * register platforms, receive real task dispatches (with their build + * scripts and artifact upload nonces) and forge task results. + * + * Returns 0 to keep waiting / on success, or -1 to drop the connection. + */ +static int +sais_link_auth_rx(struct vhd *vhd, struct pss *pss, uint8_t *buf, size_t bl, + unsigned int ss_flags) +{ + int n; + + if (ss_flags & LWSSS_FLAG_SOM) { + lejp_construct(&pss->auth_ctx, sais_link_auth_lejp_cb, pss, + link_auth_paths, + LWS_ARRAY_SIZE(link_auth_paths)); + pss->auth_secret_ok = 0; + } + + n = lejp_parse(&pss->auth_ctx, buf, (int)bl); + if (n < 0 && n != LEJP_CONTINUE) { + lwsl_notice("%s: link auth JSON invalid, dropping\n", __func__); + return -1; + } + + if (n == LEJP_CONTINUE) { + /* the auth message is not complete yet */ + + if (ss_flags & LWSSS_FLAG_EOM) { + lwsl_notice("%s: link auth incomplete at EOM, dropping\n", + __func__); + return -1; + } + + return 0; + } + + /* the auth JSON completed */ + + if (!pss->auth_secret_ok) { + lwsl_notice("%s: link secret mismatch, dropping\n", __func__); + return -1; + } + + pss->link_authed = 1; + lwsl_notice("%s: peer authenticated on %s\n", __func__, + pss->is_power ? "/power" : "/builder"); + + return 0; +} + int sai_get_head_status(struct vhd *vhd, const char *projname) { @@ -118,6 +196,16 @@ s_callback_ws(struct lws *wsi, enum lws_callback_reasons reason, void *user, return -1; } + /* + * The fleet-wide secret builders and sai-power daemons must + * present in their first ws message. Required: without it + * the builder/power endpoints are unauthenticated. + */ + if (lws_pvo_get_str(in, "link-key", &vhd->link_key)) { + lwsl_err("%s: link-key pvo required\n", __func__); + return -1; + } + if (!lws_pvo_get_str(in, "task-abandoned-timeout-mins", &num)) vhd->task_abandoned_timeout_mins = (unsigned int)atoi(num); else @@ -596,8 +684,17 @@ s_callback_ws(struct lws *wsi, enum lws_callback_reasons reason, void *user, * A ws client sent us something... it could be a builder or * it could be sai-power. We can tell which by the `is_power` * flag we set in the pss during ESTABLISHED. + * + * Either way, until it proved the link secret in its first + * message, nothing else it sends is processed. */ + if (!pss->link_authed) { + if (sais_link_auth_rx(vhd, pss, in, len, ssf) < 0) + return -1; + break; + } + if (pss->is_power) { if (sais_power_rx(vhd, pss, in, len, ssf)) { lwsl_err("%s: sais_power_rx returned error, dropping connection\n", diff --git a/src/server/s-private.h b/src/server/s-private.h index 48428a8..71c3f58 100644 --- a/src/server/s-private.h +++ b/src/server/s-private.h @@ -170,6 +170,9 @@ struct pss { lws_struct_args_t a; struct lejp_ctx ctx_power; + /* link auth state: parse of the first ws message from the peer */ + struct lejp_ctx auth_ctx; + const char *server_name; struct lwsac *query_ac; @@ -212,6 +215,8 @@ struct pss { unsigned int announced:1; unsigned int bulk_binary_data:1; unsigned int is_power:1; + unsigned int link_authed:1; + unsigned int auth_secret_ok:1; uint8_t ovstate; /* SOS_ substate when doing overview */ }; @@ -264,6 +269,7 @@ struct vhd { lws_usec_t last_check_abandoned_tasks; const char *notification_key; + const char *link_key; /* fleet secret builders / sai-power auth with */ const char *websrv_sockpath; /* control link uds we serve */ unsigned int task_abandoned_timeout_mins; diff --git a/src/virt/v-conf.c b/src/virt/v-conf.c index eb2360c..2cb9f3e 100644 --- a/src/virt/v-conf.c +++ b/src/virt/v-conf.c @@ -129,11 +129,13 @@ saiv_config(struct sai_virt *virt, const char *d) } static const char * const paths_global[] = { + "link-key", "servers[].url", "max_vms", }; enum { + VJG_LINK_KEY, VJG_SERVER_URL, VJG_MAX_VMS, }; @@ -145,6 +147,10 @@ saiv_conf_global_cb(struct lejp_ctx *ctx, char reason) if (reason == LEJPCB_VAL_STR_END) { switch (ctx->path_match - 1) { + case VJG_LINK_KEY: + v->link_key = strdup(ctx->buf); + break; + case VJG_SERVER_URL: { saiv_server_t *srv = malloc(sizeof(*srv)); diff --git a/src/virt/v-private.h b/src/virt/v-private.h index da156f3..7223762 100644 --- a/src/virt/v-private.h +++ b/src/virt/v-private.h @@ -65,6 +65,9 @@ struct sai_virt { const char *perms; /* user:group */ const char *port; /* port we listen on */ + /* fleet secret shared with sai-server ("link-key" in conf) */ + const char *link_key; + char hostname[64]; struct lwsac *pending_tasks_ac; diff --git a/src/virt/v-ws-server.c b/src/virt/v-ws-server.c index e85d5e3..fb63f77 100644 --- a/src/virt/v-ws-server.c +++ b/src/virt/v-ws-server.c @@ -266,6 +266,30 @@ saiv_server_state(void *userobj, void *sh, lws_ss_constate_t state, case LWSSSCS_CONNECTED: lwsl_notice("%s: Connected to sai-server\n", __func__); + /* + * sai-server refuses to process anything from us until we + * prove the fleet link secret, so it has to be the first + * message on the (re)connection. + */ + { + uint8_t abuf[LWS_PRE + 256]; + size_t al; + + if (!virt.link_key) + lwsl_err("%s: no link-key in conf, sai-server " + "will refuse us\n", __func__); + + al = (size_t)lws_snprintf((char *)abuf + LWS_PRE, + sizeof(abuf) - LWS_PRE, + "{\"schema\":\"" SAI_LINKAUTH_SCHEMA + "\",\"secret\":\"%s\"}", + virt.link_key ? virt.link_key : ""); + + sai_ss_queue_frag_on_buflist_REQUIRES_LWS_PRE(g->ss, + &g->bl_tx, abuf + LWS_PRE, al, + LWSSS_FLAG_SOM | LWSSS_FLAG_EOM); + } + sai_power_managed_builders_t pmb; memset(&pmb, 0, sizeof(pmb));
Page fetched 0s ago, creation time: 7ms (vhost etag hits: 0%, cache hits: 0%)