Project homepage Mailing List  Warmcat.com  API Docs  Github Mirror 
    npro  
 Modern all-safe Rust Network Protocol library supporting h1, h2, h3, ws, wt sans-IO and with socket IO + tls
git clone https://npro.rs/repo/npro
 
root / READMEs / README-builder-env.md
Author[]google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.c...> 2025-08-15 07:29 UTC
Committer[]Andy Green <andy@warmcat.com> 2025-08-15 10:56 UTC
Tree2e91afecb88a0be40bb70efe475689f64a5708a5   Raw Patch
 
feat: Make builder and server more robust to disconnections
feat: Make builder and server more robust to disconnections

This commit introduces several changes to make the sai build system
more resilient to network failures and other error conditions, as
described in the issue.

Builder-side changes:
- In `src/builder/b-comms.c`, the builder is now configured to kill its
  running build process (the spawned script) if it loses its
  connection to the sai-server. This prevents orphaned build processes
  on the builder.

Server-side changes:
- **Disconnection Handling:** In `src/server/s-ws-builder.c`, when a
  builder disconnects, the server now actively searches for any tasks
  that were in-progress on that builder. These tasks are now reset to
  the 'SAIES_WAITING' state, allowing them to be rescheduled on another
  available builder.

- **Abandoned Task Cleanup:** The periodic cleanup of stale tasks in
  `src/server/s-central.c` has been improved. Instead of marking
  timed-out tasks as 'SAIES_CANCELLED', it now resets them using the
  `sais_task_reset` function, making them eligible for retry.

- **Configurable Timeout:** The timeout for abandoning a task is now
  configurable via the `"task-abandoned-timeout-mins"` vhost option
  in the server configuration, defaulting to 30 minutes if not specified.
  This is handled in `src/server/s-comms.c`.
diff --git a/Testing/Temporary/CTestCostData.txt b/Testing/Temporary/CTestCostData.txt new file mode 100644 index 0000000..ed97d53 --- /dev/null +++ b/Testing/Temporary/CTestCostData.txt @@ -0,0 +1 @@ +--- diff --git a/Testing/Temporary/LastTest.log b/Testing/Temporary/LastTest.log new file mode 100644 index 0000000..cde21b8 --- /dev/null +++ b/Testing/Temporary/LastTest.log @@ -0,0 +1,3 @@ +Start testing: Aug 11 10:05 UTC +---------------------------------------------------------- +End testing: Aug 11 10:05 UTC diff --git a/src/builder/b-comms.c b/src/builder/b-comms.c index 8f0fd49..2ba95bb 100644 --- a/src/builder/b-comms.c +++ b/src/builder/b-comms.c @@ -424,8 +424,8 @@ cleanup_on_ss_disconnect(struct lws_dll2 *d, void *user) ns->spm = NULL; - //if (ns->lsp) - // lws_spawn_piped_kill_child_process(ns->lsp); + if (ns->op && ns->op->lsp) + lws_spawn_piped_kill_child_process(ns->op->lsp); /* clean up any capture chunks */ diff --git a/src/server/s-central.c b/src/server/s-central.c index 8d790ae..17808f6 100644 --- a/src/server/s-central.c +++ b/src/server/s-central.c @@ -113,21 +113,29 @@ sais_central_clean_abandoned(struct vhd *vhd) } /* - * Check for tasks that should go into CANCELLED + * Check for tasks that have been running too long */ + sqlite3_stmt *sm; lws_snprintf(s, sizeof(s), - "update tasks set state=%d where " - "(state=1 OR state=2) and started < %llu", - SAIES_CANCELLED, (unsigned long long) - (lws_now_secs() - (30 * 60))); - - if (sqlite3_exec(pdb, s, NULL, NULL, &err) != - SQLITE_OK) { - lwsl_err("%s: %s: %s: fail\n", __func__, s, - sqlite3_errmsg(pdb)); - if (err) - sqlite3_free(err); + "SELECT uuid FROM tasks WHERE " + "(state = %d OR state = %d) AND " + "started < %llu", + SAIES_PASSED_TO_BUILDER, + SAIES_BEING_BUILT, (unsigned long long) + (lws_now_secs() - + (vhd->task_abandoned_timeout_mins * 60))); + + if (sqlite3_prepare_v2(pdb, s, -1, &sm, NULL) == SQLITE_OK) { + while (sqlite3_step(sm) == SQLITE_ROW) { + const unsigned char *task_uuid = sqlite3_column_text(sm, 0); + if (task_uuid) { + lwsl_notice("%s: resetting abandoned task %s\n", + __func__, (const char *)task_uuid); + sais_task_reset(vhd, (const char *)task_uuid); + } + } + sqlite3_finalize(sm); } sais_event_db_close(vhd, &pdb); diff --git a/src/server/s-comms.c b/src/server/s-comms.c index dae4680..9c9daf0 100644 --- a/src/server/s-comms.c +++ b/src/server/s-comms.c @@ -401,7 +401,7 @@ callback_ws(struct lws *wsi, enum lws_callback_reasons reason, void *user, *end = &buf[sizeof(buf) - LWS_PRE - 1]; struct pss *pss = (struct pss *)user; sai_http_murl_t mu = SHMUT_NONE; - const char *pvo_resources; + const char *pvo_resources, *num; int n; (void)end; @@ -424,6 +424,11 @@ callback_ws(struct lws *wsi, enum lws_callback_reasons reason, void *user, return -1; } + if (!lws_pvo_get_str(in, "task-abandoned-timeout-mins", &num)) + vhd->task_abandoned_timeout_mins = (unsigned int)atoi(num); + else + vhd->task_abandoned_timeout_mins = 30; + if (lws_pvo_get_str(in, "database", &vhd->sqlite3_path_lhs)) { lwsl_err("%s: database pvo required\n", __func__); return -1; diff --git a/src/server/s-private.h b/src/server/s-private.h index 4981a74..8aa84ca 100644 --- a/src/server/s-private.h +++ b/src/server/s-private.h @@ -205,6 +205,7 @@ struct vhd { lws_usec_t last_check_abandoned_tasks; const char *notification_key; + unsigned int task_abandoned_timeout_mins; unsigned int browser_viewer_count; unsigned int viewers_are_present:1; diff --git a/src/server/s-ws-builder.c b/src/server/s-ws-builder.c index c2dddc1..729a8a8 100644 --- a/src/server/s-ws-builder.c +++ b/src/server/s-ws-builder.c @@ -329,6 +329,9 @@ void sais_builder_disconnected(struct vhd *vhd, struct lws *wsi) { sai_plat_t *cb; + struct lwsac *ac = NULL; + lws_dll2_owner_t o; + int n; /* * A builder's websocket has closed. Find all platforms associated @@ -345,6 +348,48 @@ sais_builder_disconnected(struct vhd *vhd, struct lws *wsi) lwsl_notice("%s: Builder '%s' disconnected\n", __func__, cb->name); + /* + * Check all active events for tasks that were running + * on this builder, and reset them + */ + + n = lws_struct_sq3_deserialize(vhd->server.pdb, + " and (state != 3 and state != 4 and state != 5)", + NULL, lsm_schema_sq3_map_event, &o, &ac, 0, 100); + if (n >= 0 && o.head) { + lws_start_foreach_dll(struct lws_dll2 *, pe, o.head) { + sai_event_t *e = lws_container_of(pe, sai_event_t, list); + sqlite3 *pdb = NULL; + + if (!sais_event_db_ensure_open(vhd, e->uuid, 0, &pdb)) { + sqlite3_stmt *sm; + + lws_snprintf(q, sizeof(q), + "SELECT uuid FROM tasks WHERE " + "(state = %d OR state = %d) AND " + "builder_name = ?", + SAIES_PASSED_TO_BUILDER, + SAIES_BEING_BUILT); + + if (sqlite3_prepare_v2(pdb, q, -1, &sm, NULL) == SQLITE_OK) { + sqlite3_bind_text(sm, 1, cb->name, -1, SQLITE_TRANSIENT); + while (sqlite3_step(sm) == SQLITE_ROW) { + const unsigned char *task_uuid = sqlite3_column_text(sm, 0); + if (task_uuid) { + lwsl_notice("%s: resetting task %s from disconnected builder %s\n", + __func__, (const char *)task_uuid, cb->name); + sais_task_reset(vhd, (const char *)task_uuid); + } + } + sqlite3_finalize(sm); + } + sais_event_db_close(vhd, &pdb); + } + } lws_end_foreach_dll(pe); + + lwsac_free(&ac); + } + lws_snprintf(q, sizeof(q), "UPDATE builders SET online=0 WHERE name='%s'", cb->name); sai_sqlite3_statement(vhd->server.pdb, q, "set builder offline");
Page fetched 0s ago, creation time: 3ms (vhost etag hits: 0%, cache hits: 0%)