| Author | ![[]](/git/avatar/c50933ca2aa61e0fe2c43d46bb6b59cb_avatar) Andy Green <andy@warmcat.com> 2026-09-30 15:34 UTC | | Committer | Andy Green <andy@warmcat.com> 2026-10-04 04:09 UTC | | Tree | c90b5eabc7d804fe65b1bbea2d8c9699d39c1398 ![]() Raw Patch | | | ![]() idle: let builders spend a share of their idle time on idle tasks | idle: let builders spend a share of their idle time on idle tasks
A .sai.json configuration can now ask for "idle": N, which gives each
event N idle tasks ("lanes") per platform for that configuration, besides
its normal task. They're for work with no natural end, like long-running
fuzzing, and never get in the way of real work.
Lanes don't count towards their event's state, so the event's result,
its feed entry and sai-push are unaffected. Only the lanes of the
"host" event for a repo and ref run: the newest event on it whose real
tasks have all finished, pass or fail. So the event stays the one being
worked on in idle time after it completed, until a newer push on the ref
completes. Each time a lane runs is a slice, a new run of the lane
task, and only the last few runs are kept, with their logs and
artifacts.
Builders opt in per platform in their conf with an "idle" object: the
percentage share of the platform's idle time to spend on idle tasks, how
many to run at once, the slice length and a settle time. sai-server
keeps the accounts, by builder platform name so they survive
disconnects: an active period of slices is followed by a rest of period
* (100 - share) / share, paid off only while the platform has no real
tasks pending. A platform with nothing real to do gets offered a lane
when it isn't resting. Running and due lanes count as pending work for
sai-power, when all the platform's builders take idle tasks, so it keeps
builders up for them, and wakes them when their rest is over.
The builder decides idleness for itself as a whole. It declines idle
tasks, with a new IDLE_DECLINED disposition that doesn't mark it busy,
while any platform has real work or had it within the settle time. When
real work is offered to any of its platforms, it stops its idle tasks to
make way, and reports them with a new YIELDED task state, which isn't a
failure. Idle task steps get SAI_IDLE_SECS, the slice length, and are
stopped as yielded if they run well past it.
sai-web shows an event's lanes after its real tasks as "idle: <config>",
leaves them out of the event's counts, and shows them at full strength
and breathing while a slice runs, fading back otherwise, eg, when they
yield to real work.
Old builders never advertise an idle share, so they are never offered
idle tasks, and a new builder's idle config is ignored by an old server.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
diff --git a/READMEs/README-idle.md b/READMEs/README-idle.md
new file mode 100644
index 0000000..4e2fbc6
--- /dev/null
+++ b/READMEs/README-idle.md
@@ -0,0 +1,116 @@
+# Idle tasks
+
+Builders spend a lot of their time with nothing to do. Idle tasks let a
+project use some of that time for work that has no natural end, like
+long-running fuzzing, without it ever getting in the way of real CI work.
+
+## Asking for idle tasks in .sai.json
+
+A configuration asks for idle tasks by giving how many it wants on each
+platform it applies to:
+
+```
+ "fuzz": {
+ "cmake": "./ci/fuzz.sh $SAI_IDLE_SECS",
+ "platforms": "none, linux-debian13/x86_64-amd/gcc",
+ "idle": 2
+ }
+```
+
+Every event still gets the configuration's normal task, which builds and counts
+towards the event's result as usual. Each event also gets, per platform, that
+many idle tasks ("lanes") for the same configuration (at most 16). Idle tasks:
+
+ - never count towards the event's state, so they don't change what the event,
+ its rss / rss.json feed entry, or sai-push, make of it
+
+ - only run on the "host" event of their repo and ref: the newest event on it
+ whose real tasks have all finished, whether they passed or failed. So the
+ event stays the one being worked on in idle time after it completed, until a
+ newer push on the same ref completes and takes over
+
+ - run in "slices": each slice is a new run of the lane, and only the last few
+ runs of a lane are kept, with their logs and artifacts
+
+## Allowing idle tasks on a builder
+
+Idle tasks only go to builder platforms that allow them in the builder conf,
+with an `"idle"` object on the platform:
+
+```
+ {
+ "name": "linux-debian13/x86_64-amd/gcc",
+ "instances": 6,
+ "idle": {
+ "share": 50,
+ "instances": 2,
+ "slice-secs": 900,
+ "settle-secs": 120
+ },
+ "servers": [ "wss://libwebsockets.org:4444/sai/builder" ]
+ }
+```
+
+|member|default|meaning|
+|---|---|---|
+|`share`|0|The percentage of this platform's idle time to spend on idle tasks. 0, or no `"idle"` object, means none|
+|`instances`|1|How many idle tasks the platform may run at once|
+|`slice-secs`|900|How long one slice of an idle task should take (at least 60)|
+|`settle-secs`|120|After real work on any platform of this builder, how long before it counts as idle again|
+
+### Share of idle time
+
+The share is spread out through the idle time, not taken as one block. From
+the first of a platform's slices starting to the last one ending is an active
+period; after it, the platform has to rest for the period times
+`(100 - share) / share` before it can start another. So with a share of 50%
+and 15 minute slices, it alternates 15 minutes of idle tasks with 15 minutes of
+rest. The rest only counts down while the platform has no real tasks pending,
+so it really is a share of the idle time.
+
+sai-server keeps the accounts, so they survive the builder going away. While
+idle tasks are running, or a builder is due to start one, the platform counts as
+having pending tasks for sai-power, the same as real tasks do. So sai-power
+keeps builders up for them, and brings builders that were turned off back up
+when their rest is over. Because sai-power can only turn on a platform, not a
+particular builder, this is only done for a platform whose builders all allow
+idle tasks.
+
+If a builder is paid for by the hour, or you don't want it running around the
+clock, don't give it an `"idle"` object.
+
+### Real work always wins
+
+"Idle" is about the builder as a whole. A builder doesn't take idle tasks while
+any of its platforms has real work, or had it within `settle-secs`: real tasks
+tend to arrive in bursts, and their steps are offered one at a time with gaps
+between.
+
+When real work is offered to any platform of a builder, it stops all the idle
+tasks it's running to make way for it, and they are reported as "yielded".
+That's not a failure, the lane just waits for its next slice. When the builder
+is idle again, idle tasks start again.
+
+## Writing an idle task
+
+An idle task's steps are told the length of their slice in `SAI_IDLE_SECS`, and
+should aim to be finished in that time. If a step goes on much past it, the
+builder stops it and the slice counts as yielded. `SAI_IDLE_SECS` isn't set in
+the normal task, so a script can use it to tell the two apart; eg,
+`./ci/fuzz.sh $SAI_IDLE_SECS` expands to plain `./ci/fuzz.sh` in the normal
+task. The whole step, including any build, has to fit in the slice, so a
+script that divides its time between several things should divide
+`SAI_IDLE_SECS` between them.
+
+Anything the task wants to keep between slices, like a fuzzing corpus, has to be
+kept somewhere under `$HOME` on the builder by the task itself for now; the job
+dir is reused by the lane's next slice on the same builder, but not preserved
+across builders.
+
+## In the web UI
+
+An event's idle tasks are shown after its real tasks, in their own group per
+configuration named "idle: <configuration>". While a slice is running, the
+lane is shown at full strength and gently "breathing"; the rest of the time,
+including when a slice yields to real work, it fades back. Idle tasks aren't
+included in the event's task counts or progress bar.
diff --git a/READMEs/README-sai-json.md b/READMEs/README-sai-json.md
index bdddc25..323e9aa 100644
--- a/READMEs/README-sai-json.md
+++ b/READMEs/README-sai-json.md
@@ -93,4 +93,18 @@ or to disable all default platforms and just use the specified ones:
You can also give a comma-separated list of build artifacts, these are
arbitrary binary files which will be uploaded to sai-server and made available
-for download over https.
\ No newline at end of file
+for download over https.
+
+#### idle
+
+A configuration can also ask for idle tasks, eg for long-running fuzzing, with
+the number of them to create on each platform it applies to:
+
+```
+ "idle": 4
+```
+
+Besides the configuration's normal task, each event then has that many idle
+tasks ("lanes") for it on each platform. They do not count towards the
+event's result, and only run in time builders would otherwise spend idle,
+when the builder conf allows it. See [README-idle.md](README-idle.md).
diff --git a/assets/index.html b/assets/index.html
index f0770bd..3424d93 100644
--- a/assets/index.html
+++ b/assets/index.html
@@ -2,10 +2,10 @@
<html lang="en">
<head>
<meta charset=utf-8 http-equiv="Content-Language" content="en"/>
- <link rel="stylesheet" type="text/css" href="sai.css?v=15"/>
+ <link rel="stylesheet" type="text/css" href="sai.css?v=16"/>
<link rel="icon" href="sai-icon.svg" sizes="any" type="image/svg+xml"/>
<link rel="alternate" type="application/rss+xml" title="Sai build events" href="rss.xml"/>
- <script type='text/javascript' src='sai.js?v=14'></script>
+ <script type='text/javascript' src='sai.js?v=15'></script>
<script type='text/javascript' src='terminal-core.js'></script>
<script type='text/javascript' src='sai-terminal.js'></script>
<script type='text/javascript' src='lws-login.js'></script>
diff --git a/assets/sai.css b/assets/sai.css
index 8841fc0..cc00c99 100644
--- a/assets/sai.css
+++ b/assets/sai.css
@@ -223,6 +223,7 @@ div.taskinfo {
#sai_task_logs_container div.taskinfo.taskstate5, div.taskinfo.taskstate5 { background: #808080; border-color: #808080; transition: background-color 500ms linear; }
#sai_task_logs_container div.taskinfo.taskstate6, div.taskinfo.taskstate6 { background: #df3030; border-color: #df3030; transition: background-color 500ms linear; }
#sai_task_logs_container div.taskinfo.taskstate10, div.taskinfo.taskstate10 { background: #ffdc00; border-color: #ffdc00; transition: background-color 500ms linear; }
+#sai_task_logs_container div.taskinfo.taskstate11, div.taskinfo.taskstate11 { background: #b0c4de; border-color: #b0c4de; transition: background-color 500ms linear; }
#sai_task_logs_container div.taskinfo.taskstate6, #sai_task_logs_container div.taskinfo.taskstate6 span.ti1, #sai_task_logs_container div.taskinfo.taskstate6 span.ti5, #sai_task_logs_container div.taskinfo.taskstate6 div.sai_arts,
div.taskinfo.taskstate6, div.taskinfo.taskstate6 span.ti1, div.taskinfo.taskstate6 span.ti5, div.taskinfo.taskstate6 div.sai_arts {
@@ -403,6 +404,53 @@ img.bico {
transition: background-color 500ms linear;
}
+/* an idle task's slice that the builder stopped, eg, for real work */
+.taskstate11 {
+ background:#b0c4de;
+ color:#202830;
+ opacity:1;
+ transition: background-color 500ms linear;
+}
+
+/*
+ * Idle tasks ("lanes") run in time the builders would otherwise spend idle.
+ * While a slice runs they are shown at full strength, gently breathing; the
+ * rest of the time, including when real work makes them yield, they fade back.
+ */
+div.taskstate.idle-lane, tr.tt-row.idle-lane {
+ opacity: 0.35;
+ transition: opacity 1.5s ease-in-out, background-color 500ms linear;
+}
+
+div.taskstate.idle-lane.taskstate1, div.taskstate.idle-lane.taskstate2,
+div.taskstate.idle-lane.taskstate9,
+tr.tt-row.idle-lane.taskstate1, tr.tt-row.idle-lane.taskstate2,
+tr.tt-row.idle-lane.taskstate9 {
+ opacity: 1;
+}
+
+div.taskstate.idle-lane.taskstate1, div.taskstate.idle-lane.taskstate2,
+div.taskstate.idle-lane.taskstate9 {
+ background: #9fd8b8;
+ animation: idle-breathe 4s ease-in-out infinite;
+}
+
+@keyframes idle-breathe {
+ 0%, 100% {
+ filter: brightness(1);
+ }
+ 50% {
+ filter: brightness(1.25);
+ }
+}
+
+@media (prefers-reduced-motion: reduce) {
+ div.taskstate.idle-lane.taskstate1, div.taskstate.idle-lane.taskstate2,
+ div.taskstate.idle-lane.taskstate9 {
+ animation: none;
+ }
+}
+
div.stats_hidden {
display: none;
}
diff --git a/assets/sai.js b/assets/sai.js
index abed574..94eed06 100644
--- a/assets/sai.js
+++ b/assets/sai.js
@@ -1342,6 +1342,8 @@ function summarize_build_situation(event_uuid)
for (var uid in run_max) {
var t = run_max[uid];
+ if (t.idle)
+ continue;
total++;
switch (t.state) {
case 0: pending++; break;
@@ -1358,28 +1360,28 @@ function summarize_build_situation(event_uuid)
if (!roo)
return { text: "" };
- var same = roo.querySelectorAll(".taskstate:not(.run-decal)");
+ var same = roo.querySelectorAll(".taskstate:not(.run-decal):not(.idle-lane)");
if (same)
total = same.length;
- same = roo.querySelectorAll(".taskstate0:not(.run-decal)");
+ same = roo.querySelectorAll(".taskstate0:not(.run-decal):not(.idle-lane)");
if (same)
pending = same.length;
- same = roo.querySelectorAll(".taskstate1:not(.run-decal)");
+ same = roo.querySelectorAll(".taskstate1:not(.run-decal):not(.idle-lane)");
if (same)
ongoing += same.length;
- same = roo.querySelectorAll(".taskstate2:not(.run-decal)");
+ same = roo.querySelectorAll(".taskstate2:not(.run-decal):not(.idle-lane)");
if (same)
ongoing += same.length;
- same = roo.querySelectorAll(".taskstate3:not(.run-decal)");
+ same = roo.querySelectorAll(".taskstate3:not(.run-decal):not(.idle-lane)");
if (same)
good = same.length;
- same = roo.querySelectorAll(".taskstate4:not(.run-decal)");
+ same = roo.querySelectorAll(".taskstate4:not(.run-decal):not(.idle-lane)");
if (same)
bad += same.length;
- same = roo.querySelectorAll(".taskstate5:not(.run-decal)");
+ same = roo.querySelectorAll(".taskstate5:not(.run-decal):not(.idle-lane)");
if (same)
bad += same.length;
- same = roo.querySelectorAll(".taskstate6:not(.run-decal)");
+ same = roo.querySelectorAll(".taskstate6:not(.run-decal):not(.idle-lane)");
if (same)
ongoing += same.length;
}
@@ -1947,6 +1949,7 @@ function sai_tt_state_rank(state)
case 3: return 6; /* passed */
case 4: return 7; /* failed */
case 5: return 8; /* cancelled */
+ case 11: return 8; /* yielded (idle task) */
case 7: return 9; /* deleted */
}
return 10;
@@ -1999,6 +2002,23 @@ function sai_tt_mark_header(tab)
});
}
+/*
+ * Idle tasks ("lanes") only run in time builders would otherwise spend idle,
+ * and aren't part of their event's result. They're shown after the real
+ * tasks, in their own group per task name.
+ */
+function sai_task_group_name(t)
+{
+ return (t.idle ? "idle: " : "") + t.taskname;
+}
+
+/* the event's tasks, real ones first, then the idle ones */
+function sai_tasks_real_then_idle(tasks)
+{
+ return tasks.filter(function(t) { return !t.idle; }).concat(
+ tasks.filter(function(t) { return !!t.idle; }));
+}
+
/* the latest run of each task uuid, keyed by uuid */
function sai_tt_latest_runs(tasks)
{
@@ -2097,6 +2117,7 @@ function sai_tt_step_html(t)
case 7: return "deleted";
case 8: return "not ready";
case 10: return "paused" + at;
+ case 11: return "yielded";
}
return "state " + t.state;
@@ -2111,7 +2132,7 @@ function sai_tt_row_keys(t)
return {
run: typeof t.run !== 'undefined' ? t.run : 0,
state: t.state,
- name: t.taskname,
+ name: sai_task_group_name(t),
plat: t.platform,
started: t.started ? t.started : 0,
dur: t.duration ? t.duration : 0,
@@ -2131,6 +2152,7 @@ function sai_tt_row_set_keys(tr, t)
function sai_tt_row_html(t, e, now_ut)
{
var s = "<tr id=\"tt_" + san(t.uuid) + "\" class=\"tt-row taskstate" + t.state +
+ (t.idle ? " idle-lane" : "") +
(t.uuid === selected_task_uuid ? " selected" : "") + "\"" +
" data-task-uuid=\"" + san(t.uuid) + "\"" +
" data-event-uuid=\"" + san(e.uuid) + "\"" +
@@ -2142,7 +2164,7 @@ function sai_tt_row_html(t, e, now_ut)
s += " data-" + n + "=\"" + san(k[n]) + "\"";
s += ">";
- s += "<td class=\"tt-name\">" + san(t.taskname) + "</td>";
+ s += "<td class=\"tt-name\">" + san(sai_task_group_name(t)) + "</td>";
s += "<td class=\"tt-plat\">" + sai_plat_icon(t.platform, 0) + " " +
san(t.platform) + "</td>";
s += "<td class=\"tt-started\">" +
@@ -2554,13 +2576,14 @@ function render_selected_event_tasks(o) {
var ctn = "";
var s1 = "";
- for (var q = 0; q < o.t.length; q++) {
- var t = o.t[q];
+ var ordered = sai_tasks_real_then_idle(o.t);
+ for (var q = 0; q < ordered.length; q++) {
+ var t = ordered[q];
if (t !== run_max[t.uuid])
continue;
- if (t.taskname !== ctn) {
+ if (sai_task_group_name(t) !== ctn) {
if (ctn !== "") {
s += "<div class=\"ib\"><table class=\"nomar\">" +
"<tr><td class=\"tn\">" + hsanitize(ctn) +
@@ -2568,10 +2591,11 @@ function render_selected_event_tasks(o) {
"</td></tr></table></div>";
s1 = "";
}
- ctn = t.taskname;
+ ctn = sai_task_group_name(t);
}
s1 += "<div id=\"taskstate_" + t.uuid + "\" class=\"taskstate taskstate" + t.state +
+ (t.idle ? " idle-lane" : "") +
(run_list[t.uuid].length > 1 ? " has_runs" : "") +
"\" data-task-uuid=\"" + san(t.uuid) +
"\" data-event-uuid=\"" + san(e.uuid) + "\" data-platform=\"" + san(t.platform) +
@@ -2817,24 +2841,26 @@ function sai_event_render(o, now_ut, reset_all_icon)
run_list[uid].sort(function(a, b) { var ar = typeof a.run !== 'undefined' ? a.run : 0; var br = typeof b.run !== 'undefined' ? b.run : 0; return ar - br; });
}
- for (q = 0; q < o.t.length; q++) {
- var t = o.t[q];
+ var ordered = sai_tasks_real_then_idle(o.t);
+ for (q = 0; q < ordered.length; q++) {
+ var t = ordered[q];
if (t !== run_max[t.uuid])
continue;
- if (t.taskname !== ctn) {
+ if (sai_task_group_name(t) !== ctn) {
if (ctn !== "") {
s += "<div class=\"ib\"><table class=\"nomar\">" +
- "<tr><td class=\"tn\">" + ctn +
+ "<tr><td class=\"tn\">" + hsanitize(ctn) +
"</td><td class=\"keepline\">" + s1 +
"</td></tr></table></div>";
s1 = "";
}
- ctn = t.taskname;
+ ctn = sai_task_group_name(t);
}
s1 += "<div id=\"taskstate_" + t.uuid + "\" class=\"taskstate taskstate" + t.state +
+ (t.idle ? " idle-lane" : "") +
(run_list[t.uuid].length > 1 ? " has_runs" : "") +
"\" data-event-uuid=\"" + san(e.uuid) + "\" data-platform=\"" + san(t.platform) +
"\" data-rebuildable=\"" + t.rebuildable + "\">";
@@ -2856,7 +2882,7 @@ function sai_event_render(o, now_ut, reset_all_icon)
if (ctn !== "") {
s += "<div class=\"ib\"><table class=\"nomar\">" +
- "<tr><td class=\"tn\">" + ctn +
+ "<tr><td class=\"tn\">" + hsanitize(ctn) +
"<td class=\"keepline\">" + s1 +
"</td></tr></table></div>";
}
@@ -2928,7 +2954,10 @@ function refresh_state(t)
tsi.classList.remove("taskstate5");
tsi.classList.remove("taskstate6");
tsi.classList.remove("taskstate7");
+ tsi.classList.remove("taskstate8");
+ tsi.classList.remove("taskstate9");
tsi.classList.remove("taskstate10");
+ tsi.classList.remove("taskstate11");
tsi.classList.add("taskstate" + task_state);
var toRemove = [];
@@ -2939,7 +2968,12 @@ function refresh_state(t)
}
toRemove.forEach(function(cls) { tsi.classList.remove(cls); });
- if (task_state === 1 || task_state === 2 || task_state === 6) {
+ /*
+ * An idle task's slice isn't progress towards anything, it
+ * shows it's running differently (see .idle-lane in sai.css)
+ */
+ if (!tsi.classList.contains("idle-lane") &&
+ (task_state === 1 || task_state === 2 || task_state === 6)) {
var total = typeof t.total_steps !== 'undefined' ? t.total_steps : t.build_step_count;
if (typeof t.build_step !== 'undefined' && typeof total !== 'undefined' && total >= 0) {
var pct = Math.round((t.build_step + 1) * 100 / (total + 2));
@@ -5374,7 +5408,7 @@ window.addEventListener("load", function() {
}
];
- const isFinalState = ["taskstate3", "taskstate4", "taskstate5", "taskstate7"].some(s => taskDiv.classList.contains(s));
+ const isFinalState = ["taskstate3", "taskstate4", "taskstate5", "taskstate7", "taskstate11"].some(s => taskDiv.classList.contains(s));
if (!isFinalState) {
if (taskDiv.classList.contains("taskstate10")) {
diff --git a/etc-sai-EXAMPLE/builder/conf b/etc-sai-EXAMPLE/builder/conf
index 5a95e47..38d1461 100644
--- a/etc-sai-EXAMPLE/builder/conf
+++ b/etc-sai-EXAMPLE/builder/conf
@@ -21,6 +21,17 @@
{
"name": "linux-ubuntu-1804/x86_64-amd/gcc",
"instances": 3,
+
+ # optional: spend up to half this platform's idle time on
+ # idle tasks (eg, fuzzing), two at a time, in 15 minute
+ # slices. See READMEs/README-idle.md
+ "idle": {
+ "share": 50,
+ "instances": 2,
+ "slice-secs": 900,
+ "settle-secs": 120
+ },
+
"servers": [ "wss://libwebsockets.org:4444/sai/builder" ]
},
{
diff --git a/src/builder/b-conf.c b/src/builder/b-conf.c
index 6701c28..7b03fcb 100644
--- a/src/builder/b-conf.c
+++ b/src/builder/b-conf.c
@@ -81,6 +81,15 @@ static const char * const paths[] = {
"platforms[].env[]",
"platforms[].servers",
"platforms[].job-limit",
+ /*
+ * Listed so the "idle" object itself matches this, rather than the
+ * "platforms[]" it's in: that would start another platform
+ */
+ "platforms[].idle",
+ "platforms[].idle.share",
+ "platforms[].idle.instances",
+ "platforms[].idle.slice-secs",
+ "platforms[].idle.settle-secs",
"platforms[]",
};
@@ -91,6 +100,11 @@ enum enum_paths {
LEJPM_PLATFORMS_ENV,
LEJPM_PLATFORMS_SERVERS,
LEJPM_PLATFORMS_JOB_LIMIT,
+ LEJPM_PLATFORMS_IDLE,
+ LEJPM_PLATFORMS_IDLE_SHARE,
+ LEJPM_PLATFORMS_IDLE_INSTANCES,
+ LEJPM_PLATFORMS_IDLE_SLICE_SECS,
+ LEJPM_PLATFORMS_IDLE_SETTLE_SECS,
LEJPM_PLATFORMS,
};
@@ -128,6 +142,9 @@ saib_conf_cb(struct lejp_ctx *ctx, char reason)
sizeof(a->sai_plat->lws_hash));
a->sai_plat->job_limit = 0;
+ a->sai_plat->idle_instances = 1;
+ a->sai_plat->idle_slice_secs = SAIB_IDLE_DEF_SLICE_SECS;
+ a->sai_plat->idle_settle_secs = SAIB_IDLE_DEF_SETTLE_SECS;
lws_dll2_add_tail(&a->sai_plat->sai_plat_list,
&a->builder->sai_plat_owner);
@@ -150,6 +167,29 @@ saib_conf_cb(struct lejp_ctx *ctx, char reason)
lwsl_err("%s: LEJPM_PLATFORMS_JOB_LIMIT %u\n", __func__, a->sai_plat->job_limit);
}
+ /*
+ * What the platform should do with its idle time, see README-idle.md
+ */
+
+ switch (ctx->path_match - 1) {
+ case LEJPM_PLATFORMS_IDLE_SHARE:
+ n = atoi(ctx->buf);
+ a->sai_plat->idle_share = n < 0 ? 0 : (n > 100 ? 100 : (unsigned int)n);
+ break;
+ case LEJPM_PLATFORMS_IDLE_INSTANCES:
+ n = atoi(ctx->buf);
+ a->sai_plat->idle_instances = n < 1 ? 1 : (unsigned int)n;
+ break;
+ case LEJPM_PLATFORMS_IDLE_SLICE_SECS:
+ n = atoi(ctx->buf);
+ a->sai_plat->idle_slice_secs = n < 60 ? 60 : (unsigned int)n;
+ break;
+ case LEJPM_PLATFORMS_IDLE_SETTLE_SECS:
+ n = atoi(ctx->buf);
+ a->sai_plat->idle_settle_secs = n < 0 ? 0 : (unsigned int)n;
+ break;
+ }
+
if (reason != LEJPCB_VAL_STR_END)
return 0;
diff --git a/src/builder/b-nspawn.c b/src/builder/b-nspawn.c
index da79338..4210da5 100644
--- a/src/builder/b-nspawn.c
+++ b/src/builder/b-nspawn.c
@@ -367,8 +367,33 @@ sai_lsp_reap_cb(void *opaque, const lws_spawn_resource_us_t *res, siginfo_t *si,
goto fail;
}
+ if (ns->idle_yield) {
+ /* we stopped this idle task to make way for real work */
+ saib_task_logf(ns->spm, ns, NULL,
+ "Idle task stopped to make way for real work");
+ exit_code = -1;
+ ns->retcode = SAISPRF_TERMINATED;
+ ns->retcode_set = 1;
+ goto fail;
+ }
+
#if !defined(WIN32)
+ if ((we_killed_him & 1) && ns->task->idle) {
+ /*
+ * An idle task that ran over its slice: it's not a failure,
+ * the slice is just over
+ */
+ saib_task_logf(ns->spm, ns, NULL,
+ "Idle task step ran past its %us slice, stopping it",
+ ns->sp->idle_slice_secs);
+ ns->idle_yield = 1;
+ exit_code = -1;
+ ns->retcode = SAISPRF_TERMINATED;
+ ns->retcode_set = 1;
+ goto fail;
+ }
+
if (we_killed_him & 1) {
lwsl_notice("%s: Process TIMED OUT by Sai\n", __func__);
exit_code = -1;
@@ -558,9 +583,11 @@ skip:
return;
fail:
- n = lws_snprintf(s, sizeof(s), "Build step %d FAILED, exit code: %d\n",
- ns->task->build_step + 1, exit_code);
- saib_log_chunk_create(ns, s, (size_t)n, 3);
+ if (!ns->idle_yield) {
+ n = lws_snprintf(s, sizeof(s), "Build step %d FAILED, exit code: %d\n",
+ ns->task->build_step + 1, exit_code);
+ saib_log_chunk_create(ns, s, (size_t)n, 3);
+ }
saib_task_grace(ns);
saib_set_ns_state(ns, NSSTATE_FAILED);
@@ -585,6 +612,7 @@ static const char * const runscript_win_first =
"set SAI_LOGPROXY=%s\n"
"set SAI_LOGPROXY_TTY0=%s\n"
"set SAI_LOGPROXY_TTY1=%s\n"
+ "%s"
"set HOME=%s\n"
"set CI=true\n"
"set BUILDKIT_PROGRESS=plain\n"
@@ -601,6 +629,7 @@ static const char * const runscript_win_next =
"set SAI_LOGPROXY=%s\n"
"set SAI_LOGPROXY_TTY0=%s\n"
"set SAI_LOGPROXY_TTY1=%s\n"
+ "%s"
"set HOME=%s\n"
"set CI=true\n"
"set BUILDKIT_PROGRESS=plain\n"
@@ -629,6 +658,7 @@ static const char * const runscript_first =
"export SAI_LOGPROXY=%s\n"
"export SAI_LOGPROXY_TTY0=%s\n"
"export SAI_LOGPROXY_TTY1=%s\n"
+ "%s"
"export CI=true\n"
"export BUILDKIT_PROGRESS=plain\n"
"set -e\n"
@@ -657,6 +687,7 @@ static const char * const runscript_next =
"export SAI_LOGPROXY=%s\n"
"export SAI_LOGPROXY_TTY0=%s\n"
"export SAI_LOGPROXY_TTY1=%s\n"
+ "%s"
"export CI=true\n"
"export BUILDKIT_PROGRESS=plain\n"
"set -e\n"
@@ -684,6 +715,7 @@ static const char * const runscript_build =
"export SAI_LOGPROXY=%s\n"
"export SAI_LOGPROXY_TTY0=%s\n"
"export SAI_LOGPROXY_TTY1=%s\n"
+ "%s"
"export CI=true\n"
"export BUILDKIT_PROGRESS=plain\n"
"set -e\n"
@@ -715,8 +747,9 @@ saib_spawn_script(struct sai_nspawn *ns)
NULL
};
#endif
- char one_step[4096];
+ char one_step[4096], idle_env[64];
char st[2048];
+ unsigned int timeout_secs;
int fd, n;
#if defined(__linux__)
int in_cgroup = 1;
@@ -765,13 +798,35 @@ saib_spawn_script(struct sai_nspawn *ns)
}
builder.wrap14 = (builder.wrap14 + 8) & 0x3fff;
+ /*
+ * An idle task is told how long its slice is, so it can aim to fit
+ * whatever it does in the time. We stop it anyway if it goes on much
+ * longer than that.
+ */
+ idle_env[0] = '\0';
+ timeout_secs = builder.build_timeout_secs;
+ if (ns->task->idle) {
+ lws_snprintf(idle_env, sizeof(idle_env),
+#if defined(WIN32)
+ "set SAI_IDLE_SECS=%u\n",
+#else
+ "export SAI_IDLE_SECS=%u\n",
+#endif
+ ns->sp->idle_slice_secs);
+ if (ns->sp->idle_slice_secs + SAIB_IDLE_SLICE_GRACE_SECS <
+ timeout_secs)
+ timeout_secs = ns->sp->idle_slice_secs +
+ SAIB_IDLE_SLICE_GRACE_SECS;
+ }
+
#if defined(WIN32)
n = lws_snprintf(st, sizeof(st),
ns->task->build_step ? runscript_win_next : runscript_win_first,
ns->instance_ordinal + 1, builder.wrap14,
ns->task->parallel ? ns->task->parallel : 1,
respath, ns->slp_control.sockpath,
- ns->slp[0].sockpath, ns->slp[1].sockpath, builder.home,
+ ns->slp[0].sockpath, ns->slp[1].sockpath, idle_env,
+ builder.home,
ns->inp, ns->task->build_step > 1 ? "\\src" : "",
one_step);
#else
@@ -794,7 +849,7 @@ saib_spawn_script(struct sai_nspawn *ns)
builder.wrap14,
ns->task->parallel ? ns->task->parallel : 1,
respath, ns->slp_control.sockpath,
- ns->slp[0].sockpath, ns->slp[1].sockpath,
+ ns->slp[0].sockpath, ns->slp[1].sockpath, idle_env,
builder.home, one_step);
#endif
@@ -835,7 +890,7 @@ saib_spawn_script(struct sai_nspawn *ns)
info.exec_array = cmd;
info.protocol_name = "sai-stdxxx";
info.max_log_lines = 10000;
- info.timeout_us = (lws_usec_t)((uint64_t)builder.build_timeout_secs * LWS_US_PER_SEC);
+ info.timeout_us = (lws_usec_t)((uint64_t)timeout_secs * LWS_US_PER_SEC);
info.reap_cb = sai_lsp_reap_cb;
#if defined(WIN32)
info.pty_mode = 0;
diff --git a/src/builder/b-private.h b/src/builder/b-private.h
index 643aca0..0d774ac 100644
--- a/src/builder/b-private.h
+++ b/src/builder/b-private.h
@@ -137,6 +137,17 @@ struct saib_opaque_spawn {
#define SAI_CLOCK_WAIT_MAX_SECS (5 * 60)
+/*
+ * Idle tasks, when the platform's conf "idle" object doesn't say: how long one
+ * slice of an idle task runs for, and how long after the last real work we
+ * wait before we count as idle again (tasks tend to arrive in bursts, and the
+ * server offers a task's steps one at a time with gaps between)
+ */
+#define SAIB_IDLE_DEF_SLICE_SECS (15 * 60)
+#define SAIB_IDLE_DEF_SETTLE_SECS 120
+/* how much longer than its slice an idle task step may run before we stop it */
+#define SAIB_IDLE_SLICE_GRACE_SECS 120
+
struct saib_ws_pss;
enum nsstate {
@@ -290,6 +301,12 @@ struct sai_builder {
*/
lws_usec_t last_log_us;
+ /*
+ * When we were last offered, or last finished, a step of a real task
+ * (not an idle one), for the idle task settle time
+ */
+ lws_usec_t last_real_us;
+
uint16_t wrap14;
unsigned int build_timeout_secs;
diff --git a/src/builder/b-task.c b/src/builder/b-task.c
index 8b961dd..75d264e 100644
--- a/src/builder/b-task.c
+++ b/src/builder/b-task.c
@@ -212,10 +212,40 @@ saib_refusal_loggable(const char *task_uuid)
return 1;
}
+/*
+ * What the idle slices we are stopping still have reserved: it's as good as
+ * free for the real work that we are stopping them for
+ */
+
+static void
+saib_idletask_yielding_res(uint64_t *ram_kib, uint64_t *disk_kib)
+{
+ *ram_kib = *disk_kib = 0;
+
+ lws_start_foreach_dll(struct lws_dll2 *, mp, builder.sai_plat_owner.head) {
+ struct sai_plat *xsp = lws_container_of(mp, struct sai_plat,
+ sai_plat_list);
+
+ lws_start_foreach_dll(struct lws_dll2 *, p, xsp->nspawn_owner.head) {
+ struct sai_nspawn *xns = lws_container_of(p,
+ struct sai_nspawn, list);
+
+ if (xns->idle_yield) {
+ *ram_kib += xns->res_ram_kib;
+ *disk_kib += xns->res_disk_kib;
+ }
+
+ } lws_end_foreach_dll(p);
+
+ } lws_end_foreach_dll(mp);
+}
+
static int
saib_can_accept_task(struct sai_plat_server *spm, sai_task_t *task,
sai_plat_t *sp)
{
+ uint64_t yield_ram_kib, yield_disk_kib, ram_reserved_kib,
+ disk_reserved_kib;
unsigned int tc = sp->job_limit ? sp->job_limit : 6u;
#if 0
unsigned int free_ram = saib_get_free_ram_kib();
@@ -259,11 +289,17 @@ saib_can_accept_task(struct sai_plat_server *spm, sai_task_t *task,
}
}
+ saib_idletask_yielding_res(&yield_ram_kib, &yield_disk_kib);
+ ram_reserved_kib = builder.ram_reserved_kib > yield_ram_kib ?
+ builder.ram_reserved_kib - yield_ram_kib : 0;
+ disk_reserved_kib = builder.disk_reserved_kib > yield_disk_kib ?
+ builder.disk_reserved_kib - yield_disk_kib : 0;
+
{
uint64_t budget = (builder.ram_limit_kib * 4) / 3;
- budget = builder.ram_reserved_kib > budget ? 0 :
- budget - builder.ram_reserved_kib;
+ budget = ram_reserved_kib > budget ? 0 :
+ budget - ram_reserved_kib;
if (budget < task->est_peak_mem_kib) {
if (saib_refusal_loggable(task->uuid))
@@ -281,7 +317,7 @@ saib_can_accept_task(struct sai_plat_server *spm, sai_task_t *task,
{
uint64_t free_disk = saib_get_free_disk_kib(builder.home);
uint64_t needed_disk = (uint64_t)task->est_disk_kib +
- builder.disk_reserved_kib;
+ disk_reserved_kib;
char vn[16];
/* leave 12.5% of free space as a safety margin */
@@ -296,7 +332,7 @@ saib_can_accept_task(struct sai_plat_server *spm, sai_task_t *task,
"free on %s", sp->name,
(unsigned long long)(needed_disk / 1024),
(unsigned int)(task->est_disk_kib / 1024),
- (unsigned long long)(builder.disk_reserved_kib / 1024),
+ (unsigned long long)(disk_reserved_kib / 1024),
(unsigned long long)(free_disk / 1024),
builder.home);
@@ -319,9 +355,10 @@ saib_can_accept_task(struct sai_plat_server *spm, sai_task_t *task,
lws_start_foreach_dll(struct lws_dll2 *, p, sp->nspawn_owner.head) {
struct sai_nspawn *xns = lws_container_of(p, struct sai_nspawn, list);
- if (xns->state == NSSTATE_INIT ||
- xns->state == NSSTATE_MOUNTING ||
- xns->state == NSSTATE_EXECUTING_STEPS)
+ if (!xns->idle_yield && /* going away to make room */
+ (xns->state == NSSTATE_INIT ||
+ xns->state == NSSTATE_MOUNTING ||
+ xns->state == NSSTATE_EXECUTING_STEPS))
executing++;
} lws_end_foreach_dll(p);
@@ -334,6 +371,112 @@ saib_can_accept_task(struct sai_plat_server *spm, sai_task_t *task,
return 0; /* acceptable */
}
+/*
+ * Idle tasks
+ *
+ * The server only offers us idle tasks for a platform when it has nothing real
+ * for that platform, and when the platform's conf share of idle time allows.
+ * But "idle" is about this whole builder: we don't take idle work while any of
+ * our platforms has real work, or had it within the settle time, and when real
+ * work is offered to any of them, we stop all our idle slices to make way.
+ */
+
+static int
+saib_idletask_should_decline(sai_plat_t *sp)
+{
+ lws_usec_t now = lws_now_usecs();
+ unsigned int ours = 0;
+
+ if (!sp->idle_share) {
+ lwsl_notice("%s: %s: no idle share\n", __func__, sp->name);
+ return 1;
+ }
+
+ /* these modes are for a builder that exists for one real thing */
+ if (builder.one_shot_active || builder.event_affinity_active)
+ return 1;
+
+ if (builder.last_real_us &&
+ now - builder.last_real_us <
+ (lws_usec_t)sp->idle_settle_secs * LWS_US_PER_SEC) {
+ lwsl_notice("%s: %s: real work too recently\n", __func__,
+ sp->name);
+ return 1;
+ }
+
+ lws_start_foreach_dll(struct lws_dll2 *, mp, builder.sai_plat_owner.head) {
+ struct sai_plat *xsp = lws_container_of(mp, struct sai_plat,
+ sai_plat_list);
+
+ lws_start_foreach_dll(struct lws_dll2 *, p, xsp->nspawn_owner.head) {
+ struct sai_nspawn *xns = lws_container_of(p,
+ struct sai_nspawn, list);
+
+ if (!xns->task)
+ continue;
+
+ if (!xns->task->idle) {
+ lwsl_notice("%s: %s: has real work\n",
+ __func__, sp->name);
+ return 1;
+ }
+
+ if (xsp == sp && !xns->idle_yield)
+ ours++;
+
+ } lws_end_foreach_dll(p);
+
+ } lws_end_foreach_dll(mp);
+
+ if (ours >= sp->idle_instances) {
+ lwsl_notice("%s: %s: already running %u idle tasks\n",
+ __func__, sp->name, ours);
+ return 1;
+ }
+
+ return 0;
+}
+
+/* real work is coming... stop every idle slice we have to make way for it */
+
+static void
+saib_idletask_yield_all(void)
+{
+ lws_start_foreach_dll(struct lws_dll2 *, mp, builder.sai_plat_owner.head) {
+ struct sai_plat *xsp = lws_container_of(mp, struct sai_plat,
+ sai_plat_list);
+
+ lws_start_foreach_dll(struct lws_dll2 *, p, xsp->nspawn_owner.head) {
+ struct sai_nspawn *xns = lws_container_of(p,
+ struct sai_nspawn, list);
+
+ if (!xns->task || !xns->task->idle || xns->idle_yield)
+ continue;
+
+ lwsl_notice("%s: yielding idle task %s\n", __func__,
+ xns->task->uuid);
+
+ xns->idle_yield = 1;
+
+ if (!xns->op || !xns->op->lsp)
+ /*
+ * Nothing running to stop, eg, between
+ * spawning and uploading; it reports as
+ * yielded when it's destroyed
+ */
+ continue;
+
+ xns->user_cancel = 1;
+ xns->term_budget = 5;
+ lws_sul_schedule(builder.context, 0,
+ &xns->sul_task_cancel,
+ saib_sul_task_cancel, 1);
+
+ } lws_end_foreach_dll(p);
+
+ } lws_end_foreach_dll(mp);
+}
+
#if !defined(WIN32)
static char csep = '/';
#else
@@ -532,9 +675,21 @@ saib_task_destroy(struct sai_nspawn *ns)
}
if (ns->task) {
+ unsigned int ecode = (unsigned int)ns->retcode;
+
+ if (ns->idle_yield)
+ /*
+ * We stopped this idle slice, it's not a failure
+ * whatever the process had to say about it
+ */
+ ecode = SAISPRF_TERMINATED | SAISPRF_YIELDED;
+
+ if (!ns->task->idle)
+ /* real work idles us only after the settle time */
+ builder.last_real_us = lws_now_usecs();
+
saib_queue_task_status_update(ns->sp, ns->spm, ns->task->uuid,
- (unsigned int)ns->retcode,
- SAI_TASK_REASON_DESTROYED);
+ ecode, SAI_TASK_REASON_DESTROYED);
/*
* Only ever give back what this ns took. Giving back the
@@ -982,7 +1137,10 @@ saib_sul_task_cancel(struct lws_sorted_usec_list *sul)
if (ns->user_killed)
n = lws_snprintf(s, sizeof(s), "\xe2\x96\xa0 >saib> Build was manually killed\n");
else
- n = lws_snprintf(s, sizeof(s), ">saib> Cancelling...\n");
+ if (ns->idle_yield)
+ n = lws_snprintf(s, sizeof(s), ">saib> Stopping idle task...\n");
+ else
+ n = lws_snprintf(s, sizeof(s), ">saib> Cancelling...\n");
saib_log_chunk_create(ns, s, (size_t)n, 3);
lws_spawn_piped_kill_child_process(ns->op->lsp);
@@ -1089,7 +1247,22 @@ saib_consider_allocating_task(struct sai_plat_server *spm, lws_struct_args_t *a,
* We're not already running it, let's consider accepting it
*/
+ if (task->idle) {
+ if (saib_idletask_should_decline(sp))
+ goto idle_decline;
+ } else {
+ /*
+ * Real work has been offered, so we're not idle: make way for
+ * it by stopping any idle tasks, whatever platform they're on
+ */
+ builder.last_real_us = lws_now_usecs();
+ saib_idletask_yield_all();
+ }
+
if (saib_can_accept_task(spm, task, sp)) {
+ if (task->idle)
+ goto idle_decline;
+
lwsl_warn("%s: builder rejects offered task\n", __func__);
if (saib_queue_task_status_update(sp, spm, task->uuid, 0,
SAI_TASK_REASON_BUSY)) {
@@ -1396,6 +1569,18 @@ saib_consider_allocating_task(struct sai_plat_server *spm, lws_struct_args_t *a,
return 0;
+idle_decline:
+ /*
+ * Unlike BUSY, this doesn't tell the server we can't take real tasks
+ */
+ lwsl_notice("%s: declining idle task %s\n", __func__, task->uuid);
+ if (saib_queue_task_status_update(sp, spm, task->uuid, 0,
+ SAI_TASK_REASON_IDLE_DECLINED))
+ return -1;
+ saib_reassess_idle_situation();
+
+ return 0;
+
ebail:
saib_task_logf(spm, ns, NULL,
"Unable to create the job dir %s: errno %d (%s)",
diff --git a/src/common/c-sqlite3.c b/src/common/c-sqlite3.c
index b1532aa..0bf57cd 100644
--- a/src/common/c-sqlite3.c
+++ b/src/common/c-sqlite3.c
@@ -128,6 +128,10 @@ sai_event_db_ensure_open(struct lws_context *cx, lws_dll2_owner_t *sqlite3_cache
err = NULL;
sqlite3_exec(*ppdb, "ALTER TABLE tasks ADD COLUMN run integer;", NULL, NULL, &err);
if (err) sqlite3_free(err);
+ err = NULL;
+ /* default 0 so every older task reads as not idle */
+ sqlite3_exec(*ppdb, "ALTER TABLE tasks ADD COLUMN idle integer default 0;", NULL, NULL, &err);
+ if (err) sqlite3_free(err);
}
sai_sqlite3_statement(*ppdb, "CREATE UNIQUE INDEX IF NOT EXISTS idx_task_uuid ON tasks(uuid, run);", "create task index");
diff --git a/src/common/include/private.h b/src/common/include/private.h
index 6315e8b..5831cd0 100644
--- a/src/common/include/private.h
+++ b/src/common/include/private.h
@@ -97,6 +97,12 @@ typedef enum {
SAIES_NOT_READY_FOR_BUILD = 8,
SAIES_STEP_SUCCESS = 9,
SAIES_PAUSED = 10,
+ /*
+ * An idle task's slice that was stopped by the builder, either to make
+ * way for real work or because the slice ran its length. It's not a
+ * failure, the lane just rests until the next slice.
+ */
+ SAIES_YIELDED = 11,
} sai_event_state_t;
enum {
@@ -104,6 +110,8 @@ enum {
SAISPRF_TERMINATED = 0x2000,
SAISPRF_EXIT = 0x8000,
SAISPRF_SIGNALLED = 0x4000,
+ /* with SAISPRF_TERMINATED: the builder stopped an idle task's slice */
+ SAISPRF_YIELDED = 0x10000,
};
typedef enum {
@@ -278,6 +286,12 @@ typedef struct {
char rebuildable;
int run;
+ /*
+ * Nonzero for an idle task: a "lane" that only runs in time builders
+ * would otherwise spend idle, one slice (run) at a time, and that
+ * does not count towards its event's state. See README-idle.md.
+ */
+ int idle;
} sai_task_t;
struct saib_logproxy {
@@ -364,6 +378,7 @@ struct sai_nspawn {
uint8_t term_budget;
uint8_t retcode_set:1;
+ uint8_t idle_yield:1; /* we stopped this idle slice */
uint8_t state_changed:1;
uint8_t user_cancel:1;
uint8_t user_killed:1;
@@ -382,6 +397,12 @@ enum {
SAI_TASK_REASON_DUPE = 1,
SAI_TASK_REASON_BUSY = 2,
SAI_TASK_REASON_DESTROYED = 3,
+ /*
+ * Builder won't take the offered idle task right now, eg, because it
+ * has real work, or recently had. Unlike BUSY, this says nothing
+ * about whether it can take real tasks.
+ */
+ SAI_TASK_REASON_IDLE_DECLINED = 4,
};
typedef struct sai_rejection {
@@ -482,6 +503,8 @@ typedef struct sai_event {
* branch".
*/
int adhoc;
+ /* how many idle tasks ("lanes") the event has, see sai_task_t .idle */
+ int idle;
lws_dll2_owner_t watcher_owner; /* sai_watcher_t */
} sai_event_t;
@@ -681,6 +704,7 @@ typedef struct sai_uuid_list {
lws_usec_t us_time_listed;
char uuid[65];
char started;
+ char idle; /* server: it's an idle task */
} sai_uuid_list_t;
/*
@@ -758,11 +782,23 @@ typedef struct sai_plat {
int powering_down;
unsigned int job_limit;
+ /*
+ * Idle tasks: from the builder conf "idle" object on the platform,
+ * and told to the server with the platform. A zero share means the
+ * platform takes no idle tasks.
+ */
+ unsigned int idle_share; /* % of idle time to fill */
+ unsigned int idle_instances; /* concurrent idle tasks */
+ unsigned int idle_slice_secs; /* length of one slice */
+ unsigned int idle_settle_secs; /* builder only */
+
/* server side only: builder resource tracking */
lws_dll2_owner_t inflight_owner; /* sai_uuid_list_t */
int avail_slots;
unsigned int avail_mem_kib;
unsigned int avail_sto_kib;
+ /* server: don't offer idle tasks before this, after a decline */
+ lws_usec_t idle_backoff_until;
unsigned int windows;
unsigned int power_managed;
@@ -1015,8 +1051,8 @@ extern const lws_struct_map_t
lsm_schema_sq3_map_artifact[1],
lsm_schema_map_ta[1],
lsm_schema_map_plat_simple[1],
- lsm_event[14],
- lsm_task[32],
+ lsm_event[15],
+ lsm_task[33],
lsm_log[8],
lsm_artifact[9],
lsm_plat_list[1],
@@ -1048,7 +1084,7 @@ extern const lws_struct_map_t
lsm_stay_state_update[2],
lsm_schema_stay_state_update[1],
lsm_build_metric[14],
- lsm_plat[14], /* +1 for pcon */
+ lsm_plat[17], /* +1 for pcon */
lsm_builder_platform[1],
lsm_builder_registration[10],
lsm_schema_sq3_map_power_controller[1],
diff --git a/src/common/struct-metadata.c b/src/common/struct-metadata.c
index 7cf34d4..7096516 100644
--- a/src/common/struct-metadata.c
+++ b/src/common/struct-metadata.c
@@ -107,6 +107,9 @@ const lws_struct_map_t lsm_plat[] = {
LSM_UNSIGNED (sai_plat_t, windows, "windows"),
LSM_UNSIGNED (sai_plat_t, power_managed, "power_managed"),
LSM_UNSIGNED (sai_plat_t, stay_on, "stay_on"),
+ LSM_JO_UNSIGNED (sai_plat_t, idle_share, "idle_share"),
+ LSM_JO_UNSIGNED (sai_plat_t, idle_instances, "idle_instances"),
+ LSM_JO_UNSIGNED (sai_plat_t, idle_slice_secs, "idle_slice_secs"),
};
const lws_struct_map_t lsm_schema_map_plat_simple[] = {
@@ -142,6 +145,7 @@ const lws_struct_map_t lsm_event[] = {
LSM_UNSIGNED (sai_event_t, last_updated, "last_updated"),
LSM_UNSIGNED (sai_event_t, sec, "sec"),
LSM_UNSIGNED (sai_event_t, adhoc, "adhoc"),
+ LSM_UNSIGNED (sai_event_t, idle, "idle"),
LSM_JO_LIST (sai_event_t, watcher_owner, sai_watcher_t, list,
NULL, lsm_watcher, "watchers"),
};
@@ -225,6 +229,7 @@ const lws_struct_map_t lsm_task[] = {
LSM_SIGNED (sai_task_t, parallel, "parallel"),
LSM_SIGNED (sai_task_t, rebuildable, "rebuildable"),
LSM_SIGNED (sai_task_t, run, "run"),
+ LSM_SIGNED (sai_task_t, idle, "idle"),
};
const lws_struct_map_t lsm_schema_json_map_task[] = {
diff --git a/src/server/CMakeLists.txt b/src/server/CMakeLists.txt
index b2ac311..1f7d7e0 100644
--- a/src/server/CMakeLists.txt
+++ b/src/server/CMakeLists.txt
@@ -11,6 +11,7 @@ set(SRCS
s-ws-builder.c
s-task.c
s-task-helpers.c
+ s-idle.c
s-central.c
s-ws-web.c
s-webops.c
diff --git a/src/server/s-central.c b/src/server/s-central.c
index 0c71519..e23a42d 100644
--- a/src/server/s-central.c
+++ b/src/server/s-central.c
@@ -250,8 +250,8 @@ sais_central_gc_deleted_events_cb(lws_sorted_usec_list_t *sul)
return;
/* Pick 1 task that is stopped to garbage collect */
- /* 0=WAITING, 3=SUCCESS, 4=FAIL, 5=CANCELLED, 8=NOT_READY, 9=STEP_SUCCESS, 10=PAUSED */
- lws_snprintf(q, sizeof(q), "SELECT uuid FROM tasks WHERE state IN (0, 3, 4, 5, 8, 9, 10) "
+ /* 0=WAITING, 3=SUCCESS, 4=FAIL, 5=CANCELLED, 8=NOT_READY, 9=STEP_SUCCESS, 10=PAUSED, 11=YIELDED */
+ lws_snprintf(q, sizeof(q), "SELECT uuid FROM tasks WHERE state IN (0, 3, 4, 5, 8, 9, 10, 11) "
"AND run=(SELECT max(run) FROM tasks t2 WHERE t2.uuid = tasks.uuid) LIMIT 1");
if (sqlite3_prepare_v2(pdb, q, -1, &tsm, NULL) == SQLITE_OK) {
diff --git a/src/server/s-comms.c b/src/server/s-comms.c
index de8f740..c1bb921 100644
--- a/src/server/s-comms.c
+++ b/src/server/s-comms.c
@@ -347,6 +347,13 @@ s_callback_ws(struct lws *wsi, enum lws_callback_reasons reason, void *user,
NULL, NULL, &err);
if (err)
sqlite3_free(err);
+
+ err = NULL;
+ sqlite3_exec(vhd->server.pdb,
+ "ALTER TABLE events ADD COLUMN idle integer default 0;",
+ NULL, NULL, &err);
+ if (err)
+ sqlite3_free(err);
}
sai_sqlite3_statement(vhd->server.pdb, "CREATE UNIQUE INDEX IF NOT EXISTS idx_event_uuid ON events(uuid);", "create event index");
@@ -432,6 +439,7 @@ s_callback_ws(struct lws *wsi, enum lws_callback_reasons reason, void *user,
if (vhd && vhd->vhost == sais_link_vhost)
sais_link_vhost = NULL;
sais_server_destroy(vhd, &vhd->server);
+ sais_idle_destroy(vhd);
goto passthru;
/*
diff --git a/src/server/s-idle.c b/src/server/s-idle.c
new file mode 100644
index 0000000..048007e
--- /dev/null
+++ b/src/server/s-idle.c
@@ -0,0 +1,800 @@
+/*
+ * Sai server - ./src/server/s-idle.c
+ *
+ * Copyright (C) 2019 - 2026 Andy Green <andy@warmcat.com>
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation:
+ * version 2.1 of the License.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+ * MA 02110-1301 USA
+ *
+ * Idle tasks
+ *
+ * A .sai.json configuration with "idle": N gets N idle tasks ("lanes") per
+ * platform in each event, besides its normal task. Lanes don't count towards
+ * the event's state, and they are only offered to a builder platform that has
+ * nothing real to do, and whose builder conf gave it an "idle" share.
+ *
+ * The lanes of the newest event on a repo + ref whose real tasks have all
+ * finished (the "host" event for that ref) are the ones that run: the event
+ * stays the one being worked on in idle time after it completed, until a newer
+ * push on the ref completes. Each time a lane runs is a "slice", which is a
+ * new run of the lane task; old runs are pruned.
+ *
+ * How much idle time a builder platform gives to lanes is its builder conf
+ * share, eg 50%. The active period (from the first of its slices starting to
+ * the last one ending) puts it in debt: it has to rest for the period *
+ * (100 - share) / share before it can start another. The debt is only paid
+ * off while that platform has no real tasks pending, so it really is a share
+ * of idle time. The debt lives here, and survives the builder disconnecting,
+ * so builders that sai-power turns off while resting are woken again when the
+ * rest is over.
+ *
+ * Making way for real work when it appears is done by the builder, which
+ * stops its slices and reports them YIELDED.
+ */
+
+#include <libwebsockets.h>
+#include <string.h>
+
+#include "s-private.h"
+
+/* the host events are found again at least this often */
+#define SAIS_IDLE_HOSTS_REFRESH_US (10 * LWS_US_PER_SEC)
+/* after a builder declines an idle task, don't offer it another for this */
+#define SAIS_IDLE_DECLINE_BACKOFF_US (60 * LWS_US_PER_SEC)
+/* how many of a lane's most recent slices keep their task row and logs */
+#define SAIS_IDLE_RUNS_KEPT 4
+/*
+ * The least an active period is charged as, so a lane that fails at once
+ * every time (eg, it doesn't build on that platform) rests like one that ran
+ * a while instead of being restarted over and over
+ */
+#define SAIS_IDLE_MIN_PERIOD_US (120 * LWS_US_PER_SEC)
+/* the most rest an active period can put a builder platform in debt for */
+#define SAIS_IDLE_MAX_DEBT_US (7ll * 24 * 3600 * LWS_US_PER_SEC)
+/* how many recent events with lanes we consider as hosts */
+#define SAIS_IDLE_HOST_EVENTS 64
+
+static sais_idle_budget_t *
+sais_idle_budget_find(struct vhd *vhd, const char *name)
+{
+ lws_start_foreach_dll(struct lws_dll2 *, p, vhd->idle_budgets.head) {
+ sais_idle_budget_t *b = lws_container_of(p,
+ sais_idle_budget_t, list);
+
+ if (!strcmp(b->name, name))
+ return b;
+
+ } lws_end_foreach_dll(p);
+
+ return NULL;
+}
+
+/*
+ * A builder told us about one of its platforms, including what, if anything,
+ * its conf says it should do with its idle time
+ */
+
+void
+sais_idle_plat_update(struct vhd *vhd, const sai_plat_t *build)
+{
+ sais_idle_budget_t *b = sais_idle_budget_find(vhd, build->name);
+
+ if (!b) {
+ if (!build->idle_share)
+ return;
+
+ b = malloc(sizeof(*b));
+ if (!b)
+ return;
+ memset(b, 0, sizeof(*b));
+ lws_strncpy(b->name, build->name, sizeof(b->name));
+ lws_strncpy(b->platform, build->platform, sizeof(b->platform));
+ b->last_tick = lws_now_usecs();
+ lws_dll2_add_tail(&b->list, &vhd->idle_budgets);
+ }
+
+ b->share = build->idle_share > 100 ? 100 : build->idle_share;
+ b->instances = build->idle_instances ? build->idle_instances : 1;
+ b->slice_secs = build->idle_slice_secs;
+
+ lwsl_notice("%s: %s: idle share %u%%, %u instances, slice %us\n",
+ __func__, b->name, b->share, b->instances, b->slice_secs);
+}
+
+void
+sais_idle_destroy(struct vhd *vhd)
+{
+ lws_start_foreach_dll_safe(struct lws_dll2 *, p, p1,
+ vhd->idle_budgets.head) {
+ sais_idle_budget_t *b = lws_container_of(p,
+ sais_idle_budget_t, list);
+
+ lws_dll2_remove(&b->list);
+ free(b);
+
+ } lws_end_foreach_dll_safe(p, p1);
+
+ lws_dll2_owner_clear(&vhd->idle_hosts);
+ lwsac_free(&vhd->ac_idle_hosts);
+}
+
+/* how many idle tasks the builder platform is working on right now */
+
+static unsigned int
+sais_idle_running(const sai_plat_t *sp, const char *except_uuid)
+{
+ unsigned int n = 0;
+
+ if (!sp)
+ return 0;
+
+ lws_start_foreach_dll(struct lws_dll2 *, p, sp->inflight_owner.head) {
+ sai_uuid_list_t *ul = lws_container_of(p, sai_uuid_list_t, list);
+
+ if (ul->idle && (!except_uuid || strcmp(ul->uuid, except_uuid)))
+ n++;
+
+ } lws_end_foreach_dll(p);
+
+ return n;
+}
+
+/*
+ * Find the host events again: for each repo + ref, the newest event with lanes
+ * whose real tasks have all finished. For each platform it has lanes on, note
+ * how many of them are not running a slice at the moment.
+ */
+
+static void
+sais_idle_hosts_refresh(struct vhd *vhd)
+{
+ struct {
+ char repo[65];
+ char ref[65];
+ } *seen;
+ sqlite3_stmt *sm, *tsm;
+ int nseen = 0, n;
+ char q[384];
+
+ lws_dll2_owner_clear(&vhd->idle_hosts);
+ lwsac_free(&vhd->ac_idle_hosts);
+ vhd->idle_hosts_refreshed = lws_now_usecs();
+ vhd->idle_hosts_stale = 0;
+
+ lws_snprintf(q, sizeof(q), "select uuid, repo_name, ref from events "
+ "where idle > 0 and adhoc = 0 and state != %d "
+ "order by created desc limit %d", SAIES_DELETED,
+ SAIS_IDLE_HOST_EVENTS);
+ if (sqlite3_prepare_v2(vhd->server.pdb, q, -1, &sm, NULL) != SQLITE_OK) {
+ lwsl_err("%s: %s\n", __func__, sqlite3_errmsg(vhd->server.pdb));
+ return;
+ }
+
+ seen = malloc(sizeof(*seen) * SAIS_IDLE_HOST_EVENTS);
+ if (!seen) {
+ sqlite3_finalize(sm);
+ return;
+ }
+
+ while (sqlite3_step(sm) == SQLITE_ROW) {
+ const char *uuid = (const char *)sqlite3_column_text(sm, 0),
+ *repo = (const char *)sqlite3_column_text(sm, 1),
+ *ref = (const char *)sqlite3_column_text(sm, 2);
+ unsigned int unfinished = 1;
+ sqlite3 *pdb = NULL;
+
+ if (!uuid || !repo || !ref)
+ continue;
+
+ for (n = 0; n < nseen; n++)
+ if (!strcmp(seen[n].repo, repo) &&
+ !strcmp(seen[n].ref, ref))
+ break;
+ if (n != nseen)
+ /* a newer event already hosts this repo + ref */
+ continue;
+
+ if (sai_event_db_ensure_open(vhd->context, &vhd->sqlite3_cache,
+ vhd->sqlite3_path_lhs, uuid, 0,
+ &pdb))
+ continue;
+
+ /*
+ * Have all its real tasks finished? If not, an older event
+ * on the ref may still be the host until it does
+ */
+
+ lws_snprintf(q, sizeof(q),
+ "select count(*) from tasks t1 where idle=0 and "
+ "state not in (%d,%d,%d) and run=(select max(run) "
+ "from tasks t2 where t2.uuid=t1.uuid)",
+ SAIES_SUCCESS, SAIES_FAIL, SAIES_CANCELLED);
+ if (sqlite3_exec(pdb, q, sql3_get_integer_cb, &unfinished,
+ NULL) != SQLITE_OK || unfinished) {
+ sai_event_db_close(&vhd->sqlite3_cache, &pdb);
+ continue;
+ }
+
+ lws_strncpy(seen[nseen].repo, repo, sizeof(seen[nseen].repo));
+ lws_strncpy(seen[nseen].ref, ref, sizeof(seen[nseen].ref));
+ nseen++;
+
+ /* a waiting lane that's bound has been offered to a builder */
+ lws_snprintf(q, sizeof(q),
+ "select platform, sum(case when state in "
+ "(%d,%d,%d,%d) or (state in (%d,%d) and "
+ "ifnull(builder_name,'')='') then 1 else 0 end) "
+ "from tasks t1 where idle=1 and run=(select "
+ "max(run) from tasks t2 where t2.uuid=t1.uuid) "
+ "group by platform",
+ SAIES_SUCCESS, SAIES_FAIL, SAIES_CANCELLED,
+ SAIES_YIELDED, SAIES_WAITING,
+ SAIES_NOT_READY_FOR_BUILD);
+
+ if (sqlite3_prepare_v2(pdb, q, -1, &tsm, NULL) == SQLITE_OK) {
+ while (sqlite3_step(tsm) == SQLITE_ROW) {
+ const char *plat = (const char *)
+ sqlite3_column_text(tsm, 0);
+ sais_idle_host_t *h;
+
+ if (!plat)
+ continue;
+
+ h = lwsac_use_zero(&vhd->ac_idle_hosts,
+ sizeof(*h), 1024);
+ if (!h)
+ break;
+
+ lws_strncpy(h->event_uuid, uuid,
+ sizeof(h->event_uuid));
+ lws_strncpy(h->platform, plat,
+ sizeof(h->platform));
+ h->free_lanes = sqlite3_column_int(tsm, 1);
+ lws_dll2_add_tail(&h->list, &vhd->idle_hosts);
+ }
+ sqlite3_finalize(tsm);
+ }
+
+ sai_event_db_close(&vhd->sqlite3_cache, &pdb);
+
+ if (nseen == SAIS_IDLE_HOST_EVENTS)
+ break;
+ }
+
+ sqlite3_finalize(sm);
+ free(seen);
+}
+
+static void
+sais_idle_hosts_ensure_fresh(struct vhd *vhd)
+{
+ if (vhd->idle_hosts_stale ||
+ lws_now_usecs() - vhd->idle_hosts_refreshed >
+ SAIS_IDLE_HOSTS_REFRESH_US)
+ sais_idle_hosts_refresh(vhd);
+}
+
+/*
+ * Called as part of recomputing the pending platforms, which happens at least
+ * once a second. At that point vhd->pending_plats holds just the platforms
+ * with real tasks waiting or running.
+ *
+ * First, builder platforms with nothing real pending get to pay off some of
+ * their rest.
+ *
+ * Then we add the idle tasks that are running, and those that builders are due
+ * to start, to the pending platforms, so sai-power keeps up, or brings up, the
+ * builders for them. sai-power can only turn on platforms, not particular
+ * builders; so we only ask it for a platform whose builders all take idle
+ * tasks, or we would be waking builders that will decline them.
+ */
+
+void
+sais_idle_add_pending_plats(struct vhd *vhd)
+{
+ lws_usec_t now = lws_now_usecs();
+
+ if (!vhd->idle_budgets.count)
+ return;
+
+ lws_start_foreach_dll(struct lws_dll2 *, p, vhd->idle_budgets.head) {
+ sais_idle_budget_t *b = lws_container_of(p,
+ sais_idle_budget_t, list);
+ lws_usec_t dt = now - b->last_tick;
+ int real = 0;
+
+ b->last_tick = now;
+
+ lws_start_foreach_dll(struct lws_dll2 *, px,
+ vhd->pending_plats.head) {
+ sais_plat_t *pl = lws_container_of(px, sais_plat_t, list);
+
+ if (!strcmp(pl->plat, b->platform) && pl->pending_count)
+ real = 1;
+
+ } lws_end_foreach_dll(px);
+
+ if (!real && b->debt_us)
+ b->debt_us = b->debt_us > dt ? b->debt_us - dt : 0;
+
+ } lws_end_foreach_dll(p);
+
+ sais_idle_hosts_ensure_fresh(vhd);
+
+ lws_start_foreach_dll(struct lws_dll2 *, p, vhd->idle_budgets.head) {
+ sais_idle_budget_t *b = lws_container_of(p,
+ sais_idle_budget_t, list);
+ sai_plat_t *sp = sais_builder_from_uuid(vhd, b->name);
+ unsigned int running = sais_idle_running(sp, NULL), due = 0,
+ lanes = 0, takers = 0, builders = 0;
+ sqlite3_stmt *sm;
+
+ if (b->share && !b->debt_us && running < b->instances) {
+ lws_start_foreach_dll(struct lws_dll2 *, ph,
+ vhd->idle_hosts.head) {
+ sais_idle_host_t *h = lws_container_of(ph,
+ sais_idle_host_t, list);
+
+ if (!strcmp(h->platform, b->platform))
+ lanes += (unsigned int)h->free_lanes;
+
+ } lws_end_foreach_dll(ph);
+
+ due = b->instances - running;
+ if (due > lanes)
+ due = lanes;
+ }
+
+ if (due) {
+ /* do all the builders of the platform take idle tasks? */
+
+ lws_start_foreach_dll(struct lws_dll2 *, p1,
+ vhd->idle_budgets.head) {
+ sais_idle_budget_t *b1 = lws_container_of(p1,
+ sais_idle_budget_t, list);
+
+ if (b1->share && !strcmp(b1->platform, b->platform))
+ takers++;
+
+ } lws_end_foreach_dll(p1);
+
+ if (sqlite3_prepare_v2(vhd->server.pdb,
+ "select count(*) from builders where "
+ "platform=?", -1, &sm, NULL) == SQLITE_OK) {
+ sqlite3_bind_text(sm, 1, b->platform, -1,
+ SQLITE_TRANSIENT);
+ if (sqlite3_step(sm) == SQLITE_ROW)
+ builders = (unsigned int)
+ sqlite3_column_int(sm, 0);
+ sqlite3_finalize(sm);
+ }
+
+ if (takers < builders)
+ due = 0;
+ }
+
+ if (running || due)
+ sais_add_pending_plat(vhd, b->platform,
+ (int)(running + due), (int)due);
+
+ } lws_end_foreach_dll(p);
+}
+
+/*
+ * Arm the next slice of an idle task whose last one is over: a new run, bound
+ * to the builder platform that's going to do it. The oldest runs beyond the
+ * last few are pruned, with their logs and artifacts, since a lane runs
+ * indefinitely.
+ */
+
+static int
+sais_idle_new_run(struct vhd *vhd, sqlite3 *pdb, const char *task_uuid,
+ const sai_plat_t *sp)
+{
+ struct lwsac *ac = NULL;
+ char esc[96], q[256];
+ lws_dll2_owner_t o;
+ sai_task_t *t;
+ int n;
+
+ lws_sql_purify(esc, task_uuid, sizeof(esc));
+ lws_snprintf(q, sizeof(q), " and uuid='%s'", esc);
+
+ n = lws_struct_sq3_deserialize(pdb, q, "run desc",
+ lsm_schema_sq3_map_task, &o, &ac, 0, 1);
+ if (n < 0 || !o.head) {
+ lwsac_free(&ac);
+ return 1;
+ }
+
+ t = lws_container_of(o.head, sai_task_t, list);
+
+ t->run++;
+ t->state = SAIES_WAITING;
+ t->started = 0;
+ t->duration = 0;
+ t->build_step = 0;
+ t->last_updated = (uint64_t)lws_now_secs();
+ t->server_name = "";
+ lws_strncpy(t->builder, sp->name, sizeof(t->builder));
+ lws_strncpy(t->builder_name, sp->name, sizeof(t->builder_name));
+
+ n = lws_struct_sq3_serialize(pdb, lsm_schema_sq3_map_task, &o, 0);
+ if (n < 0) {
+ lwsl_err("%s: unable to add run %d of %s\n", __func__, t->run,
+ task_uuid);
+ lwsac_free(&ac);
+ return 1;
+ }
+
+ n = t->run - SAIS_IDLE_RUNS_KEPT;
+ lwsac_free(&ac);
+
+ if (n >= 0) {
+ lws_snprintf(q, sizeof(q), "delete from logs where "
+ "task_uuid='%s' and run <= %d", esc, n);
+ sqlite3_exec(pdb, q, NULL, NULL, NULL);
+ lws_snprintf(q, sizeof(q), "delete from artifacts where "
+ "task_uuid='%s' and run <= %d", esc, n);
+ sqlite3_exec(pdb, q, NULL, NULL, NULL);
+ lws_snprintf(q, sizeof(q), "delete from tasks where "
+ "uuid='%s' and run <= %d", esc, n);
+ sqlite3_exec(pdb, q, NULL, NULL, NULL);
+ }
+
+ return 0;
+}
+
+/*
+ * Look for a lane on the host event for this platform that isn't running a
+ * slice, arm it and offer it to the builder platform. Returns 0 if we
+ * offered one.
+ */
+
+static int
+sais_idle_offer_lane(struct vhd *vhd, sai_plat_t *sp, const char *event_uuid)
+{
+ char task_uuid[65], q[384];
+ sqlite3 *pdb = NULL;
+ sqlite3_stmt *sm;
+ int state = -1;
+
+ if (sai_event_db_ensure_open(vhd->context, &vhd->sqlite3_cache,
+ vhd->sqlite3_path_lhs, event_uuid, 0, &pdb))
+ return 1;
+
+ lws_snprintf(q, sizeof(q),
+ "select uuid, state from tasks t1 where idle=1 and "
+ "platform=? and run=(select max(run) from tasks t2 where "
+ "t2.uuid=t1.uuid) and (state in (%d,%d,%d,%d) or "
+ "(state in (%d,%d) and (builder_name is null or "
+ "builder_name in ('', ?)))) order by uid",
+ SAIES_SUCCESS, SAIES_FAIL, SAIES_CANCELLED, SAIES_YIELDED,
+ SAIES_WAITING, SAIES_NOT_READY_FOR_BUILD);
+
+ if (sqlite3_prepare_v2(pdb, q, -1, &sm, NULL) != SQLITE_OK) {
+ lwsl_err("%s: %s\n", __func__, sqlite3_errmsg(pdb));
+ sai_event_db_close(&vhd->sqlite3_cache, &pdb);
+ return 1;
+ }
+
+ sqlite3_bind_text(sm, 1, sp->platform, -1, SQLITE_TRANSIENT);
+ sqlite3_bind_text(sm, 2, sp->name, -1, SQLITE_TRANSIENT);
+
+ task_uuid[0] = '\0';
+ while (sqlite3_step(sm) == SQLITE_ROW) {
+ const char *u = (const char *)sqlite3_column_text(sm, 0);
+
+ if (!u || strlen(u) != SAI_TASKID_LEN ||
+ sais_is_task_inflight(vhd, NULL, u, NULL))
+ continue;
+
+ lws_strncpy(task_uuid, u, sizeof(task_uuid));
+ state = sqlite3_column_int(sm, 1);
+ break;
+ }
+ sqlite3_finalize(sm);
+
+ if (!task_uuid[0]) {
+ sai_event_db_close(&vhd->sqlite3_cache, &pdb);
+ return 1;
+ }
+
+ if (state != SAIES_WAITING && state != SAIES_NOT_READY_FOR_BUILD) {
+ /* its last slice is over, it needs a new run for the next */
+ if (sais_idle_new_run(vhd, pdb, task_uuid, sp)) {
+ sai_event_db_close(&vhd->sqlite3_cache, &pdb);
+ return 1;
+ }
+ sai_event_db_close(&vhd->sqlite3_cache, &pdb);
+ sais_taskchange(vhd->h_ss_websrv, task_uuid, SAIES_WAITING);
+ } else {
+ /* its first slice, the task as created */
+ sai_event_db_close(&vhd->sqlite3_cache, &pdb);
+ if (sais_bind_task_to_builder(vhd, sp->name, sp->name,
+ task_uuid))
+ return 1;
+ if (state == SAIES_NOT_READY_FOR_BUILD)
+ sais_set_task_state(vhd, task_uuid, SAIES_WAITING, 0, 0);
+ }
+
+ lwsl_notice("%s: offering idle task %s to %s\n", __func__, task_uuid,
+ sp->name);
+
+ if (sais_create_and_offer_task_step(vhd, task_uuid)) {
+ sais_bind_task_to_builder(vhd, NULL, NULL, task_uuid);
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * The builder platform has nothing real to do... if it gives some of its idle
+ * time to idle tasks and it's not resting, offer it one. Returns 0 if we
+ * offered it something.
+ */
+
+int
+sais_idle_allocate(struct vhd *vhd, struct pss *pss, sai_plat_t *sp)
+{
+ sais_idle_budget_t *b = sais_idle_budget_find(vhd, sp->name);
+ lws_usec_t now = lws_now_usecs();
+ unsigned int running;
+
+ if (!b || !b->share || b->debt_us || sp->idle_backoff_until > now)
+ return 1;
+
+ running = sais_idle_running(sp, NULL);
+ if (running >= b->instances)
+ return 1;
+
+ /*
+ * A new slice can join the active period while it's younger than a
+ * slice, otherwise the period ends when the running slices do and the
+ * platform rests before starting any more
+ */
+ if (running && b->slice_secs &&
+ now - b->period_start > (lws_usec_t)b->slice_secs * LWS_US_PER_SEC)
+ return 1;
+
+ sais_idle_hosts_ensure_fresh(vhd);
+
+ lws_start_foreach_dll(struct lws_dll2 *, p, vhd->idle_hosts.head) {
+ sais_idle_host_t *h = lws_container_of(p, sais_idle_host_t, list);
+
+ if (h->free_lanes <= 0 || strcmp(h->platform, sp->platform))
+ continue;
+
+ if (!sais_idle_offer_lane(vhd, sp, h->event_uuid)) {
+ h->free_lanes--;
+ if (!running)
+ b->period_start = now;
+
+ sais_list_builders(vhd);
+ pss->mark_started = 1;
+
+ return 0;
+ }
+
+ /* nothing we could use there after all */
+ h->free_lanes = 0;
+
+ } lws_end_foreach_dll(p);
+
+ return 1;
+}
+
+/*
+ * An idle task's slice ended, one way or another. If it was the last of the
+ * builder platform's slices running, its active period is over and it owes
+ * rest in proportion.
+ */
+
+void
+sais_idle_slice_ended(struct vhd *vhd, sqlite3 *pdb, const char *task_uuid)
+{
+ char builder_name[96], q[192], esc[96];
+ uint64_t started = 0;
+ sais_idle_budget_t *b;
+ lws_usec_t now, period;
+ sqlite3_stmt *sm;
+
+ vhd->idle_hosts_stale = 1;
+
+ builder_name[0] = '\0';
+ lws_sql_purify(esc, task_uuid, sizeof(esc));
+ lws_snprintf(q, sizeof(q), "select builder_name, started from tasks "
+ "where uuid='%s' order by run desc limit 1", esc);
+ if (sqlite3_prepare_v2(pdb, q, -1, &sm, NULL) != SQLITE_OK)
+ return;
+ if (sqlite3_step(sm) == SQLITE_ROW) {
+ const char *bn = (const char *)sqlite3_column_text(sm, 0);
+
+ if (bn)
+ lws_strncpy(builder_name, bn, sizeof(builder_name));
+ started = (uint64_t)sqlite3_column_int64(sm, 1);
+ }
+ sqlite3_finalize(sm);
+
+ b = builder_name[0] ? sais_idle_budget_find(vhd, builder_name) : NULL;
+ if (!b || !b->share)
+ return;
+
+ if (sais_idle_running(sais_builder_from_uuid(vhd, builder_name),
+ task_uuid))
+ /* the builder platform's active period goes on */
+ return;
+
+ now = lws_now_usecs();
+ if (b->period_start)
+ period = now - b->period_start;
+ else
+ /* we restarted since it began, go by the slice itself */
+ period = started ? ((lws_usec_t)lws_now_secs() -
+ (lws_usec_t)started) * LWS_US_PER_SEC : 0;
+ if (period < SAIS_IDLE_MIN_PERIOD_US)
+ period = SAIS_IDLE_MIN_PERIOD_US;
+
+ b->debt_us += (period * (100 - (lws_usec_t)b->share)) / b->share;
+ if (b->debt_us > SAIS_IDLE_MAX_DEBT_US)
+ b->debt_us = SAIS_IDLE_MAX_DEBT_US;
+ b->period_start = 0;
+
+ lwsl_notice("%s: %s: active for %llds, resting for %llds\n", __func__,
+ b->name, (long long)(period / LWS_US_PER_SEC),
+ (long long)(b->debt_us / LWS_US_PER_SEC));
+}
+
+/*
+ * The builder platform won't take the idle task we offered right now, eg,
+ * it has real work, or had it recently. Put the task back and don't offer it
+ * idle tasks for a while.
+ */
+
+void
+sais_idle_declined(struct vhd *vhd, sai_plat_t *sp, const char *task_uuid)
+{
+ char event_uuid[33], esc[96], q[160];
+ sqlite3 *pdb = NULL;
+ int state = -1;
+
+ sp->idle_backoff_until = lws_now_usecs() + SAIS_IDLE_DECLINE_BACKOFF_US;
+ vhd->idle_hosts_stale = 1;
+
+ sai_task_uuid_to_event_uuid(event_uuid, task_uuid);
+ if (!sai_event_db_ensure_open(vhd->context, &vhd->sqlite3_cache,
+ vhd->sqlite3_path_lhs, event_uuid, 0,
+ &pdb)) {
+ lws_sql_purify(esc, task_uuid, sizeof(esc));
+ lws_snprintf(q, sizeof(q), "select state from tasks where "
+ "uuid='%s' order by run desc limit 1", esc);
+ if (sqlite3_exec(pdb, q, sql3_get_integer_cb, &state,
+ NULL) != SQLITE_OK)
+ state = -1;
+ sai_event_db_close(&vhd->sqlite3_cache, &pdb);
+ }
+
+ if (state == SAIES_STEP_SUCCESS) {
+ /*
+ * It was offered a later step of a slice that had started,
+ * eg, real work turned up between its steps. The steps have
+ * to happen in one place, so that's the end of the slice.
+ */
+ sais_task_logf(vhd, task_uuid, "builder %s declined the next "
+ "step, so this idle slice is over", sp->name);
+ sais_set_task_state(vhd, task_uuid, SAIES_YIELDED, 0, 0);
+ return;
+ }
+
+ sais_bind_task_to_builder(vhd, NULL, NULL, task_uuid);
+}
+
+/*
+ * The builder went away... any slices it had are over, and nothing else is
+ * going to tell us. Real tasks it had are dealt with separately, but they
+ * only live on events that haven't completed, and idle tasks mostly live on
+ * events that have.
+ */
+
+void
+sais_idle_builder_gone(struct vhd *vhd, sai_plat_t *sp)
+{
+ sqlite3_stmt *sm, *tsm;
+ char q[256];
+
+ lws_snprintf(q, sizeof(q), "select uuid from events where idle > 0 "
+ "and state != %d order by created desc limit %d",
+ SAIES_DELETED, SAIS_IDLE_HOST_EVENTS);
+ if (sqlite3_prepare_v2(vhd->server.pdb, q, -1, &sm, NULL) != SQLITE_OK)
+ return;
+
+ lws_snprintf(q, sizeof(q),
+ "select uuid, state from tasks t1 where idle=1 and "
+ "builder_name=? and state in (%d,%d,%d,%d) and "
+ "run=(select max(run) from tasks t2 where t2.uuid=t1.uuid)",
+ SAIES_WAITING, SAIES_PASSED_TO_BUILDER, SAIES_BEING_BUILT,
+ SAIES_STEP_SUCCESS);
+
+ while (sqlite3_step(sm) == SQLITE_ROW) {
+ const char *ev = (const char *)sqlite3_column_text(sm, 0);
+ lws_dll2_owner_t owner;
+ struct lwsac *ac = NULL;
+ sqlite3 *pdb = NULL;
+
+ if (!ev || sai_event_db_ensure_open(vhd->context,
+ &vhd->sqlite3_cache, vhd->sqlite3_path_lhs,
+ ev, 0, &pdb))
+ continue;
+
+ /*
+ * Collect them first, changing the task state reopens the db
+ */
+
+ lws_dll2_owner_clear(&owner);
+ if (sqlite3_prepare_v2(pdb, q, -1, &tsm, NULL) == SQLITE_OK) {
+ sqlite3_bind_text(tsm, 1, sp->name, -1,
+ SQLITE_TRANSIENT);
+ while (sqlite3_step(tsm) == SQLITE_ROW) {
+ const char *u = (const char *)
+ sqlite3_column_text(tsm, 0);
+ sai_uuid_list_t *ul;
+
+ if (!u)
+ continue;
+ ul = lwsac_use_zero(&ac, sizeof(*ul), 512);
+ if (!ul)
+ break;
+ lws_strncpy(ul->uuid, u, sizeof(ul->uuid));
+ /* reuse .started for "was it offered only" */
+ ul->started = sqlite3_column_int(tsm, 1) ==
+ SAIES_WAITING;
+ lws_dll2_add_tail(&ul->list, &owner);
+ }
+ sqlite3_finalize(tsm);
+ }
+ sai_event_db_close(&vhd->sqlite3_cache, &pdb);
+
+ lws_start_foreach_dll(struct lws_dll2 *, p, owner.head) {
+ sai_uuid_list_t *ul = lws_container_of(p,
+ sai_uuid_list_t, list);
+
+ sai_uuid_list_t *inf;
+
+ /*
+ * Drop it from what the builder platform is running
+ * first, so ending the last one ends its active period
+ */
+ if (sais_is_task_inflight(vhd, sp, ul->uuid, &inf))
+ sais_inflight_entry_destroy(inf);
+
+ if (ul->started)
+ sais_bind_task_to_builder(vhd, NULL, NULL,
+ ul->uuid);
+ else {
+ sais_task_logf(vhd, ul->uuid,
+ "builder %s disconnected, so this idle "
+ "slice is over", sp->name);
+ sais_set_task_state(vhd, ul->uuid,
+ SAIES_YIELDED, 0, 0);
+ }
+
+ } lws_end_foreach_dll(p);
+
+ lwsac_free(&ac);
+ }
+
+ sqlite3_finalize(sm);
+}
diff --git a/src/server/s-notification.c b/src/server/s-notification.c
index ed54b2e..0ed49d6 100644
--- a/src/server/s-notification.c
+++ b/src/server/s-notification.c
@@ -92,6 +92,7 @@ static const char * const saifile_paths[] = {
"configurations.*.cpack",
"configurations.*.branches",
"configurations.*.task_log_limit",
+ "configurations.*.idle",
"configurations.*",
};
@@ -110,6 +111,7 @@ enum enum_saifile_paths {
LEJPNSAIF_CONFIGURATIONS_CPACK,
LEJPNSAIF_CONFIGURATIONS_BRANCHES,
LEJPNSAIF_CONFIGURATIONS_TASK_LOG_LIMIT,
+ LEJPNSAIF_CONFIGURATIONS_IDLE,
LEJPNSAIF_CONFIGURATIONS_NAME,
};
@@ -293,6 +295,7 @@ sai_saifile_lejp_cb(struct lejp_ctx *ctx, char reason)
sn->t.artifacts[0] = '\0';
sn->t.branches[0] = '\0';
sn->explicit_platforms[0] = '\0';
+ sn->idle_lanes = 0;
return 0;
}
@@ -555,17 +558,28 @@ next_plat: ;
* Mint the uuids / nonces and create the task
* in the event-specific database
*/
+ pss->sn.t.idle = 0;
if (sais_task_insert(lws_get_context(pss->wsi),
pdb, &pss->sn.e, &pss->sn.t,
- pss->sn.event_task_index++) < 0) {
- lwsl_err("%s: task insert failed\n",
- __func__);
- sqlite3_exec(pdb, "END TRANSACTION", NULL, NULL, &err);
- if (err)
- sqlite3_free(err);
- sai_event_db_close(&pss->vhd->sqlite3_cache, &pdb);
- return -1;
+ pss->sn.event_task_index++) < 0)
+ goto insert_fail;
+
+ /*
+ * ... and the idle tasks ("lanes") the
+ * configuration asks for on this platform, if
+ * any. They are the same task, but only run
+ * later in idle time, see s-idle.c
+ */
+
+ for (n = 0; n < sn->idle_lanes; n++) {
+ pss->sn.t.idle = 1;
+ if (sais_task_insert(lws_get_context(pss->wsi),
+ pdb, &pss->sn.e, &pss->sn.t,
+ pss->sn.event_task_index++) < 0)
+ goto insert_fail;
+ sn->e.idle++;
}
+ pss->sn.t.idle = 0;
}
} lws_end_foreach_dll(p);
@@ -582,11 +596,18 @@ next_plat: ;
*/
sais_platforms_with_tasks_pending(pss->vhd);
-// lwsl_notice("%s: New test '%s', '%s', '%s'\n", __func__,
-// sn->t.taskname, sn->t.cmake, sn->t.packages);
-
sn->t.taskname[0] = '\0';
return 0;
+
+insert_fail:
+ lwsl_err("%s: task insert failed\n", __func__);
+ pss->sn.t.idle = 0;
+ sqlite3_exec(pdb, "END TRANSACTION", NULL, NULL, &err);
+ if (err)
+ sqlite3_free(err);
+ sai_event_db_close(&pss->vhd->sqlite3_cache, &pdb);
+
+ return -1;
}
if (reason == LEJPCB_OBJECT_START &&
@@ -742,6 +763,20 @@ next_plat: ;
sn->t.task_log_limit = (unsigned int)atoi(ctx->buf);
break;
+ case LEJPNSAIF_CONFIGURATIONS_IDLE:
+ /*
+ * How many idle tasks ("lanes") to create for this
+ * configuration on each platform it runs on, in addition to
+ * the normal task. Each lane can have an idle builder
+ * working on it at the same time.
+ */
+ sn->idle_lanes = atoi(ctx->buf);
+ if (sn->idle_lanes < 0)
+ sn->idle_lanes = 0;
+ if (sn->idle_lanes > SAI_IDLE_LANES_MAX)
+ sn->idle_lanes = SAI_IDLE_LANES_MAX;
+ break;
+
case LEJPNSAIF_PLAT_BUILD:
case LEJPNSAIF_PLAT_BUILD_STAGE:
/*
@@ -1120,6 +1155,7 @@ sai_notification_file_upload_cb(void *data, const char *name,
*/
pss->dry = 0;
+ pss->sn.e.idle = 0;
lejp_construct(&saictx, sai_saifile_lejp_cb, pss, saifile_paths,
LWS_ARRAY_SIZE(saifile_paths));
sai_lejp_enable_comments(&saictx);
@@ -1139,6 +1175,21 @@ sai_notification_file_upload_cb(void *data, const char *name,
lwsl_notice("%s: notification inserted into db\n", __func__);
+ if (pss->sn.e.idle) {
+ char q[128], esc[96];
+
+ /*
+ * The event row went in before we knew how many idle
+ * tasks the configurations would create
+ */
+ lws_sql_purify(esc, pss->sn.e.uuid, sizeof(esc));
+ lws_snprintf(q, sizeof(q),
+ "update events set idle=%d where uuid='%s'",
+ pss->sn.e.idle, esc);
+ sai_sqlite3_statement(pss->vhd->server.pdb, q,
+ "set event idle count");
+ }
+
/*
* Let sai-web know the event exists now, the same as for an
* ad-hoc event: otherwise it only hears about it once its
diff --git a/src/server/s-private.h b/src/server/s-private.h
index 13c4003..5b31cd6 100644
--- a/src/server/s-private.h
+++ b/src/server/s-private.h
@@ -34,6 +34,12 @@
*/
#define SAI_SHELLID_LEN 32
+/*
+ * The most idle tasks ("lanes") a .sai.json configuration can ask for on each
+ * platform
+ */
+#define SAI_IDLE_LANES_MAX 16
+
struct sai_plat;
/* lws_wsmsg_ array for different sources */
@@ -117,6 +123,7 @@ typedef struct {
char explicit_platforms[2048];
int event_task_index;
+ int idle_lanes; /* per platform, this config */
struct lws_b64state b64;
char *saifile;
@@ -229,6 +236,30 @@ typedef struct sais_plat {
int unmet_count;
} sais_plat_t;
+/*
+ * Server's record of a builder platform's idle task budget, see s-idle.c. It's
+ * kept by name, since it has to outlive the builder being disconnected.
+ */
+typedef struct sais_idle_budget {
+ lws_dll2_t list; /* vhd->idle_budgets */
+ lws_usec_t debt_us; /* rest owed before the next slice */
+ lws_usec_t last_tick;
+ lws_usec_t period_start; /* 0, or when its slices began */
+ unsigned int share; /* % of idle time for idle tasks */
+ unsigned int instances; /* concurrent idle tasks */
+ unsigned int slice_secs;
+ char name[96]; /* builder.platform, as sai_plat_t .name */
+ char platform[96];
+} sais_idle_budget_t;
+
+/* a platform on an event whose lanes are the ones to run for its ref */
+typedef struct sais_idle_host {
+ lws_dll2_t list; /* vhd->idle_hosts */
+ char event_uuid[33];
+ char platform[96];
+ int free_lanes; /* lanes not running a slice */
+} sais_idle_host_t;
+
typedef struct sai_shell_session {
lws_dll2_t list;
char task_uuid[65];
@@ -266,6 +297,13 @@ struct vhd {
lws_dll2_owner_t watcher_services; /* sai_watcher_service_t from config */
+ /* idle tasks, see s-idle.c */
+ lws_dll2_owner_t idle_budgets; /* sais_idle_budget_t */
+ lws_dll2_owner_t idle_hosts; /* sais_idle_host_t in ac_idle_hosts */
+ struct lwsac *ac_idle_hosts;
+ lws_usec_t idle_hosts_refreshed;
+ char idle_hosts_stale;
+
lws_usec_t last_check_abandoned_tasks;
const char *notification_key;
@@ -412,6 +450,30 @@ sais_resource_rr_destroy(sai_resource_requisition_t *rr);
int
sais_platforms_with_tasks_pending(struct vhd *vhd);
+void
+sais_add_pending_plat(struct vhd *vhd, const char *name, int count, int unmet);
+
+void
+sais_idle_plat_update(struct vhd *vhd, const sai_plat_t *build);
+
+void
+sais_idle_destroy(struct vhd *vhd);
+
+void
+sais_idle_add_pending_plats(struct vhd *vhd);
+
+int
+sais_idle_allocate(struct vhd *vhd, struct pss *pss, sai_plat_t *sp);
+
+void
+sais_idle_slice_ended(struct vhd *vhd, sqlite3 *pdb, const char *task_uuid);
+
+void
+sais_idle_declined(struct vhd *vhd, sai_plat_t *sp, const char *task_uuid);
+
+void
+sais_idle_builder_gone(struct vhd *vhd, sai_plat_t *sp);
+
sai_plat_t *
sais_builder_from_uuid(struct vhd *vhd, const char *hostname);
sai_plat_t *
@@ -431,7 +493,8 @@ sais_is_task_inflight(struct vhd *vhd, sai_plat_t *build, const char *uuid,
sai_uuid_list_t **hit);
int
-sais_add_to_inflight_list_if_absent(struct vhd *vhd, sai_plat_t *sp, const char *uuid);
+sais_add_to_inflight_list_if_absent(struct vhd *vhd, sai_plat_t *sp,
+ const char *uuid, int idle);
void
sais_inflight_entry_destroy(sai_uuid_list_t *ul);
diff --git a/src/server/s-task-helpers.c b/src/server/s-task-helpers.c
index a5e984f..a069e26 100644
--- a/src/server/s-task-helpers.c
+++ b/src/server/s-task-helpers.c
@@ -235,6 +235,7 @@ sais_set_task_state(struct vhd *vhd, const char *task_uuid,
char update[512], esc1[96], esc2[96], esc3[32], esc4[32], event_uuid[33];
sai_event_state_t oes, sta, task_ostate;
unsigned int count = 0, count_good = 0, count_bad = 0;
+ int idle = 0;
uint64_t started_orig = started;
struct lwsac *ac = NULL;
sai_event_t *e = NULL;
@@ -292,6 +293,12 @@ sais_set_task_state(struct vhd *vhd, const char *task_uuid,
goto bail;
}
+ lws_snprintf(update, sizeof(update),
+ "select idle from tasks where uuid='%s' order by run desc limit 1", esc2);
+ if (sqlite3_exec((sqlite3 *)e->pdb, update,
+ sql3_get_integer_cb, &idle, NULL) != SQLITE_OK)
+ idle = 0;
+
if (task_ostate == SAIES_PAUSED &&
(state == SAIES_BEING_BUILT || state == SAIES_PASSED_TO_BUILDER ||
state == SAIES_STEP_SUCCESS || state == SAIES_FAIL ||
@@ -347,18 +354,31 @@ sais_set_task_state(struct vhd *vhd, const char *task_uuid,
sais_taskchange(vhd->h_ss_websrv, task_uuid, state);
if (state == SAIES_SUCCESS || state == SAIES_FAIL ||
- state == SAIES_CANCELLED)
+ state == SAIES_CANCELLED || state == SAIES_YIELDED)
lws_sul_schedule(vhd->context, 0, &vhd->sul_central,
sais_central_cb, 1);
if (state != SAIES_STEP_SUCCESS)
sais_platforms_with_tasks_pending(vhd);
+ if (idle) {
+ /*
+ * Idle tasks have no say in their event's state, it
+ * was decided by the real tasks. But the end of a
+ * slice is when its builder starts to owe rest.
+ */
+ if (state == SAIES_SUCCESS || state == SAIES_FAIL ||
+ state == SAIES_CANCELLED || state == SAIES_YIELDED)
+ sais_idle_slice_ended(vhd, (sqlite3 *)e->pdb,
+ task_uuid);
+ goto done;
+ }
+
/*
* So, how many tasks for this event?
*/
- if (sqlite3_exec((sqlite3 *)e->pdb, "select count(*) from (select max(run) from tasks group by uuid)",
+ if (sqlite3_exec((sqlite3 *)e->pdb, "select count(*) from (select max(run) from tasks where idle=0 group by uuid)",
sql3_get_integer_cb, &count, NULL) != SQLITE_OK) {
lwsl_err("%s: %s: %s: fail\n", __func__, update,
sqlite3_errmsg(vhd->server.pdb));
@@ -369,7 +389,7 @@ sais_set_task_state(struct vhd *vhd, const char *task_uuid,
* ... how many completed well?
*/
- if (sqlite3_exec((sqlite3 *)e->pdb, "select count(*) from (select max(run) as mx, state from tasks group by uuid) where state == 3",
+ if (sqlite3_exec((sqlite3 *)e->pdb, "select count(*) from (select max(run) as mx, state from tasks where idle=0 group by uuid) where state == 3",
sql3_get_integer_cb, &count_good, NULL) != SQLITE_OK) {
lwsl_err("%s: %s: %s: fail\n", __func__, update,
sqlite3_errmsg(vhd->server.pdb));
@@ -380,7 +400,7 @@ sais_set_task_state(struct vhd *vhd, const char *task_uuid,
* ... how many failed?
*/
- if (sqlite3_exec((sqlite3 *)e->pdb, "select count(*) from (select max(run) as mx, state from tasks group by uuid) where state == 4",
+ if (sqlite3_exec((sqlite3 *)e->pdb, "select count(*) from (select max(run) as mx, state from tasks where idle=0 group by uuid) where state == 4",
sql3_get_integer_cb, &count_bad, NULL) != SQLITE_OK) {
lwsl_err("%s: %s: %s: fail\n", __func__, update,
sqlite3_errmsg(vhd->server.pdb));
@@ -432,6 +452,7 @@ sais_set_task_state(struct vhd *vhd, const char *task_uuid,
}
}
+done:
sai_event_db_close(&vhd->sqlite3_cache, (sqlite3 **)&e->pdb);
lwsac_free(&ac);
diff --git a/src/server/s-task.c b/src/server/s-task.c
index 830df05..1b398a2 100644
--- a/src/server/s-task.c
+++ b/src/server/s-task.c
@@ -44,7 +44,7 @@ sais_event_check_for_plat_tasks(struct vhd *vhd, const char *event_uuid,
return 0;
lws_snprintf(query, sizeof(query),
- "select count(state) from tasks where platform = '%s'",
+ "select count(state) from tasks where platform = '%s' and idle=0",
platform);
if (sqlite3_exec(check_pdb, query, sql3_get_integer_cb, &count,
@@ -113,7 +113,8 @@ sais_is_task_inflight(struct vhd *vhd, sai_plat_t *build, const char *uuid,
}
int
-sais_add_to_inflight_list_if_absent(struct vhd *vhd, sai_plat_t *sp, const char *uuid)
+sais_add_to_inflight_list_if_absent(struct vhd *vhd, sai_plat_t *sp,
+ const char *uuid, int idle)
{
sai_uuid_list_t *uuid_list;
@@ -127,6 +128,7 @@ sais_add_to_inflight_list_if_absent(struct vhd *vhd, sai_plat_t *sp, const char
memset(uuid_list, 0, sizeof(*uuid_list));
lws_strncpy(uuid_list->uuid, uuid, sizeof(uuid_list->uuid));
uuid_list->us_time_listed = lws_now_usecs();
+ uuid_list->idle = (char)!!idle;
lws_dll2_add_tail(&uuid_list->list, &sp->inflight_owner);
@@ -257,7 +259,7 @@ sais_task_pending(struct vhd *vhd, struct pss *pss, sai_plat_t *cb,
*/
lws_snprintf(query, sizeof(query), "select count(state) from tasks t1 where "
- "state IN(0,9) and platform='%s' and "
+ "state IN(0,9) and idle=0 and platform='%s' and "
"(builder_name IS NULL or builder_name IN('','%s'))"
" and run = (select max(run) from tasks t2 where t1.uuid = t2.uuid)",
esc_plat, esc_bname);
@@ -337,7 +339,7 @@ sais_task_pending(struct vhd *vhd, struct pss *pss, sai_plat_t *cb,
lws_snprintf(query, sizeof(query),
"select taskname from tasks t1 where "
- "state = 4 and platform = ?"
+ "state = 4 and idle=0 and platform = ?"
" and run = (select max(run) from tasks t2 where t1.uuid = t2.uuid)");
if (sqlite3_prepare_v2(prev_pdb, query, -1, &sm, NULL) == SQLITE_OK) {
@@ -392,7 +394,7 @@ sais_task_pending(struct vhd *vhd, struct pss *pss, sai_plat_t *cb,
lws_sql_purify(esc_taskname, fti->taskname, sizeof(esc_taskname));
lws_snprintf(pf, sizeof(pf),
- " and state IN(0,9) and platform='%s' and taskname='%s' and "
+ " and state IN(0,9) and idle=0 and platform='%s' and taskname='%s' and "
"(builder_name IS NULL or builder_name IN('','%s'))"
" and run = (select max(run) from tasks t2 where tasks.uuid = t2.uuid)",
esc_plat, esc_taskname, esc_bname);
@@ -432,7 +434,7 @@ next1: ;
/* We have fallen back to doing tasks earliest-first */
lws_snprintf(pf, sizeof(pf),
- " and state IN(0,9) and platform='%s' and "
+ " and state IN(0,9) and idle=0 and platform='%s' and "
"(builder_name IS NULL or builder_name IN('','%s'))"
" and run = (select max(run) from tasks t2 where tasks.uuid = t2.uuid)",
esc_plat, esc_bname);
@@ -515,6 +517,12 @@ sais_find_or_add_pending_plat(struct vhd *vhd, const char *name, int count, int
return 0;
}
+void
+sais_add_pending_plat(struct vhd *vhd, const char *name, int count, int unmet)
+{
+ sais_find_or_add_pending_plat(vhd, name, count, unmet);
+}
+
static void
sais_destroy_pending_plat_list(struct vhd *vhd)
{
@@ -561,10 +569,14 @@ sais_platforms_with_tasks_pending(struct vhd *vhd)
n = lws_struct_sq3_deserialize(vhd->server.pdb, pf, "created desc ",
lsm_schema_sq3_map_event, &o, &ac, 0, 20);
- if (n < 0 || !o.head) {
- /* error, or there are no events that aren't complete */
+ if (n < 0)
goto bail;
- }
+
+ /*
+ * If there are no events that aren't complete, we carry on: idle
+ * tasks live on completed events, and sai-power still needs telling
+ * that nothing is pending any more
+ */
/*
* Iterate through the events looking at his event-specific database
@@ -582,7 +594,7 @@ sais_platforms_with_tasks_pending(struct vhd *vhd)
if (sqlite3_prepare_v2(pdb, "select platform, count(*), "
"sum(case when (state = 0 or state = 9) and (builder_name IS NULL or builder_name = '') then 1 else 0 end) "
- "from tasks t1 where "
+ "from tasks t1 where idle=0 and "
"run = (select max(run) from tasks t2 where t1.uuid = t2.uuid) and "
"(state = 0 or state = 1 or state = 2 or state = 9) group by platform", -1, &sm,
NULL) != SQLITE_OK) {
@@ -619,6 +631,12 @@ sais_platforms_with_tasks_pending(struct vhd *vhd)
} lws_end_foreach_dll(p);
/*
+ * Idle tasks that builders are due to be working on count as pending
+ * too, so sai-power brings those builders up for them
+ */
+ sais_idle_add_pending_plats(vhd);
+
+ /*
* Also account for any in-memory interactive shell sessions
*/
lws_start_foreach_dll(struct lws_dll2 *, p_sh, vhd->shell_sessions.head) {
@@ -678,7 +696,11 @@ sais_allocate_task(struct vhd *vhd, struct pss *pss, sai_plat_t *sp,
if (!task_template) {
lwsl_info("%s: %s: can't identify pending task\n",
__func__, sp->name);
- return 1;
+ /*
+ * Nothing real for this builder platform to do... it may
+ * be willing to spend the time on an idle task
+ */
+ return sais_idle_allocate(vhd, pss, sp);
}
/*
@@ -888,6 +910,16 @@ sais_create_and_offer_task_step(struct vhd *vhd, const char *task_uuid)
task_template = lws_container_of(o.head, sai_task_t, list);
+ if (task_template->state == SAIES_YIELDED) {
+ /*
+ * The builder stopped this idle task's slice, it has no
+ * more steps to offer until s-idle.c starts a new slice
+ */
+ sai_event_db_close(&vhd->sqlite3_cache, &pdb);
+ lwsac_free(&ac);
+ return 0;
+ }
+
/*
* Make a copy of the lws_struct allocation in the lwsac,
* but we must retain the lwsac because the copied task_template
@@ -940,7 +972,8 @@ sais_create_and_offer_task_step(struct vhd *vhd, const char *task_uuid)
}
if (!inflight) {
- if (sais_add_to_inflight_list_if_absent(vhd, sp, task_uuid)) {
+ if (sais_add_to_inflight_list_if_absent(vhd, sp, task_uuid,
+ temp_task->idle)) {
lwsl_warn("%s: bailing as can't add to inflight %s\n", __func__, task_uuid);
sais_task_clear_build_and_logs(vhd, task_uuid, 0);
goto bail;
@@ -1047,7 +1080,8 @@ sais_create_and_offer_task_step(struct vhd *vhd, const char *task_uuid)
temp_task->server_name = pss->server_name;
if (!inflight) {
- if (sais_add_to_inflight_list_if_absent(vhd, sp, temp_task->uuid)) {
+ if (sais_add_to_inflight_list_if_absent(vhd, sp, temp_task->uuid,
+ temp_task->idle)) {
lwsl_warn("%s: bailing as can't add to inflight %s\n", __func__, task_uuid);
sais_task_clear_build_and_logs(vhd, temp_task->uuid, 0);
goto bail;
diff --git a/src/server/s-webops.c b/src/server/s-webops.c
index 2338c44..8d10a2b 100644
--- a/src/server/s-webops.c
+++ b/src/server/s-webops.c
@@ -576,7 +576,7 @@ sais_event_delete(struct vhd *vhd, const char *event_uuid)
/* 3. Drop active builders gracefully without loading huge JSON objects */
if (sai_event_db_ensure_open(vhd->context, &vhd->sqlite3_cache,
vhd->sqlite3_path_lhs, event_uuid, 0, &pdb) == 0) {
- lws_snprintf(qu, sizeof(qu), "SELECT uuid FROM tasks WHERE state != 0 AND state != 3 AND state != 4 AND state != 5 "
+ lws_snprintf(qu, sizeof(qu), "SELECT uuid FROM tasks WHERE state != 0 AND state != 3 AND state != 4 AND state != 5 AND state != 11 "
"AND run=(SELECT max(run) FROM tasks t2 WHERE t2.uuid = tasks.uuid)");
if (sqlite3_prepare_v2(pdb, qu, -1, &sm, NULL) == SQLITE_OK) {
while (sqlite3_step(sm) == SQLITE_ROW) {
diff --git a/src/server/s-ws-builder.c b/src/server/s-ws-builder.c
index 2d0a2db..bb0de25 100644
--- a/src/server/s-ws-builder.c
+++ b/src/server/s-ws-builder.c
@@ -592,6 +592,9 @@ sais_builder_disconnected(struct vhd *vhd, struct lws *wsi)
lwsac_free(&ac);
}
+ /* ... and any idle tasks it had */
+ sais_idle_builder_gone(vhd, sp);
+
/* drop any inflight task information for this builder */
lws_start_foreach_dll_safe(struct lws_dll2 *, pif, pif1,
@@ -730,11 +733,27 @@ sais_process_rej(struct vhd *vhd, struct pss *pss,
sais_plat_busy(sp, 1);
break;
+ case SAI_TASK_REASON_IDLE_DECLINED:
+ lwsl_notice("%s: SAI_TASK_REASON_IDLE_DECLINED: %s\n",
+ __func__, rej->task_uuid);
+ do_remove_uuid = 1;
+ sais_idle_declined(vhd, sp, rej->task_uuid);
+ break;
+
case SAI_TASK_REASON_DESTROYED:
lwsl_notice("%s: SAI_TASK_REASON_DESTROYED: Clear busy: %s\n",
__func__, rej->task_uuid);
do_remove_uuid = 1;
+ if (rej->ecode & SAISPRF_YIELDED) {
+ /*
+ * The builder stopped an idle task's slice, to make
+ * way for real work or because its time was up
+ */
+ n = SAIES_YIELDED;
+ lwsl_notice("%s: |||| SAIES_YIELDED: %s\n",
+ __func__, rej->task_uuid);
+ } else
if (rej->ecode & SAISPRF_EXIT) {
if ((rej->ecode & 0xff) == 0) {
n = SAIES_STEP_SUCCESS;
@@ -1105,6 +1124,9 @@ sais_ws_json_rx_builder(struct vhd *vhd, struct pss *pss, uint8_t *buf, size_t b
lws_dll2_add_tail(&live_sp->sai_plat_list, &vhd->server.builder_owner);
}
+ /* and what it wants to do with its idle time */
+ sais_idle_plat_update(vhd, build);
+
lws_sul_schedule(live_sp->cx, 0, &live_sp->sul_find_jobs,
sais_plat_find_jobs_cb, 500 * LWS_US_PER_MS);
diff --git a/src/web/w-comms.c b/src/web/w-comms.c
index ade95d6..bb076ff 100644
--- a/src/web/w-comms.c
+++ b/src/web/w-comms.c
@@ -358,6 +358,13 @@ w_callback_ws(struct lws *wsi, enum lws_callback_reasons reason, void *user,
NULL, NULL, &err);
if (err)
sqlite3_free(err);
+
+ err = NULL;
+ sqlite3_exec(vhd->pdb,
+ "ALTER TABLE events ADD COLUMN idle integer default 0;",
+ NULL, NULL, &err);
+ if (err)
+ sqlite3_free(err);
}
sai_sqlite3_statement(vhd->pdb, "CREATE UNIQUE INDEX IF NOT EXISTS idx_event_uuid ON events(uuid);", "create event index");
diff --git a/src/web/w-ws-browser.c b/src/web/w-ws-browser.c
index b4b2f03..4b1849d 100644
--- a/src/web/w-ws-browser.c
+++ b/src/web/w-ws-browser.c
@@ -1687,9 +1687,11 @@ saiw_event_summary_string(sqlite3 *pdb_event, const char *event_uuid,
* tasks holds one row per (task, run); the browser summarises on the
* latest (max) run per task uuid. Pick exactly that row per uuid using
* a correlated max-run subquery, so we count each task once.
+ *
+ * Idle tasks don't count, they're not part of the event's result.
*/
lws_snprintf(q, sizeof(q),
- "SELECT state FROM tasks t WHERE rowid = ("
+ "SELECT state FROM tasks t WHERE idle=0 AND rowid = ("
"SELECT rowid FROM tasks t2 WHERE t2.uuid = t.uuid "
"ORDER BY t2.run DESC LIMIT 1)");
|