sai-push

Overview

sai-push is an optional daemon, usually run on the git server host, that promotes commits once Sai says they're good: eg, when an event on main-dev succeeds, it pushes that commit to main, and optionally to the same branch on mirrors such as github.

It follows sai-web's public feed of events (see "RSS feed of build events" in the top level README), as JSON, using the feed's long poll: after the first fetch, it holds a request open with sai-web and only hears back when an event joins the feed or changes state, eg, from building to succeeded. So it reacts within moments of a build finishing, without polling.

How it decides what to push

Each watch in the conf covers one repository, identified by the fetch url its sai notifications give, and has rules mapping source branches to target branches:

• The first rule matching a branch decides: the branch must match the rule's wildcard match (default *) and end in its branch-suffix. The suffix is removed to name the target branch, eg, main-dev -> main. Branches no rule matches are left alone.

• For each target branch, the newest successful event on a source branch mapping to it is the one promoted. That's the latest known-good commit, even if newer events are still building or failed. Ad-hoc events (admin scratch builds from the web UI) are never promoted.

• An event whose notification arrived before the one last promoted to that target is never promoted over it. This is remembered across restarts, see "State" below.

Before pushing, sai-push fetches the primary remote into a bare repo in its cache dir and checks:

• the commit is still on the source branch: if the source branch was rewritten since, the commit isn't what the branch says is good any more, and it's skipped

• the target branch doesn't already have the commit: if it does, there's nothing to do. So an older success coming back to the top, eg, because a newer event's tasks were restarted, can't rewind the target, even where force pushing is allowed.

Then it pushes the commit to the target branch on the primary remote, forced if the rule says so, and then to each mirror in turn, each checked first for already having it. A target branch that doesn't exist yet is created.

If anything fails on the primary, the mirrors are not touched. If a mirror fails, the other mirrors are still pushed. Failures are logged with git's own output, and the same commit is tried again no sooner than 10 minutes later, when the feed next changes or the long poll's 10 minute wait expires.

Setting it up

1) Build and install sai with the SAI_PUSH cmake option (on by default on non-Windows), which installs /usr/local/bin/sai-push. It needs lws built with LWS_WITH_CLIENT, LWS_WITH_STRUCT_JSON and LWS_WITH_SPAWN, and recent enough to have

• 864f9da72 "lejp: pop an array's element parser when the array ends, at any depth": without it, a watch that has both rules and mirrors silently loses whichever of them comes second in the conf

and, in the lws front-end web server that proxies sai-web,

• fbedb4001 "proxy: a response still flowing keeps the parent's content timeout off": without it, the front-end drops each held long poll request after its timeout_secs, and sai-push goes round a fetch / wait cycle that often instead of waiting quietly.

2) Create a user just for sai-push, eg, sai-push, with a home dir and no password. sai-push is started as root, and becomes this user before it does anything on the network; git and ssh run as it.

3) As that user, create an ssh key, and give it write access to the repos it should manage on the git server, eg, in gitolite.

4) Put the git server's host key in that user's ~/.ssh/known_hosts, under exactly the host name (and port, if not 22) the remote url uses. ssh runs in BatchMode, so it will not ask, and an unknown host fails with Host key verification failed. If the git server is this host, you can take the key straight from its own host key file, rather than trusting what comes over the network the first time, eg,

` # sudo -u sai-push sh -c 'awk "{print "libwebsockets.org " $1 " " $2}" /etc/ssh/ssh_host_ed25519_key.pub >> ~/.ssh/known_hosts' `

or log in once as that user with the same name as the remote url: sudo -u sai-push ssh git@libwebsockets.org.

5) For a github mirror, the best fit is a deploy key: an ssh key that github accepts for just the one repository it's added to, with write access if you allow it. It doesn't expire, so a headless daemon can't quietly lose access to the mirror a month later, and it's revoked from the repository's settings if the host is ever compromised. It works the same way as the gitolite key, as an ssh mirror url, with no token file.

• Make a key just for this, as the sai-push user, separate from the gitolite one so either can be revoked alone (github also refuses a key that's already in use anywhere else on github):

  ```
  # sudo -u sai-push ssh-keygen -t ed25519 -N "" -C "sai-push github libwebsockets" -f /home/sai-push/.ssh/github_libwebsockets
  ```

- In the github web UI, as an admin of the repository: the repository's
  **Settings** tab, then **Deploy keys** in the sidebar (eg,
  `https://github.com/warmcat/libwebsockets/settings/keys`), then **Add
  deploy key**.  Give it a title, eg, `sai-push on libwebsockets.org`,
  paste in the contents of the **public** key,
  `/home/sai-push/.ssh/github_libwebsockets.pub`, tick **Allow write
  access**, and **Add key**.

- Tell ssh to use that key for github, in `/home/sai-push/.ssh/config`
  (owned by sai-push, mode 0600):

  ```
  Host github.com
  	IdentityFile ~/.ssh/github_libwebsockets
  	IdentitiesOnly yes
  ```

- Put github's host key in sai-push's `known_hosts`: check the
  fingerprint ssh shows against the ones github publishes at
  https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/githubs-ssh-key-fingerprints
  before accepting it, with

  ```
  # sudo -u sai-push ssh -T git@github.com
  ```

  It should answer `Hi warmcat/libwebsockets! You've successfully
  authenticated, but GitHub does not provide shell access.`: the
  repository name there confirms it's the deploy key being used.

- The mirror in the conf is then just `{ "url":
  "ssh://git@github.com/warmcat/" }`.

A deploy key covers only its one repository. To mirror another repo, make it its own key and its own watch, and give it a host alias in the ssh config, keeping github's host key entry by using HostKeyAlias:

` Host github-otherproject HostName github.com HostKeyAlias github.com IdentityFile ~/.ssh/github_otherproject IdentitiesOnly yes `

with that watch's mirror url ssh://git@github-otherproject/warmcat/.

If the target branch is protected on github by a branch protection rule or a ruleset, the deploy key must be allowed to push to it, and to force push to it if the rule forces: rulesets can list "Deploy keys" as a bypass actor.

Alternatively, an https mirror can use a token, see token-file below, eg, a github fine-grained personal access token restricted to just the mirror repositories with "Contents: Read and write". But those expire: when it does, the mirror pushes start failing, with git's message in the log, until you make a new one and replace the file.

6) Write the conf, /etc/sai/push/conf, see below.

7) Install and start the systemd unit, scripts/sai-push.service:

` # cp scripts/sai-push.service /etc/systemd/system/ # systemctl daemon-reload # systemctl enable --now sai-push `

It orders itself after sai-web.service, for when sai-web is on the same host, without depending on it: sai-push is often on the git server with sai-web elsewhere, and retries the feed with backoff anyway. If the feed url goes through a front-end web server on this host, add its unit to the After= line too.

Command line

Configuration

The conf is JSON, read from /etc/sai/push/conf unless -c says otherwise. # comments to the end of the line are allowed.

sai-push checks the whole conf before starting, and refuses to start if anything in it isn't usable, logging each problem with its line number:

• a member it doesn't know, eg, a misspelling, or one that's in the wrong place, eg, mirrors outside the watch it belongs to: it says where a misplaced member should go - a value of the wrong kind, eg, "force": "true" (a string, rather than true), or a single { } where a list [ { } ] is needed - a missing or wrong schema, or a file that isn't valid JSON

It also warns, without refusing to start, about an http(s) remote, or an http(s) mirror without a token-file, since git has no way to authenticate pushes there unless the user's own git config gives it credentials. At startup it logs, for each watch, its rules, the remote it pushes to, and its mirrors, or "no mirrors".

Example

# sai-push conf: /etc/sai/push/conf
{
	"schema": "sai-push",

	"user": "sai-push",
	"repo-cache": "/var/cache/sai-push",

	"watches": [{
		"feed": "https://libwebsockets.org/sai/rss.xml",
		"fetchurl": "https://libwebsockets.org/repo/libwebsockets",
		"remote": "ssh://git@libwebsockets.org/",

		"rules": [
			# eg, v5.0-stable-dev -> v5.0-stable, fast-forward only
			{ "match": "*-stable-dev", "branch-suffix": "-dev",
			  "force": false },
			# anything else, eg, main-dev -> main, forced
			{ "branch-suffix": "-dev", "force": true }
		],

		"mirrors": [
			# a github deploy key for the repo, see setup step 5
			{ "url": "ssh://git@github.com/warmcat/" }
		]
	}]
}

With this, when an event for libwebsockets succeeds on main-dev, its commit is force-pushed to main on ssh://git@libwebsockets.org/libwebsockets and then on ssh://git@github.com/warmcat/libwebsockets; when one succeeds on v5.0-stable-dev, its commit is pushed to v5.0-stable on both, but only if that's a fast-forward.

The same example, commented, is in etc-sai-EXAMPLE/push/conf.

Top level

Watch

Each watch covers one repository.

Rule

Mirror

State

sai-push-state.json in the repo-cache dir holds, for each target branch, the last event promoted to it on the primary remote: its uuid, its commit and the unix time its notification arrived. It's what stops an older event being promoted over a newer one, including across restarts. It's rewritten, by writing a new file and renaming it over the old one, each time a promotion reaches the primary.

If it's damaged, sai-push refuses to start rather than carry on without it; removing it starts over, with only the git checks protecting the target branches until the next promotion.

Logs

sai-push logs to stderr, so under systemd, to the journal. Among other things it logs

• the rules and mirrors it understood from the conf, at startup

• each feed update it acts on: `saip_feed_process: <fetchurl>: index ..., N events`

• each promotion started, and how each step ended, eg, `promoting to main (force), pushed <hash> to main on <remote>, main on <remote> already has <hash>, <hash> is no longer on main-dev, not promoting it, or not promoting <hash> ... it's older than event <uuid>, already promoted`

• anything git says while doing it, a line at a time

Troubleshooting