• Used for automated exp-runs and package build queue management.
• https://github.com/freebsd/PortCI

The build application will be the central place for queuing, monitoring, and viewing exp-run results. It will have an HTML and JSON REST interface. It will also display production package build status. This application will be the “master” for exp-runs.

New builds will be queued into the master. They may or may not have patches. Patches can be for the ports tree or src tree. Builds can be configured to do a comparison once completed against a known good build to find new failures.

Client builders will check in to the master and ask for work. They will occasionally report their status as well.

Lost/crashed builds will be retried a few times before being marked failed.

Results will list the following:

1. New failures
2. New skipped
3. New ignored
4. New built

• Simplify management/sharing of servers in cluster
• Remove workload for portmgr needing to manually run,track,analyse exp-runs
• Increase transparency for entire project by letting all see what the “exp-run queue” is
• Encourage committers to upload patches for testing
• Facilitate daily QAT plist builds to increase enforcement of orphans/leftovers, standards
• Facilitate daily QAT non-plist builds to find ports that no longer compile/package
• Allow all portmgr to be involved with exp-runs in a more secure manner than giving root
• Automatically retry failed package builds. Currently they require manual intervention.

1. Poudriere stability. See Stability section at https://fossil.etoilebsd.net/poudriere/doc/trunk/doc/todo.wiki
2. Patches against src/ports tree that the nightly reference is not using require running a specific reference for that patch. It could try to find the “closest” reference, but would deliver many false-positives. We don't want people looking at the list of failures and misjudging the results. We want confidence in the results so we do not introduce new failures into head.
3. Similar to 2, need to automatically run a reference build if needed. Leaving out of initial design as it complicates things too much. In general the nightly reference build has worked well. Asking submitter to rebase patch against a known reference revision seems reasonable for version 1.

1. Exp-run
   • Patch against the src or ports tree.
   • All ports will be built.
   • All existing packages will be deleted.
   • Completion will compare against a Reference build.
   • svn patch; bulk -ca
2. Reference
   • No Patch.
   • All ports will be built.
   • All existing packages will be deleted.
   • Completion will compare against previous Reference build.
   • bulk -ca
3. QAT
   • No patch.
   • All ports will be built.
   • All existing packages will be deleted.
   • Plist testing will be done.
   • Completion will compare against previous QAT build.
   • bulk -cat
4. Port
   • Patch against the src or ports tree.
   • Only the 1 port will be built.
   • Existing packages for that 1 port will be deleted.
   • Plist testing will be done.
   • svn patch; testport -n -o
5. Package
   • No patch.
   • All ports will be built.
   • Existing packages kept.
   • NO_PACKAGE packages not built, RESTRICTED packages cleaned up after bulk.
   • Completion will compare against previous build for new failures/skipped (not built as it will be different due to incremental).
   • bulk -a

• All connections will require SSL.
• The master's REST port SSL cert will be self-signed with our own CA that the builders will use exclusively to validate the cert.
• All requests and responses between builder/master will use a shared secret to ensure the commands are coming from known trusted clients.

• The system will be fully readable without authentication.
• Via LDAP/Kerberos, all committers will have access to upload a patch.
• Portmgr users can approve patches, cancel builds, retry builds, and modify configuration.

• Builders will be authenticated with known tokens that can be revoked.
• Builders will be configured (on the builder machine) to specify which job types and archs it can handle. This will allow having the package build machines claim package builds jobs, but never any kind that has a patch. This ensures the package builder is still secure in this system as it only takes an order to start a build, never executes any arbitrary data.

1. Portmgr will login to master and generate a token. This token will only display once and never again. This is the shared secret for authenticating requests and must remain secure.
2. Portmgr will setup new builder and specify the token and the master's SSL CA public key in its configuration.
3. Builder starts up and generates an id from uuidgen.
4. Builder connects to Master to register its id and hostname with the token.
   • Possibly uses DH1080 here to insert another shared secret that is never exchanged, for each to use in future communications.
5. Master registers the builder and associates its id and hostname with the token.
   • If token is already registered, deny registration.

• All builder requests to master will include its authentication key. An invalid key will be rejected. This is only so a rogue/unapproved builder cannot steal work.

id: uuid
request_key: sha256("request", builder hostname, uuid, token)

• Master responses to builder will include a validation key so that the builder can authenticate the master is valid to prevent it receiving a rogue job. Note that this key is different than the request key so that it cannot just be replayed from the request. It does contain the same shared secret token though. Any responses not containing this valid key will be ignored.

response_key: sha256("response", builder hostname, uuid, token)

• Status updates for unapproved/start/stop/crashed/cancelled/finished/analysed jobs will go to IRC.

• The IRC bot will be read-only. No queueing or approving of jobs as there is no way to authenticate securely, even with SSL the IRCD is a weak point.

• Updates to builds that have an associated PR will send an update to gnats when the build crashes, finishes, and once analyzed. There may be a long delay between finished and analyzed which is why there is an extra notice.

• Final results for builds will be emailed to appropriate parties according to the build type.

1. Patch is uploaded by committer and configured for a specified build.
   • Only exp-run and port builds are available to non-portmgr.
   • Portmgr can queue any build type
2. Build is marked unapproved until a portmgr approves it.
3. Builders check in frequently for work.
   • Once builder takes a build, a new job is created and assigned to the builder.
   • The build will be marked running. It will provide a Build URL back to the master for it to update the job object.
   • A patch id will be given in the response, along with checksum.
4. Builder will download associated patch from master if provided and compare checksum.
5. Build starts.
6. The builder will check-in every 5 minutes and on boot.
   • Missing 4 checkins will consider the job as lost and cause it to have the build's fail_cnt incremented and its status moved to retry to retry.
   • A crashed or stale build seen on startup will notify master of the failure and have the build's fail_cnt incremented and its status moved to approved to retry.
   • If a build crashes 3 times it is marked as failed and not retried again. It may be causing panics and should not continue to bring down builders.
   • A build can be cancelled at any time by a user. When the builder checks in, if the build the job is is running for is cancelled it will receive that notice and cancel its work. When completed it will report back to the master and it will mark the job and build as cancelled.
7. When a job is completed the builder will notify the master and provide a URL to a tarball of its log files, along with checksum.
   • If the reported job is successful and has been re-queued/reassigned then the new job should be aborted.
   • If the reported job is not finished, but is already considered lost, then the job will be aborted.
8. Master will download the log files, compare checksum, and then extract locally for later display and analysis.

1. When the build is completed, it will be marked as pending-analysis
2. When the master has an adequate reference build available it will compare against pending-analysis builds, update their results, and then mark them as analysed.
3. Results will be mailed to any associated PR, portmgr, and the person who queued it.

1. When the build is completed, it will be marked as pending-analysis
2. When the master has an adequate reference build available it will compare against pending-analysis builds, update their results, and then mark them as analysed.
3. New failures will be mailed to ports@, portmgr and potentially CC all committers on the hook for the commit range.

• Check queue to find timed out jobs. Increment timeout_cnt for the job. Once the timeout count reaches 4, the job will be aborted, build's fail_cnt incremented and the build re-queued by changing its status to retry.
• Check Exp-runs/Port jobs in pending-analysis state and try to compare against a Reference build. If a reference is not ready, try later. If a reference is done, update status to analyzed and send job notifications.
• Nightly queue up a Reference build
• Nightly queue up a QAT build

• token: String of token
• builder_id: (optional) ID of assigned builder
• uid_generated: UID of user who generated the token

• id: ID of builder
• uuid: UUID of builder
• hostname: Hostname of builder
• CPU: Number of CPUs
• RAM: Amount of RAM
• status: idle|running|lost

• id: ID of build
• name: Name of job (security risk, requires sanitation)
• type: exp-run|port|qat|reference|package
• pr: (optional) Associated PR
• patch_id: ID of patch
• status: unapproved|approved|assigned|failed-patch|running|crashed|cancelling|cancelled|retry|pending-analysis|analyzed|finished
• status_txt: Last status update text
• cancelled: Boolean on whether or not build is aborted
• uid_queued: UID of who queued the job
• uid_approved: UID of who approved the job
• notify_email: Comma-separated string of emails to CC on results
• arch: i386|amd64
• release: 9.1|10.0|head
• branch: Which ports branch to use
• fail_cnt: Number of times the build has failed
• priority: Secondary number to sort the queue by, lower is ran first

• id: ID of job
• build_id: ID of build
• lost: Boolean to determine if this is currently active or lost. Only 1 active should exist at once.
• status: starting|running|crashed|cancelling|cancelled|finished
• status_txt: Last status update text
• builder_id: Which builder is assigned this job
• buildurl: Build URL
• timeout_cnt: Number of times the job has timed out
• cancelled: Boolean on whether or not job is aborted

• id: ID of patch
• type: src|ports
• path: Path to patch on local system
• checksum: Checksum for patch

• All status will be queued and persistent so that outages or connectivity issues to master will not lose responses.