Cockpit Bots
These are automated bots and tools that work on Cockpit. This includes updating operating system images, updating translations or NPM modules, testing PRs, and more.
Images
In order to test Cockpit-related projects, they are staged into an operating
system image. These images are tracked in the images/
directory. For example,
you might want to test a scenario where Cockpit on one machine talks to FreeIPA
on another, and you want those two machines to use different images.
This is handled by passing a specific image to image-create and other scripts that work with test machine images. Available images include:
fedora-*
,rhel-*
,debian-*
, etc: Various operating systems for testing Cockpit related projectsservices
: Auxiliary network services for tests which are independent from the OS where Cockpit runs: FreeIPA, Samba AD, candlepin, Grafana
These well known image names are expected to contain no .
characters and have no file name extension.
Individual projects are expected to locally build their code into packages, and
install them as overlay on top of these pristine images, with image-customize
or using the machine Python API.
For managing these images:
image-download
: Download selected or all test imagesimage-create
: Create test machine images from scratch (usually through downloading a cloud image), with common build and test dependencies for Cockpit projects preinstalledimage-upload
: Upload a locally built test image to the official image servers
For running and debugging the images:
image-customize
: Install packages, upload files, or run commands in a test machine image; this keeps the original image intact, and puts the changes into an image overlay into test/images/.vm-run
: Run a test machine image; by default this happens in an ephemeral overlay. You can use the--maintain
option to write into the persistent overlay in test/images/ instead.vm-reset
: Remove all overlays from test/images/
Image location
Downloaded images are stored into ~/.cache/cockpit-images/ by default. If you
want to change that, you can set $COCKPIT_IMAGES_DATA_DIR
or the
cockpit.bots.images-data-dir
variable with git config
to a directory where
to store the pristine virtual machine images. For example:
git config cockpit.bots.images-data-dir /srv/cockpit/images
Tests
The bots automatically run the tests as needed on pull requests and branches. To check when and where tests will be run, use the tests-scan tool:
./tests-scan -vd
Note on eslintrc interaction
As eslint looks for additional configurations, eslintrc.(json|yaml) files, in
parent directories, it is recommended to have "root": true
in the eslint
configuration of any project which is using eslint and is tested through
cockpit-bots.
Integration with GitHub
A number of machines are watching our GitHub repositories and are executing tests for pull requests as well as making new images.
Most of this happens automatically, but you can influence their actions with the tests-trigger utility in this directory.
Setup
You need a GitHub token in ~/.config/cockpit-dev/github-token or from the GitHub CLI configuration in ~/.config/gh/config.yml. You can create one for your account at Developer Settings → Personal access tokens.
When generating a new personal access token, the scopes should contain
repo:status
and read:org
. Note in particular, that repo
and
public_repo
scopes each grant full push access, and should not be used.
You need at least “Write” access to the project for triggering statuses, either individually per repo (e.g. cockpit or for all cockpit-project repos.
If you’d like to download Red Hat-only internal images from S3, you’ll
need to create a key file in ~/.config/cockpit-dev/s3-keys/[domain]
.
The [domain]
can be any non-toplevel domain which contains the S3 URL
in question. The contents of this file should be a single line
containing the “access key” and the “secret key” separated by
whitespace.
For the currently configured mirrors this means that you’d likely have the following file:
~/.config/cockpit-dev/s3-keys/linodeobjects.com
For more control, you could also use the following:
~/.config/cockpit-dev/s3-keys/cockpit-images.eu-central-1.linodeobjects.com
~/.config/cockpit-dev/s3-keys/eu-central-1.linodeobjects.com
- either of the above, with
us-east
instead ofeu-central
each file would be a single line which looks like
EEVIDIDFSOQ0ABJ2LGTT 009rKOypIoqO44Q3VQGRyYPfugi84zANHF0pOW9f
The “access key” and “secret key” is unique per-developer and can be obtained by talking to Allison.
Test contexts
For describing tests which we want to run we use contexts. A context has the form:
image[/scenario][@bots#bots_pr][@owner/project/ref]
where items have the following meaning:
- image: Name of the image on which tests should run (e.g. ‘fedora-coreos’).
- scenario: Name of a specific test. This is specific for each separate project and
is passed verbatim to ‘test/run’ in
$TEST_SCENARIO
. - bots_pr: Number of pull request that exists in bots repository. When specified, bots from this PR would be used instead of main.
- owner/project: Name of github project (e.g. ‘cockpit-project/cockpit’). This part can be omitted when testing in the same project and no ‘ref’ is needed.
- ref: Reference in the project (usually branch) (e.g. ‘rhel-8.2’). Default is the project’s primary branch.
For example, context for scenario ‘firefox’ on ‘fedora-coreos’ is:
fedora-coreos/firefox
If we want to trigger it on ‘cockpit-project/cockpit’:
fedora-coreos/firefox@cockpit-project/cockpit
If we want to also not run it on the primary branch, but on ‘rhel-8-0’ branch:
fedora-coreos/firefox@cockpit-project/cockpit/rhel-8-0
If we want to run tests on ‘fedora-coreos’ but with bots from pull request ‘169’:
fedora-coreos@bots#169
Retrying a failed test
If you want to run the “fedora-coreos” testsuite again for pull request #1234 of cockpit-project/cockpit, run tests-trigger like so:
./tests-trigger --repo cockpit-project/cockpit 1234 fedora-coreos
You can also invoke bots/tests/trigger from any project checkout, in which case
you don’t need the explicit --repo
– it will default to the GitHub origin of
the current directory’s project.
Testing a pull request by a non-allowed user
If you want to run all tests on pull request #1234 that has been opened by
someone who does not have push access to the repository nor isn’t in the
allowlist
run tests-trigger with --allow
:
./tests-trigger --allow [...]
Of course, you should make sure that the pull request is proper and doesn’t execute evil code during tests.
tests-trigger with a different origin
If you need to specify –repo in tests-trigger as your remote is different from cockpit-project/cockpit, you can set a git configuration option from which tests-trigger reads the repo. This has to be set per cockpit project.
git config cockpit.bots.github-repo cockpit-project/cockpit
Refreshing a test image
Test images are refreshed automatically once per week, and even if the last refresh has failed, the machines wait one week before trying again.
If you want the machines to refresh the fedora-coreos image immediately, run image-trigger like so:
./image-trigger fedora-coreos
Creating new images for a pull request
If as part of some new feature you need to change the content of some or all images, you can ask the machines to create those images.
If you want to have a new fedora-coreos image for pull request #1234, add a bullet point to that pull request’s description like so, and add the “bot” label to the pull request.
* [ ] image-refresh fedora-coreos
The machines will post comments to the pull request about their progress and at the end there will be links to commits with the new images. You can then include these commits into the pull request in any way you like.
Updating CI to a new Fedora release
TEST_OS_DEFAULT
is usually set to the latest (stable) Fedora released,
used as default OS for test VMs.
- If this is a new image, add
_manual
test contexts for the new image tolib/testmap.py
, and land that intomain
. - Create a PR that updates
TEST_OS_DEFAULT
inlib/constants.py
, and trigger all tests for that image there.
Fedora CoreOS
The Fedora CoreOS image is updated to a new Fedora release out of our control, when this occurs:
- Update the naughty symlink
naughty/fedora-coreos
to the release CoreOS uses. - Update
OSTREE_BUILD_IMAGE
to point to the Fedora release CoreOS uses.
Pixel tests
The pixel tests used in Cockpit projects use test/reference-image
to
determine what image to run the pixel tests on.
- Create a PR which updates
test/reference-image
. - Update the pixel tests if required.