openQA for Developers and Maintainers

KDE Linux uses openQA to automatically test every system image before it is published. openQA boots each image in a virtual machine and checks that it can be installed, upgraded, and used successfully.

openQA has some specific terminology: a job is one execution of a test suite against an image, while a worker is the machine or container that runs the job. The openQA documentation explains these and other concepts, including test modules, job groups, assets, and the web UI.

How the tests work

The test worker runs alongside the CI pipeline, so it can use the images and virtual disks produced by the build without uploading those large files elsewhere. Tests are supplied to the virtual machine through a system extension and communicate with it over SSH.

Desktop tests use the accessibility API through selenium-webdriver-at-spi, allowing them to interact with applications and check their state without relying solely on screenshot matching. The suite also checks for failed system services, crashed desktop processes, networking problems, and regressions in KDE Linux commands.

CI pipeline and maintainer workflow

On the protected default branch of the upstream KDE Linux repository, the pipeline works as follows:

  1. The imaging job builds, signs, and stages the image and its sysupdate channel on storage.kde.org.
  2. The trigger-openqa job starts the pipeline in os-autoinst-distri-kdelinux, passing the staged image URL and sysupdate channel URL to it.
  3. The downstream pipeline runs the install and upgrade flows declared in tests/tests.toml, each with plain and encrypted installations.
  4. The upstream publish job runs only after the openQA pipeline has succeeded. It promotes the already-staged image for public consumption.

This makes an openQA failure a gate for image publish. Start by opening the downstream pipeline from the trigger-openqa job, then open the failing openQA job from its CI log. The job page shows the test modules in order and lets you identify whether the failure is in installation, the installed-system sanity tests, or the upgrade path.

Download a staged image

The link provided after the log message “In case of failure, inspect and download the image and sysupdate tree at…” opens a storage browser for the build under test. Download the .iso there to boot it manually or give it to a local openQA stack. The sysupdate tree contains the staged update artifacts used by the upgrade test.

Investigate a failed job

Open the job in the openQA web UI and select the failed module first. The link to the web UI will appear after the “ test job is now running.” log message. Its details include the module result, screenshots or video where applicable, and diagnostic output. In the Logs & Assets tab, download or view autoinst-log.txt for the openQA worker and test-engine log. This is the primary log for the job itself.

For tests that execute commands on the system under test, KDE Linux runs them in transient systemd services and records the service journal as diagnostic output in the test result. Inspect autoinst-log.txt for the command output and journal. You can also download the *kde-linux-collected-logs.tar.zst file for logs captured by the collect-logs tool.

Run tests locally

You can use openQA to test a locally built image before submitting a change. This is required when working from a fork outside kde-linux/kde-linux - its CI pipeline builds a test image, but does not trigger the openQA pipeline. It is also useful while developing tests for your own changes.

The KDE Linux openQA test repository includes a local stack containing both an openQA web UI and a worker. It requires Python 3.14, Podman, and podman-compose, which are included in KDE Linux.

Clone the test repository, then place the KDE Linux .iso image you want to test in its root directory. If no image is present, the test runner downloads the latest publicly available image instead.

Start the local stack from the repository root:

./qa-mock up

Once it is ready, open http://localhost:1080 to inspect the openQA web UI. Jobs are not submitted automatically, so open a shell in the container and submit the install flow. Run flow, job, and build-sysext commands inside this container.

./qa-mock enter
./qa flow install

To run the upgrade flow instead:

./qa flow upgrade

For an encrypted installation, pass FDE_INSTALL=1 as an openQA setting. The flavor suffix distinguishes encrypted jobs in the web UI:

./qa flow install --var FDE_INSTALL=1 --flavor-suffix=-encrypted
./qa flow upgrade --var FDE_INSTALL=1 --flavor-suffix=-encrypted

--var NAME=VALUE can be repeated, but each variable must be declared in the selected flow's variables list in tests/tests.toml.

The sanity-test suite uses the virtual disk created by install-system. To concentrate on particular tests, edit the suite's tests list in tests/tests.toml, or declare another suite and flow there. You can also run an individual suite with ./qa job --name <suite> while keeping its disk dependencies in mind. The flow, job, and build-sysext commands accept --manifest-path <path> to use a manifest other than tests/tests.toml. Test paths within the manifest are relative to that manifest's directory. Run ./qa job --help for its required options.

For example, build the system extension and run an install-system job with an .iso in the repository directory. Replace the build ID and variant with those of your image:

./qa build-sysext
./qa job \
    --live ./kde-linux_<build-id>.iso \
    --hdd kde-linux_<build-id>.qcow2 \
    --sysext ./openqa-sysext.img \
    --build <build-id> \
    --variant <variant> \
    --name install-system \
    --flavor live

When you are finished, exit the container shell, then stop the stack and remove its local volumes from the host.

./qa-mock down -v

./qa-mock passes through arguments to podman-compose. For example, use ./qa-mock up -d to start the stack in the background.

Test upgrades between local builds

Place the base ISO in the test repository root. When starting the stack from the host, set KDE_LINUX_OUTPUT to the mkosi.output directory containing the build you want to upgrade to:

KDE_LINUX_OUTPUT=~/Repositories/kde-linux/mkosi.output ./qa-mock up

Enter the container and run the upgrade flow, replacing &lt;build-id&gt; with the build ID of your base ISO:

./qa-mock enter
./qa flow upgrade \
    --upgrade-from /casedir/kde-linux_<build-id>.iso \
    --upgrade-to /kde-linux-output

Both options must be supplied together. The repository is mounted at /casedir, and the selected mkosi.output directory is mounted read-only at /kde-linux-output. The newest build in that directory is served to the virtual machine as a temporary sysupdate source and must be newer than the base ISO. Add --var FDE_INSTALL=1 --flavor-suffix=-encrypted to test an encrypted installation.

Declare suites and flows

tests/tests.toml defines the distribution (name and distri), suites, and flows. A suite is an ordered list of tests run in one openQA job. Its action determines how the job uses the virtual disk:

  • install boots the live image and creates the installed disk.
  • upgrade boots the installed disk and sets DO_UPGRADE for tests that perform the upgrade.
  • test boots the installed disk for validation.

A flow runs suites in the declared order, passing the installed disk between jobs. KDE Linux defines these flows:

[flows.install]
suites = ["install-system", "sanity-test"]
variables = ["FDE_INSTALL"]

[flows.upgrade]
suites = ["install-system", "upgrade-system", "sanity-test"]
variables = ["FDE_INSTALL"]

The install flow installs and checks the build under test. The upgrade flow first installs an older image, upgrades it to the build under test, and then runs sanity tests. For a staged CI build, the base is the latest published image. Without --upgrade-from and --upgrade-to, a local upgrade run will default to the latest public build as its target, using an older local ISO if one is available or downloading the previous image.

Before each job, the worker generates a temporary test directory (CASEDIR) with a main.pm schedule for the selected suite. Tests run in manifest order, including repeated entries. Edit the manifest to change the schedule.

Write a test

The test definitions and code live in os-autoinst-distri-kdelinux. Add test scripts under tests/, alongside related tests, and register them in the appropriate suite's tests array in tests/tests.toml. For example:

[suites.example]
action = "test"
tests = [
    { path = "common/bootup.py", type = "openqa" },
    { path = "common/basic_test.py", type = "python" },
    { path = "kdelinux/app/firefox.py", type = "selenium", user = "installed", timeout = 180 },
]

openqa tests execute on the worker and use openQA APIs directly. python and selenium tests execute on the system under test, the system extension packages their scripts automatically, and the worker generates the openQA test wrappers invoking them.

Choose the test type based on what you are checking:

  • Use a normal Python unittest for command-line tools, services, filesystem state, or other non-graphical behaviour.
  • Use Selenium through selenium-webdriver-at-spi when the behaviour requires interacting with a graphical application.

Normal Python tests

Create a unittest under tests/, for example tests/kdelinux/system/example.py. It should make assertions about the system under test and write its results to a JUnit XML file, which will be automatically collected as an openQA asset and a GitLab CI report.

import unittest
from lib.sut import openqa_junit_xml

class ExampleTests(unittest.TestCase):
    def test_expected_behaviour(self):
        self.assertTrue(True)

if __name__ == "__main__":
    openqa_junit_xml.run(ExampleTests, "example")

Add an entry to the relevant suite's tests array:

{ path = "kdelinux/system/example.py", type = "python" },

The generated wrapper collects the JUnit result and command output so that failures appear in the openQA job and CI artifacts.

Graphical tests with Selenium

Graphical tests are also Python unittest classes, but use the Appium/Selenium driver to find and interact with accessible UI elements. The Selenium runner enables the accessibility infrastructure and starts the driver for you.

For example, a graphical test creates a driver for its application, interacts with accessible elements, and closes the driver afterwards:

import unittest
from appium import webdriver
from appium.options.common.base import AppiumOptions
from appium.webdriver.common.appiumby import AppiumBy
from lib.sut import openqa_junit_xml

class ExampleTests(unittest.TestCase):
    @classmethod
    def setUpClass(cls):
        options = AppiumOptions()
        options.set_capability("app", "org.kde.example.desktop")
        cls.driver = webdriver.Remote("http://127.0.0.1:4723", options=options)

    @classmethod
    def tearDownClass(cls):
        cls.driver.quit()

    def test_expected_behaviour(self):
        self.driver.find_element(AppiumBy.ACCESSIBILITY_ID, "example-control").click()

if __name__ == "__main__":
    openqa_junit_xml.run(ExampleTests, "example")

Save the script as, for example, tests/kdelinux/app/example.py, and add it to the relevant suite with type = "selenium". Select the installed test user for desktop applications:

{ path = "kdelinux/app/example.py", type = "selenium", user = "installed", timeout = 180 },

Run the flow containing your suite locally while developing it.

Test options and custom wrappers

For generated SUT wrappers, manifest entries can set user (root, live, installed, or plasma_setup), timeout in seconds (defaults to 90), artifacts as a list of paths to collect, and the openQA flags fatal = true or always_run = true. Set enabled = false to leave a test declared while excluding it from the schedule.

When a test needs some form of custom orchestration to run, put a wrapper beside its SUT script and point to it with override:

{ path = "kdelinux/system/collect_logs.py", type = "selenium", override = "kdelinux/system/collect_logs.openqa.py" },

This custom wrapper is responsible for its flags, user, timeout, and artifact collection, and must pass the SUT script's relative path to CliTest.run_python() or CliTest.run_selenium(). See the existing collect_logs.openqa.py for an example.

For more information on writing Selenium tests, see Appium automation testing and GUI Testing with selenium-webdriver-at-spi.

Learn more

For a more detailed explanation of the test flow and its implementation, see openQA Testing in KDE Linux. The test infrastructure is developed in the os-autoinst-distri-kdelinux repository.

Article contributed by under the CC-BY-4.0 license.