openQA for Developers and Maintainers

KDE Linux uses openQA to automatically test every system image before it is published. openQA boots each image in a virtual machine and checks that it can be installed, upgraded, and used successfully.

openQA has some specific terminology: a job is one execution of a test suite against an image, while a worker is the machine or container that runs the job. The openQA documentation explains these and other concepts, including test modules, job groups, assets, and the web UI.

How the tests work

The test worker runs alongside the CI pipeline, so it can use the images and virtual disks produced by the build without uploading those large files elsewhere. Tests are supplied to the virtual machine through a system extension and communicate with it over SSH.

Desktop tests use the accessibility API through selenium-webdriver-at-spi, allowing them to interact with applications and check their state without relying solely on screenshot matching. The suite also checks for failed system services, crashed desktop processes, networking problems, and regressions in KDE Linux commands.

CI pipeline and maintainer workflow

On the protected default branch of the upstream KDE Linux repository, the pipeline works as follows:

  1. The imaging job builds, signs, and stages the image and its sysupdate channel on storage.kde.org.
  2. The trigger-openqa job starts the pipeline in os-autoinst-distri-kdelinux, passing the staged image URL and sysupdate channel URL to it.
  3. The downstream pipeline runs the normal install-and-sanity flow and the upgrade flow.
  4. The upstream publish job runs only after the openQA pipeline has succeeded. It promotes the already-staged image for public consumption.

This makes an openQA failure a gate for image publish. Start by opening the downstream pipeline from the trigger-openqa job, then open the failing openQA job from its CI log. The job page shows the test modules in order and lets you identify whether the failure is in installation, the installed-system sanity tests, or the upgrade path.

Download a staged image

The link provided after the log message “In case of failure, you can inspect and download the .iso image and sysupdate tree at…” opens a storage browser for the build under test. Download the .iso there to boot it manually or give it to a local openQA stack. The sysupdate tree contains the staged update artifacts used by the upgrade test.

Investigate a failed job

Open the job in the openQA web UI and select the failed module first. The link to the web UI will appear after the “ test job is now running.” log message. Its details include the module result, screenshots or video where applicable, and diagnostic output. In the Logs & Assets tab, download or view autoinst-log.txt for the openQA worker and test-engine log. This is the primary log for the job itself.

For tests that execute commands on the system under test, KDE Linux runs them in transient systemd services and records the service journal as diagnostic output in the test result. Inspect autoinst-log.txt for the command output and journal. You can also download the *kde-linux-collected-logs.tar.zst file for logs captured by the collect-logs tool.

Run tests locally

You can use openQA to test a locally built image before submitting a change. This is required when working from a fork outside kde-linux/kde-linux - its CI pipeline builds a test image, but does not trigger the openQA pipeline. It is also useful while developing tests for your own changes.

The KDE Linux openQA test repository includes a local stack containing both an openQA web UI and a worker. It requires Podman and podman-compose, which are included in KDE Linux.

Clone the test repository, then place the KDE Linux .iso image you want to test in its root directory. If no image is present, the test runner downloads the latest publicly available image instead.

Start the local stack.

./mock.sh up

Once it is ready, open http://localhost:1080 to inspect the openQA web UI. Jobs are not submitted automatically, so, open a shell in the container and submit the normal install and sanity-test flow.

podman exec -it openqa-single-instance bash
bash utils/jobs.sh

To run the upgrade flow instead, add --upgrade:

bash utils/jobs.sh --upgrade

To run the disk-encryption install-test flow, add --encrypt:

bash utils/jobs.sh --encrypt

You can also combine --upgrade and --encrypt options:

bash utils/jobs.sh --upgrade --encrypt

The sanity-test job needs the install-system job to run first because it uses the virtual disk created by the installation. To concentrate on a particular test, adjust the submitted jobs in utils/jobs.sh while keeping that dependency in mind.

When you are finished, stop the stack and remove its local volumes.

./mock.sh down -v

Write a test

The test definitions and the test code live in os-autoinst-distri-kdelinux. Add the test to the appropriate flow in main.pm - the live-install flow, installed-system sanity flow, or upgrade flow. Tests are generally composed of a small openQA wrapper in tests/ and a Python test that runs on the system under test from extensions/openqa/usr/lib/kde-linux-openqa/tests/.

Choose the test type based on what you are checking:

  • Use a normal Python unittest for command-line tools, services, filesystem state, or other non-graphical behaviour.
  • Use Selenium through selenium-webdriver-at-spi when the behaviour requires interacting with a graphical application.

Normal Python tests

Create a unittest in the aforementioned system extension test directory. It should make assertions about the system under test and write its results to a JUnit XML file, which will be automatically collected as an openQA asset and a GitLab CI report.

import unittest
from lib.sut import openqa_junit_xml

class ExampleTests(unittest.TestCase):
    def test_expected_behaviour(self):
        self.assertTrue(True)

if __name__ == "__main__":
    openqa_junit_xml.run(ExampleTests, "example")

Create a matching wrapper under tests/ that gets openQA to execute the test on the host.

from testapi import *
from lib.openqa import cli_test

def run(self):
    cli_test.CliTest("example").run_python()

Finally, register the wrapper in the relevant flow in main.pm. The wrapper collects the JUnit result and command output so that failures appear in the openQA job and CI artifacts.

Graphical tests with Selenium

Graphical tests are also Python unittest classes, but use the Appium/Selenium driver to find and interact with accessible UI elements. The Selenium runner enables the accessibility infrastructure and starts the driver for you.

For example, a graphical test creates a driver for its application, interacts with accessible elements, and closes the driver afterwards:

import unittest
from appium import webdriver
from appium.options.common.base import AppiumOptions
from appium.webdriver.common.appiumby import AppiumBy
from lib.sut import openqa_junit_xml

class ExampleTests(unittest.TestCase):
    @classmethod
    def setUpClass(cls):
        options = AppiumOptions()
        options.set_capability("app", "org.kde.example.desktop")
        cls.driver = webdriver.Remote("http://127.0.0.1:4723", options=options)

    @classmethod
    def tearDownClass(cls):
        cls.driver.quit()

    def test_expected_behaviour(self):
        self.driver.find_element(AppiumBy.ACCESSIBILITY_ID, "example-control").click()

if __name__ == "__main__":
    openqa_junit_xml.run(ExampleTests, "example")

Run the matching wrapper with run_selenium() instead of run_python(). Pass the installed test user for desktop applications:

from testapi import *
from lib.openqa import cli_test
from lib import user_manager

def run(self):
    cli_test.CliTest("example").run_selenium(user=user_manager.installed())

Similarly to normal Python tests, add its wrapper to main.pm, run it locally while developing it, and include it in the test flow that exercises the feature.

For more information on writing Selenium tests, see Appium automation testing and GUI Testing with selenium-webdriver-at-spi.

Learn more

For a more detailed explanation of the test flow and its implementation, see openQA Testing in KDE Linux. The test infrastructure is developed in the os-autoinst-distri-kdelinux repository.

Article contributed by under the CC-BY-4.0 license.