Running a single Ceph QA (Teuthology) job in containers

Posted on September 2, 2015 by ivotron

tl;dr: This describes how to run teuthology jobs using docker in three "easy" steps, with the goal of shortening the develop/build/test cycle for integration-tested code.

Teuthology is Ceph's integration testing framework. It is an awesome tool that allows to test many of the unique characteristics of Ceph (testing distributed systems is hard!)

Currently, if one wants to execute a test on the sepia lab, one has to wait for the git.ceph.com mirror to catch up with github, then wait for the gitbuilder to finish with distro-specific packages, etc. A test that runs at a ceph lab is not only testing the piece of code we've just modified but also all the components of this long pipeline (package building, installation, etc.). This is great since it allows to catch bugs early at any stage but has implication of having to wait (between 25-40 mins.) for a single job to execute.

There are existing efforts to shorten the dev/build/test cycle, like John Spray's vstart runner for cephfs and Loic Dachary's OpenStack backend for teuthology. This is another approach that relies on containers for executing a single teuthology job using containers:

  1. Write a file containing an entire teuthology job (tasks, targets and roles in a YAML file):

    sshkeys: ignore
    roles:
    - [mon.0, osd.0, osd.1, osd.2, client.0]
    tasks:
    - install.ship_utilities:
    - ceph:
        conf:
          mon:
            debug mon: 20
            debug ms: 1
            debug paxos: 20
          osd:
            debug filestore: 20
            debug journal: 20
            debug ms: 1
            debug osd: 20
    - radosbench:
        clients: [client.0]
    targets:
      'root@localhost:2222': ssh-dss ignored

    The sshkeys option is required and install.ship_utilities should be the first task to execute. Also, ~/.teuthology.yaml should look like this:

    lab_domain: ''
    lock_server: ''
  2. Initialize a cephdev container (the following assumes $PWD is the folder containing the ceph code in your machine):

    docker run \
      --name remote0
      -p 2222:22
      -d -e AUTHORIZED_KEYS="`cat ~/.ssh/id_rsa.pub`" \
      -v `pwd`:/ceph \
      -v /dev:/dev \
      -v /tmp/ceph_data/$RANDOM:/var/lib/ceph \
      --cap-add=SYS_ADMIN --privileged \
      --device /dev/fuse
      ivotron/cephdev
  3. Execute teuthology using the wip-11892-docker branch:

    teuthology \
      -a ~/archive/`date +%s` \
      --suite-path /path/to/ceph-qa-suite/ \
      ~/test.yml

Caveats: