Setting up the OCI Runtime Spec test suite

The Open Container Initiative¹ (OCI) provides a collection of tools for working with the OCI Runtime Specification². Of particular interest to us, is the runtime validation suite, which can be used to validate container runtimes according to the OCI Runtime Command Line Interface³.

The runtime tools can be found at https://github.com/opencontainers/runtime-tools but following the instructions doesn’t exactly work. Also, given where we are in the development process of our container runtime, we couldn’t just checkout the repo, build and run the tests, even if it were that simple.

Configuring the OCI Runtime Spec test suite

Running the tests in runtime-tools is a bit of a dirty process. We’ll create a Bash script to make it a bit easier. I’ll just assume that even if you’re not familiar with Bash, what’s going on here should be pretty easy to follow.

#!/bin/bash

if [[ -z ${RUNTIME} ]]; then
  echo "'RUNTIME' variable not set."
  exit 1
fi

if [[ -z ./runtimetest ]]; then
  echo "'runtimetest' not available."
  echo "Try running in 'runtime-tools' directory."
  exit 1
fi

logdir=/var/log/runtime-tools
mkdir -p $logdir

# tests to run
tests=(
    "create" # run the 'create' tests
)

# run tests
for test in "${tests[@]}"; do
  ./validation/${test}/${test}.t 2>&1 | tee ${logdir}/${test}.log
done

# check for failures
total_failures=0
for test in "${tests[@]}"; do
  failures=$(grep -F "not ok" ${logdir}/${test}.log | wc -l)

  if [ 0 -ne $failures ]; then 
    total_failures=$(($total_failures + $failures))
    echo "${test} - $failures"
  fi
done

if [ 0 -ne $total_failures ]; then
  echo "Total failures: $total_failures"
  exit 1
fi

The script can be executed like: sudo RUNTIME=/anocir/anocir test/scripts/oci-validation.sh

It will then run the tests specified in the tests array (for the moment, that’s just "create") against the container runtime specified in the RUNTIME environment variable. Output of the tests will be logged to /var/log/runtime-tools/<test_name>.log and results (including a summary of any failures) printed to stdout.

Hold off trying to run this for the time-being, though. First, let’s build a virtual machine to run the tests in.

Building a virtual machine

For virtualisation, I’m using Vagrant and VirtualBox. Since this is a series of posts about container runtimes, not virtual machines or related tooling, I’m not going to go into detail about either. And, of course, if you’re following along but prefer another virtualisation solution, there’s no reason why you can’t use that instead.

My Vagrantfile is below. I’ll just highlight the important things that will need to be done regardless of what solution you’re using.

Vagrant.configure("2") do |config|
    config.vm.box = "bento/ubuntu-24.04"
    config.vm.synced_folder '.', '/anocir'

    config.vm.provider "virtualbox" do |vb|
	vb.memory = "4096"
	vb.cpus = "2"
    end

    config.vm.provision "shell", inline: <<-SHELL
	set -e -x -o pipefail

	# Update package lists and install packages
	apt-get update && apt-get install -y git ca-certificates wget make vim gcc libseccomp-dev

	# Install go
	if ! command -v go 2>&1 >/dev/null; then
	    wget https://go.dev/dl/go1.23.4.linux-amd64.tar.gz -O go.tar.gz
	    tar -C /usr/local -xzf go.tar.gz
	    echo "PATH=$PATH:/usr/local/go/bin" >> /etc/environment
	fi

	# Clone runtime-tools repo
	if [ ! -d "runtime-tools" ]; then
	    git clone https://github.com/opencontainers/runtime-tools.git
	fi

	# Checkout a specific tree so it's not potentially changing underneath us
	git -C runtime-tools checkout f7e3563b0271e5cd52d5c915684ea11ef2779572

	# Build runtime tools
	(cd runtime-tools && make runtimetest validation-executables)
    SHELL
end

When configuring our virtual machine, we want to mount our project directory inside the VM. In my case, that’s at /anocir.

When provisioning the VM, we (probably) need to install some additional packages. Exactly what those are is going to depend on the distro you’re using in the VM. I’m using Ubuntu, so that’s git, ca-certificates, wget, make, vim, gcc, libseccomp-dev. Check you have equivalent installed.

Of course, we’ll also need to have Go installed.

The runtime-tools repo (https://github.com/opencontainers/runtime-tools.git) is where the tests are located, so we’ll need to clone that and checkout the f7e3563b0271e5cd52d5c915684ea11ef2779572 tree. There’s nothing particularly special about that tree; it’s just a stable point so that as-and-when changes go into the repo things aren’t shifting underneath us.

Finally, we can build the runtimetest and validation-executables make targets that will be used to run the tests.

Running the tests

You’ll now want to provision and bring up the virtual machine so that we can run the tests inside it. If you’re using Vagrant, just run vagrant up --provision. Now, connect to the box; with Vagrant, that’s vagrant ssh.

The moment of truth: build the anocir binary and run the test script against it.

cd /anocir
go build -o anocir

cd /home/vagrant/runtime-tools
sudo RUNTIME=/anocir/anocir /anocir/test/scripts/oci-integration.sh

TAP version 13
ok 1 - create MUST generate an error if the ID is not provided
  ---
  {
    "error": "exit status 1",
    "reference": "https://github.com/opencontainers/runtime-spec/blob/v1.1.0/runtime.md#create",
    "stderr": "Error: accepts 1 arg(s), received 0\n"
  }
  ...
ok 2 - create MUST create a new container
  ---
  {
    "reference": "https://github.com/opencontainers/runtime-spec/blob/v1.1.0/runtime.md#create"
  }
  ...
ok 3 - 'state' MUST return the state of a container
  ---
  {
    "container ID": "6f42dc0b-2fb1-4ee2-8ef7-ee371cd5dd49",
    "state ID": "6f42dc0b-2fb1-4ee2-8ef7-ee371cd5dd49"
  }
  ...
ok 4 - create MUST generate an error if the ID provided is not unique
  ---
  {
    "error": "exit status 1",
    "reference": "https://github.com/opencontainers/runtime-spec/blob/v1.1.0/runtime.md#create",
    "stderr": "Error: create container: container '6f42dc0b-2fb1-4ee2-8ef7-ee371cd5dd49' exists\n"
  }
  ...
1..4

Assuming everything was set up correctly, we should see output like the above - 4 “ok” test results for the create operation!

Looking at the list of available tests, we might assume that we could enable tests for hooks, kill or others. Unfortunately, the implementation of those tests depends on other features of the container runtime that we haven’t implemented. Those features, and the remaining features to complete our container runtime, are what we’ll start on next…

References