Building a container runtime from scratch in Go — nixpig.dev

Setting up the OCI Runtime Spec test suite

Mon, 03 Feb 2025 07:27:08 +0000

The Open Container Initiative¹ (OCI) provides a collection of tools for working with the OCI Runtime Specification². Of particular interest to us, is the runtime validation suite, which can be used to validate container runtimes according to the OCI Runtime Command Line Interface³.

List of runtime-tools tests (click to collapse/expand)

default
config_updates_without_affect
create
delete
hooks
hooks_stdin
hostname
kill
kill_no_effect
killsig
linux_devices
linux_masked_paths
linux_mount_label
linux_ns_itype
linux_ns_nopath
linux_ns_path
linux_ns_path_type
linux_process_apparmor_profile
linux_readonly_paths
linux_rootfs_propagation
linux_seccomp
linux_sysctl
linux_uid_mappings
misc_props
mounts
pidfile
poststart
poststart_fail
poststop
poststop_fail
prestart
prestart_fail
process
process_capabilities
process_capabilities_fail
process_oom_score_adj
process_rlimits
process_rlimits_fail
process_user
root_readonly_true
start
state
delete_resources
delete_only_create_resources
linux_cgroups_blkio
linux_cgroups_cpus
linux_cgroups_devices
linux_cgroups_hugetlb
linux_cgroups_memory
linux_cgroups_network
linux_cgroups_pids
linux_cgroups_relative_blkio
linux_cgroups_relative_cpus
linux_cgroups_relative_devices
linux_cgroups_relative_hugetlb
linux_cgroups_relative_memory
linux_cgroups_relative_network
linux_cgroups_relative_pids

The runtime tools can be found at https://github.com/opencontainers/runtime-tools but following the instructions doesn’t exactly work. Also, given where we are in the development process of our container runtime, we couldn’t just checkout the repo, build and run the tests, even if it were that simple.

Configuring the OCI Runtime Spec test suite

Running the tests in runtime-tools is a bit of a dirty process. We’ll create a Bash script to make it a bit easier. I’ll just assume that even if you’re not familiar with Bash, what’s going on here should be pretty easy to follow.

test/scripts/oci-integration.sh

#!/bin/bash

if [[ -z ${RUNTIME} ]]; then
 echo "'RUNTIME' variable not set."
 exit 1
fi

if [[ -z ./runtimetest ]]; then
 echo "'runtimetest' not available."
 echo "Try running in 'runtime-tools' directory."
 exit 1
fi

logdir=/var/log/runtime-tools
mkdir -p $logdir

# tests to run
tests=(
 "create" # run the 'create' tests
)

# run tests
for test in "${tests[@]}"; do
 ./validation/${test}/${test}.t 2>&1 | tee ${logdir}/${test}.log
done

# check for failures
total_failures=0
for test in "${tests[@]}"; do
 failures=$(grep -F "not ok" ${logdir}/${test}.log | wc -l)

 if [ 0 -ne $failures ]; then 
 total_failures=$(($total_failures + $failures))
 echo "${test} - $failures"
 fi
done

if [ 0 -ne $total_failures ]; then
 echo "Total failures: $total_failures"
 exit 1
fi

The script can be executed like: sudo RUNTIME=/anocir/anocir test/scripts/oci-validation.sh

It will then run the tests specified in the tests array (for the moment, that’s just "create") against the container runtime specified in the RUNTIME environment variable. Output of the tests will be logged to /var/log/runtime-tools/<test_name>.log and results (including a summary of any failures) printed to stdout.

Hold off trying to run this for the time-being, though. First, let’s build a virtual machine to run the tests in.

Building a virtual machine

Caution

It is important that we’re running tests against our runtime in a virtual machine and not on our main workstation.

Over the next few posts we’ll have partially implemented features that will be making changes to the system as the root user and (in some cases) may make changes that irreversibly break the system.

I’ll try to remember to highlight when that’s happening, but I can’t guarantee it. Just play it safe and always run the tests and the runtime in a VM during development.

For virtualisation, I’m using Vagrant and VirtualBox. Since this is a series of posts about container runtimes, not virtual machines or related tooling, I’m not going to go into detail about either. And, of course, if you’re following along but prefer another virtualisation solution, there’s no reason why you can’t use that instead.

My Vagrantfile is below. I’ll just highlight the important things that will need to be done regardless of what solution you’re using.

Vagrantfile

Vagrant.configure("2") do |config|
 config.vm.box = "bento/ubuntu-24.04"
 config.vm.synced_folder '.', '/anocir'

 config.vm.provider "virtualbox" do |vb|
	vb.memory = "4096"
	vb.cpus = "2"
 end

 config.vm.provision "shell", inline: <<-SHELL
	set -e -x -o pipefail

	# Update package lists and install packages
	apt-get update && apt-get install -y git ca-certificates wget make vim gcc libseccomp-dev

	# Install go
	if ! command -v go 2>&1 >/dev/null; then
	 wget https://go.dev/dl/go1.23.4.linux-amd64.tar.gz -O go.tar.gz
	 tar -C /usr/local -xzf go.tar.gz
	 echo "PATH=$PATH:/usr/local/go/bin" >> /etc/environment
	fi

	# Clone runtime-tools repo
	if [ ! -d "runtime-tools" ]; then
	 git clone https://github.com/opencontainers/runtime-tools.git
	fi

	# Checkout a specific tree so it's not potentially changing underneath us
	git -C runtime-tools checkout f7e3563b0271e5cd52d5c915684ea11ef2779572

	# Build runtime tools
	(cd runtime-tools && make runtimetest validation-executables)
 SHELL
end

When configuring our virtual machine, we want to mount our project directory inside the VM. In my case, that’s at /anocir.

When provisioning the VM, we (probably) need to install some additional packages. Exactly what those are is going to depend on the distro you’re using in the VM. I’m using Ubuntu, so that’s git, ca-certificates, wget, make, vim, gcc, libseccomp-dev. Check you have equivalent installed.

Of course, we’ll also need to have Go installed.

The runtime-tools repo (https://github.com/opencontainers/runtime-tools.git) is where the tests are located, so we’ll need to clone that and checkout the f7e3563b0271e5cd52d5c915684ea11ef2779572 tree. There’s nothing particularly special about that tree; it’s just a stable point so that as-and-when changes go into the repo things aren’t shifting underneath us.

Finally, we can build the runtimetest and validation-executables make targets that will be used to run the tests.

Running the tests

You’ll now want to provision and bring up the virtual machine so that we can run the tests inside it. If you’re using Vagrant, just run vagrant up --provision. Now, connect to the box; with Vagrant, that’s vagrant ssh.

The moment of truth: build the anocir binary and run the test script against it.

Build the 'anocir' binary

cd /anocir
go build -o anocir

Run the runtime-tools tests

cd /home/vagrant/runtime-tools
sudo RUNTIME=/anocir/anocir /anocir/test/scripts/oci-integration.sh

TAP version 13
ok 1 - create MUST generate an error if the ID is not provided
 ---
 {
 "error": "exit status 1",
 "reference": "https://github.com/opencontainers/runtime-spec/blob/v1.1.0/runtime.md#create",
 "stderr": "Error: accepts 1 arg(s), received 0\n"
 }
 ...
ok 2 - create MUST create a new container
 ---
 {
 "reference": "https://github.com/opencontainers/runtime-spec/blob/v1.1.0/runtime.md#create"
 }
 ...
ok 3 - 'state' MUST return the state of a container
 ---
 {
 "container ID": "6f42dc0b-2fb1-4ee2-8ef7-ee371cd5dd49",
 "state ID": "6f42dc0b-2fb1-4ee2-8ef7-ee371cd5dd49"
 }
 ...
ok 4 - create MUST generate an error if the ID provided is not unique
 ---
 {
 "error": "exit status 1",
 "reference": "https://github.com/opencontainers/runtime-spec/blob/v1.1.0/runtime.md#create",
 "stderr": "Error: create container: container '6f42dc0b-2fb1-4ee2-8ef7-ee371cd5dd49' exists\n"
 }
 ...
1..4

Assuming everything was set up correctly, we should see output like the above - 4 “ok” test results for the create operation!

Looking at the list of available tests, we might assume that we could enable tests for hooks, kill or others. Unfortunately, the implementation of those tests depends on other features of the container runtime that we haven’t implemented. Those features, and the remaining features to complete our container runtime, are what we’ll start on next…

References

https://opencontainers.org/ “Open Container Initiative” ↩︎
https://github.com/opencontainers/runtime-spec/blob/main/spec.md “OCI Runtime Specification” ↩︎
https://github.com/opencontainers/runtime-tools/blob/master/docs/command-line-interface.md “OCI Runtime Command Line Interface” ↩︎

Sending signals to a running container using 'kill'

Tue, 28 Jan 2025 15:17:20 +0000

A container on Linux is simply a running process that has some particular properties associated with it (cgroups, namespaces, etc…). As such, it can be interacted with much like any other process on the system, aforementioned properties not withstanding.

Sending signals to a container process is handled by the container runtime’s kill operation. The kill operation of the container runtime is analogous to the Linux kill command, so let’s have a quick refresher on that.

Note

In fact, we’ll use the kill syscall to implement the kill operation for our container runtime.

Signals and the Linux `kill` command

Signals¹ are a kind of software interrupt that is sent asynchronously to a process to inform it of an event. The process may then take action based on the signal it received.

A signal can be sent to a process using a number of methods - raise, kill, sigqueue, and others, including user input, like pressing CTRL+C in a terminal. For our purposes, we’re going to be focusing on kill².

The kill command takes in the PID of a process to send a signal to and the signal to send.

Send kill (9) signal to process with PID 40135

❯ kill 40135 -9

Tip

The Chromium Project has a pretty decent Linux Signal Table describing the available signals.

We can get a list of signals and their names by running kill -l. Appending a signal name or number will get the corresponding number or name, respectively.

Get signal numbers and names using 'kill -l'

❯ kill -l
 1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL 5) SIGTRAP
 6) SIGABRT 7) SIGBUS 8) SIGFPE 9) SIGKILL 10) SIGUSR1
11) SIGSEGV 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM
16) SIGSTKFLT 17) SIGCHLD 18) SIGCONT 19) SIGSTOP 20) SIGTSTP
21) SIGTTIN 22) SIGTTOU 23) SIGURG 24) SIGXCPU 25) SIGXFSZ
26) SIGVTALRM 27) SIGPROF 28) SIGWINCH 29) SIGIO 30) SIGPWR
31) SIGSYS 34) SIGRTMIN 35) SIGRTMIN+1 36) SIGRTMIN+2 37) SIGRTMIN+3
38) SIGRTMIN+4 39) SIGRTMIN+5 40) SIGRTMIN+6 41) SIGRTMIN+7 42) SIGRTMIN+8
43) SIGRTMIN+9 44) SIGRTMIN+10 45) SIGRTMIN+1146) SIGRTMIN+12 47) SIGRTMIN+13
48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-1451) SIGRTMAX-13 52) SIGRTMAX-12
53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9 56) SIGRTMAX-8 57) SIGRTMAX-7
58) SIGRTMAX-6 59) SIGRTMAX-5 60) SIGRTMAX-4 61) SIGRTMAX-3 62) SIGRTMAX-2
63) SIGRTMAX-1 64) SIGRTMAX

❯ kill -l stop
19

❯ kill -l 9
KILL

The 0 signal, which isn’t listed above (or even usually in signal tables), is a special case in that it can be used to check whether a process exists, assuming that we have the necessary privileges (more on these later) to send it a signal.

Check whether a process exists

# process exists
❯ kill 37174 -0

# process doesn't exist
❯ kill 343453 -0
bash: kill: (343453) - No such process

# process exists but don't have permissions
❯ kill 40135 -0
bash: kill: (40135) - Operation not permitted

When a signal is sent to a process, the process is able to catch, ignore or block it, with the exception of SIGSTOP and SIGKILL. If a process chooses not to catch a process then the kernel will take the default action for that signal (which is often to terminate the process).

We can see what action a process is going to take when it receives a signal, and any signals pending, by inspecting the relevant fields in the process’ status file.

Inspecting status of PID 1

❯ grep ^Sig...: /proc/1/status

SigPnd: 0000000000000000
SigBlk: 7fefc1fe28014a03
SigIgn: 0000000000001000
SigCgt: 00000000000004ec

SigPnd indicates signals that have been raised but have yet to be acted on.
SigBlk indicates signals that are blocked.
SigIgn indicates signals that are ignored.
SigCgt indicates signals that can be caught.

The value to the right of the identifier is a hex number which, when converted to binary, is a bitmask of the signals. Taking SigCgt from above as an example:

Getting signals from a hex-encoded bitmask

# https://nixpig.dev/til/convert-binary-hex/

0 0 0 0 0 0 0 0 0 0 0 0 0 4 e c
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0100 1110 1100

0100 1110 1100
 │ │││ ││
 │ │││ ││
 │ │││ │└── 3 SIGQUIT
 │ │││ └─── 4 SIGILL
 │ │││
 │ ││└────── 6 SIGABRT
 │ │└─────── 7 SIGBUS
 │ └──────── 8 SIGFPE
 │
 │
 └──────────── 11 SIGSEV
 

This indicates that the process will catch SIGQUIT, SIGILL, SIGABRT, SIGBUS, SIGFPE and SIGSEV signals.

Sending a signal to a container process

As previously mentioned, the kill operation is used to send a signal to a running container process.

Since it can only be executed against a container that’s in either the created or running state, we’ll create a helper function to check if the container can be ‘killed’.

internal/container/container.go

func (c *Container) canBeKilled() bool {
 return c.State.Status == specs.StateRunning ||
	c.State.Status == specs.StateCreated
}

Now we can implement the kill functionality on the container.

internal/container/container.go

func (c *Container) Kill(sig unix.Signal) error {
 if !c.canBeKilled() {
	return fmt.Errorf("container cannot be killed in current state (%s)", c.State.Status)
 }

 if err := syscall.Kill(c.State.Pid, sig); err != nil {
	return fmt.Errorf("send signal '%d' to process '%d': %w", sig, c.State.Pid, err)
 }

 c.State.Status = specs.StateStopped

 if c.Spec.Hooks != nil {
	if err := hooks.ExecHooks(
	 c.Spec.Hooks.Poststop, c.State,
	); err != nil {
	 fmt.Println("Warning: failed to execute poststop hooks")
	}
 }

 return nil
}

Kill receives a unix.Signal to send to the container process.

First, we check if the container is in a state that can be killed. Then, we issue the Kill syscall to the PID of the container process which we get from the state. After updating the container status to stopped, we execute the poststop hook.

Finally, we just need to hook it up to the kill operation.

internal/operations/kill.go

func Kill(opts *KillOpts) error {
 cntr, err := container.Load(opts.ID)
 if err != nil {
	return fmt.Errorf("load container: %w", err)
 }

 sig, err := strconv.Atoi(opts.Signal)
 if err != nil {
	return fmt.Errorf("convert signal to int: %w", err)
 }

 if err := cntr.Kill(unix.Signal(sig)); err != nil {
	return fmt.Errorf("kill container: %w", err)
 }

 if err := cntr.Save(); err != nil {
	return fmt.Errorf("save container: %w", err)
 }

 return nil
}

As with other operations, we first load the container using it’s ID. The signal is passed via the CLI, meaning it will be a string, so we convert it to an int. We then send that signal by calling cntr.Kill. Finally, we save the latest state of the container.

To test that the kill operation is working, we’ll first need to update the config.json from which we’re creating a container to execute a long-running process. We can just run sleep 60, which will ‘wait’ for 60 seconds - plenty of time.

alpinefs/config.json

{
 // ...

 "process": {
	"args": ["sleep", "60"],
	"env": ["PATH=/bin"],
	"user": {
	"uid": 10,
	"gid": 10
	},
	"cwd": "/"
 },

 // ...
}

To give it a try, lets first create and start a container.

Create a container

❯ ./anocir create --bundle alpinefs test1

Start the container

❯ ./anocir start test1

Then, check the state of the container.

Check the state of the container

❯ ./anocir state test1

{
 "ociVersion": "1.2.0",
 "id": "test1",
 "status": "running",
 "pid": 87799,
 "bundle": "/home/nixpig/projects/alpinefs"
}

Note that the container is running.

We can also get details of the process using ps and the pid from the state.

Get process using 'ps -p <pid>'

❯ ps -p 100249

 PID TTY TIME CMD
100249 ? 00:00:00 sleep

Now, send a SIGKILL signal to the container process.

Send SIGKILL to the container process

# SIGKILL = 9
❯ ./anocir kill test1 9

…and check the state of the container again to see that the container is stopped.

Check the state of the container

❯ ./anocir state test1

{
 "ociVersion": "1.2.0",
 "id": "test1",
 "status": "stopped",
 "pid": 87799,
 "bundle": "/home/nixpig/projects/alpinefs"
}

…and verify the process has indeed gone, using ps.

Get process using 'ps -p <pid>'

❯ ps -p 100249

 PID TTY TIME CMD
# nothing to see here

So far, for testing, we’ve just been running commands ourselves - creating, starting, stopping, deleting containers, inspecting them via querying their state and using external tools, like ps. But that’s rather inefficient. As we start to build out our runtime’s functionality, we’re going to want a much more efficient process for testing. To that end, in the next installment we’re going to set up the OCI Runtime test suite so we can:

Get immediate feedback on the changes we’re making to ensure the features we’re implementing are doing what we expect them to, without introducting regressions.
Validate that the features we’re implementing are satisfying the requirements of the OCI Runtime Spec.

Part 7: Setting up the OCI Runtime Spec test suite 🔜 Coming soon!

References

https://man7.org/linux/man-pages/man7/signal.7.html “signal(7) — Linux manual page” ↩︎
https://man7.org/linux/man-pages/man1/kill.1.html “kill(1) - Linux manual page” ↩︎

Executing container runtime lifecycle hooks

Tue, 21 Jan 2025 14:01:58 +0000

Throughout the lifecycle¹ of a container, the runtime will execute hooks² according to the phase of the lifecycle. Hooks are simply commands that run in either the runtime or container namespace.

Hook	Namespace	Lifecycle	Use
`prestart`	Runtime	During the creation of the container, after container namespaces are created and before root mount is applied.	Typically used to customise the container.
`createRuntime`	Runtime	During the creation of the container, after container namespaces are created, before the `createContainer` hooks are called and before root mount is applied.	Typically used to customise the container.
`createContainer`	Container	During the creation of the container, after container mount namespace is created, after the `createRuntime` hooks are called and before root mount is applied.
`startContainer`	Container	When starting the container, before the user-specified process is started.	Typically used to execute operations in the container before starting the user-specified process.
`poststart`	Runtime	Immediately after starting the user-specified process without waiting for it to return.	Typically used to notify callees that the container process has started.
`poststop`	Runtime	After the container is deleted.	Typically used for cleanup.

Runtime hooks called thoughout the lifecycle of a container

You’ve probably observed a couple of ‘quirks’ above:

Some of the hook names are lowercase while others are pascal case.
The prestart and createRuntime hooks are basically the same.

In the past, there was only prestart, poststart and poststop hooks. More recently, the prestart hook was deprecated in favor of the more specific createRuntime, createContainer and startContainer hooks. However, the OCI Runtime test suite, Docker, and others still use the prestart hook, so we’re going to keep it for posterity and compatibility.

Hooks are defined in the bundle’s config.json and the spec includes the following Go structs for representing them.

github.com/opencontainers/runtime-spec/specs-go/config.go

// Hook specifies a command that is run at a particular event
// in the lifecycle of a container
type Hook struct {
 Path string `json:"path"`
 Args []string `json:"args,omitempty"`
 Env []string `json:"env,omitempty"`
 Timeout *int `json:"timeout,omitempty"`
}

// ...

type Hooks struct {
 Prestart []Hook `json:"prestart,omitempty"`
 CreateRuntime []Hook `json:"createRuntime,omitempty"`
 CreateContainer []Hook `json:"createContainer,omitempty"`
 StartContainer []Hook `json:"startContainer,omitempty"`
 Poststart []Hook `json:"poststart,omitempty"`
 Poststop []Hook `json:"poststop,omitempty"`
}

The fields in Hook should look familiar; they’re what you’d typically pass to the exec syscall. Path is the command to run, Args are the arguments to pass to the command, and Env is the environment variables. Additionally, an optional Timeout can be specified to timeout the operation after a given number of seconds.

In Hooks is a field for each of the phases of the lifecycle which hold an array of Hooks to execute.

Executing the hooks

Note

When a hook is executed, the runtime is expected to pass a JSON representation of the current state of the container to stdin.

When a hook fails to execute, it must return an error, with the exception of the poststop hook which only logs a warning.

In order to execute the hooks, we’ll create a new internal/hooks/hooks.go package that exports an ExecHooks function.

internal/hooks/hooks.go

func ExecHooks(hooks []specs.Hook, state *specs.State) error {
 s, err := json.Marshal(state)
 if err != nil {
	return fmt.Errorf("marshal state: %w", err)
 }

 for _, h := range hooks {
	ctx := context.Background()

	if h.Timeout != nil {
	 var cancel context.CancelFunc
	 ctx, cancel = context.WithTimeout(
		ctx,
		time.Duration(*h.Timeout)*time.Second,
	 )
	 defer cancel()
	}

	binary, err := exec.LookPath(h.Path)
	if err != nil {
	 return fmt.Errorf("find path of hook binary: %w", err)
	}

	path := filepath.Dir(h.Path)

	cmd := exec.CommandContext(ctx, binary, path)

	cmd.Args = append(h.Args, string(s))
	cmd.Env = h.Env
	cmd.Stdin = strings.NewReader(string(s))

	if err := cmd.Run(); err != nil {
	 return fmt.Errorf("execute hook %s: %w", h.Path, err)
	}
 }

 return nil
}

The ExecHooks function takes in a slice of Hooks to execute and a pointer to the state of the container.

The first thing we do is serialise the state of the container.

We then iterate over the hooks, and for each of them:

Create a new Context to use for the execution of the hook.
If the hook has a Timeout specified, we update the Context with a timeout.
Lookup the path to the specified executable.
Create a new Cmd with the context, executable and path to it.
Apply the Args and Env to the command.
Run the command which executes the hook.

We now need to pepper calls to this function at the correct spots in our code and pass the corresponding hooks. The hooks to execute are available to us on c.Spec.Hooks with the same name as the lifecycle phase in which they’re executing. For example, the createRuntime hooks are at c.Spec.Hooks.CreateRuntime.

Let’s go through and do that now.

internal/container/container.go

func (c *Container) Init() error {
 if c.Spec.Hooks != nil {
	if err := hooks.ExecHooks(
	 c.Spec.Hooks.CreateRuntime, c.State,
	); err != nil {
	 return fmt.Errorf("exec createruntime hooks: %w", err)
	}
 }

 cmd := exec.Command("/proc/self/exe", "reexec", c.State.ID)

 // ...

 defer listener.Close()

 if c.Spec.Hooks != nil {
	if err := hooks.ExecHooks(
	 c.Spec.Hooks.CreateContainer, c.State,
	); err != nil {
	 return fmt.Errorf("exec createcontainer hooks: %w", err)
	}
 }

 if err := cmd.Start(); err != nil {
	return fmt.Errorf("reexec container process: %w", err)
 }

 // ...

 containerConn.Close()
 listener.Close()

 if c.Spec.Hooks != nil {
	if err := hooks.ExecHooks(
	 c.Spec.Hooks.StartContainer, c.State,
	); err != nil {
	 return fmt.Errorf("exec startcontainer hooks: %w", err)
	}
 }

 bin, err := exec.LookPath(c.Spec.Process.Args[0])

 // ...

 if !c.canBeStarted() {
	return fmt.Errorf("container cannot be started in current state (%s)", c.State.Status)
 }

 if c.Spec.Hooks != nil {
	if err := hooks.ExecHooks(
	 //lint:ignore SA1019 marked as deprecated, but still required by OCI Runtime integration tests and used by other tools like Docker
	 c.Spec.Hooks.Prestart, c.State,
	); err != nil {
	 return fmt.Errorf("execute prestart hooks: %w", err)
	}
 }

 conn, err := net.Dial(
	"unix",
	filepath.Join(containerRootDir, c.State.ID, containerSockFilename),
 )

 // ...

 c.State.Status = specs.StateRunning

 if c.Spec.Hooks != nil {
	if err := hooks.ExecHooks(
	 c.Spec.Hooks.Poststart, c.State,
	); err != nil {
	 return fmt.Errorf("exec poststart hooks: %w", err)
	}
 }

 return nil
}

func (c *Container) Delete(force bool) error {

 // ...

 if err := os.RemoveAll(
	filepath.Join(containerRootDir, c.State.ID),
 ); err != nil {
	return fmt.Errorf("delete container directory: %w", err)
 }

 if c.Spec.Hooks != nil {
	if err := hooks.ExecHooks(
	 c.Spec.Hooks.Poststop, c.State,
	); err != nil {
	 fmt.Printf("Warning: failed to exec poststop hooks: %s\n", err)
	}
 }

 return nil
}

Remember, Hooks is an optional field, so we need to do a nil check before trying to call ExecHooks.

With all those in place, we’re ready to give it a try. Let’s update the config.json for the container we’ll create to include some hooks.

alpinefs/config.json

{

 // ...

 "hooks": {
	"prestart": [
	 {
		"path": "/bin/sh",
		"args": ["sh", "-c", "touch /tmp/hook-prestart"]
	 }
	],
	"createRuntime": [
	 {
		"path": "/bin/sh",
		"args": ["sh", "-c", "touch /tmp/hook-createRuntime"]
	 }
	],
	"createContainer": [
	 {
		"path": "/bin/sh",
		"args": ["sh", "-c", "touch /tmp/hook-createContainer"]
	 }
	],
	"startContainer": [
	 {
		"path": "/bin/sh",
		"args": ["sh", "-c", "rm /tmp/hook-prestart /tmp/hook-createRuntime /tmp/hook-createContainer"]
	 },
	 {
		"path": "/bin/sh",
		"args": ["sh", "-c", "touch /tmp/hook-startContainer"]
	 }
	],
	"poststart": [
	 {
		"path": "/bin/sh",
		"args": ["sh", "-c", "touch /tmp/hook-poststart"]
	 }
	],
	"poststop": [
	 {
		"path": "/bin/sh",
		"args": ["sh", "-c", "rm /tmp/hook-startContainer /tmp/hook-poststart"]
	 }
	]
 }

 // ...

}

At the various phases of the lifecycle we’re creating and deleting files so we can verify the hooks are in fact being executed.

Let’s create a container.

Create a container

./anocir create --bundle alpinefs test1

After creating a container, we expect the createRuntime and createContainer hooks to have run, which should have created corresponding temporary files.

Verify hooks have run

ll -la /tmp/hook-*

Permissions Size User Date Modified Name
.rw-r--r-- 0 root 18 Jan 16:42  /tmp/hook-createContainer
.rw-r--r-- 0 root 18 Jan 16:42  /tmp/hook-createRuntime

Next, we’ll start the container.

Start the container

./anocir start test1

After starting the container, we expect the startContainer and poststart hooks to have run, which should have deleted the temporary files for createRuntime and createContainer, and created temporary files for startContainer and poststart.

Verify hooks have run

ls -la /tmp/hook-*

Permissions Size User Date Modified Name
.rw-r--r-- 0 root 18 Jan 16:43  /tmp/hook-poststart
.rw-r--r-- 0 root 18 Jan 16:43  /tmp/hook-startContainer

Finally, we’ll delete the container.

Delete the container

./anocir delete test1 --force

On deletion of the container, we expect the poststop hook to have run, which should have deleted the startContainer and poststart temporary files.

Verify hooks have run

ls -la /tmp/hook-*

"/tmp/hook-*": No such file or directory (os error 2)

That seems to be it!

There’s still one more runtime operation that we haven’t implemented yet - kill. Let’s do that next.

Part 6: Sending signals to a running container 🔜 Coming soon!

References

https://github.com/opencontainers/runtime-spec/blob/main/runtime.md#lifecycle “Runtime lifecycle” ↩︎
https://github.com/opencontainers/runtime-spec/blob/main/config.md#posix-platform-hooks “Posix platform hooks” ↩︎

Initialising a container and starting the user process

Wed, 15 Jan 2025 07:00:06 +0000

When we start a container, the process that gets run in the container is defined in the process¹ field of the config.json of the container bundle. This contains a bunch of fields we’ll cover in future posts, but the one we’re concerned about today is args.

github.com/opencontainers/runtime-spec/specs-go/config.go

// Process contains information to start a specific application 
// inside the container.
type Process struct {
 // Args specifies the binary and arguments for the application to execute.
 Args []string `json:"args,omitempty"`

 // ...
}

The args field is an array of strings with the same semantics as the argv argument of the exec² family of Linux commands. Essentially what this means is that the first item in the array is the command to execute and the remaining items are the arguments to pass to it.

There are two steps to getting the user process defined in the config.json spec running in a containerised environment.

Initialising the container (anocir create <container-id>)
- In this step the runtime creates a container process and applies the required configuration. It then sets up an IPC channel over a Unix domain socket so that it can continue to communicate with the container once the container process is detached.
Starting the container (anocir start <container-id>)
- In this step the runtime sends a message over the IPC channel to the container process instructing the container to start. The container process applies the remaining configuration, then execs the user process defined in the config.

Let’s get to it…

Initialising the container

Below is a diagram of the steps involved in ‘initialising’ a container.

Sequence diagram of initialising a container

The user (either directly or via some higher-level runtime tooling, such as Docker) issues the anocir create <container-id> command to the runtime.
The runtime does its part of the configuration of the container based on the config.json spec.
The runtime creates a Unix domain socket to use for communication between the runtime and the container process.
The runtime listens (asynchronously) on the socket for any messages coming from the container.
The runtime ‘reexecs’, applying the configuration from step 2 to the new process.
The runtime releases the container process, and continues listening on the socket, waiting to receive a ready message from the container.
The container process does its part of the configuration based on the config.json spec.
The container process sends a ready message to the socket to indicate it’s completed configuring and is ready to receive further commands.
The container process listens (synchronously) on the socket for further instructions.
The runtime receives the ready message from the socket.
The runtime exits, leaving the container process running in the ‘background’.

Note

Steps 2 & 7 are where the bulk of our work in future posts is going to be. Today, we’ll be working on everything else above.

We’re going to be referencing a few values quite frequently, so let’s first create some constants to store them in.

internal/container/container.go

const (
 containerRootDir = "/var/lib/anocir/containers"
 initSockFilename = "init.sock"
 containerSockFilename = "container.sock"
)

The containerRootDir is the directory where the runtime is storing container metadata. The initSockFilename and containerSockFilename are the filenames of the sockets we’ll be using for IPC during initialisation and container start, respectively.

Next up, we can start creating the Init function on the container. I’ll annotate in comments the steps to which the different sections relate.

internal/container/container.go

func (c *Container) Init() error {
 // 2. configure container
 // TODO: configure container

 // 3. create ipc socket
 listener, err := net.Listen(
	"unix",
	filepath.Join(containerRootDir, c.State.ID, initSockFilename),
 )
 if err != nil {
	return fmt.Errorf("listen on init sock: %w", err)
 }
 defer listener.Close()

 // 5. reexec
 cmd := exec.Command("/proc/self/exe", "reexec", c.State.ID)

 cmd.Stdin = os.Stdin
 cmd.Stdout = os.Stdout
 cmd.Stderr = os.Stderr

 if err := cmd.Start(); err != nil {
	return fmt.Errorf("reexec container process: %w", err)
 }

 c.State.Pid = cmd.Process.Pid

 // 6. release container process
 if err := cmd.Process.Release(); err != nil {
	return fmt.Errorf("release container process: %w", err)
 }

 // 4. listen
 conn, err := listener.Accept()
 if err != nil {
	return fmt.Errorf("accept on init sock: %w", err)
 }
 defer conn.Close()

 b := make([]byte, 128)
 n, err := conn.Read(b)
 if err != nil {
	return fmt.Errorf("read bytes from init sock connection: %w", err)
 }

 // 10. receive 'ready'
 msg := string(b[:n])
 if msg != "ready" {
	return fmt.Errorf("expecting 'ready' but received '%s'", msg)
 }

 c.State.Status = specs.StateCreated

 // 11. exit
 return nil
}

If you’re familiar with the net³ and os/exec⁴ packages from the standard library, that should all look pretty familiar. What might look strange is the command that’s being executed - /proc/self/exe.

On a Linux system, the /proc filesystem⁵ is a pseudo-filesystem that provides a view into the current state of the kernel, including details about system hardware and configuration. In addition, details of all of the processes running on the system are available, each in a subdirectory with the name of the process’ PID.

List process directories in /proc

ls -la -d /proc/*/

dr-xr-xr-x - root 12 Jan 04:06  /proc/1
dr-xr-xr-x - root 12 Jan 04:06  /proc/2
dr-xr-xr-x - root 12 Jan 04:06  /proc/3
dr-xr-xr-x - root 12 Jan 04:06  /proc/4
dr-xr-xr-x - root 12 Jan 04:06  /proc/5
dr-xr-xr-x - root 12 Jan 04:06  /proc/6
dr-xr-xr-x - root 12 Jan 04:06  /proc/7
dr-xr-xr-x - root 12 Jan 04:06  /proc/9
# etc...

For example, to view the status of the process with PID 1 (in my case, systemd).

View status of process with PID 1

cat /proc/1/status

Name: systemd
Umask: 0000
State: S (sleeping)
Tgid: 1
Ngid: 0
Pid: 1
PPid: 0
# truncated for brevity

As a kind of ‘handy utility’ to get details about the current process, rather than specifying it’s PID, it can be referenced as self.

Getting details about current process

ls -la /proc/self

.r--r--r-- 0 nixpig 12 Jan 07:05  arch_status
dr-xr-xr-x - nixpig 12 Jan 07:05  attr
.rw-r--r-- 0 nixpig 12 Jan 07:05  autogroup
.r-------- 0 nixpig 12 Jan 07:05  auxv
.r--r--r-- 0 nixpig 12 Jan 07:05  cgroup
.-w------- 0 nixpig 12 Jan 07:05  clear_refs
.r--r--r-- 0 nixpig 12 Jan 07:05  cmdline
.rw-r--r-- 0 nixpig 12 Jan 07:05  comm
# etc...

One of the entries (omitted above due to truncating) is exe, which is a link to the executable of the process. Thus, from our container runtime /proc/self/exe is a link to the container runtime process currently executing.

So, you might ask “why do we need to reexecute the process we’re already running? Why don’t we just execute the process defined in config.json directly?”. The nitty-gritty details will be elucidated in a future installment when we get into applying configuration to the container process. For now, understand that there are elements of the configuration (such as namespaces) which need to be applied to the process which will spawn the container process and cannot be applied from within the running process itself.

Next, we can update the Create operation to call Init on the container and save it’s state after it’s been initialised.

internal/operations/create.go

func Create(opts *CreateOpts) error {

 // ...

 cntr, err := container.New(&container.NewContainerOpts{
	ID: opts.ID,
	Bundle: bundle,
	Spec: spec,
 })
 if err != nil {
	return fmt.Errorf("create container: %w", err)
 }
 if err := cntr.Save(); err != nil {
	return fmt.Errorf("save container: %w", err)
 }

 if err := cntr.Init(); err != nil {
	return fmt.Errorf("initialise container: %w", err)
 }

 if err := cntr.Save(); err != nil {
	return fmt.Errorf("save container: %w", err)
 }

 return nil
}

Now, when we create a container with ./anocir create --bundle <bundle-path> <container-id> our runtime is going to fork off a process for the container and block waiting for a ready message on the socket in the runtime process.

Reexec’ing the container process

The /proc/self/exe process that is forked is essentially the same as executing ./anocir reexec <container-id> so now we need to handle what happens when that’s called. We’ll create a new Reexec operation to do that.

Note

It’s not technically an operation using our current definition, i.e. an operation defined by the OCI Runtime Spec. But it feels appropriate to use the same abstraction, and many other container runtimes do the same thing.

internal/operations/reexec.go

package operations

import (
 "fmt"

 "github.com/nixpig/anocir/internal/container"
)

type ReexecOpts struct {
 ID string
}

func Reexec(opts *ReexecOpts) error {
 cntr, err := container.Load(opts.ID)
 if err != nil {
	return fmt.Errorf("load container: %w", err)
 }

 if err := cntr.Reexec(); err != nil {
	return fmt.Errorf("reexec container: %w", err)
 }

 return nil
}

First, we load the container, then we call Reexec on it.

We haven’t implemented Reexec on the container yet, so let’s do that now. Again, I’ll annotate the code with the step numbers from above.

internal/container/container.go

func (c *Container) Reexec() error {
 // 7. configure container
 // TODO: configure container

 // 8. send 'ready'
 initConn, err := net.Dial(
	"unix",
	filepath.Join(containerRootDir, c.State.ID, initSockFilename),
 )
 if err != nil {
	return fmt.Errorf("dial init sock: %w", err)
 }

 if _, err := initConn.Write([]byte("ready")); err != nil {
	return fmt.Errorf("write 'ready' msg to init sock: %w", err)
 }
 // close immediately, rather than defering
 initConn.Close()

 listener, err := net.Listen(
	"unix",
	filepath.Join(containerRootDir, c.State.ID, containerSockFilename),
 )
 if err != nil {
	return fmt.Errorf("listen on container sock: %w", err)
 }

 // 9. listen for 'start'
 containerConn, err := listener.Accept()
 if err != nil {
	return fmt.Errorf("accept on container sock: %w", err)
 }

 // ...coming in just a moment!
}

The first thing we do after configuring is write the ready message to the init socket, which will be picked up by the calling runtime process. We need to immediately close the connection (often this would be done in a defer) so that the socket doesn’t leak into the container process when it’s started (remember, the container process should be isolated from the host environment).

Then, we start listening on a new socket. Spoiler: in a couple of paragraphs we’ll see what we’re listening for and handle it 😉

For now, let’s wire that up to the CLI in the same way we’ve done for all of the other operations.

internal/cli/reexec.go

package cli

import (
 "github.com/nixpig/anocir/internal/operations"
 "github.com/spf13/cobra"
)

func reexecCmd() *cobra.Command {
 cmd := &cobra.Command{
	Use: "reexec [flags] CONTAINER_ID",
	Args: cobra.ExactArgs(1),
	Hidden: true, // this command is only used internally
	RunE: func(cmd *cobra.Command, args []string) error {
	 containerID := args[0]

	 return operations.Reexec(&operations.ReexecOpts{
		ID: containerID,
	 })
	},
 }

 return cmd
}

First, creating the Cobra command. Then, adding it to the root command.

internal/cli/root.go

func RootCmd() *cobra.Command {

 // ...

 cmd.AddCommand(
	stateCmd(),
	createCmd(),
	startCmd(),
	deleteCmd(),
	killCmd(),
	reexecCmd(),
 )

 return cmd
}

Starting the container

Below is a diagram of the steps involved in ‘starting’ a container.

Sequence diagram of starting a container

The container process is already running in the ‘background’ and listening on the socket created by the runtime to receive instructions.
The user (either directly or via some higher-level runtime tooling, such as Docker) issues the anocir start <container-id> command to the runtime.
The runtime sends the start message to the socket.
The runtime exits, as it’s done it’s job of instructing the container what to do.
The container receives the start message.
The container execs the user process defined in the config, then exits when the user process exits.

Exec’ing the user process

Let’s pick back up where we left the last section…listening on the socket in the reexec.

internal/container/container.go

func (c *Container) Reexec() error {

 // ...

 // 9. listen for 'start'
 containerConn, err := listener.Accept()
 if err != nil {
	return fmt.Errorf("accept on container sock: %w", err)
 }

 b := make([]byte, 128)
 n, err := containerConn.Read(b)
 if err != nil {
	return fmt.Errorf("read bytes from container sock: %w", err)
 }

 msg := string(b[:n])
 if msg != "start" {
	return fmt.Errorf("expecting 'start' but received '%s'", msg)
 }

 // close before exec'ing the user process
 containerConn.Close()
 listener.Close()

 bin, err := exec.LookPath(c.Spec.Process.Args[0])
 if err != nil {
	return fmt.Errorf("find path of user process binary: %w", err)
 }

 args := c.Spec.Process.Args
 env := os.Environ()

 if err := syscall.Exec(bin, args, env); err != nil {
	return fmt.Errorf("execve (%s, %s, %v): %w", bin, args, env, err)
 }

 panic("if you got here then something went horribly wrong")
}

First up, we wait to receive the start message from the runtime on the socket. When we receive the start message, we immediately close the listener and connection, so as not to leak these into the container process.

Now, we get to actually execute the user process defined in the config!

The command specified in the first item of the Args array may or may not (often not) be an absolute path to the binary, so we need to get an absolute path to it (if it’s available in the $PATH).

Then, we get the environment. Bear in mind, at the moment this is the same as the host environment.

We take the absolute path to the binary, the full arguments array, and the environment array, and execute an Exec syscall with them.

At this point, the process should be replaced by the execution of the user process from the config. Nothing in our code past this point should ever get executed, thus we panic if it does.

Note

Why use syscall.Exec? Why not use the exec package?

The exec package, when executing a command, creates a new process for that command. What we need is to replace the existing process with the new command.

execve(2) is how we do that. From the Linux man pages:

execve() executes the program referred to by pathname. This causes the program that is currently being run by the calling process to be replaced with a new program, with newly initialized stack, heap, and (initialized and uninitialized) data segments.

From Go, that means making the syscall.Exec syscall. From the Go docs:

Exec invokes the execve(2) system call.

For a slightly more in depth explanation and example, see Go By Example: Exec’ing Processes.

Our container can how handle the start message when it receives it from the runtime. Let’s go ahead and actually send it!

Sending the `start` message

Before we send the start message, we need to check that the container is in a state that can be started. Per the spec, the runtime may only start the container if it’s in created state⁶. So, let’s create a receiver function on the Container struct to check that.

internal/container/container.go

func (c *Container) canBeStarted() bool {
 return c.State.Status == specs.StateCreated
}

We can use this to check if the container can be started when we call Start, which we’ll implement now.

internal/container/container.go

func (c *Container) Start() error {
 if c.Spec.Process == nil {
	// nothing to do; silent return
	return nil
 }

 if !c.canBeStarted() {
	return fmt.Errorf("container cannot be started in current state (%s)", c.State.Status)
 }

 conn, err := net.Dial(
	"unix",
	filepath.Join(containerRootDir, c.State.ID, containerSockFilename),
 )
 if err != nil {
	return fmt.Errorf("dial container sock: %w", err)
 }

 if _, err := conn.Write([]byte("start")); err != nil {
	return fmt.Errorf("write 'start' msg to container sock: %w", err)
 }
 conn.Close()

 c.State.Status = specs.StateRunning

 return nil
}

In Start, we first check if the Spec for the container has a Process defined. Then we check if the container can be started.

Assuming there’s a process defined and the container is in a state that can be started, we dial the container socket and write the start message to it. After that, we update the status of the container and return.

Let’s update our Start operation to load the container, call Start on it, and save its state.

internal/operations/start.go

func Start(opts *StartOpts) error {
 cntr, err := container.Load(opts.ID)
 if err != nil {
	return fmt.Errorf("load container: %w", err)
 }

 if err := cntr.Start(); err != nil {
	return fmt.Errorf("start container: %w", err)
 }

 if err := cntr.Save(); err != nil {
	return fmt.Errorf("save container: %w", err)
 }

 return nil
}

With everything apparently wired up, it might be tempting to try this out now, but hold on!

Updating the `delete` operation

You may have noticed way back up the top that once we execute the container process, we get the PID of it and add it to the state of the container.

Now that we have a PID for our container, we can update the delete operation to ensure any running container process is killed before we remove its resources.

We’re going to use the unix package for the kill signal, so make sure to go get it: go get golang.org/x/sys/unix.

internal/container/container.go

func (c *Container) Delete(force bool) error {
 if !force && !c.canBeDeleted() {
	return fmt.Errorf("container cannot be deleted in current state (%s) try using '--force'", c.State.Status)
 }

 process, err := os.FindProcess(c.State.Pid)
 if err != nil {
	return fmt.Errorf("find container process to delete: %w", err)
 }
 if process != nil {
	process.Signal(unix.SIGKILL)
 }

 if err := os.RemoveAll(
	filepath.Join(containerRootDir, c.State.ID),
 ); err != nil {

 return nil
}

Before removing the container directory, we check if we can find the container process. If we can, we send a kill signal to it to ensure it’s no longer running before removing any associated resources.

Now we get to the fun part - testing it works!

To recap on the expectations:

Create a container using ./anocir create --bundle <bundle-path> <container-id>
Start the container using ./anocir start <container-id>
Delete the container using ./anocir delete <container-id>

Let’s go through those steps one-by-one.

Prerequisites

We need to have a container bundle from which to create a container. If you’ve been following along, you’ll know how to do this from Reading a bundle config and saving a container’s state. To recap, we need to:
1. Create a directory for the bundle: mkdir alpinefs
2. Change into the bundle directory: cd alpinefs
3. Create the rootfs directory: mkdir rootfs
4. Use Docker to export a rootfs: docker export $(docker create alpine) | tar -C rootfs -xvf -
5. Generate a configuration: runc spec
Configure the process to run.
1. Open the bundle config: nvim config.json
2. Configure the process.args array: ["touch", "/container_test.txt"]

Create a container

Create a container

./anocir create --bundle alpinefs test1

Check the state of the container

./anocir state test1

{
 "ociVersion": "1.2.0",
 "id": "test1",
 "status": "created", # <- container is created
 "pid": 87105, # <- container has a PID
 "bundle": "/home/nixpig/projects/alpinefs"
}

Start the container

Start the container

./anocir start test1

…and, we get nothing back in the terminal. While you might have been hoping (or even expecting) to see the result of running the command in process.args in the terminal, remember - it’s running in a detached process with no stdio hooked up (we’ll hooking that up in a future installment of the series).

We can verify that it did, in fact, execute by checking for the presence of the /container_test.txt file that should have been touch’d.

Verify that container process executed

ls -la /container_test.txt

.rw-r--r-- 0 root 12 Jan 08:54  container_test.txt

…and check the state of the container.

Check the state of the container

./anocir state test1

{
 "ociVersion": "1.2.0",
 "id": "test1",
 "status": "running", # <- container is running
 "pid": 87105,
 "bundle": "/home/nixpig/projects/alpinefs"
}

Delete the container

Delete the container

./anocir delete test1 --force

It’s not very container-like

At this point, you’re probably seeing some red flags. This ‘container’ was able to write to the host filesystem. If you do some more playing around, you’ll realise it’s able to do a whole lot more that a container shouldn’t.

In fact, at this point the container is just a process running on the host system with absolutely no restrictions applied to what it can and can’t do.

You’ll remember, we haven’t actually applied any of the configuration in config.json which tells the runtime about what restrictions should be in place. These are what actually containerise a process. Things like namespaces, cgroups, capabilities, the list goes on…

We’re getting there, so stick around for the next post!

Part 5: Executing container runtime lifecycle hooks

References

https://github.com/opencontainers/runtime-spec/blob/main/config.md#process “Process config” ↩︎
https://pubs.opengroup.org/onlinepubs/9699919799/functions/exec.html “exec” ↩︎
https://pkg.go.dev/net “net package” ↩︎
https://pkg.go.dev/os/exec “os/exec package” ↩︎
https://docs.kernel.org/filesystems/proc.html “The /proc filesystem” ↩︎
https://github.com/opencontainers/runtime-spec/blob/main/runtime.md#start “Start operation” ↩︎

Loading a container, getting it's state, and deleting it

Thu, 09 Jan 2025 04:43:02 +0000

We’re now able to create container objects and persist them to disk. But, we need a way to load them, query their state, and (given we’re going to be iterating quite frequently) easily delete them.

Loading a container

Before we can get the state of a container, we need to load it. So, we create a Load function in the container package that takes in the ID of a container to load and returns the corresponding container.

internal/container/container.go

func Load(id string) (*Container, error) {
 s, err := os.ReadFile(
 filepath.Join(containerRootDir, id, "state.json"),
 )
 if err != nil {
 return nil, fmt.Errorf("read state file: %w", err)
 }

 var state *specs.State
 if err := json.Unmarshal(s, &state); err != nil {
 return nil, fmt.Errorf("unmarshal state: %w", err)
 }

 config, err := os.ReadFile(
 filepath.Join(state.Bundle, "config.json"),
 )
 if err != nil {
 return nil, fmt.Errorf("read config file: %w", err)
 }

 var spec *specs.Spec
 if err := json.Unmarshal(config, &spec); err != nil {
 return nil, fmt.Errorf("unmarshal config: %w", err)
 }

 c := &Container{
 State: state,
 Spec: spec,
 }

 return c, nil
}

First, we read the container’s state.json file from disk and unmarshal it into a specs.State struct.

Then, we read the bundle’s config.json file from disk and unmarshal it into a specs.Spec struct.

Finally, we create a Container struct with the state and spec, then return it.

Getting the container state

Now that we’re able to load a container, we can implement the state operation to retrieve the container’s state.

internal/operations/state.go

func State(opts *StateOpts) (string, error) {
 cntr, err := container.Load(opts.ID)
 if err != nil {
 return "", fmt.Errorf("load container: %w", err)
 }

 state, err := json.Marshal(cntr.State)
 if err != nil {
 return "", fmt.Errorf("marshal state: %w", err)
 }

 return string(state), nil
}

We simply load the container using the Load function we just created, then marshal the container’s state to JSON and return it as a string.

Printing the container state

We could leave the Cobra command for state as it is and let the fmt.Println(state) line handle printing the state out. However, we’ll see later why this would be problematic. For now, let’s replace the fmt.Println(state) with the code below to write the state to cmd.OutOrStdout().

internal/cli/state.go


 // ...

 if _, err := cmd.OutOrStdout().Write(
 []byte(state),
 ); err != nil {
 return fmt.Errorf("write state to stdout: %w", err)
 }

 // ...

We’ll see later exactly why it’s necessary to do this.

Deleting the container

We’ve (probably) run create a couple of times now, and those resources in /var/lib/anocir/containers not going to clean themselves up. So, let’s look at implementing the delete operation.

As defined in the spec¹, the delete operation must only delete a container if it is in the stopped state.

Attempting to delete a container that is not stopped MUST have no effect on the container and MUST generate an error.

Since we haven’t got to the point of actually running a container, our containers will never be in that state - they’re always in creating state. In order to get around this restriction, we’re going to do the same as runc and other runtimes do and add a --force flag. This will enable us to delete a container regardless of it’s state.

Before doing that, let’s first add a helper function to check whether the container can be deleted.

internal/container/container.go

func (c *Container) canBeDeleted() bool {
 return c.State.Status == specs.StateStopped
}

The logic is simply checking whether the current state of the container is stopped and returning a boolean to indicate whether it is (true) or isn’t (false) stopped.

Next, let’s implement the actual deletion of the container.

internal/container/container.go

func (c *Container) Delete(force bool) error {
 if !force && !c.canBeDeleted() {
 return fmt.Errorf("container cannot be deleted in current state (%s) try using '--force'", c.State.Status)
 }

 if err := os.RemoveAll(
 filepath.Join(containerRootDir, c.State.ID),
 ); err != nil {
 return fmt.Errorf("delete container directory: %w", err)
 }

 return nil
}

First, we check if the container can be deleted and whether the force argument is true. If the container can’t be deleted and the force argument is false, then we return an error.

If the container can be deleted or the force argument is true, then we go ahead and delete the container directory.

It’s worth noting that this is only a part of the logic needed to actually delete a container. Once we actually have running containers, we’ll need to come back and add some additional functionality in here.

We can now update our delete operation handler to delete the container.

internal/operations/delete.go

type DeleteOpts struct {
 ID string
 Force bool
}

func Delete(opts *DeleteOpts) error {
 cntr, err := container.Load(opts.ID)
 if err != nil {
 return fmt.Errorf("load container: %w", err)
 }

 if err := cntr.Delete(opts.Force); err != nil {
 return fmt.Errorf("delete container: %w", err)
 }

 return nil
}

The DeleteOpts struct is updated to include a Force field and the Delete function is updated to load then delete the container.

Finally, we need to update our delete command to add a force flag and pass that to the delete operation handler.

internal/cli/delete.go

func deleteCmd() *cobra.Command {
 cmd := &cobra.Command{
 Use: "delete [flags] CONTAINER_ID",
 Args: cobra.ExactArgs(1),
 RunE: func(cmd *cobra.Command, args []string) error {
 containerID := args[0]

 force, err := cmd.Flags().GetBool("force")
 if err != nil {
 return err
 }

 return operations.Delete(&operations.DeleteOpts{
 ID: containerID,
 Force: force,
 })
 },
 }

 cmd.Flags().BoolP("force", "f", false, "Delete container regardless of state")

 return cmd
}

That should be everything hooked up to load a container, get it’s state, and delete it. Let’s try it out.

Create a couple of containers

./anocir create --bundle alpinefs test1
./anocir create --bundle alpinefs test2

Check we get the state back

./anocir state test1

{"ociVersion":"1.2.0","id":"test1","status":"creating","bundle":"/home/nixpig/projects/alpinefs"}

Pipe it to jq to verify it's valid JSON and format it nicely

./anocir state test2 | jq

{
 "ociVersion": "1.2.0",
 "id": "test2",
 "status": "creating",
 "bundle": "/home/nixpig/projects/alpinefs"
}

Check we get a helpful error message when the container doesn't exist

./anocir state test3

Error: load container: read state file: open /var/lib/anocir/containers/test3/state.json: no such file or directory

Try to delete a container in 'creating' state

./anocir delete test1

Error: delete container: container cannot be deleted in current state (creating) try using '--force'

Force-delete the created containers

./anocir delete --force test1
./anocir delete --force test2

With a couple of the core commands now hooked up, we’re able to get to the ‘fun stuff’. In the next part of the series we’re going to initialise and start a bare-bones container in a forked process.

Part 4: Initialising and starting a container »

References

https://github.com/opencontainers/runtime-spec/blob/main/runtime.md#delete “Delete operation” ↩︎

Reading a bundle config and saving a container's state

Sat, 04 Jan 2025 05:30:02 +0000

The container bundle

The container bundle is a set of files that contain the necessary data for a runtime to build a container and perform operations against it.

It consists of two parts:

The configuration data to build the container (config.json).
The root filesystem to use for the container (rootfs).

Filesystem structure for a bundle

alpinefs/
├── config.json
└── rootfs
 ├── bin
 ├── dev
 ├── etc
 ├── home
 └── # and so on...

For more details see the Filesystem Bundle¹ section of the spec.

`config.json`

The config.json file contains the necessary configuration for a container runtime to build a container from the rootfs of the bundle. This is the file that the runtime will need to read the configuration from to apply to the container.

There are JSON schemas available for the config² and Linux platform-specific config³. We won’t be consuming these, though they’re useful as a reference.

Instead, we’ll be using the Go version of the config⁴, which will save us from the tedius task of manually (or otherwise) converting the JSON schema to Go structs.

This can be installed by running go get github.com/opencontainers/runtime-spec/specs-go@v1.2.0 from our project root.

`rootfs`

The rootfs directory contains the root filesystem to use when building the container. It typically looks like your usual Linux filesystem, with bin/, dev/, home/, and other directories you’d expect to see in the root of a Linux filesystem.

Creating a bundle

It’s the responsibility of higher-level tooling, like Docker, to provide the required bundle to the container runtime. While building out the runtime, however, we’ll create bundles ourselves.

To make things easier, we’ll use Docker to create the rootfs and runc to create a basic config. Assuming we have Docker installed, we’ll also have runc.

Create a directory for the bundle: mkdir alpinefs
Change into the bundle directory: cd alpinefs
Create the rootfs directory: mkdir rootfs
Use Docker to export a rootfs: docker export $(docker create alpine) | tar -C rootfs -xvf -
Generate a configuration: runc spec

The container state

While a container bundle provides the necessary resources for the runtime to build a container, the runtime will need to store/read/update/delete the state of the container once it’s built and operations are performed on it.

`state.json`

The state.json file is what the container runtime uses to store and retrieve the state of containers that it manages. The schema of this is also what the spec expects to be returned when querying the state of a container, e.g. by running ./anocir state mycontainer.

Broadly speaking, the state of a container includes:

ociVersion - The version of the spec.
id - The ID of the container.
status - The runtime state of the container.
pid - The ID of the container process.
bundle - The absolute path to the bundle directory.
annotations - Optional annotations.

Further details on each of these can be found in the State⁵ section of the spec.

There’s a JSON schema available⁶. As with the config, we won’t be using it, but it’s useful for reference.

We’ll be making use of the provided Go version⁷. This will be available after running go get github.com/opencontainers/runtime-spec/specs-go@v1.2.0 as previously.

The spec doesn’t specify how a runtime actually stores the state of containers it manages. For our runtime, we’re going to save it to disk as a JSON file in a directory that is the ID of the container, not dissimilar to how runc does it.

Filesystem structure for storing containers

/
└── var
 └── lib
	└── anocir
	 └── containers
		├── 234f74227159557e043bf7878686928edf24c0e8b702bfd22b28a9eb2e80f04d
		│   └── state.json
		├── 3278e73c1c4613cfe85786da35185e15cead1623415361f319aa7c1890bdedb8
		│   └── state.json
		└── 484def0fbff90d08293acee464d955fcdcb07025e635de93ddf65c37bc7d7768
		 └── state.json

Example state.json contents

cat /var/lib/anocir/containers/234f74227159557e043bf7878686928edf24c0e8b702bfd22b28a9eb2e80f04d/state.json | jq
{
 "ociVersion": "1.2.0",
 "id": "234f74227159557e043bf7878686928edf24c0e8b702bfd22b28a9eb2e80f04d",
 "bundle": "/home/nixpig/projects/alpinefs",
 "annotations": null,
 "status": "stopped",
 "pid": 50970
}

Now that we understand the key components we’ll be working with - the config and the state - and we know what we need to do - read in the config and save the state - we can get to work.

Creating a new container

From the perspective of a runtime, there are really only two things it cares about in order to build and manage a container - the config and the state. It should be no surprise then, that’s exactly what our Container struct will contain, and the function for creating a new container will receive the options needed to create those.

We also want to perform a check before creating the container whether one with the same ID already exists. If it does, then we return an error.

Let’s create a new internal/container package for this to live in.

internal/container/container.go

package container

import (
 "fmt"
 "os"
 "path/filepath"

 "github.com/opencontainers/runtime-spec/specs-go"
)

const (
 containerRootDir = "/var/lib/anocir/containers"
)

type Container struct {
 State *specs.State
 Spec *specs.Spec
}

type NewContainerOpts struct {
 ID string
 Bundle string
 Spec *specs.Spec
}

func New(opts *NewContainerOpts) (*Container, error) {
 if exists(opts.ID) {
 return nil, fmt.Errorf("container '%s' exists", opts.ID)
 }

 state := specs.State{
	Version: specs.Version,
	ID: opts.ID,
	Bundle: opts.Bundle,
	Annotations: opts.Spec.Annotations,
	Status: specs.StateCreating,
 }

 c := Container{
	State: &state,
	Spec: opts.Spec,
 }

 return &c, nil
}

func exists(containerID string) bool {
 _, err := os.Stat(filepath.Join(containerRootDir, containerID))

 return err == nil
}

You’ll also notice that we import the github.com/opencontainers/runtime-spec/specs-go package that we installed earlier to make use of the specs.Spec (config) and specs.State structs. It’s worth opening these up and having a read through. Implementing the processing for all of these configuration options is essentially what we’re going to need to do to build the runtime.

Also note that the Status of the container at this point is being set as specs.StateCreating to indicate that the container is currently in the process of being created. As we implement further stages of the lifecycle, this will be updated to indicate where in the lifecycle the container is.

We now need to hook this up to the Create operation handler we defined previously so that when the create command is run then a new container is created.

internal/operations/create.go

func Create(opts *CreateOpts) error {
 bundle, err := filepath.Abs(opts.Bundle)
 if err != nil {
	return fmt.Errorf("absolute path from bundle: %w", err)
 }

 config, err := os.ReadFile(filepath.Join(bundle, "config.json"))
 if err != nil {
	return fmt.Errorf("read config file: %w", err)
 }

 var spec *specs.Spec
 if err := json.Unmarshal(config, &spec); err != nil {
 return fmt.Errorf("unmarshall config: %w", err)
 }

 if _, err := container.New(&container.NewContainerOpts{
	ID: opts.ID,
	Bundle: bundle,
	Spec: spec,
 }); err != nil {
	return fmt.Errorf("create container: %w", err)
 }

 return nil
}

First, we get an absolute path to the bundle.

Then, we read the config.json file from the bundle location and unmarshal it into a *specs.Spec struct.

Finally, we create a new Container with the container ID, bundle and config.

Saving the container state

To save the container’s state, we need to serialise a JSON representation of it (according to the spec for state.json) and save it to disk at /var/lib/anocir/containers/{{ containerID }}/state.json.

So, we create a Save receiver function on the Container struct to do just that.

internal/container/container.go

func (c *Container) Save() error {
 if err := os.MkdirAll(
	filepath.Join(containerRootDir, c.State.ID),
	0666,
 ); err != nil {
	return fmt.Errorf("create container directory: %w", err)
 }

 state, err := json.Marshal(c.State)
 if err != nil {
	return fmt.Errorf("serialise container state: %w", err)
 }

 if err := os.WriteFile(
	filepath.Join(containerRootDir, c.State.ID, "state.json"),
	state,
	0666,
 ); err != nil {
	return fmt.Errorf("write container state: %w", err)
 }

 return nil
}

First, we create the directory for the container if it doesn’t already exist. Then, we serialise the state and save it to disk.

The state of a container will change throughout its lifecycle and Save can be called at any point to save the current state of the container. So, let’s update the Create operation to do that after newing a Container.

internal/operations/create.go

func Create(opts *CreateOpts) error {

 // ...

 cntr, err := container.New(&container.NewContainerOpts{
	ID: opts.ID,
	Bundle: bundle,
	Spec: spec,
 })
 if err != nil {
	return fmt.Errorf("create container: %w", err)
 }

 if err := cntr.Save(); err != nil {
	return fmt.Errorf("save container: %w", err)
 }

 return nil
}

Now, let’s check it works as expected.

Create a container bundle

Follow the steps from Creating a bundle. To recap:

Create a directory for the bundle: mkdir alpinefs
Change into the bundle directory: cd alpinefs
Create the rootfs directory: mkdir rootfs
Use Docker to export a rootfs: docker export $(docker create alpine) | tar -C rootfs -xvf -
Generate a configuration: runc spec
Come back out: cd ..

At least for what we’re doing now, we can continue to use this same bundle for future operations; we don’t need to create a new bundle every time.

Build the app

go build -o anocir

Create some containers

./anocir create --bundle alpinefs test1
./anocir create --bundle alpinefs test2
./anocir create --bundle alpinefs test3

Check we can't create multiple containers with the same ID

./anocir create --bundle alpinefs test1

Error: create container: container 'test1' exists

Check the content of a container state

cat /var/lib/anocir/containers/test1/state.json | jq
{
 "ociVersion": "1.2.0",
 "id": "test1",
 "status": "creating",
 "bundle": "/home/nixpig/projects/alpinefs"
}

It’s fine to cat out the state.json file for development, as above, but the runtime specifies a state operation to achieve this. We’ll implement that in the next post.

Part 3: Loading a container, getting its state, and deleting it »

References

https://github.com/opencontainers/runtime-spec/blob/main/bundle.md “Filesystem Bundle” ↩︎
https://github.com/opencontainers/runtime-spec/blob/main/schema/config-schema.json “Config JSON schema” ↩︎
https://github.com/opencontainers/runtime-spec/blob/main/schema/config-linux.json “Linux platform-specific config JSON schema” ↩︎
https://github.com/opencontainers/runtime-spec/blob/main/specs-go/config.go “Config schema in Go” ↩︎
https://github.com/opencontainers/runtime-spec/blob/main/runtime.md#state “Runtime state” ↩︎
https://github.com/opencontainers/runtime-spec/blob/main/schema/state-schema.json “State JSON schema” ↩︎
https://github.com/opencontainers/runtime-spec/blob/main/specs-go/state.go “State schema in Go” ↩︎

Building the CLI interface for a container runtime in Go

Sun, 29 Dec 2024 09:06:37 +0000

Framework for the OCI Runtime CLI

The OCI Runtime Spec defines a set of operations¹ for higher-level container tooling to interact with a container runtime, leaving individual runtimes to decide the actual implementation details. Different tooling and runtimes communicate over different APIs or protocols. For example, Docker communicates with a container runtime using a command-line interface (CLI), whereas Kubernetes communicates over gRPC.

For our runtime, we’ll be keeping things simple and creating a CLI.

Architecture of the runtime CLI

The architecture for our CLI will be structured as below.

Architecture of CLI app

For example, when running the command anocir create --bundle busybox mycontainer:

CLI app flow when running a command

The anocir create --bundle busybox mycontainer command in read in from stdin.
The anocir root command will delegate to the create subcommand.
The create subcommand will parse the busybox value from the --bundle flag and the mycontainer container ID argument then call the create handler.
The create handler will perform the necessary tasks to handle the create operation - creating the container with the ID mycontainer from the busybox bundle.

Scaffolding out the project

Let’s initialise a new project.

Create a directory for the project: mkdir anocir
Change into the directory: cd anocir
Initialise: go mod init github.com/nixpig/anocir
Create the main.go file and add the main function

main.go

package main

func main() {}

Creating handlers for the runtime operations

The handlers for the runtime operations specify a ‘generic’ interface, rather than a specific API like a CLI; from the spec:

Note: these operations are not specifying any command-line APIs, and the parameters are inputs for general operations.¹

At this point, we’re only concerned with implementing the API for these operations, so we’ll focus only on that. We’ll address the actual handling of these operations in a later part.

Each operation will be a function in an internal/operations package. The operation function will take in a pointer to a struct of options for that operation. All operations will return an error, with the exception of the state operation, which will return a (string, error); we’ll address that one first.

Query State operation

Operation	Link to spec
`state <container-id>`	Spec

The state operation is responsible for getting the state of a container. It takes in the ID of the container to get the state of.

internal/operations/state.go

package operations

import "fmt"

type StateOpts struct {
 ID string
}

func State(opts *StateOpts) (string, error) {
 fmt.Println(opts)

 return "", nil
}

Create operation

Operation	Link to spec
`create <container-id> <path-to-bundle>`	Spec

The create operation is responsible for creating a container. It takes in the ID of the new container to create and the path to a runtime bundle (more on this later) from which to create the container.

internal/operations/create.go

package operations

import "fmt"

type CreateOpts struct {
 ID string
 Bundle string
}

func Create(opts *CreateOpts) error {
 fmt.Println(opts)

 return nil
}

Start operation

Operation	Link to spec
`start <container-id>`	Spec

The start operation is responsible for starting a container. It takes in the ID of the container to start.

internal/operations/start.go

package operations

import "fmt"

type StartOpts struct {
 ID string
}

func Start(opts *StartOpts) error {
 fmt.Println(opts)

 return nil
}

Kill operation

Operation	Link to spec
`kill <container-id> <signal>`	Spec

The kill operation is responsible for sending a signal to a running container. It takes in the ID of the container to send a signal to and the signal to send.

internal/operations/kill.go

package operations

import "fmt"

type KillOpts struct {
 ID string
 Signal string
}

func Kill(opts *KillOpts) error {
 fmt.Println(opts)

 return nil
}

Delete operation

Operation	Link to spec
`delete <container-id>`	Spec

The delete operation is responsible for deleting a container. It takes in the ID of the container to delete.

internal/operatins/delete.go

package operations

import "fmt"

type DeleteOpts struct {
 ID string
}

func Delete(opts *DeleteOpts) error {
 fmt.Println(opts)

 return nil
}

With the operations API defined, we can start creating the CLI to interact with it.

Creating the runtime CLI

Since we’re interested in learning about Linux container runtimes, and not the intricacies of building command-line interfaces, we’re going to use the popular Cobra library to handle CLI interactions, such as commands, arguments and flags parsing. If you’ve ever used Kubernetes, Helm or the GitHub CLI, then you’ve interacted with it before.

Run go get github.com/spf13/cobra@v1.8.1 from the project root to install Cobra, then we’ll create an internal/cli package and scaffold out the ‘root command’.

internal/cli/root.go

package cli

import "github.com/spf13/cobra"

func RootCmd() *cobra.Command {
 cmd := &cobra.Command{
	Use: "anocir",
	SilenceUsage: true,
 }

 return cmd
}

The root command will serve as the entrypoint when a user runs the anocir command from the terminal and will be responsible for delegating to subcommands, such as when a user runs anocir create --bundle busybox mycontainer.

From the main package in main.go, import the cli package and execute the root command, handling any error by printing to the console and exiting with exit code 1.

main.go

package main

import (
 "fmt"
 "os"

 "github.com/nixpig/anocir/internal/cli"
)

func main() {
 if err := cli.RootCmd().Execute(); err != nil {
	fmt.Println(err)
	os.Exit(1)
 }
}

We now need to create a subcommand for each of the operations we previously defined.

Query State subcommand

anocir state mycontainer

internal/cli/state.go

package cli

import (
 "github.com/spf13/cobra"
 "github.com/nixpig/anocir/internal/operations"
)

func stateCmd() *cobra.Command {
 cmd := &cobra.Command{
	Use: "state [flags] CONTAINER_ID",
	Args: cobra.ExactArgs(1),
	RunE: func(cmd *cobra.Command, args []string) error {
	 containerID := args[0]

	 state, err := operations.State(&operations.StateOpts{
		ID: containerID,
	 })
	 if err != nil {
		return err
	 }

	 // TODO: do something with 'state'
	 fmt.Println(state)

	 return nil
	},
 }

 return cmd
}

Create subcommand

anocir create --bundle busybox mycontainer

anocir create mycontainer

internal/cli/create.go

package cli

import (
 "os"

 "github.com/spf13/cobra"
 "github.com/nixpig/anocir/internal/operations"
)

func createCmd() *cobra.Command {
 cmd := &cobra.Command{
	Use: "create [flags] CONTAINER_ID",
	Args: cobra.ExactArgs(1),
	RunE: func(cmd *cobra.Command, args []string) error {
	 containerID := args[0]

	 bundle, err := cmd.Flags().GetString("bundle")
	 if err != nil {
		return err
	 }

	 return operations.Create(&operations.CreateOpts{
		ID: containerID,
		Bundle: bundle,
	 })
	},
 }

 cwd, _ := os.Getwd()
 cmd.Flags().StringP("bundle", "b", cwd, "Path to bundle directory")

 return cmd
}

Start subcommand

anocir start mycontainer

internal/cli/start.go

package cli

import (
 "github.com/spf13/cobra"
 "github.com/nixpig/anocir/internal/operations"
)

func startCmd() *cobra.Command {
 cmd := &cobra.Command{
	Use: "start [flags] CONTAINER_ID",
	Args: cobra.ExactArgs(1),
	RunE: func(cmd *cobra.Command, args []string) error {
	 containerID := args[0]

	 return operations.Start(&operations.StartOpts{
		ID: containerID,
	 })
	},
 }

 return cmd
}

Kill subcommand

anocir kill mycontainer 9

internal/cli/kill.go

package cli

import (
 "github.com/spf13/cobra"
 "github.com/nixpig/anocir/internal/operations"
)

func killCmd() *cobra.Command {
 cmd := &cobra.Command{
	Use: "kill [flags] CONTAINER_ID SIGNAL",
	Args: cobra.ExactArgs(2),
	RunE: func(cmd *cobra.Command, args []string) error {
	 containerID := args[0]
	 signal := args[1]

	 return operations.Kill(&operations.KillOpts{
		ID: containerID,
		Signal: signal,
	 })
	},
 }

 return cmd
}

Delete subcommand

anocir delete mycontainer

internal/cli/delete.go

package cli

import (
 "github.com/spf13/cobra"
 "github.com/nixpig/anocir/internal/operations"
)

func deleteCmd() *cobra.Command {
 cmd := &cobra.Command{
	Use: "delete [flags] CONTAINER_ID",
	Args: cobra.ExactArgs(1),
	RunE: func(cmd *cobra.Command, args []string) error {
	 containerID := args[0]

	 return operations.Delete(&operations.DeleteOpts{
		ID: containerID,
	 })
	},
 }

 return cmd
}

Updating the root command

With all of the subcommands defined, we can now add them to the root command.

internal/cli/root.go

package cli

import "github.com/spf13/cobra"

func RootCmd() *cobra.Command {
 cmd := &cobra.Command{
	Use: "anocir",
	SilenceUsage: true,
 }

 cmd.AddCommand(
	stateCmd(),
	createCmd(),
	startCmd(),
	deleteCmd(),
	killCmd(),
 )

 return cmd
}

With the CLI and operations all wired up, let’s try compiling the app and running a few commands to ensure that the operation handlers get called with the correct arguments (which will be printed to screen).

Build the app

go build -o anocir

Create

Execute the create command, specifying the bundle directory as busybox and the container ID as mycontainer.

./anocir create --bundle busybox mycontainer

We should see the values passed to the create operation handler in the CreateOpts struct printed out.

&{mycontainer busybox}

State

Execute the state command, specifying the container ID as mycontainer.

./anocir state mycontainer

We should see the values pass to the state operation handler in the StateOpts struct printed out.

&{mycontainer}

Start

Execute the start command, specifying the container ID as mycontainer.

./anocir start mycontainer

We should see the values pass to the start operation handler in the StartOpts struct printed out.

&{mycontainer}

Kill

Execute the kill command, specifying the container ID as mycontainer and the signal as 9.

./anocir kill mycontainer 9

We should see the values pass to the kill operation handler in the KillOpts struct printed out.

&{mycontainer 9}

Delete

Execute the delete command, specifying the container ID as mycontainer.

./anocir delete mycontainer

We should see the values pass to the delete operation handler in the DeleteOpts struct printed out.

&{mycontainer}

All seems to be wired up correctly! 🎉

Next, let’s start implementing some features…

Part 2: Reading the bundle config and saving the container state »

References

https://github.com/opencontainers/runtime-spec/blob/main/runtime.md#operations “OCI Runtime operations” ↩︎ ↩︎

Building a container runtime from scratch in Go

Tue, 24 Dec 2024 05:56:48 +0000

Things started off innocently enough; “Why do we need to run Docker on Linux?,” I was asked, “Isn’t the point of Docker is that it can run anywhere?”

“Our software doesn’t work in Docker running on Windows,” I replied, “…only Docker on Linux.”

Despite being an absolutely true-and-factual answer to the first part of the question, it didn’t feel satisfactory to me. It also completely failed to address the second part, or the broader question in the context of the discussion, the details of which I won’t bore you with.

Trying to find the answer to that question is what set me down the path of building a container runtime, to understand what they are, how they work, and why they (seemingly) don’t work well in non-Linux environments. The answer being blindingly obvious, once we have an understanding of the first two.

Where to start?

A quick Google search for “build a container runtime” returns a bunch of results.

Despite the enticing titles of articles, the short runtime of videos, and the credentials of the authors, it is (I would submit) not possible to build a container runtime in 100 lines of Go, or in 30 minutes, even if you’re authority on the subject. At least, not one I would consider complete. Though, those served as a good introduction to how container runtimes work at a high-level (which, in fairness, is exactly what they aim to do).

Ultimately, I came across the Open Container Initiative (OCI) Runtime Spec¹, which “aims to specify the configuration, execution environment, and lifecycle of a container”. It’s also what is implemented by existing popular runtimes, such as:

runc- “a CLI tool for spawning and running containers on Linux according to the OCI specification”
youki - “an implementation of the OCI runtime-spec in Rust”
crun - “A fast and lightweight fully featured OCI runtime and C library”

It’s worth noting that runc is the default container runtime used by Docker and Kubernetes, so this spec definitely seemed like the place to start.

Roadmap for this series

This series of posts documents what it takes to implement (from scratch) the OCI Runtime Spec in Go, and end up with a solution that works as a drop-in replacement for runc in Docker. We shall call our runtime anocir (“an-oh-cheer”) (Another OCI Runtime).

It’s probably going to take a couple of months to get everything typed up, so keep checking back if a section you’re interested in isn’t published yet. If you have a specific question about any of the sections (published or not) then feel free to reach out!

Listing of posts (click to collapse/expand)

Building a container runtime from scratch in Go 👈️ You are here
Building the CLI interface for a container runtime
Reading a bundle config and saving a containers state
Loading a container, getting its state, and deleting it
Initialising a container and starting the user process
Executing container runtime lifecycle hooks
Sending signals to a running container using ‘kill’
Setting up the OCI Runtime Spec test suite
Isolating the container process using namespaces 🔜 Coming soon!
Managing container resources using cgroups & rlimits
Setting up the root filesystem (rootfs) of a container
Modifying runtime kernel parameters of the container
Mounting masked and readonly paths of the container
Propagating the rootfs mount of the container
Setting the hostname and domain name of the container
Setting capabilities and restricting privileges of the container
Applying scheduling policies and I/O priority to the container
Applying UID/GID and additional GIDs to the container process
Wiring up a console to the container
Writing the container process PID out to file
Adjusting the OOM score of the container process
Implementing the ‘features’ API for the container runtime
Using an alternative container runtime with Docker and other tools
Thoughts on next steps to improve our custom container runtime (CRI?)

Choice of tooling

Q. Why Go?

A. Why not? It’s like the lingua franca of container tooling development. It’s also what I’ve been working with most recently…bonus.

Note

I have some retrospective thoughts on the suitability of Go for this project, which I’ll share in a follow up post.

A note on code style

The code is structured and written in a way that makes it easier to present in a series of individually useful blog posts, not to be perfectly ‘idiomatic’ Go.

References

https://github.com/opencontainers/runtime-spec/blob/main/spec.md ↩︎

Building a container runtime from scratch in Go — nixpig.dev

Setting up the OCI Runtime Spec test suite

Configuring the OCI Runtime Spec test suite

Building a virtual machine

Running the tests

References

Sending signals to a running container using 'kill'

Signals and the Linux kill command

Sending a signal to a container process

References

Executing container runtime lifecycle hooks

Executing the hooks

References

Initialising a container and starting the user process

Initialising the container

Reexec’ing the container process

Starting the container

Exec’ing the user process

Sending the start message

Updating the delete operation

Prerequisites

Create a container

Start the container

Delete the container

It’s not very container-like

References

Loading a container, getting it's state, and deleting it

Loading a container

Getting the container state

Printing the container state

Deleting the container

References

Reading a bundle config and saving a container's state

The container bundle

config.json

rootfs

Creating a bundle

The container state

state.json

Creating a new container

Saving the container state

References

Building the CLI interface for a container runtime in Go

Framework for the OCI Runtime CLI

Architecture of the runtime CLI

Scaffolding out the project

Creating handlers for the runtime operations

Query State operation

Create operation

Start operation

Kill operation

Delete operation

Creating the runtime CLI

Query State subcommand

Create subcommand

Start subcommand

Kill subcommand

Delete subcommand

Updating the root command

Build the app

Create

State

Start

Kill

Delete

References

Building a container runtime from scratch in Go

Where to start?

Roadmap for this series

Choice of tooling

A note on code style

References

Signals and the Linux `kill` command

Sending the `start` message

Updating the `delete` operation

`config.json`

`rootfs`

`state.json`