Reading a bundle config and saving a container's state

The container bundle

The container bundle is a set of files that contain the necessary data for a runtime to build a container and perform operations against it.

It consists of two parts:

1. The configuration data to build the container (config.json).
2. The root filesystem to use for the container (rootfs).

alpinefs/
├── config.json
└── rootfs
    ├── bin
    ├── dev
    ├── etc
    ├── home
    └── # and so on...

For more details see the Filesystem Bundle¹ section of the spec.

config.json

The config.json file contains the necessary configuration for a container runtime to build a container from the rootfs of the bundle. This is the file that the runtime will need to read the configuration from to apply to the container.

There are JSON schemas available for the config² and Linux platform-specific config³. We won’t be consuming these, though they’re useful as a reference.

Instead, we’ll be using the Go version of the config⁴, which will save us from the tedius task of manually (or otherwise) converting the JSON schema to Go structs.

This can be installed by running go get github.com/opencontainers/runtime-spec/specs-go@v1.2.0 from our project root.

rootfs

The rootfs directory contains the root filesystem to use when building the container. It typically looks like your usual Linux filesystem, with bin/, dev/, home/, and other directories you’d expect to see in the root of a Linux filesystem.

Creating a bundle

It’s the responsibility of higher-level tooling, like Docker, to provide the required bundle to the container runtime. While building out the runtime, however, we’ll create bundles ourselves.

To make things easier, we’ll use Docker to create the rootfs and runc to create a basic config. Assuming we have Docker installed, we’ll also have runc.

1. Create a directory for the bundle: mkdir alpinefs
2. Change into the bundle directory: cd alpinefs
3. Create the rootfs directory: mkdir rootfs
4. Use Docker to export a rootfs: docker export $(docker create alpine) | tar -C rootfs -xvf -
5. Generate a configuration: runc spec

The container state

While a container bundle provides the necessary resources for the runtime to build a container, the runtime will need to store/read/update/delete the state of the container once it’s built and operations are performed on it.

state.json

The state.json file is what the container runtime uses to store and retrieve the state of containers that it manages. The schema of this is also what the spec expects to be returned when querying the state of a container, e.g. by running ./anocir state mycontainer.

Broadly speaking, the state of a container includes:

• ociVersion - The version of the spec.
• id - The ID of the container.
• status - The runtime state of the container.
• pid - The ID of the container process.
• bundle - The absolute path to the bundle directory.
• annotations - Optional annotations.

Further details on each of these can be found in the State⁵ section of the spec.

There’s a JSON schema available⁶. As with the config, we won’t be using it, but it’s useful for reference.

We’ll be making use of the provided Go version⁷. This will be available after running go get github.com/opencontainers/runtime-spec/specs-go@v1.2.0 as previously.

The spec doesn’t specify how a runtime actually stores the state of containers it manages. For our runtime, we’re going to save it to disk as a JSON file in a directory that is the ID of the container, not dissimilar to how runc does it.

/
└── var
    └── lib
	└── anocir
	    └── containers
		├── 234f74227159557e043bf7878686928edf24c0e8b702bfd22b28a9eb2e80f04d
		│   └── state.json
		├── 3278e73c1c4613cfe85786da35185e15cead1623415361f319aa7c1890bdedb8
		│   └── state.json
		└── 484def0fbff90d08293acee464d955fcdcb07025e635de93ddf65c37bc7d7768
		    └── state.json

cat /var/lib/anocir/containers/234f74227159557e043bf7878686928edf24c0e8b702bfd22b28a9eb2e80f04d/state.json | jq
{
  "ociVersion": "1.2.0",
  "id": "234f74227159557e043bf7878686928edf24c0e8b702bfd22b28a9eb2e80f04d",
  "bundle": "/home/nixpig/projects/alpinefs",
  "annotations": null,
  "status": "stopped",
  "pid": 50970
}

Now that we understand the key components we’ll be working with - the config and the state - and we know what we need to do - read in the config and save the state - we can get to work.

Creating a new container

From the perspective of a runtime, there are really only two things it cares about in order to build and manage a container - the config and the state. It should be no surprise then, that’s exactly what our Container struct will contain, and the function for creating a new container will receive the options needed to create those.

We also want to perform a check before creating the container whether one with the same ID already exists. If it does, then we return an error.

Let’s create a new internal/container package for this to live in.

package container

import (
    "fmt"
    "os"
    "path/filepath"

    "github.com/opencontainers/runtime-spec/specs-go"
)

const (
    containerRootDir = "/var/lib/anocir/containers"
)

type Container struct {
    State *specs.State
    Spec  *specs.Spec
}

type NewContainerOpts struct {
    ID     string
    Bundle string
    Spec   *specs.Spec
}

func New(opts *NewContainerOpts) (*Container, error) {
    if exists(opts.ID) {
        return nil, fmt.Errorf("container '%s' exists", opts.ID)
    }

    state := specs.State{
	Version:     specs.Version,
	ID:          opts.ID,
	Bundle:      opts.Bundle,
	Annotations: opts.Spec.Annotations,
	Status:      specs.StateCreating,
    }

    c := Container{
	State: &state,
	Spec:  opts.Spec,
    }

    return &c, nil
}

func exists(containerID string) bool {
    _, err := os.Stat(filepath.Join(containerRootDir, containerID))

    return err == nil
}

You’ll also notice that we import the github.com/opencontainers/runtime-spec/specs-go package that we installed earlier to make use of the specs.Spec (config) and specs.State structs. It’s worth opening these up and having a read through. Implementing the processing for all of these configuration options is essentially what we’re going to need to do to build the runtime.

Also note that the Status of the container at this point is being set as specs.StateCreating to indicate that the container is currently in the process of being created. As we implement further stages of the lifecycle, this will be updated to indicate where in the lifecycle the container is.

We now need to hook this up to the Create operation handler we defined previously so that when the create command is run then a new container is created.

func Create(opts *CreateOpts) error {
    bundle, err := filepath.Abs(opts.Bundle)
    if err != nil {
	return fmt.Errorf("absolute path from bundle: %w", err)
    }

    config, err := os.ReadFile(filepath.Join(bundle, "config.json"))
    if err != nil {
	return fmt.Errorf("read config file: %w", err)
    }

    var spec *specs.Spec
    if err := json.Unmarshal(config, &spec); err != nil {
        return fmt.Errorf("unmarshall config: %w", err)
    }

    if _, err := container.New(&container.NewContainerOpts{
	ID:     opts.ID,
	Bundle: bundle,
	Spec:   spec,
    }); err != nil {
	return fmt.Errorf("create container: %w", err)
    }

    return nil
}

First, we get an absolute path to the bundle.

Then, we read the config.json file from the bundle location and unmarshal it into a *specs.Spec struct.

Finally, we create a new Container with the container ID, bundle and config.

Saving the container state

To save the container’s state, we need to serialise a JSON representation of it (according to the spec for state.json) and save it to disk at /var/lib/anocir/containers/{{ containerID }}/state.json.

So, we create a Save receiver function on the Container struct to do just that.

func (c *Container) Save() error {
    if err := os.MkdirAll(
	filepath.Join(containerRootDir, c.State.ID),
	0666,
    ); err != nil {
	return fmt.Errorf("create container directory: %w", err)
    }

    state, err := json.Marshal(c.State)
    if err != nil {
	return fmt.Errorf("serialise container state: %w", err)
    }

    if err := os.WriteFile(
	filepath.Join(containerRootDir, c.State.ID, "state.json"),
	state,
	0666,
    ); err != nil {
	return fmt.Errorf("write container state: %w", err)
    }

    return nil
}

First, we create the directory for the container if it doesn’t already exist. Then, we serialise the state and save it to disk.

The state of a container will change throughout its lifecycle and Save can be called at any point to save the current state of the container. So, let’s update the Create operation to do that after newing a Container.

func Create(opts *CreateOpts) error {

    // ...

    cntr, err := container.New(&container.NewContainerOpts{
	ID:     opts.ID,
	Bundle: bundle,
	Spec:   spec,
    })
    if err != nil {
	return fmt.Errorf("create container: %w", err)
    }

    if err := cntr.Save(); err != nil {
	return fmt.Errorf("save container: %w", err)
    }

    return nil
}

Now, let’s check it works as expected.

Create a container bundle

Follow the steps from Creating a bundle. To recap:

1. Create a directory for the bundle: mkdir alpinefs
2. Change into the bundle directory: cd alpinefs
3. Create the rootfs directory: mkdir rootfs
4. Use Docker to export a rootfs: docker export $(docker create alpine) | tar -C rootfs -xvf -
5. Generate a configuration: runc spec
6. Come back out: cd ..

At least for what we’re doing now, we can continue to use this same bundle for future operations; we don’t need to create a new bundle every time.

go build -o anocir

./anocir create --bundle alpinefs test1
./anocir create --bundle alpinefs test2
./anocir create --bundle alpinefs test3

./anocir create --bundle alpinefs test1

Error: create container: container 'test1' exists

cat /var/lib/anocir/containers/test1/state.json | jq
{
  "ociVersion": "1.2.0",
  "id": "test1",
  "status": "creating",
  "bundle": "/home/nixpig/projects/alpinefs"
}

It’s fine to cat out the state.json file for development, as above, but the runtime specifies a state operation to achieve this. We’ll implement that in the next post.

Part 3: Loading a container, getting its state, and deleting it »

References