Executing container runtime lifecycle hooks

Jan 21st, 2025

7 min

This is Part 5 of the series Building a container runtime from scratch in Go.

In the fifth part of the series we handle the execution of hooks according to the phase of the container lifecycle.

The source code to accompany this post is available on GitHub.

Throughout the lifecycle¹ of a container, the runtime will execute hooks² according to the phase of the lifecycle. Hooks are simply commands that run in either the runtime or container namespace.

Hook	Namespace	Lifecycle	Use
`prestart`	Runtime	During the creation of the container, after container namespaces are created and before root mount is applied.	Typically used to customise the container.
`createRuntime`	Runtime	During the creation of the container, after container namespaces are created, before the `createContainer` hooks are called and before root mount is applied.	Typically used to customise the container.
`createContainer`	Container	During the creation of the container, after container mount namespace is created, after the `createRuntime` hooks are called and before root mount is applied.
`startContainer`	Container	When starting the container, before the user-specified process is started.	Typically used to execute operations in the container before starting the user-specified process.
`poststart`	Runtime	Immediately after starting the user-specified process without waiting for it to return.	Typically used to notify callees that the container process has started.
`poststop`	Runtime	After the container is deleted.	Typically used for cleanup.

Runtime hooks called thoughout the lifecycle of a container

You’ve probably observed a couple of ‘quirks’ above:

Some of the hook names are lowercase while others are pascal case.
The prestart and createRuntime hooks are basically the same.

In the past, there was only prestart, poststart and poststop hooks. More recently, the prestart hook was deprecated in favor of the more specific createRuntime, createContainer and startContainer hooks. However, the OCI Runtime test suite, Docker, and others still use the prestart hook, so we’re going to keep it for posterity and compatibility.

Hooks are defined in the bundle’s config.json and the spec includes the following Go structs for representing them.

// Hook specifies a command that is run at a particular event
// in the lifecycle of a container
type Hook struct {
    Path    string   `json:"path"`
    Args    []string `json:"args,omitempty"`
    Env     []string `json:"env,omitempty"`
    Timeout *int     `json:"timeout,omitempty"`
}

// ...

type Hooks struct {
    Prestart []Hook `json:"prestart,omitempty"`
    CreateRuntime []Hook `json:"createRuntime,omitempty"`
    CreateContainer []Hook `json:"createContainer,omitempty"`
    StartContainer []Hook `json:"startContainer,omitempty"`
    Poststart []Hook `json:"poststart,omitempty"`
    Poststop []Hook `json:"poststop,omitempty"`
}

The fields in Hook should look familiar; they’re what you’d typically pass to the exec syscall. Path is the command to run, Args are the arguments to pass to the command, and Env is the environment variables. Additionally, an optional Timeout can be specified to timeout the operation after a given number of seconds.

In Hooks is a field for each of the phases of the lifecycle which hold an array of Hooks to execute.

Executing the hooks

Notes

When a hook is executed, the runtime is expected to pass a JSON representation of the current state of the container to stdin.
When a hook fails to execute, it must return an error, with the exception of the poststop hook which only logs a warning.

In order to execute the hooks, we’ll create a new internal/hooks/hooks.go package that exports an ExecHooks function.

func ExecHooks(hooks []specs.Hook, state *specs.State) error {
    s, err := json.Marshal(state)
    if err != nil {
	return fmt.Errorf("marshal state: %w", err)
    }

    for _, h := range hooks {
	ctx := context.Background()

	if h.Timeout != nil {
	    var cancel context.CancelFunc
	    ctx, cancel = context.WithTimeout(
		ctx,
		time.Duration(*h.Timeout)*time.Second,
	    )
	    defer cancel()
	}

	binary, err := exec.LookPath(h.Path)
	if err != nil {
	    return fmt.Errorf("find path of hook binary: %w", err)
	}

	path := filepath.Dir(h.Path)

	cmd := exec.CommandContext(ctx, binary, path)

	cmd.Args = append(h.Args, string(s))
	cmd.Env = h.Env
	cmd.Stdin = strings.NewReader(string(s))

	if err := cmd.Run(); err != nil {
	    return fmt.Errorf("execute hook %s: %w", h.Path, err)
	}
    }

    return nil
}

The ExecHooks function takes in a slice of Hooks to execute and a pointer to the state of the container.

The first thing we do is serialise the state of the container.

We then iterate over the hooks, and for each of them:

Create a new Context to use for the execution of the hook.
If the hook has a Timeout specified, we update the Context with a timeout.
Lookup the path to the specified executable.
Create a new Cmd with the context, executable and path to it.
Apply the Args and Env to the command.
Run the command which executes the hook.

We now need to pepper calls to this function at the correct spots in our code and pass the corresponding hooks. The hooks to execute are available to us on c.Spec.Hooks with the same name as the lifecycle phase in which they’re executing. For example, the createRuntime hooks are at c.Spec.Hooks.CreateRuntime.

Let’s go through and do that now.

func (c *Container) Init() error {
    if c.Spec.Hooks != nil {
	if err := hooks.ExecHooks(
	    c.Spec.Hooks.CreateRuntime, c.State,
	); err != nil {
	    return fmt.Errorf("exec createruntime hooks: %w", err)
	}
    }

    cmd := exec.Command("/proc/self/exe", "reexec", c.State.ID)

    // ...

    defer listener.Close()

    if c.Spec.Hooks != nil {
	if err := hooks.ExecHooks(
	    c.Spec.Hooks.CreateContainer, c.State,
	); err != nil {
	    return fmt.Errorf("exec createcontainer hooks: %w", err)
	}
    }

    if err := cmd.Start(); err != nil {
	return fmt.Errorf("reexec container process: %w", err)
    }

    // ...

    containerConn.Close()
    listener.Close()

    if c.Spec.Hooks != nil {
	if err := hooks.ExecHooks(
	    c.Spec.Hooks.StartContainer, c.State,
	); err != nil {
	    return fmt.Errorf("exec startcontainer hooks: %w", err)
	}
    }

    bin, err := exec.LookPath(c.Spec.Process.Args[0])

    // ...

    if !c.canBeStarted() {
	return fmt.Errorf("container cannot be started in current state (%s)", c.State.Status)
    }

    if c.Spec.Hooks != nil {
	if err := hooks.ExecHooks(
	    //lint:ignore SA1019 marked as deprecated, but still required by OCI Runtime integration tests and used by other tools like Docker
	    c.Spec.Hooks.Prestart, c.State,
	); err != nil {
	    return fmt.Errorf("execute prestart hooks: %w", err)
	}
    }

    conn, err := net.Dial(
	"unix",
	filepath.Join(containerRootDir, c.State.ID, containerSockFilename),
    )

    // ...

    c.State.Status = specs.StateRunning

    if c.Spec.Hooks != nil {
	if err := hooks.ExecHooks(
	    c.Spec.Hooks.Poststart, c.State,
	); err != nil {
	    return fmt.Errorf("exec poststart hooks: %w", err)
	}
    }

    return nil
}

func (c *Container) Delete(force bool) error {

    // ...

    if err := os.RemoveAll(
	filepath.Join(containerRootDir, c.State.ID),
    ); err != nil {
	return fmt.Errorf("delete container directory: %w", err)
    }

    if c.Spec.Hooks != nil {
	if err := hooks.ExecHooks(
	    c.Spec.Hooks.Poststop, c.State,
	); err != nil {
	    fmt.Printf("Warning: failed to exec poststop hooks: %s\n", err)
	}
    }

    return nil
}

Remember, Hooks is an optional field, so we need to do a nil check before trying to call ExecHooks.

With all those in place, we’re ready to give it a try. Let’s update the config.json for the container we’ll create to include some hooks.

{

    // ...

    "hooks": {
	"prestart": [
	    {
		"path": "/bin/sh",
		"args": ["sh", "-c", "touch /tmp/hook-prestart"]
	    }
	],
	"createRuntime": [
	    {
		"path": "/bin/sh",
		"args": ["sh", "-c", "touch /tmp/hook-createRuntime"]
	    }
	],
	"createContainer": [
	    {
		"path": "/bin/sh",
		"args": ["sh", "-c", "touch /tmp/hook-createContainer"]
	    }
	],
	"startContainer": [
	    {
		"path": "/bin/sh",
		"args": ["sh", "-c", "rm /tmp/hook-prestart /tmp/hook-createRuntime /tmp/hook-createContainer"]
	    },
	    {
		"path": "/bin/sh",
		"args": ["sh", "-c", "touch /tmp/hook-startContainer"]
	    }
	],
	"poststart": [
	    {
		"path": "/bin/sh",
		"args": ["sh", "-c", "touch /tmp/hook-poststart"]
	    }
	],
	"poststop": [
	    {
		"path": "/bin/sh",
		"args": ["sh", "-c", "rm /tmp/hook-startContainer /tmp/hook-poststart"]
	    }
	]
    }

    // ...

}

At the various phases of the lifecycle we’re creating and deleting files so we can verify the hooks are in fact being executed.

Let’s create a container.

./anocir create --bundle alpinefs test1

After creating a container, we expect the createRuntime and createContainer hooks to have run, which should have created corresponding temporary files.

ll -la /tmp/hook-*

Permissions Size User Date Modified Name
.rw-r--r--     0 root 18 Jan 16:42   /tmp/hook-createContainer
.rw-r--r--     0 root 18 Jan 16:42   /tmp/hook-createRuntime

Next, we’ll start the container.

./anocir start test1

After starting the container, we expect the startContainer and poststart hooks to have run, which should have deleted the temporary files for createRuntime and createContainer, and created temporary files for startContainer and poststart.

ls -la /tmp/hook-*

Permissions Size User Date Modified Name
.rw-r--r--     0 root 18 Jan 16:43   /tmp/hook-poststart
.rw-r--r--     0 root 18 Jan 16:43   /tmp/hook-startContainer

Finally, we’ll delete the container.

./anocir delete test1 --force

On deletion of the container, we expect the poststop hook to have run, which should have deleted the startContainer and poststart temporary files.

ls -la /tmp/hook-*

"/tmp/hook-*": No such file or directory (os error 2)

That seems to be it!

There’s still one more runtime operation that we haven’t implemented yet - kill. Let’s do that next.

Part 6: Sending signals to a running container 🔜 Coming soon!

References

https://github.com/opencontainers/runtime-spec/blob/main/runtime.md#lifecycle “Runtime lifecycle” ↩︎
https://github.com/opencontainers/runtime-spec/blob/main/config.md#posix-platform-hooks “Posix platform hooks” ↩︎