Recent update to moby-runc causes execution failures on Ubuntu:Focal (20.04)

[robinmholt-daikin]

Description

I have Ubuntu focal based devices with long-term support. We are using the moby-* packages from https://packages.microsoft.com/ubuntu/20.04/prod The systems run unattended upgrades so receive updates without intervention. They are also running Microsoft's aziot-edged.service which is orchestrating containers on the system.

On 19 June, 2026, we saw moby-runc upgrade from 1.4.2-ubuntu20.04u1 to 1.4.3-ubuntu20.04u1, at which time, all containers failed to start. We quickly set the devices back to 1.4.2 and apt-mark hold moby-runc to get things back to operational.

I finally had an opportunity to investigate more thoroughly today and found the problem was really easy to reproduce as all docker run ... commands failed.

Steps to reproduce the issue

root@SOLU220609002:~# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
4f55086f7dd0: Pull complete
Digest: sha256:96498ffd522e70807ab6384a5c0485a79b9c7c08ca79ba08623edcad1054e62d
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: can't mask dir "/proc/acpi": mount src=tmpfs, dst=/proc/acpi, flags=MS_RDONLY, data=nr_blocks=1,nr_inodes=1: invalid argument

Run 'docker run --help' for more information

Describe the results you received and expected

I would have expected normal hello-world output

What version of runc are you using?

runc version 1.5.0-1
commit: c4bb59526d0c9cf3a3a46a04d08ca031749a2119
spec: 1.3.0
go: go1.25.11
libseccomp: 2.5.1

Host OS information

NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
IMAGE_ID="system-manager-full"
IMAGE_VERSION="20250108-183605"

Host kernel information

Linux SOLU220609002 5.4.0-216-generic #236-Ubuntu SMP Fri Apr 11 19:53:21 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

[robinmholt-daikin]

I had created a bug report in moby/moby project as well, since I have no idea what I am doing.

moby/moby#52972

[thaJeztah]

Could be related to;

• [1.4] libct: reuse tmpfs for directory masks #5281

[lifubang]

Thanks for the detailed report. I've investigated the root cause and would like to summarize all findings and a proposed fix.

Root cause

The error is triggered by maskDir() introduced in #5275 (backported to release-1.4 via #5281), which mounts a read-only tmpfs on masked directories with nr_blocks=1,nr_inodes=1 to enforce defense-in-depth:

func maskDir(path, mountLabel string) error {
    return mount("tmpfs", path, "tmpfs", unix.MS_RDONLY,
        label.FormatMountLabel("nr_blocks=1,nr_inodes=1", mountLabel))
}

On most kernels this works fine. However, Ubuntu 20.04 (Focal) with the official 5.4 kernel carries a private patch in mm/shmem.c that rejects nr_inodes=1(focal 5.4.0-216 kernel, as pointed out by @cpuguy83 ):

case Opt_nr_inodes:
    ctx->inodes = memparse(param->string, &rest);
    if (*rest || ctx->inodes < 2)   // Ubuntu-only check
        goto bad_value;

This check is not present in mainline Linux (torvalds/linux v5.4 nor linux-stable 5.4.y), nor in Ubuntu 22.04+ kernels. It appears to have been introduced by Canonical's kernel team as a local hardening measure and was not carried forward to newer Ubuntu kernels.

Scope

After checking multiple kernel sources:

• Affected: Ubuntu 20.04 LTS (focal) with official 5.4 kernel, all versions from 5.4.0-20 onwards
• Not affected: Ubuntu 22.04+ (5.15+ kernel), Debian, RHEL/CentOS, mainline Linux, linux-stable

So this is a kernel-specific issue rather than a runc bug. However, runc should be compatible with commonly used kernels, and Ubuntu 20.04 LTS is still widely deployed.

Why not just change nr_inodes to 2?

While nr_inodes=2 would fix this specific case, it is not the right approach. The value nr_inodes=1 has clear and intentional semantics in the context of maskDir: the tmpfs is mounted read-only purely to mask a directory, and it should consume the minimum possible resources. Changing it to nr_inodes=2 would work, but the value 2 has no logical relationship to the purpose of this function — a reader would reasonably ask "why 2, and not 1 or 3?". The answer would be "because one specific kernel version rejects 1", which is a kernel bug workaround encoded into the value itself, rather than being a property of the mask operation.

In other words, nr_inodes=1 is semantically correct for this use case. We should not weaken a deliberate design choice to accommodate a kernel that incorrectly rejects a valid mount(2) option.

Proposed fix

I propose a fallback approach: try the mount with nr_inodes=1 first (preserving defense-in-depth for kernels that support it), and fall back to mounting without nr_inodes only if the kernel rejects it:

func maskDir(path, mountLabel string) error {
    err := mount("tmpfs", path, "tmpfs", unix.MS_RDONLY,
        label.FormatMountLabel("nr_blocks=1,nr_inodes=1", mountLabel))
    if err != nil {
        // Kernels with a strict lower bound on nr_inodes (e.g. Ubuntu
        // 20.04 5.4 kernel) reject nr_inodes=1. Fall back to leaving
        // nr_inodes unbounded; nr_blocks=1 still limits the tmpfs size
        // and MS_RDONLY prevents writes.
        err = mount("tmpfs", path, "tmpfs", unix.MS_RDONLY,
            label.FormatMountLabel("nr_blocks=1", mountLabel))
    }
    return err
}

This way:

1. Kernels that support nr_inodes=1 get the full defense-in-depth
2. Kernels that reject nr_inodes=1 still get MS_RDONLY + nr_blocks=1
3. No behavior change on mainstream kernels

Happy to hear thoughts from you before sending a PR.

[cpuguy83]

Why leave it unset when you can do 2?

[cyphar]

Yeah, the LLM's conclusion is dumb -- just use nr_inodes=2 and leave a comment...?

[lifubang]

Why leave it unset when you can do 2?

My original thought was to keep the fallback behavior the same as before, since runc didn't set either of them in the past. But I doubt any custom kernel would actually want to block a specific ‘nr_blocks’ value.

Yeah, the LLM's conclusion is dumb -- just use nr_inodes=2 and leave a comment...?

I did consider this change. However, I reasoned that if a custom kernel modifies this value to 3 or something else, it would force us to make changes on our end.

[thaJeztah]

Yeah, not sure if we need to account for really custom kernels. The Ubuntu 20.04 patch is already a bit of an exception (but it's a commonly used distro, so it's reasonable to account for that). Note that Ubuntu 20.04 reached end of standard support, and is now only supported through ESM (Extended Security Maintenance) / Pro; iIRC, docker no packages for 20.04.

[lifubang]

Yeah, not sure if we need to account for really custom kernels.

So I’d prefer not to hardcode this value in runc.

but it's a commonly used distro, so it's reasonable to account for that

This adds a fallback to the previous runc behavior -- mounting tmpfs without the nr_inodes=1 option. The fallback should only be triggered on kernels that reject this option; it won’t be hit on standard mainline Linux kernels.

[robinmholt-daikin]

@lifubang - I don't know how Milestone tagging works for this project, but 1.5.0 also needs this fix.

[cyphar]

Yeah, not sure if we need to account for really custom kernels.

We had all sorts of hacks for RHEL kernels (including the incredibly broken RHEL 6 and 7 frankenkernels), so this stuff by Ubuntu is really quite pedestrian. I also don't really think of a modern-ish distro kernel as being "custom".

However, I reasoned that if a custom kernel modifies this value to 3 or something else, it would force us to make changes on our end.

But that a) seems very unlikely; and b) is equating a real distro kernel with some theoretical custom kernel that doesn't exist.

For what it's worth, I tracked it down and the patch actually comes from AUFS, of all places. Given that AUFS hasn't been touched since 2019 and nr_inodes has been supported by the kernel since the introduction of tmpfs back in Linux 2.3.51pre1, I think just special-casing this nr_inode=2 weirdness is better than removing it. (Especially since my reading of #5353 is that we will now share a tmpfs without inode restrictions for all masks?)

[lifubang]

I think just special-casing this nr_inode=2 weirdness is better than removing it.

Make sense, changed now.