Kubernetes v1.37: Hardening Container Storage with Bind Mount Options and EmptyDir Permissions (kubernetes.io)

by rss-bot · 1 month ago · 0 comments

Kubernetes v1.37 brings important storage security features: <code>emptyDir</code> permission modes and bind mount options. They help application programmers and security professionals implement rigorous security policies, for example, prohibiting deletion of files across containers or execution of arbitrary binaries from writable volumes, directly in Kubernetes without any complicated circumvention.
<h2 id="linux-storage-and-permission-fundamentals">Linux storage and permission fundamentals<a class="td-heading-self-link" href="#linux-storage-and-permission-fundamentals" aria-label="Heading self-link"></a></h2>Before diving into the new Kubernetes features, let us briefly review the low-level Linux security mechanisms that make them possible.
<h3 id="bind-mount-flags">Bind mount flags<a class="td-heading-self-link" href="#bind-mount-flags" aria-label="Heading self-link"></a></h3>When Linux mounts or remounts a directory, Virtual File System (VFS) flags control what actions are permitted on that filesystem:
<ul>
<li><code>noexec</code>: Do not permit direct execution of any binaries on the mounted filesystem.</li>
<li><code>nosuid</code>: Do not allow set-user-identifier or set-group-identifier bits to take effect.</li>
<li><code>nodev</code>: Do not interpret character or block special devices on the file system.</li>
</ul>
<h3 id="directory-permissions-and-the-sticky-bit">Directory permissions and the sticky bit<a class="td-heading-self-link" href="#directory-permissions-and-the-sticky-bit" aria-label="Heading self-link"></a></h3>Standard Unix permissions regulate access across three scopes: Owner, Group, and Others (e.g., <code>0755</code> or <code>0777</code>).
Beyond standard read, write, and execute bits, Linux supports the sticky bit (as in mode <code>01777</code>).
When applied to a directory, the sticky bit ensures that a file inside that directory can only be deleted or renamed by the file's owner or root. This is essential for shared writable directories like <code>/tmp</code>.
<h2 id="motivation-for-the-improvements">Motivation for the improvements<a class="td-heading-self-link" href="#motivation-for-the-improvements" aria-label="Heading self-link"></a></h2>Why does Kubernetes need bind mount options and <code>emptyDir</code> permissions?
The primary goal of these features is to increase the security of Kubernetes workloads by allowing security-related bind mount options on volume mounts. By default, volumes are bind-mounted into containers by the container runtime and kubelet without <code>noexec</code>, <code>nosuid</code>, or <code>nodev</code> flags. This default can undermine security. For example, with <code>noexec</code> missing, a compromised process can use any writable volume (<code>emptyDir</code>, PersistentVolume, etc.) to download, <code>chmod +x</code>, and execute arbitrary binaries even when the container has a read-only root filesystem (<code>readOnlyRootFilesystem: true</code>). Supporting <code>noexec</code>, <code>nodev</code>, and <code>nosuid</code> gives users a native way to harden volume mounts to match security benchmarks and policy.
The gap is most visible with <code>emptyDir</code> volumes, which are the most common writable volume type and have been the subject of multiple security findings:
<ul>
<li><a href="https://github.com/kubernetes/kubernetes/issues/48912">Issue #48912</a>: Recognized security gap - the inability to set mount options on <code>emptyDir</code> was flagged in an audit but remained unresolved until now.</li>
<li><a href="https://github.com/kubernetes/kubernetes/issues/119627">Issue #119627</a>: Kubernetes 1.24 Security Audit (Finding NCC-E003660-7HM) - external auditors specifically noted that the inability to mount <code>emptyDir</code> with <code>noexec</code> represents a security failure.</li>
</ul>
However, the same gap applies to all volume types. PersistentVolumes have a <code>mountOptions</code> field, but those options are filesystem-level flags applied by the CSI driver at the node, so they do not reliably translate into bind mount flags inside the container. Previously, there was no mechanism to set <code>noexec</code>, <code>nosuid</code>, or <code>nodev</code> on the bind mount that the container runtime creates for any volume type.
Additionally, the <code>emptyDir</code> volume type defaults to creating directories with a hardcoded mode of <code>0777</code>.
This previously meant that any process that can discover the volume could read, write, and delete anything in the volume, regardless of who created it.
You could - and still can - use an initial container to set a different access mode, but this is more complex, and hard to verify for compliance.
This causes real problems:
<ul>
<li>Multi-container pods sharing an <code>emptyDir</code> could not prevent one container from deleting another's files. The sticky bit (<code>01777</code>) solves this, but there was no native way to set it.</li>
<li>Some applications and security frameworks expect <code>/tmp</code> directories to have the sticky bit set (mode <code>01777</code>). Without native support for setting the <code>emptyDir</code> mode, users had to use init containers or alternative volume types to meet this requirement.</li>
<li>Platform engineers who want tighter permissions (e.g., <code>0750</code> for owner and group only) have to use init containers running <code>chmod</code>, which adds unnecessary complexity.</li>
</ul>
The <code>emptyDir</code> volume type was a notable gap.
As one of the most common writable volume types in Kubernetes, it had no way to control its creation permissions.
<h2 id="real-world-use-cases">Real-world use cases<a class="td-heading-self-link" href="#real-world-use-cases" aria-label="Heading self-link"></a></h2>Application developers, working closely with security engineers, are responsible for maintaining the security posture of their applications and ensuring workloads do not pose risks to the wider infrastructure. These features allow development teams to confidently address critical security scenarios:
Preventing Privilege Escalation on Writable Mounts: An application developer configuring temporary workspace volumes (like <code>emptyDir</code> or <code>/tmp</code> mounts) can ensure they are mounted with <code>nosuid</code> and <code>noexec</code>. This guarantees that even if the application is compromised and a malicious payload is downloaded, the workload cannot execute the payload or use it to escalate privileges on the node.
Securing Shared Scratch Space in Multi-Container Pods: A developer configuring CI/CD pipeline pods often needs multiple containers (e.g., a builder container and a sidecar logger) to share a workspace. By setting <code>mode: 01777</code> on an <code>emptyDir</code>, the developer ensures the shared workspace behaves like a traditional Unix <code>/tmp</code> directory. Each container can write files independently, but a compromised process in one container cannot delete the build artifacts produced by another.
Enforcing Principle of Least Privilege for Application Data: An application developer deploying a database pod can lock down access to the database's temporary storage. By setting <code>mode: 0750</code> on the <code>emptyDir</code>, the developer ensures that only the specific database user and group can read or write to the volume, explicitly denying access to any other processes or sidecars in the same pod.
Note: Both features are behind Alpha feature gates in Kubernetes v1.37. To use them, enable <code>VolumeBindMountOptions</code> and <code>EmptyDirVolumeMode</code> on the API server and kubelet.
<h2 id="example-1-enforcing-bind-mount-options">Example 1: Enforcing bind mount options<a class="td-heading-self-link" href="#example-1-enforcing-bind-mount-options" aria-label="Heading self-link"></a></h2>This full Pod manifest mounts an <code>emptyDir</code> volume at <code>/tmp</code> with <code>bindMountOptions: [noexec, nosuid]</code>.
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml">apiVersion: v1
kind: Pod
metadata:
 name: hardened-bindmount-pod
 namespace: default
spec:
 os:
 name: linux
 containers:
 - name: hardened-app
 image: alpine:latest
 command: ["sleep", "3600"]
 securityContext:
 readOnlyRootFilesystem: true
 volumeMounts:
 - name: temp-storage
 mountPath: /tmp
 bindMountOptions:
 - noexec
 - nosuid
 volumes:
 - name: temp-storage
 emptyDir: {}
</code></pre></div><h2 id="example-2-emptydir-volume-permission-mode-with-sticky-bit">Example 2: <code>emptyDir</code> volume permission mode with sticky bit<a class="td-heading-self-link" href="#example-2-emptydir-volume-permission-mode-with-sticky-bit" aria-label="Heading self-link"></a></h2>This full Pod manifest creates an <code>emptyDir</code> volume using <code>mode: 01777</code> to enforce standard Unix <code>/tmp</code> sticky bit protections across containers.
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml">apiVersion: v1
kind: Pod
metadata:
 name: hardened-emptydir-pod
 namespace: default
spec:
 os:
 name: linux
 containers:
 - name: app-container
 image: alpine:latest
 command: ["sleep", "3600"]
 volumeMounts:
 - name: shared-tmp
 mountPath: /tmp
 volumes:
 - name: shared-tmp
 emptyDir:
 mode: 01777
</code></pre></div><h2 id="verifying-the-features-in-linux">Verifying the features in Linux<a class="td-heading-self-link" href="#verifying-the-features-in-linux" aria-label="Heading self-link"></a></h2>To verify that these features are actively enforcing restrictions, you can run <code>kubectl exec</code> into the container. The following examples simulate attempts to perform actions that are successfully blocked by these features.
<h3 id="verifying-noexec">Verifying noexec<a class="td-heading-self-link" href="#verifying-noexec" aria-label="Heading self-link"></a></h3>Attempt to write and run a script on a volume mounted with <code>noexec</code>:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell"># 1. Exec into the pod
kubectl exec -it hardened-bindmount-pod -- sh

# 2. Create an executable script on the mounted volume
cd /tmp
echo '#!/bin/sh' > test.sh
echo 'echo "Executing untrusted code..."' >> test.sh
chmod +x test.sh

# 3. Attempt to run the script
./test.sh
</code></pre></div>Expected result:
<pre tabindex="0"><code class="language-none" data-lang="none">sh: ./test.sh: Permission denied
</code></pre>Even if an executable file is created, the Linux kernel refuses execution because <code>MS_NOEXEC</code> is enforced at the bind mount level.
<h3 id="verifying-the-sticky-bit">Verifying the sticky bit<a class="td-heading-self-link" href="#verifying-the-sticky-bit" aria-label="Heading self-link"></a></h3>Attempt to delete another user's file in an <code>emptyDir</code> with <code>01777</code> permission mode:
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell"># 1. Exec into the pod
kubectl exec -it hardened-emptydir-pod -- sh

# 2. Verify directory permissions on /tmp
ls -ld /tmp
# Output: drwxrwxrwt 2 root root ... /tmp (Notice the 't' indicating sticky bit)

# 3. Create a file as the guest user
su -s /bin/sh -c "touch /tmp/guest_file" guest

# 4. Attempt to delete that file as nobody
su -s /bin/sh -c "rm /tmp/guest_file" nobody
</code></pre></div>Expected result:
<pre tabindex="0"><code class="language-none" data-lang="none">rm: can't remove '/tmp/guest_file': Operation not permitted
</code></pre>The kernel blocks deletion because the sticky bit (<code>01777</code>) restricts file removal strictly to the owner of the file.
<h2 id="things-to-know">Things to know<a class="td-heading-self-link" href="#things-to-know" aria-label="Heading self-link"></a></h2>Keep these key details in mind as you begin using these features. Full details are available in the official documentation for <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-bind-mount-options/">bind mount options</a>, <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-emptydir-volume-mode/"><code>emptyDir</code> volume mode</a>, and <a href="https://kubernetes.io/docs/concepts/storage/volumes/#emptydir"><code>emptyDir</code> volumes</a>.
<ul>
<li>Default unchanged: If you omit <code>bindMountOptions</code> or do not set an <code>emptyDir</code> <code>mode</code>, you get standard default behaviors (like <code>0777</code> permissions) exactly as before.</li>
<li>Broad volume support: <code>bindMountOptions</code> works with <code>emptyDir</code>, PersistentVolumes, CSI volumes, projected volumes, ConfigMaps, Secrets, and more. The only exception is image volumes, which are explicitly unsupported. The <code>mode</code> field works with all <code>emptyDir</code> medium types: default (disk-backed), <code>Memory</code> (tmpfs), and <code>HugePages</code>.</li>
<li>Runtime capabilities matter (for <code>bindMountOptions</code>): The container runtime must support the CRI <code>mount_options</code> field and advertise it via <code>runtimeFeatures</code>.
The scheduler uses node declared features to avoid placing pods on incompatible nodes.
If a pod reaches such a node anyway, the kubelet rejects it. There is no silent degradation.
However, using <code>mode</code> for an <code>emptyDir</code> does not require runtime support.</li>
<li>Not the same as PV mountOptions: PersistentVolume <code>mountOptions</code> apply at the storage layer via the CSI driver. The new <code>bindMountOptions</code> controls bind mount flags applied inside the container by the runtime. They operate at different layers and do not conflict.</li>
<li>fsGroup interaction: If <code>fsGroup</code> is set in the pod's security context, the group permissions applied by <code>fsGroup</code> will override the <code>mode</code> specified for the <code>emptyDir</code> volume.
This is the same behavior that exists for <code>defaultMode</code> on Secret and ConfigMap volumes.</li>
<li>Linux only: Flags like <code>noexec</code>, <code>nosuid</code>, <code>nodev</code>, and Unix permission modes are Linux concepts.
<code>bindMountOptions</code> has no effect on Windows nodes.
On Windows, the <code>mode</code> field is also skipped for <code>emptyDir</code> volumes, since Windows does not support Unix-style file permissions.</li>
<li>Version skew safety: Both features are additive. For <code>emptyDir</code> <code>mode</code>: if the API server has the gate enabled but the kubelet does not, the field is accepted but ignored - the kubelet falls back to <code>0777</code>. For <code>bindMountOptions</code>: the scheduler uses Node Declared Features to prevent placing pods on nodes without runtime support; if a pod reaches such a node, the kubelet rejects it rather than silently ignoring the options.</li>
<li>Feature Gates: Both capabilities are available as Alpha features in Kubernetes v1.37:
<ul>
<li><code>VolumeBindMountOptions</code>: Controls bind mount flags on volume mounts.</li>
<li><code>EmptyDirVolumeMode</code>: Controls creation permission modes on <code>emptyDir</code> volumes.</li>
</ul>
</li>
</ul>
<h2 id="how-do-i-get-involved">How do I get involved?<a class="td-heading-self-link" href="#how-do-i-get-involved" aria-label="Heading self-link"></a></h2>These new features are driven by <a href="https://www.kubernetes.dev/community/community-groups/sigs/node/">SIG Node</a> and <a href="https://www.kubernetes.dev/community/community-groups/sigs/storage/">SIG Storage</a>.
You can find more details in the KEPs for these enhancements: <a href="https://www.kubernetes.dev/resources/keps/5855/">KEP-5855</a> (bind mount options) and <a href="https://www.kubernetes.dev/resources/keps/5502/">KEP-5502</a> (<code>emptyDir</code> permission mode).
Reach out to SIG Node:
<ul>
<li>Slack: <a href="https://kubernetes.slack.com/messages/sig-node">#sig-node</a></li>
<li><a href="https://groups.google.com/forum/#!forum/kubernetes-sig-node">Mailing list</a></li>
</ul>
Reach out to SIG Storage:
<ul>
<li>Slack: <a href="https://kubernetes.slack.com/messages/sig-storage">#sig-storage</a></li>
<li><a href="https://groups.google.com/forum/#!forum/kubernetes-sig-storage">Mailing list</a></li>
</ul>

AI News

Kubernetes v1.37: Hardening Container Storage with Bind Mount Options and EmptyDir Permissions (kubernetes.io)

Comments