Skip to content

Read-only root filesystem for the init containers, opt-in for the Solr and exporter pods - #885

Draft
janhoy wants to merge 8 commits into
apache:mainfrom
janhoy:readonly-rootfs
Draft

janhoy wants to merge 8 commits into
apache:mainfrom
janhoy:readonly-rootfs

Conversation

@janhoy

@janhoy janhoy commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

🤖 AI Assisted 🤖

Fixes #884, fixes #801

Stacked on #875, only the last commit belongs to this PR.

  • Every init container the operator generates now runs with readOnlyRootFilesystem: true. They only write to mounted volumes, and openssl in the Solr image was verified to export a keystore on a read-only root.
  • New podOptions.readOnlyRootFilesystem for SolrCloud and SolrPrometheusExporter sets it on the main container and mounts emptyDir volumes at /tmp and, for Solr, /var/solr/logs. A user volume at one of those paths is used instead, and an explicit containerSecurityContext.readOnlyRootFilesystem wins.

Testing

Unit tests cover both parts, and the generated pods still pass the "restricted" profile check. Verified in Docker that Solr 9.11 refuses to start when /var/solr/logs is read-only, and serves with emptyDir volumes at both paths. Not yet run in a Kubernetes cluster.

@janhoy
janhoy marked this pull request as draft November 6, 2026 01:04
Set a seccompProfile on the Solr and Prometheus exporter pods, and
runAsNonRoot, allowPrivilegeEscalation=false and capabilities.drop=[ALL]
on every container and initContainer the operator generates. The
busybox-based initContainers also get runAsUser, since that image has no
non-root user of its own.

podSecurityContext, containerSecurityContext and
defaultInitContainerSecurityContext are now merged over these defaults
field by field, so one field opts out of one default. This also extends
the initContainer securityContext to the TLS initContainers, which it
did not reach before.

The volume-prep initContainer can no longer chown a backup repository
directory, so it checks that the directory is writable and fails with a
message naming it. Setting defaultInitContainerSecurityContext.runAsUser
to 0 restores the previous behaviour.

A unit test evaluates the generated pods against the upstream profile
checks, so a new operator container that is not hardened fails the
build.

Closes apache#873
Closes apache#582
Skip the security context defaults for containers the user supplied
through podOptions, matching them by name rather than relying on the
operatorContainers map, which a user-supplied container can collide
with.

Leave allowPrivilegeEscalation unset when the user asked for privileged
mode or an added CAP_SYS_ADMIN, since Kubernetes rejects either of those
alongside allowPrivilegeEscalation: false.

Document that a root-based custom Solr image needs the init container
override too, as setup-zk and gen-pkcs12-keystore run the same image.
Send the backup repository rejection to stderr and set
FallbackToLogsOnError on the volume-prep container, so the reason shows
up in the pod status rather than only in that container's log.

Document that capabilities is overridden as a whole rather than per
sub-field, with a test pinning that, since an empty block is the only
way to keep a container's default capabilities.

Note that spec.dataStorage.ephemeral.hostPath breaks the "restricted"
profile too, not just the volumes a user adds through podOptions.
The init container override reaching the TLS init containers was the
point of moving the hardening to the end of pod assembly, and nothing
tested it: the three containers added after assembly had no coverage of
a user-supplied securityContext at all.

Also cover the exporter container, which only had pod-level assertions;
the exporter's TLS-from-secret path, which generates a keystore
conversion init container that no fixture reached; a pod that stays
compliant when the user sets an unrelated field, which is the common
case for the merging behaviour; and the volume prep container with
several repositories, where the Solr user must only be added once.
Two of the operator's security context defaults are numeric IDs: fsGroup
8983 on the pod (8080 for the exporter), and runAsUser 8983 on the init
containers whose busybox image has no non-root user of its own. OpenShift's
restricted SCCs only accept IDs from a per-namespace range and fill them in
themselves when unset, and the field-by-field merge gives the user no way
to leave a default out. Everything else the operator sets is accepted.

The new podOptions.omitDefaultIDs leaves both IDs out, for the SolrCloud
and the SolrPrometheusExporter, and the solr chart exposes it. IDs given
explicitly are still used, and the generated pods still satisfy the
"restricted" profile. The volume-prep check now reports the user and
groups it ran as, instead of assuming the default.

Fixes apache#466 without platform detection.
Code comments explained user-facing behaviour and the docs repeated
themselves. Say what each piece needs for its audience and no more. The
readOnlyRootFilesystem note now points at apache#884, which tracks setting it.
Both test harnesses give a SolrCloud a volume backupRepository backed by a
hostPath directory created by root. The kubelet applies no fsGroup to
hostPath volumes, so uid 8983 cannot write there, and the volume-prep
initContainer now fails the pod instead of chowning it as root. The Solr
pods never became ready: the smoke test timed out after 30 minutes, and the
e2e "Local Directory - Single" backup spec after 4.

Prepare the directory in both harnesses, which is what the docs tell users
to do for a volume that comes with its own ownership. The smoke test owns
its directory inside the Kind node, so it can chown it. The e2e Kind config
shares one host directory with every node, so that one is made writable
before Kind creates it as root.
Every init container the operator generates writes only to its mounted
volumes, so all of them now run with readOnlyRootFilesystem: true. openssl
in the Solr image was verified to export a keystore on a read-only root.

The Solr and exporter containers write to /tmp, and Solr to /var/solr/logs,
so they stay writable unless the user asks for podOptions.readOnlyRootFilesystem.
That sets the flag on the main container and mounts emptyDirs at those paths,
unless the user mounts a volume of their own there. Solr 9.11 was verified
to start and serve on a read-only root with those two mounts.

Closes apache#884, closes apache#801.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant