# Production-Grade MongoDB Replica Set on k3s — for Homelabs, Raspberry Pis, and Beyond

> **Heads up — this is a long one.** Grab a coffee. By the end you'll have a 3-node MongoDB replica set running on k3s, fronted by a real Let's Encrypt certificate, fully managed by ArgoCD, and reachable from DataGrip / Compass on your laptop with one paste-able URI. No private CA gymnastics. No "works on my cluster". Tested on a single Raspberry Pi 5 — designed to scale to a multi-node lab without rewriting anything.

![](https://cdn.hashnode.com/uploads/covers/658eba076064e1590b3a413e/272081d6-4b2c-43c4-b664-c2c8192041b7.png align="center")

I went looking for *the* blog on this and never found it. There are a hundred posts that say "just use Bitnami's Helm chart". There are zero that walk through the *why* behind every choice — why a StatefulSet, why three different services, why `allowTLS` and not `requireTLS`, why a per-pod ClusterIP for each replica, what `replicaSetHorizons` actually does, how to make external clients trust your cert without installing a custom CA. So I wrote the one I wished I'd found.

This post is for the homelab person who:

*   Already has k3s (or a basic kubeadm/RKE2 cluster) running on a Pi or a small NUC.
    
*   Wants a "close to production" MongoDB — replica set, TLS, secrets in git, GitOps-managed — without standing up a Kubernetes Operator just to run one DB.
    
*   Wants the same manifests to work unchanged on a 12-node Talos cluster later.
    

We're going to build it step by step, file by file, with every YAML and every shell helper included verbatim. Stick with me.

## 1\. What we're building

A 3-member MongoDB 7 **replica set** named `rs0`, deployed as a single StatefulSet on a `databases` Kubernetes namespace. From outside the cluster, each replica is reachable at its own public hostname:

```plaintext
mongo-0.home.your-domain.tld:27017
mongo-1.home.your-domain.tld:27017
mongo-2.home.your-domain.tld:27017
```

…with a real Let's Encrypt certificate — meaning DataGrip, MongoDB Compass, the official drivers, and even `curl` trust it out of the box, no custom CA install required. From inside the cluster, your apps connect to a single ClusterIP and the driver auto-discovers the topology.

The whole thing is reconciled by **ArgoCD** from a git repo, so "deploying" really means "git push". Secrets are committed too — encrypted with **Bitnami SealedSecrets** so only the controller in your cluster can decrypt them. TLS certs are issued and renewed by **cert-manager** using a Cloudflare DNS-01 challenge.

### Why each design choice

*   **StatefulSet, not Deployment.** Replica-set members must have stable names — `mongodb-0`, `mongodb-1`, `mongodb-2` — because `rs.conf().members[].host` records them by name. Random pod names from a Deployment would break the RS the first time a pod restarted.
    
*   `podManagementPolicy: Parallel`**.** The default `OrderedReady` blocks pod-N until pod-(N-1) is `Ready`. But mongod isn't `Ready` until the RS is initiated, and the RS can't be initiated until all 3 pods exist. Classic chicken-and-egg. `Parallel` starts all three concurrently.
    
*   **A** `postStart` **lifecycle hook on pod-0** does `rs.initiate(...)` once. It's idempotent — error code 23 ("already initialized") is treated as success.
    
*   **Three services for three different audiences:** the headless service for the StatefulSet's own pod DNS, a regular ClusterIP for in-cluster apps that just want "a Mongo", and three per-pod ClusterIPs for Traefik to route SNI-based external traffic to specific replicas.
    
*   **TLS in** `allowTLS` **mode** — accepts both encrypted and plain connections, but doesn't *initiate* TLS for outbound replica-set heartbeats. This matters because the LE cert only covers public hostnames, not internal cluster DNS, and we don't want intra-RS traffic failing TLS validation against names not in the SAN.
    
*   `replicaSetHorizons` so external clients see external hostnames in `db.hello()`, not internal cluster DNS they can't resolve.
    

If half of those bullets sound like Greek, don't worry — every single one is unpacked below.

* * *

## 2\. Prerequisites

This post starts where most homelab posts end. If you don't already have these pieces, here are the canonical "install this once" guides — none take more than an afternoon.

| You need | Why | Get started |
| --- | --- | --- |
| A **Kubernetes cluster** (k3s / kubeadm / RKE2 / Talos) | Where everything runs. Single-node Pi is fine. | [k3s quick install](https://docs.k3s.io/quick-start) · [kubeadm](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/) · [Talos](https://www.talos.dev/v1.7/introduction/getting-started/) |
| **kubectl** configured to point at it | The CLI you'll live in. | [kubectl install](https://kubernetes.io/docs/tasks/tools/) |
| **cert-manager** | Issues and renews TLS certs declaratively. | [cert-manager install](https://cert-manager.io/docs/installation/) |
| **Traefik** ingress controller | k3s ships it by default. We just add a TCP entrypoint. | [k3s + Traefik](https://docs.k3s.io/networking/networking-services#traefik-ingress-controller) · [standalone Helm](https://doc.traefik.io/traefik/getting-started/install-traefik/) |
| **Bitnami SealedSecrets** controller | Encrypts your Secret YAML so it can live in git. | [Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets#installation) |
| **ArgoCD** *(optional but recommended)* | GitOps reconciler. Without it, just `kubectl apply` by hand. | [Argo CD install](https://argo-cd.readthedocs.io/en/stable/getting_started/) |
| **kubeseal** CLI on your laptop | To encrypt secrets locally before committing. | [kubeseal releases](https://github.com/bitnami-labs/sealed-secrets/releases) |

I also keep [these explainers](https://www.youtube.com/c/TechnoTim) bookmarked — Techno Tim's homelab content covers k3s + Traefik + cert-manager wiring better than any blog. If GitOps is new to you, the [Argo CD ApplicationSet docs](https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/) are short and excellent.

### Choose your domain story

This is the **most flexible** part of the setup, and the part I see people get blocked on. There are three sane paths — pick whichever fits your situation. The MongoDB manifests change in only a couple of places between them.

#### Option A — You own a real domain (the path this post defaults to)

This is what I use, and what gives the cleanest experience: **public Let's Encrypt cert, no CA install on clients, works from anywhere on or off the LAN**.

*   A registered domain — `.dev`, `.com`, `.io`, `.xyz`, anything ACME-supported. Cloudflare is cheapest for `.com`\-class TLDs and free for DNS hosting.
    
*   DNS managed by a provider cert-manager supports for **DNS-01**. The popular ones: [Cloudflare](https://cert-manager.io/docs/configuration/acme/dns01/cloudflare/), [Route 53](https://cert-manager.io/docs/configuration/acme/dns01/route53/), [DigitalOcean](https://cert-manager.io/docs/configuration/acme/dns01/digitalocean/), [Hetzner via webhook](https://github.com/vadimkim/cert-manager-webhook-hetzner), [Google Cloud DNS](https://cert-manager.io/docs/configuration/acme/dns01/google/).
    
*   A **wildcard A record** `*.home.your-domain.tld → <your-cluster-ip>`. The IP can be a *private* LAN address like `192.168.0.108` — DNS-01 doesn't need the LE servers to reach you, only Cloudflare's DNS.
    
*   An **API token** for your provider (e.g. Cloudflare with `Zone:DNS:Edit`), stored as a Secret named `cloudflare-api-token` in the `cert-manager` namespace.
    

I use `home.your-domain.tld` throughout this post — search & replace it with your own domain. Anywhere you see `mongo-N.home.your-domain.tld`, mentally substitute `mongo-N.home.your-domain.tld`.

> **Why "private IP" works:** with DNS-01 the ACME server proves you own the domain by querying a TXT record at your DNS provider — it never connects back to your cluster. So your LAN IP being unroutable from the internet is irrelevant. Same trick that makes wildcard certs trivial.

#### Option B — No public domain, just `.local` / Pi-hole / `/etc/hosts`

Perfectly valid for a private homelab. You won't get a public LE cert (no DNS-01 target), but everything else works. You have two sub-options for TLS:

1.  **Skip TLS entirely.** Set `tls.mode: disabled` in `mongod.conf`, drop the cert-manager Certificate and the Traefik IngressRouteTCP, and expose the per-pod services with `type: NodePort` or via a simple TCP `LoadBalancer`. Connection string drops `tls=true`. Fine on a trusted home network.
    
2.  **Self-signed CA via cert-manager.** Issue a private root with cert-manager's `selfSigned` Issuer + a CA Certificate, sign your mongod cert from that, and install the `ca.crt` once on the laptops/tools that need to connect. Earlier drafts of this project used exactly that path — the cert chain is in the appendix.
    

For name resolution, point `mongo-0.lab.local` / `mongo-1.lab.local` / `mongo-2.lab.local` at your cluster IP via Pi-hole local DNS, your router's static DNS entries, or `/etc/hosts` on each client. The rest of the post applies unchanged — the SAN list in the certificate just becomes `mongo-N.lab.local` instead of public hostnames.

#### Option C — Local-only with a public domain you don't want to expose

Buy a cheap domain anyway. Use **DNS-only** records in Cloudflare (the grey cloud — no proxy), pointing at your private LAN IP. You get the LE cert via DNS-01, hostnames resolve only on your LAN (because the IP is RFC1918), and nothing about the cluster is reachable from the internet. This is what I do — "public name, private network". Best of both worlds.

* * *

### One Traefik tweak you must add

Mongo speaks raw TCP on 27017, not HTTP. Traefik needs an entrypoint and `allowCrossNamespace`. In your Traefik Helm values:

```yaml
ports:
  mongodb:
    port: 27017
    expose:
      default: true
    exposedPort: 27017
    protocol: TCP

providers:
  kubernetesCRD:
    allowCrossNamespace: true
```

![](https://cdn.hashnode.com/uploads/covers/658eba076064e1590b3a413e/5cde7e1c-9e1a-4cd8-842e-6c7fbd8a24df.png align="center")

The cross-namespace flag lets us put `IngressRouteTCP` in the `traefik` namespace while pointing at services in `databases`.

* * *

## 3\. The folder layout

Here's the entire MongoDB folder. Flat, no subfolders — important because **ArgoCD's path is non-recursive** by default, and a nested `services/` folder gets silently skipped.

```plaintext
k3s/databases/mongodb/
├── certificate.yaml         # cert-manager Certificate, LE via Cloudflare DNS-01
├── pv.yaml                  # 3 PersistentVolumes (hostPath, retained)
├── statefulset.yaml         # The 3-replica STS with postStart RS init
├── service.yaml             # ClusterIP "mongodb" for in-cluster apps
├── service-headless.yaml    # Headless "mongodb-headless" for STS pod DNS
├── service-per-pod.yaml     # 3 ClusterIPs (mongodb-0/1/2) as Traefik backends
├── ingressroute-tcp.yaml    # Traefik IngressRouteTCP, 3 HostSNI rules
├── secret.yaml              # Plain Secret (gitignored, dev only)
├── sealedsecret.yaml        # Encrypted version, the one ArgoCD applies
├── config/
│   └── mongod.conf          # Source of truth → auto-converted to ConfigMap
└── setup.sh                 # Local CLI tool
```

> **A note on layout philosophy.** Databases live at `k3s/databases/*` (sibling of `k3s/apps/*`), not under `apps/`. Stateful infrastructure has a fundamentally different lifecycle from "throwaway" apps. You restart a Jellyfin pod casually; you do not casually restart a database that's mid-write. Keeping them in a separate root makes the distinction obvious and lets us layer DB-specific tooling on top.

* * *

## 4\. Storage: PersistentVolumes that survive cluster wipes

For homelab use, `hostPath` is honest and simple. We pre-create one PV per pod, with `persistentVolumeReclaimPolicy: Retain` so a `kubectl delete pvc` doesn't nuke the bytes on disk.

`pv.yaml`

```yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: mongodb-pv-0
spec:
  capacity:
    storage: 20Gi
  accessModes: ["ReadWriteOnce"]
  persistentVolumeReclaimPolicy: Retain
  storageClassName: ""
  hostPath:
    path: /home/pi/db-data/mongodb-rs0/mongodb-0
    type: DirectoryOrCreate
  claimRef:
    namespace: databases
    name: data-mongodb-0
---
# (mongodb-pv-1 and mongodb-pv-2 are identical with /mongodb-1 and /mongodb-2 paths)
```

The trick that makes this rock-solid for a fresh-cluster bootstrap is the `claimRef` field. By spelling out exactly which PVC will claim the PV up front, we guarantee that when the StatefulSet's `volumeClaimTemplates` fires and creates `data-mongodb-0`, the binding is deterministic. No "PV stuck Available, PVC stuck Pending" dance. Coupled with `storageClassName: ""` (explicit empty string, not unset), the default storage class never tries to provision anything.

`type: DirectoryOrCreate` means the kubelet creates the host directory on first use if it's missing. Survives full cluster reinstalls — the next bootstrap re-binds to the same data on disk.

> If you're on a multi-node cluster, swap `hostPath` for **Longhorn**, **Ceph**, or your storage class of choice. The rest of the manifests don't change.

* * *

## 5\. Secrets: sealed and committed to git

We need three sensitive values:

*   `MONGO_INITDB_ROOT_USERNAME` — the bootstrap admin user.
    
*   `MONGO_INITDB_ROOT_PASSWORD` — its password.
    
*   `MONGO_REPLICA_SET_KEY_FILE` — a 1024-byte base64 keyfile used for **inter-member auth**. Without it, any pod could pretend to be a replica set member.
    

Generate the keyfile:

```bash
openssl rand -base64 756 | tr -d '\n' > keyfile
```

Build the plain Secret locally (this file is gitignored):

```bash
kubectl create secret generic mongodb-secret \
  -n databases \
  --from-literal=MONGO_INITDB_ROOT_USERNAME=admin \
  --from-literal=MONGO_INITDB_ROOT_PASSWORD='your-strong-pw!' \
  --from-file=MONGO_REPLICA_SET_KEY_FILE=keyfile \
  --dry-run=client -o yaml > secret.yaml
```

Then seal it for git:

```bash
kubeseal --controller-namespace=kube-system < secret.yaml > sealedsecret.yaml
```

`sealedsecret.yaml` looks like ciphertext to anyone reading the repo. Only the controller running in *your* cluster can decrypt it — they're encrypted with the controller's per-cluster key. Even if your repo is public, the secret is safe.

The sealed file goes in git, the plain `secret.yaml` does not. Add to `.gitignore`:

```plaintext
**/secret.yaml
```

* * *

## 6\. TLS: Let's Encrypt via Cloudflare DNS-01

> **Skipping this section?** Re-read the ["Choose your domain story"](#prerequisites) box. If you went with Option B (no public domain), use cert-manager's `selfSigned` Issuer chain instead — same overall shape, just swap the `issuerRef` in `certificate.yaml` and distribute the resulting `ca.crt` to your clients. If you went with Option A or C, read on.

This was the big "lightbulb moment" of the project. Most tutorials hand you a self-signed CA, then tell clients to "just trust this `ca.crt`". That works — until it doesn't. JDBC drivers want the cert in a JKS truststore. Compass wants a PEM. Your colleague's Mac doesn't trust it. It's a constant friction tax.

A real LE cert eliminates all of that. Drivers, GUIs, and curl just trust it because the LE root is in every modern OS.

![](https://cdn.hashnode.com/uploads/covers/658eba076064e1590b3a413e/1cfa7063-9aef-49be-b99e-f9ab64c8dd81.png align="center")

### The ClusterIssuer (one-time setup)

```yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: you@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-key
    solvers:
      - dns01:
          cloudflare:
            apiTokenSecretRef:
              name: cloudflare-api-token
              key: api-token
```

Verify it once:

```bash
$ kubectl get clusterissuer
NAME               READY   AGE
letsencrypt-prod   True    10d
```

### The MongoDB Certificate

`certificate.yaml`

```yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: mongodb-server
  namespace: databases
spec:
  secretName: mongodb-server-tls
  duration: 2160h        # 90 days (LE max)
  renewBefore: 360h      # renew 15 days before expiry
  privateKey:
    algorithm: RSA
    size: 2048
    rotationPolicy: Always
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  additionalOutputFormats:
    - type: CombinedPEM   # cert + key concatenated, what mongod expects
  commonName: mongo-0.home.your-domain.tld
  dnsNames:
    - mongo-0.home.your-domain.tld
    - mongo-1.home.your-domain.tld
    - mongo-2.home.your-domain.tld
```

A few things worth pointing out:

*   `additionalOutputFormats: [CombinedPEM]` — mongod's `certificateKeyFile` directive expects the cert and key in a single PEM file. cert-manager's default split-file output won't fly. This one-line opt-in produces `tls-combined.pem` in the secret alongside the standard `tls.crt`/`tls.key`.
    
*   **Only public hostnames in the SAN.** LE physically cannot sign internal cluster DNS like `mongodb-0.mongodb-headless.databases.svc.cluster.local`. So the cert covers external traffic only. Internal RS traffic stays plain TCP (more on that in the `mongod.conf` section).
    
*   **90-day duration, 15-day renewal window.** cert-manager will silently re-issue and roll the secret in your cluster forever.
    

Apply, then watch:

```bash
$ kubectl -n databases get certificate
NAME             READY   SECRET               AGE
mongodb-server   True    mongodb-server-tls   2m
```

When `READY=True` you have an LE-signed cert sitting in a Kubernetes Secret. cert-manager handles renewal automatically.

* * *

## 7\. Configuration: a real `mongod.conf` you can read

Forget cramming a multi-line YAML string into a ConfigMap. Keep `mongod.conf` as an actual file in `config/` and let your tooling auto-build the ConfigMap. (We'll see how in the scripts section.)

`config/mongod.conf`

```yaml
storage:
  dbPath: /data/db
  wiredTiger:
    engineConfig:
      cacheSizeGB: 0.5     # Pi-friendly. Bump on bigger nodes.

net:
  bindIp: 0.0.0.0
  port: 27017
  tls:
    # allowTLS = mongod accepts both plain & TLS, but does NOT initiate TLS
    # on outbound connections (e.g. replica-set member-to-member).
    # That keeps internal RS traffic plain — necessary because the LE cert
    # only covers the public hostnames, not the cluster DNS names that
    # members use to dial each other.
    mode: allowTLS
    certificateKeyFile: /etc/mongo/tls/tls-combined.pem
    # mongod 7 requires CAFile even when not validating clients. Since we
    # set allowConnectionsWithoutCertificates: true, no client cert is
    # actually checked against this CA — we just need any valid bundle.
    # Use the image's OS CA bundle (Debian path, present in mongo:7).
    CAFile: /etc/ssl/certs/ca-certificates.crt
    allowConnectionsWithoutCertificates: true

replication:
  replSetName: rs0

security:
  keyFile: /etc/mongo/keyfile
```

### Why `allowTLS` and not `preferTLS` or `requireTLS`?

This is the single most counter-intuitive line in the whole setup. The official docs nudge you toward `requireTLS` for production. Don't do it here. Here's the tradeoff matrix:

| Mode | Inbound plain | Inbound TLS | Outbound (RS heartbeats) |
| --- | --- | --- | --- |
| `disabled` | ✅ accept | ❌ reject | plain |
| `allowTLS` | ✅ accept | ✅ accept | **plain** |
| `preferTLS` | ✅ accept | ✅ accept | **TLS** |
| `requireTLS` | ❌ reject | ✅ accept | **TLS** |

For RS heartbeats, members dial each other using the names from `rs.conf()` — internal cluster DNS like `mongodb-0.mongodb-headless.databases.svc.cluster.local`. Those names **are not in the LE cert's SAN** (LE won't sign them). With `preferTLS` or `requireTLS`, mongod tries TLS for outbound, the cert fails hostname validation, heartbeats fail, the RS goes unhealthy, and you end up debugging at 2am.

`allowTLS` says "I'll *accept* TLS from clients, but I'll only *initiate* plain". Internal RS traffic stays plain (it's already inside the cluster network, so the security loss is small), external clients get full TLS through Traefik passthrough.

### Why `CAFile` is set even though we don't validate clients

mongod 7 refuses to start with TLS enabled unless `CAFile` is configured (see [SERVER-72839](https://jira.mongodb.org/browse/SERVER-72839)). Combined with `allowConnectionsWithoutCertificates: true`, no client cert is ever actually validated against this file — but the file must exist. We point at the image's built-in OS bundle. Cosmetic compliance, no security impact.

* * *

## 8\. The StatefulSet — heart of the cluster

This is the biggest manifest, so we'll walk through it in chunks.

`statefulset.yaml`

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mongodb
  namespace: databases
spec:
  serviceName: mongodb-headless     # Must match the headless Service name
  replicas: 3
  podManagementPolicy: Parallel     # Critical — see "why" below
  selector:
    matchLabels:
      app: mongodb
```

`serviceName: mongodb-headless` is what gives each pod its DNS name `mongodb-N.mongodb-headless.databases.svc.cluster.local`. `podManagementPolicy: Parallel` is the workaround for the chicken-and-egg problem we discussed.

### The init container

```yaml
initContainers:
  - name: keyfile-init
    image: busybox
    command:
      - sh
      - -c
      - |
        cp /tmp/keyfile /keyfile/keyfile && \
        chmod 400 /keyfile/keyfile && \
        cp /tls-src/tls-combined.pem /tls/tls-combined.pem && \
        chmod 400 /tls/tls-combined.pem && \
        chown 999:999 /keyfile/keyfile /tls/tls-combined.pem && \
        chown -R 999:999 /data/db
    volumeMounts:
      - name: keyfile-src
        mountPath: /tmp
      - name: keyfile
        mountPath: /keyfile
      - name: tls-src
        mountPath: /tls-src
      - name: tls
        mountPath: /tls
      - name: data
        mountPath: /data/db
```

**Why an init container?** Two reasons:

1.  `mongod` **runs as UID 999 (the** `mongo` **user)** but Kubernetes mounts secrets with permissions that mongod refuses to read for the keyfile (it requires `chmod 400`). Secret projections don't let you set `defaultMode` strictly enough on a per-key basis. The init container side-steps the entire problem by copying secrets out of the read-only secret mount into an `emptyDir`, applying the right permissions, and chowning to mongo.
    
2.  The init container also fixes ownership of the data directory on first boot.
    

### The mongod container

```yaml
containers:
  - name: mongodb
    image: mongo:7
    command:
      - mongod
      - --config
      - /etc/mongo/mongod.conf
    ports:
      - containerPort: 27017
```

Nothing exotic. We override the entrypoint to use our config file directly.

### The replica-set initiator

This is the magic that makes the whole thing self-bootstrapping:

```yaml
lifecycle:
  postStart:
    exec:
      command:
        - bash
        - -c
        - |
          # Only pod-0 bootstraps the replica set
          [[ "${HOSTNAME}" != "mongodb-0" ]] && exit 0
          # Wait up to 60s for mongod to accept connections
          for i in $(seq 1 30); do
            mongosh --quiet --eval "db.adminCommand('ping').ok" 2>/dev/null | grep -q '^1$' && break
            sleep 2
          done
          # Initiate replica set — error code 23 means already initialized, which is fine
          mongosh admin \
            -u "${MONGO_INITDB_ROOT_USERNAME}" \
            -p "${MONGO_INITDB_ROOT_PASSWORD}" \
            --quiet \
            --eval "
              var r = rs.initiate({
                _id: 'rs0',
                members: [
                  { _id: 0, host: 'mongodb-0.mongodb-headless.databases.svc.cluster.local:27017' },
                  { _id: 1, host: 'mongodb-1.mongodb-headless.databases.svc.cluster.local:27017' },
                  { _id: 2, host: 'mongodb-2.mongodb-headless.databases.svc.cluster.local:27017' }
                ]
              });
              if (r.ok !== 1 && r.code !== 23) quit(1);
            " 2>/dev/null || true
```

It runs on every pod start, but the `[[ "${HOSTNAME}" != "mongodb-0" ]] && exit 0` short-circuits everyone except pod-0. The `code !== 23` check makes it idempotent — `rs.initiate` on an already-initialized cluster returns 23, which we treat as success.

You'll notice we're **not** advertising the external horizons here. We add those after the cluster is up:

```bash
kubectl -n databases exec -it mongodb-0 -- mongosh -u admin -p
> cfg = rs.conf()
> cfg.members[0].horizons = { external: "mongo-0.home.your-domain.tld:27017" }
> cfg.members[1].horizons = { external: "mongo-1.home.your-domain.tld:27017" }
> cfg.members[2].horizons = { external: "mongo-2.home.your-domain.tld:27017" }
> rs.reconfig(cfg)
```

**What horizons do:** when a driver connects via `mongo-0.home.your-domain.tld`, mongod replies to `db.hello()` with the *external* names of all members — names the external client can actually resolve. Without horizons, the client would be told "the other members are at `mongodb-1.mongodb-headless.databases.svc.cluster.local`" and immediately fail to connect to them.

### The volume choreography

```yaml
volumeMounts:
  - { name: data,    mountPath: /data/db }
  - { name: config,  mountPath: /etc/mongo/mongod.conf, subPath: mongod.conf }
  - { name: keyfile, mountPath: /etc/mongo/keyfile,     subPath: keyfile }
  - { name: tls,     mountPath: /etc/mongo/tls,         readOnly: true }

volumes:
  - name: config
    configMap: { name: mongodb-config }

  - name: keyfile-src
    secret:
      secretName: mongodb-secret
      items:
        - key: MONGO_REPLICA_SET_KEY_FILE
          path: keyfile

  - name: keyfile
    emptyDir: {}

  - name: tls-src
    secret:
      secretName: mongodb-server-tls

  - name: tls
    emptyDir: {}

volumeClaimTemplates:
  - metadata:
      name: data
      labels: { app: mongodb }
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: ""
      resources:
        requests: { storage: 20Gi }
```

Both the keyfile and the TLS material follow the same pattern: a `*-src` secret-mount the init container reads from, an `emptyDir` it copies *into* with the right permissions, and the main container mounts the `emptyDir`. Clean and works around every Kubernetes secret-permissions footgun.

* * *

## 9\. Three services, three jobs

This is the question that confused me longest when I started: *why do I need three services for one StatefulSet?* The honest answer: each one solves a different connectivity problem.

![](https://cdn.hashnode.com/uploads/covers/658eba076064e1590b3a413e/b5b79a23-772b-428b-b5e0-36f804cd08f1.png align="center")

### 9.1 The headless service — `service-headless.yaml`

```yaml
apiVersion: v1
kind: Service
metadata:
  name: mongodb-headless
  namespace: databases
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  selector:
    app: mongodb
  ports:
    - port: 27017
      name: mongodb
```

`clusterIP: None` makes this **headless** — there is no virtual IP. Its sole job is to give each pod a DNS A record `mongodb-N.mongodb-headless.databases.svc.cluster.local`. Without this, the StatefulSet pods have no stable DNS, and the replica set falls apart.

`publishNotReadyAddresses: true` is essential during bootstrap — it lets pods discover each other *before* they're marked Ready, breaking the chicken-and-egg.

### 9.2 The regular ClusterIP — `service.yaml`

```yaml
apiVersion: v1
kind: Service
metadata:
  name: mongodb
  namespace: databases
spec:
  selector:
    app: mongodb
  ports:
    - port: 27017
      targetPort: 27017
```

This is what *internal apps* (n8n, your custom services, whatever) connect to. They use the URI `mongodb://...@mongodb.databases.svc.cluster.local:27017/?replicaSet=rs0`. The driver connects to whatever pod the Service forwards to, asks `db.hello()`, learns the topology, and starts talking to the primary directly.

### 9.3 Per-pod ClusterIPs — `service-per-pod.yaml`

```yaml
apiVersion: v1
kind: Service
metadata:
  name: mongodb-0
  namespace: databases
spec:
  selector:
    statefulset.kubernetes.io/pod-name: mongodb-0
  ports:
    - port: 27017
      targetPort: 27017
---
# (mongodb-1 and mongodb-2 are identical with the matching pod-name)
```

This is the trick that makes external SNI routing work. Each Service selects **exactly one pod** via the `statefulset.kubernetes.io/pod-name` label that Kubernetes auto-applies. Now Traefik can map "incoming connection with SNI = `mongo-0.home.your-domain.tld`" to "the Service that fronts only `mongodb-0`", giving us per-replica external addressability.

* * *

## 10\. Traefik IngressRouteTCP with SNI passthrough

![](https://cdn.hashnode.com/uploads/covers/658eba076064e1590b3a413e/1aac0512-9501-4808-ab40-41f67270100a.png align="center")

MongoDB wire protocol is binary TCP — Traefik's HTTP `Ingress` can't see it. We use `IngressRouteTCP` (Traefik's L4 CRD) with TLS passthrough:

`ingressroute-tcp.yaml` (lives in the `traefik` namespace)

```yaml
apiVersion: traefik.io/v1alpha1
kind: IngressRouteTCP
metadata:
  name: mongodb-0
  namespace: traefik
spec:
  entryPoints:
    - mongodb       # The 27017 entrypoint we added to Traefik
  routes:
    - match: HostSNI(`mongo-0.home.your-domain.tld`)
      services:
        - name: mongodb-0       # Per-pod Service from the previous section
          namespace: databases
          port: 27017
  tls:
    passthrough: true            # Traefik does NOT terminate TLS
---
# (mongodb-1 and mongodb-2 are identical, with their own SNI hostname and Service)
```

### What "passthrough" actually means

Without passthrough, Traefik would terminate TLS itself and re-encrypt to the backend (or send plain). With `passthrough: true`, Traefik treats the connection as opaque bytes — it only peeks at the unencrypted **SNI field of the TLS ClientHello** to decide where to route, then forwards the encrypted stream untouched. mongod sees the TLS handshake directly, presents its cert, terminates the session.

This is perfect for our use case: Traefik doesn't need a copy of the cert (mongod has it), the TLS path is end-to-end client→mongod, and we can route different SNI hostnames to different backend pods.

### Traffic flow

1.  DataGrip on your laptop connects to `mongo-0.home.your-domain.tld:27017`.
    
2.  DNS resolves to your cluster's IP (192.168.0.108 in my case).
    
3.  Traefik on 27017 reads the SNI from the ClientHello — sees `mongo-0.home.your-domain.tld`.
    
4.  Matches the first IngressRouteTCP rule, forwards bytes to the `mongodb-0` Service in `databases`.
    
5.  The Service forwards to the `mongodb-0` pod.
    
6.  mongod completes TLS handshake using the LE cert.
    
7.  Driver issues `db.hello()`. mongod replies with the **horizons** view: members are at `mongo-0/1/2.home.your-domain.tld`.
    
8.  Driver opens TLS connections to the other two members the same way.
    

* * *

## 11\. The shell helpers — a layered, reusable management library

You *could* manage everything with raw `kubectl`. After the third time typing `kubectl -n databases logs mongodb-0 -f --tail=200`, you won't want to.

I built a two-layer management library:

```plaintext
k3s/scripts/
├── _app-ctl.sh   ← Generic library — every app gets these commands free
└── _db-ctl.sh    ← DB-specific layer; sources _app-ctl.sh; adds DB commands
```

Each app's `setup.sh` is a tiny shim that sources the appropriate library and provides 3-4 hook functions for app-specific behavior.

### Generic commands (from `_app-ctl.sh`)

Every app gets these for free:

```plaintext
deploy        Apply all manifests in dependency order
delete        Inverse of deploy
redeploy      delete + deploy
status        Pods, PVCs, services, recent events, resource usage
logs          Tail logs (last 200 lines, then follow)
restart       Roll the workload
shell         Exec a shell into the workload pod
port-forward  kubectl port-forward to localhost
seal          Resign the secret with kubeseal
argocd-sync   Trigger an ArgoCD sync
help          Show available commands
```

The deploy logic applies a known-order list (`pv → pvc → certificate → configmap → secret → rbac → workload → service → ingress`), then **globs every other root-level YAML file** to catch helpers like `service-headless.yaml`, `ingressroute-tcp.yaml`, etc. without you having to update the script.

### DB-specific commands (from `_db-ctl.sh`)

```bash
connection-string [external|internal]
print-ca
detach-storage
attach-storage [n]
dump [file]
restore <file>
```

`_db-ctl.sh` in full (about 130 lines):

```bash
#!/usr/bin/env bash
# _db-ctl.sh — Shared database management library

DB_CTL_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$DB_CTL_DIR/_app-ctl.sh"

cmd_connection_string() {
  local scope="${1:-external}"
  if ! declare -F _db_connection_string >/dev/null; then
    err "This DB's setup.sh has not defined _db_connection_string()"
    exit 1
  fi
  _db_connection_string "$scope"
}

cmd_dump() {
  local out="${1:-${APP}-$(date +%Y%m%d-%H%M%S).dump}"
  if ! declare -F _db_dump_cmd >/dev/null; then
    err "This DB's setup.sh has not defined _db_dump_cmd()"
    exit 1
  fi
  _db_dump_cmd "$out"
  ok "Dump complete: $out"
}

cmd_restore() {
  local in="${1:-}"
  [[ -z "$in" || ! -f "$in" ]] && { err "Usage: ./setup.sh restore <dump-file>"; exit 1; }
  if ! declare -F _db_restore_cmd >/dev/null; then
    err "This DB's setup.sh has not defined _db_restore_cmd()"
    exit 1
  fi
  warn "This may overwrite existing data."
  read -r -p "  Type 'yes' to confirm: " confirm
  [[ "$confirm" != "yes" ]] && echo "  Aborted." && return 0
  _db_restore_cmd "$in"
  ok "Restore complete"
}

cmd_detach_storage() {
  local kind; kind="$(_detect_workload_kind)"
  kubectl scale "$kind/$APP" -n "$NAMESPACE" --replicas=0
  kubectl rollout status "$kind/$APP" -n "$NAMESPACE" --timeout=120s || true
  ok "Detached. Reattach with: ./setup.sh attach-storage [replicas]"
}

cmd_attach_storage() {
  local replicas="${1:-1}"
  local kind; kind="$(_detect_workload_kind)"
  kubectl scale "$kind/$APP" -n "$NAMESPACE" --replicas="$replicas"
  kubectl rollout status "$kind/$APP" -n "$NAMESPACE" --timeout=300s
  ok "Reattached at ${replicas} replica(s)"
}

db_main() {
  case "${1:-help}" in
    connection-string) shift; cmd_connection_string "${1:-external}" ;;
    print-ca)          cmd_print_ca ;;
    detach-storage)    cmd_detach_storage ;;
    attach-storage)    shift; cmd_attach_storage "${1:-1}" ;;
    dump)              shift; cmd_dump "${1:-}" ;;
    restore)           shift; cmd_restore "${1:-}" ;;
    *)                 main "$@" ;;        # Fall through to _app-ctl.sh
  esac
}
```

### MongoDB's `setup.sh` — only ~80 lines

```bash
#!/usr/bin/env bash
set -euo pipefail

APP="mongodb"
NAMESPACE="databases"
CONTAINER_PORT="27017"
HAS_PVC=true
HAS_SECRET=true
HAS_CONFIGMAP=true

DEPLOY_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

# Walk up to find k3s/scripts/_db-ctl.sh
_find_scripts() {
  local d="$1"
  while [[ "$d" != "/" ]]; do
    [[ -d "$d/scripts" && -f "$d/scripts/_db-ctl.sh" ]] && echo "$d/scripts" && return
    d="$(dirname "$d")"
  done
}
SCRIPTS_DIR="$(_find_scripts "$DEPLOY_DIR")"
source "$SCRIPTS_DIR/_db-ctl.sh"

# ─── MongoDB-specific hooks ──────────────────────────────────────────────────

_mongo_password() {
  kubectl get secret mongodb-secret -n "$NAMESPACE" \
    -o jsonpath='{.data.MONGO_INITDB_ROOT_PASSWORD}' | base64 -d | \
    sed 's/!/%21/g; s/@/%40/g; s/:/%3A/g; s|/|%2F|g'
}

_db_connection_string() {
  local scope="${1:-external}"
  local user pass
  user="$(kubectl get secret mongodb-secret -n "$NAMESPACE" -o jsonpath='{.data.MONGO_INITDB_ROOT_USERNAME}' | base64 -d)"
  pass="$(_mongo_password)"

  case "$scope" in
    external)
      echo "mongodb://${user}:${pass}@mongo-0.home.your-domain.tld,mongo-1.home.your-domain.tld,mongo-2.home.your-domain.tld:27017/?replicaSet=rs0&tls=true&authSource=admin"
      info "TLS uses Let's Encrypt — no CA file needed."
      ;;
    internal)
      echo "mongodb://${user}:${pass}@mongodb.databases.svc.cluster.local:27017/?replicaSet=rs0&authSource=admin"
      ;;
  esac
}

_db_dump_cmd() {
  local out="$1"
  local pod; pod="$(_get_pod)"
  kubectl exec -n "$NAMESPACE" "$pod" -- bash -c \
    'mongodump --quiet --uri="mongodb://${MONGO_INITDB_ROOT_USERNAME}:${MONGO_INITDB_ROOT_PASSWORD}@localhost:27017/?authSource=admin" --archive --gzip' > "$out"
}

_db_restore_cmd() {
  local in="$1"
  local pod; pod="$(_get_pod)"
  kubectl exec -i -n "$NAMESPACE" "$pod" -- bash -c \
    'mongorestore --quiet --uri="mongodb://${MONGO_INITDB_ROOT_USERNAME}:${MONGO_INITDB_ROOT_PASSWORD}@localhost:27017/?authSource=admin" --archive --gzip --drop' < "$in"
}

db_main "$@"
```

That's it. To add Postgres or Redis later, you copy this file, change `APP`, and rewrite the three hook functions. Generic commands work unchanged.

* * *

## 12\. Wiring it into ArgoCD

![](https://cdn.hashnode.com/uploads/covers/658eba076064e1590b3a413e/845e13a2-3c39-4ef9-a59d-1f4b5079b322.png align="center")

A single ApplicationSet generates one ArgoCD Application per directory entry:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: homelab
  namespace: argocd
spec:
  goTemplate: true
  generators:
    - list:
        elements:
          - { name: mongodb,  namespace: databases, path: k3s/databases/mongodb }
          - { name: postgres, namespace: databases, path: k3s/databases/postgres }
          - { name: redis,    namespace: databases, path: k3s/databases/redis }
          # ... your other apps ...
  template:
    metadata:
      name: '{{ .name }}'
      namespace: argocd
    spec:
      project: default
      source:
        repoURL: https://github.com/yourname/Home-Server-Lab
        targetRevision: main
        path: '{{ .path }}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{ .namespace }}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
```

> **The non-recursive trap.** ArgoCD's directory source is **not recursive by default**. If you nest a `services/` subfolder, ArgoCD silently skips it and you spend 45 minutes wondering why your IngressRouteTCP isn't applying. Either flatten (what we did) or add `directory: { recurse: true }` to the source.

Once committed, ArgoCD polls git, sees the new manifests, applies them, and reconciles drift forever after. Your `setup.sh` and ArgoCD are two paths to the same end state — `setup.sh` is for the moments you want to redeploy without waiting.

* * *

## 13\. Connection strings — the payoff

After all that work, here's what you actually paste. **Replace** `home.your-domain.tld` **with whatever you used in the certificate's SAN list.** If you went with Option B (no public domain) substitute your `.local` names; if you went with Option C, substitute your DNS-only names.

![](https://cdn.hashnode.com/uploads/covers/658eba076064e1590b3a413e/7d706c3d-0d79-41ef-ab2e-5bc93e406535.png align="center")

**External (TLS via Let's Encrypt — for DataGrip, Compass, your laptop, external apps):**

```plaintext
mongodb://admin:YOURPASS@mongo-0.home.your-domain.tld,mongo-1.home.your-domain.tld,mongo-2.home.your-domain.tld:27017/?replicaSet=rs0&tls=true&authSource=admin
```

No CA file. No truststore configuration. No "trust this self-signed cert" dialog. Just paste and connect. URL-encode any special characters in the password (`!` → `%21`, `@` → `%40`, etc.).

> **Without TLS (Option B, no domain):** drop `tls=true` and replace the comma-separated hostnames with your local DNS names, e.g. `mongo-0.lab.local`. The driver still discovers the replica set the same way.

**Internal (in-cluster apps, plain TCP):**

```plaintext
mongodb://admin:YOURPASS@mongodb.databases.svc.cluster.local:27017/?replicaSet=rs0&authSource=admin
```

You can quickly verify both:

```bash
# From your laptop (using local mongosh or docker)
docker run --rm mongo:7 mongosh "mongodb://admin:...@mongo-0...&tls=true&authSource=admin" \
  --quiet --eval 'db.hello().me'
# → mongo-0.home.your-domain.tld:27017

# From inside the cluster
kubectl -n databases run test --rm -i --restart=Never --image=mongo:7 -- \
  mongosh "mongodb://admin:...@mongodb.databases.svc.cluster.local:27017/?replicaSet=rs0&authSource=admin" \
  --quiet --eval 'db.hello().primary'
```

If both return a hostname instead of an error, you've made it.

* * *

## 14\. Operating it

### Daily commands

```bash
cd k3s/databases/mongodb

./setup.sh status                   # pods, RS health, PVCs, events, resource usage
./setup.sh logs                     # tail mongodb-0
./setup.sh shell                    # bash inside mongodb-0
./setup.sh restart                  # rolling restart
./setup.sh redeploy                 # full delete + apply
./setup.sh connection-string external
./setup.sh dump backup-$(date +%F).gz
./setup.sh restore backup.gz
./setup.sh detach-storage           # scale to 0 for fsck/PV work
./setup.sh attach-storage 3
```

### Syncing from a production cluster

The biggest "but does it actually work" question for a homelab DB is whether you can pull production data down. Yes — two patterns.

**Pattern A — dump locally, restore via setup.sh:**

```bash
# Dump prod into a local archive
docker run --rm -v "$PWD":/d mongo:7 \
  mongodump --uri="$PROD_URI" --archive=/d/prod.gz --gzip

cd k3s/databases/mongodb
./setup.sh restore /full/path/prod.gz
```

**Pattern B — stream prod → k3s without touching disk:**

```bash
docker run --rm mongo:7 mongodump --uri="$PROD_URI" --archive --gzip \
  | kubectl -n databases exec -i mongodb-0 -- mongorestore \
      --uri="mongodb://admin:${PASS_ENC}@localhost:27017/?authSource=admin" \
      --archive --gzip --drop
```

Mongo replicates the writes from the primary to the secondaries automatically. You'll briefly see oplog catch-up activity in the logs, then the cluster settles.

For per-project user isolation (e.g. `kai_user` with `readWrite@kai_db`), create users via mongosh after restore:

```javascript
use admin
db.createUser({
  user: "kai",
  pwd: "...",
  roles: [{ role: "readWrite", db: "kai" }]
})
```

### Backups

For real backups, schedule a CronJob:

```yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: mongodb-backup
  namespace: databases
spec:
  schedule: "0 3 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: dump
              image: mongo:7
              command:
                - sh
                - -c
                - |
                  mongodump \
                    --uri="mongodb://admin:$PASS@mongodb:27017/?authSource=admin" \
                    --archive=/backups/mongo-$(date +%Y%m%d).gz --gzip
              env:
                - name: PASS
                  valueFrom:
                    secretKeyRef:
                      name: mongodb-secret
                      key: MONGO_INITDB_ROOT_PASSWORD
              volumeMounts:
                - name: backups
                  mountPath: /backups
          volumes:
            - name: backups
              persistentVolumeClaim:
                claimName: mongodb-backups
```

Pair with rclone to push to S3, B2, or your NAS.

* * *

## 15\. What's missing — and what you'd do next

This setup is "homelab-grade production". For a real production rollout you'd extend it with:

### Sharding

A replica set is enough for tens of GB. Beyond that — or beyond a single machine's RAM — you'd shard. The skeleton:

*   Three **config server replica sets** (`csrs`) — config metadata.
    
*   N **shard replica sets** — each a 3-member RS.
    
*   Two or more **mongos routers** — the query proxies your apps connect to.
    

In Kubernetes terms: each becomes its own StatefulSet with its own headless service. The current manifests are the template; you'd parametrize `replSetName`, `dbPath`, and the keyfile is shared across all of them. The Bitnami chart automates this if you don't want the YAML maintenance.

### Monitoring

Add a `mongodb-exporter` sidecar to the StatefulSet, scrape with Prometheus, dashboard with Grafana. The MongoDB Community Operator's exporter manifests are a fine starting point.

### Resource limits

The current manifest doesn't set requests/limits — fine for a Pi where you're the only tenant. On a multi-tenant cluster, pin them. WiredTiger cache + ~1GB headroom is a safe baseline.

### True multi-zone HA

Add `topologySpreadConstraints` so the three pods land on three different nodes/zones. On a Pi homelab with one node, this is moot. On a multi-node cluster, it's essential.

### PodDisruptionBudget

A two-of-three minimum so cluster maintenance can drain at most one mongod at a time:

```yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: mongodb
  namespace: databases
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: mongodb
```

### Per-project users + RBAC

The current setup has one root admin. For multi-tenant use, create role-scoped users per app, and never let apps use the root creds.

### Velero / volume snapshots

Logical dumps are great for portability; volume-level snapshots are great for fast recovery. Velero with CSI snapshots gets you both.

* * *

## 16\. Closing thoughts

![](https://cdn.hashnode.com/uploads/covers/658eba076064e1590b3a413e/2db78eaf-e6f4-44f9-ad5f-ff9da2dfe8d5.png align="center")

The reason I wrote this is simple: I lost a weekend trying to find the seam between "spin up a Helm chart" tutorials and "operating MongoDB in production" books. Nobody explains the *why* of the small choices — the SNI passthrough, the `allowTLS` mode, the per-pod services, the horizons, the chicken-and-egg of `Parallel` pod management. So you end up with a Bitnami chart you don't understand or a hand-rolled mess that breaks the first time you cycle a pod.

This setup runs on a single Raspberry Pi 5 in my apartment. The same manifests run unchanged on a 6-node Talos cluster I tested it on. It restarts cleanly, survives full cluster reinstalls (PVs are retained, the keyfile and admin creds are sealed in git), and presents a real LE cert that DataGrip just trusts. The shell helpers turn ten-finger kubectl gymnastics into `./setup.sh status`. ArgoCD makes the source of truth your git repo.

If you're walking the same path — running real databases in your homelab, learning Kubernetes by doing — I hope this saves you the weekend it cost me.

The **full repository** with every file shown here, plus the Postgres/Redis/MySQL stubs and the rest of the homelab apps, lives at: [https://github.com/yourname/Home-Server-Lab](https://github.com/yourname/Home-Server-Lab).

Hit me up in the comments if you get stuck. And if you do extend this to sharding or write the monitoring sidecar version — please write the next post in this series. The next homelabber will thank you.

Happy hacking.

* * *

### Appendix A — `.gitignore` essentials

```plaintext
# Never commit plaintext secrets
**/secret.yaml

# Local helper output
**/*.dump
**/*.gz

# Cluster state
.kubeconfig
```

### Appendix B — Quick troubleshooting

| Symptom | Likely cause | Fix |
| --- | --- | --- |
| Pods stuck Pending forever | PV `claimRef` doesn't match PVC name | PVC names are `data-mongodb-{0,1,2}`. Match exactly. |
| `rs.initiate` exit code 23 | Already initialized | Treat as success (the script already does). |
| External clients see internal cluster DNS | `replicaSetHorizons` not configured | Run `rs.reconfig()` after first boot. |
| TLS handshake fails externally | SNI mismatch / Traefik missing TCP entrypoint | Check `Helm` values for `ports.mongodb`. |
| "InvalidOptions: TLS without specifying chain of trust" | mongod 7 wants `CAFile` even for server-only TLS | Point `CAFile` at `/etc/ssl/certs/ca-certificates.crt`. |
| Heartbeat: "SSL peer certificate validation failed" | `preferTLS`/`requireTLS` + LE cert (no internal SAN) | Use `mode: allowTLS` instead. |
| ArgoCD applies most files but not the ones in `services/` | Non-recursive directory source | Flatten the folder or set `directory.recurse: true`. |