initdb: error: directory "/var/snap/charmed-postgresql/common/var/lib/postgresql" exists but is not empty

[alanbach]

Steps to reproduce

1. Deploy a Charmed PostgreSQL 16 cluster that uses a second disk as the data storage. A sample bundle has been provided below, please customize this as per your environment configuration:

    default-base: ubuntu@24.04/stable
    applications:
      postgresql:
        charm: postgresql
        channel: 16/stable
        revision: 952
        num_units: 3
        to:
        - "0"
        - "1"
        - "2"
        options:
          profile: production
        storage:
          data: nvme,1,150G
    machines:
      "0":
        constraints: arch=amd64 instance-type=c1.large zones=az1 root-disk-source=volume root-disk=50000
      "1":
        constraints: arch=amd64 instance-type=c1.large zones=az1 root-disk-source=volume root-disk=50000
      "2":
        constraints: arch=amd64 instance-type=c1.large zones=az1 root-disk-source=volume root-disk=50000

Expected behavior

A three node cluster is expected to form and come online.

Actual behavior

Cluster never properly forms because nodes fail to initialize PostgreSQL due to a lost+found directory being present in the data storage disk (default ext4 behavior):

initdb: error: directory "/var/snap/charmed-postgresql/common/var/lib/postgresql" exists but is not empty
initdb: detail: It contains a lost+found directory, perhaps due to it being a mount point.
initdb: hint: Using a mount point directly as the data directory is not recommended.
Create a subdirectory under the mount point.

The files belonging to this database system will be owned by user "_daemon_".
This user must also own the server process.
The database cluster will be initialized with locale "C.UTF-8".
The default text search configuration will be set to "english".
Data page checksums are enabled.
initdb: error: directory "/var/snap/charmed-postgresql/common/var/lib/postgresql" exists but is not empty
initdb: detail: It contains a lost+found directory, perhaps due to it being a mount point.
initdb: hint: Using a mount point directly as the data directory is not recommended.
Create a subdirectory under the mount point.

Versions

Operating system: Ubuntu 24.04 LTS (Noble)

Juju CLI: 3.6.12

Juju agent: 3.6.12

Charm revision: 952 on 16/stable channel

LXD: Not Applicable

Workaround

Use a bash script tied to a systemd timer to monitor the data disk and delete lost+found to allow the cluster to form properly:

# Script to monitor the mountpoint
#!/usr/bin/env bash
set -euo pipefail
 
TARGET="/var/snap/charmed-postgresql/common/var/lib/postgresql"
SERVICE_NAME="pg-lostfound-watcher.service"
TIMER_NAME="pg-lostfound-watcher.timer"
 
echo "[pg-lostfound-watcher] Started, watching for mount: $TARGET"
 
while true; do
    if [ -d "$TARGET" ]; then
        if awk -v p="$TARGET" '$2 == p {found=1} END {exit !found}' /proc/mounts; then
            echo "[pg-lostfound-watcher] Detected mount at $TARGET"
            break
        fi
    fi
    sleep 5
done
 
LF="$TARGET/lost+found"
if [ -d "$LF" ]; then
    echo "[pg-lostfound-watcher] Removing $LF"
    rm -rf -- "$LF"
else
    echo "[pg-lostfound-watcher] No lost+found found at $LF (nothing to do)"
fi
 
echo "[pg-lostfound-watcher] Disabling $SERVICE_NAME and $TIMER_NAME"
systemctl disable --now "$TIMER_NAME" "$SERVICE_NAME" || true
 
echo "[pg-lostfound-watcher] Done, exiting."
exit 0
 
# Systemd Service
[Unit]
Description=Watch for charmed-postgresql data mount and remove lost+found once
After=local-fs.target
Wants=local-fs.target
 
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/pg-lostfound-watcher.sh
User=root
Group=root
 
# Systemd Timer
[Unit]
Description=Start pg-lostfound-watcher at boot
 
[Timer]
OnBootSec=10sec
Persistent=true
 
[Install]
WantedBy=timers.target

Incorporate the above elements in the juju model cloud init userdata:

cloudinit-userdata: |
  write_files:
    - path: /usr/local/sbin/pg-lostfound-watcher.sh
      permissions: '0755'
      encoding: b64
      content: |
        IyEvdXNyL2Jpbi9lbnYgYmFzaApzZXQgLWV1byBwaXBlZmFpbAoKVEFSR0VUPSIvdmFyL3NuYXAvY2hhcm1lZC1wb3N0Z3Jlc3FsL2NvbW1vbi92YXIvbGliL3Bvc3RncmVzcWwiClNFUlZJQ0VfTkFNRT0icGctbG9zdGZvdW5kLXdhdGNoZXIuc2VydmljZSIKVElNRVJfTkFNRT0icGctbG9zdGZvdW5kLXdhdGNoZXIudGltZXIiCgplY2hvICJbcGctbG9zdGZvdW5kLXdhdGNoZXJdIFN0YXJ0ZWQsIHdhdGNoaW5nIGZvciBtb3VudDogJFRBUkdFVCIKCndoaWxlIHRydWU7IGRvCiAgICBpZiBbIC1kICIkVEFSR0VUIiBdOyB0aGVuCiAgICAgICAgaWYgYXdrIC12IHA9IiRUQVJHRVQiICckMiA9PSBwIHtmb3VuZD0xfSBFTkQge2V4aXQgIWZvdW5kfScgL3Byb2MvbW91bnRzOyB0aGVuCiAgICAgICAgICAgIGVjaG8gIltwZy1sb3N0Zm91bmQtd2F0Y2hlcl0gRGV0ZWN0ZWQgbW91bnQgYXQgJFRBUkdFVCIKICAgICAgICAgICAgYnJlYWsKICAgICAgICBmaQogICAgZmkKICAgIHNsZWVwIDUKZG9uZQoKTEY9IiRUQVJHRVQvbG9zdCtmb3VuZCIKaWYgWyAtZCAiJExGIiBdOyB0aGVuCiAgICBlY2hvICJbcGctbG9zdGZvdW5kLXdhdGNoZXJdIFJlbW92aW5nICRMRiIKICAgIHJtIC1yZiAtLSAiJExGIgplbHNlCiAgICBlY2hvICJbcGctbG9zdGZvdW5kLXdhdGNoZXJdIE5vIGxvc3QrZm91bmQgZm91bmQgYXQgJExGIChub3RoaW5nIHRvIGRvKSIKZmkKCmVjaG8gIltwZy1sb3N0Zm91bmQtd2F0Y2hlcl0gRGlzYWJsaW5nICRTRVJWSUNFX05BTUUgYW5kICRUSU1FUl9OQU1FIgpzeXN0ZW1jdGwgZGlzYWJsZSAtLW5vdyAiJFRJTUVSX05BTUUiICIkU0VSVklDRV9OQU1FIiB8fCB0cnVlCgplY2hvICJbcGctbG9zdGZvdW5kLXdhdGNoZXJdIERvbmUsIGV4aXRpbmcuIgpleGl0IDAK
 
    - path: /etc/systemd/system/pg-lostfound-watcher.service
      permissions: '0644'
      encoding: b64
      content: |
        W1VuaXRdCkRlc2NyaXB0aW9uPVdhdGNoIGZvciBjaGFybWVkLXBvc3RncmVzcWwgZGF0YSBtb3VudCBhbmQgcmVtb3ZlIGxvc3QrZm91bmQgb25jZQpBZnRlcj1sb2NhbC1mcy50YXJnZXQKV2FudHM9bG9jYWwtZnMudGFyZ2V0CgpbU2VydmljZV0KVHlwZT1vbmVzaG90CkV4ZWNTdGFydD0vdXNyL2xvY2FsL3NiaW4vcGctbG9zdGZvdW5kLXdhdGNoZXIuc2gKVXNlcj1yb290Ckdyb3VwPXJvb3QK
 
    - path: /etc/systemd/system/pg-lostfound-watcher.timer
      permissions: '0644'
      encoding: b64
      content: |
        W1VuaXRdCkRlc2NyaXB0aW9uPVN0YXJ0IHBnLWxvc3Rmb3VuZC13YXRjaGVyIGF0IGJvb3QKCltUaW1lcl0KT25Cb290U2VjPTEwc2VjClBlcnNpc3RlbnQ9dHJ1ZQoKW0luc3RhbGxdCldhbnRlZEJ5PXRpbWVycy50YXJnZXQK
  postruncmd:
    - systemctl daemon-reload
    - systemctl enable --now pg-lostfound-watcher.timer

juju model-config --file model-userdata.yaml

Please let me know if need anything else.

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/DPE-9049.

This message was autogenerated

[taurus-forever]

This is a last blocker of us to release PG16 K8s as 16/stable (you are reporting 16/edge here).
WIP: https://docs.google.com/document/d/1eCVf0OH0BZ6GAt9bhxhgObWubPWttn5IZa0GXVtdvJo/edit?usp=sharing

[alanbach]

Hello @taurus-forever Do we have any updates on this bug? I just checked the latest charm on 16/edge and that is still having the same issue.

[taurus-forever]

@alanbach this is a top priority ticket.
JGYI, 16/edge for K8s charm got a fix already. VM is WIP by @marceloneppel (see open PRs linked above).

It is extremely hard to fix it in a backward compatible way, so juju refresh will work for already installed systems. We have several implementations already which have been rejected on the late testing stages.

Building a new approach currently... stay tuned!

[alanbach]

Hello @taurus-forever Thank you so much for the update and looking forward to this.