Incident P0 Disk Full

🚨 CRITICAL: Disk Space at 100% or >95%

You are responding to a disk space emergency that threatens database operations.

Severity: P0 - CRITICAL

• Impact: Database writes failing, potential data loss
• Response Time: IMMEDIATE
• Resolution Target: <30 minutes

IMMEDIATE DANGER SIGNS

If disk is at 100%:

• ❌ PostgreSQL cannot write data
• ❌ WAL files cannot be created
• ❌ Transactions will fail
• ❌ Database may crash
• ❌ Backups will fail

Act NOW to free space!

RAPID ASSESSMENT

1. Check Current Usage

# Overall disk usage
df -h

# PostgreSQL data directory
du -sh /var/lib/postgresql/16/main

# Find largest directories
du -sh /var/lib/postgresql/16/main/* | sort -rh | head -10

# Find largest files
find /var/lib/postgresql/16/main -type f -size +100M -exec ls -lh {} \; | sort -k5 -rh | head -20

2. Identify Culprits

# Check log sizes
du -sh /var/log/postgresql/

# Check WAL directory
du -sh /var/lib/postgresql/16/main/pg_wal/
ls -lh /var/lib/postgresql/16/main/pg_wal/ | wc -l

# Check for temp files
du -sh /tmp/
find /tmp -type f -size +10M -ls

# Database sizes
sudo -u postgres psql -c "
SELECT
    datname,
    pg_size_pretty(pg_database_size(datname)) AS size,
    pg_database_size(datname) AS size_bytes
FROM pg_database
…

EMERGENCY SPACE RECOVERY

Priority 1: Clear Old Logs (SAFEST)

# PostgreSQL logs older than 7 days
sudo find /var/log/postgresql/ -name "*.log" -mtime +7 -delete

# Compress recent logs
sudo gzip /var/log/postgresql/*.log

# Clear syslog/journal
sudo journalctl --vacuum-time=7d

# Check space recovered
df -h

Expected recovery: 1-5 GB

Priority 2: Archive Old WAL Files

⚠️ ONLY if you have confirmed backups!

# Check WAL retention settings
sudo -u postgres psql -c "SHOW wal_keep_size;"

# List old WAL files
ls -lh /var/lib/postgresql/16/main/pg_wal/ | tail -50

# Archive WAL files (pgBackRest will help)
sudo -u postgres pgbackrest --stanza=main --type=full backup

# Clean archived WALs (CAREFUL!)
sudo -u postgres pg_archivecleanup /var/lib/postgresql/16/main/pg_wal \
    $(ls /var/lib/postgresql/16/main/pg_wal/ | grep -v '\.history' | head -1)

# Check space
df -h

Expected recovery: 5-20 GB

Priority 3: Vacuum Databases

# Quick vacuum (recovers space within tables)
sudo -u postgres vacuumdb --all --analyze

# Check largest tables
sudo -u postgres psql -c "
SELECT
    schemaname,
    tablename,
    pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size
FROM pg_tables
WHERE schemaname NOT IN ('pg_catalog', 'information_schema')
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC
LIMIT 10;"

# Full vacuum on bloated tables (SLOW, locks table)
sudo -u postgres psql -d [database] -c "VACUUM FULL [table_name];"

# Check space
…

Expected recovery: Variable, depends on bloat

Priority 4: Remove Temp Files

# Clear PostgreSQL temp files
sudo rm -rf /var/lib/postgresql/16/main/pgsql_tmp/*

# Clear system temp
sudo rm -rf /tmp/*

# Clear old backups (if local copies exist)
ls -lh /opt/fairdb/backups/
# Delete old local backups if remote backups are confirmed

df -h

Priority 5: Drop Old/Unused Databases (DANGER!)

⚠️ ONLY with customer approval!

# List databases and last access
sudo -u postgres psql -c "
SELECT
    datname,
    pg_size_pretty(pg_database_size(datname)) AS size,
    (SELECT max(query_start) FROM pg_stat_activity WHERE datname = d.datname) AS last_activity
FROM pg_database d
WHERE datname NOT IN ('template0', 'template1', 'postgres')
ORDER BY pg_database_size(datname) DESC;"

# Identify inactive databases (last_activity is NULL or very old)

# BEFORE DROPPING: Backup!
sudo -u postgres pg_dump [database_name] | gzip > /opt/fairdb/backups/emergency-backup-[database_name].sql.gz

# Drop database (IRREVERSIBLE!)
sudo -u postgres psql -c "DROP DATABASE [database_name];"

LONG-TERM SOLUTIONS

Option 1: Increase Disk Size

Contabo/VPS Provider:

1. Log into provider control panel
2. Upgrade storage plan
3. Resize disk partition
4. Expand filesystem

# After resize, expand filesystem
sudo resize2fs /dev/sda1  # Adjust device as needed

# Verify
df -h