Automated backups for Docker containers on Linux using rsync

Written by

Automated Backups for Docker Containers on Linux using rsync

Data persistence and reliable backups are critical for any production environment, especially when dealing with containerized applications managed by Docker. While Docker provides mechanisms like volumes and bind mounts for data persistence, it doesn’t inherently offer a backup solution. This guide provides Linux System Administrators with a comprehensive, step-by-step approach to implementing automated backups for Docker container data using rsync, a powerful and efficient file synchronization tool.

This guide covers common Linux distributions including Debian/Ubuntu and RHEL/AlmaLinux/Fedora, ensuring a broad applicability for system administrators.

Prerequisites

A running Linux server (Ubuntu, Debian, RHEL, AlmaLinux, Fedora).
Docker installed and configured.
Basic familiarity with Linux command-line operations and shell scripting.
rsync installed (usually present by default, but installation steps will be provided).
Root or sudo privileges on the server.

Understanding the Challenge and Solution

Docker containers are designed to be ephemeral. Their state, including any data generated or modified by applications, must be stored externally to persist beyond the container’s lifecycle. This is typically achieved using:

Docker Volumes: Managed by Docker, these are the preferred way to store data. They are located in a specific directory on the host filesystem (e.g., /var/lib/docker/volumes/).
Bind Mounts: Directly mount a file or directory from the host filesystem into the container.

The challenge lies in backing up this data reliably. Simply copying container files using docker cp is often insufficient because it only copies files within the container’s writable layer, not its volumes or bind mounts, and doesn’t guarantee data consistency for applications that are actively writing data (e.g., databases).

rsync is an excellent choice for this task due to its:

Efficiency: It only transfers the changed parts of files, making subsequent backups much faster.
Flexibility: Supports local and remote synchronization.
Robustness: Can preserve file attributes (permissions, ownership, timestamps).
Incremental Backups: Ideal for creating snapshot-like backups over time.

Backup Strategy Overview

Our strategy involves the following key steps:

Identify the exact location of Docker volumes and bind mounts on the host system.
Ensure data consistency by temporarily stopping the container (recommended for critical data) or using application-specific dump tools (for databases).
Use rsync to copy the data from the host’s volume paths to a designated backup directory.
Restart the container.
Automate this process using a shell script and a cron job.

Step-by-Step Guide

Step 1: Identify Docker Volumes and Bind Mounts

To back up your data, you first need to know where it resides on the host filesystem. Use docker inspect to find this information for your specific container.

docker inspect <container_name_or_id> | grep -A 10 "Mounts"

Look for the "Source" field within the "Mounts" section.

Example Output Snippet:

        "Mounts": [
            {
                "Type": "volume",
                "Name": "my_app_data",
                "Source": "/var/lib/docker/volumes/my_app_data/_data",
                "Destination": "/app/data",
                "Driver": "local",
                "Mode": "rw",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "bind",
                "Source": "/opt/my_app/config",
                "Destination": "/etc/my_app",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],

In this example:

A Docker volume named my_app_data has its data located at /var/lib/docker/volumes/my_app_data/_data on the host.
A bind mount is mapping /opt/my_app/config on the host to /etc/my_app inside the container.

Note down all relevant "Source" paths for the data you wish to back up.

Step 2: Install rsync (if not present)

rsync is typically installed by default on most Linux distributions. If not, you can install it using your distribution’s package manager.

For Debian/Ubuntu:

sudo apt update
sudo apt install rsync

For RHEL/AlmaLinux/Fedora:

sudo dnf install rsync

(For older RHEL/CentOS versions, use sudo yum install rsync).

Step 3: Choose a Backup Destination

Decide where you want to store your backups. This could be:

A dedicated local directory (e.g., /mnt/backups/docker/).
A mounted network share (NFS, SMB/CIFS).
A remote server accessible via SSH (can be used with rsync directly, but this guide focuses on local operations for simplicity).

Ensure the backup destination has sufficient disk space and appropriate permissions.

Step 4: Manual Backup Procedure

Before automating, it’s good practice to perform a manual backup to understand the process.

Option A: Stop Container (Recommended for Data Consistency)

For applications like databases or any service that writes frequently to disk, stopping the container ensures that the data is in a consistent state during the backup, preventing corruption.

# 1. Stop the Docker container
sudo docker stop <container_name>

# 2. Perform the rsync backup
# The trailing slash on the source path (/path/to/volume/_data/) is important.
# It tells rsync to copy the *contents* of the directory, not the directory itself.
# --delete removes files in the destination that no longer exist in the source.
# -a (archive mode) preserves permissions, ownership, timestamps, and handles recursion.
# -v (verbose) shows details of the transfer.
# -h (human-readable) shows file sizes in a readable format.
sudo rsync -avh --delete /var/lib/docker/volumes/my_app_data/_data/ /mnt/backups/docker/my_app_data/

# 3. Start the Docker container
sudo docker start <container_name>

Replace <container_name>, /var/lib/docker/volumes/my_app_data/_data/, and /mnt/backups/docker/my_app_data/ with your actual values.

Option B: Live Backup (Use with Caution, or for Stateless/Read-Only data)

If your application can handle inconsistent reads (e.g., a static file server, or a log volume where occasional missed logs are acceptable), you might perform a live backup without stopping the container. However, for critical data, especially databases, this approach is highly discouraged as it can lead to corrupt backups.

For databases, the correct approach is to use the database’s native dump utility *before* backing up any associated volumes.

# Example for PostgreSQL (assuming 'postgres' is the username and 'my_db' is the database)
# Replace <db_container_name> and credentials as necessary.
# This dumps the database to a file inside the container, then copies it to the host.
sudo docker exec <db_container_name> pg_dump -U postgres my_db > /tmp/my_db_backup.sql
sudo docker cp <db_container_name>:/tmp/my_db_backup.sql /mnt/backups/docker/db_dumps/my_db_backup_$(date +"%Y%m%d%H%M%S").sql

# After dumping, you can then rsync the volume, but the dump is the most critical part for consistency.
# If the volume contains other persistent data, rsync it.
sudo rsync -avh --delete /var/lib/docker/volumes/my_db_data/_data/ /mnt/backups/docker/my_db_data/

Step 5: Create a Backup Script

To automate this, create a bash script that handles the entire process. This script will include stopping the container, performing the rsync, and restarting the container, along with logging.

Create a file, for example, /usr/local/bin/docker-backup.sh:

#!/bin/bash

# --- Configuration ---
# Name of the Docker container to back up
CONTAINER_NAME="my_web_app"

# Absolute path to the Docker volume or bind mount on the host.
# IMPORTANT: Ensure this path ends with a trailing slash to copy contents.
VOLUME_SOURCE_PATH="/var/lib/docker/volumes/my_app_data/_data/" # Example for a Docker volume
# If backing up a bind mount, it might look like: VOLUME_SOURCE_PATH="/opt/my_app/config/"

# Directory where backups will be stored.
# This directory will be created if it doesn't exist.
BACKUP_DEST_BASE="/mnt/backups/docker"

# Path for the log file
LOG_FILE="/var/log/docker_backup_${CONTAINER_NAME}.log"

# --- Script Logic ---
DATE_FORMAT=$(date +"%Y-%m-%d_%H-%M-%S")
BACKUP_DEST_FULL="${BACKUP_DEST_BASE}/${CONTAINER_NAME}"

# Redirect all stdout and stderr to the log file, and also print to console (tee)
exec > >(tee -a ${LOG_FILE}) 2>&1

echo "[$DATE_FORMAT] --- Starting backup for container: ${CONTAINER_NAME} ---"
echo "[$DATE_FORMAT] Source volume path: ${VOLUME_SOURCE_PATH}"
echo "[$DATE_FORMAT] Backup destination: ${BACKUP_DEST_FULL}"

# Create backup directory if it doesn't exist
if [ ! -d "${BACKUP_DEST_FULL}" ]; then
    echo "[$DATE_FORMAT] Creating backup directory: ${BACKUP_DEST_FULL}"
    mkdir -p "${BACKUP_DEST_FULL}"
    if [ $? -ne 0 ]; then
        echo "[$DATE_FORMAT] ERROR: Failed to create backup directory. Exiting."
        exit 1
    fi
fi

# Stop the container for data consistency
echo "[$DATE_FORMAT] Stopping container ${CONTAINER_NAME}..."
sudo docker stop "${CONTAINER_NAME}"
if [ $? -ne 0 ]; then
    echo "[$DATE_FORMAT] WARNING: Failed to stop container ${CONTAINER_NAME}. Attempting backup anyway, but consistency may be compromised."
    # Decide if you want to exit here or continue with a warning. For critical data, you might exit.
    # exit 1
fi

# Perform the rsync backup
echo "[$DATE_FORMAT] Starting rsync for ${VOLUME_SOURCE_PATH} to ${BACKUP_DEST_FULL}..."
sudo rsync -avh --delete "${VOLUME_SOURCE_PATH}" "${BACKUP_DEST_FULL}"
RSYNC_EXIT_CODE=$?

if [ $RSYNC_EXIT_CODE -eq 0 ]; then
    echo "[$DATE_FORMAT] rsync completed successfully."
elif [ $RSYNC_EXIT_CODE -eq 24 ]; then
    echo "[$DATE_FORMAT] rsync completed with warnings (some files vanished before they could be transferred). This might be acceptable depending on usage."
else
    echo "[$DATE_FORMAT] ERROR: rsync failed with exit code: ${RSYNC_EXIT_CODE}. Please check logs for details."
    # Decide if you want to exit here, or continue to restart container.
    # For critical backups, you might want to stop here and investigate.
    # exit 1
fi

# Start the container
echo "[$DATE_FORMAT] Starting container ${CONTAINER_NAME}..."
sudo docker start "${CONTAINER_NAME}"
if [ $? -ne 0 ]; then
    echo "[$DATE_FORMAT] ERROR: Failed to start container ${CONTAINER_NAME}. Manual intervention required!"
    exit 1
fi

echo "[$DATE_FORMAT] Backup process for ${CONTAINER_NAME} finished."
echo "[$DATE_FORMAT] ---------------------------------------------------"

Make the script executable:

sudo chmod +x /usr/local/bin/docker-backup.sh

Important Notes for the script:

Trailing Slash: Double-check the VOLUME_SOURCE_PATH variable. It MUST end with a trailing slash (/) if you want to copy the *contents* of that directory into the destination. Without it, rsync would create a subdirectory named after the source directory inside the destination.
Multiple Volumes: If your container uses multiple volumes or bind mounts, you can either create separate scripts for each or expand this script to loop through an array of source/destination pairs.
Database Dumps: For databases, integrate the docker exec ... pg_dump / mysqldump commands (as shown in Step 4 Option B) *before* the docker stop and rsync commands for the database volume. This ensures a consistent database dump.

Step 6: Automate with Cron

Cron is a time-based job scheduler in Unix-like operating systems. You can use it to run your backup script automatically at specified intervals.

Edit the cron table for the root user (recommended, as Docker commands often require root privileges):

sudo crontab -e

Add a line at the end of the file to schedule your backup. For example, to run the script daily at 2:00 AM:

0 2 * * * /usr/local/bin/docker-backup.sh

Explanation of the cron schedule (0 2 * * *):

0: Minute (0-59)
2: Hour (0-23, 0 is midnight)
*: Day of month (1-31) – every day
*: Month (1-12) – every month
*: Day of week (0-7, 0 or 7 is Sunday) – every day

After saving the crontab file, cron will automatically pick up the new job. The script’s output (including errors) will be logged to the LOG_FILE specified in the script.

Step 7: Testing and Monitoring

Automated backups are useless if they don’t work or if you don’t know they’re failing.

Run Manually: Execute the script manually a few times to ensure it runs without errors and produces the expected output.
Check Logs: Regularly review the LOG_FILE (e.g., /var/log/docker_backup_my_web_app.log) for success messages or any errors/warnings.
Verify Backups: Periodically inspect the backup destination (/mnt/backups/docker/my_web_app/) to confirm that files are being copied correctly and appear intact.
Test Restoration: The most crucial step! Periodically perform a test restoration on a separate machine or a non-production environment to ensure your backups are actually usable. This validates your entire backup strategy.

Advanced Considerations

Retention Policies: Implement a strategy to delete old backups to save disk space. Tools like find with -mtime +N -delete or more sophisticated backup solutions (e.g., BorgBackup, Restic) can manage this.
Remote Backups: Extend the script to use rsync over SSH for off-site backups:
```
rsync -avh --delete "${VOLUME_SOURCE_PATH}" user@remote_host:/path/to/remote/backup/
```
For this, ensure passwordless SSH login is configured using SSH keys.
Security: Ensure backup directories have appropriate permissions. If storing sensitive data, consider encrypting your backups.
Docker Compose: If using Docker Compose, identify volumes by inspecting the services in your docker-compose.yml file. The volume names (and thus their host paths) are usually defined there.
Read-Only Snapshots: For extremely large or critical databases, consider using filesystem snapshots (e.g., LVM snapshots or cloud provider block storage snapshots) to create a consistent point-in-time copy before running rsync. This can minimize downtime.
Monitoring & Alerting: Integrate log monitoring tools (e.g., Splunk, ELK stack, Prometheus/Grafana) to alert you automatically if backup jobs fail or report errors.

Restoration Strategy

A backup is only as good as its restore process. Here’s a general strategy for restoring data:

Stop the Container: Stop the Docker container that uses the volume you wish to restore.
```
sudo docker stop <container_name>
```
Clear Existing Data (Optional but often necessary): If you’re restoring to a clean state or overwriting corrupted data, clear the current volume’s contents.
```
sudo rm -rf /var/lib/docker/volumes/my_app_data/_data/*
```
(WARNING: Be extremely careful with rm -rf and ensure you are targeting the correct path.)
Restore Data with rsync: Copy the data from your backup directory back to the original volume path.
```
sudo rsync -avh /mnt/backups/docker/my_app_data/ /var/lib/docker/volumes/my_app_data/_data/
```
Note: The --delete flag is typically omitted here, as you usually don’t want to delete files from the target that might not be in the specific backup snapshot.
Start the Container:
```
sudo docker start <container_name>
```
Verify Restoration: Check the application logs and functionality to ensure the data has been restored correctly and the application is operating as expected.
Database Restores: If you used native database dumps, restore them using the database’s specific restore commands (e.g., psql -U postgres my_db < /path/to/dump.sql or mysql -u user -p my_db < /path/to/dump.sql). This usually happens after restoring the general volume data.

Conclusion

Implementing an automated backup solution for Docker containers is a fundamental aspect of maintaining robust and reliable services. By leveraging rsync, Linux System Administrators can establish an efficient, incremental, and highly configurable backup process for their Docker volumes and bind mounts. Remember to prioritize data consistency, thoroughly test your restoration procedures, and continuously monitor your backup jobs to ensure the integrity and recoverability of your critical containerized data.

Automated backups for Docker containers on Linux using rsync