deployment templates for lichen
add optional backup.yml: borgbase backups via systemd timers
backup.yml (opt-in) installs borg, uploads the borgbase SSH key,
initializes the repo on first run if empty, and registers two systemd
timers:
- lichen-backup.timer (daily at 03:00 by default) — borg create +
prune + compact against the lichen_data docker volume
- lichen-borg-check.timer (first Sunday of the month) — borg check
--verify-data to catch archive corruption early
No app downtime — backups run hot against the live volume. Site data is
git-managed with atomic renames, so a mid-commit archive restores cleanly.
Retention defaults: 7 daily / 4 weekly / 6 monthly. Variables are in the
playbook header; overridable via group_vars.
Also adds inventory.yml to the submodule .gitignore to keep operators'
real inventories out of the repo.
+304
-1
+1
.gitignore
+1
.gitignore
···
1
+
inventory.yml
+57
-1
ansible/README.md
+57
-1
ansible/README.md
···
63
63
| `lichen_deploy_dir` | `/srv/lichen` | target directory on the server |
64
64
| `lichen_rust_log` | `info` | `RUST_LOG` for the container |
65
65
66
+
## Backups (optional)
67
+
68
+
`backup.yml` installs Borg + systemd timers that back up the `lichen_data`
69
+
docker volume to a BorgBase repository daily, and verify archive integrity
70
+
monthly. The app is not stopped during the backup — lichen's site data is
71
+
git-managed and uses atomic writes, so hot archives restore cleanly.
72
+
73
+
### Setup
74
+
75
+
1. Create a BorgBase repo and upload your SSH public key in their dashboard.
76
+
2. Add these to `group_vars/all.yml` (vault-encrypt the secrets):
77
+
```yaml
78
+
borg_repo: "ssh://xxxxxx@xxxxxx.repo.borgbase.com/./repo"
79
+
borg_passphrase: "your-borg-repo-passphrase"
80
+
borg_ssh_key_src: "~/.ssh/borgbase_ed25519"
81
+
```
82
+
3. Run the playbook:
83
+
```bash
84
+
ansible-playbook -i inventory.yml backup.yml --ask-vault-pass
85
+
```
86
+
87
+
First run uploads the key, writes `/etc/lichen-borg.env`, initializes the repo
88
+
if empty, installs `/usr/local/bin/lichen-backup` + `/usr/local/bin/lichen-borg-check`,
89
+
and enables the two timers.
90
+
91
+
### Trigger a backup manually
92
+
93
+
```bash
94
+
ssh root@server 'systemctl start lichen-backup.service'
95
+
journalctl -u lichen-backup.service --no-pager -n 50
96
+
```
97
+
98
+
### Restore
99
+
100
+
```bash
101
+
# list archives
102
+
borg list "$BORG_REPO"
103
+
104
+
# extract the latest into a temp dir
105
+
mkdir /tmp/restore && cd /tmp/restore
106
+
borg extract "$BORG_REPO::lichen-2026-04-17T03:00:00"
107
+
108
+
# stop the app, rsync the contents back over the live volume, restart
109
+
docker compose -f /srv/lichen/docker-compose.yml stop app
110
+
rsync -a --delete var/lib/docker/volumes/lichen_lichen_data/_data/ \
111
+
/var/lib/docker/volumes/lichen_lichen_data/_data/
112
+
docker compose -f /srv/lichen/docker-compose.yml start app
113
+
```
114
+
115
+
### Retention
116
+
117
+
Defaults: 7 daily / 4 weekly / 6 monthly. Override via
118
+
`borg_keep_daily`, `borg_keep_weekly`, `borg_keep_monthly`.
119
+
66
120
## Files
67
121
68
122
```
···
70
124
├── ansible.cfg
71
125
├── deploy.yml # install docker + deploy from scratch
72
126
├── update.yml # pull latest image + restart
127
+
├── backup.yml # install borg + timers (optional)
73
128
├── inventory.example.yml
74
129
├── roles/
75
130
│ ├── docker/ # install docker engine + compose plugin
76
-
│ └── lichen/ # sync compose files, render .env, bring up stack
131
+
│ ├── lichen/ # sync compose files, render .env, bring up stack
132
+
│ └── borg/ # borg install, scripts, systemd units
77
133
└── group_vars/
78
134
└── .gitkeep # put vault-encrypted vars here
79
135
```
+40
ansible/backup.yml
+40
ansible/backup.yml
···
1
+
---
2
+
- name: Set up BorgBase backups for a lichen deployment
3
+
hosts: all
4
+
become: true
5
+
vars:
6
+
borg_ssh_key_dest: /root/.ssh/borgbase_ed25519
7
+
borg_schedule: "*-*-* 03:00:00"
8
+
borg_check_schedule: "Sun *-*-01..07 04:00:00"
9
+
borg_keep_daily: 7
10
+
borg_keep_weekly: 4
11
+
borg_keep_monthly: 6
12
+
borg_source_path: /var/lib/docker/volumes/lichen_lichen_data/_data
13
+
pre_tasks:
14
+
- name: Require borg_repo
15
+
assert:
16
+
that:
17
+
- borg_repo is defined
18
+
- borg_repo | length > 0
19
+
fail_msg: >-
20
+
borg_repo must be set (e.g.
21
+
"ssh://xxx@xxx.repo.borgbase.com/./repo"). Put it in group_vars
22
+
or pass via --extra-vars.
23
+
- name: Require borg_passphrase
24
+
assert:
25
+
that:
26
+
- borg_passphrase is defined
27
+
- borg_passphrase | length > 0
28
+
fail_msg: >-
29
+
borg_passphrase must be supplied (vault-encrypted in group_vars
30
+
or passed via --extra-vars).
31
+
- name: Require borg_ssh_key_src
32
+
assert:
33
+
that:
34
+
- borg_ssh_key_src is defined
35
+
- borg_ssh_key_src | length > 0
36
+
fail_msg: >-
37
+
borg_ssh_key_src must point at the private key (on your workstation)
38
+
that BorgBase has the matching public key for.
39
+
roles:
40
+
- borg
+4
ansible/roles/borg/handlers/main.yml
+4
ansible/roles/borg/handlers/main.yml
+104
ansible/roles/borg/tasks/main.yml
+104
ansible/roles/borg/tasks/main.yml
···
1
+
---
2
+
- name: Install borgbackup
3
+
package:
4
+
name: borgbackup
5
+
state: present
6
+
7
+
- name: Ensure /root/.ssh exists
8
+
file:
9
+
path: /root/.ssh
10
+
state: directory
11
+
mode: "0700"
12
+
owner: root
13
+
group: root
14
+
15
+
- name: Upload borgbase private key
16
+
copy:
17
+
src: "{{ borg_ssh_key_src }}"
18
+
dest: "{{ borg_ssh_key_dest }}"
19
+
mode: "0600"
20
+
owner: root
21
+
group: root
22
+
no_log: true
23
+
24
+
- name: Write borg environment file
25
+
template:
26
+
src: lichen-borg.env.j2
27
+
dest: /etc/lichen-borg.env
28
+
mode: "0600"
29
+
owner: root
30
+
group: root
31
+
no_log: true
32
+
33
+
- name: Install backup script
34
+
template:
35
+
src: lichen-backup.sh.j2
36
+
dest: /usr/local/bin/lichen-backup
37
+
mode: "0755"
38
+
39
+
- name: Install borg-check script
40
+
template:
41
+
src: lichen-borg-check.sh.j2
42
+
dest: /usr/local/bin/lichen-borg-check
43
+
mode: "0755"
44
+
45
+
- name: Check if borg repo is initialized
46
+
shell: |
47
+
set -a
48
+
. /etc/lichen-borg.env
49
+
borg info "$BORG_REPO" >/dev/null 2>&1
50
+
register: borg_info
51
+
failed_when: false
52
+
changed_when: false
53
+
no_log: true
54
+
55
+
- name: Initialize borg repo on first run
56
+
shell: |
57
+
set -a
58
+
. /etc/lichen-borg.env
59
+
borg init --encryption=repokey-blake2 "$BORG_REPO"
60
+
when: borg_info.rc != 0
61
+
no_log: true
62
+
63
+
- name: Install lichen-backup systemd service
64
+
template:
65
+
src: lichen-backup.service.j2
66
+
dest: /etc/systemd/system/lichen-backup.service
67
+
mode: "0644"
68
+
notify: reload systemd
69
+
70
+
- name: Install lichen-backup systemd timer
71
+
template:
72
+
src: lichen-backup.timer.j2
73
+
dest: /etc/systemd/system/lichen-backup.timer
74
+
mode: "0644"
75
+
notify: reload systemd
76
+
77
+
- name: Install lichen-borg-check systemd service
78
+
template:
79
+
src: lichen-borg-check.service.j2
80
+
dest: /etc/systemd/system/lichen-borg-check.service
81
+
mode: "0644"
82
+
notify: reload systemd
83
+
84
+
- name: Install lichen-borg-check systemd timer
85
+
template:
86
+
src: lichen-borg-check.timer.j2
87
+
dest: /etc/systemd/system/lichen-borg-check.timer
88
+
mode: "0644"
89
+
notify: reload systemd
90
+
91
+
- name: Flush handlers before enabling timers
92
+
meta: flush_handlers
93
+
94
+
- name: Enable and start lichen-backup timer
95
+
systemd:
96
+
name: lichen-backup.timer
97
+
enabled: true
98
+
state: started
99
+
100
+
- name: Enable and start lichen-borg-check timer
101
+
systemd:
102
+
name: lichen-borg-check.timer
103
+
enabled: true
104
+
state: started
+11
ansible/roles/borg/templates/lichen-backup.service.j2
+11
ansible/roles/borg/templates/lichen-backup.service.j2
+40
ansible/roles/borg/templates/lichen-backup.sh.j2
+40
ansible/roles/borg/templates/lichen-backup.sh.j2
···
1
+
#!/bin/bash
2
+
# lichen-backup — daily borg backup of the lichen_data docker volume
3
+
#
4
+
# runs the backup "hot" without stopping the app. site data is git-managed,
5
+
# which uses atomic rename for writes, so an archive captured mid-commit
6
+
# restores cleanly (worst case a `git fsck` on the restored copy).
7
+
8
+
set -euo pipefail
9
+
set -a
10
+
. /etc/lichen-borg.env
11
+
set +a
12
+
13
+
SOURCE="{{ borg_source_path }}"
14
+
15
+
if [ ! -d "$SOURCE" ]; then
16
+
echo "source path $SOURCE does not exist — is the lichen stack deployed?" >&2
17
+
exit 1
18
+
fi
19
+
20
+
echo "=> creating archive"
21
+
borg create \
22
+
--stats \
23
+
--compression zstd \
24
+
--exclude-caches \
25
+
"$BORG_REPO::lichen-{now:%Y-%m-%dT%H:%M:%S}" \
26
+
"$SOURCE"
27
+
28
+
echo "=> pruning old archives"
29
+
borg prune \
30
+
--list \
31
+
--glob-archives 'lichen-*' \
32
+
--keep-daily {{ borg_keep_daily }} \
33
+
--keep-weekly {{ borg_keep_weekly }} \
34
+
--keep-monthly {{ borg_keep_monthly }} \
35
+
"$BORG_REPO"
36
+
37
+
echo "=> compacting"
38
+
borg compact "$BORG_REPO"
39
+
40
+
echo "=> done"
+10
ansible/roles/borg/templates/lichen-backup.timer.j2
+10
ansible/roles/borg/templates/lichen-backup.timer.j2
+10
ansible/roles/borg/templates/lichen-borg-check.service.j2
+10
ansible/roles/borg/templates/lichen-borg-check.service.j2
+14
ansible/roles/borg/templates/lichen-borg-check.sh.j2
+14
ansible/roles/borg/templates/lichen-borg-check.sh.j2
···
1
+
#!/bin/bash
2
+
# lichen-borg-check — monthly integrity check of the borg repo
3
+
#
4
+
# slow but worth it. catches archive corruption before you need the backup.
5
+
6
+
set -euo pipefail
7
+
set -a
8
+
. /etc/lichen-borg.env
9
+
set +a
10
+
11
+
echo "=> verifying archives and data"
12
+
borg check --verify-data "$BORG_REPO"
13
+
14
+
echo "=> done"
+10
ansible/roles/borg/templates/lichen-borg-check.timer.j2
+10
ansible/roles/borg/templates/lichen-borg-check.timer.j2