My nix-darwin and NixOS config
docs: add RAID doc for later
+316
+316
docs/raid-migration.md
+316
docs/raid-migration.md
···
1
+
# Migrating /srv to RAID
2
+
3
+
This guide covers moving the server's `/srv` data disk from a single drive to a
4
+
software RAID array when new hardware arrives. No data should be lost if the
5
+
steps are followed in order.
6
+
7
+
---
8
+
9
+
## Choosing a RAID Level
10
+
11
+
| Level | Drives needed | Usable space | Fault tolerance | Use case |
12
+
|---|---|---|---|---|
13
+
| **RAID 1** | 2 | 50% | 1 drive failure | Best for a 2-drive home server — simple, safe |
14
+
| **RAID 5** | 3+ | n−1 drives | 1 drive failure | Good space efficiency at 3+ drives |
15
+
| **RAID 6** | 4+ | n−2 drives | 2 drive failures | Only worth it at 4+ drives |
16
+
| **RAID 10** | 4 | 50% | 1 per mirrored pair | Fast + redundant, needs 4 drives |
17
+
18
+
**Recommendation for home use: RAID 1 with 2 drives.** It's the most
19
+
straightforward to set up and recover from, and NixOS supports it natively
20
+
via `mdadm`. If you end up with 3+ drives, RAID 5 is a reasonable step up.
21
+
22
+
---
23
+
24
+
## Overview of Changes
25
+
26
+
1. Back up all `/srv` data off-server
27
+
2. Wipe and assemble the new RAID array with `mdadm`
28
+
3. Update `modules/server/storage.nix` and `modules/options.nix` to point at
29
+
the new RAID device
30
+
4. Restore data and rebuild
31
+
32
+
---
33
+
34
+
## Step 1 — Back Up /srv
35
+
36
+
Do this **before touching any disks**.
37
+
38
+
```bash
39
+
# On the server — snapshot /srv to an external drive or remote destination
40
+
# (adjust destination as needed)
41
+
42
+
# Option A: rsync to external USB drive mounted at /mnt/backup
43
+
sudo rsync -aHAXv /srv/ /mnt/backup/srv-snapshot/
44
+
45
+
# Option B: rsync over SSH to your macmini
46
+
rsync -aHAXz -e ssh ewan@server:/srv/ ~/srv-backup/
47
+
48
+
# Verify the backup looks complete
49
+
ls -la /mnt/backup/srv-snapshot/
50
+
du -sh /mnt/backup/srv-snapshot/
51
+
```
52
+
53
+
Don't proceed until you're confident the backup is good.
54
+
55
+
---
56
+
57
+
## Step 2 — Identify the New Drives
58
+
59
+
Boot with the new drives attached and identify their device paths:
60
+
61
+
```bash
62
+
lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE
63
+
```
64
+
65
+
You're looking for the two (or more) new blank drives. Note their paths —
66
+
e.g. `/dev/sdb` and `/dev/sdc`. Double-check by size and model; **do not
67
+
mistake the OS drive or your current `/srv` drive for a blank one**.
68
+
69
+
```bash
70
+
# Confirm a drive is blank / has no important filesystem
71
+
sudo wipefs /dev/sdb
72
+
sudo wipefs /dev/sdc
73
+
```
74
+
75
+
---
76
+
77
+
## Step 3 — Assemble the RAID Array
78
+
79
+
### RAID 1 (2 drives — recommended)
80
+
81
+
```bash
82
+
# Create the array — this is destructive on /dev/sdb and /dev/sdc
83
+
sudo mdadm --create /dev/md0 \
84
+
--level=1 \
85
+
--raid-devices=2 \
86
+
/dev/sdb /dev/sdc
87
+
88
+
# Monitor initial sync progress (takes minutes to hours depending on drive size)
89
+
watch cat /proc/mdstat
90
+
```
91
+
92
+
### RAID 5 (3+ drives)
93
+
94
+
```bash
95
+
sudo mdadm --create /dev/md0 \
96
+
--level=5 \
97
+
--raid-devices=3 \
98
+
/dev/sdb /dev/sdc /dev/sdd
99
+
100
+
watch cat /proc/mdstat
101
+
```
102
+
103
+
> You can use the array before the initial sync finishes — it just runs slower.
104
+
105
+
### Format and label the array
106
+
107
+
```bash
108
+
sudo mkfs.ext4 -L srv /dev/md0
109
+
```
110
+
111
+
> The `storage.nix` auto-format service checks for `TYPE=` from `blkid` and
112
+
> skips formatting if the filesystem is already present, so it is safe to
113
+
> format manually here. The service will see it's already formatted and do nothing.
114
+
115
+
### Get the RAID device UUID
116
+
117
+
```bash
118
+
sudo blkid /dev/md0
119
+
# → /dev/md0: LABEL="srv" UUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" TYPE="ext4"
120
+
121
+
# Also note the mdadm array UUID for the config:
122
+
sudo mdadm --detail /dev/md0 | grep "UUID"
123
+
```
124
+
125
+
---
126
+
127
+
## Step 4 — Update the NixOS Config
128
+
129
+
### 4a. Add the RAID device path to `options.nix`
130
+
131
+
In `modules/options.nix`, update the storage device default to point at the
132
+
RAID device instead of the raw disk UUID:
133
+
134
+
```nix
135
+
# modules/options.nix
136
+
server.storage.srv = {
137
+
device = mkOption {
138
+
type = str;
139
+
# Was: "/dev/disk/by-uuid/<old-single-drive-uuid>"
140
+
default = "/dev/disk/by-label/srv"; # ← RAID array is labelled "srv"
141
+
description = "Block device for /srv (RAID md0 or raw disk).";
142
+
};
143
+
# fsType and options stay the same
144
+
};
145
+
```
146
+
147
+
Using `by-label` rather than `by-uuid` is fine here because the label `srv`
148
+
is set on the RAID array itself, not an individual drive. Alternatively, use
149
+
`/dev/disk/by-uuid/<md0-uuid>` if you prefer UUID-stable references.
150
+
151
+
### 4b. Enable mdadm in `modules/server/storage.nix`
152
+
153
+
Add mdadm configuration so NixOS assembles the array at boot:
154
+
155
+
```nix
156
+
# modules/server/storage.nix
157
+
{ config, pkgs, ... }:
158
+
let
159
+
srv = config.myConfig.server.storage.srv;
160
+
device = srv.device;
161
+
in
162
+
{
163
+
# ── mdadm — assemble RAID at boot ──────────────────────────────────────────
164
+
boot.swraid = {
165
+
enable = true;
166
+
mdadmConf = ''
167
+
MAILADDR root
168
+
'';
169
+
};
170
+
171
+
# ── 1. Auto-format (unchanged — skips if already formatted) ────────────────
172
+
systemd.services."srv-autoformat" = {
173
+
# ... (no changes needed here)
174
+
};
175
+
176
+
# ── 2. /srv mount (unchanged) ──────────────────────────────────────────────
177
+
fileSystems."/srv" = {
178
+
# ... (no changes needed here)
179
+
};
180
+
181
+
# ── 3. Subdirectory creation (unchanged) ────────────────────────────────────
182
+
systemd.tmpfiles.rules = [
183
+
# ... (no changes needed here)
184
+
];
185
+
}
186
+
```
187
+
188
+
The only real addition is the `boot.swraid` block. Everything else in the
189
+
module stays the same because it already references `device` from the option.
190
+
191
+
### 4c. (Optional) Add mdadm monitoring
192
+
193
+
For email alerts on drive failure, add to `modules/server/storage.nix`:
194
+
195
+
```nix
196
+
# Requires postfix or another MTA to be configured separately
197
+
environment.etc."mdadm.conf".text = ''
198
+
MAILADDR root
199
+
ARRAY /dev/md0 UUID=<array-uuid-from-step-3>
200
+
'';
201
+
202
+
systemd.services.mdmonitor = {
203
+
wantedBy = [ "multi-user.target" ];
204
+
};
205
+
```
206
+
207
+
---
208
+
209
+
## Step 5 — Restore Data
210
+
211
+
Mount the new array, then rsync the backup back onto it:
212
+
213
+
```bash
214
+
# Test-mount the new array
215
+
sudo mkdir -p /mnt/newraid
216
+
sudo mount /dev/md0 /mnt/newraid
217
+
218
+
# Restore from your backup (adjust source path)
219
+
sudo rsync -aHAXv /mnt/backup/srv-snapshot/ /mnt/newraid/
220
+
221
+
# Check ownership is preserved correctly
222
+
ls -la /mnt/newraid/forgejo
223
+
ls -la /mnt/newraid/postgresql
224
+
225
+
sudo umount /mnt/newraid
226
+
```
227
+
228
+
---
229
+
230
+
## Step 6 — Apply the Config and Reboot
231
+
232
+
```bash
233
+
cd /home/ewan/.config/nix-config
234
+
235
+
# Rebuild and switch (this writes the new fstab entry and mdadm config)
236
+
sudo nixos-rebuild switch --flake .#server
237
+
238
+
# Reboot so the array is assembled cleanly from the start
239
+
sudo reboot
240
+
```
241
+
242
+
After reboot, verify:
243
+
244
+
```bash
245
+
# Array is up and healthy
246
+
cat /proc/mdstat
247
+
sudo mdadm --detail /dev/md0
248
+
249
+
# /srv is mounted correctly
250
+
mount | grep /srv
251
+
df -h /srv
252
+
253
+
# Services came back up
254
+
systemctl status forgejo nextcloud bluesky-pds
255
+
256
+
# Data looks right
257
+
ls -la /srv/
258
+
```
259
+
260
+
---
261
+
262
+
## Ongoing Maintenance
263
+
264
+
### Check array health
265
+
266
+
```bash
267
+
cat /proc/mdstat
268
+
sudo mdadm --detail /dev/md0
269
+
```
270
+
271
+
### Trigger a manual scrub (checks for silent corruption)
272
+
273
+
```bash
274
+
echo check | sudo tee /sys/block/md0/md/sync_action
275
+
watch cat /proc/mdstat
276
+
```
277
+
278
+
It's worth adding a weekly scrub as a systemd timer — NixOS has
279
+
`boot.swraid`-compatible options for this, or you can write a simple oneshot
280
+
timer in `modules/server/maintenance.nix`.
281
+
282
+
### Replacing a failed drive
283
+
284
+
```bash
285
+
# Mark the failed drive as faulty and remove it
286
+
sudo mdadm /dev/md0 --fail /dev/sdb
287
+
sudo mdadm /dev/md0 --remove /dev/sdb
288
+
289
+
# Physically replace the drive, then add the new one
290
+
sudo mdadm /dev/md0 --add /dev/sdb
291
+
292
+
# Watch the rebuild
293
+
watch cat /proc/mdstat
294
+
```
295
+
296
+
---
297
+
298
+
## Updating This Config for a Fresh Deploy
299
+
300
+
If you ever rebuild the server from scratch on RAID hardware (rather than
301
+
migrating from a single disk), the process is simpler:
302
+
303
+
1. Assemble and format the RAID array before running `nixos-install`
304
+
2. Set `myConfig.server.storage.srv.device` in `hosts/server/default.nix` to
305
+
the correct device path
306
+
3. Ensure `boot.swraid.enable = true` is in `storage.nix`
307
+
4. The auto-format service will see the existing ext4 and skip formatting
308
+
5. Proceed with the normal deploy runbook in `hosts-server.md`
309
+
310
+
---
311
+
312
+
## Resources
313
+
314
+
- [NixOS `boot.swraid` options](https://search.nixos.org/options?query=boot.swraid)
315
+
- [Arch Wiki — mdadm](https://wiki.archlinux.org/title/RAID) — comprehensive mdadm reference
316
+
- [mdadm man page](https://man.archlinux.org/man/mdadm.8)