My personal blog
hauleth.dev
blog
ft: write the part II of article about systemd
+429
-7
-1
config.toml
-1
config.toml
+8
content/post/indie-web.md
+8
content/post/indie-web.md
+2
-2
content/post/who-watches-watchmen-i.md
+2
-2
content/post/who-watches-watchmen-i.md
···
508
508
This blog post is already quite lengthy, so I will split it into separate parts.
509
509
There probably will be 3 of them:
510
510
511
-
- <a href="#top">Part 1 - Basics, security, and FD passing (this one)</a>
512
-
- Part 2 - Socket activation
511
+
- [Part 1 - Basics, security, and FD passing (this one)](./#top)
512
+
- [Part 2 - Socket activation](@/post/who-watches-watchmen-ii.md)
513
513
- Part 3 - Logging
+414
content/post/who-watches-watchmen-ii.md
+414
content/post/who-watches-watchmen-ii.md
···
1
+
+++
2
+
title = "Who Watches Watchmen? - Part 2"
3
+
date = 2023-07-04
4
+
5
+
description = """
6
+
Continuation of travel into making systemd to work for us, not against us. This
7
+
time we will talk about socket activation and how to make our application run
8
+
only when we need it to run.
9
+
"""
10
+
11
+
[taxonomies]
12
+
tags = [
13
+
"elixir",
14
+
"programming",
15
+
"systemd",
16
+
"deployment"
17
+
]
18
+
19
+
[[extra.thanks]]
20
+
name = "Nicodemus"
21
+
why = "Helping me with my poor English"
22
+
+++
23
+
24
+
This is continuation of [Part I][part-i] where I described the basics of the
25
+
supervising BEAM applications with systemd and how to create basic, secure
26
+
service for your Elixir application with it. In this article I will assume that
27
+
you have read [the previous one][part-i].
28
+
29
+
______________________________________________________________________
30
+
31
+
We already have our super simple service description. Just to refresh your
32
+
memory, it is the `hello.service` file once again:
33
+
34
+
```ini
35
+
[Unit]
36
+
Description=Hello World service
37
+
Requires=network.target
38
+
39
+
[Service]
40
+
Type=notify
41
+
Environment=PORT=80
42
+
ExecStart=/opt/hello/bin/hello start
43
+
WatchdogSec=1min
44
+
45
+
# We need to add capability to be able to bind on port 80
46
+
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
47
+
48
+
# Hardening
49
+
DynamicUser=true
50
+
PrivateDevices=true
51
+
Environment=ERL_CRASH_DUMP_SECONDS=0
52
+
```
53
+
54
+
However there is one small problem. It allows our service to listen on **any**
55
+
restricted port, not just `80` that we want to listen on. This can be
56
+
troublesome as an attacker that gains RCE on our server can then capture any
57
+
traffic on any port that we do not want to open (for example exposing port 22
58
+
using the [`ssh`] module).
59
+
60
+
It would be nice if we could somehow inject sockets for only the ports we want
61
+
to listen to into our application.
62
+
63
+
## Socket passing
64
+
65
+
Thanks to the [`systemd.socket`][systemd.socket] feature we can achieve that
66
+
with a little work on our side.
67
+
68
+
First we need to create new unit named `hello.socket` next to our
69
+
`hello.service`:
70
+
71
+
```ini
72
+
[Unit]
73
+
Description=Listening socket
74
+
Requires=sockets.target
75
+
76
+
[Socket]
77
+
ListenStream=80
78
+
BindIPv6Only=both
79
+
ReusePort=true
80
+
NoDelay=true
81
+
```
82
+
83
+
It will create a socket connected to TCP 80 (because we used `ListenStream=`,
84
+
and TCP is the stream protocol). By default it will bind that socket to a
85
+
service named the same as our socket, so now we need to edit our `hello.service`
86
+
a little bit:
87
+
88
+
```ini
89
+
[Unit]
90
+
Description=Hello World service
91
+
Requires=network.target
92
+
93
+
[Service]
94
+
Type=notify
95
+
Environment=PORT=80
96
+
ExecStart=/opt/hello/bin/hello start
97
+
WatchdogSec=1min
98
+
99
+
# See, we no longer need to insecurely allow binding to any port
100
+
# CapabilityBoundingSet=CAP_NET_BIND_SERVICE
101
+
102
+
# Hardening
103
+
DynamicUser=true
104
+
PrivateDevices=true
105
+
Environment=ERL_CRASH_DUMP_SECONDS=0
106
+
```
107
+
108
+
And we need to modify our `Hello.Application.cowboy_opts/0` to handle the socket
109
+
which is passed to us a file descriptor:
110
+
111
+
```elixir
112
+
# hello/application.ex
113
+
defmodule Hello.Application do
114
+
use Application
115
+
116
+
def start(_type, _opts) do
117
+
fds = :systemd.listen_fds()
118
+
119
+
children = [
120
+
{Plug.Cowboy, [scheme: :http, plug: Hello.Router] ++ cowboy_opts(fds)},
121
+
{Plug.Cowboy.Drainer, refs: :all}
122
+
]
123
+
124
+
Supervisor.start_link(children, strategy: :one_for_one)
125
+
end
126
+
127
+
# If there are no sockets passed to the application, then start listening on
128
+
# the port specified by the `PORT` environment variable
129
+
defp cowboy_opts([]) do
130
+
[port: String.to_integer(System.get_env("PORT", "5000"))]
131
+
end
132
+
133
+
# If there are any socket passed, then use first one
134
+
defp cowboy_opts([socket | _]) do
135
+
fd =
136
+
case socket do
137
+
# Sockets can be named, which will be passed as the second element in
138
+
# a tuple
139
+
{fd, _name} -> fd
140
+
# Or unnamed, and then it will be just the file descriptor
141
+
fd -> fd
142
+
end
143
+
144
+
[
145
+
net: :inet6, # (1)
146
+
port: 0, # (2)
147
+
fd: fd # (3)
148
+
]
149
+
end
150
+
end
151
+
```
152
+
153
+
1. Systemd sockets are IPv6 enabled (we explicitly said that we want to listen
154
+
on both). That means, that we need to mark our connection as an INET6
155
+
connection. This will not affect IPv4 (INET) connections.
156
+
1. We are required to pass `:port` key, but its value will be ignored, so we
157
+
just pass `0`.
158
+
1. We pass the file descriptor that will be then passed to the Cowboy listener.
159
+
160
+
Now when we will start our service:
161
+
162
+
```txt
163
+
# systemctl start hello.service
164
+
```
165
+
166
+
It will be available at `https://localhost/` while still running as an
167
+
unprivileged user.
168
+
169
+
### Multiple ports
170
+
171
+
The question may arise - how to allow our service to listen on more than one
172
+
port, for example you want to have your website available as HTTPS alongside
173
+
"regular" HTTP. This means that our application needs to listen on two
174
+
restricted ports:
175
+
176
+
- 80 - for HTTP
177
+
- 443 - for HTTPS
178
+
179
+
Now we need to slightly modify a little our socket service and add another one.
180
+
First rename our `hello.socket` to `hello-http.socket` and add a line
181
+
`Service=hello.service` and `FileDescriptorName=http` to `[Socket]` section, so
182
+
we end with:
183
+
184
+
```ini
185
+
[Unit]
186
+
Description=HTTP Socket
187
+
Requires=sockets.target
188
+
189
+
[Socket]
190
+
# We declare the name of the file descriptor here to simplify extraction in
191
+
# the application afterwards. By default it will be the socket name (so
192
+
# `hello-http` in our case), but `http` is much cleaner.
193
+
FileDescriptorName=http
194
+
ListenStream=80
195
+
Service=hello.service
196
+
BindIPv6Only=both
197
+
ReusePort=true
198
+
NoDelay=true
199
+
```
200
+
201
+
Next we create a similar file, but for HTTPS named `hello-https.socket`
202
+
203
+
```ini
204
+
[Unit]
205
+
Description=HTTPS Socket
206
+
Requires=sockets.target
207
+
208
+
[Socket]
209
+
FileDescriptorName=https
210
+
ListenStream=443
211
+
Service=hello.service
212
+
BindIPv6Only=both
213
+
ReusePort=true
214
+
NoDelay=true
215
+
```
216
+
217
+
And we add the dependency on both of our sockets to the `hello.service`:
218
+
219
+
```ini
220
+
[Unit]
221
+
Description=Hello World service
222
+
After=hello-http.socket hello-https.socket
223
+
BindTo=hello-http.socket hello-https.socket
224
+
225
+
[Service]
226
+
ExecStart=/opt/hello/bin/hello start
227
+
228
+
# Hardening
229
+
DynamicUser=true
230
+
PrivateDevices=true
231
+
Environment=ERL_CRASH_DUMB_SECONDS=0
232
+
```
233
+
234
+
Now we need to somehow differentiate between our sockets in the
235
+
`Hello.Application`, so we will be able to pass the proper FD to each of the
236
+
listeners. The `:systemd.listen_fds/0` will return a list of file descriptors,
237
+
and if they are named, the format will be a 2-tuple where the first element is
238
+
the file descriptor and the second is the name as a string:
239
+
240
+
```elixir
241
+
# hello/application.ex
242
+
defmodule Hello.Application do
243
+
use Application
244
+
245
+
def start(_type, _opts) do
246
+
fds = :systemd.listen_fds()
247
+
248
+
router = Hello.Router
249
+
250
+
children = [
251
+
{Plug.Cowboy, [
252
+
scheme: :http,
253
+
plug: router
254
+
] ++ cowboy_opts(fds, "http")},
255
+
{Plug.Cowboy, [
256
+
scheme: :https,
257
+
plug: router,
258
+
keyfile: "path/to/keyfile.pem",
259
+
certfile: "path/to/certfile.pem",
260
+
dhfile: "path/to/dhfile.pem"
261
+
] ++ cowboy_opts(fds, "https")},
262
+
{Plug.Cowboy.Drainer, refs: :all}
263
+
]
264
+
265
+
Supervisor.start_link(children, strategy: :one_for_one)
266
+
end
267
+
268
+
defp cowboy_opts(fds, protocol) do
269
+
case List.keyfind(fds, protocol, 1) do
270
+
# If there is socket passed for given protocol, then use that one
271
+
{fd, ^protocol} ->
272
+
[
273
+
net: :inet6,
274
+
port: 0,
275
+
fd: fd
276
+
]
277
+
278
+
# If there are no sockets passed to the application that match
279
+
# the protocol, then start listening on the port specified by
280
+
# `PORT_{protocol}` environment variable
281
+
_ ->
282
+
[
283
+
port: String.to_integer(System.get_env("PORT_#{protocol}", "5000"))
284
+
]
285
+
end
286
+
end
287
+
```
288
+
289
+
Now our application will listen on both - HTTP and HTTPS, despite running as
290
+
unprivileged user.
291
+
292
+
## Socket activation
293
+
294
+
Now, that we can inject sockets to our application with ease we can achieve even
295
+
more fascinating feature - socket activation.
296
+
297
+
Some of you may used `inetd` in the past, that allows you to dynamically start
298
+
processes on network requests. It is quite an interesting tool that detects
299
+
traffic on certain ports, then spawns a new process to handle it, passing data
300
+
to and from that process via `STDIN` and `STDOUT`. There was a quirk though, it
301
+
required the spawned process to shutdown after it handled the request and it was
302
+
starting a new instance for each request. That works poorly with VMs like BEAM
303
+
that have substantial startup time and are expected to be long-running systems.
304
+
BEAM is capable of handling network requests on it's own.
305
+
306
+
Fortunately for us, the way that we have implemented our systemd service is all
307
+
that we need to have our application dynamically activated. To observe that we
308
+
just need to shutdown everything:
309
+
310
+
```txt
311
+
# systemctl stop hello-http.socket hello-https.socket hello.service
312
+
```
313
+
314
+
And now relaunch **only the sockets**:
315
+
316
+
```txt
317
+
# systemctl start hello-http.socket hello-https.socket
318
+
```
319
+
320
+
We can check, that our service is not running:
321
+
322
+
```txt
323
+
$ systemctl status hello.service
324
+
● hello.service - Hello World service
325
+
Loaded: loaded (/usr/local/lib/systemd/system/hello.service; static; vendor preset: enabled)
326
+
Active: inactive (dead)
327
+
TriggeredBy: ● hello-http.socket ● hello-https.socket
328
+
```
329
+
330
+
We can see the `TriggeredBy` section that tells us, that this service will be
331
+
started by one of the sockets listed there. Let see what will happen when we
332
+
will try to request anything from our application:
333
+
334
+
```txt
335
+
$ curl http://localhost/
336
+
Hello World!
337
+
```
338
+
339
+
You can see that we got a response from our application. This mean that our
340
+
application must have started, and indeed when we check:
341
+
342
+
```txt
343
+
$ systemctl status hello.service
344
+
● hello.service - Hello
345
+
Loaded: loaded (/usr/local/lib/systemd/system/hello.service; static; vendor preset: enabled)
346
+
Active: active (running) since Thu 2022-02-03 13:20:27 CET; 4s ago
347
+
TriggeredBy: ● hello-http.socket ● hello-https.socket
348
+
Main PID: 1106 (beam.smp)
349
+
Tasks: 19 (limit: 1136)
350
+
Memory: 116.7M
351
+
CGroup: /system.slice/hello.service
352
+
├─1106 /opt/hello/erts-12.2/bin/beam.smp -- -root /opt/hello -progname erl -- -home /run/hello -- -noshell -s elixir start_cli -mode embedded -setcookie CR63SVI6L5JAMJSDL3H4XPNMOPHEWSV2FPHCHCAN65CY6ASHMXBA==== -sname hello -c>
353
+
└─1138 erl_child_setup 1024
354
+
```
355
+
356
+
It seems to be running, and if we stop it, then we will get information that it
357
+
still can be activated by our sockets:
358
+
359
+
```txt
360
+
# systemctl stop hello.service
361
+
Warning: Stopping hello.service, but it can still be activated by:
362
+
hello-http.socket hello-https.socket
363
+
```
364
+
365
+
That means, that systemd is still listening on the sockets that we defined, even
366
+
when our application is down, and will start our application again as soon as
367
+
there are any incoming requests.
368
+
369
+
Let test that out again:
370
+
371
+
```txt
372
+
$ curl http://localhost/
373
+
Hello World!
374
+
$ systemctl status hello.service
375
+
● hello.service - Hello
376
+
Loaded: loaded (/usr/local/lib/systemd/system/hello.service; static; vendor preset: enabled)
377
+
Active: active (running) since Thu 2022-02-03 13:22:27 CET; 4s ago
378
+
TriggeredBy: ● hello-http.socket ● hello-https.socket
379
+
Main PID: 3452 (beam.smp)
380
+
Tasks: 19 (limit: 1136)
381
+
Memory: 116.7M
382
+
CGroup: /system.slice/hello.service
383
+
├─3452 /opt/hello/erts-12.2/bin/beam.smp -- -root /opt/hello -progname erl -- -home /run/hello -- -noshell -s elixir start_cli -mode embedded -setcookie CR63SVI6L5JAMJSDL3H4XPNMOPHEWSV2FPHCHCAN65CY6ASHMXBA==== -sname hello -c>
384
+
└─3453 erl_child_setup 1024
385
+
```
386
+
387
+
Our application got launched again, automatically, just by the fact that
388
+
there was incoming TCP connection.
389
+
390
+
Does it work for HTTPS connection as well?
391
+
392
+
```txt
393
+
# systemctl stop hello.service
394
+
$ curl -k https://localhost/
395
+
Hello World!
396
+
```
397
+
398
+
It seems so. Independently of which port we try to reach our application on, it
399
+
will be automatically launched for us and the connection will be properly
400
+
handled. Do note that systemd will not shut down our process after serving the
401
+
request. It will continue to run from that point forward.
402
+
403
+
## Summary
404
+
405
+
I know that it took quite while since the last post (ca. 1.5 years), but I hope
406
+
that I will be able to write the final part much sooner than this.
407
+
408
+
- [Part 1 - Basics, security, and FD passing][part-i]
409
+
- [Part 2 - Socket activation (this one)](./#top)
410
+
- Part 3 - Logging
411
+
412
+
[part-i]: @/post/who-watches-watchmen-i.md
413
+
[systemd.socket]: https://www.freedesktop.org/software/systemd/man/systemd.socket.html
414
+
[`ssh`]: https://erlang.org/doc/man/ssh.html
+3
-4
flake.nix
+3
-4
flake.nix
···
19
19
buildPhase = ''
20
20
git submodule update --init --recursive --depth=1
21
21
zola build -o $out
22
-
'';
22
+
'';
23
23
24
24
dontInstall = true;
25
25
};
26
-
in rec {
26
+
in
27
+
{
27
28
packages = {
28
29
inherit blog;
29
30
};
30
31
defaultPackage = blog;
31
-
/* apps.hello = flake-utils.lib.mkApp { drv = packages.hello; }; */
32
-
/* defaultApp = apps.hello; */
33
32
34
33
devShells.default = pkgs.mkShell {
35
34
inputsFrom = [ blog ];