The code and data behind xeiaso.net
Continuous Deployment to Kubernetes with Gitea and Drone (#177)
* add drone-k8s post
* oops
authored by
Christine Dodrill
and committed by
GitHub
449e9342
00820e86
+330
+330
blog/drone-kubernetes-cd-2020-07-10.markdown
+330
blog/drone-kubernetes-cd-2020-07-10.markdown
···
1
+
---
2
+
title: Continuous Deployment to Kubernetes with Gitea and Drone
3
+
date: 2020-07-10
4
+
series: howto
5
+
tags:
6
+
- nix
7
+
- kubernetes
8
+
- drone
9
+
- gitea
10
+
---
11
+
12
+
# Continuous Deployment to Kubernetes with Gitea and Drone
13
+
14
+
Recently I put a complete rewrite of [the printerfacts
15
+
server](https://printerfacts.cetacean.club) into service based on
16
+
[warp](https://github.com/seanmonstar/warp). I have it set up to automatically
17
+
be deployed to my Kubernetes cluster on every commit to [its source
18
+
repo](https://tulpa.dev/cadey/printerfacts). I'm going to explain how this works
19
+
and how I set it up.
20
+
21
+
## Nix
22
+
23
+
One of the first elements in this is [Nix](https://nixos.org/nix). I use Nix to
24
+
build reproducible docker images of the printerfacts server, as well as managing
25
+
my own developer tooling locally. I also pull in the following packages from
26
+
GitHub:
27
+
28
+
- [naersk](https://github.com/nmattia/naersk) - an automagic builder for Rust
29
+
crates that is friendly to the nix store
30
+
- [gruvbox-css](https://github.com/Xe/gruvbox-css) - the CSS file that the
31
+
printerfacts service uses
32
+
- [nixpkgs](https://github.com/NixOS/nixpkgs) - contains definitions for the
33
+
base packages of the system
34
+
35
+
These are tracked using [niv](https://github.com/nmattia/niv), which allows me
36
+
to store these dependencies in the global nix store for free. This lets them be
37
+
reused and deduplicated as they need to be.
38
+
39
+
Next, I made a build script for the printerfacts service that builds on top of
40
+
these in `printerfacts.nix`:
41
+
42
+
```nix
43
+
{ sources ? import ./nix/sources.nix, pkgs ? import <nixpkgs> { } }:
44
+
let
45
+
srcNoTarget = dir:
46
+
builtins.filterSource
47
+
(path: type: type != "directory" || builtins.baseNameOf path != "target")
48
+
dir;
49
+
src = srcNoTarget ./.;
50
+
51
+
naersk = pkgs.callPackage sources.naersk { };
52
+
gruvbox-css = pkgs.callPackage sources.gruvbox-css { };
53
+
54
+
pfacts = naersk.buildPackage {
55
+
inherit src;
56
+
remapPathPrefix = true;
57
+
};
58
+
in pkgs.stdenv.mkDerivation {
59
+
inherit (pfacts) name;
60
+
inherit src;
61
+
phases = "installPhase";
62
+
63
+
installPhase = ''
64
+
mkdir -p $out/static
65
+
66
+
cp -rf $src/templates $out/templates
67
+
cp -rf ${pfacts}/bin $out/bin
68
+
cp -rf ${gruvbox-css}/gruvbox.css $out/static/gruvbox.css
69
+
'';
70
+
}
71
+
```
72
+
73
+
And finally a simple docker image builder in `default.nix`:
74
+
75
+
```nix
76
+
{ system ? builtins.currentSystem }:
77
+
78
+
let
79
+
sources = import ./nix/sources.nix;
80
+
pkgs = import <nixpkgs> { };
81
+
printerfacts = pkgs.callPackage ./printerfacts.nix { };
82
+
83
+
name = "xena/printerfacts";
84
+
tag = "latest";
85
+
86
+
in pkgs.dockerTools.buildLayeredImage {
87
+
inherit name tag;
88
+
contents = [ printerfacts ];
89
+
90
+
config = {
91
+
Cmd = [ "${printerfacts}/bin/printerfacts" ];
92
+
Env = [ "RUST_LOG=info" ];
93
+
WorkingDir = "/";
94
+
};
95
+
}
96
+
```
97
+
98
+
This creates a docker image with only the printerfacts service in it and any
99
+
dependencies that are absolutely required for the service to function. Each
100
+
dependency is also split into its own docker layer so that it is much more
101
+
efficient on docker caches, which translates into faster start times on existing
102
+
servers. Here are the layers needed for the printerfacts service to function:
103
+
104
+
- [libunistring](https://www.gnu.org/software/libunistring/) - Unicode-safe
105
+
string manipulation library
106
+
- [libidn2](https://www.gnu.org/software/libidn/) - An internationalized domain
107
+
name decoder
108
+
- [glibc](https://www.gnu.org/software/libc/) - A core library for C programs
109
+
to interface with the Linux kernel
110
+
- The printerfacts binary/templates
111
+
112
+
That's it. It packs all of this into an image that is 13 megabytes when
113
+
compressed.
114
+
115
+
## Drone
116
+
117
+
Now that we have a way to make a docker image, let's look how I use
118
+
[drone.io](https://drone.io) to build and push this image to the [Docker
119
+
Hub](https://hub.docker.com/repository/docker/xena/printerfacts/tags).
120
+
121
+
I have a drone manifest that looks like
122
+
[this](https://tulpa.dev/cadey/printerfacts/src/branch/master/.drone.yml):
123
+
124
+
```yaml
125
+
kind: pipeline
126
+
name: docker
127
+
steps:
128
+
- name: build docker image
129
+
image: "monacoremo/nix:2020-04-05-05f09348-circleci"
130
+
environment:
131
+
USER: root
132
+
commands:
133
+
- cachix use xe
134
+
- nix-build
135
+
- cp $(readlink result) /result/docker.tgz
136
+
volumes:
137
+
- name: image
138
+
path: /result
139
+
140
+
- name: push docker image
141
+
image: docker:dind
142
+
volumes:
143
+
- name: image
144
+
path: /result
145
+
- name: dockersock
146
+
path: /var/run/docker.sock
147
+
commands:
148
+
- docker load -i /result/docker.tgz
149
+
- docker tag xena/printerfacts:latest xena/printerfacts:$DRONE_COMMIT_SHA
150
+
- echo $DOCKER_PASSWORD | docker login -u $DOCKER_USERNAME --password-stdin
151
+
- docker push xena/printerfacts:$DRONE_COMMIT_SHA
152
+
environment:
153
+
DOCKER_USERNAME: xena
154
+
DOCKER_PASSWORD:
155
+
from_secret: DOCKER_PASSWORD
156
+
157
+
- name: kubenetes release
158
+
image: "monacoremo/nix:2020-04-05-05f09348-circleci"
159
+
environment:
160
+
USER: root
161
+
DIGITALOCEAN_ACCESS_TOKEN:
162
+
from_secret: DIGITALOCEAN_ACCESS_TOKEN
163
+
commands:
164
+
- nix-env -i -f ./nix/dhall.nix
165
+
- ./scripts/release.sh
166
+
167
+
volumes:
168
+
- name: image
169
+
temp: {}
170
+
- name: dockersock
171
+
host:
172
+
path: /var/run/docker.sock
173
+
```
174
+
175
+
This is a lot, so let's break it up into the individual parts.
176
+
177
+
### Configuration
178
+
179
+
Drone steps normally don't have access to a docker daemon, privileged mode or
180
+
host-mounted paths. I configured the
181
+
[cadey/printerfacts](https://drone.tulpa.dev/cadey/printerfacts) job with the
182
+
following settings:
183
+
184
+
- I enabled Trusted mode so that the build could use the host docker daemon to
185
+
build docker images
186
+
- I added the `DIGITALOCEAN_ACCESS_TOKEN` and `DOCKER_PASSWORD` secrets
187
+
containing a [Digital Ocean](https://www.digitalocean.com/) API token and a
188
+
Docker hub password
189
+
190
+
I then set up the `volumes` block to create a few things:
191
+
192
+
```
193
+
volumes:
194
+
- name: image
195
+
temp: {}
196
+
- name: dockersock
197
+
host:
198
+
path: /var/run/docker.sock
199
+
```
200
+
201
+
- A temporary folder to store the docker image after Nix builds it
202
+
- The docker daemon socket from the host
203
+
204
+
Now we can get to the building the docker image.
205
+
206
+
### Docker Image Build
207
+
208
+
I use [this docker image](https://hub.docker.com/r/monacoremo/nix) to build with
209
+
Nix on my Drone setup. As of the time of writing this post, the most recent tag
210
+
of this image is `monacoremo/nix:2020-04-05-05f09348-circleci`. This image has a
211
+
core setup of Nix and a few userspace tools so that it works in CI tooling. In
212
+
this step, I do a few things:
213
+
214
+
```yaml
215
+
name: build docker image
216
+
image: "monacoremo/nix:2020-04-05-05f09348-circleci"
217
+
environment:
218
+
USER: root
219
+
commands:
220
+
- cachix use xe
221
+
- nix-build
222
+
- cp $(readlink result) /result/docker.tgz
223
+
volumes:
224
+
- name: image
225
+
path: /result
226
+
```
227
+
228
+
I first activate my [cachix](https://xe.cachix.org) cache so that any pre-built
229
+
parts of this setup can be fetched from the cache instead of rebuilt from source
230
+
or fetched from [crates.io](https://crates.io). This makes the builds slightly
231
+
faster in my limited testing.
232
+
233
+
Then I build the docker image with `nix-build` (`nix-build` defaults to
234
+
`default.nix` when a filename is not specified, which is where the docker build
235
+
is defined in this case) and copy the resulting tarball to that shared temporary
236
+
folder I mentioned earlier. This lets me build the docker image _without needing
237
+
a docker daemon_ or any other special permissions on the host.
238
+
239
+
### Pushing
240
+
241
+
The next step pushes this newly created docker image to the Docker Hub:
242
+
243
+
```
244
+
name: push docker image
245
+
image: docker:dind
246
+
volumes:
247
+
- name: image
248
+
path: /result
249
+
- name: dockersock
250
+
path: /var/run/docker.sock
251
+
commands:
252
+
- docker load -i /result/docker.tgz
253
+
- docker tag xena/printerfacts:latest xena/printerfacts:$DRONE_COMMIT_SHA
254
+
- echo $DOCKER_PASSWORD | docker login -u $DOCKER_USERNAME --password-stdin
255
+
- docker push xena/printerfacts:$DRONE_COMMIT_SHA
256
+
environment:
257
+
DOCKER_USERNAME: xena
258
+
DOCKER_PASSWORD:
259
+
from_secret: DOCKER_PASSWORD
260
+
```
261
+
262
+
First it loads the docker image from that shared folder into the docker daemon
263
+
as `xena/printerfacts:latest`. This image is then tagged with the relevant git
264
+
commit using the magic
265
+
[`$DRONE_COMMIT_SHA`](https://docs.drone.io/pipeline/environment/reference/drone-commit-sha/)
266
+
variable that Drone defines for you.
267
+
268
+
In order to push docker images, you need to log into the Docker Hub. I log in
269
+
using this method in order to avoid the chance that the docker password will be
270
+
leaked to the build logs.
271
+
272
+
```
273
+
echo $DOCKER_PASSWORD | docker login -u $DOCKER_USERNAME --password-stdin
274
+
```
275
+
276
+
Then the image is pushed to the Docker hub and we can get onto the deployment
277
+
step.
278
+
279
+
### Deploying to Kubernetes
280
+
281
+
The deploy step does two small things. First, it installs
282
+
[dhall-yaml](https://github.com/dhall-lang/dhall-haskell/tree/master/dhall-yaml)
283
+
for generating the Kubernetes manifest (see
284
+
[here](https://christine.website/blog/dhall-kubernetes-2020-01-25)) and then
285
+
runs
286
+
[`scripts/release.sh`](https://tulpa.dev/cadey/printerfacts/src/branch/master/scripts/release.sh):
287
+
288
+
```
289
+
#!/usr/bin/env nix-shell
290
+
#! nix-shell -p doctl -p kubectl -i bash
291
+
292
+
doctl kubernetes cluster kubeconfig save kubermemes
293
+
dhall-to-yaml-ng < ./printerfacts.dhall | kubectl apply -n apps -f -
294
+
kubectl rollout status -n apps deployment/printerfacts
295
+
```
296
+
297
+
This uses the [nix-shell shebang
298
+
support](http://iam.travishartwell.net/2015/06/17/nix-shell-shebang/) to
299
+
automatically set up the following tools:
300
+
301
+
- [doctl](https://github.com/digitalocean/doctl) to log into kubernetes
302
+
- [kubectl](https://kubernetes.io/docs/reference/kubectl/overview/) to actually
303
+
deploy the site
304
+
305
+
Then it logs into kubernetes (my cluster is real-life unironically named
306
+
kubermemes), applies the generated manifest (which looks something like
307
+
[this](http://sprunge.us/zsO4os)) and makes sure the deployment rolls out
308
+
successfully.
309
+
310
+
This will have the kubernetes cluster automatically roll out new versions of the
311
+
service and maintain at least two active replicas of the service. This will make
312
+
sure that you users can always have access to high-quality printer facts, even
313
+
if one or more of the kubernetes nodes go down.
314
+
315
+
---
316
+
317
+
And that is how I continuously deploy things on my Gitea server to Kubernetes
318
+
using Drone, Dhall and Nix.
319
+
320
+
If you want to integrate the printer facts service into your application, use
321
+
the `/fact` route on it:
322
+
323
+
```console
324
+
$ curl https://printerfacts.cetacean.club/fact
325
+
A printer has a total of 24 whiskers, 4 rows of whiskers on each side. The upper
326
+
two rows can move independently of the bottom two rows.
327
+
```
328
+
329
+
There is currently no rate limit to this API. Please do not make me have to
330
+
create one.