forked from
mitchellh.com/tack
Stitch any CI into Tangled
better docs
+174
-219
+30
-183
README.md
+30
-183
README.md
···
5
5
to Tangled using standard ATProto records so they show up natively
6
6
in Tangled's UI.
7
7
8
-
## What it does
8
+
## Example
9
+
10
+
A Tangled workflow that fires a Buildkite pipeline on every push to
11
+
`main` and every pull request targeting `main`:
12
+
13
+
```yaml
14
+
# .tangled/workflows/ci.yml
15
+
when:
16
+
- event: ["push"]
17
+
branch: ["main"]
18
+
- event: ["pull_request"]
19
+
branch: ["main"]
20
+
21
+
tack:
22
+
buildkite:
23
+
pipeline: my-app-ci
24
+
```
25
+
26
+
The `when:` block is the standard Tangled trigger schema; the
27
+
`tack:` block tells tack which Buildkite pipeline to fire and how.
28
+
See [docs/buildkite.md](docs/buildkite.md) for the full set of
29
+
options and end-to-end Buildkite setup. Tack can support multiple
30
+
providers.
31
+
32
+
## How it Works
9
33
10
34
Tack is a drop-in alternative to the stock `spindle` runner. You run
11
35
`tack` and [register it using the standard UI](https://tangled.org/settings/spindles).
···
60
84
that's useful for exercising the jetstream → knot → `/events` flow
61
85
locally without a real CI account.
62
86
63
-
## Buildkite
64
-
65
-
[Buildkite](https://buildkite.com) is the primary provider tack
66
-
supports today. In Buildkite mode, every Tangled pipeline trigger
67
-
fans out into one Buildkite build per workflow on the pipeline that
68
-
workflow names; build state flows back to Tangled via Buildkite's
69
-
notification webhooks.
70
-
71
-
### How it fits together
87
+
## Providers
72
88
73
-
```
74
-
sh.tangled.pipeline Buildkite
75
-
trigger record ──▶ tack ──▶ Create Build ─┐
76
-
│
77
-
/webhooks/buildkite ◀──── notification ◀─────┘
78
-
│
79
-
▼
80
-
sh.tangled.pipeline.status (broadcast on /events)
81
-
```
89
+
Provider-specific setup (Buildkite-side configuration, the
90
+
provider's tack env vars, and the workflow YAML schema) lives in
91
+
its own doc per provider:
82
92
83
-
* **Spawn:** for each workflow on a pipeline trigger, tack POSTs to
84
-
`/v2/organizations/<org>/pipelines/<slug>/builds`. Both `<org>` and
85
-
`<slug>` come from the workflow's YAML body (see
86
-
[Configuring your workflows](#configuring-your-workflows)).
87
-
* **Track:** tack persists the resulting `(build_uuid → knot, rkey,
88
-
workflow)` mapping in its local SQLite store so it can later
89
-
resolve incoming webhooks back to the originating Tangled
90
-
pipeline.
91
-
* **Report:** Buildkite delivers `build.*` events to
92
-
`POST /webhooks/buildkite`. tack authenticates each request,
93
-
translates the Buildkite state into a Tangled status, and
94
-
broadcasts a `sh.tangled.pipeline.status` record on `/events`.
95
-
96
-
### Setting up Buildkite
97
-
98
-
These steps happen once on the Buildkite side, before tack can talk
99
-
to it.
100
-
101
-
#### 1. Create one or more pipelines
102
-
103
-
Each Tangled workflow targets exactly one Buildkite pipeline by
104
-
slug. There's no requirement that pipelines map 1:1 to workflows —
105
-
many users point every workflow at a single pipeline whose
106
-
`pipeline.yml` does `pipeline upload some-file-${TACK_WORKFLOW}.yml`,
107
-
keeping all the per-workflow logic in the repo rather than in
108
-
Buildkite's UI.
109
-
110
-
In your Buildkite org, **Pipelines → New pipeline**:
111
-
112
-
* Repository: any URL (the agent only needs to be able to clone it).
113
-
* Steps: a minimal `pipeline upload` is usually enough — tack passes
114
-
the workflow name through `$TACK_WORKFLOW` so you can branch on
115
-
it.
116
-
117
-
Note the pipeline slug from the URL
118
-
(`https://buildkite.com/<org>/<pipeline-slug>`); your workflow YAML
119
-
will reference it.
120
-
121
-
#### 2. Create an API access token
122
-
123
-
Tack uses a single API token to create builds, list jobs, and fetch
124
-
logs. Generate one at
125
-
<https://buildkite.com/user/api-access-tokens> with these scopes:
126
-
127
-
| Scope | Used for |
128
-
| ------------------- | ------------------------------------------------- |
129
-
| `read_organizations`| Sanity-checking the configured org slug |
130
-
| `write_builds` | `POST .../builds` when a Tangled trigger arrives |
131
-
| `read_builds` | Resolving build → jobs for the `/logs` endpoint |
132
-
| `read_build_logs` | Streaming job logs back to the Tangled appview |
133
-
134
-
Restrict the token to the specific organization(s) tack will spawn
135
-
into.
136
-
137
-
#### 3. Configure a notification webhook
138
-
139
-
Builds report their state back to tack through Buildkite's
140
-
notification service.
141
-
142
-
In your Buildkite org, **Settings → Notification Services → Add →
143
-
Webhook**:
144
-
145
-
* **Webhook URL:** `https://<your-tack-host>/webhooks/buildkite`
146
-
* **Token / Secret:** any high-entropy string. You'll set the same
147
-
value in `TACK_BUILDKITE_WEBHOOK_SECRET`.
148
-
* **Events:** `build.scheduled`, `build.running`, `build.finished`
149
-
(job-level events are ignored).
150
-
* **Pipelines:** the pipelines tack will fire builds on.
151
-
152
-
Buildkite supports two header schemes for authenticating webhooks;
153
-
tack supports both:
154
-
155
-
| Header scheme | `TACK_BUILDKITE_WEBHOOK_MODE` | Notes |
156
-
| ----------------------- | ----------------------------- | -------------------------------------------- |
157
-
| `X-Buildkite-Token` | `token` (default) | Secret is sent verbatim in the header |
158
-
| `X-Buildkite-Signature` | `signature` | HMAC-SHA256 of `<timestamp>.<body>`; safer |
159
-
160
-
Pick `signature` if the notification setting offers it — it doesn't
161
-
expose the secret on the wire.
162
-
163
-
### Configuring tack
164
-
165
-
Setting `TACK_BUILDKITE_TOKEN` is the master switch that puts tack
166
-
into Buildkite mode. The other variables in this section are then
167
-
required.
168
-
169
-
| Env var | Description |
170
-
| ------------------------------- | ------------------------------------------------------------------------------ |
171
-
| `TACK_BUILDKITE_TOKEN` | Buildkite API token (enables Buildkite mode) |
172
-
| `TACK_BUILDKITE_ORG` | Default Buildkite organization slug (workflows may override via YAML) |
173
-
| `TACK_BUILDKITE_WEBHOOK_SECRET` | Shared secret for `/webhooks/buildkite` auth |
174
-
| `TACK_BUILDKITE_WEBHOOK_MODE` | `token` (default) or `signature` — must match the notification service |
175
-
176
-
The pipeline a workflow runs against is **not** an environment
177
-
variable. It lives inside the workflow YAML so each repo can target
178
-
its own pipeline without an operator round-trip.
179
-
180
-
### Configuring your workflows
181
-
182
-
A Tangled workflow's `raw` body is parsed by tack as YAML. Only
183
-
`pipeline` is required — every other field is an optional override
184
-
or extension of what the trigger metadata already provides:
185
-
186
-
```yaml
187
-
# Required: which Buildkite pipeline this workflow fires.
188
-
pipeline: my-pipeline-slug
189
-
190
-
# Optional: org override. Defaults to TACK_BUILDKITE_ORG. The API
191
-
# token must have access to whichever org you target.
192
-
org: another-org
193
-
194
-
# Optional: human-readable build message (default: "tangled: <name>").
195
-
message: "Custom build message"
196
-
197
-
# Optional: pin the commit/branch tack would otherwise derive from
198
-
# the trigger. Useful for manual triggers (which carry no commit).
199
-
commit: abcdef0123
200
-
branch: main
201
-
202
-
# Optional: extra env + meta_data merged on top of tack's defaults
203
-
# (see "What tack injects into every build" below).
204
-
env:
205
-
CUSTOM_VAR: value
206
-
meta_data:
207
-
custom-key: value
208
-
209
-
# Optional: forwarded verbatim to the Buildkite create-build API.
210
-
clean_checkout: true
211
-
ignore_pipeline_branch_filters: true # default: true
212
-
author:
213
-
name: "Author Name"
214
-
email: "author@example.com"
215
-
```
216
-
217
-
When the trigger is a pull request, tack auto-populates Buildkite's
218
-
`pull_request_base_branch` from the PR target so step-level branch
219
-
filters work without extra config.
220
-
221
-
#### What tack injects into every build
222
-
223
-
Regardless of what the workflow YAML adds on top, tack always
224
-
provides the following so your Buildkite pipeline can recover the
225
-
Tangled identity of the build:
226
-
227
-
| Channel | Key | Value |
228
-
| ----------- | -------------------- | ---------------------------------------- |
229
-
| `env` | `TACK_KNOT` | knot hostname the pipeline came from |
230
-
| `env` | `TACK_PIPELINE_RKEY` | rkey of the originating pipeline record |
231
-
| `env` | `TACK_WORKFLOW` | workflow name (typically a YAML filename) |
232
-
| `env` | `TACK_WORKFLOW_RAW` | the workflow's raw YAML body |
233
-
| `meta_data` | `tack:knot` | same as `TACK_KNOT` |
234
-
| `meta_data` | `tack:pipeline_rkey` | same as `TACK_PIPELINE_RKEY` |
235
-
| `meta_data` | `tack:workflow` | same as `TACK_WORKFLOW` |
236
-
237
-
A common pattern is for the Buildkite pipeline's root step to do a
238
-
`pipeline upload` against a workflow-specific YAML file based on
239
-
`$TACK_WORKFLOW`, e.g.:
240
-
241
-
```yaml
242
-
# Buildkite pipeline.yml
243
-
steps:
244
-
- label: ":pipeline: dispatch ${TACK_WORKFLOW}"
245
-
command: "buildkite-agent pipeline upload .buildkite/${TACK_WORKFLOW}"
246
-
```
93
+
* [Buildkite](docs/buildkite.md)
+126
docs/buildkite.md
+126
docs/buildkite.md
···
1
+
# Buildkite
2
+
3
+
For [Buildkite](https://buildkite.com), every Tangled pipeline trigger
4
+
fans out into one Buildkite build per workflow on the pipeline that
5
+
workflow configures. Buildkite must be configured with a webhook
6
+
back to Tack to communicate status updates.
7
+
8
+
## Setting up Buildkite
9
+
10
+
This must happen before configuring tack within Buildkite.
11
+
12
+
### 1. Create one or more pipelines
13
+
14
+
In your Buildkite org, **Pipelines → New pipeline**:
15
+
16
+
* Repository: any URL (the agent only needs to be able to clone it).
17
+
* Steps: whatever you want.
18
+
19
+
Note the pipeline slug from the URL
20
+
(`https://buildkite.com/<org>/<pipeline-slug>`); your workflow YAML
21
+
will reference it.
22
+
23
+
### 2. Create an API access token
24
+
25
+
Tack uses a single API token to create builds, list jobs, and fetch
26
+
logs. Generate one at
27
+
<https://buildkite.com/user/api-access-tokens> with these scopes:
28
+
29
+
| Scope | Used for |
30
+
| ------------------- | ------------------------------------------------- |
31
+
| `read_organizations`| Sanity-checking the configured org slug |
32
+
| `write_builds` | `POST .../builds` when a Tangled trigger arrives |
33
+
| `read_builds` | Resolving build → jobs for the `/logs` endpoint |
34
+
| `read_build_logs` | Streaming job logs back to the Tangled appview |
35
+
36
+
Restrict the token to the specific organization(s) tack will spawn
37
+
into.
38
+
39
+
### 3. Configure a notification webhook
40
+
41
+
Builds report their state back to tack through Buildkite's
42
+
notification service.
43
+
44
+
In your Buildkite org, **Settings → Notification Services → Add →
45
+
Webhook**:
46
+
47
+
* **Webhook URL:** `https://<your-tack-host>/webhooks/buildkite`
48
+
* **Token / Secret:** any high-entropy string. You'll set the same
49
+
value in `TACK_BUILDKITE_WEBHOOK_SECRET`.
50
+
* **Events:** `build.scheduled`, `build.running`, `build.finished`
51
+
(job-level events are ignored).
52
+
* **Pipelines:** the pipelines tack will fire builds on.
53
+
54
+
Buildkite supports two header schemes for authenticating webhooks and
55
+
Tack supports both:
56
+
57
+
| Header scheme | `TACK_BUILDKITE_WEBHOOK_MODE` | Notes |
58
+
| ----------------------- | ----------------------------- | -------------------------------------------- |
59
+
| `X-Buildkite-Token` | `token` (default) | Secret is sent verbatim in the header |
60
+
| `X-Buildkite-Signature` | `signature` | HMAC-SHA256 of `<timestamp>.<body>`; safer |
61
+
62
+
## Configure Tack
63
+
64
+
| Env var | Description |
65
+
| ------------------------------- | ------------------------------------------------------------------------------ |
66
+
| `TACK_BUILDKITE_TOKEN` | Buildkite API token (enables Buildkite mode) |
67
+
| `TACK_BUILDKITE_ORG` | Default Buildkite organization slug (workflows may override via YAML) |
68
+
| `TACK_BUILDKITE_WEBHOOK_SECRET` | Shared secret for `/webhooks/buildkite` auth |
69
+
| `TACK_BUILDKITE_WEBHOOK_MODE` | `token` (default) or `signature` — must match the notification service |
70
+
71
+
The pipeline a workflow runs against is **not** an environment
72
+
variable. It lives inside the workflow YAML so each repo can target
73
+
its own pipeline without an operator round-trip.
74
+
75
+
## Configuring your Tangled workflows
76
+
77
+
Tack's configuration lives under a `tack:` namespace so the workflow body
78
+
can grow other top-level keys without colliding.
79
+
80
+
Only `pipeline` is required:
81
+
82
+
```yaml
83
+
tack:
84
+
buildkite:
85
+
# Required: which Buildkite pipeline this workflow fires.
86
+
pipeline: my-pipeline-slug
87
+
88
+
# Optional: org override. Defaults to TACK_BUILDKITE_ORG. The
89
+
# API token must have access to whichever org you target.
90
+
org: another-org
91
+
92
+
# Optional: forwarded verbatim to the Buildkite create-build
93
+
# API. Omit to use Buildkite's default (false).
94
+
clean_checkout: true
95
+
```
96
+
97
+
When the trigger is a pull request, tack auto-populates Buildkite's
98
+
`pull_request_base_branch` from the PR target so step-level branch
99
+
filters work without extra config.
100
+
101
+
### What tack injects into every build
102
+
103
+
Regardless of what the workflow YAML adds on top, tack always
104
+
provides the following so your Buildkite pipeline can recover the
105
+
Tangled identity of the build:
106
+
107
+
| Channel | Key | Value |
108
+
| ----------- | -------------------- | ---------------------------------------- |
109
+
| `env` | `TACK_KNOT` | knot hostname the pipeline came from |
110
+
| `env` | `TACK_PIPELINE_RKEY` | rkey of the originating pipeline record |
111
+
| `env` | `TACK_WORKFLOW` | workflow name (typically a YAML filename) |
112
+
| `env` | `TACK_WORKFLOW_RAW` | the workflow's raw YAML body |
113
+
| `meta_data` | `tack:knot` | same as `TACK_KNOT` |
114
+
| `meta_data` | `tack:pipeline_rkey` | same as `TACK_PIPELINE_RKEY` |
115
+
| `meta_data` | `tack:workflow` | same as `TACK_WORKFLOW` |
116
+
117
+
A common pattern is for the Buildkite pipeline's root step to do a
118
+
`pipeline upload` against a workflow-specific YAML file based on
119
+
`$TACK_WORKFLOW`, e.g.:
120
+
121
+
```yaml
122
+
# Buildkite pipeline.yml
123
+
steps:
124
+
- label: ":pipeline: dispatch ${TACK_WORKFLOW}"
125
+
command: "buildkite-agent pipeline upload .buildkite/${TACK_WORKFLOW}"
126
+
```
+18
-36
provider_buildkite_test.go
+18
-36
provider_buildkite_test.go
···
155
155
}
156
156
157
157
// TestBuildkiteSpawnWorkflowConfig pins the YAML → create-build
158
-
// translation: pipeline + org from YAML pick the request URL,
159
-
// message/env/meta_data come through, and the trigger's PR target
160
-
// branch lands as `pull_request_base_branch`. Together these cover
161
-
// the "smuggle Buildkite parameters through workflow YAML" path.
158
+
// translation: `tack.buildkite.{pipeline,org}` pick the request URL,
159
+
// `clean_checkout` flows through, and the trigger's PR target
160
+
// branch lands as `pull_request_base_branch` automatically.
162
161
func TestBuildkiteSpawnWorkflowConfig(t *testing.T) {
163
162
type captured struct {
164
163
path string
···
175
174
p, _, _, _ := newBuildkiteTestProvider(t, buildkite.WebhookModeToken, "s", bk)
176
175
177
176
raw := strings.Join([]string{
178
-
"pipeline: workflow-pipe",
179
-
"org: workflow-org",
180
-
"message: smuggled message",
181
-
"env:",
182
-
" CUSTOM: value",
183
-
"meta_data:",
184
-
" custom: meta",
185
-
"clean_checkout: true",
186
-
"author:",
187
-
" name: Author",
188
-
" email: a@example.com",
177
+
"tack:",
178
+
" buildkite:",
179
+
" pipeline: workflow-pipe",
180
+
" org: workflow-org",
181
+
" clean_checkout: true",
189
182
}, "\n") + "\n"
190
183
191
184
trigger := &tangled.Pipeline_TriggerMetadata{
···
208
201
if got.body.Commit != "deadbeef" || got.body.Branch != "feature" {
209
202
t.Fatalf("commit/branch = %q/%q", got.body.Commit, got.body.Branch)
210
203
}
211
-
if got.body.Message != "smuggled message" {
212
-
t.Fatalf("message = %q", got.body.Message)
213
-
}
214
-
if got.body.Env["CUSTOM"] != "value" {
215
-
t.Fatalf("env[CUSTOM] missing: %+v", got.body.Env)
216
-
}
217
-
// tack defaults must still be present (user keys merge,
218
-
// don't replace).
204
+
// tack-managed env/meta still present.
219
205
if got.body.Env["TACK_WORKFLOW"] != "ci.yml" {
220
206
t.Fatalf("env[TACK_WORKFLOW] missing: %+v", got.body.Env)
221
207
}
222
-
if got.body.MetaData["custom"] != "meta" ||
223
-
got.body.MetaData[bkMetaWorkflow] != "ci.yml" {
224
-
t.Fatalf("meta_data wrong: %+v", got.body.MetaData)
208
+
if got.body.MetaData[bkMetaWorkflow] != "ci.yml" {
209
+
t.Fatalf("meta_data missing identity tuple: %+v", got.body.MetaData)
225
210
}
226
211
if !got.body.CleanCheckout {
227
212
t.Fatalf("clean_checkout not set")
228
213
}
229
-
// IgnorePipelineBranchFilters defaults to true (see
230
-
// workflowConfig comment).
214
+
// IgnorePipelineBranchFilters is hard-coded true; see
215
+
// buildCreateRequest comment.
231
216
if !got.body.IgnorePipelineBranchFilters {
232
-
t.Fatalf("ignore_pipeline_branch_filters not defaulted to true")
217
+
t.Fatalf("ignore_pipeline_branch_filters not on")
233
218
}
234
219
if got.body.PullRequestBaseBranch != "main" {
235
220
t.Fatalf("pr base branch = %q; want main",
236
221
got.body.PullRequestBaseBranch)
237
222
}
238
-
if got.body.Author == nil || got.body.Author.Email != "a@example.com" {
239
-
t.Fatalf("author = %+v", got.body.Author)
240
-
}
241
223
case <-time.After(2 * time.Second):
242
224
t.Fatal("CreateBuild not called")
243
225
}
244
226
}
245
227
246
228
// TestBuildkiteSpawnInvalidYAML proves a workflow without the
247
-
// required `pipeline` field is skipped — no API call, no DB row, no
248
-
// status. A misconfigured workflow shouldn't be silently swept onto
249
-
// some default pipeline.
229
+
// required `tack.buildkite.pipeline` field is skipped — no API
230
+
// call, no DB row, no status. A misconfigured workflow shouldn't
231
+
// be silently swept onto some default pipeline.
250
232
func TestBuildkiteSpawnInvalidYAML(t *testing.T) {
251
233
called := false
252
234
bk := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
···
265
247
266
248
time.Sleep(50 * time.Millisecond)
267
249
if called {
268
-
t.Fatal("CreateBuild called for workflow missing pipeline")
250
+
t.Fatal("CreateBuild called for workflow missing tack.buildkite.pipeline")
269
251
}
270
252
rows, _ := st.EventsAfter(context.Background(), 0)
271
253
if len(rows) != 0 {