forked from
mitchellh.com/tack
Stitch any CI into Tangled
better buildkite schema
+531
-112
+184
-7
README.md
+184
-7
README.md
···
37
37
38
38
## Configuration
39
39
40
+
Core configuration controls how tack talks to Tangled. Provider-specific
41
+
configuration (e.g. Buildkite) lives in its own section below.
42
+
40
43
### Required
41
44
42
45
| Env var | Description |
···
53
56
| `TACK_JETSTREAM_URL` | Tangled Jetstream WebSocket URL |
54
57
| `TACK_DEV` | Use `ws://` for knot event-streams (any non-empty value) |
55
58
56
-
### Buildkite
59
+
When no provider is configured, tack runs an in-process fake provider
60
+
that's useful for exercising the jetstream → knot → `/events` flow
61
+
locally without a real CI account.
62
+
63
+
## Buildkite
64
+
65
+
[Buildkite](https://buildkite.com) is the primary provider tack
66
+
supports today. In Buildkite mode, every Tangled pipeline trigger
67
+
fans out into one Buildkite build per workflow on the pipeline that
68
+
workflow names; build state flows back to Tangled via Buildkite's
69
+
notification webhooks.
70
+
71
+
### How it fits together
72
+
73
+
```
74
+
sh.tangled.pipeline Buildkite
75
+
trigger record ──▶ tack ──▶ Create Build ─┐
76
+
│
77
+
/webhooks/buildkite ◀──── notification ◀─────┘
78
+
│
79
+
▼
80
+
sh.tangled.pipeline.status (broadcast on /events)
81
+
```
82
+
83
+
* **Spawn:** for each workflow on a pipeline trigger, tack POSTs to
84
+
`/v2/organizations/<org>/pipelines/<slug>/builds`. Both `<org>` and
85
+
`<slug>` come from the workflow's YAML body (see
86
+
[Configuring your workflows](#configuring-your-workflows)).
87
+
* **Track:** tack persists the resulting `(build_uuid → knot, rkey,
88
+
workflow)` mapping in its local SQLite store so it can later
89
+
resolve incoming webhooks back to the originating Tangled
90
+
pipeline.
91
+
* **Report:** Buildkite delivers `build.*` events to
92
+
`POST /webhooks/buildkite`. tack authenticates each request,
93
+
translates the Buildkite state into a Tangled status, and
94
+
broadcasts a `sh.tangled.pipeline.status` record on `/events`.
95
+
96
+
### Setting up Buildkite
97
+
98
+
These steps happen once on the Buildkite side, before tack can talk
99
+
to it.
100
+
101
+
#### 1. Create one or more pipelines
102
+
103
+
Each Tangled workflow targets exactly one Buildkite pipeline by
104
+
slug. There's no requirement that pipelines map 1:1 to workflows —
105
+
many users point every workflow at a single pipeline whose
106
+
`pipeline.yml` does `pipeline upload some-file-${TACK_WORKFLOW}.yml`,
107
+
keeping all the per-workflow logic in the repo rather than in
108
+
Buildkite's UI.
109
+
110
+
In your Buildkite org, **Pipelines → New pipeline**:
111
+
112
+
* Repository: any URL (the agent only needs to be able to clone it).
113
+
* Steps: a minimal `pipeline upload` is usually enough — tack passes
114
+
the workflow name through `$TACK_WORKFLOW` so you can branch on
115
+
it.
116
+
117
+
Note the pipeline slug from the URL
118
+
(`https://buildkite.com/<org>/<pipeline-slug>`); your workflow YAML
119
+
will reference it.
120
+
121
+
#### 2. Create an API access token
122
+
123
+
Tack uses a single API token to create builds, list jobs, and fetch
124
+
logs. Generate one at
125
+
<https://buildkite.com/user/api-access-tokens> with these scopes:
126
+
127
+
| Scope | Used for |
128
+
| ------------------- | ------------------------------------------------- |
129
+
| `read_organizations`| Sanity-checking the configured org slug |
130
+
| `write_builds` | `POST .../builds` when a Tangled trigger arrives |
131
+
| `read_builds` | Resolving build → jobs for the `/logs` endpoint |
132
+
| `read_build_logs` | Streaming job logs back to the Tangled appview |
133
+
134
+
Restrict the token to the specific organization(s) tack will spawn
135
+
into.
136
+
137
+
#### 3. Configure a notification webhook
138
+
139
+
Builds report their state back to tack through Buildkite's
140
+
notification service.
141
+
142
+
In your Buildkite org, **Settings → Notification Services → Add →
143
+
Webhook**:
144
+
145
+
* **Webhook URL:** `https://<your-tack-host>/webhooks/buildkite`
146
+
* **Token / Secret:** any high-entropy string. You'll set the same
147
+
value in `TACK_BUILDKITE_WEBHOOK_SECRET`.
148
+
* **Events:** `build.scheduled`, `build.running`, `build.finished`
149
+
(job-level events are ignored).
150
+
* **Pipelines:** the pipelines tack will fire builds on.
151
+
152
+
Buildkite supports two header schemes for authenticating webhooks;
153
+
tack supports both:
57
154
58
-
Setting `TACK_BUILDKITE_TOKEN` enables Buildkite mode; when unset, tack
59
-
runs the in-process fake provider for local development. When
60
-
Buildkite mode is enabled, every other variable in this section is
155
+
| Header scheme | `TACK_BUILDKITE_WEBHOOK_MODE` | Notes |
156
+
| ----------------------- | ----------------------------- | -------------------------------------------- |
157
+
| `X-Buildkite-Token` | `token` (default) | Secret is sent verbatim in the header |
158
+
| `X-Buildkite-Signature` | `signature` | HMAC-SHA256 of `<timestamp>.<body>`; safer |
159
+
160
+
Pick `signature` if the notification setting offers it — it doesn't
161
+
expose the secret on the wire.
162
+
163
+
### Configuring tack
164
+
165
+
Setting `TACK_BUILDKITE_TOKEN` is the master switch that puts tack
166
+
into Buildkite mode. The other variables in this section are then
61
167
required.
62
168
63
169
| Env var | Description |
64
170
| ------------------------------- | ------------------------------------------------------------------------------ |
65
171
| `TACK_BUILDKITE_TOKEN` | Buildkite API token (enables Buildkite mode) |
66
-
| `TACK_BUILDKITE_ORG` | Buildkite organization slug |
67
-
| `TACK_BUILDKITE_PIPELINE` | Buildkite pipeline slug to fire builds on |
172
+
| `TACK_BUILDKITE_ORG` | Default Buildkite organization slug (workflows may override via YAML) |
68
173
| `TACK_BUILDKITE_WEBHOOK_SECRET` | Shared secret for `/webhooks/buildkite` auth |
69
-
| `TACK_BUILDKITE_WEBHOOK_MODE` | `token` (default) or `signature` — must match Buildkite's notification setting |
174
+
| `TACK_BUILDKITE_WEBHOOK_MODE` | `token` (default) or `signature` — must match the notification service |
175
+
176
+
The pipeline a workflow runs against is **not** an environment
177
+
variable. It lives inside the workflow YAML so each repo can target
178
+
its own pipeline without an operator round-trip.
179
+
180
+
### Configuring your workflows
181
+
182
+
A Tangled workflow's `raw` body is parsed by tack as YAML. Only
183
+
`pipeline` is required — every other field is an optional override
184
+
or extension of what the trigger metadata already provides:
185
+
186
+
```yaml
187
+
# Required: which Buildkite pipeline this workflow fires.
188
+
pipeline: my-pipeline-slug
189
+
190
+
# Optional: org override. Defaults to TACK_BUILDKITE_ORG. The API
191
+
# token must have access to whichever org you target.
192
+
org: another-org
193
+
194
+
# Optional: human-readable build message (default: "tangled: <name>").
195
+
message: "Custom build message"
196
+
197
+
# Optional: pin the commit/branch tack would otherwise derive from
198
+
# the trigger. Useful for manual triggers (which carry no commit).
199
+
commit: abcdef0123
200
+
branch: main
201
+
202
+
# Optional: extra env + meta_data merged on top of tack's defaults
203
+
# (see "What tack injects into every build" below).
204
+
env:
205
+
CUSTOM_VAR: value
206
+
meta_data:
207
+
custom-key: value
208
+
209
+
# Optional: forwarded verbatim to the Buildkite create-build API.
210
+
clean_checkout: true
211
+
ignore_pipeline_branch_filters: true # default: true
212
+
author:
213
+
name: "Author Name"
214
+
email: "author@example.com"
215
+
```
216
+
217
+
When the trigger is a pull request, tack auto-populates Buildkite's
218
+
`pull_request_base_branch` from the PR target so step-level branch
219
+
filters work without extra config.
220
+
221
+
#### What tack injects into every build
222
+
223
+
Regardless of what the workflow YAML adds on top, tack always
224
+
provides the following so your Buildkite pipeline can recover the
225
+
Tangled identity of the build:
226
+
227
+
| Channel | Key | Value |
228
+
| ----------- | -------------------- | ---------------------------------------- |
229
+
| `env` | `TACK_KNOT` | knot hostname the pipeline came from |
230
+
| `env` | `TACK_PIPELINE_RKEY` | rkey of the originating pipeline record |
231
+
| `env` | `TACK_WORKFLOW` | workflow name (typically a YAML filename) |
232
+
| `env` | `TACK_WORKFLOW_RAW` | the workflow's raw YAML body |
233
+
| `meta_data` | `tack:knot` | same as `TACK_KNOT` |
234
+
| `meta_data` | `tack:pipeline_rkey` | same as `TACK_PIPELINE_RKEY` |
235
+
| `meta_data` | `tack:workflow` | same as `TACK_WORKFLOW` |
236
+
237
+
A common pattern is for the Buildkite pipeline's root step to do a
238
+
`pipeline upload` against a workflow-specific YAML file based on
239
+
`$TACK_WORKFLOW`, e.g.:
240
+
241
+
```yaml
242
+
# Buildkite pipeline.yml
243
+
steps:
244
+
- label: ":pipeline: dispatch ${TACK_WORKFLOW}"
245
+
command: "buildkite-agent pipeline upload .buildkite/${TACK_WORKFLOW}"
246
+
```
+1
-1
go.mod
+1
-1
go.mod
···
7
7
github.com/charmbracelet/log v1.0.0
8
8
github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674
9
9
github.com/mattn/go-sqlite3 v1.14.44
10
+
go.yaml.in/yaml/v2 v2.4.3
10
11
tangled.org/core v1.13.0-alpha
11
12
)
12
13
···
50
51
github.com/whyrusleeping/cbor-gen v0.3.1 // indirect
51
52
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect
52
53
go.uber.org/atomic v1.11.0 // indirect
53
-
go.yaml.in/yaml/v2 v2.4.3 // indirect
54
54
golang.org/x/crypto v0.48.0 // indirect
55
55
golang.org/x/exp v0.0.0-20260112195511-716be5621a96 // indirect
56
56
golang.org/x/net v0.50.0 // indirect
+21
-12
internal/buildkite/buildkite.go
+21
-12
internal/buildkite/buildkite.go
···
39
39
// maps it onto its own ErrLogsNotFound for the /logs handler).
40
40
var ErrNotFound = errors.New("buildkite: not found")
41
41
42
-
// Client is a thin wrapper around net/http carrying API credentials
43
-
// + organization context so call sites don't repeat them. Safe for
44
-
// concurrent use; the embedded http.Client is goroutine-safe.
42
+
// Client is a thin wrapper around net/http carrying API credentials.
43
+
// Organisation slug is *not* held on the client — every call takes
44
+
// its target org explicitly so a single client can address multiple
45
+
// orgs the token has access to. Safe for concurrent use; the
46
+
// embedded http.Client is goroutine-safe.
45
47
type Client struct {
46
48
http *http.Client
47
49
token string
48
-
org string
49
50
}
50
51
51
52
// NewClient builds a Client with sensible defaults. The 30s timeout
52
53
// covers individual requests, not the whole client lifetime —
53
54
// long-poll-style endpoints aren't used here, so a generous-but-bounded
54
55
// per-request timeout is the right default.
55
-
func NewClient(token, org string) *Client {
56
+
func NewClient(token string) *Client {
56
57
return &Client{
57
58
http: &http.Client{Timeout: 30 * time.Second},
58
59
token: token,
59
-
org: org,
60
60
}
61
61
}
62
62
···
95
95
// fields callers actively use are exposed — the upstream API accepts
96
96
// many more, but we'd just be passing through dead options.
97
97
//
98
-
// IgnorePipelineBranchFilters defaults to false on the wire (omitempty
99
-
// elides the zero value); callers that want pipeline-level branch
100
-
// filters bypassed should set it to true.
98
+
// IgnorePipelineBranchFilters / CleanCheckout default to false on the
99
+
// wire (omitempty elides the zero value); callers that want them
100
+
// turned on should set the field explicitly.
101
+
//
102
+
// PullRequestBaseBranch surfaces a Tangled PR trigger's target
103
+
// branch so Buildkite step filters keyed on the PR base behave the
104
+
// way users expect.
101
105
type CreateBuildRequest struct {
102
106
Commit string `json:"commit"`
103
107
Branch string `json:"branch"`
104
108
Message string `json:"message,omitempty"`
105
109
Env map[string]string `json:"env,omitempty"`
106
110
MetaData map[string]string `json:"meta_data,omitempty"`
111
+
CleanCheckout bool `json:"clean_checkout,omitempty"`
107
112
IgnorePipelineBranchFilters bool `json:"ignore_pipeline_branch_filters,omitempty"`
113
+
PullRequestBaseBranch string `json:"pull_request_base_branch,omitempty"`
108
114
}
109
115
110
116
// CreateBuild fires a build on the named pipeline. Returns the
···
117
123
// the caller's log.
118
124
func (c *Client) CreateBuild(
119
125
ctx context.Context,
126
+
org string,
120
127
pipelineSlug string,
121
128
req CreateBuildRequest,
122
129
) (*Build, error) {
···
126
133
}
127
134
128
135
url := fmt.Sprintf("%s/v2/organizations/%s/pipelines/%s/builds",
129
-
APIBase, c.org, pipelineSlug,
136
+
APIBase, org, pipelineSlug,
130
137
)
131
138
httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(body))
132
139
if err != nil {
···
162
169
// Returns ErrNotFound when Buildkite responds 404.
163
170
func (c *Client) GetBuild(
164
171
ctx context.Context,
172
+
org string,
165
173
pipelineSlug string,
166
174
buildNumber int64,
167
175
) (*Build, error) {
168
176
url := fmt.Sprintf("%s/v2/organizations/%s/pipelines/%s/builds/%d",
169
-
APIBase, c.org, pipelineSlug, buildNumber,
177
+
APIBase, org, pipelineSlug, buildNumber,
170
178
)
171
179
httpReq, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
172
180
if err != nil {
···
205
213
// job that hasn't started yet — it has no log to serve).
206
214
func (c *Client) GetJobLog(
207
215
ctx context.Context,
216
+
org string,
208
217
pipelineSlug string,
209
218
buildNumber int64,
210
219
jobID string,
211
220
) (string, error) {
212
221
url := fmt.Sprintf("%s/v2/organizations/%s/pipelines/%s/builds/%d/jobs/%s/log",
213
-
APIBase, c.org, pipelineSlug, buildNumber, jobID,
222
+
APIBase, org, pipelineSlug, buildNumber, jobID,
214
223
)
215
224
httpReq, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
216
225
if err != nil {
+8
-8
internal/buildkite/buildkite_test.go
+8
-8
internal/buildkite/buildkite_test.go
···
121
121
APIBase = srv.URL
122
122
defer func() { APIBase = prev }()
123
123
124
-
c := NewClient("tok", "myorg")
125
-
build, err := c.CreateBuild(context.Background(), "mypipe", CreateBuildRequest{
124
+
c := NewClient("tok")
125
+
build, err := c.CreateBuild(context.Background(), "myorg", "mypipe", CreateBuildRequest{
126
126
Commit: "abc",
127
127
Branch: "main",
128
128
MetaData: map[string]string{"k": "v"},
···
150
150
APIBase = srv.URL
151
151
defer func() { APIBase = prev }()
152
152
153
-
c := NewClient("tok", "myorg")
154
-
_, err := c.CreateBuild(context.Background(), "mypipe", CreateBuildRequest{})
153
+
c := NewClient("tok")
154
+
_, err := c.CreateBuild(context.Background(), "myorg", "mypipe", CreateBuildRequest{})
155
155
if err == nil {
156
156
t.Fatal("expected error")
157
157
}
···
180
180
APIBase = srv.URL
181
181
defer func() { APIBase = prev }()
182
182
183
-
body, err := NewClient("tok", "myorg").
184
-
GetJobLog(context.Background(), "mypipe", 7, "job-1")
183
+
body, err := NewClient("tok").
184
+
GetJobLog(context.Background(), "myorg", "mypipe", 7, "job-1")
185
185
if err != nil {
186
186
t.Fatalf("GetJobLog: %v", err)
187
187
}
···
198
198
APIBase = srv.URL
199
199
defer func() { APIBase = prev }()
200
200
201
-
_, err := NewClient("tok", "myorg").
202
-
GetJobLog(context.Background(), "mypipe", 7, "job-1")
201
+
_, err := NewClient("tok").
202
+
GetJobLog(context.Background(), "myorg", "mypipe", 7, "job-1")
203
203
if err != ErrNotFound {
204
204
t.Fatalf("err = %v; want ErrNotFound", err)
205
205
}
+8
-9
main.go
+8
-9
main.go
···
37
37
// (useful for local development against a real Tangled
38
38
// jetstream); when set, the other Buildkite fields are
39
39
// required and tack will refuse to start without them.
40
+
//
41
+
// BuildkiteOrg is the *default* org used when a workflow YAML
42
+
// doesn't specify one of its own. The pipeline a workflow runs
43
+
// against is no longer global — it's pulled from the workflow
44
+
// body itself (see workflowConfig in provider_buildkite.go).
40
45
BuildkiteToken string
41
46
BuildkiteOrg string
42
-
BuildkitePipeline string
43
47
BuildkiteWebhookSecret string
44
48
BuildkiteWebhookMode buildkite.WebhookMode
45
49
}
···
54
58
Dev: os.Getenv("TACK_DEV") != "",
55
59
BuildkiteToken: os.Getenv("TACK_BUILDKITE_TOKEN"),
56
60
BuildkiteOrg: os.Getenv("TACK_BUILDKITE_ORG"),
57
-
BuildkitePipeline: os.Getenv("TACK_BUILDKITE_PIPELINE"),
58
61
BuildkiteWebhookSecret: os.Getenv("TACK_BUILDKITE_WEBHOOK_SECRET"),
59
62
BuildkiteWebhookMode: buildkite.WebhookMode(
60
63
envOr("TACK_BUILDKITE_WEBHOOK_MODE", string(buildkite.WebhookModeToken)),
···
84
87
if cfg.BuildkiteToken != "" {
85
88
if cfg.BuildkiteOrg == "" {
86
89
return cfg, errors.New("TACK_BUILDKITE_ORG is required when TACK_BUILDKITE_TOKEN is set")
87
-
}
88
-
if cfg.BuildkitePipeline == "" {
89
-
return cfg, errors.New("TACK_BUILDKITE_PIPELINE is required when TACK_BUILDKITE_TOKEN is set")
90
90
}
91
91
if cfg.BuildkiteWebhookSecret == "" {
92
92
return cfg, errors.New("TACK_BUILDKITE_WEBHOOK_SECRET is required when TACK_BUILDKITE_TOKEN is set")
···
176
176
if cfg.BuildkiteToken != "" {
177
177
bkProvider = newBuildkiteProvider(
178
178
br, st,
179
-
buildkite.NewClient(cfg.BuildkiteToken, cfg.BuildkiteOrg),
180
-
cfg.BuildkitePipeline,
179
+
buildkite.NewClient(cfg.BuildkiteToken),
180
+
cfg.BuildkiteOrg,
181
181
cfg.BuildkiteWebhookSecret,
182
182
cfg.BuildkiteWebhookMode,
183
183
logger,
184
184
)
185
185
provider = bkProvider
186
186
logger.Info("buildkite provider enabled",
187
-
"org", cfg.BuildkiteOrg,
188
-
"pipeline", cfg.BuildkitePipeline,
187
+
"default_org", cfg.BuildkiteOrg,
189
188
"webhook_mode", cfg.BuildkiteWebhookMode,
190
189
)
191
190
} else {
+185
-71
provider_buildkite.go
+185
-71
provider_buildkite.go
···
9
9
// Spawn time and publishes a sh.tangled.pipeline.status record on
10
10
// the in-process broker.
11
11
//
12
-
// Only one Buildkite pipeline is used per spindle (TACK_BUILDKITE_PIPELINE).
13
-
// Every Tangled workflow runs as a build on that single pipeline, with
14
-
// the workflow identity plumbed through env + meta_data. The operator
15
-
// configures their Buildkite pipeline to read those env vars and
16
-
// dispatch accordingly (e.g. via `pipeline upload`). Mapping every
17
-
// Tangled workflow to its own Buildkite pipeline would force operators
18
-
// to provision Buildkite resources for each workflow file in every
19
-
// repo that points at the spindle — friction we don't want to impose.
12
+
// The Buildkite *pipeline slug* a workflow targets is carried inside
13
+
// the workflow's YAML body (Pipeline_Workflow.Raw), not configured
14
+
// globally on the spindle. That keeps tack a thin translator: the
15
+
// repo author decides which Buildkite pipeline runs each Tangled
16
+
// workflow without an operator round-trip. See workflowConfig below
17
+
// for the supported YAML schema.
20
18
21
19
import (
22
20
"context"
···
28
26
"strings"
29
27
"time"
30
28
29
+
"go.yaml.in/yaml/v2"
31
30
"tangled.org/core/api/tangled"
32
31
33
32
"github.com/mitchellh/tack/internal/buildkite"
···
44
43
bkMetaWorkflow = "tack:workflow"
45
44
)
46
45
46
+
// workflowConfig is the tack-flavoured schema we expect inside each
47
+
// Tangled workflow's Raw YAML body. Only the Buildkite `pipeline`
48
+
// slug is required; everything else is optional. Fields nest under
49
+
// `tack: { buildkite: ... }` so the workflow YAML can grow other
50
+
// top-level keys (Tangled's own scheduling fields, future provider
51
+
// blocks) without colliding with our namespace.
52
+
//
53
+
// Fields map onto the Buildkite REST "Create a build" request
54
+
// properties documented at
55
+
// https://buildkite.com/docs/apis/rest-api/builds#create-a-build
56
+
// (see also the comment block on buildkite.CreateBuildRequest). We
57
+
// expose only the small subset users genuinely need to override —
58
+
// trigger metadata supplies commit/branch, and tack supplies the
59
+
// identity env+meta the webhook handler relies on, so there's no
60
+
// reason to let users re-specify those.
61
+
type workflowConfig struct {
62
+
Tack tackConfig `yaml:"tack"`
63
+
}
64
+
65
+
// tackConfig is the per-provider block under the top-level `tack:`
66
+
// key. Right now the only nested provider is Buildkite.
67
+
type tackConfig struct {
68
+
Buildkite buildkiteConfig `yaml:"buildkite"`
69
+
}
70
+
71
+
// buildkiteConfig is the Buildkite-specific subset of workflowConfig.
72
+
//
73
+
// `org` lets a workflow target a Buildkite organisation other than
74
+
// the spindle's default — useful when one tack instance fronts
75
+
// multiple orgs. The configured API token must have access to that
76
+
// org or the build creation request will 401/403; we surface that
77
+
// error verbatim rather than guessing.
78
+
//
79
+
// `clean_checkout` is forwarded verbatim to Buildkite. CleanCheckout
80
+
// is a *bool so omitting it leaves Buildkite's own default in place
81
+
// instead of always shipping `false`.
82
+
type buildkiteConfig struct {
83
+
Pipeline string `yaml:"pipeline"`
84
+
Org string `yaml:"org"`
85
+
CleanCheckout *bool `yaml:"clean_checkout"`
86
+
}
87
+
88
+
// parseWorkflowConfig decodes a workflow YAML body into workflowConfig.
89
+
// An empty body is treated as a structural error so spawnWorkflow can
90
+
// short-circuit cleanly: a workflow with no body has nothing for tack
91
+
// to do anyway.
92
+
func parseWorkflowConfig(raw string) (*buildkiteConfig, error) {
93
+
if strings.TrimSpace(raw) == "" {
94
+
return nil, errors.New("workflow body is empty")
95
+
}
96
+
var cfg workflowConfig
97
+
if err := yaml.Unmarshal([]byte(raw), &cfg); err != nil {
98
+
return nil, fmt.Errorf("parse workflow yaml: %w", err)
99
+
}
100
+
bk := cfg.Tack.Buildkite
101
+
if bk.Pipeline == "" {
102
+
return nil, errors.New("workflow yaml: `tack.buildkite.pipeline` is required")
103
+
}
104
+
return &bk, nil
105
+
}
106
+
47
107
// buildkiteProvider implements Provider.
48
108
//
49
109
// webhookSecret + webhookMode live on the provider rather than on
···
51
111
// "everything Buildkite-y": colocating the auth knob with the API
52
112
// client and the state translator keeps configuration drift to one
53
113
// place and makes the http.go side pure transport.
114
+
//
115
+
// defaultOrg is the Buildkite organisation the configured API token
116
+
// belongs to. Workflows may opt into a different org via their YAML
117
+
// `org` field; the API token then needs to be authorised against it.
54
118
type buildkiteProvider struct {
55
119
br *broker
56
120
st *store
57
121
log *slog.Logger
58
122
client *buildkite.Client
59
-
pipelineSlug string
123
+
defaultOrg string
60
124
webhookSecret string
61
125
webhookMode buildkite.WebhookMode
62
126
}
···
65
129
var _ Provider = (*buildkiteProvider)(nil)
66
130
67
131
// newBuildkiteProvider wires a provider to its Buildkite client and
68
-
// to the broker it publishes pipeline.status records on. pipelineSlug
69
-
// is the Buildkite pipeline that all builds get fired on (see file
70
-
// header for why there's only one). webhookSecret/webhookMode govern
71
-
// inbound /webhooks/buildkite request authentication.
132
+
// to the broker it publishes pipeline.status records on. defaultOrg
133
+
// is the org the API token authenticates against and the org used
134
+
// when a workflow doesn't specify its own. webhookSecret/webhookMode
135
+
// govern inbound /webhooks/buildkite request authentication.
72
136
func newBuildkiteProvider(
73
137
br *broker,
74
138
st *store,
75
139
client *buildkite.Client,
76
-
pipelineSlug string,
140
+
defaultOrg string,
77
141
webhookSecret string,
78
142
webhookMode buildkite.WebhookMode,
79
143
log *slog.Logger,
···
83
147
st: st,
84
148
log: log.With("component", "provider", "kind", "buildkite"),
85
149
client: client,
86
-
pipelineSlug: pipelineSlug,
150
+
defaultOrg: defaultOrg,
87
151
webhookSecret: webhookSecret,
88
152
webhookMode: webhookMode,
89
153
}
···
111
175
}
112
176
113
177
// Spawn satisfies Provider. For each workflow it fires a separate
114
-
// Buildkite build off the configured pipeline so each workflow gets
115
-
// its own status timeline. The actual API call runs on a goroutine —
116
-
// CreateBuild is one HTTP round-trip, but we still want Spawn to be
117
-
// non-blocking per the interface contract.
178
+
// Buildkite build off the pipeline named in that workflow's YAML so
179
+
// each workflow gets its own status timeline. The actual API call
180
+
// runs on a goroutine — CreateBuild is one HTTP round-trip, but we
181
+
// still want Spawn to be non-blocking per the interface contract.
118
182
//
119
183
// On a successful create we persist the build UUID → (knot, rkey,
120
184
// workflow) mapping and publish a "pending" pipeline.status so the
···
134
198
return
135
199
}
136
200
137
-
// Derive build inputs once. Every workflow on this trigger
138
-
// targets the same commit/branch — only the workflow name
139
-
// varies between the per-workflow goroutines below.
140
-
commit, branch := triggerCommitAndBranch(trigger)
141
-
if commit == "" {
142
-
// Buildkite's create-build API requires a commit; we'd
143
-
// rather log loudly and skip than fire builds on "HEAD"
144
-
// and silently get whatever main happens to look like.
145
-
p.log.Error("trigger has no commit; refusing to spawn",
146
-
"knot", knot, "rkey", pipelineRkey,
147
-
)
148
-
return
149
-
}
150
-
151
201
for _, wf := range workflows {
152
202
if wf == nil || wf.Name == "" {
153
203
continue
154
204
}
155
205
wf := wf
156
-
go p.spawnWorkflow(ctx, knot, pipelineRkey, commit, branch, wf)
206
+
go p.spawnWorkflow(ctx, knot, pipelineRkey, trigger, wf)
157
207
}
158
208
}
159
209
···
166
216
ctx context.Context,
167
217
knot string,
168
218
pipelineRkey string,
169
-
commit string,
170
-
branch string,
219
+
trigger *tangled.Pipeline_TriggerMetadata,
171
220
wf *tangled.Pipeline_Workflow,
172
221
) {
173
222
logger := p.log.With(
···
176
225
"workflow", wf.Name,
177
226
)
178
227
179
-
pipelineURI := pipelineATURI(knot, pipelineRkey)
180
-
meta := map[string]string{
181
-
bkMetaKnot: knot,
182
-
bkMetaPipelineRkey: pipelineRkey,
183
-
bkMetaWorkflow: wf.Name,
228
+
cfg, err := parseWorkflowConfig(wf.Raw)
229
+
if err != nil {
230
+
// Bad workflow YAML is a user-facing config error: log it
231
+
// loudly and skip. Firing a build off some default would
232
+
// be more confusing than doing nothing.
233
+
logger.Error("invalid workflow config; refusing to spawn", "err", err)
234
+
return
184
235
}
185
-
env := envFromTuple(knot, pipelineRkey, wf)
236
+
logger = logger.With("pipeline", cfg.Pipeline)
186
237
187
-
req := buildkite.CreateBuildRequest{
188
-
Commit: commit,
189
-
Branch: branch,
190
-
Message: fmt.Sprintf("tangled: %s", wf.Name),
191
-
Env: env,
192
-
MetaData: meta,
193
-
IgnorePipelineBranchFilters: true,
238
+
req, err := p.buildCreateRequest(cfg, trigger, knot, pipelineRkey, wf)
239
+
if err != nil {
240
+
logger.Error("build create request", "err", err)
241
+
return
194
242
}
195
243
196
-
build, err := p.client.CreateBuild(ctx, p.pipelineSlug, req)
244
+
org := cfg.Org
245
+
if org == "" {
246
+
org = p.defaultOrg
247
+
}
248
+
249
+
build, err := p.client.CreateBuild(ctx, org, cfg.Pipeline, req)
197
250
if err != nil {
198
-
logger.Error("create buildkite build", "err", err)
251
+
logger.Error("create buildkite build", "err", err, "org", org)
199
252
return
200
253
}
201
254
logger.Info("buildkite build created",
202
255
"build_uuid", build.ID,
203
256
"build_number", build.Number,
204
257
"web_url", build.WebURL,
258
+
"org", org,
205
259
)
206
260
261
+
pipelineURI := pipelineATURI(knot, pipelineRkey)
207
262
if err := p.st.InsertBuildkiteBuild(ctx, BuildkiteBuildRef{
208
263
BuildUUID: build.ID,
209
264
BuildNumber: build.Number,
210
-
PipelineSlug: p.pipelineSlug,
265
+
PipelineSlug: cfg.Pipeline,
211
266
Knot: knot,
212
267
PipelineRkey: pipelineRkey,
213
268
Workflow: wf.Name,
···
234
289
}
235
290
}
236
291
292
+
// buildCreateRequest folds the parsed workflow config and the
293
+
// Tangled trigger metadata into a single Buildkite create-build
294
+
// payload. Trigger metadata supplies commit/branch; the workflow
295
+
// YAML supplies the Buildkite routing knobs (pipeline/org) and the
296
+
// small handful of build options we expose.
297
+
//
298
+
// `ignore_pipeline_branch_filters` is hard-coded to true: Tangled
299
+
// refs frequently don't match arbitrary Buildkite pipeline branch
300
+
// filters, and a build silently dropped at create time is a worse
301
+
// failure mode than running one we shouldn't have. Users wanting
302
+
// the filter back are expected to drop the filter on the Buildkite
303
+
// pipeline itself.
304
+
//
305
+
// Returns an error when the trigger lacks a commit — Buildkite's
306
+
// API requires one and we'd rather log+skip than fire a build that
307
+
// resolves to "whatever main happens to be".
308
+
func (p *buildkiteProvider) buildCreateRequest(
309
+
cfg *buildkiteConfig,
310
+
trigger *tangled.Pipeline_TriggerMetadata,
311
+
knot, pipelineRkey string,
312
+
wf *tangled.Pipeline_Workflow,
313
+
) (buildkite.CreateBuildRequest, error) {
314
+
commit, branch := triggerCommitAndBranch(trigger)
315
+
if commit == "" {
316
+
return buildkite.CreateBuildRequest{}, errors.New(
317
+
"trigger has no commit",
318
+
)
319
+
}
320
+
321
+
cleanCheckout := false
322
+
if cfg.CleanCheckout != nil {
323
+
cleanCheckout = *cfg.CleanCheckout
324
+
}
325
+
326
+
req := buildkite.CreateBuildRequest{
327
+
Commit: commit,
328
+
Branch: branch,
329
+
Message: fmt.Sprintf("tangled: %s", wf.Name),
330
+
Env: envFromTuple(knot, pipelineRkey, wf),
331
+
MetaData: map[string]string{
332
+
bkMetaKnot: knot,
333
+
bkMetaPipelineRkey: pipelineRkey,
334
+
bkMetaWorkflow: wf.Name,
335
+
},
336
+
CleanCheckout: cleanCheckout,
337
+
IgnorePipelineBranchFilters: true,
338
+
}
339
+
340
+
// Auto-populate Buildkite's PR fields from the Tangled PR
341
+
// trigger when present. Buildkite doesn't get a PR number from
342
+
// us (Tangled doesn't surface one through the trigger), but
343
+
// the base branch alone is enough for `pull_request_base_branch`-
344
+
// gated step filters to work.
345
+
if trigger != nil && trigger.PullRequest != nil {
346
+
req.PullRequestBaseBranch = trigger.PullRequest.TargetBranch
347
+
}
348
+
349
+
return req, nil
350
+
}
351
+
237
352
// Logs satisfies Provider. We resolve the (knot, rkey, workflow)
238
353
// tuple to a Buildkite build via the store, fetch the current jobs
239
354
// list, then drain each job's plain-text log into the channel as one
···
256
371
) (<-chan LogLine, error) {
257
372
ref, err := p.st.LookupBuildkiteBuildByTuple(ctx, knot, pipelineRkey, workflow)
258
373
if err != nil {
259
-
return nil, fmt.Errorf("lookup build for logs: %w", err)
374
+
return nil, fmt.Errorf("lookup build mapping: %w", err)
260
375
}
261
376
if ref == nil {
262
377
return nil, ErrLogsNotFound
263
378
}
264
379
265
-
// Fresh fetch so we get the current job set, not whatever was
266
-
// returned at create time (when most jobs are still nil). The
267
-
// upstream's not-found is mapped to the Provider-shaped one
268
-
// here because the /logs handler only knows about ErrLogsNotFound.
269
-
build, err := p.client.GetBuild(ctx, ref.PipelineSlug, ref.BuildNumber)
380
+
// Resolve the org against which we should pull jobs/logs. We
381
+
// don't persist it on BuildkiteBuildRef today (the slug + token
382
+
// have always been enough); fall back to the provider default,
383
+
// which is correct for the common single-org install. A
384
+
// future migration can add a column when multi-org installs
385
+
// need it.
386
+
org := p.defaultOrg
387
+
388
+
build, err := p.client.GetBuild(ctx, org, ref.PipelineSlug, ref.BuildNumber)
270
389
if err != nil {
271
390
if errors.Is(err, buildkite.ErrNotFound) {
272
391
return nil, ErrLogsNotFound
273
392
}
274
-
return nil, fmt.Errorf("get build for logs: %w", err)
393
+
return nil, fmt.Errorf("get build: %w", err)
275
394
}
276
395
277
-
out := make(chan LogLine, 64)
396
+
out := make(chan LogLine, 32)
278
397
go func() {
279
398
defer close(out)
280
399
stepID := 0
281
400
for _, job := range build.Jobs {
282
-
// Only "script" jobs have agent-produced logs.
283
-
// Waiter / manual / trigger jobs have no body to
284
-
// fetch; skip them so we don't hit Buildkite with
285
-
// 404-bound requests.
286
401
if job.Type != "" && job.Type != "script" {
402
+
// Skip non-script jobs (waiter, manual,
403
+
// trigger). They have no log to fetch and
404
+
// surfacing empty steps just clutters the
405
+
// appview.
287
406
continue
288
407
}
289
-
290
408
name := job.Name
291
409
if name == "" {
292
-
name = job.ID
410
+
name = fmt.Sprintf("job %s", job.ID)
293
411
}
294
412
295
-
// Job-level start frame so the appview can bound
296
-
// timing per job.
297
413
if !sendLine(ctx, out, LogLine{
298
414
Kind: LogKindControl,
299
415
Time: time.Now(),
···
304
420
return
305
421
}
306
422
307
-
body, err := p.client.GetJobLog(ctx, ref.PipelineSlug, ref.BuildNumber, job.ID)
423
+
body, err := p.client.GetJobLog(ctx, org, ref.PipelineSlug, ref.BuildNumber, job.ID)
308
424
if err != nil {
309
425
p.log.Debug("fetch job log",
310
426
"err", err,
···
512
628
return trigger.Push.NewSha, refToBranch(trigger.Push.Ref)
513
629
case trigger.PullRequest != nil:
514
630
// PRs build the source commit on the source branch.
515
-
// Buildkite's pipeline can opt into PR-aware behaviour
516
-
// via pull_request_id (not currently plumbed through).
517
631
return trigger.PullRequest.SourceSha, trigger.PullRequest.SourceBranch
518
632
default:
519
633
// Manual triggers and any future kinds: fall back to the
+124
-4
provider_buildkite_test.go
+124
-4
provider_buildkite_test.go
···
47
47
logger := slog.Default()
48
48
p := newBuildkiteProvider(
49
49
br, st,
50
-
buildkite.NewClient("tok", "myorg"),
51
-
"mypipe",
50
+
buildkite.NewClient("tok"),
51
+
"myorg",
52
52
secret, mode,
53
53
logger,
54
54
)
···
74
74
},
75
75
}
76
76
workflows := []*tangled.Pipeline_Workflow{
77
-
{Name: "test.yml", Raw: "steps:\n - run: true\n"},
77
+
{Name: "test.yml", Raw: "tack:\n buildkite:\n pipeline: mypipe\n"},
78
78
}
79
79
80
80
p.Spawn(context.Background(), "knot.example.com", "rkey-1", trigger, workflows)
···
139
139
140
140
p.Spawn(context.Background(), "knot.example.com", "rkey-1",
141
141
&tangled.Pipeline_TriggerMetadata{Manual: &tangled.Pipeline_ManualTriggerData{}},
142
-
[]*tangled.Pipeline_Workflow{{Name: "test.yml"}},
142
+
[]*tangled.Pipeline_Workflow{{Name: "test.yml",
143
+
Raw: "tack:\n buildkite:\n pipeline: mypipe\n"}},
143
144
)
144
145
145
146
// Give any rogue goroutine a moment.
146
147
time.Sleep(50 * time.Millisecond)
147
148
if called {
148
149
t.Fatal("CreateBuild called despite missing commit")
150
+
}
151
+
rows, _ := st.EventsAfter(context.Background(), 0)
152
+
if len(rows) != 0 {
153
+
t.Fatalf("got %d events, want 0", len(rows))
154
+
}
155
+
}
156
+
157
+
// TestBuildkiteSpawnWorkflowConfig pins the YAML → create-build
158
+
// translation: pipeline + org from YAML pick the request URL,
159
+
// message/env/meta_data come through, and the trigger's PR target
160
+
// branch lands as `pull_request_base_branch`. Together these cover
161
+
// the "smuggle Buildkite parameters through workflow YAML" path.
162
+
func TestBuildkiteSpawnWorkflowConfig(t *testing.T) {
163
+
type captured struct {
164
+
path string
165
+
body buildkite.CreateBuildRequest
166
+
}
167
+
gotCh := make(chan captured, 1)
168
+
bk := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
169
+
var body buildkite.CreateBuildRequest
170
+
_ = json.NewDecoder(r.Body).Decode(&body)
171
+
gotCh <- captured{path: r.URL.Path, body: body}
172
+
w.WriteHeader(http.StatusCreated)
173
+
_ = json.NewEncoder(w).Encode(buildkite.Build{ID: "uuid-9", Number: 9})
174
+
})
175
+
p, _, _, _ := newBuildkiteTestProvider(t, buildkite.WebhookModeToken, "s", bk)
176
+
177
+
raw := strings.Join([]string{
178
+
"pipeline: workflow-pipe",
179
+
"org: workflow-org",
180
+
"message: smuggled message",
181
+
"env:",
182
+
" CUSTOM: value",
183
+
"meta_data:",
184
+
" custom: meta",
185
+
"clean_checkout: true",
186
+
"author:",
187
+
" name: Author",
188
+
" email: a@example.com",
189
+
}, "\n") + "\n"
190
+
191
+
trigger := &tangled.Pipeline_TriggerMetadata{
192
+
PullRequest: &tangled.Pipeline_PullRequestTriggerData{
193
+
SourceSha: "deadbeef",
194
+
SourceBranch: "feature",
195
+
TargetBranch: "main",
196
+
},
197
+
}
198
+
199
+
p.Spawn(context.Background(), "knot.example.com", "rkey-x", trigger,
200
+
[]*tangled.Pipeline_Workflow{{Name: "ci.yml", Raw: raw}})
201
+
202
+
select {
203
+
case got := <-gotCh:
204
+
// URL must reflect YAML org + pipeline.
205
+
if !strings.Contains(got.path, "/organizations/workflow-org/pipelines/workflow-pipe/") {
206
+
t.Fatalf("path = %q", got.path)
207
+
}
208
+
if got.body.Commit != "deadbeef" || got.body.Branch != "feature" {
209
+
t.Fatalf("commit/branch = %q/%q", got.body.Commit, got.body.Branch)
210
+
}
211
+
if got.body.Message != "smuggled message" {
212
+
t.Fatalf("message = %q", got.body.Message)
213
+
}
214
+
if got.body.Env["CUSTOM"] != "value" {
215
+
t.Fatalf("env[CUSTOM] missing: %+v", got.body.Env)
216
+
}
217
+
// tack defaults must still be present (user keys merge,
218
+
// don't replace).
219
+
if got.body.Env["TACK_WORKFLOW"] != "ci.yml" {
220
+
t.Fatalf("env[TACK_WORKFLOW] missing: %+v", got.body.Env)
221
+
}
222
+
if got.body.MetaData["custom"] != "meta" ||
223
+
got.body.MetaData[bkMetaWorkflow] != "ci.yml" {
224
+
t.Fatalf("meta_data wrong: %+v", got.body.MetaData)
225
+
}
226
+
if !got.body.CleanCheckout {
227
+
t.Fatalf("clean_checkout not set")
228
+
}
229
+
// IgnorePipelineBranchFilters defaults to true (see
230
+
// workflowConfig comment).
231
+
if !got.body.IgnorePipelineBranchFilters {
232
+
t.Fatalf("ignore_pipeline_branch_filters not defaulted to true")
233
+
}
234
+
if got.body.PullRequestBaseBranch != "main" {
235
+
t.Fatalf("pr base branch = %q; want main",
236
+
got.body.PullRequestBaseBranch)
237
+
}
238
+
if got.body.Author == nil || got.body.Author.Email != "a@example.com" {
239
+
t.Fatalf("author = %+v", got.body.Author)
240
+
}
241
+
case <-time.After(2 * time.Second):
242
+
t.Fatal("CreateBuild not called")
243
+
}
244
+
}
245
+
246
+
// TestBuildkiteSpawnInvalidYAML proves a workflow without the
247
+
// required `pipeline` field is skipped — no API call, no DB row, no
248
+
// status. A misconfigured workflow shouldn't be silently swept onto
249
+
// some default pipeline.
250
+
func TestBuildkiteSpawnInvalidYAML(t *testing.T) {
251
+
called := false
252
+
bk := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
253
+
called = true
254
+
})
255
+
p, st, _, _ := newBuildkiteTestProvider(t, buildkite.WebhookModeToken, "s", bk)
256
+
257
+
p.Spawn(context.Background(), "knot.example.com", "rkey-z",
258
+
&tangled.Pipeline_TriggerMetadata{
259
+
Push: &tangled.Pipeline_PushTriggerData{NewSha: "abc", Ref: "refs/heads/main"},
260
+
},
261
+
[]*tangled.Pipeline_Workflow{
262
+
{Name: "broken.yml", Raw: "steps:\n - run: true\n"},
263
+
},
264
+
)
265
+
266
+
time.Sleep(50 * time.Millisecond)
267
+
if called {
268
+
t.Fatal("CreateBuild called for workflow missing pipeline")
149
269
}
150
270
rows, _ := st.EventsAfter(context.Background(), 0)
151
271
if len(rows) != 0 {