forked from
ewancroft.uk/tangled-sync
Move from GitHub to Tangled
Refactor repo-utils into main, add Tangled record management and README sync
- Removed `repo-utils.ts` and moved its functions (`run`, `ensureDir`, `generateTid`, `ensureTangledRecord`, `updateReadme`) into `src/index.ts`.
- Improved Tangled record creation with caching and safe TID generation.
- Updated GitHub repo fetching and filtering logic.
- Added automatic README updates with Tangled mirror references.
- Upgraded TypeScript and ts-node versions in package.json.
- Expanded README.md with detailed setup, usage, and best practices instructions.
- Added VSCode spellcheck settings for project-specific words.
+244
-109
+7
.vscode/settings.json
+7
.vscode/settings.json
+95
-6
README.md
+95
-6
README.md
···
1
1
# Tangled Sync
2
2
3
-
a TypeScript project that syncs GitHub repos to Tangled and publishes ATProto records for each repository.
3
+
**Tangled Sync** is a TypeScript project that automates the process of syncing GitHub repositories to Tangled and publishing ATProto records for each repository. It is designed to streamline your workflow if you want your GitHub projects mirrored on Tangled while also maintaining structured metadata in ATProto.
4
+
5
+
This tool is particularly useful for developers and organisations that want a decentralized or alternative hosting layer for their code repositories while keeping them discoverable via ATProto.
6
+
7
+
---
8
+
9
+
## Getting Started
10
+
11
+
### Configuration
12
+
13
+
Before running any scripts, you need to configure the project. See `src/.env` (or `src/config.env` if you prefer to keep a template) for the required environment variables:
14
+
15
+
* `BASE_DIR` – the local directory where GitHub repositories will be cloned.
16
+
* `GITHUB_USER` – your GitHub username or organisation.
17
+
* `ATPROTO_DID` – your ATProto DID (Decentralized Identifier).
18
+
* `BLUESKY_PDS` – the URL of your Bluesky PDS instance.
19
+
* `BLUESKY_USERNAME` – your Bluesky username.
20
+
* `BLUESKY_PASSWORD` – your Bluesky password.
21
+
22
+
Make sure this file is properly set up before proceeding.
23
+
24
+
---
25
+
26
+
### Installation
27
+
28
+
1. Clone this repository locally.
29
+
2. Navigate to the project directory.
30
+
3. Run:
31
+
32
+
```bash
33
+
npm install
34
+
```
35
+
36
+
This will install all dependencies required for syncing GitHub repositories and interacting with ATProto.
37
+
38
+
---
39
+
40
+
### Verify SSH Connection to Tangled
41
+
42
+
* If the Tangled remote does not exist for a repository, the script will attempt to create it on first run. This requires a working SSH key associated with your account.
43
+
44
+
Without proper SSH authentication, repository creation and pushing will fail.
45
+
46
+
---
47
+
48
+
### Running the Sync Script
49
+
50
+
Once configuration and SSH verification are complete, run:
51
+
52
+
```bash
53
+
npm run sync
54
+
```
55
+
56
+
What happens during the sync:
57
+
58
+
1. **Login to Bluesky:** The script authenticates using your credentials to allow publishing ATProto records.
59
+
2. **Clone GitHub Repositories:** All repositories under your configured GitHub user are cloned locally (excluding a repository with the same name as your username to avoid recursion).
60
+
3. **Ensure Tangled Remotes:** For each repository, a `tangled` remote is added if it doesn’t exist.
61
+
4. **Push to Tangled:** The script pushes the `main` branch to Tangled. If your `origin` remote’s push URL points to Tangled, it will reset it back to GitHub.
62
+
5. **Update README:** Each repository’s README is updated to include a link to its Tangled mirror, if it isn’t already present.
63
+
6. **Create ATProto Records:** Each repository gets a structured record published in ATProto under your DID, including metadata like description, creation date, and source URL.
64
+
65
+
---
66
+
67
+
### Notes & Best Practices
4
68
5
-
See `src/config.env` for configuration. After running this script, run `npm install`
6
-
and then `npm run sync` from the project directory.
69
+
* **Directory Management:** The script ensures that your `BASE_DIR` exists and creates it if necessary.
70
+
* **Record Uniqueness:** ATProto records use a time-based, sortable ID (TID) to ensure uniqueness. Duplicate IDs are avoided automatically.
71
+
* **Error Handling:** If a repository cannot be pushed to Tangled, the script logs a warning but continues processing the remaining repositories.
72
+
* **Idempotency:** Running the script multiple times is safe; existing remotes and ATProto records are checked before creation to prevent duplicates.
7
73
8
-
**Crucially**, before running `npm run sync`, you must **verify your SSH connection** to Tangled:
74
+
---
9
75
10
-
1. Run `ssh -T git@tangled.sh` and ensure it succeeds.
11
-
2. If the tangled remote does not exist for a GitHub repo, the script will attempt to create it on first run, but this requires an active, working SSH key.
76
+
### Example Workflow
77
+
78
+
```bash
79
+
# Run the sync script
80
+
npm run sync
81
+
```
82
+
83
+
After execution, you’ll see logs detailing which repositories were cloned, which remotes were added, which READMEs were updated, and which ATProto records were created.
84
+
85
+
This allows you to quickly confirm that all GitHub repositories have been mirrored and documented properly on Tangled.
86
+
87
+
---
88
+
89
+
### Contribution & Development
90
+
91
+
If you plan to contribute:
92
+
93
+
* Ensure Node.js v18+ and npm v9+ are installed.
94
+
* Test the script in a separate directory to avoid accidentally overwriting your production repositories.
95
+
* Use `console.log` statements to debug or track progress during development.
96
+
* Maintain proper `.env` configuration to avoid leaking credentials.
97
+
98
+
---
99
+
100
+
**Tangled Sync** bridges GitHub and Tangled efficiently, providing automatic mirroring, record management, and easy discoverability. Following these steps will ensure a smooth, automated workflow for syncing and publishing your repositories.
+2
-2
package-lock.json
+2
-2
package-lock.json
+2
-4
package.json
+2
-4
package.json
···
1
-
2
1
{
3
2
"name": "tangled-sync",
4
3
"version": "1.0.0",
5
4
"description": "Sync GitHub repos to Tangled with ATProto records",
6
5
"main": "src/index.ts",
7
-
"type": "module",
8
6
"scripts": {
9
7
"sync": "ts-node src/index.ts"
10
8
},
···
13
11
"dotenv": "^16.0.0"
14
12
},
15
13
"devDependencies": {
16
-
"typescript": "^5.0.0",
17
-
"ts-node": "^10.0.0"
14
+
"ts-node": "^10.9.2",
15
+
"typescript": "^5.9.3"
18
16
},
19
17
"author": "",
20
18
"license": "MIT"
+138
-4
src/index.ts
+138
-4
src/index.ts
···
2
2
import dotenv from "dotenv";
3
3
import fs from "fs";
4
4
import path from "path";
5
-
import { run, ensureDir, ensureTangledRecord, updateReadme } from "./repo-utils";
5
+
import { execSync } from "child_process";
6
6
7
7
dotenv.config({ path: "./src/.env" });
8
8
···
22
22
console.log("[LOGIN] Logged in to Bluesky");
23
23
}
24
24
25
-
async function getGitHubRepos(): Promise<{ clone_url: string, name: string, description?: string }[]> {
25
+
async function getGitHubRepos(): Promise<{ clone_url: string; name: string; description?: string }[]> {
26
26
const curl = `curl -s "https://api.github.com/users/${GITHUB_USER}/repos?per_page=200"`;
27
27
const output = run(curl);
28
28
const json = JSON.parse(output);
29
-
return json.filter((r: any) => r.name !== GITHUB_USER)
30
-
.map((r: any) => ({ clone_url: r.clone_url, name: r.name, description: r.description }));
29
+
return json
30
+
.filter((r: any) => r.name !== GITHUB_USER)
31
+
.map((r: any) => ({ clone_url: r.clone_url, name: r.name, description: r.description }));
31
32
}
32
33
33
34
async function ensureTangledRemoteAndPush(repoDir: string, repoName: string, cloneUrl: string) {
···
49
50
console.log(`[PUSH] Pushed main to Tangled`);
50
51
} catch (error) {
51
52
console.warn(`[WARN] Could not push ${repoName} to Tangled. Check SSH or repo existence.`);
53
+
}
54
+
}
55
+
56
+
const BASE32_SORTABLE = "234567abcdefghijklmnopqrstuvwxyz";
57
+
58
+
function run(cmd: string, cwd?: string): string {
59
+
const options: import("child_process").ExecSyncOptions = {
60
+
cwd,
61
+
stdio: "pipe",
62
+
shell: process.env.SHELL || "/bin/bash",
63
+
encoding: "utf-8",
64
+
};
65
+
return execSync(cmd, options).toString().trim();
66
+
}
67
+
68
+
function ensureDir(dir: string) {
69
+
if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true });
70
+
}
71
+
72
+
function generateClockId(): number {
73
+
return Math.floor(Math.random() * 1024);
74
+
}
75
+
76
+
function toBase32Sortable(num: bigint): string {
77
+
if (num === 0n) return "2222222222222";
78
+
let result = "";
79
+
while (num > 0n) {
80
+
result = BASE32_SORTABLE[Number(num % 32n)] + result;
81
+
num = num / 32n;
82
+
}
83
+
return result.padStart(13, "2");
84
+
}
85
+
86
+
function generateTid(): string {
87
+
const nowMicroseconds = BigInt(Date.now()) * 1000n;
88
+
const clockId = generateClockId();
89
+
const tidBigInt = (nowMicroseconds << 10n) | BigInt(clockId);
90
+
return toBase32Sortable(tidBigInt);
91
+
}
92
+
93
+
// Tangled repo schema typing
94
+
interface TangledRepoRecord {
95
+
$type: "sh.tangled.repo";
96
+
knot: string;
97
+
name: string;
98
+
spindle: string;
99
+
description: string;
100
+
source: string;
101
+
labels: string[];
102
+
createdAt: string;
103
+
}
104
+
105
+
// Cache for existing repo records
106
+
const recordCache: Record<string, string> = {};
107
+
108
+
async function ensureTangledRecord(
109
+
agent: AtpAgent,
110
+
atprotoDid: string,
111
+
githubUser: string,
112
+
repoName: string,
113
+
description?: string
114
+
): Promise<string> {
115
+
if (recordCache[repoName]) return recordCache[repoName];
116
+
117
+
let cursor: string | undefined = undefined;
118
+
let tid: string | null = null;
119
+
120
+
do {
121
+
const res: any = await agent.api.com.atproto.repo.listRecords({
122
+
repo: atprotoDid,
123
+
collection: "sh.tangled.repo",
124
+
limit: 50,
125
+
cursor,
126
+
});
127
+
128
+
for (const record of res.data.records) {
129
+
const value = record.value as TangledRepoRecord;
130
+
if (value.name === repoName && record.rkey) {
131
+
tid = record.rkey;
132
+
recordCache[repoName] = tid;
133
+
console.log(`[FOUND] Existing record for ${repoName} (TID: ${tid})`);
134
+
break;
135
+
}
136
+
}
137
+
138
+
cursor = res.data.cursor;
139
+
} while (!tid && cursor);
140
+
141
+
if (!tid) {
142
+
tid = generateTid();
143
+
const record: TangledRepoRecord = {
144
+
$type: "sh.tangled.repo",
145
+
knot: "knot1.tangled.sh",
146
+
name: repoName,
147
+
spindle: "",
148
+
description: description ?? repoName,
149
+
source: `https://github.com/${githubUser}/${repoName}`,
150
+
labels: [],
151
+
createdAt: new Date().toISOString(),
152
+
};
153
+
154
+
await agent.api.com.atproto.repo.putRecord({
155
+
repo: atprotoDid,
156
+
collection: "sh.tangled.repo",
157
+
rkey: tid,
158
+
record,
159
+
});
160
+
161
+
recordCache[repoName] = tid;
162
+
console.log(`[CREATED] Tangled record for ${repoName} (TID: ${tid})`);
163
+
}
164
+
165
+
return tid;
166
+
}
167
+
168
+
function updateReadme(baseDir: string, repoName: string, atprotoDid: string) {
169
+
const repoDir = path.join(baseDir, repoName);
170
+
const readmeFiles = ["README.md", "README.MD", "README.txt", "README"];
171
+
const readmeFile = readmeFiles.find((f) => fs.existsSync(path.join(repoDir, f)));
172
+
if (!readmeFile) return;
173
+
const readmePath = path.join(repoDir, readmeFile);
174
+
const content = fs.readFileSync(readmePath, "utf-8");
175
+
if (!/tangled\.org/i.test(content)) {
176
+
fs.appendFileSync(
177
+
readmePath,
178
+
`
179
+
Mirrored on Tangled: https://tangled.org/${atprotoDid}/${repoName}
180
+
`
181
+
);
182
+
run(`git add ${readmeFile}`, repoDir);
183
+
run(`git commit -m "Add Tangled mirror reference to README"`, repoDir);
184
+
run(`git push origin main`, repoDir);
185
+
console.log(`[README] Updated for ${repoName}`);
52
186
}
53
187
}
54
188
-93
src/repo-utils.ts
-93
src/repo-utils.ts
···
1
-
import { AtpAgent } from "@atproto/api";
2
-
import { execSync, ExecSyncOptions } from "child_process";
3
-
import fs from "fs";
4
-
import path from "path";
5
-
6
-
const BASE32_SORTABLE = '234567abcdefghijklmnopqrstuvwxyz';
7
-
8
-
export function run(cmd: string, cwd?: string): string {
9
-
const options: ExecSyncOptions = {
10
-
cwd,
11
-
stdio: "pipe",
12
-
shell: process.env.SHELL || "/bin/bash",
13
-
encoding: "utf-8"
14
-
};
15
-
return execSync(cmd, options).toString().trim();
16
-
}
17
-
18
-
export function ensureDir(dir: string) {
19
-
if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true });
20
-
}
21
-
22
-
function generateClockId(): number {
23
-
return Math.floor(Math.random() * 1024);
24
-
}
25
-
26
-
function toBase32Sortable(num: bigint): string {
27
-
if (num === 0n) return '2222222222222';
28
-
let result = '';
29
-
while (num > 0n) {
30
-
result = BASE32_SORTABLE[Number(num % 32n)] + result;
31
-
num = num / 32n;
32
-
}
33
-
return result.padStart(13, '2');
34
-
}
35
-
36
-
export function generateTid(): string {
37
-
const nowMicroseconds = BigInt(Date.now()) * 1000n;
38
-
const clockId = generateClockId();
39
-
const tidBigInt = (nowMicroseconds << 10n) | BigInt(clockId);
40
-
return toBase32Sortable(tidBigInt);
41
-
}
42
-
43
-
export async function ensureTangledRecord(agent: AtpAgent, atprotoDid: string, githubUser: string, repoName: string, description?: string) {
44
-
let tid: string = generateTid();
45
-
let exists = true;
46
-
47
-
while (exists) {
48
-
tid = generateTid();
49
-
try {
50
-
await agent.api.com.atproto.repo.getRecord({ repo: atprotoDid, collection: "sh.tangled.repo", rkey: tid });
51
-
exists = true;
52
-
} catch (e: any) {
53
-
if (e.message && e.message.includes('Record not found')) {
54
-
exists = false;
55
-
} else {
56
-
console.error("Error checking ATProto record existence:", e);
57
-
throw e;
58
-
}
59
-
}
60
-
}
61
-
62
-
const record = {
63
-
$type: "sh.tangled.repo",
64
-
knot: "knot1.tangled.sh",
65
-
name: repoName,
66
-
createdAt: new Date().toISOString(),
67
-
description: description || repoName,
68
-
labels: [],
69
-
source: `https://github.com/${githubUser}/${repoName}`,
70
-
spindle: "",
71
-
};
72
-
73
-
await agent.api.com.atproto.repo.putRecord({ repo: atprotoDid, collection: "sh.tangled.repo", rkey: tid, record });
74
-
console.log(`[CREATED] Tangled record for ${repoName} (TID: ${tid})`);
75
-
}
76
-
77
-
export function updateReadme(baseDir: string, repoName: string, atprotoDid: string) {
78
-
const repoDir = path.join(baseDir, repoName);
79
-
const readmeFiles = ["README.md", "README.MD", "README.txt", "README"];
80
-
const readmeFile = readmeFiles.find(f => fs.existsSync(path.join(repoDir, f)));
81
-
if (!readmeFile) return;
82
-
const readmePath = path.join(repoDir, readmeFile);
83
-
const content = fs.readFileSync(readmePath, "utf-8");
84
-
if (!/tangled\.org/i.test(content)) {
85
-
fs.appendFileSync(readmePath, `
86
-
Mirrored on Tangled: https://tangled.org/${atprotoDid}/${repoName}
87
-
`);
88
-
run(`git add ${readmeFile}`, repoDir);
89
-
run(`git commit -m "Add Tangled mirror reference to README"`, repoDir);
90
-
run(`git push origin main`, repoDir);
91
-
console.log(`[README] Updated for ${repoName}`);
92
-
}
93
-
}