Mirror of https://github.com/roostorg/coop
github.com/roostorg/coop
1# Local Development Guide
2
3This guide covers configuration details and troubleshooting for local development. For the quickstart, see the [root README](../README.md).
4
5## Prerequisites
6
7- **Operating System**: macOS, Linux, or Windows with WSL2
8- **Node.js 24** (use `nvm install && nvm use` so local matches `.nvmrc`)
9- **npm** (included with Node.js)
10- **Docker and Docker Compose**
11- **16 GB RAM** or more recommended
12
13## Environment Setup
14
15### Server Configuration
16
17Copy `server/.env.example` to `server/.env`. The example file contains all available options with documentation. Key sections:
18
19- **Database connections**: PostgreSQL, ClickHouse, ScyllaDB, Redis
20- **External APIs**: OpenAI, SendGrid, Google APIs (optional)
21- **Security**: Session secrets, JWT signing keys
22
23The default values work with Docker Compose services out of the box.
24
25### Client Configuration
26
27Copy `client/.env.example` to `client/.env`. Defaults work for local development.
28
29## Docker Services
30
31Start all backing services:
32
33```bash
34npm run up
35```
36
37This starts:
38
39Service | Port | Notes
40----------------|------------|---------------------------------
41PostgreSQL | 5432 | Primary DB (with pgvector)
42ClickHouse | 8123, 9000 | Analytics warehouse
43ScyllaDB | 9042 | Item submission history
44Redis | 6379 | Caching and job queues
45Jaeger | 16686 | Tracing UI (opens automatically)
46OTEL Collector | 4317 | Telemetry collection
47
48Check service health:
49
50```bash
51docker ps
52docker logs <container-name>
53```
54
55Stop services:
56
57```bash
58npm run down
59```
60
61## Database Operations
62
63### Running Migrations
64
65```bash
66npm run db:update -- --env staging --db api-server-pg
67npm run db:update -- --env staging --db clickhouse
68#Creating keyspace
69npm run db:create -- --env staging --db scylla
70#Running migrations
71npm run db:update -- --env staging --db scylla
72```
73
74### Other Commands
75
76```bash
77npm run db:add -- --name <migration-name> --db api-server-pg
78npm run db:clean # Drop and recreate (destructive)
79npm run db:create # Create database
80npm run db:drop # Drop database
81```
82
83### Migration Locations
84
85```
86db/src/scripts/
87├── api-server-pg/ # PostgreSQL
88├── clickhouse/ # ClickHouse
89└── scylla/ # ScyllaDB
90```
91
92## Running the Application
93
94### All Services Together
95
96```bash
97npm run start # Client + server + GraphQL codegen (opens browser)
98npm run compile # Same, without opening browser
99```
100
101### Individual Services (Recommended for Debugging)
102
103```bash
104# Terminal 1
105npm run client:start
106
107# Terminal 2
108npm run server:start
109
110# Terminal 3 (optional, for GraphQL schema changes)
111npm run generate:watch
112```
113
114### Background Workers
115
116Item submissions are processed asynchronously via a BullMQ worker that consumes from Redis. To process items locally, run the worker in a separate terminal:
117
118```bash
119cd server
120npm run runWorkerOrJob ItemProcessingWorker
121```
122
123Without this running, submitted items will be enqueued in Redis but not processed. Other available workers/jobs can be found in `server/iocContainer/services/workersAndJobs.ts`.
124
125### With Distributed Tracing
126
127```bash
128cd server && npm run start:trace
129```
130
131View traces at http://localhost:16686
132
133### Access Points
134
135Service | URL
136-----------|------------------------------
137Client | http://localhost:3000
138API Server | http://localhost:8080
139GraphQL | http://localhost:8080/graphql
140Jaeger UI | http://localhost:16686
141
142## Testing
143
144```bash
145# Server
146cd server
147npm run test # Watch mode
148npm run test:prepush # Single run
149npm run test:integ # Integration tests
150
151# Client
152cd client
153npm run test # Watch mode
154npm run test:prepush # Single run
155
156# Full validation (run before pushing)
157npm run check:prepush
158```
159
160## GraphQL Development
161
162Coop uses schema-first GraphQL with bidirectional code generation.
163
164```bash
165npm run generate # One-time
166npm run generate:watch # Watch mode
167```
168
169Generated files:
170- `client/src/graphql/generated.ts`
171- `server/graphql/generated.ts`
172
173Schema changes trigger recompilation of both client and server. If you experience regeneration loops, stop watch mode and run manually.
174
175## HMA Development
176HMA is not started automatically with `npm run up`. Start it separately if you're doing hash matching: `docker compose up --build -d hma`
177
178HMA is pre-configured in `server/.env` with `HMA_SERVICE_URL=http://localhost:9876`. No additional environment setup is needed for local development.
179
180### Image URL Accessibility
181When submitting items to Coop, image URLs must be reachable by the HMA Docker container and not just your browser or the Node.js server.
182HMA fetches the image itself to compute the hash. This means localhost URLs will silently fail: HMA will return empty hashes, the image similarity signal will not evaluate, and no rule will fire.
183
184For local development, if you're serving images from your host machine, add the following to /etc/hosts:
185
186127.0.0.1 host.docker.internal
187
188Then use `host.docker.internal:<port>` in image URLs when submitting items. This URL resolves correctly from both the browser and inside Docker.
189
190## Troubleshooting
191
192### ScyllaDB Not Ready
193
194ScyllaDB takes 30-60 seconds to initialize. If migrations fail immediately after `npm run up`, wait and retry.
195
196### ClickHouse Migration Fails
197
198Ensure `CLICKHOUSE_USERNAME` and `CLICKHOUSE_PASSWORD` are set in your `.env`.
199
200### Port Conflicts
201
202```bash
203lsof -i :3000 # Client
204lsof -i :8080 # Server
205lsof -i :5432 # PostgreSQL
206```
207
208### Reset Everything
209
210```bash
211npm run down
212docker volume prune # Warning: removes all Docker volumes
213npm run up
214npm run db:update -- --env staging --db api-server-pg
215npm run db:update -- --env staging --db clickhouse
216npm run create-org
217```
218
219### Connecting to Databases Directly
220
221```bash
222# PostgreSQL
223psql -h localhost -U postgres -d postgres
224# Password: postgres123
225
226# ClickHouse
227clickhouse-client --host localhost --user default --password clickhouse
228
229# Redis
230redis-cli
231```
232
233## Code Quality
234
235```bash
236npm run lint # ESLint
237npm run format # Prettier
238npm run check:prepush # Run before pushing
239```