Mirror of https://github.com/roostorg/osprey
github.com/roostorg/osprey
1# Development
2Welcome to the development guide for Osprey. This document will help you get started with contributing to the project.
3
4## Reporting a Bug or Issue
5Found a bug or have a feature request? We'd love to hear from you! When opening an issue, please use our templates:
6
7* [Bug Report](https://github.com/roostorg/osprey/issues/new?template=bug_report.md)
8* [Feature Request](https://github.com/roostorg/osprey/issues/new?template=feature_request.md)
9* [Submit an Egg (new tool idea) to ROOST!](https://github.com/roostorg/osprey/issues/new?template=documentation.md)
10
11# Setup Guide
12
13This guide provides comprehensive instructions for setting up a development environment for Osprey.
14
15## Prerequisites
16
17### System Requirements
18
19- **Python 3.11 or higher** - Check with `python --version`
20- **Git** - Version control system
21- **Operating System**: macOS, Linux, or Windows (with WSL recommended)
22
23### Package Manager: UV
24
25Osprey uses [uv](https://docs.astral.sh/uv/) as the Python package manager for fast, reliable dependency management.
26
27#### Install UV
28
29**macOS/Linux:**
30
31```bash
32curl -LsSf https://astral.sh/uv/install.sh | sh
33```
34
35**Windows:**
36
37```bash
38powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
39```
40
41**Alternative (via pip):**
42
43```bash
44pip install uv
45```
46
47**Verify installation:**
48
49```bash
50uv --version
51```
52
53## Project Setup
54
55### 1. Clone the Repository
56
57```bash
58git clone git@github.com:roostorg/osprey.git
59cd osprey
60```
61
62### 2. Install Dependencies
63
64```bash
65# Install all dependencies including development tools
66uv sync
67```
68
69This command will:
70
71- Create a virtual environment automatically
72- Install all production dependencies
73- Install development dependencies (ruff, mypy, pre-commit) automatically
74- Use the locked versions from `uv.lock` for reproducible builds
75
76**Note**: `uv sync` includes development dependencies by default. Use `uv sync --no-dev` if you only want production dependencies.
77
78### 3. Set Up Pre-commit Hooks
79
80```bash
81uv run pre-commit install
82```
83
84This installs git hooks that automatically run code quality checks before each commit.
85
86### 4. Verify Setup
87
88Run these commands to ensure everything is working correctly:
89
90```bash
91# Check linting configuration
92uv run ruff check
93
94# Check formatting
95uv run ruff format --diff
96
97# Run type checking
98uv run mypy .
99
100# Test pre-commit hooks
101uv run pre-commit run --all-files
102```
103
104**Expected Results:**
105
106- Ruff should report "All checks passed!" or show specific issues to fix
107- MyPy should run without errors
108- Pre-commit should run all hooks successfully
109
110### 5. Getting Started
111
112```bash
113docker compose up -d
114```
115
116This starts up many services, including:
117- **Osprey Worker**: The main engine that processes input events given the rules and UDFs
118 - **Test Data Producer**: Optional with `--profile test_data`
119- **Osprey UI**: Frontend service that hosts the react code for the web interface and communicates to the UI API
120- **Osprey UI API**: Backend service that provides data and functionality to the web interface
121- **Kafka** (KRaft mode): Message streaming for user generated events
122- **Postgres**: A database that the Worker, UI API, and Druid use for various reasons, such as the Postgres-backed Labels Service (in the example plugins)
123- **Druid**: A database that consumes Osprey Worker outputs to power the UI API for real-time querying
124
125
126### 6. Access the Application
127The UI will automatically connect to the backend services running in Docker containers.
128
129 - Osprey UI: http://localhost:5002
130 - Backend API: http://localhost:5004
131 - Worker Service: http://localhost:5001
132
133
134#### Plugins
135
136In Osprey, UDFs and output sinks are designed to be easily portable. This is done through a plugin system based on pluggy. An example plugin package has been provided for reference, see `example_plugins/register_plugins.py`:
137
138```python
139@hookimpl_osprey
140def register_udfs() -> Sequence[Type[UDFBase[Any, Any]]]:
141 # Register custom user-defined functions
142
143@hookimpl_osprey
144def register_output_sinks(config: Config) -> Sequence[BaseOutputSink]:
145 # Define output destinations
146 # By default it prints the execution results to the console
147
148@hookimpl_osprey
149def register_ast_validators() -> None:
150 # Register AST validators
151```
152
153#### Rules
154
155Rules are written in SML, some examples are provided in `example_rules/` with YAML config, the rules are mounted to the worker processes when the containers start via environment variables. ex:
156
157```bash
158OSPREY_RULES=./example_rules uv run python3.11 osprey_worker/src/osprey/worker/cli/sinks.py run-rules-sink
159```
160
161#### Test Data
162
163Generate sample JSON actions:
164```bash
165docker compose --profile test_data up kafka_test_data_producer -d
166```
167
168Produces user login events with timestamps, user IDs, and IP addresses to `osprey.actions_input` topic.
169
170## Development Workflow
171
172### Branch Management
173
174- **Branch naming convention**: Use `github_username/description` format (e.g., `caidanw/feature-auth`, `caidanw/fix-database-timeout`)
175- **Base branch**: Always branch from `main`
176- **Create new branch**: `git checkout -b username/feature-name`
177
178### Code Quality Standards
179
180#### Automated Checks
181
182Every commit automatically runs:
183
1841. **Trailing whitespace removal**
1852. **End-of-file fixing**
1863. **YAML/JSON/TOML validation**
1874. **Ruff linting and formatting**
188
189#### Manual Checks
190
191Before pushing, run:
192
193```bash
194# Comprehensive linting check
195uv run ruff check
196
197# Format all code
198uv run ruff format
199
200# Type checking (on specific files/modules)
201uv run mypy osprey_worker/src/osprey_worker/lib
202# Or you can type check every module (this will happen in CI)
203uv run mypy .
204
205# Run all pre-commit hooks
206uv run pre-commit run --all-files
207```
208
209### Commit Standards
210
211Follow [Conventional Commits](https://www.conventionalcommits.org/) format:
212
213```
214feat: add user authentication system
215fix: resolve database connection timeout
216docs: update API documentation
217refactor: simplify rule evaluation logic
218```
219
220**Examples:**
221
222- `feat:` - New features
223- `fix:` - Bug fixes
224- `docs:` - Documentation changes
225- `refactor:` - Code refactoring
226- `test:` - Adding or updating tests
227- `chore:` - Maintenance tasks
228
229### Making Changes
230
2311. **Create a new branch:**
232
233 ```bash
234 git checkout -b username/feature-name
235 ```
236
2372. **Make your changes**
238
2393. **Run quality checks:**
240
241 ```bash
242 uv run ruff check --fix
243 uv run ruff format
244 ```
245
2464. **Test your changes** (if tests exist)
247
2485. **Commit your changes:**
249
250 ```bash
251 git add .
252 git commit -m "feat: descriptive commit message"
253 ```
254
255 Pre-commit hooks will run automatically and may fix formatting issues.
256
2576. **Push your branch:**
258
259 ```bash
260 git push origin username/feature-name
261 ```
262
263## Development Tools Overview
264
265### Ruff - Linting and Formatting
266
267**Purpose**: Replaces Black, isort, Flake8, and other tools
268
269**Configuration**: Located in `pyproject.toml` under `[tool.ruff]`
270
271**Key Rules Enabled**:
272
273- `E` - pycodestyle errors
274- `F` - pyflakes
275- `I` - isort (import sorting)
276- `B006` - flake8-bugbear (mutable default arguments)
277
278**Commands**:
279
280```bash
281# Check for issues
282uv run ruff check
283
284# Fix auto-fixable issues
285uv run ruff check --fix
286
287# Format code
288uv run ruff format
289
290# Check specific files
291uv run ruff check path/to/file.py
292```
293
294### MyPy - Type Checking
295
296**Purpose**: Static type checking for Python
297
298**Configuration**: Located in `pyproject.toml` under `[tool.mypy]`
299
300**Key Features**:
301
302- Pydantic plugin support
303- SQLAlchemy plugin support
304- Relaxed strict mode (matching legacy codebase)
305- Ignores protobuf generated files
306
307**Commands**:
308
309```bash
310# Type check entire project
311uv run mypy .
312
313# Type check specific files
314uv run mypy path/to/file.py
315
316# Type check entire module
317uv run mypy osprey_worker/
318
319# Check with verbose output
320uv run mypy --show-traceback path/to/file.py
321```
322
323### Pre-commit - Git Hooks
324
325**Purpose**: Automated quality checks before commits
326
327**Configuration**: Located in `.pre-commit-config.yaml`
328
329**Commands**:
330
331```bash
332# Run all hooks on staged files
333uv run pre-commit run
334
335# Run all hooks on all files
336uv run pre-commit run --all-files
337
338# Run specific hook
339uv run pre-commit run ruff
340
341# Update hook versions
342uv run pre-commit autoupgrade
343
344# Bypass hooks (emergency only)
345git commit --no-verify
346```
347
348### UV - Package Management
349
350**Purpose**: Fast Python package manager and environment management
351
352**Key Commands**:
353
354```bash
355# Install dependencies
356uv sync
357
358# Add new dependency
359uv add package-name
360
361# Add development dependency
362uv add --group dev package-name
363
364# Remove dependency
365uv remove package-name
366
367# Run command in environment
368uv run command-name
369
370# Update dependencies
371uv lock --upgrade
372```
373
374## Troubleshooting
375
376### Common Issues
377
378#### "uv: command not found"
379
380**Solution**: Install uv using the installation script or pip:
381
382```bash
383curl -LsSf https://astral.sh/uv/install.sh | sh
384# Then restart your terminal
385```
386
387#### Pre-commit hooks failing
388
389**Solution**: Run hooks manually to see detailed errors:
390
391```bash
392uv run pre-commit run --all-files
393```
394
395#### MyPy errors on protobuf files
396
397**Solution**: Protobuf generated files are excluded in configuration. If you see errors, check that files match the exclusion patterns in `pyproject.toml`.
398
399#### Import errors during type checking
400
401**Solution**: Ensure all dependencies are installed:
402
403```bash
404uv sync
405```
406
407### Getting Help
408
4091. **Check this documentation** for common setup issues
4102. **Review error messages** carefully - they often contain solutions
4113. **Run commands with verbose flags** for more detailed output
4124. **Check configuration files** (`pyproject.toml`, `.pre-commit-config.yaml`)
413
414## Next Steps
415
416- Check the [contributing guidelines](https://github.com/roostorg/.github/blob/main/CONTRIBUTING.md) for project-specific rules
417- Explore the codebase structure in `osprey_worker/`, `osprey_common/`, and `osprey_rpc/`
418
419## IDE Setup Recommendations
420
421### VS Code
422
423Install these extensions for the best development experience:
424
425- **Python** (Microsoft)
426- **Ruff** (Astral Software)
427- **MyPy Type Checker** (Microsoft)
428
429**Settings** (add to `.vscode/settings.json`):
430
431```json
432{
433 "python.defaultInterpreterPath": ".venv/bin/python",
434 "ruff.enable": true,
435 "ruff.organizeImports": true,
436 "python.analysis.typeCheckingMode": "basic"
437}
438```
439
440This completes the development setup! You're now ready to contribute to Osprey.