Development#

Welcome to the development guide for Osprey. This document will help you get started with contributing to the project.

Reporting a Bug or Issue#

Found a bug or have a feature request? We'd love to hear from you! When opening an issue, please use our templates:

• Bug Report
• Feature Request
• Submit an Egg (new tool idea) to ROOST!

Setup Guide#

This guide provides comprehensive instructions for setting up a development environment for Osprey.

Prerequisites#

System Requirements#

• Python 3.11 or higher - Check with python --version
• Git - Version control system
• Operating System: macOS, Linux, or Windows (with WSL recommended)

Package Manager: UV#

Osprey uses uv as the Python package manager for fast, reliable dependency management.

Install UV#

macOS/Linux:

curl -LsSf https://astral.sh/uv/install.sh | sh

Windows:

powershell -c "irm https://astral.sh/uv/install.ps1 | iex"

Alternative (via pip):

pip install uv

Verify installation:

uv --version

Project Setup#

1. Clone the Repository#

git clone git@github.com:roostorg/osprey.git
cd osprey

2. Install Dependencies#

# Install all dependencies including development tools
uv sync

This command will:

• Create a virtual environment automatically
• Install all production dependencies
• Install development dependencies (ruff, mypy, pre-commit) automatically
• Use the locked versions from uv.lock for reproducible builds

Note: uv sync includes development dependencies by default. Use uv sync --no-dev if you only want production dependencies.

3. Set Up Pre-commit Hooks#

uv run pre-commit install

This installs git hooks that automatically run code quality checks before each commit.

4. Verify Setup#

Run these commands to ensure everything is working correctly:

# Check linting configuration
uv run ruff check

# Check formatting
uv run ruff format --diff

# Run type checking
uv run mypy .

# Test pre-commit hooks
uv run pre-commit run --all-files

Expected Results:

• Ruff should report "All checks passed!" or show specific issues to fix
• MyPy should run without errors
• Pre-commit should run all hooks successfully

5. Getting Started#

docker compose up -d

This starts up many services, including:

• Osprey Worker: The main engine that processes input events given the rules and UDFs
  • Test Data Producer: Optional with --profile test_data
• Osprey UI: Frontend service that hosts the react code for the web interface and communicates to the UI API
• Osprey UI API: Backend service that provides data and functionality to the web interface
• Kafka (KRaft mode): Message streaming for user generated events
• Postgres: A database that the Worker, UI API, and Druid use for various reasons, such as the Postgres-backed Labels Service (in the example plugins)
• Druid: A database that consumes Osprey Worker outputs to power the UI API for real-time querying

6. Access the Application#

The UI will automatically connect to the backend services running in Docker containers.

• Osprey UI: http://localhost:5002
• Backend API: http://localhost:5004
• Worker Service: http://localhost:5001

Plugins#

In Osprey, UDFs and output sinks are designed to be easily portable. This is done through a plugin system based on pluggy. An example plugin package has been provided for reference, see example_plugins/register_plugins.py:

@hookimpl_osprey
def register_udfs() -> Sequence[Type[UDFBase[Any, Any]]]:
    # Register custom user-defined functions

@hookimpl_osprey
def register_output_sinks(config: Config) -> Sequence[BaseOutputSink]:
    # Define output destinations
    # By default it prints the execution results to the console

@hookimpl_osprey
def register_ast_validators() -> None:
    # Register AST validators

Rules#

Rules are written in SML, some examples are provided in example_rules/ with YAML config, the rules are mounted to the worker processes when the containers start via environment variables. ex:

OSPREY_RULES=./example_rules uv run python3.11 osprey_worker/src/osprey/worker/cli/sinks.py run-rules-sink

Test Data#

Generate sample JSON actions:

docker compose --profile test_data up kafka_test_data_producer -d

Produces user login events with timestamps, user IDs, and IP addresses to osprey.actions_input topic.

Development Workflow#

Branch Management#

• Branch naming convention: Use github_username/description format (e.g., caidanw/feature-auth, caidanw/fix-database-timeout)
• Base branch: Always branch from main
• Create new branch: git checkout -b username/feature-name

Code Quality Standards#

Automated Checks#

Every commit automatically runs:

1. Trailing whitespace removal
2. End-of-file fixing
3. YAML/JSON/TOML validation
4. Ruff linting and formatting

Manual Checks#

Before pushing, run:

# Comprehensive linting check
uv run ruff check

# Format all code
uv run ruff format

# Type checking (on specific files/modules)
uv run mypy osprey_worker/src/osprey_worker/lib
# Or you can type check every module (this will happen in CI)
uv run mypy .

# Run all pre-commit hooks
uv run pre-commit run --all-files

Commit Standards#

Follow Conventional Commits format:

feat: add user authentication system
fix: resolve database connection timeout
docs: update API documentation
refactor: simplify rule evaluation logic

Examples:

• feat: - New features
• fix: - Bug fixes
• docs: - Documentation changes
• refactor: - Code refactoring
• test: - Adding or updating tests
• chore: - Maintenance tasks

Making Changes#

1. Create a new branch:

   git checkout -b username/feature-name
   

2. Make your changes

3. Run quality checks:

   uv run ruff check --fix
   uv run ruff format
   

4. Test your changes (if tests exist)

5. Commit your changes:

   git add .
   git commit -m "feat: descriptive commit message"
   

   Pre-commit hooks will run automatically and may fix formatting issues.

6. Push your branch:

   git push origin username/feature-name
   

Development Tools Overview#

Ruff - Linting and Formatting#

Purpose: Replaces Black, isort, Flake8, and other tools

Configuration: Located in pyproject.toml under [tool.ruff]

Key Rules Enabled:

• E - pycodestyle errors
• F - pyflakes
• I - isort (import sorting)
• B006 - flake8-bugbear (mutable default arguments)

Commands:

# Check for issues
uv run ruff check

# Fix auto-fixable issues
uv run ruff check --fix

# Format code
uv run ruff format

# Check specific files
uv run ruff check path/to/file.py

MyPy - Type Checking#

Purpose: Static type checking for Python

Configuration: Located in pyproject.toml under [tool.mypy]

Key Features:

• Pydantic plugin support
• SQLAlchemy plugin support
• Relaxed strict mode (matching legacy codebase)
• Ignores protobuf generated files

Commands:

# Type check entire project
uv run mypy .

# Type check specific files
uv run mypy path/to/file.py

# Type check entire module
uv run mypy osprey_worker/

# Check with verbose output
uv run mypy --show-traceback path/to/file.py

Pre-commit - Git Hooks#

Purpose: Automated quality checks before commits

Configuration: Located in .pre-commit-config.yaml

Commands:

# Run all hooks on staged files
uv run pre-commit run

# Run all hooks on all files
uv run pre-commit run --all-files

# Run specific hook
uv run pre-commit run ruff

# Update hook versions
uv run pre-commit autoupgrade

# Bypass hooks (emergency only)
git commit --no-verify

UV - Package Management#

Purpose: Fast Python package manager and environment management

Key Commands:

# Install dependencies
uv sync

# Add new dependency
uv add package-name

# Add development dependency
uv add --group dev package-name

# Remove dependency
uv remove package-name

# Run command in environment
uv run command-name

# Update dependencies
uv lock --upgrade

Troubleshooting#

Common Issues#

"uv: command not found"#

Solution: Install uv using the installation script or pip:

curl -LsSf https://astral.sh/uv/install.sh | sh
# Then restart your terminal

Pre-commit hooks failing#

Solution: Run hooks manually to see detailed errors:

uv run pre-commit run --all-files

MyPy errors on protobuf files#

Solution: Protobuf generated files are excluded in configuration. If you see errors, check that files match the exclusion patterns in pyproject.toml.

Import errors during type checking#

Solution: Ensure all dependencies are installed:

uv sync

Getting Help#

1. Check this documentation for common setup issues
2. Review error messages carefully - they often contain solutions
3. Run commands with verbose flags for more detailed output
4. Check configuration files (pyproject.toml, .pre-commit-config.yaml)

Next Steps#

• Check the contributing guidelines for project-specific rules
• Explore the codebase structure in osprey_worker/, osprey_common/, and osprey_rpc/

IDE Setup Recommendations#

VS Code#

Install these extensions for the best development experience:

• Python (Microsoft)
• Ruff (Astral Software)
• MyPy Type Checker (Microsoft)

Settings (add to .vscode/settings.json):

{
  "python.defaultInterpreterPath": ".venv/bin/python",
  "ruff.enable": true,
  "ruff.organizeImports": true,
  "python.analysis.typeCheckingMode": "basic"
}

This completes the development setup! You're now ready to contribute to Osprey.