···11+name: Deploy mdBook site to Pages
22+33+on:
44+ # Runs on pushes targeting the default branch
55+ push:
66+ branches: ["main"]
77+88+ # Allows you to run this workflow manually from the Actions tab
99+ workflow_dispatch:
1010+1111+# Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages
1212+permissions:
1313+ contents: read
1414+ pages: write
1515+ id-token: write
1616+1717+# Allow only one concurrent deployment, skipping runs queued between the run in-progress and latest queued.
1818+# However, do NOT cancel in-progress runs as we want to allow these production deployments to complete.
1919+concurrency:
2020+ group: "pages"
2121+ cancel-in-progress: false
2222+2323+jobs:
2424+ # Build job
2525+ build:
2626+ runs-on: ubuntu-latest
2727+ env:
2828+ MDBOOK_VERSION: 0.5.2
2929+ steps:
3030+ - uses: actions/checkout@v4
3131+ - name: Install mdBook
3232+ run: |
3333+ curl --proto '=https' --tlsv1.2 https://sh.rustup.rs -sSf -y | sh
3434+ rustup update
3535+ cargo install --version ${MDBOOK_VERSION} mdbook
3636+ - name: Setup Pages
3737+ id: pages
3838+ uses: actions/configure-pages@v5
3939+ - name: Build with mdBook
4040+ run: mdbook build docs
4141+ - name: Upload artifact
4242+ uses: actions/upload-pages-artifact@v3
4343+ with:
4444+ path: ./docs/book
4545+4646+ # Deployment job
4747+ deploy:
4848+ environment:
4949+ name: github-pages
5050+ url: ${{ steps.deployment.outputs.page_url }}
5151+ runs-on: ubuntu-latest
5252+ needs: build
5353+ steps:
5454+ - name: Deploy to GitHub Pages
5555+ id: deployment
5656+ uses: actions/deploy-pages@v4
+3
.gitignore
···306306# End of https://www.toptal.com/developers/gitignore/api/python,rust,pycharm
307307308308.claude
309309+310310+# docs output
311311+book/
+22-99
README.md
···11-# ROOST - Osprey
22-33-<img width="200" height="64" alt="Copy of ROOST-Mark-Yellow" src="/images/ROOST-Horizontal-Yellow.png" />
44-55-# Welcome to Osprey
66-77-<img alt="Osprey UI Sample" src="./images/query-and-charts.png" />
88-99-1010-## Overview
11+<img width="200" height="64" alt="ROOST logo" src="images/ROOST-Horizontal-Yellow.png" />
1121212-This repository is home to one of ROOST’s safety tools, the Osprey rules engine. Osprey is an open-source event stream decisions engine and analysis UI designed to investigate and take automatic action on events and their properties as they happen in real-time. Originally developed internally at [Discord](https://discord.com/) to combat spam, abuse, botting, and scripting across its platform, Osprey has now been open-sourced to help other platforms facing similar challenges.
33+# Osprey
1341414-Osprey is a library for processing actions through human-written rules and outputting verdicts & custom effects back to configurable output sinks. It evaluates events using structured rule logic (SML) that is extendable via user-defined functions (UDFs). Osprey can also track state across events by labelling entities if implementers provide a labels service backend (see [labels_service.py](./example_plugins/src/services/labels_service.py) for a Postgres-backed labels service example)
1515-1616-This 'Rules \+ Investigation' tool is able to:
55+**Automate the obvious and investigate the ambiguous.** High-performance safety rules engine for real-time event processing at scale.
176187- take action based on user behavior
198- combine actions with human written rules
209- let operators query human actions and past decisions
2110- perform investigations or write new rules based on decisions
22112323-## v0 Release Notes
2424-The v0 release is built for engineers and Trust & Safety teams who want to explore, test, and integrate Osprey's core capabilities into their platform for incident response and Trust & Safety investigation.
1212+Osprey is an event stream decisions engine and analysis UI designed to investigate and take automatic action on events and their properties as they happen in real-time. Originally developed internally at [Discord](https://discord.com/) to combat spam, abuse, botting, and scripting across its platform, Osprey has been open-sourced to help other platforms facing similar challenges.
25132626-Some UI features have been planned for the v1 release to ensure a solid foundation for experimentation:
1414+
27152828-- **Bulk actions** - Mass investigation and response operations through the UI
2929-- **Labels** - Visual entity tagging and categorization systems in the interface
3030-- **Event stream in UI** - Real-time event monitoring and alert dashboard
3131-- **Load Balancing** - Better performance efficiency and management of sync/async rules
1616+Osprey is a library for processing actions through human-written rules and outputting verdicts & custom effects back to configurable output sinks. It evaluates events using structured rule logic (SML) that is extendable via user-defined functions (UDFs). Osprey can also track state across events by labelling entities if implementers provide a labels service backend (see [labels_service.py](./example_plugins/src/services/labels_service.py) for a Postgres-backed labels service example).
32173333-### Feedback Wanted
3434-This is a working system, not a prototype. Try it locally, connect your data, write some rules, and tell us what's missing for your use case. We're particularly interested in:
1818+Osprey is built for engineers and Trust & Safety teams who want to explore, test, and integrate its core capabilities into their platform for incident response and Trust & Safety investigation. [Read more about user research and personas](docs/user_personas.md).
35193636-- Integration challenges with your existing platform infrastructure
3737-- Performance characteristics with your event volumes and rule complexity
3838-- Missing detection capabilities or response actions you need
3939-- API improvements that would make adoption easier for your team
2020+## Development
40214141-Your experimentation feedback will directly shape v1 priorities and help us build the most useful Trust & Safety tooling for the community.
2222+- See [DEVELOPMENT.md](./docs/DEVELOPMENT.md) for comprehensive development setup and workflow documentation
2323+- All code changes should pass linting (Ruff) and type checking (MyPy)
2424+- Pre-commit hooks automatically run on each commit to maintain code quality
42254326## Join Us
2727+4428Writing code is not the only way to help the project. Reviewing pull requests, answering questions to help others on mailing lists or issues, providing feedback from a domain expert perspective, organizing and teaching tutorials, working on the website, improving the documentation, are all priceless contributions.
45294646-- Join us on [Discord server](https://discord.gg/5Csqnw2FSQ)
3030+- Join us in [our Discord server](https://discord.gg/5Csqnw2FSQ)
4731- Join our [newsletter](https://roost.tools/#get-started) for more announcements and information
4832- Follow us on [Bluesky](https://bsky.app/profile/roost.tools) or [LinkedIn](https://www.linkedin.com/company/roost-tools/)
49335050-_ROOST (Robust Open Online Safety Tools), a non-profit organization that brings together expertise, resources, and investments from major technology companies and philanthropies to build scalable, interoperable safety infrastructure for the AI era._
3434+_[ROOST](https://roost.tools) (Robust Open Online Safety Tools) is a non-profit organization that brings together expertise, resources, and investments from major technology companies and philanthropies to build scalable, interoperable safety infrastructure for the AI era._
51355252-## 🚀 Quick Start
3636+### Feedback Wanted
53375454-### Prerequisites
3838+This is a working system, not a prototype. Try it locally, connect your data, write some rules, and tell us what's missing for your use case. We're particularly interested in:
55395656-- Python 3.11 or higher
5757-- Git
5858-- [uv](https://docs.astral.sh/uv/) (Python package manager) or python package manager of choice
4040+- Integration challenges with your existing platform infrastructure
4141+- Performance characteristics with your event volumes and rule complexity
4242+- Missing detection capabilities or response actions you need
4343+- API improvements that would make adoption easier for your team
59446060-### Installation
4545+Your experimentation feedback will directly shape future priorities and help us build the most useful Trust & Safety tooling for the community.
61466262-1. **Clone the repository:**
4747+## Recognition
63486464- ```bash
6565- git clone git@github.com:roostorg/osprey.git
6666- cd osprey
6767- ```
6868-6969-2. **Install dependencies using uv:**
7070-7171- ```bash
7272- uv sync
7373- ```
7474-7575-3. **Set up development tools:**
7676-7777- ```bash
7878- uv run pre-commit install
7979- ```
8080-8181-4. **Verify setup:**
8282-8383- ```bash
8484- # Test linting
8585- uv run ruff check
8686- uv run mypy .
8787-8888- # Test formatting
8989- uv run ruff format --diff
9090-9191- # Test pre-commit hooks
9292- uv run pre-commit run --all-files
9393-9494-5. **Start Services:**
9595-9696- ```bash
9797- docker compose up -d
9898- ```
9999-100100- or using the wrapper script
101101-102102- ```bash
103103- ./start.sh
104104- ```
105105-106106- this starts the osprey-worker on its own along with all its required dependencies.
107107-108108- alternatively, you can start Osprey with `osprey-coordinator`, refer to the [Coordinator README](./example_docker_compose/run_osprey_with_coordinator/README.md) for more information
109109-110110-6. (Optional) **Open ports for the UI/UI API:**
111111-112112- By default, the `docker-compose.yaml` binds running services to `127.0.0.1`. If you are running the docker compose on a headless machine, you may need to modify this configuration and/or make changes to your firewall, specifically for ports `5002` and `5004`.
113113-114114- For example, if you use Tailscale to access your Osprey instance, you may change `127.0.0.1:5002:5002` to `<Tailscale IP>:5002:5002`. Alternatively, if you wish for your instance to be accessible from the public internet, you may set it simply to `5002:5002` to bind to `0.0.0.0`.
115115-116116- Be aware that some firewalls like iptables/UFW do _not_ prevent access to ports being used by Docker networking. Not explicitly setting a bind address with only UFW as a firewall will not prevent access from the public internet unless [properly configured](https://github.com/chaifeng/ufw-docker).
117117-118118-### Development Workflow
119119-120120-- See [DEVELOPMENT.md](./docs/DEVELOPMENT.md) for comprehensive development setup and workflow documentation
121121-- All code changes should pass linting (Ruff) and type checking (MyPy)
122122-- Pre-commit hooks automatically run on each commit to maintain code quality
123123-124124-125125-## Recognition
12649Discord uses Osprey to quickly detect and remove new types of harm that put users at risk. Rather than leaving other platforms to build similar tools from scratch, ROOST and Discord have open-sourced this powerful rule engine in collaboration with [internet.dev](https://internet.dev/) to make it available for anyone who needs it.
···11-# Development
22-Welcome to the development guide for Osprey. This document will help you get started with contributing to the project.
33-44-## Reporting a Bug or Issue
55-Found a bug or have a feature request? We'd love to hear from you! When opening an issue, please use our templates:
66-77-* [Bug Report](https://github.com/roostorg/osprey/issues/new?template=bug_report.md)
88-* [Feature Request](https://github.com/roostorg/osprey/issues/new?template=feature_request.md)
99-* [Submit an Egg (new tool idea) to ROOST!](https://github.com/roostorg/osprey/issues/new?template=documentation.md)
1010-111# Setup Guide
122133This guide provides comprehensive instructions for setting up a development environment for Osprey.
144155## Prerequisites
1661717-### System Requirements
1818-1919-- **Python 3.11 or higher** - Check with `python --version`
2020-- **Git** - Version control system
217- **Operating System**: macOS, Linux, or Windows (with WSL recommended)
2222-2323-### Package Manager: UV
2424-2525-Osprey uses [uv](https://docs.astral.sh/uv/) as the Python package manager for fast, reliable dependency management.
2626-2727-#### Install UV
2828-2929-**macOS/Linux:**
3030-3131-```bash
3232-curl -LsSf https://astral.sh/uv/install.sh | sh
3333-```
3434-3535-**Windows:**
3636-3737-```bash
3838-powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
3939-```
4040-4141-**Alternative (via pip):**
4242-4343-```bash
4444-pip install uv
4545-```
4646-4747-**Verify installation:**
4848-4949-```bash
5050-uv --version
5151-```
88+- **[Python](https://www.python.org/) 3.11 or higher** (check with `python --version`)
99+- **[Git](https://git-scm.com/)** for version control
1010+- **[uv](https://docs.astral.sh/uv/)** for Python package management
1111+- **[npm](https://nodejs.org/en/download)**
52125313## Project Setup
5414···11373docker compose up -d
11474```
115757676+or using the wrapper script
7777+7878+```bash
7979+./start.sh
8080+```
8181+11682This starts up many services, including:
11783- **Osprey Worker**: The main engine that processes input events given the rules and UDFs
11884 - **Test Data Producer**: Optional with `--profile test_data`
···12288- **Postgres**: A database that the Worker, UI API, and Druid use for various reasons, such as the Postgres-backed Labels Service (in the example plugins)
12389- **Druid**: A database that consumes Osprey Worker outputs to power the UI API for real-time querying
124909191+Alternatively, you can start Osprey with `osprey-coordinator`, refer to the [Coordinator README](./example_docker_compose/run_osprey_with_coordinator/README.md) for more information
12592126126-### 6. Access the Application
127127-The UI will automatically connect to the backend services running in Docker containers.
9393+### 6. (Optional) Open ports for the UI/UI API
9494+9595+By default, the `docker-compose.yaml` binds running services to `127.0.0.1`. If you are running the docker compose on a headless machine, you may need to modify this configuration and/or make changes to your firewall, specifically for ports `5002` and `5004`.
12896129129- - Osprey UI: http://localhost:5002
130130- - Backend API: http://localhost:5004
131131- - Worker Service: http://localhost:5001
9797+For example, if you use Tailscale to access your Osprey instance, you may change `127.0.0.1:5002:5002` to `<Tailscale IP>:5002:5002`. Alternatively, if you wish for your instance to be accessible from the public internet, you may set it simply to `5002:5002` to bind to `0.0.0.0`.
132989999+Be aware that some firewalls like iptables/UFW do _not_ prevent access to ports being used by Docker networking. Not explicitly setting a bind address with only UFW as a firewall will not prevent access from the public internet unless [properly configured](https://github.com/chaifeng/ufw-docker).
133100134134-#### Plugins
101101+### 7. Access the Application
102102+103103+The UI will automatically connect to the backend services running in Docker containers.
104104+105105+- Osprey UI: [localhost:5002](http://localhost:5002)
106106+- Backend API: [localhost:5004](http://localhost:5004)
107107+- Worker Service: [localhost:5001](http://localhost:5001)
108108+109109+## Plugins
135110136111In Osprey, UDFs and output sinks are designed to be easily portable. This is done through a plugin system based on pluggy. An example plugin package has been provided for reference, see `example_plugins/register_plugins.py`:
137112···150125 # Register AST validators
151126```
152127153153-#### Rules
128128+## Rules
154129155130Rules are written in SML, some examples are provided in `example_rules/` with YAML config, the rules are mounted to the worker processes when the containers start via environment variables. ex:
156131···158133OSPREY_RULES=./example_rules uv run python3.11 osprey_worker/src/osprey/worker/cli/sinks.py run-rules-sink
159134```
160135161161-#### Test Data
136136+[More about rules →](rules.md)
137137+138138+## Test Data
162139163140Generate sample JSON actions:
164141```bash
165165-docker compose --profile test_data up kafka-test-data-producer -d
142142+docker compose --profile test_data up osprey-kafka-test-data-producer -d
166143```
167144168145Produces user login events with timestamps, user IDs, and IP addresses to `osprey.actions_input` topic.
169169-170170-## Development Workflow
171171-172172-### Branch Management
173173-174174-- **Branch naming convention**: Use `github_username/description` format (e.g., `caidanw/feature-auth`, `caidanw/fix-database-timeout`)
175175-- **Base branch**: Always branch from `main`
176176-- **Create new branch**: `git checkout -b username/feature-name`
177177-178178-### Code Quality Standards
179179-180180-#### Automated Checks
181181-182182-Every commit automatically runs:
183183-184184-1. **Trailing whitespace removal**
185185-2. **End-of-file fixing**
186186-3. **YAML/JSON/TOML validation**
187187-4. **Ruff linting and formatting**
188188-189189-#### Manual Checks
190190-191191-Before pushing, run:
192192-193193-```bash
194194-# Comprehensive linting check
195195-uv run ruff check
196196-197197-# Format all code
198198-uv run ruff format
199199-200200-# Type checking (on specific files/modules)
201201-uv run mypy osprey_worker/src/osprey_worker/lib
202202-# Or you can type check every module (this will happen in CI)
203203-uv run mypy .
204204-205205-# Run all pre-commit hooks
206206-uv run pre-commit run --all-files
207207-```
208208-209209-### Commit Standards
210210-211211-Follow [Conventional Commits](https://www.conventionalcommits.org/) format:
212212-213213-```
214214-feat: add user authentication system
215215-fix: resolve database connection timeout
216216-docs: update API documentation
217217-refactor: simplify rule evaluation logic
218218-```
219219-220220-**Examples:**
221221-222222-- `feat:` - New features
223223-- `fix:` - Bug fixes
224224-- `docs:` - Documentation changes
225225-- `refactor:` - Code refactoring
226226-- `test:` - Adding or updating tests
227227-- `chore:` - Maintenance tasks
228228-229229-### Making Changes
230230-231231-1. **Create a new branch:**
232232-233233- ```bash
234234- git checkout -b username/feature-name
235235- ```
236236-237237-2. **Make your changes**
238238-239239-3. **Run quality checks:**
240240-241241- ```bash
242242- uv run ruff check --fix
243243- uv run ruff format
244244- ```
245245-246246-4. **Test your changes** (if tests exist)
247247-248248-5. **Commit your changes:**
249249-250250- ```bash
251251- git add .
252252- git commit -m "feat: descriptive commit message"
253253- ```
254254-255255- Pre-commit hooks will run automatically and may fix formatting issues.
256256-257257-6. **Push your branch:**
258258-259259- ```bash
260260- git push origin username/feature-name
261261- ```
262262-263263-## Development Tools Overview
264264-265265-### Ruff - Linting and Formatting
266266-267267-**Purpose**: Replaces Black, isort, Flake8, and other tools
268268-269269-**Configuration**: Located in `pyproject.toml` under `[tool.ruff]`
270270-271271-**Key Rules Enabled**:
272272-273273-- `E` - pycodestyle errors
274274-- `F` - pyflakes
275275-- `I` - isort (import sorting)
276276-- `B006` - flake8-bugbear (mutable default arguments)
277277-278278-**Commands**:
279279-280280-```bash
281281-# Check for issues
282282-uv run ruff check
283283-284284-# Fix auto-fixable issues
285285-uv run ruff check --fix
286286-287287-# Format code
288288-uv run ruff format
289289-290290-# Check specific files
291291-uv run ruff check path/to/file.py
292292-```
293293-294294-### MyPy - Type Checking
295295-296296-**Purpose**: Static type checking for Python
297297-298298-**Configuration**: Located in `pyproject.toml` under `[tool.mypy]`
299299-300300-**Key Features**:
301301-302302-- Pydantic plugin support
303303-- SQLAlchemy plugin support
304304-- Relaxed strict mode (matching legacy codebase)
305305-- Ignores protobuf generated files
306306-307307-**Commands**:
308308-309309-```bash
310310-# Type check entire project
311311-uv run mypy .
312312-313313-# Type check specific files
314314-uv run mypy path/to/file.py
315315-316316-# Type check entire module
317317-uv run mypy osprey_worker/
318318-319319-# Check with verbose output
320320-uv run mypy --show-traceback path/to/file.py
321321-```
322322-323323-### Pre-commit - Git Hooks
324324-325325-**Purpose**: Automated quality checks before commits
326326-327327-**Configuration**: Located in `.pre-commit-config.yaml`
328328-329329-**Commands**:
330330-331331-```bash
332332-# Run all hooks on staged files
333333-uv run pre-commit run
334334-335335-# Run all hooks on all files
336336-uv run pre-commit run --all-files
337337-338338-# Run specific hook
339339-uv run pre-commit run ruff
340340-341341-# Update hook versions
342342-uv run pre-commit autoupgrade
343343-344344-# Bypass hooks (emergency only)
345345-git commit --no-verify
346346-```
347347-348348-### UV - Package Management
349349-350350-**Purpose**: Fast Python package manager and environment management
351351-352352-**Key Commands**:
353353-354354-```bash
355355-# Install dependencies
356356-uv sync
357357-358358-# Add new dependency
359359-uv add package-name
360360-361361-# Add development dependency
362362-uv add --group dev package-name
363363-364364-# Remove dependency
365365-uv remove package-name
366366-367367-# Run command in environment
368368-uv run command-name
369369-370370-# Update dependencies
371371-uv lock --upgrade
372372-```
373373-374374-## Troubleshooting
375375-376376-### Common Issues
377377-378378-#### "uv: command not found"
379379-380380-**Solution**: Install uv using the installation script or pip:
381381-382382-```bash
383383-curl -LsSf https://astral.sh/uv/install.sh | sh
384384-# Then restart your terminal
385385-```
386386-387387-#### Pre-commit hooks failing
388388-389389-**Solution**: Run hooks manually to see detailed errors:
390390-391391-```bash
392392-uv run pre-commit run --all-files
393393-```
394394-395395-#### MyPy errors on protobuf files
396396-397397-**Solution**: Protobuf generated files are excluded in configuration. If you see errors, check that files match the exclusion patterns in `pyproject.toml`.
398398-399399-#### Import errors during type checking
400400-401401-**Solution**: Ensure all dependencies are installed:
402402-403403-```bash
404404-uv sync
405405-```
406406-407407-### Getting Help
408408-409409-1. **Check this documentation** for common setup issues
410410-2. **Review error messages** carefully - they often contain solutions
411411-3. **Run commands with verbose flags** for more detailed output
412412-4. **Check configuration files** (`pyproject.toml`, `.pre-commit-config.yaml`)
413413-414414-## Next Steps
415415-416416-- Check the [contributing guidelines](https://github.com/roostorg/.github/blob/main/CONTRIBUTING.md) for project-specific rules
417417-- Explore the codebase structure in `osprey_worker/`, `osprey_common/`, and `osprey_rpc/`
418418-419419-## IDE Setup Recommendations
420420-421421-### VS Code
422422-423423-Install these extensions for the best development experience:
424424-425425-- **Python** (Microsoft)
426426-- **Ruff** (Astral Software)
427427-- **MyPy Type Checker** (Microsoft)
428428-429429-**Settings** (add to `.vscode/settings.json`):
430430-431431-```json
432432-{
433433- "python.defaultInterpreterPath": ".venv/bin/python",
434434- "ruff.enable": true,
435435- "ruff.organizeImports": true,
436436- "python.analysis.typeCheckingMode": "basic"
437437-}
438438-```
439439-440440-This completes the development setup! You're now ready to contribute to Osprey.
+8
docs/README.md
···11113. **[Development Guide](DEVELOPMENT.md)** - Set up your development environment and get started using Osprey
12124. **[Osprey UI](UI.md)** - Understand how to use and navigate the UI
13131414+## Reporting a Bug or Issue
1515+1616+Found a bug or have a feature request? We'd love to hear from you! When opening an issue, please use our templates:
1717+1818+* [Bug Report](https://github.com/roostorg/osprey/issues/new?template=bug_report.md)
1919+* [Feature Request](https://github.com/roostorg/osprey/issues/new?template=feature_request.md)
2020+* [Submit an Egg (new tool idea) to ROOST!](https://github.com/roostorg/osprey/issues/new?template=documentation.md)
2121+1422## Need Help?
15231624If you can't find what you're looking for in these documents, please:
···11-# **Osprey User Interface Guide**
22-11+# Osprey User Interface Guide
3243## Getting Started
54···11101211The Osprey UI has several pages accessible by a left-hand menu:
13121414-
1515-1313+
16141715Home will bring you to the default page of Osprey, with three main columns.
18161919-**NOTE: The Event Stream in the right column is not yet in v0, and will be available before or in v1.**
2020-2121-
1717+
22182319### Left Column: Query
24202525-#### **Query Box**
2121+#### Query Box
26222723The Osprey Query UI uses the same SML syntax as rules, but for searching and filtering near-real-time and historical data rather than creating new rules. Using the test data generator, you can try writing a query to look for an action called “create\_post” specifically from a given User ID.
2828-
29242525+
30263127You can also use a UDF in your query. If you ever forget what a UDF does, you can hover on the information symbol for a tip:
3232-
2828+2929+
33303431A query can be run against a time window ranging from the last second to the last 3 months (and also a custom range):
3535-
36323333+
37343835The Osprey UI is designed to be dynamic and update in real-time. If any other component in the other two columns is interacted with, the query will automatically update and vice versa. The query also automatically populates the URL. This can be handy for sharing a specific query with someone on a team, but may present privacy risks.
39364040-
3737+
41384242-#### **History**
3939+#### History
43404441Every query is logged in the Query History view, and there is a dropdown filter to only show queries that you have run.
45424643When you hover over the query, it will also show the Top N Charts used during the query session (more on that below).
47444848-
4545+
49465047The Query History can also be accessed and seen in a different format via the left-side menu. From here you can filter by the user who ran the query, view the original query, and run it using the same time range the original query used.
51485252-
4949+
53505454-#### **Saved Queries**
5151+#### Saved Queries
55525653If there are specific queries that are used often, Osprey provides the ability to save a query:
57545858-
5555+
59566057The user who initiated the query and when the query was first run is logged as part of the Saved Query. Saved Queries can also be accessed via the left-side menu. The user who saved the query and what time it was saved is logged and visible. There is a drop-down menu at the top to filter saved queries by users.
61586262-
5959+
63606461### Middle Column: Charts
65626663The middle column in Osprey shows two types of charts: **Time Series** and **Top N Results**. Both sections provide the ability to add extra charts to see different slices of time or types of top results.
67646868-
6565+
69667070-#### **Time Series Chart**
6767+#### Time Series Chart
71687269The Time Series chart shows a visualization of the results in the query over a period of time. The time ranges include:
7370···8077* Month
81788279Hovering over a bar in the time series chart shows how many events took place during that time.
8383-
84808181+
85828683There is also a time and date picker above the time series chart where you can set a custom range:
8787-
8484+8585+
88868987An extra table can be added for another view of a different unit of time. To get rid of the table, you can “[yeet](https://www.urbandictionary.com/define.php?term=Yeet) it”.
9090-
91888989+
92909393-#### **Top N Results**
9191+#### Top N Results
94929593Adding a Top N Results table populates a table with the top results for the results of the query. You can view and assign labels to a specific entity by hovering over it and clicking “Edit Labels”
9696-
97949898-**NOTE: Labels are not yet in v0**
9595+
9996100100-
9797+
1019810299You can also select PoP (Period over Period) to compare the query results with results from a window of time in the past to see the delta.
103100104104-
101101+
105102106103### Right Column: Event Stream
107104108108-**The Event Stream is not yet in v0, and will be available before or in v1.**
109109-110105The Event Stream is essentially Osprey's "live feed" and investigation dashboard where security teams can:
111106112107* Monitor real-time activity
···116111* Drill down into specific users/entities
117112118113It provides a more detailed view of each event that matches the query. The Event Stream can show metadata related to accounts that can link to other internal tools that provide detailed information about an account and/or further enforcement actions.
119119-
120120-114114+
121115122116The event stream is also viewable in a card format vs a list format (list format shown in the screenshot).
123117124118Osprey users may have personal preferences on how to do investigations and what information is most helpful for them. Osprey makes it easy to customize the types of information shown in the Event Stream by clicking “Summary Features”
125125-
126126-119119+
127120128121### Labeling
129122130130-**Note: Labels are not yet in v0, but will come in v1**
131123Any unique entity can be labeled in the Osprey UI. This manual labeling tool is used by Safety teams to tag individual entities (users, IPs, emails, etc.) with labels. Labels are essentially the manual annotation tool that feeds into Osprey's automated rule system, allowing human judgment to enhance machine detection. Labels can be positive, negative, or neutral. Examples:
132124133125**Negative Labels: Harmful/problematic behavior**
···139131**Neutral Labels: Informational tags**
140132* Examples: "new\_user", "from\_mobile", "beta\_tester"
141133142142-143143-Below are examples of a new label interface from v0, and an example from Discord’s usage of labels (coming in v1).
144144-
145145-
146146-134134+
135135+
147136148137### UDF Documentation
149138150139The UDF Documentation page can be accessed via the left-side menu. It dynamically updates based on the code, so any new UDFs added will show up on this page. This page essentially serves as the "API reference" for the SML language, making it easy for users to discover and properly use all available functions when writing rules and queries.
151151-
140140+
152141153142This page can be used as a manual for writing SML rules or queries, guide for understanding parameter types and requirements, and act as a plugin discovery portal to explore what custom UDFs are loaded.
154143155144### Bulk Labeling
156156-157157-**Note: Since Bulk Labeling relies on Labels, it does not yet work in v0.**
158145159146There are two ways to bulk label items in Osprey: the left-side menu and via the chart column. In this example, you can bulk label all the users that have posted a message that is not empty:
160160-
161147148148+
162149163150**Bulk labels can be dangerous if there’s a false positive\!** Osprey provides a counter of how many unique entities are about to be bulk labeled at the top. Labels can be positive, negative, or neutral. A reason must be provided when labeling anything. Each bulk job will create a unique task ID and log the user who initiated the bulk job, the status of the bulk labeling, and a link to the query that the bulk job originated from.
164151165152To view all bulk labeling jobs that have been done, click into “Bulk Job History” from the left-side menu. You’ll need the unique task ID to look up a bulk job.
166166-
167153154154+
168155169156### Rule Visualizer
170170-171171-**Note: Since the Rule Visualizer relies on Labels, it does not yet work in v0.**
172157173158The Rule Visualizer shows how upstream labels, rules, and downstream labels interact with one another. To use it, select an Action or a Label. A graph view will appear showing the relations between rules and labels.
174159···176161* **Blue square:** rule
177162* **Green circle:** label downstream of a rule
178163179179-
180180-164164+
181165182166### Query Syntax
183167184184-#### **Actions**
168168+#### Actions
185169186170Actions are events that are sent to Osprey. An event is simply something that happens. When a user does something like create a post, send a message, change their username, etc an event happens to represent that. There are probably a lot of events emitted in your org, and Osprey doesn’t need to consume all of them.
187171···212196* Post Text
213197* Internet Service Provider
214198215215-#### **Effects**
199199+#### Effects
216200217201Effects can be triggered when one or many rules are evaluated to be true. These are validated and handled in aggregate at the end of an execution output. For example, an effect might apply a label to an entity marking it as a “Spammer”.
218202···227211MessageText != Null
228212```
229213230230-#### **Combining Conditions**
214214+#### Combining Conditions
231215232216Let’s say you’re looking for any matches where a user tried to login more than 3 times. You can create a query to check for two types of data fields: “EventType” and “LoginAttempts”.
233217···242226 (UserId == 123) or (UserId == 456)
243227```
244228245245-#### **Using UDFs in Queries**
229229+#### Using UDFs in Queries
246230247247-UDFs (read more [here](https://github.com/roostorg/osprey/blob/f16da6e5c32ae124c3cc6e2d7efded7cea1ac726/docs/rules.md#user-defined-functions-udfs)) are a powerful part of queries. Once you define a UDF with the specific desired logic, you can reference it in a query.
231231+UDFs (read more [here](rules.md#user-defined-functions-udfs)) are a powerful part of queries. Once you define a UDF with the specific desired logic, you can reference it in a query.
248232249249-**NOTE: If you try to query a UDF that doesn’t exist, Osprey will silently fail with a 500 error.**
233233+> [!NOTE]
234234+> If you try to query a UDF that doesn’t exist, Osprey will silently fail with a 500 error.
250235251236```py
252237 # Text search
···257242 ListLength(list=UserConnections) > 10
258243```
259244260260-#### **Label Queries**
261261-262262-**Important Note: Labels are not yet in v0, so these will not work in the UI.**
245245+#### Label Queries
263246264247Since the UI searches across actions/events:
265248266266-* **Don't use:** HasLabel() \- won't work in Query UI
267267-* **Use instead**: DidAddLabel() \- shows when an action added a label
249249+* **Don't use:** HasLabel() - won't work in Query UI
250250+* **Use instead**: DidAddLabel() - shows when an action added a label
268251269252```py
270253 # Find actions that added specific labels
271254 DidAddLabel(entity_type="UserId", label_name="likely_spammer")
272255 DidAddLabel(entity_type="IpAddress", label_name="suspicious")
273256```
274274-275275-###
276257277258### Example Queries
278259
+20
docs/development/ide.md
···11+# IDE Setup Recommendations
22+33+## VS Code
44+55+Install these extensions for the best development experience:
66+77+- **Python** (Microsoft)
88+- **Ruff** (Astral Software)
99+- **MyPy Type Checker** (Microsoft)
1010+1111+**Settings** (add to `.vscode/settings.json`):
1212+1313+```json
1414+{
1515+ "python.defaultInterpreterPath": ".venv/bin/python",
1616+ "ruff.enable": true,
1717+ "ruff.organizeImports": true,
1818+ "python.analysis.typeCheckingMode": "basic"
1919+}
2020+```
+110
docs/development/tools.md
···11+# Development Tools Overview
22+33+## Ruff - Linting and Formatting
44+55+**Purpose**: Replaces Black, isort, Flake8, and other tools
66+77+**Configuration**: Located in `pyproject.toml` under `[tool.ruff]`
88+99+**Key Rules Enabled**:
1010+1111+- `E` - pycodestyle errors
1212+- `F` - pyflakes
1313+- `I` - isort (import sorting)
1414+- `B006` - flake8-bugbear (mutable default arguments)
1515+1616+**Commands**:
1717+1818+```bash
1919+# Check for issues
2020+uv run ruff check
2121+2222+# Fix auto-fixable issues
2323+uv run ruff check --fix
2424+2525+# Format code
2626+uv run ruff format
2727+2828+# Check specific files
2929+uv run ruff check path/to/file.py
3030+```
3131+3232+## MyPy - Type Checking
3333+3434+**Purpose**: Static type checking for Python
3535+3636+**Configuration**: Located in `pyproject.toml` under `[tool.mypy]`
3737+3838+**Key Features**:
3939+4040+- Pydantic plugin support
4141+- SQLAlchemy plugin support
4242+- Relaxed strict mode (matching legacy codebase)
4343+- Ignores protobuf generated files
4444+4545+**Commands**:
4646+4747+```bash
4848+# Type check entire project
4949+uv run mypy .
5050+5151+# Type check specific files
5252+uv run mypy path/to/file.py
5353+5454+# Type check entire module
5555+uv run mypy osprey_worker/
5656+5757+# Check with verbose output
5858+uv run mypy --show-traceback path/to/file.py
5959+```
6060+6161+## Pre-commit - Git Hooks
6262+6363+**Purpose**: Automated quality checks before commits
6464+6565+**Configuration**: Located in `.pre-commit-config.yaml`
6666+6767+**Commands**:
6868+6969+```bash
7070+# Run all hooks on staged files
7171+uv run pre-commit run
7272+7373+# Run all hooks on all files
7474+uv run pre-commit run --all-files
7575+7676+# Run specific hook
7777+uv run pre-commit run ruff
7878+7979+# Update hook versions
8080+uv run pre-commit autoupgrade
8181+8282+# Bypass hooks (emergency only)
8383+git commit --no-verify
8484+```
8585+8686+## UV - Package Management
8787+8888+**Purpose**: Fast Python package manager and environment management
8989+9090+**Key Commands**:
9191+9292+```bash
9393+# Install dependencies
9494+uv sync
9595+9696+# Add new dependency
9797+uv add package-name
9898+9999+# Add development dependency
100100+uv add --group dev package-name
101101+102102+# Remove dependency
103103+uv remove package-name
104104+105105+# Run command in environment
106106+uv run command-name
107107+108108+# Update dependencies
109109+uv lock --upgrade
110110+```
+32
docs/development/troubleshooting.md
···11+# Troubleshooting
22+33+## Common Issues
44+55+### "uv: command not found"
66+77+**Solution**: Install uv using the installation script or pip:
88+99+```bash
1010+curl -LsSf https://astral.sh/uv/install.sh | sh
1111+# Then restart your terminal
1212+```
1313+1414+### Pre-commit hooks failing
1515+1616+**Solution**: Run hooks manually to see detailed errors:
1717+1818+```bash
1919+uv run pre-commit run --all-files
2020+```
2121+2222+### MyPy errors on protobuf files
2323+2424+**Solution**: Protobuf generated files are excluded in configuration. If you see errors, check that files match the exclusion patterns in `pyproject.toml`.
2525+2626+### Import errors during type checking
2727+2828+**Solution**: Ensure all dependencies are installed:
2929+3030+```bash
3131+uv sync
3232+```
+92
docs/development/workflow.md
···11+# Development Workflow
22+33+## Branch Management
44+55+- **Branch naming convention**: Use `github_username/description` format (e.g., `caidanw/feature-auth`, `caidanw/fix-database-timeout`)
66+- **Base branch**: Always branch from `main`
77+- **Create new branch**: `git checkout -b username/feature-name`
88+99+## Code Quality Standards
1010+1111+### Automated Checks
1212+1313+Every commit automatically runs:
1414+1515+1. **Trailing whitespace removal**
1616+2. **End-of-file fixing**
1717+3. **YAML/JSON/TOML validation**
1818+4. **Ruff linting and formatting**
1919+2020+### Manual Checks
2121+2222+Before pushing, run:
2323+2424+```bash
2525+# Comprehensive linting check
2626+uv run ruff check
2727+2828+# Format all code
2929+uv run ruff format
3030+3131+# Type checking (on specific files/modules)
3232+uv run mypy osprey_worker/src/osprey_worker/lib
3333+# Or you can type check every module (this will happen in CI)
3434+uv run mypy .
3535+3636+# Run all pre-commit hooks
3737+uv run pre-commit run --all-files
3838+```
3939+4040+## Commit Standards
4141+4242+Follow [Conventional Commits](https://www.conventionalcommits.org/) format:
4343+4444+```
4545+feat: add user authentication system
4646+fix: resolve database connection timeout
4747+docs: update API documentation
4848+refactor: simplify rule evaluation logic
4949+```
5050+5151+**Examples:**
5252+5353+- `feat:` - New features
5454+- `fix:` - Bug fixes
5555+- `docs:` - Documentation changes
5656+- `refactor:` - Code refactoring
5757+- `test:` - Adding or updating tests
5858+- `chore:` - Maintenance tasks
5959+6060+## Making Changes
6161+6262+1. **Create a new branch:**
6363+6464+ ```bash
6565+ git checkout -b username/feature-name
6666+ ```
6767+6868+2. **Make your changes**
6969+7070+3. **Run quality checks:**
7171+7272+ ```bash
7373+ uv run ruff check --fix
7474+ uv run ruff format
7575+ ```
7676+7777+4. **Test your changes** (if tests exist)
7878+7979+5. **Commit your changes:**
8080+8181+ ```bash
8282+ git add .
8383+ git commit -m "feat: descriptive commit message"
8484+ ```
8585+8686+ Pre-commit hooks will run automatically and may fix formatting issues.
8787+8888+6. **Push your branch:**
8989+9090+ ```bash
9191+ git push origin username/feature-name
9292+ ```
+20
docs/docs.md
···11+# Contribute to these docs
22+33+This documentation site is built using [mdBook](https://rust-lang.github.io/mdBook/) and deployed to GitHub Pages. Changes merged into the `main` branch will automatically be built and deployed.
44+55+Documentation can be edited directly in the GitHub web UI for existing pages, or by selecting the **🖉 Suggest an edit** icon at the top of the docs site. To create a new page, be sure to update `SUMMARY.md` as well. Once you're done with your changes, open a pull request for review.
66+77+To understand more about how mdBook works, learn about the [anatomy of a book](https://rust-lang.github.io/mdBook/guide/creating.html#anatomy-of-a-book).
88+99+## Developing locally
1010+1111+To build the site locally, clone this repository and install `mdbook` (follow the [official installation instructions](https://rust-lang.github.io/mdBook/guide/installation.html)).
1212+1313+Once installed, use the `mdbook` command-line tool from the root of this repo. For example, to automatically start watching, building, and serving the site:
1414+1515+```shell
1616+mdbook serve
1717+```
1818+1919+Then make your changes, preview them in your web browser (at [http://localhost:3000](http://localhost:3000) by default), commit, push, and open a pull request like any other git project.
2020+
+3-8
docs/rules.md
···11-# Osprey Docs
22-33-# Osprey Docs
11+# Osprey Rules
4253
6477-## Rules
88-99-### Creating Rules
55+## Creating Rules
106117Rules in Osprey are written in `Some Madeup Language (SML)` and follow most syntax conventions present in the Osprey Query UI. SML is a subset of Python with additional restrictions to make the rules simpler to craft.
128···5854)
5955```
60566161-### Instrumenting Rules with WhenRules
5757+## Instrumenting Rules with WhenRules
62586359The `WhenRules()` function allows for the connection of rules with external services, declarations or internal label modifications by listing Rule objects in sequence within the `rules_any=[]` parameter and `EffectBase`. By default, operators and designers can utilize UDFs with predefined effects such as `DeclareVerdict()`, `LabelAdd()`, and `LabelRemove()` on positive rule evaluations.
6460···179175UDF outputs can also implement the `CustomExtractedFeature` interface - which get persisted in the outputs for the UI. `EffectToCustomExtractedFeatureBase` can also be used when effects need additional processing for use in the UI.
180176181177## Labels
182182-**NOTE: Labels are currently not in v0, so users will be unable to add or edit labels via the UI**
183178184179Labels are a standard plugin that enable stateful rules, and touch many parts of Osprey. They are effectively tags on various entities, which can be arbitrarily defined.
185180