Mirror of https://github.com/roostorg/osprey
github.com/roostorg/osprey
Add detailed documentation for UI navigation
Add detailed docs for:
- how to navigate the UI
- what different parts of the UI do
- query syntax
- added warnings on what's not going to be working in v0
- added intdev to recognition
+299
-12
+3
-3
README.md
+3
-3
README.md
···
1
1
# ROOST - Osprey
2
2
3
-
<img width="36" height="46" alt="Copy of ROOST-Mark-Yellow" src="/images/ROOST-Horizontal-Yellow .png" />
3
+
<img width="150" height="191" alt="Copy of ROOST-Mark-Yellow" src="/images/ROOST-Horizontal-Yellow .png" />
4
4
5
5
# Welcome to Osprey
6
6
···
87
87
# Test pre-commit hooks
88
88
uv run pre-commit run --all-files
89
89
90
-
5. ** Start Services: **
90
+
5. **Start Services:**
91
91
```bash
92
92
docker compose up -d
93
93
```
···
101
101
102
102
103
103
## Recognition
104
-
Discord uses Osprey to quickly detect and remove new types of harm that put users at risk. Rather than leaving other platforms to build similar tools from scratch, ROOST and Discord have open-sourced this powerful rule engine making it available for anyone who needs it.
104
+
Discord uses Osprey to quickly detect and remove new types of harm that put users at risk. Rather than leaving other platforms to build similar tools from scratch, ROOST and Discord have open-sourced this powerful rule engine in collaboration with [internet.dev](https://internet.dev/) to make it available for anyone who needs it.
+1
-1
docs/README.md
+1
-1
docs/README.md
···
9
9
1. **[Code of Conduct](CODE_OF_CONDUCT.md)** - Understand our community standards
10
10
2. **[Contributing Guidelines](CONTRIBUTING.md)** - Learn how to contribute
11
11
3. **[Development Guide](DEVELOPMENT.md)** - Set up your development environment and get started using Osprey
12
-
4. **[Osprey UI](https://github.com/roostorg/osprey/blob/main/osprey_ui/README.md)** - Understand how to get the UI running
12
+
4. **[Osprey UI](UI.md)** - Understand how to use and navigate the UI
13
13
14
14
## Need Help?
15
15
+292
docs/UI.md
+292
docs/UI.md
···
1
+
# **Osprey User Interface Guide**
2
+
3
+
4
+
## Getting Started
5
+
6
+
```bash
7
+
cd osprey_ui
8
+
npm install
9
+
npm start
10
+
```
11
+
12
+
The Osprey UI has several pages accessible by a left-hand menu:
13
+
14
+
15
+
16
+
17
+
Home will bring you to the default page of Osprey, with three main columns.
18
+
19
+
**NOTE: The Event Stream in the right column is not yet in v0, and will be available before or in v1.**
20
+
21
+
22
+
23
+
### Left Column: Query
24
+
25
+
#### **Query Box**
26
+
27
+
The Osprey Query UI uses the same SML syntax as rules, but for searching and filtering near-real-time and historical data rather than creating new rules. Using the test data generator, you can try writing a query to look for an action called “create\_post” specifically from a given User ID.
28
+
29
+
30
+
31
+
You can also use a UDF in your query. If you ever forget what a UDF does, you can hover on the information symbol for a tip:
32
+
33
+
34
+
A query can be run against a time window ranging from the last second to the last 3 months (and also a custom range):
35
+
36
+
37
+
38
+
The Osprey UI is designed to be dynamic and update in real-time. If any other component in the other two columns is interacted with, the query will automatically update and vice versa. The query also automatically populates the URL. This can be handy for sharing a specific query with someone on a team, but may present privacy risks.
39
+
40
+
41
+
42
+
#### **History**
43
+
44
+
Every query is logged in the Query History view, and there is a dropdown filter to only show queries that you have run.
45
+
46
+
When you hover over the query, it will also show the Top N Charts used during the query session (more on that below).
47
+
48
+
49
+
50
+
The Query History can also be accessed and seen in a different format via the left-side menu. From here you can filter by the user who ran the query, view the original query, and run it using the same time range the original query used.
51
+
52
+
53
+
54
+
#### **Saved Queries**
55
+
56
+
If there are specific queries that are used often, Osprey provides the ability to save a query:
57
+
58
+
59
+
60
+
The user who initiated the query and when the query was first run is logged as part of the Saved Query. Saved Queries can also be accessed via the left-side menu. The user who saved the query and what time it was saved is logged and visible. There is a drop-down menu at the top to filter saved queries by users.
61
+
62
+
63
+
64
+
### Middle Column: Charts
65
+
66
+
The middle column in Osprey shows two types of charts: **Time Series** and **Top N Results**. Both sections provide the ability to add extra charts to see different slices of time or types of top results.
67
+
68
+
69
+
70
+
#### **Time Series Chart**
71
+
72
+
The Time Series chart shows a visualization of the results in the query over a period of time. The time ranges include:
73
+
74
+
* Minute
75
+
* Fifteen minutes
76
+
* Half hour
77
+
* Hour
78
+
* Day
79
+
* Week
80
+
* Month
81
+
82
+
Hovering over a bar in the time series chart shows how many events took place during that time.
83
+
84
+
85
+
86
+
There is also a time and date picker above the time series chart where you can set a custom range:
87
+
88
+
89
+
An extra table can be added for another view of a different unit of time. To get rid of the table, you can “[yeet](https://www.urbandictionary.com/define.php?term=Yeet) it”.
90
+
91
+
92
+
93
+
#### **Top N Results**
94
+
95
+
Adding a Top N Results table populates a table with the top results for the results of the query. You can view and assign labels to a specific entity by hovering over it and clicking “Edit Labels”
96
+
97
+
98
+
**NOTE: Labels are not yet in v0**
99
+
100
+
101
+
102
+
You can also select PoP (Period over Period) to compare the query results with results from a window of time in the past to see the delta.
103
+
104
+
105
+
106
+
### Right Column: Event Stream
107
+
108
+
**The Event Stream is not yet in v0, and will be available before or in v1.**
109
+
110
+
The Event Stream is essentially Osprey's "live feed" and investigation dashboard where security teams can:
111
+
112
+
* Monitor real-time activity
113
+
* Search historical events using SML queries
114
+
* Investigate suspicious patterns
115
+
* Track rule execution results
116
+
* Drill down into specific users/entities
117
+
118
+
It provides a more detailed view of each event that matches the query. The Event Stream can show metadata related to accounts that can link to other internal tools that provide detailed information about an account and/or further enforcement actions.
119
+
120
+
121
+
122
+
The event stream is also viewable in a card format vs a list format (list format shown in the screenshot).
123
+
124
+
Osprey users may have personal preferences on how to do investigations and what information is most helpful for them. Osprey makes it easy to customize the types of information shown in the Event Stream by clicking “Summary Features”
125
+
126
+
127
+
128
+
### Labeling
129
+
130
+
**Note: Labels are not yet in v0, but will come in v1**
131
+
Any unique entity can be labeled in the Osprey UI. This manual labeling tool is used by Safety teams to tag individual entities (users, IPs, emails, etc.) with labels. Labels are essentially the manual annotation tool that feeds into Osprey's automated rule system, allowing human judgment to enhance machine detection. Labels can be positive, negative, or neutral. Examples:
132
+
133
+
**Negative Labels: Harmful/problematic behavior**
134
+
* Examples: "spammer", "bot", "banned", "suspicious"
135
+
136
+
**Positive Labels: Good/trusted behavior**
137
+
* Examples: "verified", "trusted", "premium\_user"
138
+
139
+
**Neutral Labels: Informational tags**
140
+
* Examples: "new\_user", "from\_mobile", "beta\_tester"
141
+
142
+
143
+
Below are examples of a new label interface from v0, and an example from Discord’s usage of labels (coming in v1).
144
+
145
+
146
+
147
+
148
+
### UDF Documentation
149
+
150
+
The UDF Documentation page can be accessed via the left-side menu. It dynamically updates based on the code, so any new UDFs added will show up on this page. This page essentially serves as the "API reference" for the SML language, making it easy for users to discover and properly use all available functions when writing rules and queries.
151
+
152
+
153
+
This page can be used as a manual for writing SML rules or queries, guide for understanding parameter types and requirements, and act as a plugin discovery portal to explore what custom UDFs are loaded.
154
+
155
+
### Bulk Labeling
156
+
157
+
**Note: Since Bulk Labeling relies on Labels, it does not yet work in v0.**
158
+
159
+
There are two ways to bulk label items in Osprey: the left-side menu and via the chart column. In this example, you can bulk label all the users that have posted a message that is not empty:
160
+
161
+
162
+
163
+
**Bulk labels can be dangerous if there’s a false positive\!** Osprey provides a counter of how many unique entities are about to be bulk labeled at the top. Labels can be positive, negative, or neutral. A reason must be provided when labeling anything. Each bulk job will create a unique task ID and log the user who initiated the bulk job, the status of the bulk labeling, and a link to the query that the bulk job originated from.
164
+
165
+
To view all bulk labeling jobs that have been done, click into “Bulk Job History” from the left-side menu. You’ll need the unique task ID to look up a bulk job.
166
+
167
+
168
+
169
+
### Rule Visualizer
170
+
171
+
**Note: Since the Rule Visualizer relies on Labels, it does not yet work in v0.**
172
+
173
+
The Rule Visualizer shows how upstream labels, rules, and downstream labels interact with one another. To use it, select an Action or a Label. A graph view will appear showing the relations between rules and labels.
174
+
175
+
* **Red circle:** label upstream of a rule
176
+
* **Blue square:** rule
177
+
* **Green circle:** label downstream of a rule
178
+
179
+
180
+
181
+
182
+
### Query Syntax
183
+
184
+
#### **Actions**
185
+
186
+
Actions are events that are sent to Osprey. An event is simply something that happens. When a user does something like create a post, send a message, change their username, etc an event happens to represent that. There are probably a lot of events emitted in your org, and Osprey doesn’t need to consume all of them.
187
+
188
+
##### Features
189
+
190
+
A feature is any variable in the global namespace in Osprey. All features must be uniquely named. However, prefixing a \`\_\` at the start of a variable name prevents it from being exported as a feature and keeps the variable within the local file’s namespace.
191
+
192
+
Features are outputs of Osprey executions. Downstream, they are sent to and indexed by Druid, so users can query for events based on feature names later, i.e. \`UserEmail \== '`` despicable@example.com` ``.
193
+
194
+
```py
195
+
UserId: Entity[int] = EntityJson(type='User', path='$.user.id', coerce_type=True)
196
+
197
+
UserEmail: str = JsonData(type='Email', path='$.user.email', required=False)
198
+
```
199
+
200
+
In the example above, both `UserId` and `UserEmail` are features.
201
+
202
+
##### Entities
203
+
204
+
Entities are a special type of Feature. All entities are features, but not all features are entities. An entity can have effects applied to it, such as labels, classifications, or signals). Every entity has a type that determines which effects can be applied to it based on static validations.
205
+
206
+
Entities get special treatment within the Osprey UI. Clicking on an entity in the tool will take you to an Entity View, providing a deep dive into its history.
207
+
208
+
This could be:
209
+
210
+
* User ID
211
+
* IP Address
212
+
* Post Text
213
+
* Internet Service Provider
214
+
215
+
#### **Effects**
216
+
217
+
Effects can be triggered when one or many rules are evaluated to be true. These are validated and handled in aggregate at the end of an execution output. For example, an effect might apply a label to an entity marking it as a “Spammer”.
218
+
219
+
### Basic Query Structure
220
+
221
+
Let’s say you have a data field called “EventType” and one of the events is to “create\_post”. You are looking for posts by a user whose ID is 12345 and you don’t want to see empty posts. Your query would look like:
222
+
223
+
```py
224
+
# Simple field comparisons
225
+
EventType == "create_post"
226
+
UserId == 12345
227
+
MessageText != Null
228
+
```
229
+
230
+
#### **Combining Conditions**
231
+
232
+
Let’s say you’re looking for any matches where a user tried to login more than 3 times. You can create a query to check for two types of data fields: “EventType” and “LoginAttempts”.
233
+
234
+
If you’re looking for multiple types of events in “EventType” like for posts AND messages, you can use brackets to list the types of events.
235
+
236
+
```py
237
+
# Multiple conditions (AND)
238
+
EventType == "user_login" and LoginAttempts >= 3
239
+
240
+
# OR conditions
241
+
EventType in ["create_post", "send_message"]
242
+
(UserId == 123) or (UserId == 456)
243
+
```
244
+
245
+
#### **Using UDFs in Queries**
246
+
247
+
UDFs (read more [here](https://github.com/roostorg/osprey/blob/f16da6e5c32ae124c3cc6e2d7efded7cea1ac726/docs/rules.md#user-defined-functions-udfs)) are a powerful part of queries. Once you define a UDF with the specific desired logic, you can reference it in a query.
248
+
249
+
**NOTE: If you try to query a UDF that doesn’t exist, Osprey will silently fail with a 500 error.**
250
+
251
+
```py
252
+
# Text search
253
+
TextContains(text=PostContent, phrase="spam")
254
+
RegexMatch(target=MessageText, pattern="(buy|sell|deal)")
255
+
256
+
# List operations
257
+
ListLength(list=UserConnections) > 10
258
+
```
259
+
260
+
#### **Label Queries**
261
+
262
+
**Important Note: Labels are not yet in v0, so these will not work in the UI.**
263
+
264
+
Since the UI searches across actions/events:
265
+
266
+
* **Don't use:** HasLabel() \- won't work in Query UI
267
+
* **Use instead**: DidAddLabel() \- shows when an action added a label
268
+
269
+
```py
270
+
# Find actions that added specific labels
271
+
DidAddLabel(entity_type="UserId", label_name="likely_spammer")
272
+
DidAddLabel(entity_type="IpAddress", label_name="suspicious")
273
+
```
274
+
275
+
###
276
+
277
+
### Example Queries
278
+
279
+
```py
280
+
# Find suspicious login attempts:
281
+
EventType == "user_login" and LoginAttempts >= 5
282
+
283
+
# Find posts containing specific words:
284
+
EventType == "create_post" and TextContains(text=PostContent, phrase="urgent")
285
+
286
+
# Find users who were flagged:
287
+
DidAddLabel(entity_type="UserId", label_name="flagged")
288
+
289
+
# Complex search:
290
+
EventType == "send_message" and
291
+
RegexMatch(target=MessageText, pattern="(click|link|urgent)") and
292
+
not DidAddLabel(entity_type="UserId", label_name="verified")
+1
docs/rules.md
+1
docs/rules.md
···
179
179
UDF outputs can also implement the `CustomExtractedFeature` interface - which get persisted in the outputs for the UI. `EffectToCustomExtractedFeatureBase` can also be used when effects need additional processing for use in the UI.
180
180
181
181
## Labels
182
+
**NOTE: Labels are currently not in v0, so users will be unable to add or edit labels via the UI**
182
183
183
184
Labels are a standard plugin that enable stateful rules, and touch many parts of Osprey. They are effectively tags on various entities, which can be arbitrarily defined.
184
185
+1
-1
example_data/template.json
+1
-1
example_data/template.json
···
1
-
{"send_time": "$timestamp","data": {"action_id": "$action_id","action_name": "create_post","data": {"user_id": "$user_id","ip_address": "$ip_address","event_type": "create_post","post": {"text": "$text"}}}}
1
+
{"send_time": "$timestamp","data": {"action_id": "$action_id","action_name": "create_post","data": {"user_id": "$user_id","ip_address": "$ip_address","event_type": "create_post","post": {"text": "$text"}}}}
images/add-labels.png
images/add-labels.png
This is a binary file and will not be displayed.
images/bulk-job-history.png
images/bulk-job-history.png
This is a binary file and will not be displayed.
images/bulk-label.png
images/bulk-label.png
This is a binary file and will not be displayed.
images/complete-label.png
images/complete-label.png
This is a binary file and will not be displayed.
images/empty-label.png
images/empty-label.png
This is a binary file and will not be displayed.
images/hover-query-history.png
images/hover-query-history.png
This is a binary file and will not be displayed.
images/hover-time-series.png
images/hover-time-series.png
This is a binary file and will not be displayed.
images/multiple-time-series.png
images/multiple-time-series.png
This is a binary file and will not be displayed.
images/multiple-top-charts.png
images/multiple-top-charts.png
This is a binary file and will not be displayed.
images/osprey-home.png
images/osprey-home.png
This is a binary file and will not be displayed.
images/query-and-charts.png
images/query-and-charts.png
This is a binary file and will not be displayed.
images/query-box.png
images/query-box.png
This is a binary file and will not be displayed.
images/query-history-page.png
images/query-history-page.png
This is a binary file and will not be displayed.
images/query-history-search.png
images/query-history-search.png
This is a binary file and will not be displayed.
images/query-history.png
images/query-history.png
This is a binary file and will not be displayed.
images/query-time-range.png
images/query-time-range.png
This is a binary file and will not be displayed.
images/rules-visualizer.png
images/rules-visualizer.png
This is a binary file and will not be displayed.
images/saved-queries-page.png
images/saved-queries-page.png
This is a binary file and will not be displayed.
images/summary-features.png
images/summary-features.png
This is a binary file and will not be displayed.
images/time-date-picker.png
images/time-date-picker.png
This is a binary file and will not be displayed.
images/udf-documentation.png
images/udf-documentation.png
This is a binary file and will not be displayed.
-1
osprey_rpc/src/osprey/rpc/common/v1/execution_result_pb2_grpc.py
-1
osprey_rpc/src/osprey/rpc/common/v1/execution_result_pb2_grpc.py
-1
osprey_rpc/src/osprey/rpc/common/v1/take_data_pb2_grpc.py
-1
osprey_rpc/src/osprey/rpc/common/v1/take_data_pb2_grpc.py
-1
osprey_rpc/src/osprey/rpc/common/v1/verdicts_pb2_grpc.py
-1
osprey_rpc/src/osprey/rpc/common/v1/verdicts_pb2_grpc.py
-1
osprey_rpc/src/osprey/rpc/pigeon/v1/options_pb2_grpc.py
-1
osprey_rpc/src/osprey/rpc/pigeon/v1/options_pb2_grpc.py
-1
osprey_worker/src/osprey/worker/lib/utils/dates.py
-1
osprey_worker/src/osprey/worker/lib/utils/dates.py
-1
osprey_worker/src/osprey/worker/sinks/sink/kafka_output_sink.py
-1
osprey_worker/src/osprey/worker/sinks/sink/kafka_output_sink.py
+1
-1
osprey_worker/src/osprey/worker/sinks/sink/osprey_coordinator_input_stream.py
+1
-1
osprey_worker/src/osprey/worker/sinks/sink/osprey_coordinator_input_stream.py