Mirror of https://github.com/roostorg/osprey
github.com/roostorg/osprey
1# **Osprey User Interface Guide**
2
3
4## Getting Started
5
6```bash
7cd osprey_ui
8npm install
9npm start
10```
11
12The Osprey UI has several pages accessible by a left-hand menu:
13
14
15
16
17Home will bring you to the default page of Osprey, with three main columns.
18
19**NOTE: The Event Stream in the right column is not yet in v0, and will be available before or in v1.**
20
21
22
23### Left Column: Query
24
25#### **Query Box**
26
27The Osprey Query UI uses the same SML syntax as rules, but for searching and filtering near-real-time and historical data rather than creating new rules. Using the test data generator, you can try writing a query to look for an action called “create\_post” specifically from a given User ID.
28
29
30
31You can also use a UDF in your query. If you ever forget what a UDF does, you can hover on the information symbol for a tip:
32
33
34A query can be run against a time window ranging from the last second to the last 3 months (and also a custom range):
35
36
37
38The Osprey UI is designed to be dynamic and update in real-time. If any other component in the other two columns is interacted with, the query will automatically update and vice versa. The query also automatically populates the URL. This can be handy for sharing a specific query with someone on a team, but may present privacy risks.
39
40
41
42#### **History**
43
44Every query is logged in the Query History view, and there is a dropdown filter to only show queries that you have run.
45
46When you hover over the query, it will also show the Top N Charts used during the query session (more on that below).
47
48
49
50The Query History can also be accessed and seen in a different format via the left-side menu. From here you can filter by the user who ran the query, view the original query, and run it using the same time range the original query used.
51
52
53
54#### **Saved Queries**
55
56If there are specific queries that are used often, Osprey provides the ability to save a query:
57
58
59
60The user who initiated the query and when the query was first run is logged as part of the Saved Query. Saved Queries can also be accessed via the left-side menu. The user who saved the query and what time it was saved is logged and visible. There is a drop-down menu at the top to filter saved queries by users.
61
62
63
64### Middle Column: Charts
65
66The middle column in Osprey shows two types of charts: **Time Series** and **Top N Results**. Both sections provide the ability to add extra charts to see different slices of time or types of top results.
67
68
69
70#### **Time Series Chart**
71
72The Time Series chart shows a visualization of the results in the query over a period of time. The time ranges include:
73
74* Minute
75* Fifteen minutes
76* Half hour
77* Hour
78* Day
79* Week
80* Month
81
82Hovering over a bar in the time series chart shows how many events took place during that time.
83
84
85
86There is also a time and date picker above the time series chart where you can set a custom range:
87
88
89An extra table can be added for another view of a different unit of time. To get rid of the table, you can “[yeet](https://www.urbandictionary.com/define.php?term=Yeet) it”.
90
91
92
93#### **Top N Results**
94
95Adding a Top N Results table populates a table with the top results for the results of the query. You can view and assign labels to a specific entity by hovering over it and clicking “Edit Labels”
96
97
98**NOTE: Labels are not yet in v0**
99
100
101
102You can also select PoP (Period over Period) to compare the query results with results from a window of time in the past to see the delta.
103
104
105
106### Right Column: Event Stream
107
108**The Event Stream is not yet in v0, and will be available before or in v1.**
109
110The Event Stream is essentially Osprey's "live feed" and investigation dashboard where security teams can:
111
112* Monitor real-time activity
113* Search historical events using SML queries
114* Investigate suspicious patterns
115* Track rule execution results
116* Drill down into specific users/entities
117
118It provides a more detailed view of each event that matches the query. The Event Stream can show metadata related to accounts that can link to other internal tools that provide detailed information about an account and/or further enforcement actions.
119
120
121
122The event stream is also viewable in a card format vs a list format (list format shown in the screenshot).
123
124Osprey users may have personal preferences on how to do investigations and what information is most helpful for them. Osprey makes it easy to customize the types of information shown in the Event Stream by clicking “Summary Features”
125
126
127
128### Labeling
129
130**Note: Labels are not yet in v0, but will come in v1**
131Any unique entity can be labeled in the Osprey UI. This manual labeling tool is used by Safety teams to tag individual entities (users, IPs, emails, etc.) with labels. Labels are essentially the manual annotation tool that feeds into Osprey's automated rule system, allowing human judgment to enhance machine detection. Labels can be positive, negative, or neutral. Examples:
132
133**Negative Labels: Harmful/problematic behavior**
134* Examples: "spammer", "bot", "banned", "suspicious"
135
136**Positive Labels: Good/trusted behavior**
137* Examples: "verified", "trusted", "premium\_user"
138
139**Neutral Labels: Informational tags**
140* Examples: "new\_user", "from\_mobile", "beta\_tester"
141
142
143Below are examples of a new label interface from v0, and an example from Discord’s usage of labels (coming in v1).
144
145
146
147
148### UDF Documentation
149
150The UDF Documentation page can be accessed via the left-side menu. It dynamically updates based on the code, so any new UDFs added will show up on this page. This page essentially serves as the "API reference" for the SML language, making it easy for users to discover and properly use all available functions when writing rules and queries.
151
152
153This page can be used as a manual for writing SML rules or queries, guide for understanding parameter types and requirements, and act as a plugin discovery portal to explore what custom UDFs are loaded.
154
155### Bulk Labeling
156
157**Note: Since Bulk Labeling relies on Labels, it does not yet work in v0.**
158
159There are two ways to bulk label items in Osprey: the left-side menu and via the chart column. In this example, you can bulk label all the users that have posted a message that is not empty:
160
161
162
163**Bulk labels can be dangerous if there’s a false positive\!** Osprey provides a counter of how many unique entities are about to be bulk labeled at the top. Labels can be positive, negative, or neutral. A reason must be provided when labeling anything. Each bulk job will create a unique task ID and log the user who initiated the bulk job, the status of the bulk labeling, and a link to the query that the bulk job originated from.
164
165To view all bulk labeling jobs that have been done, click into “Bulk Job History” from the left-side menu. You’ll need the unique task ID to look up a bulk job.
166
167
168
169### Rule Visualizer
170
171**Note: Since the Rule Visualizer relies on Labels, it does not yet work in v0.**
172
173The Rule Visualizer shows how upstream labels, rules, and downstream labels interact with one another. To use it, select an Action or a Label. A graph view will appear showing the relations between rules and labels.
174
175* **Red circle:** label upstream of a rule
176* **Blue square:** rule
177* **Green circle:** label downstream of a rule
178
179
180
181
182### Query Syntax
183
184#### **Actions**
185
186Actions are events that are sent to Osprey. An event is simply something that happens. When a user does something like create a post, send a message, change their username, etc an event happens to represent that. There are probably a lot of events emitted in your org, and Osprey doesn’t need to consume all of them.
187
188##### Features
189
190A feature is any variable in the global namespace in Osprey. All features must be uniquely named. However, prefixing a \`\_\` at the start of a variable name prevents it from being exported as a feature and keeps the variable within the local file’s namespace.
191
192Features are outputs of Osprey executions. Downstream, they are sent to and indexed by Druid, so users can query for events based on feature names later, i.e. \`UserEmail \== '`` despicable@example.com` ``.
193
194```py
195UserId: Entity[int] = EntityJson(type='User', path='$.user.id', coerce_type=True)
196
197UserEmail: str = JsonData(type='Email', path='$.user.email', required=False)
198```
199
200In the example above, both `UserId` and `UserEmail` are features.
201
202##### Entities
203
204Entities are a special type of Feature. All entities are features, but not all features are entities. An entity can have effects applied to it, such as labels, classifications, or signals). Every entity has a type that determines which effects can be applied to it based on static validations.
205
206Entities get special treatment within the Osprey UI. Clicking on an entity in the tool will take you to an Entity View, providing a deep dive into its history.
207
208This could be:
209
210* User ID
211* IP Address
212* Post Text
213* Internet Service Provider
214
215#### **Effects**
216
217Effects can be triggered when one or many rules are evaluated to be true. These are validated and handled in aggregate at the end of an execution output. For example, an effect might apply a label to an entity marking it as a “Spammer”.
218
219### Basic Query Structure
220
221Let’s say you have a data field called “EventType” and one of the events is to “create\_post”. You are looking for posts by a user whose ID is 12345 and you don’t want to see empty posts. Your query would look like:
222
223```py
224# Simple field comparisons
225EventType == "create_post"
226UserId == 12345
227MessageText != Null
228```
229
230#### **Combining Conditions**
231
232Let’s say you’re looking for any matches where a user tried to login more than 3 times. You can create a query to check for two types of data fields: “EventType” and “LoginAttempts”.
233
234If you’re looking for multiple types of events in “EventType” like for posts AND messages, you can use brackets to list the types of events.
235
236```py
237 # Multiple conditions (AND)
238 EventType == "user_login" and LoginAttempts >= 3
239
240 # OR conditions
241 EventType in ["create_post", "send_message"]
242 (UserId == 123) or (UserId == 456)
243```
244
245#### **Using UDFs in Queries**
246
247UDFs (read more [here](https://github.com/roostorg/osprey/blob/f16da6e5c32ae124c3cc6e2d7efded7cea1ac726/docs/rules.md#user-defined-functions-udfs)) are a powerful part of queries. Once you define a UDF with the specific desired logic, you can reference it in a query.
248
249**NOTE: If you try to query a UDF that doesn’t exist, Osprey will silently fail with a 500 error.**
250
251```py
252 # Text search
253 TextContains(text=PostContent, phrase="spam")
254 RegexMatch(target=MessageText, pattern="(buy|sell|deal)")
255
256 # List operations
257 ListLength(list=UserConnections) > 10
258```
259
260#### **Label Queries**
261
262**Important Note: Labels are not yet in v0, so these will not work in the UI.**
263
264Since the UI searches across actions/events:
265
266* **Don't use:** HasLabel() \- won't work in Query UI
267* **Use instead**: DidAddLabel() \- shows when an action added a label
268
269```py
270 # Find actions that added specific labels
271 DidAddLabel(entity_type="UserId", label_name="likely_spammer")
272 DidAddLabel(entity_type="IpAddress", label_name="suspicious")
273```
274
275###
276
277### Example Queries
278
279```py
280# Find suspicious login attempts:
281 EventType == "user_login" and LoginAttempts >= 5
282
283# Find posts containing specific words:
284 EventType == "create_post" and TextContains(text=PostContent, phrase="urgent")
285
286# Find users who were flagged:
287 DidAddLabel(entity_type="UserId", label_name="flagged")
288
289# Complex search:
290 EventType == "send_message" and
291 RegexMatch(target=MessageText, pattern="(click|link|urgent)") and
292 not DidAddLabel(entity_type="UserId", label_name="verified")