loading up the forgejo repo on tangled to test page performance
Expanding documentation in queue.go (#26889)
A set of terminology, along with a broader description, can help more
people engage with the Gitea queue system, providing insights and
ensuring its correct use.
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
authored by
zareck
wxiaoguang
and committed by
GitHub
4e240f23
4f32abaf
+48
-13
+48
-13
modules/queue/queue.go
+48
-13
modules/queue/queue.go
···
1
1
// Copyright 2023 The Gitea Authors. All rights reserved.
2
2
// SPDX-License-Identifier: MIT
3
3
4
-
// Package queue implements a specialized queue system for Gitea.
4
+
// Package queue implements a specialized concurrent queue system for Gitea.
5
+
//
6
+
// Terminology:
7
+
//
8
+
// 1. Item:
9
+
// - An item can be a simple value, such as an integer, or a more complex structure that has multiple fields.
10
+
// Usually a item serves as a task or a message. Sets of items will be sent to a queue handler to be processed.
11
+
// - It's represented as a JSON-marshaled binary slice in the queue
12
+
//
13
+
// 2. Batch:
14
+
// - A collection of items that are grouped together for processing. Each worker receives a batch of items.
15
+
//
16
+
// 3. Worker:
17
+
// - Individual unit of execution designed to process items from the queue. It's a goroutine that calls the Handler.
18
+
// - Workers will get new items through a channel (WorkerPoolQueue is responsible for the distribution).
19
+
// - Workers operate in parallel. The default value of max workers is determined by the setting system.
20
+
//
21
+
// 4. Handler (represented by HandlerFuncT type):
22
+
// - It's the function responsible for processing items. Each active worker will call it.
23
+
// - If an item or some items are not psuccessfully rocessed, the handler could return them as "unhandled items".
24
+
// In such scenarios, the queue system ensures these unhandled items are returned to the base queue after a brief delay.
25
+
// This mechanism is particularly beneficial in cases where the processing entity (like a document indexer) is
26
+
// temporarily unavailable. It ensures that no item is skipped or lost due to transient failures in the processing
27
+
// mechanism.
5
28
//
6
-
// There are two major kinds of concepts:
29
+
// 5. Base queue:
30
+
// - Represents the underlying storage mechanism for the queue. There are several implementations:
31
+
// - Channel: Uses Go's native channel constructs to manage the queue, suitable for in-memory queuing.
32
+
// - LevelDB: Especially useful in persistent queues for single instances.
33
+
// - Redis: Suitable for clusters, where we may have multiple nodes.
34
+
// - Dummy: This is special, it's not a real queue, it's a immediate no-op queue, which is useful for tests.
35
+
// - They all have the same abstraction, the same interface, and they are tested by the same testing code.
7
36
//
8
-
// * The "base queue": channel, level, redis:
9
-
// - They have the same abstraction, the same interface, and they are tested by the same testing code.
10
-
// - The dummy(immediate) queue is special, it's not a real queue, it's only used as a no-op queue or a testing queue.
37
+
// 6. WorkerPoolQueue:
38
+
// - It's responsible to glue all together, using the "base queue" to provide "worker pool" functionality. It creates
39
+
// new workers if needed and can flush the queue, running all the items synchronously till it finishes.
40
+
// - Its "Push" function doesn't block forever, it will return an error if the queue is full after the timeout.
11
41
//
12
-
// * The WorkerPoolQueue: it uses the "base queue" to provide "worker pool" function.
13
-
// - It calls the "handler" to process the data in the base queue.
14
-
// - Its "Push" function doesn't block forever,
15
-
// it will return an error if the queue is full after the timeout.
42
+
// 7. Manager:
43
+
// - The purpose of it is to serve as a centralized manager for multiple WorkerPoolQueue instances. Whenever we want
44
+
// to create a new queue, flush, or get a specific queue, we could use it.
16
45
//
17
46
// A queue can be "simple" or "unique". A unique queue will try to avoid duplicate items.
18
47
// Unique queue's "Has" function can be used to check whether an item is already in the queue,
19
-
// although it's not 100% reliable due to there is no proper transaction support.
48
+
// although it's not 100% reliable due to the lack of proper transaction support.
20
49
// Simple queue's "Has" function always returns "has=false".
21
50
//
22
-
// The HandlerFuncT function is called by the WorkerPoolQueue to process the data in the base queue.
23
-
// If the handler returns "unhandled" items, they will be re-queued to the base queue after a slight delay,
24
-
// in case the item processor (eg: document indexer) is not available.
51
+
// A WorkerPoolQueue is a generic struct; this means it will work with any type but just for that type.
52
+
// If you want another kind of items to run, you would have to call the manager to create a new WorkerPoolQueue for you
53
+
// with a different handler that works with this new type of item. As an example of this:
54
+
//
55
+
// func Init() error {
56
+
// itemQueue = queue.CreateSimpleQueue(graceful.GetManager().ShutdownContext(), "queue-name", handler)
57
+
// ...
58
+
// }
59
+
// func handler(items ...*mypkg.QueueItem) []*mypkg.QueueItem { ... }
25
60
package queue
26
61
27
62
import "code.gitea.io/gitea/modules/util"