loading up the forgejo repo on tangled to test page performance
Merge pull request 'tests: Improve in-repo documentation' (#5235) from fnetx/testing-docs into forgejo
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/5235
Reviewed-by: thefox <thefox@noreply.codeberg.org>
+190
-137
+3
.forgejo/workflows/e2e.yml
+3
.forgejo/workflows/e2e.yml
+143
-42
tests/e2e/README.md
+143
-42
tests/e2e/README.md
···
1
1
# End to end tests
2
2
3
-
E2e tests largely follow the same syntax as [integration tests](../integration).
4
-
Whereas integration tests are intended to mock and stress the back-end, server-side code, e2e tests the interface between front-end and back-end, as well as visual regressions with both assertions and visual comparisons.
5
-
They can be run with make commands for the appropriate backends, namely:
6
-
```shell
7
-
make test-sqlite
8
-
make test-pgsql
9
-
make test-mysql
10
-
```
3
+
Thank you for your effort to provide good software tests for Forgejo.
4
+
Please also read the general testing instructions in the
5
+
[Forgejo contributor documentation](https://forgejo.org/docs/next/contributor/testing/)
6
+
and make sure to also check the
7
+
[Playwright documentation](https://playwright.dev/docs/intro)
8
+
for further information.
9
+
10
+
This file is meant to provide specific information for the integration tests
11
+
as well as some tips and tricks you should know.
12
+
13
+
Feel free to extend this file with more instructions if you feel like you have something to share!
14
+
15
+
16
+
## How to run the tests?
17
+
18
+
Before running any tests, please ensure you perform a clean frontend build:
11
19
12
-
Make sure to perform a clean front-end build before running tests:
13
20
```
14
21
make clean frontend
15
22
```
16
23
17
-
## Install playwright system dependencies
24
+
Whenever you modify frontend code (i.e. JavaScript and CSS files),
25
+
you need to create a new frontend build.
26
+
27
+
For tests that require interactive Git repos,
28
+
you also need to ensure a Forgejo binary is ready to be used by Git hooks.
29
+
For this, you additionally need to run
30
+
31
+
~~~
32
+
make TAGS="sqlite sqlite_unlock_notify" backend
33
+
~~~
34
+
35
+
### Install dependencies
36
+
37
+
Browsertesting is performed by playwright.
38
+
You need certain system libraries and playwright will download required browsers.
39
+
Playwright takes care of this when you run:
40
+
18
41
```
19
42
npx playwright install-deps
20
43
```
21
44
22
-
## Interactive testing
45
+
> **Note**
46
+
> On some operating systems, the installation of missing libraries can complicate testing certain browsers.
47
+
> It is often not necessary to test with all browsers locally.
48
+
> Choosing either Firefox or Chromium is fine.
49
+
23
50
24
-
You can make use of Playwright's integrated UI mode to run individual tests,
25
-
get feedback and visually trace what your browser is doing.
51
+
### Run all tests
52
+
53
+
If you want to run the full test suite, you can use
54
+
55
+
```
56
+
make test-e2e-sqlite
57
+
```
58
+
59
+
### Interactive testing
26
60
27
-
To do so, launch the debugserver using:
61
+
We recommend that you use interactive testing for the development.
62
+
After you performed the required builds,
63
+
you should use one shell to start the debugserver (and leave it running):
28
64
29
65
```
30
66
make test-e2e-debugserver
31
67
```
32
68
33
-
Then launch the Playwright UI:
69
+
It allows you to explore the test data in your local browser,
70
+
and playwright to perform tests on it.
71
+
72
+
> **Note**
73
+
> The modifications persist while the debugserver is running.
74
+
> If you modified things, it might be useful to restart it to get back to a fresh state.
75
+
> While writing playwright tests, you either
76
+
> need to ensure they are resilient against repeated runs
77
+
> (e.g. when only creating new content),
78
+
> or that they restore the initial state for the next browser run.
79
+
80
+
#### With the playwright UI:
81
+
82
+
Playwright ships with an integrated UI mode which allows you to
83
+
run individual tests and to debug them by seeing detailed traces of what playwright does.
84
+
Launch it with:
34
85
35
86
```
36
87
npx playwright test --ui
37
88
```
38
89
39
-
You can also run individual tests while the debugserver using:
90
+
#### Running individual tests
40
91
41
92
```
42
93
npx playwright test actions.test.e2e.js:9
···
46
97
and after the colon you can put the linenumber where the test is defined.
47
98
48
99
49
-
## Run all tests via local act_runner
100
+
#### With VSCodium or VSCode
101
+
102
+
To debug a test, you can also use "Playwright Test" for
103
+
[VScodium](https://open-vsx.org/extension/ms-playwright/playwright)
104
+
or [VSCode](https://marketplace.visualstudio.com/items?itemName=ms-playwright.playwright).
105
+
106
+
107
+
### Run all tests via local act_runner
108
+
109
+
If you have a [forgejo runner](https://code.forgejo.org/forgejo/runner/),
110
+
you can use it to run the test jobs:
111
+
50
112
```
51
-
act_runner exec -W ./.github/workflows/pull-e2e-tests.yml --event=pull_request --default-actions-url="https://github.com" -i catthehacker/ubuntu:runner-latest
113
+
forgejo-runner exec -W .forgejo/workflows/e2e.yml --event=pull_request
52
114
```
53
115
54
-
## Run sqlite e2e tests
55
-
Start tests
56
-
```
57
-
make test-e2e-sqlite
58
-
```
116
+
### Run e2e tests with another database
117
+
118
+
This approach is not currently used,
119
+
neither in the CI/CD nor by core contributors on their lcoal machines.
120
+
It is still documented for the sake of completeness:
121
+
You can also perform e2e tests using MariaDB/MySQL or PostgreSQL if you want.
59
122
60
-
## Run MySQL e2e tests
61
123
Setup a MySQL database inside docker
62
124
```
63
125
docker run -e "MYSQL_DATABASE=test" -e "MYSQL_ALLOW_EMPTY_PASSWORD=yes" -p 3306:3306 --rm --name mysql mysql:latest #(just ctrl-c to stop db and clean the container)
···
68
130
TEST_MYSQL_HOST=localhost:3306 TEST_MYSQL_DBNAME=test TEST_MYSQL_USERNAME=root TEST_MYSQL_PASSWORD='' make test-e2e-mysql
69
131
```
70
132
71
-
## Run pgsql e2e tests
72
133
Setup a pgsql database inside docker
73
134
```
74
135
docker run -e "POSTGRES_DB=test" -p 5432:5432 --rm --name pgsql postgres:latest #(just ctrl-c to stop db and clean the container)
···
78
139
TEST_PGSQL_HOST=localhost:5432 TEST_PGSQL_DBNAME=test TEST_PGSQL_USERNAME=postgres TEST_PGSQL_PASSWORD=postgres make test-e2e-pgsql
79
140
```
80
141
81
-
## Running individual tests
142
+
### Running individual tests
82
143
83
144
Example command to run `example.test.e2e.js` test file:
84
145
85
-
_Note: unlike integration tests, this filtering is at the file level, not function_
146
+
> **Note**
147
+
> Unlike integration tests, this filtering is at the file level, not function
86
148
87
149
For SQLite:
88
150
···
90
152
make test-e2e-sqlite#example
91
153
```
92
154
93
-
For PostgreSQL databases(replace `mysql` to `pgsql`):
94
-
95
-
```
96
-
TEST_MYSQL_HOST=localhost:1433 TEST_MYSQL_DBNAME=test TEST_MYSQL_USERNAME=sa TEST_MYSQL_PASSWORD=MwantsaSecurePassword1 make test-e2e-mysql#example
97
-
```
155
+
### Visual testing
98
156
99
-
## Visual testing
157
+
> **Warning**
158
+
> This is not currently used by most Forgejo contributors.
159
+
> Your help to improve the situation and allow for visual testing is appreciated.
100
160
101
161
Although the main goal of e2e is assertion testing, we have added a framework for visual regress testing. If you are working on front-end features, please use the following:
102
162
- Check out `main`, `make clean frontend`, and run e2e tests with `VISUAL_TEST=1` to generate outputs. This will initially fail, as no screenshots exist. You can run the e2e tests again to assert it passes.
···
106
166
107
167
ACCEPT_VISUAL=1 will overwrite the snapshot images with new images.
108
168
109
-
## With VSCodium or VSCode
169
+
170
+
## Tips and tricks
110
171
111
-
To debug a test, you can use "Playwright Test" for
112
-
[VScodium](https://open-vsx.org/extension/ms-playwright/playwright)
113
-
or [VSCode](https://marketplace.visualstudio.com/items?itemName=ms-playwright.playwright).
114
-
Before doing that you will need to manually start a Forgejo instance and populate it
115
-
with data from `models/fixtures` by running:
172
+
If you know noteworthy tests that can act as an inspiration for new tests,
173
+
please add some details here.
116
174
117
-
```sh
118
-
make TAGS='sqlite sqlite_unlock_notify' 'test-e2e-debugserver'
119
-
```
175
+
### Run tests very selectively
176
+
177
+
Browser testing can take some time.
178
+
If you want to iterate fast,
179
+
save your time and only run very selected tests.
180
+
Use only one browser.
181
+
182
+
### Skip Safari if it doesn't work
183
+
184
+
Many contributors have issues getting Safari (webkit)
185
+
and especially Safari Mobile to work.
186
+
187
+
At the top of your test function, you can use:
188
+
189
+
~~~javascript
190
+
test.skip(workerInfo.project.name === 'Mobile Safari', 'Unable to get tests working on Safari Mobile.');
191
+
~~~
192
+
193
+
### Don't forget the formatting.
194
+
195
+
When writing tests without modifying other frontend code,
196
+
it is easy to forget that the JavaScript test files also need formatting.
197
+
198
+
Run `make lint-frontend-fix`.
199
+
200
+
### Define new repos
201
+
202
+
Take a look at `declare_repos_test.go` to see how to add your repositories.
203
+
Feel free to improve the logic used there if you need more advanced functionality
204
+
(it is a simplified version of the code used in the integration tests).
205
+
206
+
### Accessibility testing
207
+
208
+
If you can, perform automated accessibility testing using
209
+
[AxeCore](https://github.com/dequelabs/axe-core-npm/blob/develop/packages/playwright/README.md).
210
+
211
+
Take a look at `shared/forms.js` and some other places for inspiration.
212
+
213
+
### List related files coverage
214
+
215
+
If you think your playwright tests covers an important aspect of some template, CSS or backend files,
216
+
consider adding the paths to `.forgejo/workflows/e2e.yml` in the path filter.
217
+
218
+
It ensures that future modifications to this file will be tested as well.
219
+
220
+
Currently, we do not run the e2e tests on all changes.
+8
-4
tests/e2e/declare_repos_test.go
+8
-4
tests/e2e/declare_repos_test.go
···
21
21
"github.com/stretchr/testify/require"
22
22
)
23
23
24
+
// first entry represents filename
25
+
// the following entries define the full file content over time
24
26
type FileChanges [][]string
25
27
26
28
// put your Git repo declarations in here
27
29
// feel free to amend the helper function below or use the raw variant directly
28
30
func DeclareGitRepos(t *testing.T) func() {
29
-
var cleanupFunctions []func()
30
-
cleanupFunctions = append(cleanupFunctions, newRepo(t, 2, "diff-test", FileChanges{
31
-
{"testfile", "hello", "hallo", "hola", "native", "ubuntu-latest", "- runs-on: ubuntu-latest", "- runs-on: debian-latest"},
32
-
}))
31
+
cleanupFunctions := []func(){
32
+
newRepo(t, 2, "diff-test", FileChanges{
33
+
{"testfile", "hello", "hallo", "hola", "native", "ubuntu-latest", "- runs-on: ubuntu-latest", "- runs-on: debian-latest"},
34
+
}),
35
+
// add your repo declarations here
36
+
}
33
37
34
38
return func() {
35
39
for _, cleanup := range cleanupFunctions {
+36
-14
tests/integration/README.md
+36
-14
tests/integration/README.md
···
1
1
# Integration tests
2
2
3
+
Thank you for your effort to provide good software tests for Forgejo.
4
+
Please also read the general testing instructions in the
5
+
[Forgejo contributor documentation](https://forgejo.org/docs/next/contributor/testing/).
6
+
7
+
This file is meant to provide specific information for the integration tests
8
+
as well as some tips and tricks you should know.
9
+
10
+
Feel free to extend this file with more instructions if you feel like you have something to share!
11
+
12
+
13
+
## How to run the tests?
14
+
15
+
Before running any tests, please ensure you perform a clean build:
16
+
17
+
```
18
+
make clean build
19
+
```
20
+
3
21
Integration tests can be run with make commands for the
4
22
appropriate backends, namely:
5
23
```shell
···
8
26
make test-mysql
9
27
```
10
28
11
-
Make sure to perform a clean build before running tests:
12
-
```
13
-
make clean build
14
-
```
15
29
16
-
## Run tests via local act_runner
30
+
### Run tests via local forgejo runner
17
31
18
-
### Run all jobs
32
+
If you have a [forgejo runner](https://code.forgejo.org/forgejo/runner/),
33
+
you can use it to run the test jobs:
34
+
35
+
#### Run all jobs
19
36
20
37
```
21
-
act_runner exec -W ./.github/workflows/pull-db-tests.yml --event=pull_request --default-actions-url="https://github.com" -i catthehacker/ubuntu:runner-latest
38
+
forgejo-runner exec -W .forgejo/workflows/testing.yml --event=pull_request
22
39
```
23
40
24
41
Warning: This file defines many jobs, so it will be resource-intensive and therefore not recommended.
25
42
26
-
### Run single job
43
+
#### Run single job
27
44
28
45
```SHELL
29
-
act_runner exec -W ./.github/workflows/pull-db-tests.yml --event=pull_request --default-actions-url="https://github.com" -i catthehacker/ubuntu:runner-latest -j <job_name>
46
+
forgejo-runner exec -W .forgejo/workflows/testing.yml --event=pull_request -j <job_name>
30
47
```
31
48
32
49
You can list all job names via:
33
50
34
51
```SHELL
35
-
act_runner exec -W ./.github/workflows/pull-db-tests.yml --event=pull_request --default-actions-url="https://github.com" -i catthehacker/ubuntu:runner-latest -l
52
+
forgejo-runner exec -W .forgejo/workflows/testing.yml --event=pull_request -l
36
53
```
37
54
38
-
## Run sqlite integration tests
55
+
### Run sqlite integration tests
39
56
Start tests
40
57
```
41
58
make test-sqlite
42
59
```
43
60
44
-
## Run MySQL integration tests
61
+
### Run MySQL integration tests
45
62
Setup a MySQL database inside docker
46
63
```
47
64
docker run -e "MYSQL_DATABASE=test" -e "MYSQL_ALLOW_EMPTY_PASSWORD=yes" -p 3306:3306 --rm --name mysql mysql:latest #(just ctrl-c to stop db and clean the container)
···
52
69
TEST_MYSQL_HOST=localhost:3306 TEST_MYSQL_DBNAME=test TEST_MYSQL_USERNAME=root TEST_MYSQL_PASSWORD='' make test-mysql
53
70
```
54
71
55
-
## Run pgsql integration tests
72
+
### Run pgsql integration tests
56
73
Setup a pgsql database inside docker
57
74
```
58
75
docker run -e "POSTGRES_DB=test" -p 5432:5432 --rm --name pgsql postgres:latest #(just ctrl-c to stop db and clean the container)
···
62
79
TEST_PGSQL_HOST=localhost:5432 TEST_PGSQL_DBNAME=test TEST_PGSQL_USERNAME=postgres TEST_PGSQL_PASSWORD=postgres make test-pgsql
63
80
```
64
81
65
-
## Running individual tests
82
+
### Running individual tests
66
83
67
84
Example command to run GPG test:
68
85
···
99
116
```bash
100
117
GITEA_SLOW_TEST_TIME="10s" GITEA_SLOW_FLUSH_TIME="5s" make test-sqlite
101
118
```
119
+
120
+
## Tips and tricks
121
+
122
+
If you know noteworthy tests that can act as an inspiration for new tests,
123
+
please add some details here.
-77
tests/integration/README_ZH.md
-77
tests/integration/README_ZH.md
···
1
-
# 关于集成测试
2
-
3
-
使用如下 make 命令可以运行指定的集成测试：
4
-
```shell
5
-
make test-mysql
6
-
make test-pgsql
7
-
make test-sqlite
8
-
```
9
-
10
-
在执行集成测试命令前请确保清理了之前的构建环境，清理命令如下：
11
-
```
12
-
make clean build
13
-
```
14
-
15
-
## 如何在本地 act_runner 上运行测试
16
-
17
-
### 运行所有任务
18
-
19
-
```
20
-
act_runner exec -W ./.github/workflows/pull-db-tests.yml --event=pull_request --default-actions-url="https://github.com" -i catthehacker/ubuntu:runner-latest
21
-
```
22
-
23
-
警告:由于在此文件中定义了许多任务，因此此操作将花费太多的CPU和内存来运行。所以不建议这样做。
24
-
25
-
### 运行单个任务
26
-
27
-
```SHELL
28
-
act_runner exec -W ./.github/workflows/pull-db-tests.yml --event=pull_request --default-actions-url="https://github.com" -i catthehacker/ubuntu:runner-latest -j <job_name>
29
-
```
30
-
31
-
您可以通过以下方式列出所有任务名称:
32
-
```SHELL
33
-
act_runner exec -W ./.github/workflows/pull-db-tests.yml --event=pull_request --default-actions-url="https://github.com" -i catthehacker/ubuntu:runner-latest -l
34
-
```
35
-
36
-
## 如何使用 sqlite 数据库进行集成测试
37
-
使用该命令执行集成测试
38
-
```
39
-
make test-sqlite
40
-
```
41
-
42
-
## 如何使用 mysql 数据库进行集成测试
43
-
首先在docker容器里部署一个 mysql 数据库
44
-
```
45
-
docker run -e "MYSQL_DATABASE=test" -e "MYSQL_ALLOW_EMPTY_PASSWORD=yes" -p 3306:3306 --rm --name mysql mysql:8 #(just ctrl-c to stop db and clean the container)
46
-
```
47
-
之后便可以基于这个数据库进行集成测试
48
-
```
49
-
TEST_MYSQL_HOST=localhost:3306 TEST_MYSQL_DBNAME=test TEST_MYSQL_USERNAME=root TEST_MYSQL_PASSWORD='' make test-mysql
50
-
```
51
-
52
-
## 如何使用 pgsql 数据库进行集成测试
53
-
同上，首先在 docker 容器里部署一个 pgsql 数据库
54
-
```
55
-
docker run -e "POSTGRES_DB=test" -p 5432:5432 --rm --name pgsql postgres:14 #(just ctrl-c to stop db and clean the container)
56
-
```
57
-
之后便可以基于这个数据库进行集成测试
58
-
```
59
-
TEST_PGSQL_HOST=localhost:5432 TEST_PGSQL_DBNAME=test TEST_PGSQL_USERNAME=postgres TEST_PGSQL_PASSWORD=postgres make test-pgsql
60
-
```
61
-
62
-
## 如何进行自定义的集成测试
63
-
64
-
下面的示例展示了怎样在集成测试中只进行 GPG 测试：
65
-
66
-
sqlite 数据库:
67
-
68
-
```
69
-
make test-sqlite#GPG
70
-
```
71
-
72
-
其它数据库 (用 PGSQL 取代 MYSQL）:
73
-
74
-
```
75
-
TEST_MYSQL_HOST=localhost:1433 TEST_MYSQL_DBNAME=test TEST_MYSQL_USERNAME=sa TEST_MYSQL_PASSWORD=MwantsaSecurePassword1 make test-mysql#GPG
76
-
```
77
-