Linux kernel mirror (for testing)
git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel
os
linux
media: Documentation: Add Mali-C55 ISP Documentation
Add a documentation page for the mali-c55 driver, which gives a brief
overview of the hardware and explains how to use the driver's capture
devices and the crop/scaler functions.
Tested-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Acked-by: Nayden Kanchev <nayden.kanchev@arm.com>
Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>
Signed-off-by: Hans Verkuil <hverkuil+cisco@kernel.org>
authored by
Daniel Scally
and committed by
Hans Verkuil
dfb999e7
d5f281f3
+360
+19
Documentation/admin-guide/media/mali-c55-graph.dot
+19
Documentation/admin-guide/media/mali-c55-graph.dot
···
1
+
digraph board {
2
+
rankdir=TB
3
+
n00000001 [label="{{} | mali-c55 tpg\n/dev/v4l-subdev0 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green]
4
+
n00000001:port0 -> n00000003:port0 [style=dashed]
5
+
n00000003 [label="{{<port0> 0} | mali-c55 isp\n/dev/v4l-subdev1 | {<port1> 1 | <port2> 2}}", shape=Mrecord, style=filled, fillcolor=green]
6
+
n00000003:port1 -> n00000007:port0 [style=bold]
7
+
n00000003:port2 -> n00000007:port2 [style=bold]
8
+
n00000003:port1 -> n0000000b:port0 [style=bold]
9
+
n00000007 [label="{{<port0> 0 | <port2> 2} | mali-c55 resizer fr\n/dev/v4l-subdev2 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
10
+
n00000007:port1 -> n0000000e [style=bold]
11
+
n0000000b [label="{{<port0> 0} | mali-c55 resizer ds\n/dev/v4l-subdev3 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
12
+
n0000000b:port1 -> n00000012 [style=bold]
13
+
n0000000e [label="mali-c55 fr\n/dev/video0", shape=box, style=filled, fillcolor=yellow]
14
+
n00000012 [label="mali-c55 ds\n/dev/video1", shape=box, style=filled, fillcolor=yellow]
15
+
n00000022 [label="{{<port0> 0} | csi2-rx\n/dev/v4l-subdev4 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
16
+
n00000022:port1 -> n00000003:port0
17
+
n00000027 [label="{{} | imx415 1-001a\n/dev/v4l-subdev5 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green]
18
+
n00000027:port0 -> n00000022:port0 [style=bold]
19
+
}
+340
Documentation/admin-guide/media/mali-c55.rst
+340
Documentation/admin-guide/media/mali-c55.rst
···
1
+
.. SPDX-License-Identifier: GPL-2.0
2
+
3
+
==========================================
4
+
ARM Mali-C55 Image Signal Processor driver
5
+
==========================================
6
+
7
+
Introduction
8
+
============
9
+
10
+
This file documents the driver for ARM's Mali-C55 Image Signal Processor. The
11
+
driver is located under drivers/media/platform/arm/mali-c55.
12
+
13
+
The Mali-C55 ISP receives data in either raw Bayer format or RGB/YUV format from
14
+
sensors through either a parallel interface or a memory bus before processing it
15
+
and outputting it through an internal DMA engine. Two output pipelines are
16
+
possible (though one may not be fitted, depending on the implementation). These
17
+
are referred to as "Full resolution" and "Downscale", but the naming is historic
18
+
and both pipes are capable of cropping/scaling operations. The full resolution
19
+
pipe is also capable of outputting RAW data, bypassing much of the ISP's
20
+
processing. The downscale pipe cannot output RAW data. An integrated test
21
+
pattern generator can be used to drive the ISP and produce image data in the
22
+
absence of a connected camera sensor. The driver module is named mali_c55, and
23
+
is enabled through the CONFIG_VIDEO_MALI_C55 config option.
24
+
25
+
The driver implements V4L2, Media Controller and V4L2 Subdevice interfaces and
26
+
expects camera sensors connected to the ISP to have V4L2 subdevice interfaces.
27
+
28
+
Mali-C55 ISP hardware
29
+
=====================
30
+
31
+
A high level functional view of the Mali-C55 ISP is presented below. The ISP
32
+
takes input from either a live source or through a DMA engine for memory input,
33
+
depending on the SoC integration.::
34
+
35
+
+---------+ +----------+ +--------+
36
+
| Sensor |--->| CSI-2 Rx | "Full Resolution" | DMA |
37
+
+---------+ +----------+ |\ Output +--->| Writer |
38
+
| | \ | +--------+
39
+
| | \ +----------+ +------+---> Streaming I/O
40
+
+------------+ +------->| | | | |
41
+
| | | |-->| Mali-C55 |--+
42
+
| DMA Reader |--------------->| | | ISP | |
43
+
| | | / | | | +---> Streaming I/O
44
+
+------------+ | / +----------+ | |
45
+
|/ +------+
46
+
| +--------+
47
+
+--->| DMA |
48
+
"Downscaled" | Writer |
49
+
Output +--------+
50
+
51
+
Media Controller Topology
52
+
=========================
53
+
54
+
An example of the ISP's topology (as implemented in a system with an IMX415
55
+
camera sensor and generic CSI-2 receiver) is below:
56
+
57
+
58
+
.. kernel-figure:: mali-c55-graph.dot
59
+
:alt: mali-c55-graph.dot
60
+
:align: center
61
+
62
+
The driver has 4 V4L2 subdevices:
63
+
64
+
- `mali_c55 isp`: Responsible for configuring input crop and color space
65
+
conversion
66
+
- `mali_c55 tpg`: The test pattern generator, emulating a camera sensor.
67
+
- `mali_c55 resizer fr`: The Full-Resolution pipe resizer
68
+
- `mali_c55 resizer ds`: The Downscale pipe resizer
69
+
70
+
The driver has 2 V4L2 video devices:
71
+
72
+
- `mali-c55 fr`: The full-resolution pipe's capture device
73
+
- `mali-c55 ds`: The downscale pipe's capture device
74
+
75
+
Frame sequences are synchronised across to two capture devices, meaning if one
76
+
pipe is started later than the other the sequence numbers returned in its
77
+
buffers will match those of the other pipe rather than starting from zero.
78
+
79
+
Idiosyncrasies
80
+
--------------
81
+
82
+
**mali-c55 isp**
83
+
The `mali-c55 isp` subdevice has a single sink pad to which all sources of data
84
+
should be connected. The active source is selected by enabling the appropriate
85
+
media link and disabling all others. The ISP has two source pads, reflecting the
86
+
different paths through which it can internally route data. Tap points within
87
+
the ISP allow users to divert data to avoid processing by some or all of the
88
+
hardware's processing steps. The diagram below is intended only to highlight how
89
+
the bypassing works and is not a true reflection of those processing steps; for
90
+
a high-level functional block diagram see ARM's developer page for the
91
+
ISP [3]_::
92
+
93
+
+--------------------------------------------------------------+
94
+
| Possible Internal ISP Data Routes |
95
+
| +------------+ +----------+ +------------+ |
96
+
+---+ | | | | | Colour | +---+
97
+
| 0 |--+-->| Processing |->| Demosaic |->| Space |--->| 1 |
98
+
+---+ | | | | | | Conversion | +---+
99
+
| | +------------+ +----------+ +------------+ |
100
+
| | +---+
101
+
| +---------------------------------------------------| 2 |
102
+
| +---+
103
+
| |
104
+
+--------------------------------------------------------------+
105
+
106
+
107
+
.. flat-table::
108
+
:header-rows: 1
109
+
110
+
* - Pad
111
+
- Direction
112
+
- Purpose
113
+
114
+
* - 0
115
+
- sink
116
+
- Data input, connected to the TPG and camera sensors
117
+
118
+
* - 1
119
+
- source
120
+
- RGB/YUV data, connected to the FR and DS V4L2 subdevices
121
+
122
+
* - 2
123
+
- source
124
+
- RAW bayer data, connected to the FR V4L2 subdevices
125
+
126
+
The ISP is limited to both input and output resolutions between 640x480 and
127
+
8192x8192, and this is reflected in the ISP and resizer subdevice's .set_fmt()
128
+
operations.
129
+
130
+
**mali-c55 resizer fr**
131
+
The `mali-c55 resizer fr` subdevice has two _sink_ pads to reflect the different
132
+
insertion points in the hardware (either RAW or demosaiced data):
133
+
134
+
.. flat-table::
135
+
:header-rows: 1
136
+
137
+
* - Pad
138
+
- Direction
139
+
- Purpose
140
+
141
+
* - 0
142
+
- sink
143
+
- Data input connected to the ISP's demosaiced stream.
144
+
145
+
* - 1
146
+
- source
147
+
- Data output connected to the capture video device
148
+
149
+
* - 2
150
+
- sink
151
+
- Data input connected to the ISP's raw data stream
152
+
153
+
The data source in use is selected through the routing API; two routes each of a
154
+
single stream are available:
155
+
156
+
.. flat-table::
157
+
:header-rows: 1
158
+
159
+
* - Sink Pad
160
+
- Source Pad
161
+
- Purpose
162
+
163
+
* - 0
164
+
- 1
165
+
- Demosaiced data route
166
+
167
+
* - 2
168
+
- 1
169
+
- Raw data route
170
+
171
+
172
+
If the demosaiced route is active then the FR pipe is only capable of output
173
+
in RGB/YUV formats. If the raw route is active then the output reflects the
174
+
input (which may be either Bayer or RGB/YUV data).
175
+
176
+
Using the driver to capture video
177
+
=================================
178
+
179
+
Using the media controller APIs we can configure the input source and ISP to
180
+
capture images in a variety of formats. In the examples below, configuring the
181
+
media graph is done with the v4l-utils [1]_ package's media-ctl utility.
182
+
Capturing the images is done with yavta [2]_.
183
+
184
+
Configuring the input source
185
+
----------------------------
186
+
187
+
The first step is to set the input source that we wish by enabling the correct
188
+
media link. Using the example topology above, we can select the TPG as follows:
189
+
190
+
.. code-block:: none
191
+
192
+
media-ctl -l "'lte-csi2-rx':1->'mali-c55 isp':0[0]"
193
+
media-ctl -l "'mali-c55 tpg':0->'mali-c55 isp':0[1]"
194
+
195
+
Configuring which video devices will stream data
196
+
------------------------------------------------
197
+
198
+
The driver will wait for all video devices to have their VIDIOC_STREAMON ioctl
199
+
called before it tells the sensor to start streaming. To facilitate this we need
200
+
to enable links to the video devices that we want to use. In the example below
201
+
we enable the links to both of the image capture video devices
202
+
203
+
.. code-block:: none
204
+
205
+
media-ctl -l "'mali-c55 resizer fr':1->'mali-c55 fr':0[1]"
206
+
media-ctl -l "'mali-c55 resizer ds':1->'mali-c55 ds':0[1]"
207
+
208
+
Capturing bayer data from the source and processing to RGB/YUV
209
+
--------------------------------------------------------------
210
+
211
+
To capture 1920x1080 bayer data from the source and push it through the ISP's
212
+
full processing pipeline, we configure the data formats appropriately on the
213
+
source, ISP and resizer subdevices and set the FR resizer's routing to select
214
+
processed data. The media bus format on the resizer's source pad will be either
215
+
RGB121212_1X36 or YUV10_1X30, depending on whether you want to capture RGB or
216
+
YUV. The ISP's debayering block outputs RGB data natively, setting the source
217
+
pad format to YUV10_1X30 enables the colour space conversion block.
218
+
219
+
In this example we target RGB565 output, so select RGB121212_1X36 as the resizer
220
+
source pad's format:
221
+
222
+
.. code-block:: none
223
+
224
+
# Set formats on the TPG and ISP
225
+
media-ctl -V "'mali-c55 tpg':0[fmt:SRGGB20_1X20/1920x1080]"
226
+
media-ctl -V "'mali-c55 isp':0[fmt:SRGGB20_1X20/1920x1080]"
227
+
media-ctl -V "'mali-c55 isp':1[fmt:SRGGB20_1X20/1920x1080]"
228
+
229
+
# Set routing on the FR resizer
230
+
media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[1],2/0->1/0[0]]"
231
+
232
+
# Set format on the resizer, must be done AFTER the routing.
233
+
media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/1920x1080]"
234
+
235
+
The downscale output can also be used to stream data at the same time. In this
236
+
case since only processed data can be captured through the downscale output no
237
+
routing need be set:
238
+
239
+
.. code-block:: none
240
+
241
+
# Set format on the resizer
242
+
media-ctl -V "'mali-c55 resizer ds':1[fmt:RGB121212_1X36/1920x1080]"
243
+
244
+
Following which images can be captured from both the FR and DS output's video
245
+
devices (simultaneously, if desired):
246
+
247
+
.. code-block:: none
248
+
249
+
yavta -f RGB565 -s 1920x1080 -c10 /dev/video0
250
+
yavta -f RGB565 -s 1920x1080 -c10 /dev/video1
251
+
252
+
Cropping the image
253
+
~~~~~~~~~~~~~~~~~~
254
+
255
+
Both the full resolution and downscale pipes can crop to a minimum resolution of
256
+
640x480. To crop the image simply configure the resizer's sink pad's crop and
257
+
compose rectangles and set the format on the video device:
258
+
259
+
.. code-block:: none
260
+
261
+
media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(480,270)/640x480 compose:(0,0)/640x480]"
262
+
media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]"
263
+
yavta -f RGB565 -s 640x480 -c10 /dev/video0
264
+
265
+
Downscaling the image
266
+
~~~~~~~~~~~~~~~~~~~~~
267
+
268
+
Both the full resolution and downscale pipes can downscale the image by up to 8x
269
+
provided the minimum 640x480 output resolution is adhered to. For the best image
270
+
result the scaling ratio for each direction should be the same. To configure
271
+
scaling we use the compose rectangle on the resizer's sink pad:
272
+
273
+
.. code-block:: none
274
+
275
+
media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(0,0)/1920x1080 compose:(0,0)/640x480]"
276
+
media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]"
277
+
yavta -f RGB565 -s 640x480 -c10 /dev/video0
278
+
279
+
Capturing images in YUV formats
280
+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
281
+
282
+
If we need to output YUV data rather than RGB the color space conversion block
283
+
needs to be active, which is achieved by setting MEDIA_BUS_FMT_YUV10_1X30 on the
284
+
resizer's source pad. We can then configure a capture format like NV12 (here in
285
+
its multi-planar variant)
286
+
287
+
.. code-block:: none
288
+
289
+
media-ctl -V "'mali-c55 resizer fr':1[fmt:YUV10_1X30/1920x1080]"
290
+
yavta -f NV12M -s 1920x1080 -c10 /dev/video0
291
+
292
+
Capturing RGB data from the source and processing it with the resizers
293
+
----------------------------------------------------------------------
294
+
295
+
The Mali-C55 ISP can work with sensors capable of outputting RGB data. In this
296
+
case although none of the image quality blocks would be used it can still
297
+
crop/scale the data in the usual way. For this reason RGB data input to the ISP
298
+
still goes through the ISP subdevice's pad 1 to the resizer.
299
+
300
+
To achieve this, the ISP's sink pad's format is set to
301
+
MEDIA_BUS_FMT_RGB202020_1X60 - this reflects the format that data must be in to
302
+
work with the ISP. Converting the camera sensor's output to that format is the
303
+
responsibility of external hardware.
304
+
305
+
In this example we ask the test pattern generator to give us RGB data instead of
306
+
bayer.
307
+
308
+
.. code-block:: none
309
+
310
+
media-ctl -V "'mali-c55 tpg':0[fmt:RGB202020_1X60/1920x1080]"
311
+
media-ctl -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080]"
312
+
313
+
Cropping or scaling the data can be done in exactly the same way as outlined
314
+
earlier.
315
+
316
+
Capturing raw data from the source and outputting it unmodified
317
+
-----------------------------------------------------------------
318
+
319
+
The ISP can additionally capture raw data from the source and output it on the
320
+
full resolution pipe only, completely unmodified. In this case the downscale
321
+
pipe can still process the data normally and be used at the same time.
322
+
323
+
To configure raw bypass the FR resizer's subdevice's routing table needs to be
324
+
configured, followed by formats in the appropriate places:
325
+
326
+
.. code-block:: none
327
+
328
+
media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[0],2/0->1/0[1]]"
329
+
media-ctl -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080]"
330
+
media-ctl -V "'mali-c55 resizer fr':2[fmt:RGB202020_1X60/1920x1080]"
331
+
media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB202020_1X60/1920x1080]"
332
+
333
+
# Set format on the video device and stream
334
+
yavta -f RGB565 -s 1920x1080 -c10 /dev/video0
335
+
336
+
References
337
+
==========
338
+
.. [1] https://git.linuxtv.org/v4l-utils.git/
339
+
.. [2] https://git.ideasonboard.org/yavta.git
340
+
.. [3] https://developer.arm.com/Processors/Mali-C55