Linux kernel mirror (for testing)
git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel
os
linux
Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux
Pull arm64 updates from Will Deacon:
"We've got a little less than normal thanks to the holidays in
December, but there's the usual summary below. The highlight is
probably the 52-bit physical addressing (LPA2) clean-up from Ard.
Confidential Computing:
- Register a platform device when running in CCA realm mode to enable
automatic loading of dependent modules
CPU Features:
- Update a bunch of system register definitions to pick up new field
encodings from the architectural documentation
- Add hwcaps and selftests for the new (2024) dpISA extensions
Documentation:
- Update EL3 (firmware) requirements for booting Linux on modern
arm64 designs
- Remove stale information about the kernel virtual memory map
Miscellaneous:
- Minor cleanups and typo fixes
Memory management:
- Fix vmemmap_check_pmd() to look at the PMD type bits
- LPA2 (52-bit physical addressing) cleanups and minor fixes
- Adjust physical address space depending upon whether or not LPA2 is
enabled
Perf and PMUs:
- Add port filtering support for NVIDIA's NVLINK-C2C Coresight PMU
- Extend AXI filtering support for the DDR PMU on NXP IMX SoCs
- Fix Designware PCIe PMU event numbering
- Add generic branch events for the Apple M1 CPU PMU
- Add support for Marvell Odyssey DDR and LLC-TAD PMUs
- Cleanups to the Hisilicon DDRC and Uncore PMU code
- Advertise discard mode for the SPE PMU
- Add the perf users mailing list to our MAINTAINERS entry"
* tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux: (64 commits)
Documentation: arm64: Remove stale and redundant virtual memory diagrams
perf docs: arm_spe: Document new discard mode
perf: arm_spe: Add format option for discard mode
MAINTAINERS: Add perf list for drivers/perf/
arm64: Remove duplicate included header
drivers/perf: apple_m1: Map generic branch events
arm64: rsi: Add automatic arm-cca-guest module loading
kselftest/arm64: Add 2024 dpISA extensions to hwcap test
KVM: arm64: Allow control of dpISA extensions in ID_AA64ISAR3_EL1
arm64/hwcap: Describe 2024 dpISA extensions to userspace
arm64/sysreg: Update ID_AA64SMFR0_EL1 to DDI0601 2024-12
arm64: Filter out SVE hwcaps when FEAT_SVE isn't implemented
drivers/perf: hisi: Set correct IRQ affinity for PMUs with no association
arm64/sme: Move storage of reg_smidr to __cpuinfo_store_cpu()
arm64: mm: Test for pmd_sect() in vmemmap_check_pmd()
arm64/mm: Replace open encodings with PXD_TABLE_BIT
arm64/mm: Rename pte_mkpresent() as pte_mkvalid()
arm64/sysreg: Update ID_AA64ISAR2_EL1 to DDI0601 2024-09
arm64/sysreg: Update ID_AA64ZFR0_EL1 to DDI0601 2024-09
arm64/sysreg: Update ID_AA64FPFR0_EL1 to DDI0601 2024-09
...
+1754
-750
+3
-3
Documentation/admin-guide/perf/dwc_pcie_pmu.rst
+3
-3
Documentation/admin-guide/perf/dwc_pcie_pmu.rst
···
60
60
The "format" directory describes format of the config fields of the
61
61
perf_event_attr structure. The "events" directory provides configuration
62
62
templates for all documented events. For example,
63
-
"Rx_PCIe_TLP_Data_Payload" is an equivalent of "eventid=0x22,type=0x1".
63
+
"rx_pcie_tlp_data_payload" is an equivalent of "eventid=0x21,type=0x0".
64
64
65
65
The "perf list" command shall list the available events from sysfs, e.g.::
66
66
···
79
79
80
80
The average RX/TX bandwidth can be calculated using the following formula:
81
81
82
-
PCIe RX Bandwidth = Rx_PCIe_TLP_Data_Payload / Measure_Time_Window
83
-
PCIe TX Bandwidth = Tx_PCIe_TLP_Data_Payload / Measure_Time_Window
82
+
PCIe RX Bandwidth = rx_pcie_tlp_data_payload / Measure_Time_Window
83
+
PCIe TX Bandwidth = tx_pcie_tlp_data_payload / Measure_Time_Window
84
84
85
85
Lane Event Usage
86
86
-------------------------------
+4
-1
Documentation/admin-guide/perf/hisi-pmu.rst
+4
-1
Documentation/admin-guide/perf/hisi-pmu.rst
···
35
35
SCCL ID #1.
36
36
37
37
The driver also provides a "cpumask" sysfs attribute, which shows the CPU core
38
-
ID used to count the uncore PMU event.
38
+
ID used to count the uncore PMU event. An "associated_cpus" sysfs attribute is
39
+
also provided to show the CPUs associated with this PMU. The "cpumask" indicates
40
+
the CPUs to open the events, usually as a hint for userspaces tools like perf.
41
+
It only contains one associated CPU from the "associated_cpus".
39
42
40
43
Example usage of perf::
41
44
+2
Documentation/admin-guide/perf/index.rst
+2
Documentation/admin-guide/perf/index.rst
+80
Documentation/admin-guide/perf/mrvl-odyssey-ddr-pmu.rst
+80
Documentation/admin-guide/perf/mrvl-odyssey-ddr-pmu.rst
···
1
+
===================================================================
2
+
Marvell Odyssey DDR PMU Performance Monitoring Unit (PMU UNCORE)
3
+
===================================================================
4
+
5
+
Odyssey DRAM Subsystem supports eight counters for monitoring performance
6
+
and software can program those counters to monitor any of the defined
7
+
performance events. Supported performance events include those counted
8
+
at the interface between the DDR controller and the PHY, interface between
9
+
the DDR Controller and the CHI interconnect, or within the DDR Controller.
10
+
11
+
Additionally DSS also supports two fixed performance event counters, one
12
+
for ddr reads and the other for ddr writes.
13
+
14
+
The counter will be operating in either manual or auto mode.
15
+
16
+
The PMU driver exposes the available events and format options under sysfs::
17
+
18
+
/sys/bus/event_source/devices/mrvl_ddr_pmu_<>/events/
19
+
/sys/bus/event_source/devices/mrvl_ddr_pmu_<>/format/
20
+
21
+
Examples::
22
+
23
+
$ perf list | grep ddr
24
+
mrvl_ddr_pmu_<>/ddr_act_bypass_access/ [Kernel PMU event]
25
+
mrvl_ddr_pmu_<>/ddr_bsm_alloc/ [Kernel PMU event]
26
+
mrvl_ddr_pmu_<>/ddr_bsm_starvation/ [Kernel PMU event]
27
+
mrvl_ddr_pmu_<>/ddr_cam_active_access/ [Kernel PMU event]
28
+
mrvl_ddr_pmu_<>/ddr_cam_mwr/ [Kernel PMU event]
29
+
mrvl_ddr_pmu_<>/ddr_cam_rd_active_access/ [Kernel PMU event]
30
+
mrvl_ddr_pmu_<>/ddr_cam_rd_or_wr_access/ [Kernel PMU event]
31
+
mrvl_ddr_pmu_<>/ddr_cam_read/ [Kernel PMU event]
32
+
mrvl_ddr_pmu_<>/ddr_cam_wr_access/ [Kernel PMU event]
33
+
mrvl_ddr_pmu_<>/ddr_cam_write/ [Kernel PMU event]
34
+
mrvl_ddr_pmu_<>/ddr_capar_error/ [Kernel PMU event]
35
+
mrvl_ddr_pmu_<>/ddr_crit_ref/ [Kernel PMU event]
36
+
mrvl_ddr_pmu_<>/ddr_ddr_reads/ [Kernel PMU event]
37
+
mrvl_ddr_pmu_<>/ddr_ddr_writes/ [Kernel PMU event]
38
+
mrvl_ddr_pmu_<>/ddr_dfi_cmd_is_retry/ [Kernel PMU event]
39
+
mrvl_ddr_pmu_<>/ddr_dfi_cycles/ [Kernel PMU event]
40
+
mrvl_ddr_pmu_<>/ddr_dfi_parity_poison/ [Kernel PMU event]
41
+
mrvl_ddr_pmu_<>/ddr_dfi_rd_data_access/ [Kernel PMU event]
42
+
mrvl_ddr_pmu_<>/ddr_dfi_wr_data_access/ [Kernel PMU event]
43
+
mrvl_ddr_pmu_<>/ddr_dqsosc_mpc/ [Kernel PMU event]
44
+
mrvl_ddr_pmu_<>/ddr_dqsosc_mrr/ [Kernel PMU event]
45
+
mrvl_ddr_pmu_<>/ddr_enter_mpsm/ [Kernel PMU event]
46
+
mrvl_ddr_pmu_<>/ddr_enter_powerdown/ [Kernel PMU event]
47
+
mrvl_ddr_pmu_<>/ddr_enter_selfref/ [Kernel PMU event]
48
+
mrvl_ddr_pmu_<>/ddr_hif_pri_rdaccess/ [Kernel PMU event]
49
+
mrvl_ddr_pmu_<>/ddr_hif_rd_access/ [Kernel PMU event]
50
+
mrvl_ddr_pmu_<>/ddr_hif_rd_or_wr_access/ [Kernel PMU event]
51
+
mrvl_ddr_pmu_<>/ddr_hif_rmw_access/ [Kernel PMU event]
52
+
mrvl_ddr_pmu_<>/ddr_hif_wr_access/ [Kernel PMU event]
53
+
mrvl_ddr_pmu_<>/ddr_hpri_sched_rd_crit_access/ [Kernel PMU event]
54
+
mrvl_ddr_pmu_<>/ddr_load_mode/ [Kernel PMU event]
55
+
mrvl_ddr_pmu_<>/ddr_lpri_sched_rd_crit_access/ [Kernel PMU event]
56
+
mrvl_ddr_pmu_<>/ddr_precharge/ [Kernel PMU event]
57
+
mrvl_ddr_pmu_<>/ddr_precharge_for_other/ [Kernel PMU event]
58
+
mrvl_ddr_pmu_<>/ddr_precharge_for_rdwr/ [Kernel PMU event]
59
+
mrvl_ddr_pmu_<>/ddr_raw_hazard/ [Kernel PMU event]
60
+
mrvl_ddr_pmu_<>/ddr_rd_bypass_access/ [Kernel PMU event]
61
+
mrvl_ddr_pmu_<>/ddr_rd_crc_error/ [Kernel PMU event]
62
+
mrvl_ddr_pmu_<>/ddr_rd_uc_ecc_error/ [Kernel PMU event]
63
+
mrvl_ddr_pmu_<>/ddr_rdwr_transitions/ [Kernel PMU event]
64
+
mrvl_ddr_pmu_<>/ddr_refresh/ [Kernel PMU event]
65
+
mrvl_ddr_pmu_<>/ddr_retry_fifo_full/ [Kernel PMU event]
66
+
mrvl_ddr_pmu_<>/ddr_spec_ref/ [Kernel PMU event]
67
+
mrvl_ddr_pmu_<>/ddr_tcr_mrr/ [Kernel PMU event]
68
+
mrvl_ddr_pmu_<>/ddr_war_hazard/ [Kernel PMU event]
69
+
mrvl_ddr_pmu_<>/ddr_waw_hazard/ [Kernel PMU event]
70
+
mrvl_ddr_pmu_<>/ddr_win_limit_reached_rd/ [Kernel PMU event]
71
+
mrvl_ddr_pmu_<>/ddr_win_limit_reached_wr/ [Kernel PMU event]
72
+
mrvl_ddr_pmu_<>/ddr_wr_crc_error/ [Kernel PMU event]
73
+
mrvl_ddr_pmu_<>/ddr_wr_trxn_crit_access/ [Kernel PMU event]
74
+
mrvl_ddr_pmu_<>/ddr_write_combine/ [Kernel PMU event]
75
+
mrvl_ddr_pmu_<>/ddr_zqcl/ [Kernel PMU event]
76
+
mrvl_ddr_pmu_<>/ddr_zqlatch/ [Kernel PMU event]
77
+
mrvl_ddr_pmu_<>/ddr_zqstart/ [Kernel PMU event]
78
+
79
+
$ perf stat -e ddr_cam_read,ddr_cam_write,ddr_cam_active_access,ddr_cam
80
+
rd_or_wr_access,ddr_cam_rd_active_access,ddr_cam_mwr <workload>
+37
Documentation/admin-guide/perf/mrvl-odyssey-tad-pmu.rst
+37
Documentation/admin-guide/perf/mrvl-odyssey-tad-pmu.rst
···
1
+
====================================================================
2
+
Marvell Odyssey LLC-TAD Performance Monitoring Unit (PMU UNCORE)
3
+
====================================================================
4
+
5
+
Each TAD provides eight 64-bit counters for monitoring
6
+
cache behavior.The driver always configures the same counter for
7
+
all the TADs. The user would end up effectively reserving one of
8
+
eight counters in every TAD to look across all TADs.
9
+
The occurrences of events are aggregated and presented to the user
10
+
at the end of running the workload. The driver does not provide a
11
+
way for the user to partition TADs so that different TADs are used for
12
+
different applications.
13
+
14
+
The performance events reflect various internal or interface activities.
15
+
By combining the values from multiple performance counters, cache
16
+
performance can be measured in terms such as: cache miss rate, cache
17
+
allocations, interface retry rate, internal resource occupancy, etc.
18
+
19
+
The PMU driver exposes the available events and format options under sysfs::
20
+
21
+
/sys/bus/event_source/devices/tad/events/
22
+
/sys/bus/event_source/devices/tad/format/
23
+
24
+
Examples::
25
+
26
+
$ perf list | grep tad
27
+
tad/tad_alloc_any/ [Kernel PMU event]
28
+
tad/tad_alloc_dtg/ [Kernel PMU event]
29
+
tad/tad_alloc_ltg/ [Kernel PMU event]
30
+
tad/tad_hit_any/ [Kernel PMU event]
31
+
tad/tad_hit_dtg/ [Kernel PMU event]
32
+
tad/tad_hit_ltg/ [Kernel PMU event]
33
+
tad/tad_req_msh_in_exlmn/ [Kernel PMU event]
34
+
tad/tad_tag_rd/ [Kernel PMU event]
35
+
tad/tad_tot_cycle/ [Kernel PMU event]
36
+
37
+
$ perf stat -e tad_alloc_dtg,tad_alloc_ltg,tad_alloc_any,tad_hit_dtg,tad_hit_ltg,tad_hit_any,tad_tag_rd <workload>
+43
-9
Documentation/admin-guide/perf/nvidia-pmu.rst
+43
-9
Documentation/admin-guide/perf/nvidia-pmu.rst
···
34
34
traffic coverage.
35
35
36
36
The events and configuration options of this PMU device are described in sysfs,
37
-
see /sys/bus/event_sources/devices/nvidia_scf_pmu_<socket-id>.
37
+
see /sys/bus/event_source/devices/nvidia_scf_pmu_<socket-id>.
38
38
39
39
Example usage:
40
40
···
66
66
the PMU traffic coverage.
67
67
68
68
The events and configuration options of this PMU device are described in sysfs,
69
-
see /sys/bus/event_sources/devices/nvidia_nvlink_c2c0_pmu_<socket-id>.
69
+
see /sys/bus/event_source/devices/nvidia_nvlink_c2c0_pmu_<socket-id>.
70
70
71
71
Example usage:
72
72
···
86
86
87
87
perf stat -a -e nvidia_nvlink_c2c0_pmu_3/event=0x0/
88
88
89
+
The NVLink-C2C has two ports that can be connected to one GPU (occupying both
90
+
ports) or to two GPUs (one GPU per port). The user can use "port" bitmap
91
+
parameter to select the port(s) to monitor. Each bit represents the port number,
92
+
e.g. "port=0x1" corresponds to port 0 and "port=0x3" is for port 0 and 1. The
93
+
PMU will monitor both ports by default if not specified.
94
+
95
+
Example for port filtering:
96
+
97
+
* Count event id 0x0 from the GPU connected with socket 0 on port 0::
98
+
99
+
perf stat -a -e nvidia_nvlink_c2c0_pmu_0/event=0x0,port=0x1/
100
+
101
+
* Count event id 0x0 from the GPUs connected with socket 0 on port 0 and port 1::
102
+
103
+
perf stat -a -e nvidia_nvlink_c2c0_pmu_0/event=0x0,port=0x3/
104
+
89
105
NVLink-C2C1 PMU
90
106
-------------------
91
107
···
112
96
the PMU traffic coverage.
113
97
114
98
The events and configuration options of this PMU device are described in sysfs,
115
-
see /sys/bus/event_sources/devices/nvidia_nvlink_c2c1_pmu_<socket-id>.
99
+
see /sys/bus/event_source/devices/nvidia_nvlink_c2c1_pmu_<socket-id>.
116
100
117
101
Example usage:
118
102
···
132
116
133
117
perf stat -a -e nvidia_nvlink_c2c1_pmu_3/event=0x0/
134
118
119
+
The NVLink-C2C has two ports that can be connected to one GPU (occupying both
120
+
ports) or to two GPUs (one GPU per port). The user can use "port" bitmap
121
+
parameter to select the port(s) to monitor. Each bit represents the port number,
122
+
e.g. "port=0x1" corresponds to port 0 and "port=0x3" is for port 0 and 1. The
123
+
PMU will monitor both ports by default if not specified.
124
+
125
+
Example for port filtering:
126
+
127
+
* Count event id 0x0 from the GPU connected with socket 0 on port 0::
128
+
129
+
perf stat -a -e nvidia_nvlink_c2c1_pmu_0/event=0x0,port=0x1/
130
+
131
+
* Count event id 0x0 from the GPUs connected with socket 0 on port 0 and port 1::
132
+
133
+
perf stat -a -e nvidia_nvlink_c2c1_pmu_0/event=0x0,port=0x3/
134
+
135
135
CNVLink PMU
136
136
---------------
137
137
···
157
125
for more info about the PMU traffic coverage.
158
126
159
127
The events and configuration options of this PMU device are described in sysfs,
160
-
see /sys/bus/event_sources/devices/nvidia_cnvlink_pmu_<socket-id>.
128
+
see /sys/bus/event_source/devices/nvidia_cnvlink_pmu_<socket-id>.
161
129
162
130
Each SoC socket can be connected to one or more sockets via CNVLink. The user can
163
131
use "rem_socket" bitmap parameter to select the remote socket(s) to monitor.
164
132
Each bit represents the socket number, e.g. "rem_socket=0xE" corresponds to
165
-
socket 1 to 3.
166
-
/sys/bus/event_sources/devices/nvidia_cnvlink_pmu_<socket-id>/format/rem_socket
133
+
socket 1 to 3. The PMU will monitor all remote sockets by default if not
134
+
specified.
135
+
/sys/bus/event_source/devices/nvidia_cnvlink_pmu_<socket-id>/format/rem_socket
167
136
shows the valid bits that can be set in the "rem_socket" parameter.
168
137
169
138
The PMU can not distinguish the remote traffic initiator, therefore it does not
···
198
165
for more info about the PMU traffic coverage.
199
166
200
167
The events and configuration options of this PMU device are described in sysfs,
201
-
see /sys/bus/event_sources/devices/nvidia_pcie_pmu_<socket-id>.
168
+
see /sys/bus/event_source/devices/nvidia_pcie_pmu_<socket-id>.
202
169
203
170
Each SoC socket can support multiple root ports. The user can use
204
171
"root_port" bitmap parameter to select the port(s) to monitor, i.e.
205
-
"root_port=0xF" corresponds to root port 0 to 3.
206
-
/sys/bus/event_sources/devices/nvidia_pcie_pmu_<socket-id>/format/root_port
172
+
"root_port=0xF" corresponds to root port 0 to 3. The PMU will monitor all root
173
+
ports by default if not specified.
174
+
/sys/bus/event_source/devices/nvidia_pcie_pmu_<socket-id>/format/root_port
207
175
shows the valid bits that can be set in the "root_port" parameter.
208
176
209
177
Example usage:
+12
Documentation/arch/arm64/booting.rst
+12
Documentation/arch/arm64/booting.rst
···
449
449
450
450
- HFGWTR_EL2.nGCS_EL0 (bit 52) must be initialised to 0b1.
451
451
452
+
- For CPUs with debug architecture i.e FEAT_Debugv8pN (all versions):
453
+
454
+
- If EL3 is present:
455
+
456
+
- MDCR_EL3.TDA (bit 9) must be initialized to 0b0
457
+
458
+
- For CPUs with FEAT_PMUv3:
459
+
460
+
- If EL3 is present:
461
+
462
+
- MDCR_EL3.TPM (bit 6) must be initialized to 0b0
463
+
452
464
The requirements described above for CPU mode, caches, MMUs, architected
453
465
timers, coherency and system registers apply to all CPUs. All CPUs must
454
466
enter the kernel in the same exception level. Where the values documented
+76
-13
Documentation/arch/arm64/elf_hwcaps.rst
+76
-13
Documentation/arch/arm64/elf_hwcaps.rst
···
174
174
Functionality implied by ID_AA64PFR1_EL1.GCS == 0b1, as
175
175
described by Documentation/arch/arm64/gcs.rst.
176
176
177
+
HWCAP_CMPBR
178
+
Functionality implied by ID_AA64ISAR2_EL1.CSSC == 0b0010.
179
+
180
+
HWCAP_FPRCVT
181
+
Functionality implied by ID_AA64ISAR3_EL1.FPRCVT == 0b0001.
182
+
183
+
HWCAP_F8MM8
184
+
Functionality implied by ID_AA64FPFR0_EL1.F8MM8 == 0b0001.
185
+
186
+
HWCAP_F8MM4
187
+
Functionality implied by ID_AA64FPFR0_EL1.F8MM4 == 0b0001.
188
+
189
+
HWCAP_SVE_F16MM
190
+
Functionality implied by ID_AA64PFR0_EL1.SVE == 0b0001 and
191
+
ID_AA64ZFR0_EL1.F16MM == 0b0001.
192
+
193
+
HWCAP_SVE_ELTPERM
194
+
Functionality implied by ID_AA64PFR0_EL1.SVE == 0b0001 and
195
+
ID_AA64ZFR0_EL1.ELTPERM == 0b0001.
196
+
197
+
HWCAP_SVE_AES2
198
+
Functionality implied by ID_AA64PFR0_EL1.SVE == 0b0001 and
199
+
ID_AA64ZFR0_EL1.AES == 0b0011.
200
+
201
+
HWCAP_SVE_BFSCALE
202
+
Functionality implied by ID_AA64PFR0_EL1.SVE == 0b0001 and
203
+
ID_AA64ZFR0_EL1.B16B16 == 0b0010.
204
+
205
+
HWCAP_SVE2P2
206
+
Functionality implied by ID_AA64PFR0_EL1.SVE == 0b0001 and
207
+
ID_AA64ZFR0_EL1.SVEver == 0b0011.
208
+
209
+
HWCAP_SME2P2
210
+
Functionality implied by ID_AA64SMFR0_EL1.SMEver == 0b0011.
211
+
212
+
HWCAP_SME_SBITPERM
213
+
Functionality implied by ID_AA64SMFR0_EL1.SBitPerm == 0b1.
214
+
215
+
HWCAP_SME_AES
216
+
Functionality implied by ID_AA64SMFR0_EL1.AES == 0b1.
217
+
218
+
HWCAP_SME_SFEXPA
219
+
Functionality implied by ID_AA64SMFR0_EL1.SFEXPA == 0b1.
220
+
221
+
HWCAP_SME_STMOP
222
+
Functionality implied by ID_AA64SMFR0_EL1.STMOP == 0b1.
223
+
224
+
HWCAP_SME_SMOP4
225
+
Functionality implied by ID_AA64SMFR0_EL1.SMOP4 == 0b1.
226
+
177
227
HWCAP2_DCPODP
178
228
Functionality implied by ID_AA64ISAR1_EL1.DPB == 0b0010.
179
229
180
230
HWCAP2_SVE2
181
-
Functionality implied by ID_AA64ZFR0_EL1.SVEver == 0b0001.
231
+
Functionality implied by ID_AA64PFR0_EL1.SVE == 0b0001 and
232
+
ID_AA64ZFR0_EL1.SVEver == 0b0001.
182
233
183
234
HWCAP2_SVEAES
184
-
Functionality implied by ID_AA64ZFR0_EL1.AES == 0b0001.
235
+
Functionality implied by ID_AA64PFR0_EL1.SVE == 0b0001 and
236
+
ID_AA64ZFR0_EL1.AES == 0b0001.
185
237
186
238
HWCAP2_SVEPMULL
187
-
Functionality implied by ID_AA64ZFR0_EL1.AES == 0b0010.
239
+
Functionality implied by ID_AA64PFR0_EL1.SVE == 0b0001 and
240
+
ID_AA64ZFR0_EL1.AES == 0b0010.
188
241
189
242
HWCAP2_SVEBITPERM
190
-
Functionality implied by ID_AA64ZFR0_EL1.BitPerm == 0b0001.
243
+
Functionality implied by ID_AA64PFR0_EL1.SVE == 0b0001 and
244
+
ID_AA64ZFR0_EL1.BitPerm == 0b0001.
191
245
192
246
HWCAP2_SVESHA3
193
-
Functionality implied by ID_AA64ZFR0_EL1.SHA3 == 0b0001.
247
+
Functionality implied by ID_AA64PFR0_EL1.SVE == 0b0001 and
248
+
ID_AA64ZFR0_EL1.SHA3 == 0b0001.
194
249
195
250
HWCAP2_SVESM4
196
-
Functionality implied by ID_AA64ZFR0_EL1.SM4 == 0b0001.
251
+
Functionality implied by ID_AA64PFR0_EL1.SVE == 0b0001 and
252
+
ID_AA64ZFR0_EL1.SM4 == 0b0001.
197
253
198
254
HWCAP2_FLAGM2
199
255
Functionality implied by ID_AA64ISAR0_EL1.TS == 0b0010.
···
258
202
Functionality implied by ID_AA64ISAR1_EL1.FRINTTS == 0b0001.
259
203
260
204
HWCAP2_SVEI8MM
261
-
Functionality implied by ID_AA64ZFR0_EL1.I8MM == 0b0001.
205
+
Functionality implied by ID_AA64PFR0_EL1.SVE == 0b0001 and
206
+
ID_AA64ZFR0_EL1.I8MM == 0b0001.
262
207
263
208
HWCAP2_SVEF32MM
264
-
Functionality implied by ID_AA64ZFR0_EL1.F32MM == 0b0001.
209
+
Functionality implied by ID_AA64PFR0_EL1.SVE == 0b0001 and
210
+
ID_AA64ZFR0_EL1.F32MM == 0b0001.
265
211
266
212
HWCAP2_SVEF64MM
267
-
Functionality implied by ID_AA64ZFR0_EL1.F64MM == 0b0001.
213
+
Functionality implied by ID_AA64PFR0_EL1.SVE == 0b0001 and
214
+
ID_AA64ZFR0_EL1.F64MM == 0b0001.
268
215
269
216
HWCAP2_SVEBF16
270
-
Functionality implied by ID_AA64ZFR0_EL1.BF16 == 0b0001.
217
+
Functionality implied by ID_AA64PFR0_EL1.SVE == 0b0001 and
218
+
ID_AA64ZFR0_EL1.BF16 == 0b0001.
271
219
272
220
HWCAP2_I8MM
273
221
Functionality implied by ID_AA64ISAR1_EL1.I8MM == 0b0001.
···
337
277
Functionality implied by ID_AA64ISAR1_EL1.BF16 == 0b0010.
338
278
339
279
HWCAP2_SVE_EBF16
340
-
Functionality implied by ID_AA64ZFR0_EL1.BF16 == 0b0010.
280
+
Functionality implied by ID_AA64PFR0_EL1.SVE == 0b0001 and
281
+
ID_AA64ZFR0_EL1.BF16 == 0b0010.
341
282
342
283
HWCAP2_CSSC
343
284
Functionality implied by ID_AA64ISAR2_EL1.CSSC == 0b0001.
···
347
286
Functionality implied by ID_AA64ISAR2_EL1.RPRFM == 0b0001.
348
287
349
288
HWCAP2_SVE2P1
350
-
Functionality implied by ID_AA64ZFR0_EL1.SVEver == 0b0010.
289
+
Functionality implied by ID_AA64PFR0_EL1.SVE == 0b0001 and
290
+
ID_AA64ZFR0_EL1.SVEver == 0b0010.
351
291
352
292
HWCAP2_SME2
353
293
Functionality implied by ID_AA64SMFR0_EL1.SMEver == 0b0001.
···
375
313
Functionality implied by ID_AA64ISAR2_EL1.BC == 0b0001.
376
314
377
315
HWCAP2_SVE_B16B16
378
-
Functionality implied by ID_AA64ZFR0_EL1.B16B16 == 0b0001.
316
+
Functionality implied by ID_AA64PFR0_EL1.SVE == 0b0001 and
317
+
ID_AA64ZFR0_EL1.B16B16 == 0b0001.
379
318
380
319
HWCAP2_LRCPC3
381
320
Functionality implied by ID_AA64ISAR1_EL1.LRCPC == 0b0011.
-65
Documentation/arch/arm64/memory.rst
-65
Documentation/arch/arm64/memory.rst
···
23
23
contains only user (non-global) mappings. The swapper_pg_dir address is
24
24
written to TTBR1 and never written to TTBR0.
25
25
26
-
27
-
AArch64 Linux memory layout with 4KB pages + 4 levels (48-bit)::
28
-
29
-
Start End Size Use
30
-
-----------------------------------------------------------------------
31
-
0000000000000000 0000ffffffffffff 256TB user
32
-
ffff000000000000 ffff7fffffffffff 128TB kernel logical memory map
33
-
[ffff600000000000 ffff7fffffffffff] 32TB [kasan shadow region]
34
-
ffff800000000000 ffff80007fffffff 2GB modules
35
-
ffff800080000000 fffffbffefffffff 124TB vmalloc
36
-
fffffbfff0000000 fffffbfffdffffff 224MB fixed mappings (top down)
37
-
fffffbfffe000000 fffffbfffe7fffff 8MB [guard region]
38
-
fffffbfffe800000 fffffbffff7fffff 16MB PCI I/O space
39
-
fffffbffff800000 fffffbffffffffff 8MB [guard region]
40
-
fffffc0000000000 fffffdffffffffff 2TB vmemmap
41
-
fffffe0000000000 ffffffffffffffff 2TB [guard region]
42
-
43
-
44
-
AArch64 Linux memory layout with 64KB pages + 3 levels (52-bit with HW support)::
45
-
46
-
Start End Size Use
47
-
-----------------------------------------------------------------------
48
-
0000000000000000 000fffffffffffff 4PB user
49
-
fff0000000000000 ffff7fffffffffff ~4PB kernel logical memory map
50
-
[fffd800000000000 ffff7fffffffffff] 512TB [kasan shadow region]
51
-
ffff800000000000 ffff80007fffffff 2GB modules
52
-
ffff800080000000 fffffbffefffffff 124TB vmalloc
53
-
fffffbfff0000000 fffffbfffdffffff 224MB fixed mappings (top down)
54
-
fffffbfffe000000 fffffbfffe7fffff 8MB [guard region]
55
-
fffffbfffe800000 fffffbffff7fffff 16MB PCI I/O space
56
-
fffffbffff800000 fffffbffffffffff 8MB [guard region]
57
-
fffffc0000000000 ffffffdfffffffff ~4TB vmemmap
58
-
ffffffe000000000 ffffffffffffffff 128GB [guard region]
59
-
60
-
61
-
Translation table lookup with 4KB pages::
62
-
63
-
+--------+--------+--------+--------+--------+--------+--------+--------+
64
-
|63 56|55 48|47 40|39 32|31 24|23 16|15 8|7 0|
65
-
+--------+--------+--------+--------+--------+--------+--------+--------+
66
-
| | | | | |
67
-
| | | | | v
68
-
| | | | | [11:0] in-page offset
69
-
| | | | +-> [20:12] L3 index
70
-
| | | +-----------> [29:21] L2 index
71
-
| | +---------------------> [38:30] L1 index
72
-
| +-------------------------------> [47:39] L0 index
73
-
+----------------------------------------> [55] TTBR0/1
74
-
75
-
76
-
Translation table lookup with 64KB pages::
77
-
78
-
+--------+--------+--------+--------+--------+--------+--------+--------+
79
-
|63 56|55 48|47 40|39 32|31 24|23 16|15 8|7 0|
80
-
+--------+--------+--------+--------+--------+--------+--------+--------+
81
-
| | | | |
82
-
| | | | v
83
-
| | | | [15:0] in-page offset
84
-
| | | +----------> [28:16] L3 index
85
-
| | +--------------------------> [41:29] L2 index
86
-
| +-------------------------------> [47:42] L1 index (48-bit)
87
-
| [51:42] L1 index (52-bit)
88
-
+----------------------------------------> [55] TTBR0/1
89
-
90
-
91
26
When using KVM without the Virtualization Host Extensions, the
92
27
hypervisor maps kernel pages in EL2 at a fixed (and potentially
93
28
random) offset from the linear mapping. See the kern_hyp_va macro and
+1
MAINTAINERS
+1
MAINTAINERS
···
1918
1918
M: Will Deacon <will@kernel.org>
1919
1919
M: Mark Rutland <mark.rutland@arm.com>
1920
1920
L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers)
1921
+
L: linux-perf-users@vger.kernel.org
1921
1922
S: Maintained
1922
1923
F: Documentation/devicetree/bindings/arm/pmu.yaml
1923
1924
F: Documentation/devicetree/bindings/perf/
+3
-5
arch/arm64/Kconfig
+3
-5
arch/arm64/Kconfig
···
113
113
select ARCH_WANT_FRAME_POINTERS
114
114
select ARCH_WANT_HUGE_PMD_SHARE if ARM64_4K_PAGES || (ARM64_16K_PAGES && !ARM64_VA_BITS_36)
115
115
select ARCH_WANT_LD_ORPHAN_WARN
116
-
select ARCH_WANTS_EXECMEM_LATE if EXECMEM
116
+
select ARCH_WANTS_EXECMEM_LATE
117
117
select ARCH_WANTS_NO_INSTR
118
118
select ARCH_WANTS_THP_SWAP if ARM64_4K_PAGES
119
119
select ARCH_HAS_UBSAN
···
1379
1379
1380
1380
config ARM64_VA_BITS_52
1381
1381
bool "52-bit"
1382
-
depends on ARM64_PAN || !ARM64_SW_TTBR0_PAN
1383
1382
help
1384
1383
Enable 52-bit virtual addressing for userspace when explicitly
1385
1384
requested via a hint to mmap(). The kernel will also use 52-bit
···
1430
1431
config ARM64_PA_BITS_52
1431
1432
bool "52-bit"
1432
1433
depends on ARM64_64K_PAGES || ARM64_VA_BITS_52
1433
-
depends on ARM64_PAN || !ARM64_SW_TTBR0_PAN
1434
1434
help
1435
1435
Enable support for a 52-bit physical address space, introduced as
1436
1436
part of the ARMv8.2-LPA extension.
···
1679
1681
config ARM64_SW_TTBR0_PAN
1680
1682
bool "Emulate Privileged Access Never using TTBR0_EL1 switching"
1681
1683
depends on !KCSAN
1684
+
select ARM64_PAN
1682
1685
help
1683
1686
Enabling this option prevents the kernel from accessing
1684
1687
user-space memory directly by pointing TTBR0_EL1 to a reserved
···
1936
1937
config ARM64_CNP
1937
1938
bool "Enable support for Common Not Private (CNP) translations"
1938
1939
default y
1939
-
depends on ARM64_PAN || !ARM64_SW_TTBR0_PAN
1940
1940
help
1941
1941
Common Not Private (CNP) allows translation table entries to
1942
1942
be shared between different PEs in the same inner shareable
···
2130
2132
depends on AS_HAS_ARMV8_5
2131
2133
depends on AS_HAS_LSE_ATOMICS
2132
2134
# Required for tag checking in the uaccess routines
2133
-
depends on ARM64_PAN
2135
+
select ARM64_PAN
2134
2136
select ARCH_HAS_SUBPAGE_FAULTS
2135
2137
select ARCH_USES_HIGH_VMA_FLAGS
2136
2138
select ARCH_USES_PG_ARCH_2
+5
arch/arm64/include/asm/assembler.h
+5
arch/arm64/include/asm/assembler.h
···
343
343
// Narrow PARange to fit the PS field in TCR_ELx
344
344
ubfx \tmp0, \tmp0, #ID_AA64MMFR0_EL1_PARANGE_SHIFT, #3
345
345
mov \tmp1, #ID_AA64MMFR0_EL1_PARANGE_MAX
346
+
#ifdef CONFIG_ARM64_LPA2
347
+
alternative_if_not ARM64_HAS_VA52
348
+
mov \tmp1, #ID_AA64MMFR0_EL1_PARANGE_48
349
+
alternative_else_nop_endif
350
+
#endif
346
351
cmp \tmp0, \tmp1
347
352
csel \tmp0, \tmp1, \tmp0, hi
348
353
bfi \tcr, \tmp0, \pos, #3
+2
arch/arm64/include/asm/cpucaps.h
+2
arch/arm64/include/asm/cpucaps.h
···
46
46
return IS_ENABLED(CONFIG_ARM64_POE);
47
47
case ARM64_HAS_GCS:
48
48
return IS_ENABLED(CONFIG_ARM64_GCS);
49
+
case ARM64_HAFT:
50
+
return IS_ENABLED(CONFIG_ARM64_HAFT);
49
51
case ARM64_UNMAP_KERNEL_AT_EL0:
50
52
return IS_ENABLED(CONFIG_UNMAP_KERNEL_AT_EL0);
51
53
case ARM64_WORKAROUND_843419:
+1
-2
arch/arm64/include/asm/cpufeature.h
+1
-2
arch/arm64/include/asm/cpufeature.h
+3
-3
arch/arm64/include/asm/el2_setup.h
+3
-3
arch/arm64/include/asm/el2_setup.h
···
154
154
/* Coprocessor traps */
155
155
.macro __init_el2_cptr
156
156
__check_hvhe .LnVHE_\@, x1
157
-
mov x0, #CPACR_ELx_FPEN
157
+
mov x0, #CPACR_EL1_FPEN
158
158
msr cpacr_el1, x0
159
159
b .Lskip_set_cptr_\@
160
160
.LnVHE_\@:
···
332
332
333
333
// (h)VHE case
334
334
mrs x0, cpacr_el1 // Disable SVE traps
335
-
orr x0, x0, #CPACR_ELx_ZEN
335
+
orr x0, x0, #CPACR_EL1_ZEN
336
336
msr cpacr_el1, x0
337
337
b .Lskip_set_cptr_\@
338
338
···
353
353
354
354
// (h)VHE case
355
355
mrs x0, cpacr_el1 // Disable SME traps
356
-
orr x0, x0, #CPACR_ELx_SMEN
356
+
orr x0, x0, #CPACR_EL1_SMEN
357
357
msr cpacr_el1, x0
358
358
b .Lskip_set_cptr_sme_\@
359
359
+15
arch/arm64/include/asm/hwcap.h
+15
arch/arm64/include/asm/hwcap.h
···
93
93
#define KERNEL_HWCAP_PACA __khwcap_feature(PACA)
94
94
#define KERNEL_HWCAP_PACG __khwcap_feature(PACG)
95
95
#define KERNEL_HWCAP_GCS __khwcap_feature(GCS)
96
+
#define KERNEL_HWCAP_CMPBR __khwcap_feature(CMPBR)
97
+
#define KERNEL_HWCAP_FPRCVT __khwcap_feature(FPRCVT)
98
+
#define KERNEL_HWCAP_F8MM8 __khwcap_feature(F8MM8)
99
+
#define KERNEL_HWCAP_F8MM4 __khwcap_feature(F8MM4)
100
+
#define KERNEL_HWCAP_SVE_F16MM __khwcap_feature(SVE_F16MM)
101
+
#define KERNEL_HWCAP_SVE_ELTPERM __khwcap_feature(SVE_ELTPERM)
102
+
#define KERNEL_HWCAP_SVE_AES2 __khwcap_feature(SVE_AES2)
103
+
#define KERNEL_HWCAP_SVE_BFSCALE __khwcap_feature(SVE_BFSCALE)
104
+
#define KERNEL_HWCAP_SVE2P2 __khwcap_feature(SVE2P2)
105
+
#define KERNEL_HWCAP_SME2P2 __khwcap_feature(SME2P2)
106
+
#define KERNEL_HWCAP_SME_SBITPERM __khwcap_feature(SME_SBITPERM)
107
+
#define KERNEL_HWCAP_SME_AES __khwcap_feature(SME_AES)
108
+
#define KERNEL_HWCAP_SME_SFEXPA __khwcap_feature(SME_SFEXPA)
109
+
#define KERNEL_HWCAP_SME_STMOP __khwcap_feature(SME_STMOP)
110
+
#define KERNEL_HWCAP_SME_SMOP4 __khwcap_feature(SME_SMOP4)
96
111
97
112
#define __khwcap2_feature(x) (const_ilog2(HWCAP2_ ## x) + 64)
98
113
#define KERNEL_HWCAP_DCPODP __khwcap2_feature(DCPODP)
-2
arch/arm64/include/asm/kvm_arm.h
-2
arch/arm64/include/asm/kvm_arm.h
+20
-20
arch/arm64/include/asm/kvm_emulate.h
+20
-20
arch/arm64/include/asm/kvm_emulate.h
···
556
556
({ \
557
557
u64 cptr = 0; \
558
558
\
559
-
if ((set) & CPACR_ELx_FPEN) \
559
+
if ((set) & CPACR_EL1_FPEN) \
560
560
cptr |= CPTR_EL2_TFP; \
561
-
if ((set) & CPACR_ELx_ZEN) \
561
+
if ((set) & CPACR_EL1_ZEN) \
562
562
cptr |= CPTR_EL2_TZ; \
563
-
if ((set) & CPACR_ELx_SMEN) \
563
+
if ((set) & CPACR_EL1_SMEN) \
564
564
cptr |= CPTR_EL2_TSM; \
565
-
if ((clr) & CPACR_ELx_TTA) \
565
+
if ((clr) & CPACR_EL1_TTA) \
566
566
cptr |= CPTR_EL2_TTA; \
567
567
if ((clr) & CPTR_EL2_TAM) \
568
568
cptr |= CPTR_EL2_TAM; \
···
576
576
({ \
577
577
u64 cptr = 0; \
578
578
\
579
-
if ((clr) & CPACR_ELx_FPEN) \
579
+
if ((clr) & CPACR_EL1_FPEN) \
580
580
cptr |= CPTR_EL2_TFP; \
581
-
if ((clr) & CPACR_ELx_ZEN) \
581
+
if ((clr) & CPACR_EL1_ZEN) \
582
582
cptr |= CPTR_EL2_TZ; \
583
-
if ((clr) & CPACR_ELx_SMEN) \
583
+
if ((clr) & CPACR_EL1_SMEN) \
584
584
cptr |= CPTR_EL2_TSM; \
585
-
if ((set) & CPACR_ELx_TTA) \
585
+
if ((set) & CPACR_EL1_TTA) \
586
586
cptr |= CPTR_EL2_TTA; \
587
587
if ((set) & CPTR_EL2_TAM) \
588
588
cptr |= CPTR_EL2_TAM; \
···
595
595
#define cpacr_clear_set(clr, set) \
596
596
do { \
597
597
BUILD_BUG_ON((set) & CPTR_VHE_EL2_RES0); \
598
-
BUILD_BUG_ON((clr) & CPACR_ELx_E0POE); \
599
-
__build_check_all_or_none((clr), CPACR_ELx_FPEN); \
600
-
__build_check_all_or_none((set), CPACR_ELx_FPEN); \
601
-
__build_check_all_or_none((clr), CPACR_ELx_ZEN); \
602
-
__build_check_all_or_none((set), CPACR_ELx_ZEN); \
603
-
__build_check_all_or_none((clr), CPACR_ELx_SMEN); \
604
-
__build_check_all_or_none((set), CPACR_ELx_SMEN); \
598
+
BUILD_BUG_ON((clr) & CPACR_EL1_E0POE); \
599
+
__build_check_all_or_none((clr), CPACR_EL1_FPEN); \
600
+
__build_check_all_or_none((set), CPACR_EL1_FPEN); \
601
+
__build_check_all_or_none((clr), CPACR_EL1_ZEN); \
602
+
__build_check_all_or_none((set), CPACR_EL1_ZEN); \
603
+
__build_check_all_or_none((clr), CPACR_EL1_SMEN); \
604
+
__build_check_all_or_none((set), CPACR_EL1_SMEN); \
605
605
\
606
606
if (has_vhe() || has_hvhe()) \
607
607
sysreg_clear_set(cpacr_el1, clr, set); \
···
624
624
u64 val;
625
625
626
626
if (has_vhe()) {
627
-
val = (CPACR_ELx_FPEN | CPACR_EL1_ZEN_EL1EN);
627
+
val = (CPACR_EL1_FPEN | CPACR_EL1_ZEN_EL1EN);
628
628
if (cpus_have_final_cap(ARM64_SME))
629
629
val |= CPACR_EL1_SMEN_EL1EN;
630
630
} else if (has_hvhe()) {
631
-
val = CPACR_ELx_FPEN;
631
+
val = CPACR_EL1_FPEN;
632
632
633
633
if (!vcpu_has_sve(vcpu) || !guest_owns_fp_regs())
634
-
val |= CPACR_ELx_ZEN;
634
+
val |= CPACR_EL1_ZEN;
635
635
if (cpus_have_final_cap(ARM64_SME))
636
-
val |= CPACR_ELx_SMEN;
636
+
val |= CPACR_EL1_SMEN;
637
637
} else {
638
638
val = CPTR_NVHE_EL2_RES1;
639
639
···
685
685
#define __guest_hyp_cptr_xen_trap_enabled(vcpu, xen) \
686
686
(!vcpu_has_nv(vcpu) ? false : \
687
687
____cptr_xen_trap_enabled(vcpu, \
688
-
SYS_FIELD_GET(CPACR_ELx, xen, \
688
+
SYS_FIELD_GET(CPACR_EL1, xen, \
689
689
vcpu_sanitised_cptr_el2(vcpu))))
690
690
691
691
static inline bool guest_hyp_fpsimd_traps_enabled(const struct kvm_vcpu *vcpu)
+4
-4
arch/arm64/include/asm/kvm_nested.h
+4
-4
arch/arm64/include/asm/kvm_nested.h
···
33
33
34
34
static inline u64 translate_cptr_el2_to_cpacr_el1(u64 cptr_el2)
35
35
{
36
-
u64 cpacr_el1 = CPACR_ELx_RES1;
36
+
u64 cpacr_el1 = CPACR_EL1_RES1;
37
37
38
38
if (cptr_el2 & CPTR_EL2_TTA)
39
-
cpacr_el1 |= CPACR_ELx_TTA;
39
+
cpacr_el1 |= CPACR_EL1_TTA;
40
40
if (!(cptr_el2 & CPTR_EL2_TFP))
41
-
cpacr_el1 |= CPACR_ELx_FPEN;
41
+
cpacr_el1 |= CPACR_EL1_FPEN;
42
42
if (!(cptr_el2 & CPTR_EL2_TZ))
43
-
cpacr_el1 |= CPACR_ELx_ZEN;
43
+
cpacr_el1 |= CPACR_EL1_ZEN;
44
44
45
45
cpacr_el1 |= cptr_el2 & (CPTR_EL2_TCPAC | CPTR_EL2_TAM);
46
46
-3
arch/arm64/include/asm/mmu.h
-3
arch/arm64/include/asm/mmu.h
-6
arch/arm64/include/asm/pgtable-hwdef.h
-6
arch/arm64/include/asm/pgtable-hwdef.h
···
222
222
*/
223
223
#define S1_TABLE_AP (_AT(pmdval_t, 3) << 61)
224
224
225
-
/*
226
-
* Highest possible physical address supported.
227
-
*/
228
-
#define PHYS_MASK_SHIFT (CONFIG_ARM64_PA_BITS)
229
-
#define PHYS_MASK ((UL(1) << PHYS_MASK_SHIFT) - 1)
230
-
231
225
#define TTBR_CNP_BIT (UL(1) << 0)
232
226
233
227
/*
+7
arch/arm64/include/asm/pgtable-prot.h
+7
arch/arm64/include/asm/pgtable-prot.h
···
81
81
#define lpa2_is_enabled() false
82
82
#define PTE_MAYBE_SHARED PTE_SHARED
83
83
#define PMD_MAYBE_SHARED PMD_SECT_S
84
+
#define PHYS_MASK_SHIFT (CONFIG_ARM64_PA_BITS)
84
85
#else
85
86
static inline bool __pure lpa2_is_enabled(void)
86
87
{
···
90
89
91
90
#define PTE_MAYBE_SHARED (lpa2_is_enabled() ? 0 : PTE_SHARED)
92
91
#define PMD_MAYBE_SHARED (lpa2_is_enabled() ? 0 : PMD_SECT_S)
92
+
#define PHYS_MASK_SHIFT (lpa2_is_enabled() ? CONFIG_ARM64_PA_BITS : 48)
93
93
#endif
94
+
95
+
/*
96
+
* Highest possible physical address supported.
97
+
*/
98
+
#define PHYS_MASK ((UL(1) << PHYS_MASK_SHIFT) - 1)
94
99
95
100
/*
96
101
* If we have userspace only BTI we don't want to mark kernel pages
+4
-4
arch/arm64/include/asm/pgtable.h
+4
-4
arch/arm64/include/asm/pgtable.h
···
273
273
return clear_pte_bit(pte, __pgprot(PTE_CONT));
274
274
}
275
275
276
-
static inline pte_t pte_mkpresent(pte_t pte)
276
+
static inline pte_t pte_mkvalid(pte_t pte)
277
277
{
278
278
return set_pte_bit(pte, __pgprot(PTE_VALID));
279
279
}
···
896
896
pr_err("%s:%d: bad pud %016llx.\n", __FILE__, __LINE__, pud_val(e))
897
897
898
898
#define p4d_none(p4d) (pgtable_l4_enabled() && !p4d_val(p4d))
899
-
#define p4d_bad(p4d) (pgtable_l4_enabled() && !(p4d_val(p4d) & 2))
899
+
#define p4d_bad(p4d) (pgtable_l4_enabled() && !(p4d_val(p4d) & P4D_TABLE_BIT))
900
900
#define p4d_present(p4d) (!p4d_none(p4d))
901
901
902
902
static inline void set_p4d(p4d_t *p4dp, p4d_t p4d)
···
1023
1023
pr_err("%s:%d: bad p4d %016llx.\n", __FILE__, __LINE__, p4d_val(e))
1024
1024
1025
1025
#define pgd_none(pgd) (pgtable_l5_enabled() && !pgd_val(pgd))
1026
-
#define pgd_bad(pgd) (pgtable_l5_enabled() && !(pgd_val(pgd) & 2))
1026
+
#define pgd_bad(pgd) (pgtable_l5_enabled() && !(pgd_val(pgd) & PGD_TABLE_BIT))
1027
1027
#define pgd_present(pgd) (!pgd_none(pgd))
1028
1028
1029
1029
static inline void set_pgd(pgd_t *pgdp, pgd_t pgd)
···
1345
1345
}
1346
1346
1347
1347
/*
1348
-
* __ptep_set_wrprotect - mark read-only while trasferring potential hardware
1348
+
* __ptep_set_wrprotect - mark read-only while transferring potential hardware
1349
1349
* dirty status (PTE_DBM && !PTE_RDONLY) to the software PTE_DIRTY bit.
1350
1350
*/
1351
1351
static inline void __ptep_set_wrprotect(struct mm_struct *mm,
+2
arch/arm64/include/asm/rsi.h
+2
arch/arm64/include/asm/rsi.h
-1
arch/arm64/include/asm/seccomp.h
-1
arch/arm64/include/asm/seccomp.h
···
23
23
#define SECCOMP_ARCH_NATIVE_NR NR_syscalls
24
24
#define SECCOMP_ARCH_NATIVE_NAME "aarch64"
25
25
#ifdef CONFIG_COMPAT
26
-
#include <asm/unistd_compat_32.h>
27
26
# define SECCOMP_ARCH_COMPAT AUDIT_ARCH_ARM
28
27
# define SECCOMP_ARCH_COMPAT_NR __NR_compat32_syscalls
29
28
# define SECCOMP_ARCH_COMPAT_NAME "arm"
+4
-1
arch/arm64/include/asm/sparsemem.h
+4
-1
arch/arm64/include/asm/sparsemem.h
···
5
5
#ifndef __ASM_SPARSEMEM_H
6
6
#define __ASM_SPARSEMEM_H
7
7
8
-
#define MAX_PHYSMEM_BITS CONFIG_ARM64_PA_BITS
8
+
#include <asm/pgtable-prot.h>
9
+
10
+
#define MAX_PHYSMEM_BITS PHYS_MASK_SHIFT
11
+
#define MAX_POSSIBLE_PHYSMEM_BITS (52)
9
12
10
13
/*
11
14
* Section size must be at least 512MB for 64K base
+15
arch/arm64/include/uapi/asm/hwcap.h
+15
arch/arm64/include/uapi/asm/hwcap.h
···
56
56
#define HWCAP_PACA (1 << 30)
57
57
#define HWCAP_PACG (1UL << 31)
58
58
#define HWCAP_GCS (1UL << 32)
59
+
#define HWCAP_CMPBR (1UL << 33)
60
+
#define HWCAP_FPRCVT (1UL << 34)
61
+
#define HWCAP_F8MM8 (1UL << 35)
62
+
#define HWCAP_F8MM4 (1UL << 36)
63
+
#define HWCAP_SVE_F16MM (1UL << 37)
64
+
#define HWCAP_SVE_ELTPERM (1UL << 38)
65
+
#define HWCAP_SVE_AES2 (1UL << 39)
66
+
#define HWCAP_SVE_BFSCALE (1UL << 40)
67
+
#define HWCAP_SVE2P2 (1UL << 41)
68
+
#define HWCAP_SME2P2 (1UL << 42)
69
+
#define HWCAP_SME_SBITPERM (1UL << 43)
70
+
#define HWCAP_SME_AES (1UL << 44)
71
+
#define HWCAP_SME_SFEXPA (1UL << 45)
72
+
#define HWCAP_SME_STMOP (1UL << 46)
73
+
#define HWCAP_SME_SMOP4 (1UL << 47)
59
74
60
75
/*
61
76
* HWCAP2 flags - for AT_HWCAP2
+67
-36
arch/arm64/kernel/cpufeature.c
+67
-36
arch/arm64/kernel/cpufeature.c
···
268
268
};
269
269
270
270
static const struct arm64_ftr_bits ftr_id_aa64isar3[] = {
271
+
ARM64_FTR_BITS(FTR_VISIBLE, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64ISAR3_EL1_FPRCVT_SHIFT, 4, 0),
271
272
ARM64_FTR_BITS(FTR_VISIBLE, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64ISAR3_EL1_FAMINMAX_SHIFT, 4, 0),
272
273
ARM64_FTR_END,
273
274
};
···
319
318
ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_SVE),
320
319
FTR_STRICT, FTR_LOWER_SAFE, ID_AA64ZFR0_EL1_F32MM_SHIFT, 4, 0),
321
320
ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_SVE),
321
+
FTR_STRICT, FTR_LOWER_SAFE, ID_AA64ZFR0_EL1_F16MM_SHIFT, 4, 0),
322
+
ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_SVE),
322
323
FTR_STRICT, FTR_LOWER_SAFE, ID_AA64ZFR0_EL1_I8MM_SHIFT, 4, 0),
323
324
ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_SVE),
324
325
FTR_STRICT, FTR_LOWER_SAFE, ID_AA64ZFR0_EL1_SM4_SHIFT, 4, 0),
···
332
329
FTR_STRICT, FTR_LOWER_SAFE, ID_AA64ZFR0_EL1_BF16_SHIFT, 4, 0),
333
330
ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_SVE),
334
331
FTR_STRICT, FTR_LOWER_SAFE, ID_AA64ZFR0_EL1_BitPerm_SHIFT, 4, 0),
332
+
ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_SVE),
333
+
FTR_STRICT, FTR_LOWER_SAFE, ID_AA64ZFR0_EL1_EltPerm_SHIFT, 4, 0),
335
334
ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_SVE),
336
335
FTR_STRICT, FTR_LOWER_SAFE, ID_AA64ZFR0_EL1_AES_SHIFT, 4, 0),
337
336
ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_SVE),
···
378
373
FTR_STRICT, FTR_EXACT, ID_AA64SMFR0_EL1_SF8DP4_SHIFT, 1, 0),
379
374
ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_SME),
380
375
FTR_STRICT, FTR_EXACT, ID_AA64SMFR0_EL1_SF8DP2_SHIFT, 1, 0),
376
+
ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_SME),
377
+
FTR_STRICT, FTR_EXACT, ID_AA64SMFR0_EL1_SBitPerm_SHIFT, 1, 0),
378
+
ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_SME),
379
+
FTR_STRICT, FTR_EXACT, ID_AA64SMFR0_EL1_AES_SHIFT, 1, 0),
380
+
ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_SME),
381
+
FTR_STRICT, FTR_EXACT, ID_AA64SMFR0_EL1_SFEXPA_SHIFT, 1, 0),
382
+
ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_SME),
383
+
FTR_STRICT, FTR_EXACT, ID_AA64SMFR0_EL1_STMOP_SHIFT, 1, 0),
384
+
ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_SME),
385
+
FTR_STRICT, FTR_EXACT, ID_AA64SMFR0_EL1_SMOP4_SHIFT, 1, 0),
381
386
ARM64_FTR_END,
382
387
};
383
388
···
396
381
ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_EXACT, ID_AA64FPFR0_EL1_F8FMA_SHIFT, 1, 0),
397
382
ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_EXACT, ID_AA64FPFR0_EL1_F8DP4_SHIFT, 1, 0),
398
383
ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_EXACT, ID_AA64FPFR0_EL1_F8DP2_SHIFT, 1, 0),
384
+
ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_EXACT, ID_AA64FPFR0_EL1_F8MM8_SHIFT, 1, 0),
385
+
ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_EXACT, ID_AA64FPFR0_EL1_F8MM4_SHIFT, 1, 0),
399
386
ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_EXACT, ID_AA64FPFR0_EL1_F8E4M3_SHIFT, 1, 0),
400
387
ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_EXACT, ID_AA64FPFR0_EL1_F8E5M2_SHIFT, 1, 0),
401
388
ARM64_FTR_END,
···
1021
1004
/* Override was valid */
1022
1005
ftr_new = tmp;
1023
1006
str = "forced";
1024
-
} else if (ftr_ovr == tmp) {
1007
+
} else {
1025
1008
/* Override was the safe value */
1026
1009
str = "already set";
1027
1010
}
1028
1011
1029
-
if (str)
1030
-
pr_warn("%s[%d:%d]: %s to %llx\n",
1031
-
reg->name,
1032
-
ftrp->shift + ftrp->width - 1,
1033
-
ftrp->shift, str,
1034
-
tmp & (BIT(ftrp->width) - 1));
1012
+
pr_warn("%s[%d:%d]: %s to %llx\n",
1013
+
reg->name,
1014
+
ftrp->shift + ftrp->width - 1,
1015
+
ftrp->shift, str,
1016
+
tmp & (BIT(ftrp->width) - 1));
1035
1017
} else if ((ftr_mask & reg->override->val) == ftr_mask) {
1036
1018
reg->override->val &= ~ftr_mask;
1037
1019
pr_warn("%s[%d:%d]: impossible override, ignored\n",
···
1183
1167
id_aa64pfr1_sme(read_sanitised_ftr_reg(SYS_ID_AA64PFR1_EL1))) {
1184
1168
unsigned long cpacr = cpacr_save_enable_kernel_sme();
1185
1169
1186
-
/*
1187
-
* We mask out SMPS since even if the hardware
1188
-
* supports priorities the kernel does not at present
1189
-
* and we block access to them.
1190
-
*/
1191
-
info->reg_smidr = read_cpuid(SMIDR_EL1) & ~SMIDR_EL1_SMPS;
1192
1170
vec_init_vq_map(ARM64_VEC_SME);
1193
1171
1194
1172
cpacr_restore(cpacr);
···
1432
1422
if (IS_ENABLED(CONFIG_ARM64_SME) &&
1433
1423
id_aa64pfr1_sme(read_sanitised_ftr_reg(SYS_ID_AA64PFR1_EL1))) {
1434
1424
unsigned long cpacr = cpacr_save_enable_kernel_sme();
1435
-
1436
-
/*
1437
-
* We mask out SMPS since even if the hardware
1438
-
* supports priorities the kernel does not at present
1439
-
* and we block access to them.
1440
-
*/
1441
-
info->reg_smidr = read_cpuid(SMIDR_EL1) & ~SMIDR_EL1_SMPS;
1442
1425
1443
1426
/* Probe vector lengths */
1444
1427
if (!system_capabilities_finalized())
···
2379
2376
#ifdef CONFIG_ARM64_POE
2380
2377
static void cpu_enable_poe(const struct arm64_cpu_capabilities *__unused)
2381
2378
{
2382
-
sysreg_clear_set(REG_TCR2_EL1, 0, TCR2_EL1x_E0POE);
2383
-
sysreg_clear_set(CPACR_EL1, 0, CPACR_ELx_E0POE);
2379
+
sysreg_clear_set(REG_TCR2_EL1, 0, TCR2_EL1_E0POE);
2380
+
sysreg_clear_set(CPACR_EL1, 0, CPACR_EL1_E0POE);
2384
2381
}
2385
2382
#endif
2386
2383
···
3025
3022
.matches = match, \
3026
3023
}
3027
3024
3025
+
#define HWCAP_CAP_MATCH_ID(match, reg, field, min_value, cap_type, cap) \
3026
+
{ \
3027
+
__HWCAP_CAP(#cap, cap_type, cap) \
3028
+
HWCAP_CPUID_MATCH(reg, field, min_value) \
3029
+
.matches = match, \
3030
+
}
3031
+
3028
3032
#ifdef CONFIG_ARM64_PTR_AUTH
3029
3033
static const struct arm64_cpu_capabilities ptr_auth_hwcap_addr_matches[] = {
3030
3034
{
···
3058
3048
},
3059
3049
{},
3060
3050
};
3051
+
#endif
3052
+
3053
+
#ifdef CONFIG_ARM64_SVE
3054
+
static bool has_sve_feature(const struct arm64_cpu_capabilities *cap, int scope)
3055
+
{
3056
+
return system_supports_sve() && has_user_cpuid_feature(cap, scope);
3057
+
}
3061
3058
#endif
3062
3059
3063
3060
static const struct arm64_cpu_capabilities arm64_elf_hwcaps[] = {
···
3109
3092
HWCAP_CAP(ID_AA64MMFR2_EL1, AT, IMP, CAP_HWCAP, KERNEL_HWCAP_USCAT),
3110
3093
#ifdef CONFIG_ARM64_SVE
3111
3094
HWCAP_CAP(ID_AA64PFR0_EL1, SVE, IMP, CAP_HWCAP, KERNEL_HWCAP_SVE),
3112
-
HWCAP_CAP(ID_AA64ZFR0_EL1, SVEver, SVE2p1, CAP_HWCAP, KERNEL_HWCAP_SVE2P1),
3113
-
HWCAP_CAP(ID_AA64ZFR0_EL1, SVEver, SVE2, CAP_HWCAP, KERNEL_HWCAP_SVE2),
3114
-
HWCAP_CAP(ID_AA64ZFR0_EL1, AES, IMP, CAP_HWCAP, KERNEL_HWCAP_SVEAES),
3115
-
HWCAP_CAP(ID_AA64ZFR0_EL1, AES, PMULL128, CAP_HWCAP, KERNEL_HWCAP_SVEPMULL),
3116
-
HWCAP_CAP(ID_AA64ZFR0_EL1, BitPerm, IMP, CAP_HWCAP, KERNEL_HWCAP_SVEBITPERM),
3117
-
HWCAP_CAP(ID_AA64ZFR0_EL1, B16B16, IMP, CAP_HWCAP, KERNEL_HWCAP_SVE_B16B16),
3118
-
HWCAP_CAP(ID_AA64ZFR0_EL1, BF16, IMP, CAP_HWCAP, KERNEL_HWCAP_SVEBF16),
3119
-
HWCAP_CAP(ID_AA64ZFR0_EL1, BF16, EBF16, CAP_HWCAP, KERNEL_HWCAP_SVE_EBF16),
3120
-
HWCAP_CAP(ID_AA64ZFR0_EL1, SHA3, IMP, CAP_HWCAP, KERNEL_HWCAP_SVESHA3),
3121
-
HWCAP_CAP(ID_AA64ZFR0_EL1, SM4, IMP, CAP_HWCAP, KERNEL_HWCAP_SVESM4),
3122
-
HWCAP_CAP(ID_AA64ZFR0_EL1, I8MM, IMP, CAP_HWCAP, KERNEL_HWCAP_SVEI8MM),
3123
-
HWCAP_CAP(ID_AA64ZFR0_EL1, F32MM, IMP, CAP_HWCAP, KERNEL_HWCAP_SVEF32MM),
3124
-
HWCAP_CAP(ID_AA64ZFR0_EL1, F64MM, IMP, CAP_HWCAP, KERNEL_HWCAP_SVEF64MM),
3095
+
HWCAP_CAP_MATCH_ID(has_sve_feature, ID_AA64ZFR0_EL1, SVEver, SVE2p2, CAP_HWCAP, KERNEL_HWCAP_SVE2P2),
3096
+
HWCAP_CAP_MATCH_ID(has_sve_feature, ID_AA64ZFR0_EL1, SVEver, SVE2p1, CAP_HWCAP, KERNEL_HWCAP_SVE2P1),
3097
+
HWCAP_CAP_MATCH_ID(has_sve_feature, ID_AA64ZFR0_EL1, SVEver, SVE2, CAP_HWCAP, KERNEL_HWCAP_SVE2),
3098
+
HWCAP_CAP_MATCH_ID(has_sve_feature, ID_AA64ZFR0_EL1, AES, IMP, CAP_HWCAP, KERNEL_HWCAP_SVEAES),
3099
+
HWCAP_CAP_MATCH_ID(has_sve_feature, ID_AA64ZFR0_EL1, AES, PMULL128, CAP_HWCAP, KERNEL_HWCAP_SVEPMULL),
3100
+
HWCAP_CAP_MATCH_ID(has_sve_feature, ID_AA64ZFR0_EL1, AES, AES2, CAP_HWCAP, KERNEL_HWCAP_SVE_AES2),
3101
+
HWCAP_CAP_MATCH_ID(has_sve_feature, ID_AA64ZFR0_EL1, BitPerm, IMP, CAP_HWCAP, KERNEL_HWCAP_SVEBITPERM),
3102
+
HWCAP_CAP_MATCH_ID(has_sve_feature, ID_AA64ZFR0_EL1, B16B16, IMP, CAP_HWCAP, KERNEL_HWCAP_SVE_B16B16),
3103
+
HWCAP_CAP_MATCH_ID(has_sve_feature, ID_AA64ZFR0_EL1, B16B16, BFSCALE, CAP_HWCAP, KERNEL_HWCAP_SVE_BFSCALE),
3104
+
HWCAP_CAP_MATCH_ID(has_sve_feature, ID_AA64ZFR0_EL1, BF16, IMP, CAP_HWCAP, KERNEL_HWCAP_SVEBF16),
3105
+
HWCAP_CAP_MATCH_ID(has_sve_feature, ID_AA64ZFR0_EL1, BF16, EBF16, CAP_HWCAP, KERNEL_HWCAP_SVE_EBF16),
3106
+
HWCAP_CAP_MATCH_ID(has_sve_feature, ID_AA64ZFR0_EL1, SHA3, IMP, CAP_HWCAP, KERNEL_HWCAP_SVESHA3),
3107
+
HWCAP_CAP_MATCH_ID(has_sve_feature, ID_AA64ZFR0_EL1, SM4, IMP, CAP_HWCAP, KERNEL_HWCAP_SVESM4),
3108
+
HWCAP_CAP_MATCH_ID(has_sve_feature, ID_AA64ZFR0_EL1, I8MM, IMP, CAP_HWCAP, KERNEL_HWCAP_SVEI8MM),
3109
+
HWCAP_CAP_MATCH_ID(has_sve_feature, ID_AA64ZFR0_EL1, F32MM, IMP, CAP_HWCAP, KERNEL_HWCAP_SVEF32MM),
3110
+
HWCAP_CAP_MATCH_ID(has_sve_feature, ID_AA64ZFR0_EL1, F64MM, IMP, CAP_HWCAP, KERNEL_HWCAP_SVEF64MM),
3111
+
HWCAP_CAP_MATCH_ID(has_sve_feature, ID_AA64ZFR0_EL1, F16MM, IMP, CAP_HWCAP, KERNEL_HWCAP_SVE_F16MM),
3112
+
HWCAP_CAP_MATCH_ID(has_sve_feature, ID_AA64ZFR0_EL1, EltPerm, IMP, CAP_HWCAP, KERNEL_HWCAP_SVE_ELTPERM),
3125
3113
#endif
3126
3114
#ifdef CONFIG_ARM64_GCS
3127
3115
HWCAP_CAP(ID_AA64PFR1_EL1, GCS, IMP, CAP_HWCAP, KERNEL_HWCAP_GCS),
···
3146
3124
HWCAP_CAP(ID_AA64MMFR0_EL1, ECV, IMP, CAP_HWCAP, KERNEL_HWCAP_ECV),
3147
3125
HWCAP_CAP(ID_AA64MMFR1_EL1, AFP, IMP, CAP_HWCAP, KERNEL_HWCAP_AFP),
3148
3126
HWCAP_CAP(ID_AA64ISAR2_EL1, CSSC, IMP, CAP_HWCAP, KERNEL_HWCAP_CSSC),
3127
+
HWCAP_CAP(ID_AA64ISAR2_EL1, CSSC, CMPBR, CAP_HWCAP, KERNEL_HWCAP_CMPBR),
3149
3128
HWCAP_CAP(ID_AA64ISAR2_EL1, RPRFM, IMP, CAP_HWCAP, KERNEL_HWCAP_RPRFM),
3150
3129
HWCAP_CAP(ID_AA64ISAR2_EL1, RPRES, IMP, CAP_HWCAP, KERNEL_HWCAP_RPRES),
3151
3130
HWCAP_CAP(ID_AA64ISAR2_EL1, WFxT, IMP, CAP_HWCAP, KERNEL_HWCAP_WFXT),
···
3156
3133
HWCAP_CAP(ID_AA64PFR1_EL1, SME, IMP, CAP_HWCAP, KERNEL_HWCAP_SME),
3157
3134
HWCAP_CAP(ID_AA64SMFR0_EL1, FA64, IMP, CAP_HWCAP, KERNEL_HWCAP_SME_FA64),
3158
3135
HWCAP_CAP(ID_AA64SMFR0_EL1, LUTv2, IMP, CAP_HWCAP, KERNEL_HWCAP_SME_LUTV2),
3136
+
HWCAP_CAP(ID_AA64SMFR0_EL1, SMEver, SME2p2, CAP_HWCAP, KERNEL_HWCAP_SME2P2),
3159
3137
HWCAP_CAP(ID_AA64SMFR0_EL1, SMEver, SME2p1, CAP_HWCAP, KERNEL_HWCAP_SME2P1),
3160
3138
HWCAP_CAP(ID_AA64SMFR0_EL1, SMEver, SME2, CAP_HWCAP, KERNEL_HWCAP_SME2),
3161
3139
HWCAP_CAP(ID_AA64SMFR0_EL1, I16I64, IMP, CAP_HWCAP, KERNEL_HWCAP_SME_I16I64),
···
3174
3150
HWCAP_CAP(ID_AA64SMFR0_EL1, SF8FMA, IMP, CAP_HWCAP, KERNEL_HWCAP_SME_SF8FMA),
3175
3151
HWCAP_CAP(ID_AA64SMFR0_EL1, SF8DP4, IMP, CAP_HWCAP, KERNEL_HWCAP_SME_SF8DP4),
3176
3152
HWCAP_CAP(ID_AA64SMFR0_EL1, SF8DP2, IMP, CAP_HWCAP, KERNEL_HWCAP_SME_SF8DP2),
3153
+
HWCAP_CAP(ID_AA64SMFR0_EL1, SF8MM8, IMP, CAP_HWCAP, KERNEL_HWCAP_SME_SF8MM8),
3154
+
HWCAP_CAP(ID_AA64SMFR0_EL1, SF8MM4, IMP, CAP_HWCAP, KERNEL_HWCAP_SME_SF8MM4),
3155
+
HWCAP_CAP(ID_AA64SMFR0_EL1, SBitPerm, IMP, CAP_HWCAP, KERNEL_HWCAP_SME_SBITPERM),
3156
+
HWCAP_CAP(ID_AA64SMFR0_EL1, AES, IMP, CAP_HWCAP, KERNEL_HWCAP_SME_AES),
3157
+
HWCAP_CAP(ID_AA64SMFR0_EL1, SFEXPA, IMP, CAP_HWCAP, KERNEL_HWCAP_SME_SFEXPA),
3158
+
HWCAP_CAP(ID_AA64SMFR0_EL1, STMOP, IMP, CAP_HWCAP, KERNEL_HWCAP_SME_STMOP),
3159
+
HWCAP_CAP(ID_AA64SMFR0_EL1, SMOP4, IMP, CAP_HWCAP, KERNEL_HWCAP_SME_SMOP4),
3177
3160
#endif /* CONFIG_ARM64_SME */
3178
3161
HWCAP_CAP(ID_AA64FPFR0_EL1, F8CVT, IMP, CAP_HWCAP, KERNEL_HWCAP_F8CVT),
3179
3162
HWCAP_CAP(ID_AA64FPFR0_EL1, F8FMA, IMP, CAP_HWCAP, KERNEL_HWCAP_F8FMA),
···
3509
3478
return;
3510
3479
3511
3480
safe_mmfr1 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR1_EL1);
3512
-
mmfr0 = read_cpuid(ID_AA64MMFR0_EL1);
3481
+
mmfr0 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1);
3513
3482
mmfr1 = read_cpuid(ID_AA64MMFR1_EL1);
3514
3483
3515
3484
/* Verify VMID bits */
+25
arch/arm64/kernel/cpuinfo.c
+25
arch/arm64/kernel/cpuinfo.c
···
145
145
[KERNEL_HWCAP_SME_SF8DP4] = "smesf8dp4",
146
146
[KERNEL_HWCAP_SME_SF8DP2] = "smesf8dp2",
147
147
[KERNEL_HWCAP_POE] = "poe",
148
+
[KERNEL_HWCAP_CMPBR] = "cmpbr",
149
+
[KERNEL_HWCAP_FPRCVT] = "fprcvt",
150
+
[KERNEL_HWCAP_F8MM8] = "f8mm8",
151
+
[KERNEL_HWCAP_F8MM4] = "f8mm4",
152
+
[KERNEL_HWCAP_SVE_F16MM] = "svef16mm",
153
+
[KERNEL_HWCAP_SVE_ELTPERM] = "sveeltperm",
154
+
[KERNEL_HWCAP_SVE_AES2] = "sveaes2",
155
+
[KERNEL_HWCAP_SVE_BFSCALE] = "svebfscale",
156
+
[KERNEL_HWCAP_SVE2P2] = "sve2p2",
157
+
[KERNEL_HWCAP_SME2P2] = "sme2p2",
158
+
[KERNEL_HWCAP_SME_SBITPERM] = "smesbitperm",
159
+
[KERNEL_HWCAP_SME_AES] = "smeaes",
160
+
[KERNEL_HWCAP_SME_SFEXPA] = "smesfexpa",
161
+
[KERNEL_HWCAP_SME_STMOP] = "smestmop",
162
+
[KERNEL_HWCAP_SME_SMOP4] = "smesmop4",
148
163
};
149
164
150
165
#ifdef CONFIG_COMPAT
···
496
481
497
482
if (id_aa64pfr0_mpam(info->reg_id_aa64pfr0))
498
483
info->reg_mpamidr = read_cpuid(MPAMIDR_EL1);
484
+
485
+
if (IS_ENABLED(CONFIG_ARM64_SME) &&
486
+
id_aa64pfr1_sme(info->reg_id_aa64pfr1)) {
487
+
/*
488
+
* We mask out SMPS since even if the hardware
489
+
* supports priorities the kernel does not at present
490
+
* and we block access to them.
491
+
*/
492
+
info->reg_smidr = read_cpuid(SMIDR_EL1) & ~SMIDR_EL1_SMPS;
493
+
}
499
494
500
495
cpuinfo_detect_icache_policy(info);
501
496
}
+9
arch/arm64/kernel/pi/idreg-override.c
+9
arch/arm64/kernel/pi/idreg-override.c
···
83
83
id_aa64mmfr0_override.val |=
84
84
(ID_AA64MMFR0_EL1_TGRAN_LPA2 - 1) << ID_AA64MMFR0_EL1_TGRAN_SHIFT;
85
85
id_aa64mmfr0_override.mask |= 0xfU << ID_AA64MMFR0_EL1_TGRAN_SHIFT;
86
+
87
+
/*
88
+
* Override PARange to 48 bits - the override will just be
89
+
* ignored if the actual PARange is smaller, but this is
90
+
* unlikely to be the case for LPA2 capable silicon.
91
+
*/
92
+
id_aa64mmfr0_override.val |=
93
+
ID_AA64MMFR0_EL1_PARANGE_48 << ID_AA64MMFR0_EL1_PARANGE_SHIFT;
94
+
id_aa64mmfr0_override.mask |= 0xfU << ID_AA64MMFR0_EL1_PARANGE_SHIFT;
86
95
}
87
96
#endif
88
97
return true;
+6
arch/arm64/kernel/pi/map_kernel.c
+6
arch/arm64/kernel/pi/map_kernel.c
···
136
136
{
137
137
u64 sctlr = read_sysreg(sctlr_el1);
138
138
u64 tcr = read_sysreg(tcr_el1) | TCR_DS;
139
+
u64 mmfr0 = read_sysreg(id_aa64mmfr0_el1);
140
+
u64 parange = cpuid_feature_extract_unsigned_field(mmfr0,
141
+
ID_AA64MMFR0_EL1_PARANGE_SHIFT);
142
+
143
+
tcr &= ~TCR_IPS_MASK;
144
+
tcr |= parange << TCR_IPS_SHIFT;
139
145
140
146
asm(" msr sctlr_el1, %0 ;"
141
147
" isb ;"
+15
arch/arm64/kernel/rsi.c
+15
arch/arm64/kernel/rsi.c
···
8
8
#include <linux/psci.h>
9
9
#include <linux/swiotlb.h>
10
10
#include <linux/cc_platform.h>
11
+
#include <linux/platform_device.h>
11
12
12
13
#include <asm/io.h>
13
14
#include <asm/mem_encrypt.h>
···
141
140
static_branch_enable(&rsi_present);
142
141
}
143
142
143
+
static struct platform_device rsi_dev = {
144
+
.name = RSI_PDEV_NAME,
145
+
.id = PLATFORM_DEVID_NONE
146
+
};
147
+
148
+
static int __init arm64_create_dummy_rsi_dev(void)
149
+
{
150
+
if (is_realm_world() &&
151
+
platform_device_register(&rsi_dev))
152
+
pr_err("failed to register rsi platform device\n");
153
+
return 0;
154
+
}
155
+
156
+
arch_initcall(arm64_create_dummy_rsi_dev)
+4
-4
arch/arm64/kvm/arm.c
+4
-4
arch/arm64/kvm/arm.c
···
1990
1990
static void __init cpu_prepare_hyp_mode(int cpu, u32 hyp_va_bits)
1991
1991
{
1992
1992
struct kvm_nvhe_init_params *params = per_cpu_ptr_nvhe_sym(kvm_init_params, cpu);
1993
-
u64 mmfr0 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1);
1994
-
unsigned long tcr;
1993
+
unsigned long tcr, ips;
1995
1994
1996
1995
/*
1997
1996
* Calculate the raw per-cpu offset without a translation from the
···
2004
2005
params->mair_el2 = read_sysreg(mair_el1);
2005
2006
2006
2007
tcr = read_sysreg(tcr_el1);
2008
+
ips = FIELD_GET(TCR_IPS_MASK, tcr);
2007
2009
if (cpus_have_final_cap(ARM64_KVM_HVHE)) {
2008
2010
tcr |= TCR_EPD1_MASK;
2009
2011
} else {
···
2014
2014
tcr &= ~TCR_T0SZ_MASK;
2015
2015
tcr |= TCR_T0SZ(hyp_va_bits);
2016
2016
tcr &= ~TCR_EL2_PS_MASK;
2017
-
tcr |= FIELD_PREP(TCR_EL2_PS_MASK, kvm_get_parange(mmfr0));
2018
-
if (kvm_lpa2_is_enabled())
2017
+
tcr |= FIELD_PREP(TCR_EL2_PS_MASK, ips);
2018
+
if (lpa2_is_enabled())
2019
2019
tcr |= TCR_EL2_DS;
2020
2020
params->tcr_el2 = tcr;
2021
2021
+3
-3
arch/arm64/kvm/at.c
+3
-3
arch/arm64/kvm/at.c
···
111
111
return vcpu_read_sys_reg(vcpu, TCR2_EL2) & TCR2_EL2_PIE;
112
112
case TR_EL10:
113
113
return (__vcpu_sys_reg(vcpu, HCRX_EL2) & HCRX_EL2_TCR2En) &&
114
-
(__vcpu_sys_reg(vcpu, TCR2_EL1) & TCR2_EL1x_PIE);
114
+
(__vcpu_sys_reg(vcpu, TCR2_EL1) & TCR2_EL1_PIE);
115
115
default:
116
116
BUG();
117
117
}
···
140
140
}
141
141
142
142
val = __vcpu_sys_reg(vcpu, TCR2_EL1);
143
-
wi->poe = val & TCR2_EL1x_POE;
144
-
wi->e0poe = val & TCR2_EL1x_E0POE;
143
+
wi->poe = val & TCR2_EL1_POE;
144
+
wi->e0poe = val & TCR2_EL1_E0POE;
145
145
}
146
146
}
147
147
+1
-1
arch/arm64/kvm/emulate-nested.c
+1
-1
arch/arm64/kvm/emulate-nested.c
+1
-1
arch/arm64/kvm/fpsimd.c
+1
-1
arch/arm64/kvm/fpsimd.c
···
169
169
if (has_vhe() && system_supports_sme()) {
170
170
/* Also restore EL0 state seen on entry */
171
171
if (vcpu_get_flag(vcpu, HOST_SME_ENABLED))
172
-
sysreg_clear_set(CPACR_EL1, 0, CPACR_ELx_SMEN);
172
+
sysreg_clear_set(CPACR_EL1, 0, CPACR_EL1_SMEN);
173
173
else
174
174
sysreg_clear_set(CPACR_EL1,
175
175
CPACR_EL1_SMEN_EL0EN,
+2
-2
arch/arm64/kvm/hyp/include/hyp/switch.h
+2
-2
arch/arm64/kvm/hyp/include/hyp/switch.h
···
419
419
420
420
/* First disable enough traps to allow us to update the registers */
421
421
if (sve_guest || (is_protected_kvm_enabled() && system_supports_sve()))
422
-
cpacr_clear_set(0, CPACR_ELx_FPEN | CPACR_ELx_ZEN);
422
+
cpacr_clear_set(0, CPACR_EL1_FPEN | CPACR_EL1_ZEN);
423
423
else
424
-
cpacr_clear_set(0, CPACR_ELx_FPEN);
424
+
cpacr_clear_set(0, CPACR_EL1_FPEN);
425
425
isb();
426
426
427
427
/* Write out the host state if it's in the registers */
+2
-2
arch/arm64/kvm/hyp/nvhe/hyp-main.c
+2
-2
arch/arm64/kvm/hyp/nvhe/hyp-main.c
···
68
68
if (!guest_owns_fp_regs())
69
69
return;
70
70
71
-
cpacr_clear_set(0, CPACR_ELx_FPEN | CPACR_ELx_ZEN);
71
+
cpacr_clear_set(0, CPACR_EL1_FPEN | CPACR_EL1_ZEN);
72
72
isb();
73
73
74
74
if (vcpu_has_sve(vcpu))
···
481
481
handle_host_smc(host_ctxt);
482
482
break;
483
483
case ESR_ELx_EC_SVE:
484
-
cpacr_clear_set(0, CPACR_ELx_ZEN);
484
+
cpacr_clear_set(0, CPACR_EL1_ZEN);
485
485
isb();
486
486
sve_cond_update_zcr_vq(sve_vq_from_vl(kvm_host_sve_max_vl) - 1,
487
487
SYS_ZCR_EL2);
+1
-1
arch/arm64/kvm/hyp/nvhe/pkvm.c
+1
-1
arch/arm64/kvm/hyp/nvhe/pkvm.c
+3
-3
arch/arm64/kvm/hyp/nvhe/switch.c
+3
-3
arch/arm64/kvm/hyp/nvhe/switch.c
···
48
48
val |= has_hvhe() ? CPACR_EL1_TTA : CPTR_EL2_TTA;
49
49
if (cpus_have_final_cap(ARM64_SME)) {
50
50
if (has_hvhe())
51
-
val &= ~CPACR_ELx_SMEN;
51
+
val &= ~CPACR_EL1_SMEN;
52
52
else
53
53
val |= CPTR_EL2_TSM;
54
54
}
55
55
56
56
if (!guest_owns_fp_regs()) {
57
57
if (has_hvhe())
58
-
val &= ~(CPACR_ELx_FPEN | CPACR_ELx_ZEN);
58
+
val &= ~(CPACR_EL1_FPEN | CPACR_EL1_ZEN);
59
59
else
60
60
val |= CPTR_EL2_TFP | CPTR_EL2_TZ;
61
61
···
192
192
193
193
/* Re-enable SVE traps if not supported for the guest vcpu. */
194
194
if (!vcpu_has_sve(vcpu))
195
-
cpacr_clear_set(CPACR_ELx_ZEN, 0);
195
+
cpacr_clear_set(CPACR_EL1_ZEN, 0);
196
196
197
197
} else {
198
198
__fpsimd_save_state(*host_data_ptr(fpsimd_state));
+10
-23
arch/arm64/kvm/hyp/pgtable.c
+10
-23
arch/arm64/kvm/hyp/pgtable.c
···
35
35
return unlikely(ctx->flags & KVM_PGTABLE_WALK_SKIP_CMO);
36
36
}
37
37
38
-
static bool kvm_phys_is_valid(u64 phys)
39
-
{
40
-
u64 parange_max = kvm_get_parange_max();
41
-
u8 shift = id_aa64mmfr0_parange_to_phys_shift(parange_max);
42
-
43
-
return phys < BIT(shift);
44
-
}
45
-
46
38
static bool kvm_block_mapping_supported(const struct kvm_pgtable_visit_ctx *ctx, u64 phys)
47
39
{
48
40
u64 granule = kvm_granule_size(ctx->level);
···
45
53
if (granule > (ctx->end - ctx->addr))
46
54
return false;
47
55
48
-
if (kvm_phys_is_valid(phys) && !IS_ALIGNED(phys, granule))
56
+
if (!IS_ALIGNED(phys, granule))
49
57
return false;
50
58
51
59
return IS_ALIGNED(ctx->addr, granule);
···
579
587
580
588
/* Force mappings to page granularity */
581
589
bool force_pte;
590
+
591
+
/* Walk should update owner_id only */
592
+
bool annotation;
582
593
};
583
594
584
595
u64 kvm_get_vtcr(u64 mmfr0, u64 mmfr1, u32 phys_shift)
···
880
885
{
881
886
u64 phys = data->phys;
882
887
883
-
/*
884
-
* Stage-2 walks to update ownership data are communicated to the map
885
-
* walker using an invalid PA. Avoid offsetting an already invalid PA,
886
-
* which could overflow and make the address valid again.
887
-
*/
888
-
if (!kvm_phys_is_valid(phys))
889
-
return phys;
890
-
891
-
/*
892
-
* Otherwise, work out the correct PA based on how far the walk has
893
-
* gotten.
894
-
*/
888
+
/* Work out the correct PA based on how far the walk has gotten */
895
889
return phys + (ctx->addr - ctx->start);
896
890
}
897
891
···
891
907
892
908
if (data->force_pte && ctx->level < KVM_PGTABLE_LAST_LEVEL)
893
909
return false;
910
+
911
+
if (data->annotation)
912
+
return true;
894
913
895
914
return kvm_block_mapping_supported(ctx, phys);
896
915
}
···
910
923
if (!stage2_leaf_mapping_allowed(ctx, data))
911
924
return -E2BIG;
912
925
913
-
if (kvm_phys_is_valid(phys))
926
+
if (!data->annotation)
914
927
new = kvm_init_valid_leaf_pte(phys, data->attr, ctx->level);
915
928
else
916
929
new = kvm_init_invalid_leaf_owner(data->owner_id);
···
1072
1085
{
1073
1086
int ret;
1074
1087
struct stage2_map_data map_data = {
1075
-
.phys = KVM_PHYS_INVALID,
1076
1088
.mmu = pgt->mmu,
1077
1089
.memcache = mc,
1078
1090
.owner_id = owner_id,
1079
1091
.force_pte = true,
1092
+
.annotation = true,
1080
1093
};
1081
1094
struct kvm_pgtable_walker walker = {
1082
1095
.cb = stage2_map_walker,
+8
-8
arch/arm64/kvm/hyp/vhe/switch.c
+8
-8
arch/arm64/kvm/hyp/vhe/switch.c
···
77
77
* VHE (HCR.E2H == 1) which allows us to use here the CPTR_EL2.TAM
78
78
* shift value for trapping the AMU accesses.
79
79
*/
80
-
u64 val = CPACR_ELx_TTA | CPTR_EL2_TAM;
80
+
u64 val = CPACR_EL1_TTA | CPTR_EL2_TAM;
81
81
82
82
if (guest_owns_fp_regs()) {
83
-
val |= CPACR_ELx_FPEN;
83
+
val |= CPACR_EL1_FPEN;
84
84
if (vcpu_has_sve(vcpu))
85
-
val |= CPACR_ELx_ZEN;
85
+
val |= CPACR_EL1_ZEN;
86
86
} else {
87
87
__activate_traps_fpsimd32(vcpu);
88
88
}
···
122
122
* hypervisor has traps enabled to dispel any illusion of something more
123
123
* complicated taking place.
124
124
*/
125
-
if (!(SYS_FIELD_GET(CPACR_ELx, FPEN, cptr) & BIT(0)))
126
-
val &= ~CPACR_ELx_FPEN;
127
-
if (!(SYS_FIELD_GET(CPACR_ELx, ZEN, cptr) & BIT(0)))
128
-
val &= ~CPACR_ELx_ZEN;
125
+
if (!(SYS_FIELD_GET(CPACR_EL1, FPEN, cptr) & BIT(0)))
126
+
val &= ~CPACR_EL1_FPEN;
127
+
if (!(SYS_FIELD_GET(CPACR_EL1, ZEN, cptr) & BIT(0)))
128
+
val &= ~CPACR_EL1_ZEN;
129
129
130
130
if (kvm_has_feat(vcpu->kvm, ID_AA64MMFR3_EL1, S2POE, IMP))
131
-
val |= cptr & CPACR_ELx_E0POE;
131
+
val |= cptr & CPACR_EL1_E0POE;
132
132
133
133
val |= cptr & CPTR_EL2_TCPAC;
134
134
+5
-1
arch/arm64/kvm/sys_regs.c
+5
-1
arch/arm64/kvm/sys_regs.c
···
1602
1602
if (!cpus_have_final_cap(ARM64_HAS_WFXT))
1603
1603
val &= ~ARM64_FEATURE_MASK(ID_AA64ISAR2_EL1_WFxT);
1604
1604
break;
1605
+
case SYS_ID_AA64ISAR3_EL1:
1606
+
val &= ID_AA64ISAR3_EL1_FPRCVT | ID_AA64ISAR3_EL1_FAMINMAX;
1607
+
break;
1605
1608
case SYS_ID_AA64MMFR2_EL1:
1606
1609
val &= ~ID_AA64MMFR2_EL1_CCIDX_MASK;
1607
1610
break;
···
2629
2626
ID_WRITABLE(ID_AA64ISAR2_EL1, ~(ID_AA64ISAR2_EL1_RES0 |
2630
2627
ID_AA64ISAR2_EL1_APA3 |
2631
2628
ID_AA64ISAR2_EL1_GPA3)),
2632
-
ID_UNALLOCATED(6,3),
2629
+
ID_WRITABLE(ID_AA64ISAR3_EL1, (ID_AA64ISAR3_EL1_FPRCVT |
2630
+
ID_AA64ISAR3_EL1_FAMINMAX)),
2633
2631
ID_UNALLOCATED(6,4),
2634
2632
ID_UNALLOCATED(6,5),
2635
2633
ID_UNALLOCATED(6,6),
+12
arch/arm64/mm/hugetlbpage.c
+12
arch/arm64/mm/hugetlbpage.c
···
519
519
520
520
static int __init hugetlbpage_init(void)
521
521
{
522
+
/*
523
+
* HugeTLB pages are supported on maximum four page table
524
+
* levels (PUD, CONT PMD, PMD, CONT PTE) for a given base
525
+
* page size, corresponding to hugetlb_add_hstate() calls
526
+
* here.
527
+
*
528
+
* HUGE_MAX_HSTATE should at least match maximum supported
529
+
* HugeTLB page sizes on the platform. Any new addition to
530
+
* supported HugeTLB page sizes will also require changing
531
+
* HUGE_MAX_HSTATE as well.
532
+
*/
533
+
BUILD_BUG_ON(HUGE_MAX_HSTATE < 4);
522
534
if (pud_sect_supported())
523
535
hugetlb_add_hstate(PUD_SHIFT - PAGE_SHIFT);
524
536
+6
-1
arch/arm64/mm/init.c
+6
-1
arch/arm64/mm/init.c
···
279
279
280
280
if (IS_ENABLED(CONFIG_RANDOMIZE_BASE)) {
281
281
extern u16 memstart_offset_seed;
282
-
u64 mmfr0 = read_cpuid(ID_AA64MMFR0_EL1);
282
+
283
+
/*
284
+
* Use the sanitised version of id_aa64mmfr0_el1 so that linear
285
+
* map randomization can be enabled by shrinking the IPA space.
286
+
*/
287
+
u64 mmfr0 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1);
283
288
int parange = cpuid_feature_extract_unsigned_field(
284
289
mmfr0, ID_AA64MMFR0_EL1_PARANGE_SHIFT);
285
290
s64 range = linear_region_size -
+2
-1
arch/arm64/mm/mmu.c
+2
-1
arch/arm64/mm/mmu.c
···
1169
1169
unsigned long addr, unsigned long next)
1170
1170
{
1171
1171
vmemmap_verify((pte_t *)pmdp, node, addr, next);
1172
-
return 1;
1172
+
1173
+
return pmd_sect(READ_ONCE(*pmdp));
1173
1174
}
1174
1175
1175
1176
int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,
+3
-2
arch/arm64/mm/proc.S
+3
-2
arch/arm64/mm/proc.S
···
501
501
#ifdef CONFIG_ARM64_HAFT
502
502
cmp x9, ID_AA64MMFR1_EL1_HAFDBS_HAFT
503
503
b.lt 1f
504
-
orr tcr2, tcr2, TCR2_EL1x_HAFT
504
+
orr tcr2, tcr2, TCR2_EL1_HAFT
505
505
#endif /* CONFIG_ARM64_HAFT */
506
506
1:
507
507
#endif /* CONFIG_ARM64_HW_AFDBM */
···
532
532
#undef PTE_MAYBE_NG
533
533
#undef PTE_MAYBE_SHARED
534
534
535
-
orr tcr2, tcr2, TCR2_EL1x_PIE
535
+
orr tcr2, tcr2, TCR2_EL1_PIE
536
+
msr REG_TCR2_EL1, x0
536
537
537
538
.Lskip_indirection:
538
539
+1
-1
arch/arm64/mm/trans_pgd.c
+1
-1
arch/arm64/mm/trans_pgd.c
+1
-1
arch/arm64/tools/gen-sysreg.awk
+1
-1
arch/arm64/tools/gen-sysreg.awk
···
206
206
207
207
# Currently this is effectivey a comment, in future we may want to emit
208
208
# defines for the fields.
209
-
/^Fields/ && block_current() == "Sysreg" {
209
+
(/^Fields/ || /^Mapping/) && block_current() == "Sysreg" {
210
210
expect_fields(2)
211
211
212
212
if (next_bit != 63)
+95
-30
arch/arm64/tools/sysreg
+95
-30
arch/arm64/tools/sysreg
···
24
24
# ...
25
25
# EndEnum
26
26
27
-
# Alternatively if multiple registers share the same layout then
28
-
# a SysregFields block can be used to describe the shared layout
27
+
# For VHE aliases (*_EL12, *_EL02) of system registers, a Mapping
28
+
# entry describes the register the alias actually accesses:
29
+
30
+
# Sysreg <name_EL12> <op0> <op1> <crn> <crm> <op2>
31
+
# Mapping <name_EL1>
32
+
# EndSysreg
33
+
34
+
# Where multiple system regsiters are not VHE aliases but share a
35
+
# common layout, a SysregFields block can be used to describe the
36
+
# shared layout:
29
37
30
38
# SysregFields <fieldsname>
31
39
# <field>
···
1018
1010
0b0000 NI
1019
1011
0b0001 IMP
1020
1012
EndEnum
1021
-
Res0 31:12
1013
+
Res0 31:20
1014
+
UnsignedEnum 19:16 UINJ
1015
+
0b0000 NI
1016
+
0b0001 IMP
1017
+
EndEnum
1018
+
Res0 15:12
1022
1019
UnsignedEnum 11:8 MTEFAR
1023
1020
0b0000 NI
1024
1021
0b0001 IMP
···
1048
1035
0b0000 NI
1049
1036
0b0001 IMP
1050
1037
EndEnum
1051
-
Res0 51:48
1038
+
UnsignedEnum 51:48 F16MM
1039
+
0b0000 NI
1040
+
0b0001 IMP
1041
+
EndEnum
1052
1042
UnsignedEnum 47:44 I8MM
1053
1043
0b0000 NI
1054
1044
0b0001 IMP
···
1069
1053
UnsignedEnum 27:24 B16B16
1070
1054
0b0000 NI
1071
1055
0b0001 IMP
1056
+
0b0010 BFSCALE
1072
1057
EndEnum
1073
1058
UnsignedEnum 23:20 BF16
1074
1059
0b0000 NI
···
1080
1063
0b0000 NI
1081
1064
0b0001 IMP
1082
1065
EndEnum
1083
-
Res0 15:8
1066
+
UnsignedEnum 15:12 EltPerm
1067
+
0b0000 NI
1068
+
0b0001 IMP
1069
+
EndEnum
1070
+
Res0 11:8
1084
1071
UnsignedEnum 7:4 AES
1085
1072
0b0000 NI
1086
1073
0b0001 IMP
1087
1074
0b0010 PMULL128
1075
+
0b0011 AES2
1088
1076
EndEnum
1089
1077
UnsignedEnum 3:0 SVEver
1090
1078
0b0000 IMP
1091
1079
0b0001 SVE2
1092
1080
0b0010 SVE2p1
1081
+
0b0011 SVE2p2
1093
1082
EndEnum
1094
1083
EndSysreg
1095
1084
···
1113
1090
0b0000 SME
1114
1091
0b0001 SME2
1115
1092
0b0010 SME2p1
1116
-
0b0000 IMP
1093
+
0b0011 SME2p2
1117
1094
EndEnum
1118
1095
UnsignedEnum 55:52 I16I64
1119
1096
0b0000 NI
···
1177
1154
0b0 NI
1178
1155
0b1 IMP
1179
1156
EndEnum
1180
-
Res0 27:0
1157
+
Res0 27:26
1158
+
UnsignedEnum 25 SBitPerm
1159
+
0b0 NI
1160
+
0b1 IMP
1161
+
EndEnum
1162
+
UnsignedEnum 24 AES
1163
+
0b0 NI
1164
+
0b1 IMP
1165
+
EndEnum
1166
+
UnsignedEnum 23 SFEXPA
1167
+
0b0 NI
1168
+
0b1 IMP
1169
+
EndEnum
1170
+
Res0 22:17
1171
+
UnsignedEnum 16 STMOP
1172
+
0b0 NI
1173
+
0b1 IMP
1174
+
EndEnum
1175
+
Res0 15:1
1176
+
UnsignedEnum 0 SMOP4
1177
+
0b0 NI
1178
+
0b1 IMP
1179
+
EndEnum
1181
1180
EndSysreg
1182
1181
1183
1182
Sysreg ID_AA64FPFR0_EL1 3 0 0 4 7
···
1220
1175
0b0 NI
1221
1176
0b1 IMP
1222
1177
EndEnum
1223
-
Res0 27:2
1178
+
UnsignedEnum 27 F8MM8
1179
+
0b0 NI
1180
+
0b1 IMP
1181
+
EndEnum
1182
+
UnsignedEnum 26 F8MM4
1183
+
0b0 NI
1184
+
0b1 IMP
1185
+
EndEnum
1186
+
Res0 25:2
1224
1187
UnsignedEnum 1 F8E4M3
1225
1188
0b0 NI
1226
1189
0b1 IMP
···
1556
1503
UnsignedEnum 55:52 CSSC
1557
1504
0b0000 NI
1558
1505
0b0001 IMP
1506
+
0b0010 CMPBR
1559
1507
EndEnum
1560
1508
UnsignedEnum 51:48 RPRFM
1561
1509
0b0000 NI
1562
1510
0b0001 IMP
1563
1511
EndEnum
1564
-
Res0 47:44
1512
+
UnsignedEnum 47:44 PCDPHINT
1513
+
0b0000 NI
1514
+
0b0001 IMP
1515
+
EndEnum
1565
1516
UnsignedEnum 43:40 PRFMSLC
1566
1517
0b0000 NI
1567
1518
0b0001 IMP
···
1618
1561
EndSysreg
1619
1562
1620
1563
Sysreg ID_AA64ISAR3_EL1 3 0 0 6 3
1621
-
Res0 63:16
1564
+
Res0 63:32
1565
+
UnsignedEnum 31:28 FPRCVT
1566
+
0b0000 NI
1567
+
0b0001 IMP
1568
+
EndEnum
1569
+
UnsignedEnum 27:24 LSUI
1570
+
0b0000 NI
1571
+
0b0001 IMP
1572
+
EndEnum
1573
+
UnsignedEnum 23:20 OCCMO
1574
+
0b0000 NI
1575
+
0b0001 IMP
1576
+
EndEnum
1577
+
UnsignedEnum 19:16 LSFE
1578
+
0b0000 NI
1579
+
0b0001 IMP
1580
+
EndEnum
1622
1581
UnsignedEnum 15:12 PACM
1623
1582
0b0000 NI
1624
1583
0b0001 TRIVIAL_IMP
···
2051
1978
Field 0 M
2052
1979
EndSysreg
2053
1980
2054
-
SysregFields CPACR_ELx
1981
+
Sysreg CPACR_EL1 3 0 1 0 2
2055
1982
Res0 63:30
2056
1983
Field 29 E0POE
2057
1984
Field 28 TTA
···
2062
1989
Res0 19:18
2063
1990
Field 17:16 ZEN
2064
1991
Res0 15:0
2065
-
EndSysregFields
2066
-
2067
-
Sysreg CPACR_EL1 3 0 1 0 2
2068
-
Fields CPACR_ELx
2069
1992
EndSysreg
2070
1993
2071
1994
Sysreg SMPRI_EL1 3 0 1 2 4
···
3016
2947
EndSysreg
3017
2948
3018
2949
Sysreg CPACR_EL12 3 5 1 0 2
3019
-
Fields CPACR_ELx
2950
+
Mapping CPACR_EL1
3020
2951
EndSysreg
3021
2952
3022
2953
Sysreg ZCR_EL12 3 5 1 2 0
3023
-
Fields ZCR_ELx
2954
+
Mapping ZCR_EL1
3024
2955
EndSysreg
3025
2956
3026
2957
Sysreg SMCR_EL12 3 5 1 2 6
3027
-
Fields SMCR_ELx
2958
+
Mapping SMCR_EL1
3028
2959
EndSysreg
3029
2960
3030
2961
Sysreg GCSCR_EL12 3 5 2 5 0
3031
-
Fields GCSCR_ELx
2962
+
Mapping GCSCR_EL1
3032
2963
EndSysreg
3033
2964
3034
2965
Sysreg GCSPR_EL12 3 5 2 5 1
3035
-
Fields GCSPR_ELx
2966
+
Mapping GCSPR_EL1
3036
2967
EndSysreg
3037
2968
3038
2969
Sysreg FAR_EL12 3 5 6 0 0
···
3044
2975
EndSysreg
3045
2976
3046
2977
Sysreg CONTEXTIDR_EL12 3 5 13 0 1
3047
-
Fields CONTEXTIDR_ELx
2978
+
Mapping CONTEXTIDR_EL1
3048
2979
EndSysreg
3049
2980
3050
2981
SysregFields TTBRx_EL1
···
3061
2992
Fields TTBRx_EL1
3062
2993
EndSysreg
3063
2994
3064
-
SysregFields TCR2_EL1x
2995
+
Sysreg TCR2_EL1 3 0 2 0 3
3065
2996
Res0 63:16
3066
2997
Field 15 DisCH1
3067
2998
Field 14 DisCH0
···
3075
3006
Field 2 E0POE
3076
3007
Field 1 PIE
3077
3008
Field 0 PnCH
3078
-
EndSysregFields
3079
-
3080
-
Sysreg TCR2_EL1 3 0 2 0 3
3081
-
Fields TCR2_EL1x
3082
3009
EndSysreg
3083
3010
3084
3011
Sysreg TCR2_EL12 3 5 2 0 3
3085
-
Fields TCR2_EL1x
3012
+
Mapping TCR2_EL1
3086
3013
EndSysreg
3087
3014
3088
3015
Sysreg TCR2_EL2 3 4 2 0 3
···
3149
3084
EndSysreg
3150
3085
3151
3086
Sysreg PIRE0_EL12 3 5 10 2 2
3152
-
Fields PIRx_ELx
3087
+
Mapping PIRE0_EL1
3153
3088
EndSysreg
3154
3089
3155
3090
Sysreg PIRE0_EL2 3 4 10 2 2
···
3161
3096
EndSysreg
3162
3097
3163
3098
Sysreg PIR_EL12 3 5 10 2 3
3164
-
Fields PIRx_ELx
3099
+
Mapping PIR_EL1
3165
3100
EndSysreg
3166
3101
3167
3102
Sysreg PIR_EL2 3 4 10 2 3
···
3181
3116
EndSysreg
3182
3117
3183
3118
Sysreg POR_EL12 3 5 10 2 4
3184
-
Fields PIRx_ELx
3119
+
Mapping POR_EL1
3185
3120
EndSysreg
3186
3121
3187
3122
Sysreg S2POR_EL1 3 0 10 2 5
+2
drivers/perf/apple_m1_cpu_pmu.c
+2
drivers/perf/apple_m1_cpu_pmu.c
···
168
168
PERF_MAP_ALL_UNSUPPORTED,
169
169
[PERF_COUNT_HW_CPU_CYCLES] = M1_PMU_PERFCTR_CORE_ACTIVE_CYCLE,
170
170
[PERF_COUNT_HW_INSTRUCTIONS] = M1_PMU_PERFCTR_INST_ALL,
171
+
[PERF_COUNT_HW_BRANCH_INSTRUCTIONS] = M1_PMU_PERFCTR_INST_BRANCH,
172
+
[PERF_COUNT_HW_BRANCH_MISSES] = M1_PMU_PERFCTR_BRANCH_MISPRED_NONSPEC,
171
173
};
172
174
173
175
/* sysfs definitions */
+2
-2
drivers/perf/arm-cmn.c
+2
-2
drivers/perf/arm-cmn.c
···
1713
1713
goto done;
1714
1714
}
1715
1715
1716
-
for (i = 0; i < CMN_MAX_DTCS; i++)
1717
-
if (val->dtc_count[i] == CMN_DT_NUM_COUNTERS)
1716
+
for_each_hw_dtc_idx(hw, dtc, idx)
1717
+
if (val->dtc_count[dtc] == CMN_DT_NUM_COUNTERS)
1718
1718
goto done;
1719
1719
1720
1720
for_each_hw_dn(hw, dn, i) {
+7
-68
drivers/perf/arm_cspmu/nvidia_cspmu.c
+7
-68
drivers/perf/arm_cspmu/nvidia_cspmu.c
···
54
54
ARM_CSPMU_EVENT_ATTR(scf_cache_wb, 0xF3),
55
55
56
56
NV_CSPMU_EVENT_ATTR_4(socket, rd_data, 0x101),
57
-
NV_CSPMU_EVENT_ATTR_4(socket, dl_rsp, 0x105),
58
57
NV_CSPMU_EVENT_ATTR_4(socket, wb_data, 0x109),
59
-
NV_CSPMU_EVENT_ATTR_4(socket, ev_rsp, 0x10d),
60
-
NV_CSPMU_EVENT_ATTR_4(socket, prb_data, 0x111),
61
58
62
59
NV_CSPMU_EVENT_ATTR_4(socket, rd_outstanding, 0x115),
63
-
NV_CSPMU_EVENT_ATTR_4(socket, dl_outstanding, 0x119),
64
-
NV_CSPMU_EVENT_ATTR_4(socket, wb_outstanding, 0x11d),
65
-
NV_CSPMU_EVENT_ATTR_4(socket, wr_outstanding, 0x121),
66
-
NV_CSPMU_EVENT_ATTR_4(socket, ev_outstanding, 0x125),
67
-
NV_CSPMU_EVENT_ATTR_4(socket, prb_outstanding, 0x129),
68
60
69
61
NV_CSPMU_EVENT_ATTR_4(socket, rd_access, 0x12d),
70
-
NV_CSPMU_EVENT_ATTR_4(socket, dl_access, 0x131),
71
62
NV_CSPMU_EVENT_ATTR_4(socket, wb_access, 0x135),
72
63
NV_CSPMU_EVENT_ATTR_4(socket, wr_access, 0x139),
73
-
NV_CSPMU_EVENT_ATTR_4(socket, ev_access, 0x13d),
74
-
NV_CSPMU_EVENT_ATTR_4(socket, prb_access, 0x141),
75
-
76
-
NV_CSPMU_EVENT_ATTR_4(ocu, gmem_rd_data, 0x145),
77
-
NV_CSPMU_EVENT_ATTR_4(ocu, gmem_rd_access, 0x149),
78
-
NV_CSPMU_EVENT_ATTR_4(ocu, gmem_wb_access, 0x14d),
79
-
NV_CSPMU_EVENT_ATTR_4(ocu, gmem_rd_outstanding, 0x151),
80
-
NV_CSPMU_EVENT_ATTR_4(ocu, gmem_wr_outstanding, 0x155),
81
-
82
-
NV_CSPMU_EVENT_ATTR_4(ocu, rem_rd_data, 0x159),
83
-
NV_CSPMU_EVENT_ATTR_4(ocu, rem_rd_access, 0x15d),
84
-
NV_CSPMU_EVENT_ATTR_4(ocu, rem_wb_access, 0x161),
85
-
NV_CSPMU_EVENT_ATTR_4(ocu, rem_rd_outstanding, 0x165),
86
-
NV_CSPMU_EVENT_ATTR_4(ocu, rem_wr_outstanding, 0x169),
87
64
88
65
ARM_CSPMU_EVENT_ATTR(gmem_rd_data, 0x16d),
89
66
ARM_CSPMU_EVENT_ATTR(gmem_rd_access, 0x16e),
90
67
ARM_CSPMU_EVENT_ATTR(gmem_rd_outstanding, 0x16f),
91
-
ARM_CSPMU_EVENT_ATTR(gmem_dl_rsp, 0x170),
92
-
ARM_CSPMU_EVENT_ATTR(gmem_dl_access, 0x171),
93
-
ARM_CSPMU_EVENT_ATTR(gmem_dl_outstanding, 0x172),
94
68
ARM_CSPMU_EVENT_ATTR(gmem_wb_data, 0x173),
95
69
ARM_CSPMU_EVENT_ATTR(gmem_wb_access, 0x174),
96
-
ARM_CSPMU_EVENT_ATTR(gmem_wb_outstanding, 0x175),
97
-
ARM_CSPMU_EVENT_ATTR(gmem_ev_rsp, 0x176),
98
-
ARM_CSPMU_EVENT_ATTR(gmem_ev_access, 0x177),
99
-
ARM_CSPMU_EVENT_ATTR(gmem_ev_outstanding, 0x178),
100
70
ARM_CSPMU_EVENT_ATTR(gmem_wr_data, 0x179),
101
-
ARM_CSPMU_EVENT_ATTR(gmem_wr_outstanding, 0x17a),
102
71
ARM_CSPMU_EVENT_ATTR(gmem_wr_access, 0x17b),
103
72
104
73
NV_CSPMU_EVENT_ATTR_4(socket, wr_data, 0x17c),
105
-
106
-
NV_CSPMU_EVENT_ATTR_4(ocu, gmem_wr_data, 0x180),
107
-
NV_CSPMU_EVENT_ATTR_4(ocu, gmem_wb_data, 0x184),
108
-
NV_CSPMU_EVENT_ATTR_4(ocu, gmem_wr_access, 0x188),
109
-
NV_CSPMU_EVENT_ATTR_4(ocu, gmem_wb_outstanding, 0x18c),
110
-
111
-
NV_CSPMU_EVENT_ATTR_4(ocu, rem_wr_data, 0x190),
112
-
NV_CSPMU_EVENT_ATTR_4(ocu, rem_wb_data, 0x194),
113
-
NV_CSPMU_EVENT_ATTR_4(ocu, rem_wr_access, 0x198),
114
-
NV_CSPMU_EVENT_ATTR_4(ocu, rem_wb_outstanding, 0x19c),
115
74
116
75
ARM_CSPMU_EVENT_ATTR(gmem_wr_total_bytes, 0x1a0),
117
76
ARM_CSPMU_EVENT_ATTR(remote_socket_wr_total_bytes, 0x1a1),
···
81
122
ARM_CSPMU_EVENT_ATTR(cmem_rd_data, 0x1a5),
82
123
ARM_CSPMU_EVENT_ATTR(cmem_rd_access, 0x1a6),
83
124
ARM_CSPMU_EVENT_ATTR(cmem_rd_outstanding, 0x1a7),
84
-
ARM_CSPMU_EVENT_ATTR(cmem_dl_rsp, 0x1a8),
85
-
ARM_CSPMU_EVENT_ATTR(cmem_dl_access, 0x1a9),
86
-
ARM_CSPMU_EVENT_ATTR(cmem_dl_outstanding, 0x1aa),
87
125
ARM_CSPMU_EVENT_ATTR(cmem_wb_data, 0x1ab),
88
126
ARM_CSPMU_EVENT_ATTR(cmem_wb_access, 0x1ac),
89
-
ARM_CSPMU_EVENT_ATTR(cmem_wb_outstanding, 0x1ad),
90
-
ARM_CSPMU_EVENT_ATTR(cmem_ev_rsp, 0x1ae),
91
-
ARM_CSPMU_EVENT_ATTR(cmem_ev_access, 0x1af),
92
-
ARM_CSPMU_EVENT_ATTR(cmem_ev_outstanding, 0x1b0),
93
127
ARM_CSPMU_EVENT_ATTR(cmem_wr_data, 0x1b1),
94
-
ARM_CSPMU_EVENT_ATTR(cmem_wr_outstanding, 0x1b2),
95
-
96
-
NV_CSPMU_EVENT_ATTR_4(ocu, cmem_rd_data, 0x1b3),
97
-
NV_CSPMU_EVENT_ATTR_4(ocu, cmem_rd_access, 0x1b7),
98
-
NV_CSPMU_EVENT_ATTR_4(ocu, cmem_wb_access, 0x1bb),
99
-
NV_CSPMU_EVENT_ATTR_4(ocu, cmem_rd_outstanding, 0x1bf),
100
-
NV_CSPMU_EVENT_ATTR_4(ocu, cmem_wr_outstanding, 0x1c3),
101
-
102
-
ARM_CSPMU_EVENT_ATTR(ocu_prb_access, 0x1c7),
103
-
ARM_CSPMU_EVENT_ATTR(ocu_prb_data, 0x1c8),
104
-
ARM_CSPMU_EVENT_ATTR(ocu_prb_outstanding, 0x1c9),
105
128
106
129
ARM_CSPMU_EVENT_ATTR(cmem_wr_access, 0x1ca),
107
-
108
-
NV_CSPMU_EVENT_ATTR_4(ocu, cmem_wr_access, 0x1cb),
109
-
NV_CSPMU_EVENT_ATTR_4(ocu, cmem_wb_data, 0x1cf),
110
-
NV_CSPMU_EVENT_ATTR_4(ocu, cmem_wr_data, 0x1d3),
111
-
NV_CSPMU_EVENT_ATTR_4(ocu, cmem_wb_outstanding, 0x1d7),
112
130
113
131
ARM_CSPMU_EVENT_ATTR(cmem_wr_total_bytes, 0x1db),
114
132
···
130
194
131
195
static struct attribute *nvlink_c2c_pmu_format_attrs[] = {
132
196
ARM_CSPMU_FORMAT_EVENT_ATTR,
197
+
ARM_CSPMU_FORMAT_ATTR(port, "config1:0-1"),
133
198
NULL,
134
199
};
135
200
···
175
238
const struct nv_cspmu_ctx *ctx =
176
239
to_nv_cspmu_ctx(to_arm_cspmu(event->pmu));
177
240
178
-
if (ctx->filter_mask == 0)
241
+
const u32 filter_val = event->attr.config1 & ctx->filter_mask;
242
+
243
+
if (filter_val == 0)
179
244
return ctx->filter_default_val;
180
245
181
-
return event->attr.config1 & ctx->filter_mask;
246
+
return filter_val;
182
247
}
183
248
184
249
enum nv_cspmu_name_fmt {
···
213
274
{
214
275
.prodid = 0x104,
215
276
.prodid_mask = NV_PRODID_MASK,
216
-
.filter_mask = 0x0,
277
+
.filter_mask = NV_NVL_C2C_FILTER_ID_MASK,
217
278
.filter_default_val = NV_NVL_C2C_FILTER_ID_MASK,
218
279
.name_pattern = "nvidia_nvlink_c2c1_pmu_%u",
219
280
.name_fmt = NAME_FMT_SOCKET,
···
223
284
{
224
285
.prodid = 0x105,
225
286
.prodid_mask = NV_PRODID_MASK,
226
-
.filter_mask = 0x0,
287
+
.filter_mask = NV_NVL_C2C_FILTER_ID_MASK,
227
288
.filter_default_val = NV_NVL_C2C_FILTER_ID_MASK,
228
289
.name_pattern = "nvidia_nvlink_c2c0_pmu_%u",
229
290
.name_fmt = NAME_FMT_SOCKET,
+22
drivers/perf/arm_spe_pmu.c
+22
drivers/perf/arm_spe_pmu.c
···
85
85
#define SPE_PMU_FEAT_LDS (1UL << 4)
86
86
#define SPE_PMU_FEAT_ERND (1UL << 5)
87
87
#define SPE_PMU_FEAT_INV_FILT_EVT (1UL << 6)
88
+
#define SPE_PMU_FEAT_DISCARD (1UL << 7)
88
89
#define SPE_PMU_FEAT_DEV_PROBED (1UL << 63)
89
90
u64 features;
90
91
···
194
193
#define ATTR_CFG_FLD_store_filter_CFG config /* PMSFCR_EL1.ST */
195
194
#define ATTR_CFG_FLD_store_filter_LO 34
196
195
#define ATTR_CFG_FLD_store_filter_HI 34
196
+
#define ATTR_CFG_FLD_discard_CFG config /* PMBLIMITR_EL1.FM = DISCARD */
197
+
#define ATTR_CFG_FLD_discard_LO 35
198
+
#define ATTR_CFG_FLD_discard_HI 35
197
199
198
200
#define ATTR_CFG_FLD_event_filter_CFG config1 /* PMSEVFR_EL1 */
199
201
#define ATTR_CFG_FLD_event_filter_LO 0
···
220
216
GEN_PMU_FORMAT_ATTR(event_filter);
221
217
GEN_PMU_FORMAT_ATTR(inv_event_filter);
222
218
GEN_PMU_FORMAT_ATTR(min_latency);
219
+
GEN_PMU_FORMAT_ATTR(discard);
223
220
224
221
static struct attribute *arm_spe_pmu_formats_attr[] = {
225
222
&format_attr_ts_enable.attr,
···
233
228
&format_attr_event_filter.attr,
234
229
&format_attr_inv_event_filter.attr,
235
230
&format_attr_min_latency.attr,
231
+
&format_attr_discard.attr,
236
232
NULL,
237
233
};
238
234
···
243
237
{
244
238
struct device *dev = kobj_to_dev(kobj);
245
239
struct arm_spe_pmu *spe_pmu = dev_get_drvdata(dev);
240
+
241
+
if (attr == &format_attr_discard.attr && !(spe_pmu->features & SPE_PMU_FEAT_DISCARD))
242
+
return 0;
246
243
247
244
if (attr == &format_attr_inv_event_filter.attr && !(spe_pmu->features & SPE_PMU_FEAT_INV_FILT_EVT))
248
245
return 0;
···
511
502
u64 base, limit;
512
503
struct arm_spe_pmu_buf *buf;
513
504
505
+
if (ATTR_CFG_GET_FLD(&event->attr, discard)) {
506
+
limit = FIELD_PREP(PMBLIMITR_EL1_FM, PMBLIMITR_EL1_FM_DISCARD);
507
+
limit |= PMBLIMITR_EL1_E;
508
+
goto out_write_limit;
509
+
}
510
+
514
511
/* Start a new aux session */
515
512
buf = perf_aux_output_begin(handle, event);
516
513
if (!buf) {
···
756
741
757
742
if ((FIELD_GET(PMSFCR_EL1_FL, reg)) &&
758
743
!(spe_pmu->features & SPE_PMU_FEAT_FILT_LAT))
744
+
return -EOPNOTSUPP;
745
+
746
+
if (ATTR_CFG_GET_FLD(&event->attr, discard) &&
747
+
!(spe_pmu->features & SPE_PMU_FEAT_DISCARD))
759
748
return -EOPNOTSUPP;
760
749
761
750
set_spe_event_has_cx(event);
···
1045
1026
1046
1027
if (FIELD_GET(PMSIDR_EL1_ERND, reg))
1047
1028
spe_pmu->features |= SPE_PMU_FEAT_ERND;
1029
+
1030
+
if (spe_pmu->pmsver >= ID_AA64DFR0_EL1_PMSVer_V1P2)
1031
+
spe_pmu->features |= SPE_PMU_FEAT_DISCARD;
1048
1032
1049
1033
/* This field has a spaced out encoding, so just use a look-up */
1050
1034
fld = FIELD_GET(PMSIDR_EL1_INTERVAL, reg);
+39
-33
drivers/perf/dwc_pcie_pmu.c
+39
-33
drivers/perf/dwc_pcie_pmu.c
···
20
20
#include <linux/sysfs.h>
21
21
#include <linux/types.h>
22
22
23
-
#define DWC_PCIE_VSEC_RAS_DES_ID 0x02
24
23
#define DWC_PCIE_EVENT_CNT_CTL 0x8
25
24
26
25
/*
···
99
100
struct list_head dev_node;
100
101
};
101
102
102
-
struct dwc_pcie_vendor_id {
103
-
int vendor_id;
103
+
struct dwc_pcie_pmu_vsec_id {
104
+
u16 vendor_id;
105
+
u16 vsec_id;
106
+
u8 vsec_rev;
104
107
};
105
108
106
-
static const struct dwc_pcie_vendor_id dwc_pcie_vendor_ids[] = {
107
-
{.vendor_id = PCI_VENDOR_ID_ALIBABA },
108
-
{.vendor_id = PCI_VENDOR_ID_AMPERE },
109
-
{.vendor_id = PCI_VENDOR_ID_QCOM },
109
+
/*
110
+
* VSEC IDs are allocated by the vendor, so a given ID may mean different
111
+
* things to different vendors. See PCIe r6.0, sec 7.9.5.2.
112
+
*/
113
+
static const struct dwc_pcie_pmu_vsec_id dwc_pcie_pmu_vsec_ids[] = {
114
+
{ .vendor_id = PCI_VENDOR_ID_ALIBABA,
115
+
.vsec_id = 0x02, .vsec_rev = 0x4 },
116
+
{ .vendor_id = PCI_VENDOR_ID_AMPERE,
117
+
.vsec_id = 0x02, .vsec_rev = 0x4 },
118
+
{ .vendor_id = PCI_VENDOR_ID_QCOM,
119
+
.vsec_id = 0x02, .vsec_rev = 0x4 },
110
120
{} /* terminator */
111
121
};
112
122
···
207
199
DWC_PCIE_PMU_TIME_BASE_EVENT_ATTR(L1_1, 0x05),
208
200
DWC_PCIE_PMU_TIME_BASE_EVENT_ATTR(L1_2, 0x06),
209
201
DWC_PCIE_PMU_TIME_BASE_EVENT_ATTR(CFG_RCVRY, 0x07),
210
-
DWC_PCIE_PMU_TIME_BASE_EVENT_ATTR(TX_RX_L0S, 0x08),
211
-
DWC_PCIE_PMU_TIME_BASE_EVENT_ATTR(L1_AUX, 0x09),
202
+
DWC_PCIE_PMU_TIME_BASE_EVENT_ATTR(L1_AUX, 0x08),
203
+
DWC_PCIE_PMU_TIME_BASE_EVENT_ATTR(TX_RX_L0S, 0x09),
212
204
213
205
/* Group #1 */
214
206
DWC_PCIE_PMU_TIME_BASE_EVENT_ATTR(tx_pcie_tlp_data_payload, 0x20),
···
527
519
perf_pmu_unregister(&pcie_pmu->pmu);
528
520
}
529
521
530
-
static bool dwc_pcie_match_des_cap(struct pci_dev *pdev)
522
+
static u16 dwc_pcie_des_cap(struct pci_dev *pdev)
531
523
{
532
-
const struct dwc_pcie_vendor_id *vid;
533
-
u16 vsec = 0;
524
+
const struct dwc_pcie_pmu_vsec_id *vid;
525
+
u16 vsec;
534
526
u32 val;
535
527
536
528
if (!pci_is_pcie(pdev) || !(pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT))
537
-
return false;
529
+
return 0;
538
530
539
-
for (vid = dwc_pcie_vendor_ids; vid->vendor_id; vid++) {
531
+
for (vid = dwc_pcie_pmu_vsec_ids; vid->vendor_id; vid++) {
540
532
vsec = pci_find_vsec_capability(pdev, vid->vendor_id,
541
-
DWC_PCIE_VSEC_RAS_DES_ID);
542
-
if (vsec)
543
-
break;
533
+
vid->vsec_id);
534
+
if (vsec) {
535
+
pci_read_config_dword(pdev, vsec + PCI_VNDR_HEADER,
536
+
&val);
537
+
if (PCI_VNDR_HEADER_REV(val) == vid->vsec_rev) {
538
+
pci_dbg(pdev, "Detected PCIe Vendor-Specific Extended Capability RAS DES\n");
539
+
return vsec;
540
+
}
541
+
}
544
542
}
545
-
if (!vsec)
546
-
return false;
547
-
548
-
pci_read_config_dword(pdev, vsec + PCI_VNDR_HEADER, &val);
549
-
if (PCI_VNDR_HEADER_REV(val) != 0x04)
550
-
return false;
551
-
552
-
pci_dbg(pdev,
553
-
"Detected PCIe Vendor-Specific Extended Capability RAS DES\n");
554
-
return true;
543
+
return 0;
555
544
}
556
545
557
546
static void dwc_pcie_unregister_dev(struct dwc_pcie_dev_info *dev_info)
···
592
587
593
588
switch (action) {
594
589
case BUS_NOTIFY_ADD_DEVICE:
595
-
if (!dwc_pcie_match_des_cap(pdev))
590
+
if (!dwc_pcie_des_cap(pdev))
596
591
return NOTIFY_DONE;
597
592
if (dwc_pcie_register_dev(pdev))
598
593
return NOTIFY_BAD;
···
617
612
struct pci_dev *pdev = plat_dev->dev.platform_data;
618
613
struct dwc_pcie_pmu *pcie_pmu;
619
614
char *name;
620
-
u32 sbdf, val;
615
+
u32 sbdf;
621
616
u16 vsec;
622
617
int ret;
623
618
624
-
vsec = pci_find_vsec_capability(pdev, pdev->vendor,
625
-
DWC_PCIE_VSEC_RAS_DES_ID);
626
-
pci_read_config_dword(pdev, vsec + PCI_VNDR_HEADER, &val);
619
+
vsec = dwc_pcie_des_cap(pdev);
620
+
if (!vsec)
621
+
return -ENODEV;
622
+
627
623
sbdf = plat_dev->id;
628
624
name = devm_kasprintf(&plat_dev->dev, GFP_KERNEL, "dwc_rootport_%x", sbdf);
629
625
if (!name)
···
736
730
int ret;
737
731
738
732
for_each_pci_dev(pdev) {
739
-
if (!dwc_pcie_match_des_cap(pdev))
733
+
if (!dwc_pcie_des_cap(pdev))
740
734
continue;
741
735
742
736
ret = dwc_pcie_register_dev(pdev);
+25
-8
drivers/perf/fsl_imx9_ddr_perf.c
+25
-8
drivers/perf/fsl_imx9_ddr_perf.c
···
63
63
64
64
static DEFINE_IDA(ddr_ida);
65
65
66
+
/*
67
+
* V1 support 1 read transaction, 1 write transaction and 1 read beats
68
+
* event which corresponding respecitively to counter 2, 3 and 4.
69
+
*/
70
+
#define DDR_PERF_AXI_FILTER_V1 0x1
71
+
72
+
/*
73
+
* V2 support 1 read beats and 3 write beats events which corresponding
74
+
* respecitively to counter 2-5.
75
+
*/
76
+
#define DDR_PERF_AXI_FILTER_V2 0x2
77
+
66
78
struct imx_ddr_devtype_data {
67
79
const char *identifier; /* system PMU identifier for userspace */
80
+
unsigned int filter_ver; /* AXI filter version */
68
81
};
69
82
70
83
struct ddr_pmu {
···
96
83
97
84
static const struct imx_ddr_devtype_data imx91_devtype_data = {
98
85
.identifier = "imx91",
86
+
.filter_ver = DDR_PERF_AXI_FILTER_V1
99
87
};
100
88
101
89
static const struct imx_ddr_devtype_data imx93_devtype_data = {
102
90
.identifier = "imx93",
91
+
.filter_ver = DDR_PERF_AXI_FILTER_V1
103
92
};
104
93
105
94
static const struct imx_ddr_devtype_data imx95_devtype_data = {
106
95
.identifier = "imx95",
96
+
.filter_ver = DDR_PERF_AXI_FILTER_V2
107
97
};
108
98
109
-
static inline bool is_imx93(struct ddr_pmu *pmu)
99
+
static inline bool axi_filter_v1(struct ddr_pmu *pmu)
110
100
{
111
-
return pmu->devtype_data == &imx93_devtype_data;
101
+
return pmu->devtype_data->filter_ver == DDR_PERF_AXI_FILTER_V1;
112
102
}
113
103
114
-
static inline bool is_imx95(struct ddr_pmu *pmu)
104
+
static inline bool axi_filter_v2(struct ddr_pmu *pmu)
115
105
{
116
-
return pmu->devtype_data == &imx95_devtype_data;
106
+
return pmu->devtype_data->filter_ver == DDR_PERF_AXI_FILTER_V2;
117
107
}
118
108
119
109
static const struct of_device_id imx_ddr_pmu_dt_ids[] = {
···
171
155
struct imx9_pmu_events_attr {
172
156
struct device_attribute attr;
173
157
u64 id;
174
-
const void *devtype_data;
158
+
const struct imx_ddr_devtype_data *devtype_data;
175
159
};
176
160
177
161
static ssize_t ddr_pmu_event_show(struct device *dev,
···
323
307
if (!eattr->devtype_data)
324
308
return attr->mode;
325
309
326
-
if (eattr->devtype_data != ddr_pmu->devtype_data)
310
+
if (eattr->devtype_data != ddr_pmu->devtype_data &&
311
+
eattr->devtype_data->filter_ver != ddr_pmu->devtype_data->filter_ver)
327
312
return 0;
328
313
329
314
return attr->mode;
···
641
624
hwc->idx = counter;
642
625
hwc->state |= PERF_HES_STOPPED;
643
626
644
-
if (is_imx93(pmu))
627
+
if (axi_filter_v1(pmu))
645
628
/* read trans, write trans, read beat */
646
629
imx93_ddr_perf_monitor_config(pmu, event_id, counter, cfg1, cfg2);
647
630
648
-
if (is_imx95(pmu))
631
+
if (axi_filter_v2(pmu))
649
632
/* write beat, read beat2, read beat1, read beat */
650
633
imx95_ddr_perf_monitor_config(pmu, event_id, counter, cfg1, cfg2);
651
634
+9
-33
drivers/perf/hisilicon/hisi_uncore_cpa_pmu.c
+9
-33
drivers/perf/hisilicon/hisi_uncore_cpa_pmu.c
···
180
180
static int hisi_cpa_pmu_init_data(struct platform_device *pdev,
181
181
struct hisi_pmu *cpa_pmu)
182
182
{
183
-
if (device_property_read_u32(&pdev->dev, "hisilicon,scl-id",
184
-
&cpa_pmu->sicl_id)) {
183
+
hisi_uncore_pmu_init_topology(cpa_pmu, &pdev->dev);
184
+
185
+
if (cpa_pmu->topo.sicl_id < 0) {
185
186
dev_err(&pdev->dev, "Can not read sicl-id\n");
186
187
return -EINVAL;
187
188
}
188
189
189
-
if (device_property_read_u32(&pdev->dev, "hisilicon,idx-id",
190
-
&cpa_pmu->index_id)) {
190
+
if (cpa_pmu->topo.index_id < 0) {
191
191
dev_err(&pdev->dev, "Cannot read idx-id\n");
192
192
return -EINVAL;
193
193
}
194
194
195
-
cpa_pmu->ccl_id = -1;
196
-
cpa_pmu->sccl_id = -1;
197
195
cpa_pmu->base = devm_platform_ioremap_resource(pdev, 0);
198
196
if (IS_ERR(cpa_pmu->base))
199
197
return PTR_ERR(cpa_pmu->base);
···
225
227
.attrs = hisi_cpa_pmu_events_attr,
226
228
};
227
229
228
-
static DEVICE_ATTR(cpumask, 0444, hisi_cpumask_sysfs_show, NULL);
229
-
230
-
static struct attribute *hisi_cpa_pmu_cpumask_attrs[] = {
231
-
&dev_attr_cpumask.attr,
232
-
NULL
233
-
};
234
-
235
-
static const struct attribute_group hisi_cpa_pmu_cpumask_attr_group = {
236
-
.attrs = hisi_cpa_pmu_cpumask_attrs,
237
-
};
238
-
239
-
static struct device_attribute hisi_cpa_pmu_identifier_attr =
240
-
__ATTR(identifier, 0444, hisi_uncore_pmu_identifier_attr_show, NULL);
241
-
242
-
static struct attribute *hisi_cpa_pmu_identifier_attrs[] = {
243
-
&hisi_cpa_pmu_identifier_attr.attr,
244
-
NULL
245
-
};
246
-
247
-
static const struct attribute_group hisi_cpa_pmu_identifier_group = {
248
-
.attrs = hisi_cpa_pmu_identifier_attrs,
249
-
};
250
-
251
230
static const struct attribute_group *hisi_cpa_pmu_attr_groups[] = {
252
231
&hisi_cpa_pmu_format_group,
253
232
&hisi_cpa_pmu_events_group,
254
-
&hisi_cpa_pmu_cpumask_attr_group,
255
-
&hisi_cpa_pmu_identifier_group,
233
+
&hisi_pmu_cpumask_attr_group,
234
+
&hisi_pmu_identifier_group,
256
235
NULL
257
236
};
258
237
···
286
311
if (ret)
287
312
return ret;
288
313
289
-
name = devm_kasprintf(&pdev->dev, GFP_KERNEL, "hisi_sicl%d_cpa%u",
290
-
cpa_pmu->sicl_id, cpa_pmu->index_id);
314
+
name = devm_kasprintf(&pdev->dev, GFP_KERNEL, "hisi_sicl%d_cpa%d",
315
+
cpa_pmu->topo.sicl_id, cpa_pmu->topo.index_id);
291
316
if (!name)
292
317
return -ENOMEM;
293
318
···
364
389
}
365
390
module_exit(hisi_cpa_pmu_module_exit);
366
391
392
+
MODULE_IMPORT_NS("HISI_PMU");
367
393
MODULE_DESCRIPTION("HiSilicon SoC CPA PMU driver");
368
394
MODULE_LICENSE("GPL v2");
369
395
MODULE_AUTHOR("Qi Liu <liuqi115@huawei.com>");
+18
-43
drivers/perf/hisilicon/hisi_uncore_ddrc_pmu.c
+18
-43
drivers/perf/hisilicon/hisi_uncore_ddrc_pmu.c
···
111
111
* so there is no need to write event type, while it is programmable counter in
112
112
* PMU v2.
113
113
*/
114
-
static void hisi_ddrc_pmu_write_evtype(struct hisi_pmu *hha_pmu, int idx,
114
+
static void hisi_ddrc_pmu_write_evtype(struct hisi_pmu *ddrc_pmu, int idx,
115
115
u32 type)
116
116
{
117
117
u32 offset;
118
118
119
-
if (hha_pmu->identifier >= HISI_PMU_V2) {
119
+
if (ddrc_pmu->identifier >= HISI_PMU_V2) {
120
120
offset = DDRC_V2_EVENT_TYPE + 4 * idx;
121
-
writel(type, hha_pmu->base + offset);
121
+
writel(type, ddrc_pmu->base + offset);
122
122
}
123
123
}
124
124
···
297
297
static int hisi_ddrc_pmu_init_data(struct platform_device *pdev,
298
298
struct hisi_pmu *ddrc_pmu)
299
299
{
300
+
hisi_uncore_pmu_init_topology(ddrc_pmu, &pdev->dev);
301
+
300
302
/*
301
303
* Use the SCCL_ID and DDRC channel ID to identify the
302
304
* DDRC PMU, while SCCL_ID is in MPIDR[aff2].
303
305
*/
304
306
if (device_property_read_u32(&pdev->dev, "hisilicon,ch-id",
305
-
&ddrc_pmu->index_id)) {
307
+
&ddrc_pmu->topo.index_id)) {
306
308
dev_err(&pdev->dev, "Can not read ddrc channel-id!\n");
307
309
return -EINVAL;
308
310
}
309
311
310
-
if (device_property_read_u32(&pdev->dev, "hisilicon,scl-id",
311
-
&ddrc_pmu->sccl_id)) {
312
+
if (ddrc_pmu->topo.sccl_id < 0) {
312
313
dev_err(&pdev->dev, "Can not read ddrc sccl-id!\n");
313
314
return -EINVAL;
314
315
}
315
-
/* DDRC PMUs only share the same SCCL */
316
-
ddrc_pmu->ccl_id = -1;
317
316
318
317
ddrc_pmu->base = devm_platform_ioremap_resource(pdev, 0);
319
318
if (IS_ERR(ddrc_pmu->base)) {
···
322
323
323
324
ddrc_pmu->identifier = readl(ddrc_pmu->base + DDRC_VERSION);
324
325
if (ddrc_pmu->identifier >= HISI_PMU_V2) {
325
-
if (device_property_read_u32(&pdev->dev, "hisilicon,sub-id",
326
-
&ddrc_pmu->sub_id)) {
326
+
if (ddrc_pmu->topo.sub_id < 0) {
327
327
dev_err(&pdev->dev, "Can not read sub-id!\n");
328
328
return -EINVAL;
329
329
}
···
380
382
.attrs = hisi_ddrc_pmu_v2_events_attr,
381
383
};
382
384
383
-
static DEVICE_ATTR(cpumask, 0444, hisi_cpumask_sysfs_show, NULL);
384
-
385
-
static struct attribute *hisi_ddrc_pmu_cpumask_attrs[] = {
386
-
&dev_attr_cpumask.attr,
387
-
NULL,
388
-
};
389
-
390
-
static const struct attribute_group hisi_ddrc_pmu_cpumask_attr_group = {
391
-
.attrs = hisi_ddrc_pmu_cpumask_attrs,
392
-
};
393
-
394
-
static struct device_attribute hisi_ddrc_pmu_identifier_attr =
395
-
__ATTR(identifier, 0444, hisi_uncore_pmu_identifier_attr_show, NULL);
396
-
397
-
static struct attribute *hisi_ddrc_pmu_identifier_attrs[] = {
398
-
&hisi_ddrc_pmu_identifier_attr.attr,
399
-
NULL
400
-
};
401
-
402
-
static const struct attribute_group hisi_ddrc_pmu_identifier_group = {
403
-
.attrs = hisi_ddrc_pmu_identifier_attrs,
404
-
};
405
-
406
385
static const struct attribute_group *hisi_ddrc_pmu_v1_attr_groups[] = {
407
386
&hisi_ddrc_pmu_v1_format_group,
408
387
&hisi_ddrc_pmu_v1_events_group,
409
-
&hisi_ddrc_pmu_cpumask_attr_group,
410
-
&hisi_ddrc_pmu_identifier_group,
388
+
&hisi_pmu_cpumask_attr_group,
389
+
&hisi_pmu_identifier_group,
411
390
NULL,
412
391
};
413
392
414
393
static const struct attribute_group *hisi_ddrc_pmu_v2_attr_groups[] = {
415
394
&hisi_ddrc_pmu_v2_format_group,
416
395
&hisi_ddrc_pmu_v2_events_group,
417
-
&hisi_ddrc_pmu_cpumask_attr_group,
418
-
&hisi_ddrc_pmu_identifier_group,
396
+
&hisi_pmu_cpumask_attr_group,
397
+
&hisi_pmu_identifier_group,
419
398
NULL
420
399
};
421
400
···
476
501
477
502
if (ddrc_pmu->identifier >= HISI_PMU_V2)
478
503
name = devm_kasprintf(&pdev->dev, GFP_KERNEL,
479
-
"hisi_sccl%u_ddrc%u_%u",
480
-
ddrc_pmu->sccl_id, ddrc_pmu->index_id,
481
-
ddrc_pmu->sub_id);
504
+
"hisi_sccl%d_ddrc%d_%d",
505
+
ddrc_pmu->topo.sccl_id, ddrc_pmu->topo.index_id,
506
+
ddrc_pmu->topo.sub_id);
482
507
else
483
508
name = devm_kasprintf(&pdev->dev, GFP_KERNEL,
484
-
"hisi_sccl%u_ddrc%u", ddrc_pmu->sccl_id,
485
-
ddrc_pmu->index_id);
509
+
"hisi_sccl%d_ddrc%d", ddrc_pmu->topo.sccl_id,
510
+
ddrc_pmu->topo.index_id);
486
511
487
512
if (!name)
488
513
return -ENOMEM;
···
550
575
{
551
576
platform_driver_unregister(&hisi_ddrc_pmu_driver);
552
577
cpuhp_remove_multi_state(CPUHP_AP_PERF_ARM_HISI_DDRC_ONLINE);
553
-
554
578
}
555
579
module_exit(hisi_ddrc_pmu_module_exit);
556
580
581
+
MODULE_IMPORT_NS("HISI_PMU");
557
582
MODULE_DESCRIPTION("HiSilicon SoC DDRC uncore PMU driver");
558
583
MODULE_LICENSE("GPL v2");
559
584
MODULE_AUTHOR("Shaokun Zhang <zhangshaokun@hisilicon.com>");
+12
-36
drivers/perf/hisilicon/hisi_uncore_hha_pmu.c
+12
-36
drivers/perf/hisilicon/hisi_uncore_hha_pmu.c
···
295
295
unsigned long long id;
296
296
acpi_status status;
297
297
298
+
hisi_uncore_pmu_init_topology(hha_pmu, &pdev->dev);
299
+
298
300
/*
299
301
* Use SCCL_ID and UID to identify the HHA PMU, while
300
302
* SCCL_ID is in MPIDR[aff2].
301
303
*/
302
-
if (device_property_read_u32(&pdev->dev, "hisilicon,scl-id",
303
-
&hha_pmu->sccl_id)) {
304
+
if (hha_pmu->topo.sccl_id < 0) {
304
305
dev_err(&pdev->dev, "Can not read hha sccl-id!\n");
305
306
return -EINVAL;
306
307
}
···
310
309
* Early versions of BIOS support _UID by mistake, so we support
311
310
* both "hisilicon, idx-id" as preference, if available.
312
311
*/
313
-
if (device_property_read_u32(&pdev->dev, "hisilicon,idx-id",
314
-
&hha_pmu->index_id)) {
312
+
if (hha_pmu->topo.index_id < 0) {
315
313
status = acpi_evaluate_integer(ACPI_HANDLE(&pdev->dev),
316
314
"_UID", NULL, &id);
317
315
if (ACPI_FAILURE(status)) {
···
318
318
return -EINVAL;
319
319
}
320
320
321
-
hha_pmu->index_id = id;
321
+
hha_pmu->topo.index_id = id;
322
322
}
323
-
/* HHA PMUs only share the same SCCL */
324
-
hha_pmu->ccl_id = -1;
325
323
326
324
hha_pmu->base = devm_platform_ioremap_resource(pdev, 0);
327
325
if (IS_ERR(hha_pmu->base)) {
···
405
407
.attrs = hisi_hha_pmu_v2_events_attr,
406
408
};
407
409
408
-
static DEVICE_ATTR(cpumask, 0444, hisi_cpumask_sysfs_show, NULL);
409
-
410
-
static struct attribute *hisi_hha_pmu_cpumask_attrs[] = {
411
-
&dev_attr_cpumask.attr,
412
-
NULL,
413
-
};
414
-
415
-
static const struct attribute_group hisi_hha_pmu_cpumask_attr_group = {
416
-
.attrs = hisi_hha_pmu_cpumask_attrs,
417
-
};
418
-
419
-
static struct device_attribute hisi_hha_pmu_identifier_attr =
420
-
__ATTR(identifier, 0444, hisi_uncore_pmu_identifier_attr_show, NULL);
421
-
422
-
static struct attribute *hisi_hha_pmu_identifier_attrs[] = {
423
-
&hisi_hha_pmu_identifier_attr.attr,
424
-
NULL
425
-
};
426
-
427
-
static const struct attribute_group hisi_hha_pmu_identifier_group = {
428
-
.attrs = hisi_hha_pmu_identifier_attrs,
429
-
};
430
-
431
410
static const struct attribute_group *hisi_hha_pmu_v1_attr_groups[] = {
432
411
&hisi_hha_pmu_v1_format_group,
433
412
&hisi_hha_pmu_v1_events_group,
434
-
&hisi_hha_pmu_cpumask_attr_group,
435
-
&hisi_hha_pmu_identifier_group,
413
+
&hisi_pmu_cpumask_attr_group,
414
+
&hisi_pmu_identifier_group,
436
415
NULL,
437
416
};
438
417
439
418
static const struct attribute_group *hisi_hha_pmu_v2_attr_groups[] = {
440
419
&hisi_hha_pmu_v2_format_group,
441
420
&hisi_hha_pmu_v2_events_group,
442
-
&hisi_hha_pmu_cpumask_attr_group,
443
-
&hisi_hha_pmu_identifier_group,
421
+
&hisi_pmu_cpumask_attr_group,
422
+
&hisi_pmu_identifier_group,
444
423
NULL
445
424
};
446
425
···
485
510
if (ret)
486
511
return ret;
487
512
488
-
name = devm_kasprintf(&pdev->dev, GFP_KERNEL, "hisi_sccl%u_hha%u",
489
-
hha_pmu->sccl_id, hha_pmu->index_id);
513
+
name = devm_kasprintf(&pdev->dev, GFP_KERNEL, "hisi_sccl%d_hha%d",
514
+
hha_pmu->topo.sccl_id, hha_pmu->topo.index_id);
490
515
if (!name)
491
516
return -ENOMEM;
492
517
···
556
581
}
557
582
module_exit(hisi_hha_pmu_module_exit);
558
583
584
+
MODULE_IMPORT_NS("HISI_PMU");
559
585
MODULE_DESCRIPTION("HiSilicon SoC HHA uncore PMU driver");
560
586
MODULE_LICENSE("GPL v2");
561
587
MODULE_AUTHOR("Shaokun Zhang <zhangshaokun@hisilicon.com>");
+11
-33
drivers/perf/hisilicon/hisi_uncore_l3c_pmu.c
+11
-33
drivers/perf/hisilicon/hisi_uncore_l3c_pmu.c
···
355
355
static int hisi_l3c_pmu_init_data(struct platform_device *pdev,
356
356
struct hisi_pmu *l3c_pmu)
357
357
{
358
+
hisi_uncore_pmu_init_topology(l3c_pmu, &pdev->dev);
359
+
358
360
/*
359
361
* Use the SCCL_ID and CCL_ID to identify the L3C PMU, while
360
362
* SCCL_ID is in MPIDR[aff2] and CCL_ID is in MPIDR[aff1].
361
363
*/
362
-
if (device_property_read_u32(&pdev->dev, "hisilicon,scl-id",
363
-
&l3c_pmu->sccl_id)) {
364
+
if (l3c_pmu->topo.sccl_id < 0) {
364
365
dev_err(&pdev->dev, "Can not read l3c sccl-id!\n");
365
366
return -EINVAL;
366
367
}
367
368
368
-
if (device_property_read_u32(&pdev->dev, "hisilicon,ccl-id",
369
-
&l3c_pmu->ccl_id)) {
369
+
if (l3c_pmu->topo.ccl_id < 0) {
370
370
dev_err(&pdev->dev, "Can not read l3c ccl-id!\n");
371
371
return -EINVAL;
372
372
}
···
441
441
.attrs = hisi_l3c_pmu_v2_events_attr,
442
442
};
443
443
444
-
static DEVICE_ATTR(cpumask, 0444, hisi_cpumask_sysfs_show, NULL);
445
-
446
-
static struct attribute *hisi_l3c_pmu_cpumask_attrs[] = {
447
-
&dev_attr_cpumask.attr,
448
-
NULL,
449
-
};
450
-
451
-
static const struct attribute_group hisi_l3c_pmu_cpumask_attr_group = {
452
-
.attrs = hisi_l3c_pmu_cpumask_attrs,
453
-
};
454
-
455
-
static struct device_attribute hisi_l3c_pmu_identifier_attr =
456
-
__ATTR(identifier, 0444, hisi_uncore_pmu_identifier_attr_show, NULL);
457
-
458
-
static struct attribute *hisi_l3c_pmu_identifier_attrs[] = {
459
-
&hisi_l3c_pmu_identifier_attr.attr,
460
-
NULL
461
-
};
462
-
463
-
static const struct attribute_group hisi_l3c_pmu_identifier_group = {
464
-
.attrs = hisi_l3c_pmu_identifier_attrs,
465
-
};
466
-
467
444
static const struct attribute_group *hisi_l3c_pmu_v1_attr_groups[] = {
468
445
&hisi_l3c_pmu_v1_format_group,
469
446
&hisi_l3c_pmu_v1_events_group,
470
-
&hisi_l3c_pmu_cpumask_attr_group,
471
-
&hisi_l3c_pmu_identifier_group,
447
+
&hisi_pmu_cpumask_attr_group,
448
+
&hisi_pmu_identifier_group,
472
449
NULL,
473
450
};
474
451
475
452
static const struct attribute_group *hisi_l3c_pmu_v2_attr_groups[] = {
476
453
&hisi_l3c_pmu_v2_format_group,
477
454
&hisi_l3c_pmu_v2_events_group,
478
-
&hisi_l3c_pmu_cpumask_attr_group,
479
-
&hisi_l3c_pmu_identifier_group,
455
+
&hisi_pmu_cpumask_attr_group,
456
+
&hisi_pmu_identifier_group,
480
457
NULL
481
458
};
482
459
···
521
544
if (ret)
522
545
return ret;
523
546
524
-
name = devm_kasprintf(&pdev->dev, GFP_KERNEL, "hisi_sccl%u_l3c%u",
525
-
l3c_pmu->sccl_id, l3c_pmu->ccl_id);
547
+
name = devm_kasprintf(&pdev->dev, GFP_KERNEL, "hisi_sccl%d_l3c%d",
548
+
l3c_pmu->topo.sccl_id, l3c_pmu->topo.ccl_id);
526
549
if (!name)
527
550
return -ENOMEM;
528
551
···
592
615
}
593
616
module_exit(hisi_l3c_pmu_module_exit);
594
617
618
+
MODULE_IMPORT_NS("HISI_PMU");
595
619
MODULE_DESCRIPTION("HiSilicon SoC L3C uncore PMU driver");
596
620
MODULE_LICENSE("GPL v2");
597
621
MODULE_AUTHOR("Anurup M <anurup.m@huawei.com>");
+14
-39
drivers/perf/hisilicon/hisi_uncore_pa_pmu.c
+14
-39
drivers/perf/hisilicon/hisi_uncore_pa_pmu.c
···
269
269
static int hisi_pa_pmu_init_data(struct platform_device *pdev,
270
270
struct hisi_pmu *pa_pmu)
271
271
{
272
+
hisi_uncore_pmu_init_topology(pa_pmu, &pdev->dev);
273
+
272
274
/*
273
275
* As PA PMU is in a SICL, use the SICL_ID and the index ID
274
276
* to identify the PA PMU.
275
277
*/
276
-
if (device_property_read_u32(&pdev->dev, "hisilicon,scl-id",
277
-
&pa_pmu->sicl_id)) {
278
+
if (pa_pmu->topo.sicl_id < 0) {
278
279
dev_err(&pdev->dev, "Cannot read sicl-id!\n");
279
280
return -EINVAL;
280
281
}
281
282
282
-
if (device_property_read_u32(&pdev->dev, "hisilicon,idx-id",
283
-
&pa_pmu->index_id)) {
283
+
if (pa_pmu->topo.index_id < 0) {
284
284
dev_err(&pdev->dev, "Cannot read idx-id!\n");
285
285
return -EINVAL;
286
286
}
287
-
288
-
pa_pmu->ccl_id = -1;
289
-
pa_pmu->sccl_id = -1;
290
287
291
288
pa_pmu->dev_info = device_get_match_data(&pdev->dev);
292
289
if (!pa_pmu->dev_info)
···
353
356
.attrs = hisi_h60pa_pmu_events_attr,
354
357
};
355
358
356
-
static DEVICE_ATTR(cpumask, 0444, hisi_cpumask_sysfs_show, NULL);
357
-
358
-
static struct attribute *hisi_pa_pmu_cpumask_attrs[] = {
359
-
&dev_attr_cpumask.attr,
360
-
NULL
361
-
};
362
-
363
-
static const struct attribute_group hisi_pa_pmu_cpumask_attr_group = {
364
-
.attrs = hisi_pa_pmu_cpumask_attrs,
365
-
};
366
-
367
-
static struct device_attribute hisi_pa_pmu_identifier_attr =
368
-
__ATTR(identifier, 0444, hisi_uncore_pmu_identifier_attr_show, NULL);
369
-
370
-
static struct attribute *hisi_pa_pmu_identifier_attrs[] = {
371
-
&hisi_pa_pmu_identifier_attr.attr,
372
-
NULL
373
-
};
374
-
375
-
static const struct attribute_group hisi_pa_pmu_identifier_group = {
376
-
.attrs = hisi_pa_pmu_identifier_attrs,
377
-
};
378
-
379
359
static struct hisi_pa_pmu_int_regs hisi_pa_pmu_regs = {
380
360
.mask_offset = PA_INT_MASK,
381
361
.clear_offset = PA_INT_CLEAR,
···
362
388
static const struct attribute_group *hisi_pa_pmu_v2_attr_groups[] = {
363
389
&hisi_pa_pmu_v2_format_group,
364
390
&hisi_pa_pmu_v2_events_group,
365
-
&hisi_pa_pmu_cpumask_attr_group,
366
-
&hisi_pa_pmu_identifier_group,
391
+
&hisi_pmu_cpumask_attr_group,
392
+
&hisi_pmu_identifier_group,
367
393
NULL
368
394
};
369
395
···
376
402
static const struct attribute_group *hisi_pa_pmu_v3_attr_groups[] = {
377
403
&hisi_pa_pmu_v2_format_group,
378
404
&hisi_pa_pmu_v3_events_group,
379
-
&hisi_pa_pmu_cpumask_attr_group,
380
-
&hisi_pa_pmu_identifier_group,
405
+
&hisi_pmu_cpumask_attr_group,
406
+
&hisi_pmu_identifier_group,
381
407
NULL
382
408
};
383
409
···
396
422
static const struct attribute_group *hisi_h60pa_pmu_attr_groups[] = {
397
423
&hisi_pa_pmu_v2_format_group,
398
424
&hisi_h60pa_pmu_events_group,
399
-
&hisi_pa_pmu_cpumask_attr_group,
400
-
&hisi_pa_pmu_identifier_group,
425
+
&hisi_pmu_cpumask_attr_group,
426
+
&hisi_pmu_identifier_group,
401
427
NULL
402
428
};
403
429
···
462
488
if (ret)
463
489
return ret;
464
490
465
-
name = devm_kasprintf(&pdev->dev, GFP_KERNEL, "hisi_sicl%d_%s%u",
466
-
pa_pmu->sicl_id, pa_pmu->dev_info->name,
467
-
pa_pmu->index_id);
491
+
name = devm_kasprintf(&pdev->dev, GFP_KERNEL, "hisi_sicl%d_%s%d",
492
+
pa_pmu->topo.sicl_id, pa_pmu->dev_info->name,
493
+
pa_pmu->topo.index_id);
468
494
if (!name)
469
495
return -ENOMEM;
470
496
···
543
569
}
544
570
module_exit(hisi_pa_pmu_module_exit);
545
571
572
+
MODULE_IMPORT_NS("HISI_PMU");
546
573
MODULE_DESCRIPTION("HiSilicon Protocol Adapter uncore PMU driver");
547
574
MODULE_LICENSE("GPL v2");
548
575
MODULE_AUTHOR("Shaokun Zhang <zhangshaokun@hisilicon.com>");
+119
-41
drivers/perf/hisilicon/hisi_uncore_pmu.c
+119
-41
drivers/perf/hisilicon/hisi_uncore_pmu.c
···
14
14
#include <linux/err.h>
15
15
#include <linux/errno.h>
16
16
#include <linux/interrupt.h>
17
+
#include <linux/property.h>
17
18
18
19
#include <asm/cputype.h>
19
20
#include <asm/local64.h>
···
35
34
36
35
return sysfs_emit(page, "config=0x%lx\n", (unsigned long)eattr->var);
37
36
}
38
-
EXPORT_SYMBOL_GPL(hisi_event_sysfs_show);
37
+
EXPORT_SYMBOL_NS_GPL(hisi_event_sysfs_show, "HISI_PMU");
39
38
40
39
/*
41
40
* sysfs cpumask attributes. For uncore PMU, we only have a single CPU to show
···
47
46
48
47
return sysfs_emit(buf, "%d\n", hisi_pmu->on_cpu);
49
48
}
50
-
EXPORT_SYMBOL_GPL(hisi_cpumask_sysfs_show);
49
+
EXPORT_SYMBOL_NS_GPL(hisi_cpumask_sysfs_show, "HISI_PMU");
50
+
51
+
static DEVICE_ATTR(cpumask, 0444, hisi_cpumask_sysfs_show, NULL);
52
+
53
+
static ssize_t hisi_associated_cpus_sysfs_show(struct device *dev,
54
+
struct device_attribute *attr, char *buf)
55
+
{
56
+
struct hisi_pmu *hisi_pmu = to_hisi_pmu(dev_get_drvdata(dev));
57
+
58
+
return cpumap_print_to_pagebuf(true, buf, &hisi_pmu->associated_cpus);
59
+
}
60
+
static DEVICE_ATTR(associated_cpus, 0444, hisi_associated_cpus_sysfs_show, NULL);
61
+
62
+
static struct attribute *hisi_pmu_cpumask_attrs[] = {
63
+
&dev_attr_cpumask.attr,
64
+
&dev_attr_associated_cpus.attr,
65
+
NULL
66
+
};
67
+
68
+
const struct attribute_group hisi_pmu_cpumask_attr_group = {
69
+
.attrs = hisi_pmu_cpumask_attrs,
70
+
};
71
+
EXPORT_SYMBOL_NS_GPL(hisi_pmu_cpumask_attr_group, "HISI_PMU");
72
+
73
+
ssize_t hisi_uncore_pmu_identifier_attr_show(struct device *dev,
74
+
struct device_attribute *attr,
75
+
char *page)
76
+
{
77
+
struct hisi_pmu *hisi_pmu = to_hisi_pmu(dev_get_drvdata(dev));
78
+
79
+
return sysfs_emit(page, "0x%08x\n", hisi_pmu->identifier);
80
+
}
81
+
EXPORT_SYMBOL_NS_GPL(hisi_uncore_pmu_identifier_attr_show, "HISI_PMU");
82
+
83
+
static struct device_attribute hisi_pmu_identifier_attr =
84
+
__ATTR(identifier, 0444, hisi_uncore_pmu_identifier_attr_show, NULL);
85
+
86
+
static struct attribute *hisi_pmu_identifier_attrs[] = {
87
+
&hisi_pmu_identifier_attr.attr,
88
+
NULL
89
+
};
90
+
91
+
const struct attribute_group hisi_pmu_identifier_group = {
92
+
.attrs = hisi_pmu_identifier_attrs,
93
+
};
94
+
EXPORT_SYMBOL_NS_GPL(hisi_pmu_identifier_group, "HISI_PMU");
51
95
52
96
static bool hisi_validate_event_group(struct perf_event *event)
53
97
{
···
142
96
143
97
return idx;
144
98
}
145
-
EXPORT_SYMBOL_GPL(hisi_uncore_pmu_get_event_idx);
146
-
147
-
ssize_t hisi_uncore_pmu_identifier_attr_show(struct device *dev,
148
-
struct device_attribute *attr,
149
-
char *page)
150
-
{
151
-
struct hisi_pmu *hisi_pmu = to_hisi_pmu(dev_get_drvdata(dev));
152
-
153
-
return sysfs_emit(page, "0x%08x\n", hisi_pmu->identifier);
154
-
}
155
-
EXPORT_SYMBOL_GPL(hisi_uncore_pmu_identifier_attr_show);
99
+
EXPORT_SYMBOL_NS_GPL(hisi_uncore_pmu_get_event_idx, "HISI_PMU");
156
100
157
101
static void hisi_uncore_pmu_clear_event_idx(struct hisi_pmu *hisi_pmu, int idx)
158
102
{
···
201
165
202
166
return 0;
203
167
}
204
-
EXPORT_SYMBOL_GPL(hisi_uncore_pmu_init_irq);
168
+
EXPORT_SYMBOL_NS_GPL(hisi_uncore_pmu_init_irq, "HISI_PMU");
205
169
206
170
int hisi_uncore_pmu_event_init(struct perf_event *event)
207
171
{
···
255
219
256
220
return 0;
257
221
}
258
-
EXPORT_SYMBOL_GPL(hisi_uncore_pmu_event_init);
222
+
EXPORT_SYMBOL_NS_GPL(hisi_uncore_pmu_event_init, "HISI_PMU");
259
223
260
224
/*
261
225
* Set the counter to count the event that we're interested in,
···
309
273
/* Write start value to the hardware event counter */
310
274
hisi_pmu->ops->write_counter(hisi_pmu, hwc, val);
311
275
}
312
-
EXPORT_SYMBOL_GPL(hisi_uncore_pmu_set_event_period);
276
+
EXPORT_SYMBOL_NS_GPL(hisi_uncore_pmu_set_event_period, "HISI_PMU");
313
277
314
278
void hisi_uncore_pmu_event_update(struct perf_event *event)
315
279
{
···
330
294
HISI_MAX_PERIOD(hisi_pmu->counter_bits);
331
295
local64_add(delta, &event->count);
332
296
}
333
-
EXPORT_SYMBOL_GPL(hisi_uncore_pmu_event_update);
297
+
EXPORT_SYMBOL_NS_GPL(hisi_uncore_pmu_event_update, "HISI_PMU");
334
298
335
299
void hisi_uncore_pmu_start(struct perf_event *event, int flags)
336
300
{
···
353
317
hisi_uncore_pmu_enable_event(event);
354
318
perf_event_update_userpage(event);
355
319
}
356
-
EXPORT_SYMBOL_GPL(hisi_uncore_pmu_start);
320
+
EXPORT_SYMBOL_NS_GPL(hisi_uncore_pmu_start, "HISI_PMU");
357
321
358
322
void hisi_uncore_pmu_stop(struct perf_event *event, int flags)
359
323
{
···
370
334
hisi_uncore_pmu_event_update(event);
371
335
hwc->state |= PERF_HES_UPTODATE;
372
336
}
373
-
EXPORT_SYMBOL_GPL(hisi_uncore_pmu_stop);
337
+
EXPORT_SYMBOL_NS_GPL(hisi_uncore_pmu_stop, "HISI_PMU");
374
338
375
339
int hisi_uncore_pmu_add(struct perf_event *event, int flags)
376
340
{
···
393
357
394
358
return 0;
395
359
}
396
-
EXPORT_SYMBOL_GPL(hisi_uncore_pmu_add);
360
+
EXPORT_SYMBOL_NS_GPL(hisi_uncore_pmu_add, "HISI_PMU");
397
361
398
362
void hisi_uncore_pmu_del(struct perf_event *event, int flags)
399
363
{
···
405
369
perf_event_update_userpage(event);
406
370
hisi_pmu->pmu_events.hw_events[hwc->idx] = NULL;
407
371
}
408
-
EXPORT_SYMBOL_GPL(hisi_uncore_pmu_del);
372
+
EXPORT_SYMBOL_NS_GPL(hisi_uncore_pmu_del, "HISI_PMU");
409
373
410
374
void hisi_uncore_pmu_read(struct perf_event *event)
411
375
{
412
376
/* Read hardware counter and update the perf counter statistics */
413
377
hisi_uncore_pmu_event_update(event);
414
378
}
415
-
EXPORT_SYMBOL_GPL(hisi_uncore_pmu_read);
379
+
EXPORT_SYMBOL_NS_GPL(hisi_uncore_pmu_read, "HISI_PMU");
416
380
417
381
void hisi_uncore_pmu_enable(struct pmu *pmu)
418
382
{
···
425
389
426
390
hisi_pmu->ops->start_counters(hisi_pmu);
427
391
}
428
-
EXPORT_SYMBOL_GPL(hisi_uncore_pmu_enable);
392
+
EXPORT_SYMBOL_NS_GPL(hisi_uncore_pmu_enable, "HISI_PMU");
429
393
430
394
void hisi_uncore_pmu_disable(struct pmu *pmu)
431
395
{
···
433
397
434
398
hisi_pmu->ops->stop_counters(hisi_pmu);
435
399
}
436
-
EXPORT_SYMBOL_GPL(hisi_uncore_pmu_disable);
400
+
EXPORT_SYMBOL_NS_GPL(hisi_uncore_pmu_disable, "HISI_PMU");
437
401
438
402
439
403
/*
···
480
444
*/
481
445
static bool hisi_pmu_cpu_is_associated_pmu(struct hisi_pmu *hisi_pmu)
482
446
{
447
+
struct hisi_pmu_topology *topo = &hisi_pmu->topo;
483
448
int sccl_id, ccl_id;
484
449
485
-
/* If SCCL_ID is -1, the PMU is in a SICL and has no CPU affinity */
486
-
if (hisi_pmu->sccl_id == -1)
487
-
return true;
488
-
489
-
if (hisi_pmu->ccl_id == -1) {
450
+
if (topo->ccl_id == -1) {
490
451
/* If CCL_ID is -1, the PMU only shares the same SCCL */
491
452
hisi_read_sccl_and_ccl_id(&sccl_id, NULL);
492
453
493
-
return sccl_id == hisi_pmu->sccl_id;
454
+
return sccl_id == topo->sccl_id;
494
455
}
495
456
496
457
hisi_read_sccl_and_ccl_id(&sccl_id, &ccl_id);
497
458
498
-
return sccl_id == hisi_pmu->sccl_id && ccl_id == hisi_pmu->ccl_id;
459
+
return sccl_id == topo->sccl_id && ccl_id == topo->ccl_id;
499
460
}
500
461
501
462
int hisi_uncore_pmu_online_cpu(unsigned int cpu, struct hlist_node *node)
···
500
467
struct hisi_pmu *hisi_pmu = hlist_entry_safe(node, struct hisi_pmu,
501
468
node);
502
469
503
-
if (!hisi_pmu_cpu_is_associated_pmu(hisi_pmu))
470
+
/*
471
+
* If the CPU is not associated to PMU, initialize the hisi_pmu->on_cpu
472
+
* based on the locality if it hasn't been initialized yet. For PMUs
473
+
* do have associated CPUs, it'll be updated later.
474
+
*/
475
+
if (!hisi_pmu_cpu_is_associated_pmu(hisi_pmu)) {
476
+
if (hisi_pmu->on_cpu != -1)
477
+
return 0;
478
+
479
+
hisi_pmu->on_cpu = cpumask_local_spread(0, dev_to_node(hisi_pmu->dev));
480
+
WARN_ON(irq_set_affinity(hisi_pmu->irq, cpumask_of(hisi_pmu->on_cpu)));
504
481
return 0;
482
+
}
505
483
506
484
cpumask_set_cpu(cpu, &hisi_pmu->associated_cpus);
507
485
508
-
/* If another CPU is already managing this PMU, simply return. */
509
-
if (hisi_pmu->on_cpu != -1)
486
+
/* If another associated CPU is already managing this PMU, simply return. */
487
+
if (hisi_pmu->on_cpu != -1 &&
488
+
cpumask_test_cpu(hisi_pmu->on_cpu, &hisi_pmu->associated_cpus))
510
489
return 0;
511
490
512
491
/* Use this CPU in cpumask for event counting */
···
529
484
530
485
return 0;
531
486
}
532
-
EXPORT_SYMBOL_GPL(hisi_uncore_pmu_online_cpu);
487
+
EXPORT_SYMBOL_NS_GPL(hisi_uncore_pmu_online_cpu, "HISI_PMU");
533
488
534
489
int hisi_uncore_pmu_offline_cpu(unsigned int cpu, struct hlist_node *node)
535
490
{
536
491
struct hisi_pmu *hisi_pmu = hlist_entry_safe(node, struct hisi_pmu,
537
492
node);
538
493
unsigned int target;
539
-
540
-
if (!cpumask_test_and_clear_cpu(cpu, &hisi_pmu->associated_cpus))
541
-
return 0;
542
494
543
495
/* Nothing to do if this CPU doesn't own the PMU */
544
496
if (hisi_pmu->on_cpu != cpu)
···
544
502
/* Give up ownership of the PMU */
545
503
hisi_pmu->on_cpu = -1;
546
504
547
-
/* Choose a new CPU to migrate ownership of the PMU to */
505
+
/*
506
+
* Migrate ownership of the PMU to a new CPU chosen from PMU's online
507
+
* associated CPUs if possible, if no associated CPU online then
508
+
* migrate to one online CPU.
509
+
*/
548
510
target = cpumask_any_and_but(&hisi_pmu->associated_cpus,
549
511
cpu_online_mask, cpu);
512
+
if (target >= nr_cpu_ids)
513
+
target = cpumask_any_but(cpu_online_mask, cpu);
514
+
550
515
if (target >= nr_cpu_ids)
551
516
return 0;
552
517
···
564
515
565
516
return 0;
566
517
}
567
-
EXPORT_SYMBOL_GPL(hisi_uncore_pmu_offline_cpu);
518
+
EXPORT_SYMBOL_NS_GPL(hisi_uncore_pmu_offline_cpu, "HISI_PMU");
519
+
520
+
/*
521
+
* Retrieve the topology information from the firmware for the hisi_pmu device.
522
+
* The topology ID will be -1 if we cannot initialize it, it may either due to
523
+
* the PMU doesn't locate on this certain topology or the firmware needs to be
524
+
* fixed.
525
+
*/
526
+
void hisi_uncore_pmu_init_topology(struct hisi_pmu *hisi_pmu, struct device *dev)
527
+
{
528
+
struct hisi_pmu_topology *topo = &hisi_pmu->topo;
529
+
530
+
topo->sccl_id = -1;
531
+
topo->ccl_id = -1;
532
+
topo->index_id = -1;
533
+
topo->sub_id = -1;
534
+
535
+
if (device_property_read_u32(dev, "hisilicon,scl-id", &topo->sccl_id))
536
+
dev_dbg(dev, "no scl-id present\n");
537
+
538
+
if (device_property_read_u32(dev, "hisilicon,ccl-id", &topo->ccl_id))
539
+
dev_dbg(dev, "no ccl-id present\n");
540
+
541
+
if (device_property_read_u32(dev, "hisilicon,idx-id", &topo->index_id))
542
+
dev_dbg(dev, "no idx-id present\n");
543
+
544
+
if (device_property_read_u32(dev, "hisilicon,sub-id", &topo->sub_id))
545
+
dev_dbg(dev, "no sub-id present\n");
546
+
}
547
+
EXPORT_SYMBOL_NS_GPL(hisi_uncore_pmu_init_topology, "HISI_PMU");
568
548
569
549
void hisi_pmu_init(struct hisi_pmu *hisi_pmu, struct module *module)
570
550
{
···
613
535
pmu->attr_groups = hisi_pmu->pmu_events.attr_groups;
614
536
pmu->capabilities = PERF_PMU_CAP_NO_EXCLUDE;
615
537
}
616
-
EXPORT_SYMBOL_GPL(hisi_pmu_init);
538
+
EXPORT_SYMBOL_NS_GPL(hisi_pmu_init, "HISI_PMU");
617
539
618
540
MODULE_DESCRIPTION("HiSilicon SoC uncore Performance Monitor driver framework");
619
541
MODULE_LICENSE("GPL v2");
+41
-8
drivers/perf/hisilicon/hisi_uncore_pmu.h
+41
-8
drivers/perf/hisilicon/hisi_uncore_pmu.h
···
81
81
const struct attribute_group **attr_groups;
82
82
};
83
83
84
+
/**
85
+
* struct hisi_pmu_topology - Describe the topology hierarchy on which the PMU
86
+
* is located.
87
+
* @sccl_id: ID of the SCCL on which the PMU locate is located.
88
+
* @sicl_id: ID of the SICL on which the PMU locate is located.
89
+
* @scl_id: ID used by the core which is unaware of the SCCL/SICL.
90
+
* @ccl_id: ID of the CCL (CPU cluster) on which the PMU is located.
91
+
* @index_id: the ID of the PMU module if there're several PMUs at a
92
+
* particularly location in the topology.
93
+
* @sub_id: submodule ID of the PMU. For example we use this for DDRC PMU v2
94
+
* since each DDRC has more than one DMC
95
+
*
96
+
* The ID will be -1 if the PMU isn't located on a certain topology.
97
+
*/
98
+
struct hisi_pmu_topology {
99
+
/*
100
+
* SCCL (Super CPU CLuster) and SICL (Super I/O Cluster) are parallel
101
+
* so a PMU cannot locate on a SCCL and a SICL. If the SCCL/SICL
102
+
* distinction is not relevant, use scl_id instead.
103
+
*/
104
+
union {
105
+
int sccl_id;
106
+
int sicl_id;
107
+
int scl_id;
108
+
};
109
+
int ccl_id;
110
+
int index_id;
111
+
int sub_id;
112
+
};
113
+
84
114
/* Generic pmu struct for different pmu types */
85
115
struct hisi_pmu {
86
116
struct pmu pmu;
87
117
const struct hisi_uncore_ops *ops;
88
118
const struct hisi_pmu_dev_info *dev_info;
89
119
struct hisi_pmu_hwevents pmu_events;
90
-
/* associated_cpus: All CPUs associated with the PMU */
120
+
struct hisi_pmu_topology topo;
121
+
/*
122
+
* CPUs associated to the PMU and are preferred to use for counting.
123
+
* Could be empty if PMU has no association (e.g. PMU on SICL), in
124
+
* which case any online CPU will be used.
125
+
*/
91
126
cpumask_t associated_cpus;
92
127
/* CPU used for counting */
93
128
int on_cpu;
94
129
int irq;
95
130
struct device *dev;
96
131
struct hlist_node node;
97
-
int sccl_id;
98
-
int sicl_id;
99
-
int ccl_id;
100
132
void __iomem *base;
101
-
/* the ID of the PMU modules */
102
-
u32 index_id;
103
-
/* For DDRC PMU v2: each DDRC has more than one DMC */
104
-
u32 sub_id;
105
133
int num_counters;
106
134
int counter_bits;
107
135
/* check event code range */
108
136
int check_event;
109
137
u32 identifier;
110
138
};
139
+
140
+
/* Generic implementation of cpumask/identifier group */
141
+
extern const struct attribute_group hisi_pmu_cpumask_attr_group;
142
+
extern const struct attribute_group hisi_pmu_identifier_group;
111
143
112
144
int hisi_uncore_pmu_get_event_idx(struct perf_event *event);
113
145
void hisi_uncore_pmu_read(struct perf_event *event);
···
164
132
char *page);
165
133
int hisi_uncore_pmu_init_irq(struct hisi_pmu *hisi_pmu,
166
134
struct platform_device *pdev);
135
+
void hisi_uncore_pmu_init_topology(struct hisi_pmu *hisi_pmu, struct device *dev);
167
136
168
137
void hisi_pmu_init(struct hisi_pmu *hisi_pmu, struct module *module);
169
138
#endif /* __HISI_UNCORE_PMU_H__ */
+9
-34
drivers/perf/hisilicon/hisi_uncore_sllc_pmu.c
+9
-34
drivers/perf/hisilicon/hisi_uncore_sllc_pmu.c
···
288
288
static int hisi_sllc_pmu_init_data(struct platform_device *pdev,
289
289
struct hisi_pmu *sllc_pmu)
290
290
{
291
+
hisi_uncore_pmu_init_topology(sllc_pmu, &pdev->dev);
292
+
291
293
/*
292
294
* Use the SCCL_ID and the index ID to identify the SLLC PMU,
293
295
* while SCCL_ID is from MPIDR_EL1 by CPU.
294
296
*/
295
-
if (device_property_read_u32(&pdev->dev, "hisilicon,scl-id",
296
-
&sllc_pmu->sccl_id)) {
297
+
if (sllc_pmu->topo.sccl_id < 0) {
297
298
dev_err(&pdev->dev, "Cannot read sccl-id!\n");
298
299
return -EINVAL;
299
300
}
300
301
301
-
if (device_property_read_u32(&pdev->dev, "hisilicon,idx-id",
302
-
&sllc_pmu->index_id)) {
302
+
if (sllc_pmu->topo.index_id < 0) {
303
303
dev_err(&pdev->dev, "Cannot read idx-id!\n");
304
304
return -EINVAL;
305
305
}
306
-
307
-
/* SLLC PMUs only share the same SCCL */
308
-
sllc_pmu->ccl_id = -1;
309
306
310
307
sllc_pmu->base = devm_platform_ioremap_resource(pdev, 0);
311
308
if (IS_ERR(sllc_pmu->base)) {
···
344
347
.attrs = hisi_sllc_pmu_v2_events_attr,
345
348
};
346
349
347
-
static DEVICE_ATTR(cpumask, 0444, hisi_cpumask_sysfs_show, NULL);
348
-
349
-
static struct attribute *hisi_sllc_pmu_cpumask_attrs[] = {
350
-
&dev_attr_cpumask.attr,
351
-
NULL
352
-
};
353
-
354
-
static const struct attribute_group hisi_sllc_pmu_cpumask_attr_group = {
355
-
.attrs = hisi_sllc_pmu_cpumask_attrs,
356
-
};
357
-
358
-
static struct device_attribute hisi_sllc_pmu_identifier_attr =
359
-
__ATTR(identifier, 0444, hisi_uncore_pmu_identifier_attr_show, NULL);
360
-
361
-
static struct attribute *hisi_sllc_pmu_identifier_attrs[] = {
362
-
&hisi_sllc_pmu_identifier_attr.attr,
363
-
NULL
364
-
};
365
-
366
-
static const struct attribute_group hisi_sllc_pmu_identifier_group = {
367
-
.attrs = hisi_sllc_pmu_identifier_attrs,
368
-
};
369
-
370
350
static const struct attribute_group *hisi_sllc_pmu_v2_attr_groups[] = {
371
351
&hisi_sllc_pmu_v2_format_group,
372
352
&hisi_sllc_pmu_v2_events_group,
373
-
&hisi_sllc_pmu_cpumask_attr_group,
374
-
&hisi_sllc_pmu_identifier_group,
353
+
&hisi_pmu_cpumask_attr_group,
354
+
&hisi_pmu_identifier_group,
375
355
NULL
376
356
};
377
357
···
407
433
if (ret)
408
434
return ret;
409
435
410
-
name = devm_kasprintf(&pdev->dev, GFP_KERNEL, "hisi_sccl%u_sllc%u",
411
-
sllc_pmu->sccl_id, sllc_pmu->index_id);
436
+
name = devm_kasprintf(&pdev->dev, GFP_KERNEL, "hisi_sccl%d_sllc%d",
437
+
sllc_pmu->topo.sccl_id, sllc_pmu->topo.index_id);
412
438
if (!name)
413
439
return -ENOMEM;
414
440
···
481
507
}
482
508
module_exit(hisi_sllc_pmu_module_exit);
483
509
510
+
MODULE_IMPORT_NS("HISI_PMU");
484
511
MODULE_DESCRIPTION("HiSilicon SLLC uncore PMU driver");
485
512
MODULE_LICENSE("GPL v2");
486
513
MODULE_AUTHOR("Shaokun Zhang <zhangshaokun@hisilicon.com>");
+11
-34
drivers/perf/hisilicon/hisi_uncore_uc_pmu.c
+11
-34
drivers/perf/hisilicon/hisi_uncore_uc_pmu.c
···
11
11
#include <linux/irq.h>
12
12
#include <linux/list.h>
13
13
#include <linux/mod_devicetable.h>
14
-
#include <linux/property.h>
15
14
16
15
#include "hisi_uncore_pmu.h"
17
16
···
365
366
static int hisi_uc_pmu_init_data(struct platform_device *pdev,
366
367
struct hisi_pmu *uc_pmu)
367
368
{
369
+
hisi_uncore_pmu_init_topology(uc_pmu, &pdev->dev);
370
+
368
371
/*
369
372
* Use SCCL (Super CPU Cluster) ID and CCL (CPU Cluster) ID to
370
373
* identify the topology information of UC PMU devices in the chip.
371
374
* They have some CCLs per SCCL and then 4 UC PMU per CCL.
372
375
*/
373
-
if (device_property_read_u32(&pdev->dev, "hisilicon,scl-id",
374
-
&uc_pmu->sccl_id)) {
376
+
if (uc_pmu->topo.sccl_id < 0) {
375
377
dev_err(&pdev->dev, "Can not read uc sccl-id!\n");
376
378
return -EINVAL;
377
379
}
378
380
379
-
if (device_property_read_u32(&pdev->dev, "hisilicon,ccl-id",
380
-
&uc_pmu->ccl_id)) {
381
+
if (uc_pmu->topo.ccl_id < 0) {
381
382
dev_err(&pdev->dev, "Can not read uc ccl-id!\n");
382
383
return -EINVAL;
383
384
}
384
385
385
-
if (device_property_read_u32(&pdev->dev, "hisilicon,sub-id",
386
-
&uc_pmu->sub_id)) {
386
+
if (uc_pmu->topo.sub_id < 0) {
387
387
dev_err(&pdev->dev, "Can not read sub-id!\n");
388
388
return -EINVAL;
389
389
}
···
437
439
.attrs = hisi_uc_pmu_events_attr,
438
440
};
439
441
440
-
static DEVICE_ATTR(cpumask, 0444, hisi_cpumask_sysfs_show, NULL);
441
-
442
-
static struct attribute *hisi_uc_pmu_cpumask_attrs[] = {
443
-
&dev_attr_cpumask.attr,
444
-
NULL,
445
-
};
446
-
447
-
static const struct attribute_group hisi_uc_pmu_cpumask_attr_group = {
448
-
.attrs = hisi_uc_pmu_cpumask_attrs,
449
-
};
450
-
451
-
static struct device_attribute hisi_uc_pmu_identifier_attr =
452
-
__ATTR(identifier, 0444, hisi_uncore_pmu_identifier_attr_show, NULL);
453
-
454
-
static struct attribute *hisi_uc_pmu_identifier_attrs[] = {
455
-
&hisi_uc_pmu_identifier_attr.attr,
456
-
NULL
457
-
};
458
-
459
-
static const struct attribute_group hisi_uc_pmu_identifier_group = {
460
-
.attrs = hisi_uc_pmu_identifier_attrs,
461
-
};
462
-
463
442
static const struct attribute_group *hisi_uc_pmu_attr_groups[] = {
464
443
&hisi_uc_pmu_format_group,
465
444
&hisi_uc_pmu_events_group,
466
-
&hisi_uc_pmu_cpumask_attr_group,
467
-
&hisi_uc_pmu_identifier_group,
445
+
&hisi_pmu_cpumask_attr_group,
446
+
&hisi_pmu_identifier_group,
468
447
NULL
469
448
};
470
449
···
513
538
if (ret)
514
539
return ret;
515
540
516
-
name = devm_kasprintf(&pdev->dev, GFP_KERNEL, "hisi_sccl%d_uc%d_%u",
517
-
uc_pmu->sccl_id, uc_pmu->ccl_id, uc_pmu->sub_id);
541
+
name = devm_kasprintf(&pdev->dev, GFP_KERNEL, "hisi_sccl%d_uc%d_%d",
542
+
uc_pmu->topo.sccl_id, uc_pmu->topo.ccl_id,
543
+
uc_pmu->topo.sub_id);
518
544
if (!name)
519
545
return -ENOMEM;
520
546
···
589
613
}
590
614
module_exit(hisi_uc_pmu_module_exit);
591
615
616
+
MODULE_IMPORT_NS("HISI_PMU");
592
617
MODULE_DESCRIPTION("HiSilicon SoC UC uncore PMU driver");
593
618
MODULE_LICENSE("GPL");
594
619
MODULE_AUTHOR("Junhao He <hejunhao3@huawei.com>");
+458
-70
drivers/perf/marvell_cn10k_ddr_pmu.c
+458
-70
drivers/perf/marvell_cn10k_ddr_pmu.c
···
1
1
// SPDX-License-Identifier: GPL-2.0
2
-
/* Marvell CN10K DRAM Subsystem (DSS) Performance Monitor Driver
2
+
/*
3
+
* Marvell CN10K DRAM Subsystem (DSS) Performance Monitor Driver
3
4
*
4
-
* Copyright (C) 2021 Marvell.
5
+
* Copyright (C) 2021-2024 Marvell.
5
6
*/
6
7
7
8
#include <linux/init.h>
···
15
14
#include <linux/platform_device.h>
16
15
17
16
/* Performance Counters Operating Mode Control Registers */
18
-
#define DDRC_PERF_CNT_OP_MODE_CTRL 0x8020
19
-
#define OP_MODE_CTRL_VAL_MANNUAL 0x1
17
+
#define CN10K_DDRC_PERF_CNT_OP_MODE_CTRL 0x8020
18
+
#define ODY_DDRC_PERF_CNT_OP_MODE_CTRL 0x20020
19
+
#define OP_MODE_CTRL_VAL_MANUAL 0x1
20
20
21
21
/* Performance Counters Start Operation Control Registers */
22
-
#define DDRC_PERF_CNT_START_OP_CTRL 0x8028
22
+
#define CN10K_DDRC_PERF_CNT_START_OP_CTRL 0x8028
23
+
#define ODY_DDRC_PERF_CNT_START_OP_CTRL 0x200A0
23
24
#define START_OP_CTRL_VAL_START 0x1ULL
24
25
#define START_OP_CTRL_VAL_ACTIVE 0x2
25
26
26
27
/* Performance Counters End Operation Control Registers */
27
-
#define DDRC_PERF_CNT_END_OP_CTRL 0x8030
28
+
#define CN10K_DDRC_PERF_CNT_END_OP_CTRL 0x8030
29
+
#define ODY_DDRC_PERF_CNT_END_OP_CTRL 0x200E0
28
30
#define END_OP_CTRL_VAL_END 0x1ULL
29
31
30
32
/* Performance Counters End Status Registers */
31
-
#define DDRC_PERF_CNT_END_STATUS 0x8038
33
+
#define CN10K_DDRC_PERF_CNT_END_STATUS 0x8038
34
+
#define ODY_DDRC_PERF_CNT_END_STATUS 0x20120
32
35
#define END_STATUS_VAL_END_TIMER_MODE_END 0x1
33
36
34
37
/* Performance Counters Configuration Registers */
35
-
#define DDRC_PERF_CFG_BASE 0x8040
38
+
#define CN10K_DDRC_PERF_CFG_BASE 0x8040
39
+
#define ODY_DDRC_PERF_CFG_BASE 0x20160
36
40
37
41
/* 8 Generic event counter + 2 fixed event counters */
38
42
#define DDRC_PERF_NUM_GEN_COUNTERS 8
···
48
42
DDRC_PERF_NUM_FIX_COUNTERS)
49
43
50
44
/* Generic event counter registers */
51
-
#define DDRC_PERF_CFG(n) (DDRC_PERF_CFG_BASE + 8 * (n))
45
+
#define DDRC_PERF_CFG(base, n) ((base) + 8 * (n))
52
46
#define EVENT_ENABLE BIT_ULL(63)
53
47
54
48
/* Two dedicated event counters for DDR reads and writes */
55
49
#define EVENT_DDR_READS 101
56
50
#define EVENT_DDR_WRITES 100
57
51
52
+
#define DDRC_PERF_REG(base, n) ((base) + 8 * (n))
58
53
/*
59
54
* programmable events IDs in programmable event counters.
60
55
* DO NOT change these event-id numbers, they are used to
61
56
* program event bitmap in h/w.
62
57
*/
58
+
#define EVENT_DFI_CMD_IS_RETRY 61
59
+
#define EVENT_RD_UC_ECC_ERROR 60
60
+
#define EVENT_RD_CRC_ERROR 59
61
+
#define EVENT_CAPAR_ERROR 58
62
+
#define EVENT_WR_CRC_ERROR 57
63
+
#define EVENT_DFI_PARITY_POISON 56
64
+
#define EVENT_RETRY_FIFO_FULL 46
65
+
#define EVENT_DFI_CYCLES 45
66
+
63
67
#define EVENT_OP_IS_ZQLATCH 55
64
68
#define EVENT_OP_IS_ZQSTART 54
65
69
#define EVENT_OP_IS_TCR_MRR 53
···
118
102
#define EVENT_HIF_RD_OR_WR 1
119
103
120
104
/* Event counter value registers */
121
-
#define DDRC_PERF_CNT_VALUE_BASE 0x8080
122
-
#define DDRC_PERF_CNT_VALUE(n) (DDRC_PERF_CNT_VALUE_BASE + 8 * (n))
105
+
#define CN10K_DDRC_PERF_CNT_VALUE_BASE 0x8080
106
+
#define ODY_DDRC_PERF_CNT_VALUE_BASE 0x201C0
123
107
124
108
/* Fixed event counter enable/disable register */
125
-
#define DDRC_PERF_CNT_FREERUN_EN 0x80C0
109
+
#define CN10K_DDRC_PERF_CNT_FREERUN_EN 0x80C0
126
110
#define DDRC_PERF_FREERUN_WRITE_EN 0x1
127
111
#define DDRC_PERF_FREERUN_READ_EN 0x2
128
112
129
113
/* Fixed event counter control register */
130
-
#define DDRC_PERF_CNT_FREERUN_CTRL 0x80C8
114
+
#define CN10K_DDRC_PERF_CNT_FREERUN_CTRL 0x80C8
115
+
#define ODY_DDRC_PERF_CNT_FREERUN_CTRL 0x20240
131
116
#define DDRC_FREERUN_WRITE_CNT_CLR 0x1
132
117
#define DDRC_FREERUN_READ_CNT_CLR 0x2
133
118
134
-
/* Fixed event counter value register */
135
-
#define DDRC_PERF_CNT_VALUE_WR_OP 0x80D0
136
-
#define DDRC_PERF_CNT_VALUE_RD_OP 0x80D8
119
+
/* Fixed event counter clear register, defined only for Odyssey */
120
+
#define ODY_DDRC_PERF_CNT_FREERUN_CLR 0x20248
121
+
137
122
#define DDRC_PERF_CNT_VALUE_OVERFLOW BIT_ULL(48)
138
123
#define DDRC_PERF_CNT_MAX_VALUE GENMASK_ULL(48, 0)
124
+
125
+
/* Fixed event counter value register */
126
+
#define CN10K_DDRC_PERF_CNT_VALUE_WR_OP 0x80D0
127
+
#define CN10K_DDRC_PERF_CNT_VALUE_RD_OP 0x80D8
128
+
#define ODY_DDRC_PERF_CNT_VALUE_WR_OP 0x20250
129
+
#define ODY_DDRC_PERF_CNT_VALUE_RD_OP 0x20258
139
130
140
131
struct cn10k_ddr_pmu {
141
132
struct pmu pmu;
142
133
void __iomem *base;
134
+
const struct ddr_pmu_platform_data *p_data;
135
+
const struct ddr_pmu_ops *ops;
143
136
unsigned int cpu;
144
137
struct device *dev;
145
138
int active_events;
···
157
132
struct hlist_node node;
158
133
};
159
134
135
+
struct ddr_pmu_ops {
136
+
void (*enable_read_freerun_counter)(struct cn10k_ddr_pmu *pmu,
137
+
bool enable);
138
+
void (*enable_write_freerun_counter)(struct cn10k_ddr_pmu *pmu,
139
+
bool enable);
140
+
void (*clear_read_freerun_counter)(struct cn10k_ddr_pmu *pmu);
141
+
void (*clear_write_freerun_counter)(struct cn10k_ddr_pmu *pmu);
142
+
void (*pmu_overflow_handler)(struct cn10k_ddr_pmu *pmu, int evt_idx);
143
+
};
144
+
160
145
#define to_cn10k_ddr_pmu(p) container_of(p, struct cn10k_ddr_pmu, pmu)
146
+
147
+
struct ddr_pmu_platform_data {
148
+
u64 counter_overflow_val;
149
+
u64 counter_max_val;
150
+
u64 cnt_base;
151
+
u64 cfg_base;
152
+
u64 cnt_op_mode_ctrl;
153
+
u64 cnt_start_op_ctrl;
154
+
u64 cnt_end_op_ctrl;
155
+
u64 cnt_end_status;
156
+
u64 cnt_freerun_en;
157
+
u64 cnt_freerun_ctrl;
158
+
u64 cnt_freerun_clr;
159
+
u64 cnt_value_wr_op;
160
+
u64 cnt_value_rd_op;
161
+
bool is_cn10k;
162
+
bool is_ody;
163
+
};
161
164
162
165
static ssize_t cn10k_ddr_pmu_event_show(struct device *dev,
163
166
struct device_attribute *attr,
···
262
209
NULL
263
210
};
264
211
212
+
static struct attribute *odyssey_ddr_perf_events_attrs[] = {
213
+
/* Programmable */
214
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_hif_rd_or_wr_access, EVENT_HIF_RD_OR_WR),
215
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_hif_wr_access, EVENT_HIF_WR),
216
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_hif_rd_access, EVENT_HIF_RD),
217
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_hif_rmw_access, EVENT_HIF_RMW),
218
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_hif_pri_rdaccess, EVENT_HIF_HI_PRI_RD),
219
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_rd_bypass_access, EVENT_READ_BYPASS),
220
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_act_bypass_access, EVENT_ACT_BYPASS),
221
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_dfi_wr_data_access,
222
+
EVENT_DFI_WR_DATA_CYCLES),
223
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_dfi_rd_data_access,
224
+
EVENT_DFI_RD_DATA_CYCLES),
225
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_hpri_sched_rd_crit_access,
226
+
EVENT_HPR_XACT_WHEN_CRITICAL),
227
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_lpri_sched_rd_crit_access,
228
+
EVENT_LPR_XACT_WHEN_CRITICAL),
229
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_wr_trxn_crit_access,
230
+
EVENT_WR_XACT_WHEN_CRITICAL),
231
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_cam_active_access, EVENT_OP_IS_ACTIVATE),
232
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_cam_rd_or_wr_access,
233
+
EVENT_OP_IS_RD_OR_WR),
234
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_cam_rd_active_access,
235
+
EVENT_OP_IS_RD_ACTIVATE),
236
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_cam_read, EVENT_OP_IS_RD),
237
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_cam_write, EVENT_OP_IS_WR),
238
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_cam_mwr, EVENT_OP_IS_MWR),
239
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_precharge, EVENT_OP_IS_PRECHARGE),
240
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_precharge_for_rdwr,
241
+
EVENT_PRECHARGE_FOR_RDWR),
242
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_precharge_for_other,
243
+
EVENT_PRECHARGE_FOR_OTHER),
244
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_rdwr_transitions, EVENT_RDWR_TRANSITIONS),
245
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_write_combine, EVENT_WRITE_COMBINE),
246
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_war_hazard, EVENT_WAR_HAZARD),
247
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_raw_hazard, EVENT_RAW_HAZARD),
248
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_waw_hazard, EVENT_WAW_HAZARD),
249
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_enter_selfref, EVENT_OP_IS_ENTER_SELFREF),
250
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_enter_powerdown,
251
+
EVENT_OP_IS_ENTER_POWERDOWN),
252
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_enter_mpsm, EVENT_OP_IS_ENTER_MPSM),
253
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_refresh, EVENT_OP_IS_REFRESH),
254
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_crit_ref, EVENT_OP_IS_CRIT_REF),
255
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_spec_ref, EVENT_OP_IS_SPEC_REF),
256
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_load_mode, EVENT_OP_IS_LOAD_MODE),
257
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_zqcl, EVENT_OP_IS_ZQCL),
258
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_cam_wr_access, EVENT_OP_IS_ZQCS),
259
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_dfi_cycles, EVENT_DFI_CYCLES),
260
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_retry_fifo_full,
261
+
EVENT_RETRY_FIFO_FULL),
262
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_bsm_alloc, EVENT_BSM_ALLOC),
263
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_bsm_starvation, EVENT_BSM_STARVATION),
264
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_win_limit_reached_rd,
265
+
EVENT_VISIBLE_WIN_LIMIT_REACHED_RD),
266
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_win_limit_reached_wr,
267
+
EVENT_VISIBLE_WIN_LIMIT_REACHED_WR),
268
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_dqsosc_mpc, EVENT_OP_IS_DQSOSC_MPC),
269
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_dqsosc_mrr, EVENT_OP_IS_DQSOSC_MRR),
270
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_tcr_mrr, EVENT_OP_IS_TCR_MRR),
271
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_zqstart, EVENT_OP_IS_ZQSTART),
272
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_zqlatch, EVENT_OP_IS_ZQLATCH),
273
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_dfi_parity_poison,
274
+
EVENT_DFI_PARITY_POISON),
275
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_wr_crc_error, EVENT_WR_CRC_ERROR),
276
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_capar_error, EVENT_CAPAR_ERROR),
277
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_rd_crc_error, EVENT_RD_CRC_ERROR),
278
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_rd_uc_ecc_error, EVENT_RD_UC_ECC_ERROR),
279
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_dfi_cmd_is_retry, EVENT_DFI_CMD_IS_RETRY),
280
+
/* Free run event counters */
281
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_ddr_reads, EVENT_DDR_READS),
282
+
CN10K_DDR_PMU_EVENT_ATTR(ddr_ddr_writes, EVENT_DDR_WRITES),
283
+
NULL
284
+
};
285
+
286
+
static struct attribute_group odyssey_ddr_perf_events_attr_group = {
287
+
.name = "events",
288
+
.attrs = odyssey_ddr_perf_events_attrs,
289
+
};
290
+
265
291
static struct attribute_group cn10k_ddr_perf_events_attr_group = {
266
292
.name = "events",
267
293
.attrs = cn10k_ddr_perf_events_attrs,
···
386
254
NULL,
387
255
};
388
256
257
+
static const struct attribute_group *odyssey_attr_groups[] = {
258
+
&odyssey_ddr_perf_events_attr_group,
259
+
&cn10k_ddr_perf_format_attr_group,
260
+
&cn10k_ddr_perf_cpumask_attr_group,
261
+
NULL
262
+
};
263
+
389
264
/* Default poll timeout is 100 sec, which is very sufficient for
390
265
* 48 bit counter incremented max at 5.6 GT/s, which may take many
391
266
* hours to overflow.
···
405
266
return ms_to_ktime((u64)cn10k_ddr_pmu_poll_period_sec * USEC_PER_SEC);
406
267
}
407
268
408
-
static int ddr_perf_get_event_bitmap(int eventid, u64 *event_bitmap)
269
+
static int ddr_perf_get_event_bitmap(int eventid, u64 *event_bitmap,
270
+
struct cn10k_ddr_pmu *ddr_pmu)
409
271
{
272
+
int err = 0;
273
+
410
274
switch (eventid) {
275
+
case EVENT_DFI_PARITY_POISON ...EVENT_DFI_CMD_IS_RETRY:
276
+
if (!ddr_pmu->p_data->is_ody) {
277
+
err = -EINVAL;
278
+
break;
279
+
}
280
+
fallthrough;
411
281
case EVENT_HIF_RD_OR_WR ... EVENT_WAW_HAZARD:
412
282
case EVENT_OP_IS_REFRESH ... EVENT_OP_IS_ZQLATCH:
413
283
*event_bitmap = (1ULL << (eventid - 1));
···
427
279
*event_bitmap = (0xFULL << (eventid - 1));
428
280
break;
429
281
default:
430
-
pr_err("%s Invalid eventid %d\n", __func__, eventid);
431
-
return -EINVAL;
282
+
err = -EINVAL;
432
283
}
433
284
434
-
return 0;
285
+
if (err)
286
+
pr_err("%s Invalid eventid %d\n", __func__, eventid);
287
+
return err;
435
288
}
436
289
437
290
static int cn10k_ddr_perf_alloc_counter(struct cn10k_ddr_pmu *pmu,
···
500
351
return 0;
501
352
}
502
353
354
+
static void cn10k_ddr_perf_counter_start(struct cn10k_ddr_pmu *ddr_pmu,
355
+
int counter)
356
+
{
357
+
const struct ddr_pmu_platform_data *p_data = ddr_pmu->p_data;
358
+
u64 ctrl_reg = p_data->cnt_start_op_ctrl;
359
+
360
+
writeq_relaxed(START_OP_CTRL_VAL_START, ddr_pmu->base +
361
+
DDRC_PERF_REG(ctrl_reg, counter));
362
+
}
363
+
364
+
static void cn10k_ddr_perf_counter_stop(struct cn10k_ddr_pmu *ddr_pmu,
365
+
int counter)
366
+
{
367
+
const struct ddr_pmu_platform_data *p_data = ddr_pmu->p_data;
368
+
u64 ctrl_reg = p_data->cnt_end_op_ctrl;
369
+
370
+
writeq_relaxed(END_OP_CTRL_VAL_END, ddr_pmu->base +
371
+
DDRC_PERF_REG(ctrl_reg, counter));
372
+
}
373
+
503
374
static void cn10k_ddr_perf_counter_enable(struct cn10k_ddr_pmu *pmu,
504
375
int counter, bool enable)
505
376
{
377
+
const struct ddr_pmu_platform_data *p_data = pmu->p_data;
378
+
u64 ctrl_reg = pmu->p_data->cnt_op_mode_ctrl;
379
+
const struct ddr_pmu_ops *ops = pmu->ops;
380
+
bool is_ody = pmu->p_data->is_ody;
506
381
u32 reg;
507
382
u64 val;
508
383
···
536
363
}
537
364
538
365
if (counter < DDRC_PERF_NUM_GEN_COUNTERS) {
539
-
reg = DDRC_PERF_CFG(counter);
366
+
reg = DDRC_PERF_CFG(p_data->cfg_base, counter);
540
367
val = readq_relaxed(pmu->base + reg);
541
368
542
369
if (enable)
···
545
372
val &= ~EVENT_ENABLE;
546
373
547
374
writeq_relaxed(val, pmu->base + reg);
548
-
} else {
549
-
val = readq_relaxed(pmu->base + DDRC_PERF_CNT_FREERUN_EN);
550
-
if (enable) {
551
-
if (counter == DDRC_PERF_READ_COUNTER_IDX)
552
-
val |= DDRC_PERF_FREERUN_READ_EN;
553
-
else
554
-
val |= DDRC_PERF_FREERUN_WRITE_EN;
555
-
} else {
556
-
if (counter == DDRC_PERF_READ_COUNTER_IDX)
557
-
val &= ~DDRC_PERF_FREERUN_READ_EN;
558
-
else
559
-
val &= ~DDRC_PERF_FREERUN_WRITE_EN;
375
+
376
+
if (is_ody) {
377
+
if (enable) {
378
+
/*
379
+
* Setup the PMU counter to work in
380
+
* manual mode
381
+
*/
382
+
reg = DDRC_PERF_REG(ctrl_reg, counter);
383
+
writeq_relaxed(OP_MODE_CTRL_VAL_MANUAL,
384
+
pmu->base + reg);
385
+
386
+
cn10k_ddr_perf_counter_start(pmu, counter);
387
+
} else {
388
+
cn10k_ddr_perf_counter_stop(pmu, counter);
389
+
}
560
390
}
561
-
writeq_relaxed(val, pmu->base + DDRC_PERF_CNT_FREERUN_EN);
391
+
} else {
392
+
if (counter == DDRC_PERF_READ_COUNTER_IDX)
393
+
ops->enable_read_freerun_counter(pmu, enable);
394
+
else
395
+
ops->enable_write_freerun_counter(pmu, enable);
562
396
}
563
397
}
564
398
565
399
static u64 cn10k_ddr_perf_read_counter(struct cn10k_ddr_pmu *pmu, int counter)
566
400
{
401
+
const struct ddr_pmu_platform_data *p_data = pmu->p_data;
567
402
u64 val;
568
403
569
404
if (counter == DDRC_PERF_READ_COUNTER_IDX)
570
-
return readq_relaxed(pmu->base + DDRC_PERF_CNT_VALUE_RD_OP);
405
+
return readq_relaxed(pmu->base +
406
+
p_data->cnt_value_rd_op);
571
407
572
408
if (counter == DDRC_PERF_WRITE_COUNTER_IDX)
573
-
return readq_relaxed(pmu->base + DDRC_PERF_CNT_VALUE_WR_OP);
409
+
return readq_relaxed(pmu->base +
410
+
p_data->cnt_value_wr_op);
574
411
575
-
val = readq_relaxed(pmu->base + DDRC_PERF_CNT_VALUE(counter));
412
+
val = readq_relaxed(pmu->base +
413
+
DDRC_PERF_REG(p_data->cnt_base, counter));
576
414
return val;
577
415
}
578
416
579
417
static void cn10k_ddr_perf_event_update(struct perf_event *event)
580
418
{
581
419
struct cn10k_ddr_pmu *pmu = to_cn10k_ddr_pmu(event->pmu);
420
+
const struct ddr_pmu_platform_data *p_data = pmu->p_data;
582
421
struct hw_perf_event *hwc = &event->hw;
583
422
u64 prev_count, new_count, mask;
584
423
···
599
414
new_count = cn10k_ddr_perf_read_counter(pmu, hwc->idx);
600
415
} while (local64_xchg(&hwc->prev_count, new_count) != prev_count);
601
416
602
-
mask = DDRC_PERF_CNT_MAX_VALUE;
417
+
mask = p_data->counter_max_val;
603
418
604
419
local64_add((new_count - prev_count) & mask, &event->count);
605
420
}
···
620
435
static int cn10k_ddr_perf_event_add(struct perf_event *event, int flags)
621
436
{
622
437
struct cn10k_ddr_pmu *pmu = to_cn10k_ddr_pmu(event->pmu);
438
+
const struct ddr_pmu_platform_data *p_data = pmu->p_data;
439
+
const struct ddr_pmu_ops *ops = pmu->ops;
623
440
struct hw_perf_event *hwc = &event->hw;
624
441
u8 config = event->attr.config;
625
442
int counter, ret;
···
641
454
642
455
if (counter < DDRC_PERF_NUM_GEN_COUNTERS) {
643
456
/* Generic counters, configure event id */
644
-
reg_offset = DDRC_PERF_CFG(counter);
645
-
ret = ddr_perf_get_event_bitmap(config, &val);
457
+
reg_offset = DDRC_PERF_CFG(p_data->cfg_base, counter);
458
+
ret = ddr_perf_get_event_bitmap(config, &val, pmu);
646
459
if (ret)
647
460
return ret;
648
461
···
650
463
} else {
651
464
/* fixed event counter, clear counter value */
652
465
if (counter == DDRC_PERF_READ_COUNTER_IDX)
653
-
val = DDRC_FREERUN_READ_CNT_CLR;
466
+
ops->clear_read_freerun_counter(pmu);
654
467
else
655
-
val = DDRC_FREERUN_WRITE_CNT_CLR;
656
-
657
-
writeq_relaxed(val, pmu->base + DDRC_PERF_CNT_FREERUN_CTRL);
468
+
ops->clear_write_freerun_counter(pmu);
658
469
}
659
470
660
471
hwc->state |= PERF_HES_STOPPED;
···
697
512
static void cn10k_ddr_perf_pmu_enable(struct pmu *pmu)
698
513
{
699
514
struct cn10k_ddr_pmu *ddr_pmu = to_cn10k_ddr_pmu(pmu);
515
+
const struct ddr_pmu_platform_data *p_data = ddr_pmu->p_data;
700
516
701
517
writeq_relaxed(START_OP_CTRL_VAL_START, ddr_pmu->base +
702
-
DDRC_PERF_CNT_START_OP_CTRL);
518
+
p_data->cnt_start_op_ctrl);
703
519
}
704
520
705
521
static void cn10k_ddr_perf_pmu_disable(struct pmu *pmu)
706
522
{
707
523
struct cn10k_ddr_pmu *ddr_pmu = to_cn10k_ddr_pmu(pmu);
524
+
const struct ddr_pmu_platform_data *p_data = ddr_pmu->p_data;
708
525
709
526
writeq_relaxed(END_OP_CTRL_VAL_END, ddr_pmu->base +
710
-
DDRC_PERF_CNT_END_OP_CTRL);
527
+
p_data->cnt_end_op_ctrl);
711
528
}
712
529
713
530
static void cn10k_ddr_perf_event_update_all(struct cn10k_ddr_pmu *pmu)
···
734
547
}
735
548
}
736
549
550
+
static void ddr_pmu_enable_read_freerun(struct cn10k_ddr_pmu *pmu, bool enable)
551
+
{
552
+
const struct ddr_pmu_platform_data *p_data = pmu->p_data;
553
+
u64 val;
554
+
555
+
val = readq_relaxed(pmu->base + p_data->cnt_freerun_en);
556
+
if (enable)
557
+
val |= DDRC_PERF_FREERUN_READ_EN;
558
+
else
559
+
val &= ~DDRC_PERF_FREERUN_READ_EN;
560
+
561
+
writeq_relaxed(val, pmu->base + p_data->cnt_freerun_en);
562
+
}
563
+
564
+
static void ddr_pmu_enable_write_freerun(struct cn10k_ddr_pmu *pmu, bool enable)
565
+
{
566
+
const struct ddr_pmu_platform_data *p_data = pmu->p_data;
567
+
u64 val;
568
+
569
+
val = readq_relaxed(pmu->base + p_data->cnt_freerun_en);
570
+
if (enable)
571
+
val |= DDRC_PERF_FREERUN_WRITE_EN;
572
+
else
573
+
val &= ~DDRC_PERF_FREERUN_WRITE_EN;
574
+
575
+
writeq_relaxed(val, pmu->base + p_data->cnt_freerun_en);
576
+
}
577
+
578
+
static void ddr_pmu_read_clear_freerun(struct cn10k_ddr_pmu *pmu)
579
+
{
580
+
const struct ddr_pmu_platform_data *p_data = pmu->p_data;
581
+
u64 val;
582
+
583
+
val = DDRC_FREERUN_READ_CNT_CLR;
584
+
writeq_relaxed(val, pmu->base + p_data->cnt_freerun_ctrl);
585
+
}
586
+
587
+
static void ddr_pmu_write_clear_freerun(struct cn10k_ddr_pmu *pmu)
588
+
{
589
+
const struct ddr_pmu_platform_data *p_data = pmu->p_data;
590
+
u64 val;
591
+
592
+
val = DDRC_FREERUN_WRITE_CNT_CLR;
593
+
writeq_relaxed(val, pmu->base + p_data->cnt_freerun_ctrl);
594
+
}
595
+
596
+
static void ddr_pmu_overflow_hander(struct cn10k_ddr_pmu *pmu, int evt_idx)
597
+
{
598
+
cn10k_ddr_perf_event_update_all(pmu);
599
+
cn10k_ddr_perf_pmu_disable(&pmu->pmu);
600
+
cn10k_ddr_perf_pmu_enable(&pmu->pmu);
601
+
}
602
+
603
+
static void ddr_pmu_ody_enable_read_freerun(struct cn10k_ddr_pmu *pmu,
604
+
bool enable)
605
+
{
606
+
const struct ddr_pmu_platform_data *p_data = pmu->p_data;
607
+
u64 val;
608
+
609
+
val = readq_relaxed(pmu->base + p_data->cnt_freerun_ctrl);
610
+
if (enable)
611
+
val |= DDRC_PERF_FREERUN_READ_EN;
612
+
else
613
+
val &= ~DDRC_PERF_FREERUN_READ_EN;
614
+
615
+
writeq_relaxed(val, pmu->base + p_data->cnt_freerun_ctrl);
616
+
}
617
+
618
+
static void ddr_pmu_ody_enable_write_freerun(struct cn10k_ddr_pmu *pmu,
619
+
bool enable)
620
+
{
621
+
const struct ddr_pmu_platform_data *p_data = pmu->p_data;
622
+
u64 val;
623
+
624
+
val = readq_relaxed(pmu->base + p_data->cnt_freerun_ctrl);
625
+
if (enable)
626
+
val |= DDRC_PERF_FREERUN_WRITE_EN;
627
+
else
628
+
val &= ~DDRC_PERF_FREERUN_WRITE_EN;
629
+
630
+
writeq_relaxed(val, pmu->base + p_data->cnt_freerun_ctrl);
631
+
}
632
+
633
+
static void ddr_pmu_ody_read_clear_freerun(struct cn10k_ddr_pmu *pmu)
634
+
{
635
+
const struct ddr_pmu_platform_data *p_data = pmu->p_data;
636
+
u64 val;
637
+
638
+
val = DDRC_FREERUN_READ_CNT_CLR;
639
+
writeq_relaxed(val, pmu->base + p_data->cnt_freerun_clr);
640
+
}
641
+
642
+
static void ddr_pmu_ody_write_clear_freerun(struct cn10k_ddr_pmu *pmu)
643
+
{
644
+
const struct ddr_pmu_platform_data *p_data = pmu->p_data;
645
+
u64 val;
646
+
647
+
val = DDRC_FREERUN_WRITE_CNT_CLR;
648
+
writeq_relaxed(val, pmu->base + p_data->cnt_freerun_clr);
649
+
}
650
+
651
+
static void ddr_pmu_ody_overflow_hander(struct cn10k_ddr_pmu *pmu, int evt_idx)
652
+
{
653
+
/*
654
+
* On reaching the maximum value of the counter, the counter freezes
655
+
* there. The particular event is updated and the respective counter
656
+
* is stopped and started again so that it starts counting from zero
657
+
*/
658
+
cn10k_ddr_perf_event_update(pmu->events[evt_idx]);
659
+
cn10k_ddr_perf_counter_stop(pmu, evt_idx);
660
+
cn10k_ddr_perf_counter_start(pmu, evt_idx);
661
+
}
662
+
737
663
static irqreturn_t cn10k_ddr_pmu_overflow_handler(struct cn10k_ddr_pmu *pmu)
738
664
{
665
+
const struct ddr_pmu_platform_data *p_data = pmu->p_data;
666
+
const struct ddr_pmu_ops *ops = pmu->ops;
739
667
struct perf_event *event;
740
668
struct hw_perf_event *hwc;
741
669
u64 prev_count, new_count;
···
888
586
continue;
889
587
890
588
value = cn10k_ddr_perf_read_counter(pmu, i);
891
-
if (value == DDRC_PERF_CNT_MAX_VALUE) {
589
+
if (value == p_data->counter_max_val) {
892
590
pr_info("Counter-(%d) reached max value\n", i);
893
-
cn10k_ddr_perf_event_update_all(pmu);
894
-
cn10k_ddr_perf_pmu_disable(&pmu->pmu);
895
-
cn10k_ddr_perf_pmu_enable(&pmu->pmu);
591
+
ops->pmu_overflow_handler(pmu, i);
896
592
}
897
593
}
898
594
···
929
629
return 0;
930
630
}
931
631
632
+
static const struct ddr_pmu_ops ddr_pmu_ops = {
633
+
.enable_read_freerun_counter = ddr_pmu_enable_read_freerun,
634
+
.enable_write_freerun_counter = ddr_pmu_enable_write_freerun,
635
+
.clear_read_freerun_counter = ddr_pmu_read_clear_freerun,
636
+
.clear_write_freerun_counter = ddr_pmu_write_clear_freerun,
637
+
.pmu_overflow_handler = ddr_pmu_overflow_hander,
638
+
};
639
+
640
+
#if defined(CONFIG_ACPI) || defined(CONFIG_OF)
641
+
static const struct ddr_pmu_platform_data cn10k_ddr_pmu_pdata = {
642
+
.counter_overflow_val = BIT_ULL(48),
643
+
.counter_max_val = GENMASK_ULL(48, 0),
644
+
.cnt_base = CN10K_DDRC_PERF_CNT_VALUE_BASE,
645
+
.cfg_base = CN10K_DDRC_PERF_CFG_BASE,
646
+
.cnt_op_mode_ctrl = CN10K_DDRC_PERF_CNT_OP_MODE_CTRL,
647
+
.cnt_start_op_ctrl = CN10K_DDRC_PERF_CNT_START_OP_CTRL,
648
+
.cnt_end_op_ctrl = CN10K_DDRC_PERF_CNT_END_OP_CTRL,
649
+
.cnt_end_status = CN10K_DDRC_PERF_CNT_END_STATUS,
650
+
.cnt_freerun_en = CN10K_DDRC_PERF_CNT_FREERUN_EN,
651
+
.cnt_freerun_ctrl = CN10K_DDRC_PERF_CNT_FREERUN_CTRL,
652
+
.cnt_freerun_clr = 0,
653
+
.cnt_value_wr_op = CN10K_DDRC_PERF_CNT_VALUE_WR_OP,
654
+
.cnt_value_rd_op = CN10K_DDRC_PERF_CNT_VALUE_RD_OP,
655
+
.is_cn10k = TRUE,
656
+
};
657
+
#endif
658
+
659
+
static const struct ddr_pmu_ops ddr_pmu_ody_ops = {
660
+
.enable_read_freerun_counter = ddr_pmu_ody_enable_read_freerun,
661
+
.enable_write_freerun_counter = ddr_pmu_ody_enable_write_freerun,
662
+
.clear_read_freerun_counter = ddr_pmu_ody_read_clear_freerun,
663
+
.clear_write_freerun_counter = ddr_pmu_ody_write_clear_freerun,
664
+
.pmu_overflow_handler = ddr_pmu_ody_overflow_hander,
665
+
};
666
+
667
+
#ifdef CONFIG_ACPI
668
+
static const struct ddr_pmu_platform_data odyssey_ddr_pmu_pdata = {
669
+
.counter_overflow_val = 0,
670
+
.counter_max_val = GENMASK_ULL(63, 0),
671
+
.cnt_base = ODY_DDRC_PERF_CNT_VALUE_BASE,
672
+
.cfg_base = ODY_DDRC_PERF_CFG_BASE,
673
+
.cnt_op_mode_ctrl = ODY_DDRC_PERF_CNT_OP_MODE_CTRL,
674
+
.cnt_start_op_ctrl = ODY_DDRC_PERF_CNT_START_OP_CTRL,
675
+
.cnt_end_op_ctrl = ODY_DDRC_PERF_CNT_END_OP_CTRL,
676
+
.cnt_end_status = ODY_DDRC_PERF_CNT_END_STATUS,
677
+
.cnt_freerun_en = 0,
678
+
.cnt_freerun_ctrl = ODY_DDRC_PERF_CNT_FREERUN_CTRL,
679
+
.cnt_freerun_clr = ODY_DDRC_PERF_CNT_FREERUN_CLR,
680
+
.cnt_value_wr_op = ODY_DDRC_PERF_CNT_VALUE_WR_OP,
681
+
.cnt_value_rd_op = ODY_DDRC_PERF_CNT_VALUE_RD_OP,
682
+
.is_ody = TRUE,
683
+
};
684
+
#endif
685
+
932
686
static int cn10k_ddr_perf_probe(struct platform_device *pdev)
933
687
{
688
+
const struct ddr_pmu_platform_data *dev_data;
934
689
struct cn10k_ddr_pmu *ddr_pmu;
935
690
struct resource *res;
936
691
void __iomem *base;
692
+
bool is_cn10k;
693
+
bool is_ody;
937
694
char *name;
938
695
int ret;
939
696
···
1001
644
ddr_pmu->dev = &pdev->dev;
1002
645
platform_set_drvdata(pdev, ddr_pmu);
1003
646
647
+
dev_data = device_get_match_data(&pdev->dev);
648
+
if (!dev_data) {
649
+
dev_err(&pdev->dev, "Error: No device match data found\n");
650
+
return -ENODEV;
651
+
}
652
+
1004
653
base = devm_platform_get_and_ioremap_resource(pdev, 0, &res);
1005
654
if (IS_ERR(base))
1006
655
return PTR_ERR(base);
1007
656
1008
657
ddr_pmu->base = base;
1009
658
1010
-
/* Setup the PMU counter to work in manual mode */
1011
-
writeq_relaxed(OP_MODE_CTRL_VAL_MANNUAL, ddr_pmu->base +
1012
-
DDRC_PERF_CNT_OP_MODE_CTRL);
659
+
ddr_pmu->p_data = dev_data;
660
+
is_cn10k = ddr_pmu->p_data->is_cn10k;
661
+
is_ody = ddr_pmu->p_data->is_ody;
1013
662
1014
-
ddr_pmu->pmu = (struct pmu) {
1015
-
.module = THIS_MODULE,
1016
-
.capabilities = PERF_PMU_CAP_NO_EXCLUDE,
1017
-
.task_ctx_nr = perf_invalid_context,
1018
-
.attr_groups = cn10k_attr_groups,
1019
-
.event_init = cn10k_ddr_perf_event_init,
1020
-
.add = cn10k_ddr_perf_event_add,
1021
-
.del = cn10k_ddr_perf_event_del,
1022
-
.start = cn10k_ddr_perf_event_start,
1023
-
.stop = cn10k_ddr_perf_event_stop,
1024
-
.read = cn10k_ddr_perf_event_update,
1025
-
.pmu_enable = cn10k_ddr_perf_pmu_enable,
1026
-
.pmu_disable = cn10k_ddr_perf_pmu_disable,
1027
-
};
663
+
if (is_cn10k) {
664
+
ddr_pmu->ops = &ddr_pmu_ops;
665
+
/* Setup the PMU counter to work in manual mode */
666
+
writeq_relaxed(OP_MODE_CTRL_VAL_MANUAL, ddr_pmu->base +
667
+
ddr_pmu->p_data->cnt_op_mode_ctrl);
668
+
669
+
ddr_pmu->pmu = (struct pmu) {
670
+
.module = THIS_MODULE,
671
+
.capabilities = PERF_PMU_CAP_NO_EXCLUDE,
672
+
.task_ctx_nr = perf_invalid_context,
673
+
.attr_groups = cn10k_attr_groups,
674
+
.event_init = cn10k_ddr_perf_event_init,
675
+
.add = cn10k_ddr_perf_event_add,
676
+
.del = cn10k_ddr_perf_event_del,
677
+
.start = cn10k_ddr_perf_event_start,
678
+
.stop = cn10k_ddr_perf_event_stop,
679
+
.read = cn10k_ddr_perf_event_update,
680
+
.pmu_enable = cn10k_ddr_perf_pmu_enable,
681
+
.pmu_disable = cn10k_ddr_perf_pmu_disable,
682
+
};
683
+
}
684
+
685
+
if (is_ody) {
686
+
ddr_pmu->ops = &ddr_pmu_ody_ops;
687
+
688
+
ddr_pmu->pmu = (struct pmu) {
689
+
.module = THIS_MODULE,
690
+
.capabilities = PERF_PMU_CAP_NO_EXCLUDE,
691
+
.task_ctx_nr = perf_invalid_context,
692
+
.attr_groups = odyssey_attr_groups,
693
+
.event_init = cn10k_ddr_perf_event_init,
694
+
.add = cn10k_ddr_perf_event_add,
695
+
.del = cn10k_ddr_perf_event_del,
696
+
.start = cn10k_ddr_perf_event_start,
697
+
.stop = cn10k_ddr_perf_event_stop,
698
+
.read = cn10k_ddr_perf_event_update,
699
+
};
700
+
}
1028
701
1029
702
/* Choose this cpu to collect perf data */
1030
703
ddr_pmu->cpu = raw_smp_processor_id();
···
1075
688
if (ret)
1076
689
goto error;
1077
690
1078
-
pr_info("CN10K DDR PMU Driver for ddrc@%llx\n", res->start);
691
+
pr_info("DDR PMU Driver for ddrc@%llx\n", res->start);
1079
692
return 0;
1080
693
error:
1081
694
cpuhp_state_remove_instance_nocalls(
···
1097
710
1098
711
#ifdef CONFIG_OF
1099
712
static const struct of_device_id cn10k_ddr_pmu_of_match[] = {
1100
-
{ .compatible = "marvell,cn10k-ddr-pmu", },
713
+
{ .compatible = "marvell,cn10k-ddr-pmu", .data = &cn10k_ddr_pmu_pdata },
1101
714
{ },
1102
715
};
1103
716
MODULE_DEVICE_TABLE(of, cn10k_ddr_pmu_of_match);
···
1105
718
1106
719
#ifdef CONFIG_ACPI
1107
720
static const struct acpi_device_id cn10k_ddr_pmu_acpi_match[] = {
1108
-
{"MRVL000A", 0},
721
+
{"MRVL000A", (kernel_ulong_t)&cn10k_ddr_pmu_pdata },
722
+
{"MRVL000C", (kernel_ulong_t)&odyssey_ddr_pmu_pdata},
1109
723
{},
1110
724
};
1111
725
MODULE_DEVICE_TABLE(acpi, cn10k_ddr_pmu_acpi_match);
+63
-3
drivers/perf/marvell_cn10k_tad_pmu.c
+63
-3
drivers/perf/marvell_cn10k_tad_pmu.c
···
37
37
DECLARE_BITMAP(counters_map, TAD_MAX_COUNTERS);
38
38
};
39
39
40
+
enum mrvl_tad_pmu_version {
41
+
TAD_PMU_V1 = 1,
42
+
TAD_PMU_V2,
43
+
};
44
+
45
+
struct tad_pmu_data {
46
+
int id;
47
+
};
48
+
40
49
static int tad_pmu_cpuhp_state;
41
50
42
51
static void tad_pmu_event_counter_read(struct perf_event *event)
···
223
214
.attrs = tad_pmu_event_attrs,
224
215
};
225
216
217
+
static struct attribute *ody_tad_pmu_event_attrs[] = {
218
+
TAD_PMU_EVENT_ATTR(tad_req_msh_in_exlmn, 0x3),
219
+
TAD_PMU_EVENT_ATTR(tad_alloc_dtg, 0x1a),
220
+
TAD_PMU_EVENT_ATTR(tad_alloc_ltg, 0x1b),
221
+
TAD_PMU_EVENT_ATTR(tad_alloc_any, 0x1c),
222
+
TAD_PMU_EVENT_ATTR(tad_hit_dtg, 0x1d),
223
+
TAD_PMU_EVENT_ATTR(tad_hit_ltg, 0x1e),
224
+
TAD_PMU_EVENT_ATTR(tad_hit_any, 0x1f),
225
+
TAD_PMU_EVENT_ATTR(tad_tag_rd, 0x20),
226
+
TAD_PMU_EVENT_ATTR(tad_tot_cycle, 0xFF),
227
+
NULL
228
+
};
229
+
230
+
static const struct attribute_group ody_tad_pmu_events_attr_group = {
231
+
.name = "events",
232
+
.attrs = ody_tad_pmu_event_attrs,
233
+
};
234
+
226
235
PMU_FORMAT_ATTR(event, "config:0-7");
227
236
228
237
static struct attribute *tad_pmu_format_attrs[] = {
···
279
252
NULL
280
253
};
281
254
255
+
static const struct attribute_group *ody_tad_pmu_attr_groups[] = {
256
+
&ody_tad_pmu_events_attr_group,
257
+
&tad_pmu_format_attr_group,
258
+
&tad_pmu_cpumask_attr_group,
259
+
NULL
260
+
};
261
+
282
262
static int tad_pmu_probe(struct platform_device *pdev)
283
263
{
264
+
const struct tad_pmu_data *dev_data;
284
265
struct device *dev = &pdev->dev;
285
266
struct tad_region *regions;
286
267
struct tad_pmu *tad_pmu;
···
296
261
u32 tad_pmu_page_size;
297
262
u32 tad_page_size;
298
263
u32 tad_cnt;
264
+
int version;
299
265
int i, ret;
300
266
char *name;
301
267
···
305
269
return -ENOMEM;
306
270
307
271
platform_set_drvdata(pdev, tad_pmu);
272
+
273
+
dev_data = device_get_match_data(&pdev->dev);
274
+
if (!dev_data) {
275
+
dev_err(&pdev->dev, "Error: No device match data found\n");
276
+
return -ENODEV;
277
+
}
278
+
version = dev_data->id;
308
279
309
280
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
310
281
if (!res) {
···
362
319
tad_pmu->pmu = (struct pmu) {
363
320
364
321
.module = THIS_MODULE,
365
-
.attr_groups = tad_pmu_attr_groups,
366
322
.capabilities = PERF_PMU_CAP_NO_EXCLUDE |
367
323
PERF_PMU_CAP_NO_INTERRUPT,
368
324
.task_ctx_nr = perf_invalid_context,
···
373
331
.stop = tad_pmu_event_counter_stop,
374
332
.read = tad_pmu_event_counter_read,
375
333
};
334
+
335
+
if (version == TAD_PMU_V1)
336
+
tad_pmu->pmu.attr_groups = tad_pmu_attr_groups;
337
+
else
338
+
tad_pmu->pmu.attr_groups = ody_tad_pmu_attr_groups;
376
339
377
340
tad_pmu->cpu = raw_smp_processor_id();
378
341
···
407
360
perf_pmu_unregister(&pmu->pmu);
408
361
}
409
362
363
+
#if defined(CONFIG_OF) || defined(CONFIG_ACPI)
364
+
static const struct tad_pmu_data tad_pmu_data = {
365
+
.id = TAD_PMU_V1,
366
+
};
367
+
#endif
368
+
369
+
#ifdef CONFIG_ACPI
370
+
static const struct tad_pmu_data tad_pmu_v2_data = {
371
+
.id = TAD_PMU_V2,
372
+
};
373
+
#endif
374
+
410
375
#ifdef CONFIG_OF
411
376
static const struct of_device_id tad_pmu_of_match[] = {
412
-
{ .compatible = "marvell,cn10k-tad-pmu", },
377
+
{ .compatible = "marvell,cn10k-tad-pmu", .data = &tad_pmu_data },
413
378
{},
414
379
};
415
380
#endif
416
381
417
382
#ifdef CONFIG_ACPI
418
383
static const struct acpi_device_id tad_pmu_acpi_match[] = {
419
-
{"MRVL000B", 0},
384
+
{"MRVL000B", (kernel_ulong_t)&tad_pmu_data},
385
+
{"MRVL000D", (kernel_ulong_t)&tad_pmu_v2_data},
420
386
{},
421
387
};
422
388
MODULE_DEVICE_TABLE(acpi, tad_pmu_acpi_match);
+8
drivers/virt/coco/arm-cca-guest/arm-cca-guest.c
+8
drivers/virt/coco/arm-cca-guest/arm-cca-guest.c
···
6
6
#include <linux/arm-smccc.h>
7
7
#include <linux/cc_platform.h>
8
8
#include <linux/kernel.h>
9
+
#include <linux/mod_devicetable.h>
9
10
#include <linux/module.h>
10
11
#include <linux/smp.h>
11
12
#include <linux/tsm.h>
···
220
219
}
221
220
module_exit(arm_cca_guest_exit);
222
221
222
+
/* modalias, so userspace can autoload this module when RSI is available */
223
+
static const struct platform_device_id arm_cca_match[] __maybe_unused = {
224
+
{ RSI_PDEV_NAME, 0},
225
+
{ }
226
+
};
227
+
228
+
MODULE_DEVICE_TABLE(platform, arm_cca_match);
223
229
MODULE_AUTHOR("Sami Mujawar <sami.mujawar@arm.com>");
224
230
MODULE_DESCRIPTION("Arm CCA Guest TSM Driver");
225
231
MODULE_LICENSE("GPL");
+26
tools/perf/Documentation/perf-arm-spe.txt
+26
tools/perf/Documentation/perf-arm-spe.txt
···
150
150
pct_enable=1 - collect physical timestamp instead of virtual timestamp (PMSCR.PCT) - requires privilege
151
151
store_filter=1 - collect stores only (PMSFCR.ST)
152
152
ts_enable=1 - enable timestamping with value of generic timer (PMSCR.TS)
153
+
discard=1 - enable SPE PMU events but don't collect sample data - see 'Discard mode' (PMBLIMITR.FM = DISCARD)
153
154
154
155
+++*+++ Latency is the total latency from the point at which sampling started on that instruction, rather
155
156
than only the execution latency.
···
221
220
222
221
Increase sampling interval (see above)
223
222
223
+
PMU events
224
+
~~~~~~~~~~
225
+
226
+
SPE has events that can be counted on core PMUs. These are prefixed with
227
+
SAMPLE_, for example SAMPLE_POP, SAMPLE_FEED, SAMPLE_COLLISION and
228
+
SAMPLE_FEED_BR.
229
+
230
+
These events will only count when an SPE event is running on the same core that
231
+
the PMU event is opened on, otherwise they read as 0. There are various ways to
232
+
ensure that the PMU event and SPE event are scheduled together depending on the
233
+
way the event is opened. For example opening both events as per-process events
234
+
on the same process, although it's not guaranteed that the PMU event is enabled
235
+
first when context switching. For that reason it may be better to open the PMU
236
+
event as a systemwide event and then open SPE on the process of interest.
237
+
238
+
Discard mode
239
+
~~~~~~~~~~~~
240
+
241
+
SPE related (SAMPLE_* etc) core PMU events can be used without the overhead of
242
+
collecting sample data if discard mode is supported (optional from Armv8.6).
243
+
First run a system wide SPE session (or on the core of interest) using options
244
+
to minimize output. Then run perf stat:
245
+
246
+
perf record -e arm_spe/discard/ -a -N -B --no-bpf-event -o - > /dev/null &
247
+
perf stat -e SAMPLE_FEED_LD
224
248
225
249
SEE ALSO
226
250
--------
+233
-2
tools/testing/selftests/arm64/abi/hwcap.c
+233
-2
tools/testing/selftests/arm64/abi/hwcap.c
···
46
46
asm volatile(".inst 0xb82003ff" : : : );
47
47
}
48
48
49
+
static void cmpbr_sigill(void)
50
+
{
51
+
/* Not implemented, too complicated and unreliable anyway */
52
+
}
53
+
54
+
49
55
static void crc32_sigill(void)
50
56
{
51
57
/* CRC32W W0, W0, W1 */
···
88
82
asm volatile(".inst 0xec0fc00");
89
83
}
90
84
85
+
static void f8mm4_sigill(void)
86
+
{
87
+
/* FMMLA V0.4SH, V0.16B, V0.16B */
88
+
asm volatile(".inst 0x6e00ec00");
89
+
}
90
+
91
+
static void f8mm8_sigill(void)
92
+
{
93
+
/* FMMLA V0.4S, V0.16B, V0.16B */
94
+
asm volatile(".inst 0x6e80ec00");
95
+
}
96
+
91
97
static void faminmax_sigill(void)
92
98
{
93
99
/* FAMIN V0.4H, V0.4H, V0.4H */
···
114
96
static void fpmr_sigill(void)
115
97
{
116
98
asm volatile("mrs x0, S3_3_C4_C4_2" : : : "x0");
99
+
}
100
+
101
+
static void fprcvt_sigill(void)
102
+
{
103
+
/* FCVTAS S0, H0 */
104
+
asm volatile(".inst 0x1efa0000");
117
105
}
118
106
119
107
static void gcs_sigill(void)
···
250
226
asm volatile("msr S0_3_C4_C6_3, xzr" : : : );
251
227
}
252
228
229
+
static void sme2p2_sigill(void)
230
+
{
231
+
/* SMSTART SM */
232
+
asm volatile("msr S0_3_C4_C3_3, xzr" : : : );
233
+
234
+
/* UXTB Z0.D, P0/Z, Z0.D */
235
+
asm volatile(".inst 0x4c1a000" : : : );
236
+
237
+
/* SMSTOP */
238
+
asm volatile("msr S0_3_C4_C6_3, xzr" : : : );
239
+
}
240
+
241
+
static void sme_aes_sigill(void)
242
+
{
243
+
/* SMSTART SM */
244
+
asm volatile("msr S0_3_C4_C3_3, xzr" : : : );
245
+
246
+
/* AESD z0.b, z0.b, z0.b */
247
+
asm volatile(".inst 0x4522e400" : : : "z0");
248
+
249
+
/* SMSTOP */
250
+
asm volatile("msr S0_3_C4_C6_3, xzr" : : : );
251
+
}
252
+
253
+
static void sme_sbitperm_sigill(void)
254
+
{
255
+
/* SMSTART SM */
256
+
asm volatile("msr S0_3_C4_C3_3, xzr" : : : );
257
+
258
+
/* BDEP Z0.B, Z0.B, Z0.B */
259
+
asm volatile(".inst 0x4500b400" : : : "z0");
260
+
261
+
/* SMSTOP */
262
+
asm volatile("msr S0_3_C4_C6_3, xzr" : : : );
263
+
}
264
+
253
265
static void smei16i32_sigill(void)
254
266
{
255
267
/* SMSTART */
···
399
339
/* SMSTART */
400
340
asm volatile("msr S0_3_C4_C7_3, xzr" : : : );
401
341
402
-
/* FMLALB V0.8H, V0.16B, V0.16B */
403
-
asm volatile(".inst 0xec0fc00");
342
+
/* FMLALB Z0.8H, Z0.B, Z0.B */
343
+
asm volatile(".inst 0x64205000");
344
+
345
+
/* SMSTOP */
346
+
asm volatile("msr S0_3_C4_C6_3, xzr" : : : );
347
+
}
348
+
349
+
static void smesfexpa_sigill(void)
350
+
{
351
+
/* SMSTART */
352
+
asm volatile("msr S0_3_C4_C7_3, xzr" : : : );
353
+
354
+
/* FEXPA Z0.D, Z0.D */
355
+
asm volatile(".inst 0x04e0b800");
356
+
357
+
/* SMSTOP */
358
+
asm volatile("msr S0_3_C4_C6_3, xzr" : : : );
359
+
}
360
+
361
+
static void smesmop4_sigill(void)
362
+
{
363
+
/* SMSTART */
364
+
asm volatile("msr S0_3_C4_C7_3, xzr" : : : );
365
+
366
+
/* SMOP4A ZA0.S, Z0.B, { Z0.B - Z1.B } */
367
+
asm volatile(".inst 0x80108000");
368
+
369
+
/* SMSTOP */
370
+
asm volatile("msr S0_3_C4_C6_3, xzr" : : : );
371
+
}
372
+
373
+
static void smestmop_sigill(void)
374
+
{
375
+
/* SMSTART */
376
+
asm volatile("msr S0_3_C4_C7_3, xzr" : : : );
377
+
378
+
/* STMOPA ZA0.S, { Z0.H - Z1.H }, Z0.H, Z20[0] */
379
+
asm volatile(".inst 0x80408008");
404
380
405
381
/* SMSTOP */
406
382
asm volatile("msr S0_3_C4_C6_3, xzr" : : : );
···
460
364
asm volatile(".inst 0x65000000" : : : "z0");
461
365
}
462
366
367
+
static void sve2p2_sigill(void)
368
+
{
369
+
/* NOT Z0.D, P0/Z, Z0.D */
370
+
asm volatile(".inst 0x4cea000" : : : "z0");
371
+
}
372
+
463
373
static void sveaes_sigill(void)
464
374
{
465
375
/* AESD z0.b, z0.b, z0.b */
466
376
asm volatile(".inst 0x4522e400" : : : "z0");
467
377
}
468
378
379
+
static void sveaes2_sigill(void)
380
+
{
381
+
/* AESD {Z0.B - Z1.B }, { Z0.B - Z1.B }, Z0.Q */
382
+
asm volatile(".inst 0x4522ec00" : : : "z0");
383
+
}
384
+
469
385
static void sveb16b16_sigill(void)
470
386
{
471
387
/* BFADD Z0.H, Z0.H, Z0.H */
472
388
asm volatile(".inst 0x65000000" : : : );
389
+
}
390
+
391
+
static void svebfscale_sigill(void)
392
+
{
393
+
/* BFSCALE Z0.H, P0/M, Z0.H, Z0.H */
394
+
asm volatile(".inst 0x65098000" : : : "z0");
395
+
}
396
+
397
+
static void svef16mm_sigill(void)
398
+
{
399
+
/* FMMLA Z0.S, Z0.H, Z0.H */
400
+
asm volatile(".inst 0x6420e400");
473
401
}
474
402
475
403
static void svepmull_sigill(void)
···
512
392
{
513
393
/* EOR3 Z0.D, Z0.D, Z0.D, Z0.D */
514
394
asm volatile(".inst 0x4203800" : : : "z0");
395
+
}
396
+
397
+
static void sveeltperm_sigill(void)
398
+
{
399
+
/* COMPACT Z0.B, P0, Z0.B */
400
+
asm volatile(".inst 0x5218000" : : : "x0");
515
401
}
516
402
517
403
static void svesm4_sigill(void)
···
596
470
.sigill_fn = aes_sigill,
597
471
},
598
472
{
473
+
.name = "CMPBR",
474
+
.at_hwcap = AT_HWCAP,
475
+
.hwcap_bit = HWCAP_CMPBR,
476
+
.cpuinfo = "cmpbr",
477
+
.sigill_fn = cmpbr_sigill,
478
+
},
479
+
{
599
480
.name = "CRC32",
600
481
.at_hwcap = AT_HWCAP,
601
482
.hwcap_bit = HWCAP_CRC32,
···
657
524
.sigill_fn = f8fma_sigill,
658
525
},
659
526
{
527
+
.name = "F8MM8",
528
+
.at_hwcap = AT_HWCAP,
529
+
.hwcap_bit = HWCAP_F8MM8,
530
+
.cpuinfo = "f8mm8",
531
+
.sigill_fn = f8mm8_sigill,
532
+
},
533
+
{
534
+
.name = "F8MM4",
535
+
.at_hwcap = AT_HWCAP,
536
+
.hwcap_bit = HWCAP_F8MM4,
537
+
.cpuinfo = "f8mm4",
538
+
.sigill_fn = f8mm4_sigill,
539
+
},
540
+
{
660
541
.name = "FAMINMAX",
661
542
.at_hwcap = AT_HWCAP2,
662
543
.hwcap_bit = HWCAP2_FAMINMAX,
···
691
544
.cpuinfo = "fpmr",
692
545
.sigill_fn = fpmr_sigill,
693
546
.sigill_reliable = true,
547
+
},
548
+
{
549
+
.name = "FPRCVT",
550
+
.at_hwcap = AT_HWCAP,
551
+
.hwcap_bit = HWCAP_FPRCVT,
552
+
.cpuinfo = "fprcvt",
553
+
.sigill_fn = fprcvt_sigill,
694
554
},
695
555
{
696
556
.name = "GCS",
···
846
692
.sigill_fn = sme2p1_sigill,
847
693
},
848
694
{
695
+
.name = "SME 2.2",
696
+
.at_hwcap = AT_HWCAP,
697
+
.hwcap_bit = HWCAP_SME2P2,
698
+
.cpuinfo = "sme2p2",
699
+
.sigill_fn = sme2p2_sigill,
700
+
},
701
+
{
702
+
.name = "SME AES",
703
+
.at_hwcap = AT_HWCAP,
704
+
.hwcap_bit = HWCAP_SME_AES,
705
+
.cpuinfo = "smeaes",
706
+
.sigill_fn = sme_aes_sigill,
707
+
},
708
+
{
849
709
.name = "SME I16I32",
850
710
.at_hwcap = AT_HWCAP2,
851
711
.hwcap_bit = HWCAP2_SME_I16I32,
···
909
741
.sigill_fn = smelutv2_sigill,
910
742
},
911
743
{
744
+
.name = "SME SBITPERM",
745
+
.at_hwcap = AT_HWCAP,
746
+
.hwcap_bit = HWCAP_SME_SBITPERM,
747
+
.cpuinfo = "smesbitperm",
748
+
.sigill_fn = sme_sbitperm_sigill,
749
+
},
750
+
{
912
751
.name = "SME SF8FMA",
913
752
.at_hwcap = AT_HWCAP2,
914
753
.hwcap_bit = HWCAP2_SME_SF8FMA,
···
935
760
.hwcap_bit = HWCAP2_SME_SF8DP4,
936
761
.cpuinfo = "smesf8dp4",
937
762
.sigill_fn = smesf8dp4_sigill,
763
+
},
764
+
{
765
+
.name = "SME SFEXPA",
766
+
.at_hwcap = AT_HWCAP,
767
+
.hwcap_bit = HWCAP_SME_SFEXPA,
768
+
.cpuinfo = "smesfexpa",
769
+
.sigill_fn = smesfexpa_sigill,
770
+
},
771
+
{
772
+
.name = "SME SMOP4",
773
+
.at_hwcap = AT_HWCAP,
774
+
.hwcap_bit = HWCAP_SME_SMOP4,
775
+
.cpuinfo = "smesmop4",
776
+
.sigill_fn = smesmop4_sigill,
777
+
},
778
+
{
779
+
.name = "SME STMOP",
780
+
.at_hwcap = AT_HWCAP,
781
+
.hwcap_bit = HWCAP_SME_STMOP,
782
+
.cpuinfo = "smestmop",
783
+
.sigill_fn = smestmop_sigill,
938
784
},
939
785
{
940
786
.name = "SVE",
···
980
784
.sigill_fn = sve2p1_sigill,
981
785
},
982
786
{
787
+
.name = "SVE 2.2",
788
+
.at_hwcap = AT_HWCAP,
789
+
.hwcap_bit = HWCAP_SVE2P2,
790
+
.cpuinfo = "sve2p2",
791
+
.sigill_fn = sve2p2_sigill,
792
+
},
793
+
{
983
794
.name = "SVE AES",
984
795
.at_hwcap = AT_HWCAP2,
985
796
.hwcap_bit = HWCAP2_SVEAES,
986
797
.cpuinfo = "sveaes",
987
798
.sigill_fn = sveaes_sigill,
799
+
},
800
+
{
801
+
.name = "SVE AES2",
802
+
.at_hwcap = AT_HWCAP,
803
+
.hwcap_bit = HWCAP_SVE_AES2,
804
+
.cpuinfo = "sveaes2",
805
+
.sigill_fn = sveaes2_sigill,
806
+
},
807
+
{
808
+
.name = "SVE BFSCALE",
809
+
.at_hwcap = AT_HWCAP,
810
+
.hwcap_bit = HWCAP_SVE_BFSCALE,
811
+
.cpuinfo = "svebfscale",
812
+
.sigill_fn = svebfscale_sigill,
813
+
},
814
+
{
815
+
.name = "SVE ELTPERM",
816
+
.at_hwcap = AT_HWCAP,
817
+
.hwcap_bit = HWCAP_SVE_ELTPERM,
818
+
.cpuinfo = "sveeltperm",
819
+
.sigill_fn = sveeltperm_sigill,
820
+
},
821
+
{
822
+
.name = "SVE F16MM",
823
+
.at_hwcap = AT_HWCAP,
824
+
.hwcap_bit = HWCAP_SVE_F16MM,
825
+
.cpuinfo = "svef16mm",
826
+
.sigill_fn = svef16mm_sigill,
988
827
},
989
828
{
990
829
.name = "SVE2 B16B16",