+53 -26

Documentation/ABI/testing/sysfs-bus-cxl

reviewed

··· 90 90 capability. 91 91 92 92 93 93 + What: /sys/bus/cxl/devices/{port,endpoint}X/parent_dport 94 94 + Date: January, 2023 95 95 + KernelVersion: v6.3 96 96 + Contact: linux-cxl@vger.kernel.org 97 97 + Description: 98 98 + (RO) CXL port objects are instantiated for each upstream port in 99 99 + a CXL/PCIe switch, and for each endpoint to map the 100 100 + corresponding memory device into the CXL port hierarchy. When a 101 101 + descendant CXL port (switch or endpoint) is enumerated it is 102 102 + useful to know which 'dport' object in the parent CXL port 103 103 + routes to this descendant. The 'parent_dport' symlink points to 104 104 + the device representing the downstream port of a CXL switch that 105 105 + routes to {port,endpoint}X. 106 106 + 107 107 + 93 108 What: /sys/bus/cxl/devices/portX/dportY 94 109 Date: June, 2021 95 110 KernelVersion: v5.14 ··· 198 183 199 184 What: /sys/bus/cxl/devices/endpointX/CDAT 200 185 Date: July, 2022 201 201 - KernelVersion: v5.20 186 186 + KernelVersion: v6.0 202 187 Contact: linux-cxl@vger.kernel.org 203 188 Description: 204 189 (RO) If this sysfs entry is not present no DOE mailbox was ··· 209 194 210 195 What: /sys/bus/cxl/devices/decoderX.Y/mode 211 196 Date: May, 2022 212 212 - KernelVersion: v5.20 197 197 + KernelVersion: v6.0 213 198 Contact: linux-cxl@vger.kernel.org 214 199 Description: 215 200 (RW) When a CXL decoder is of devtype "cxl_decoder_endpoint" it ··· 229 214 230 215 What: /sys/bus/cxl/devices/decoderX.Y/dpa_resource 231 216 Date: May, 2022 232 232 - KernelVersion: v5.20 217 217 + KernelVersion: v6.0 233 218 Contact: linux-cxl@vger.kernel.org 234 219 Description: 235 220 (RO) When a CXL decoder is of devtype "cxl_decoder_endpoint", ··· 240 225 241 226 What: /sys/bus/cxl/devices/decoderX.Y/dpa_size 242 227 Date: May, 2022 243 243 - KernelVersion: v5.20 228 228 + KernelVersion: v6.0 244 229 Contact: linux-cxl@vger.kernel.org 245 230 Description: 246 231 (RW) When a CXL decoder is of devtype "cxl_decoder_endpoint" it ··· 260 245 261 246 What: /sys/bus/cxl/devices/decoderX.Y/interleave_ways 262 247 Date: May, 2022 263 263 - KernelVersion: v5.20 248 248 + KernelVersion: v6.0 264 249 Contact: linux-cxl@vger.kernel.org 265 250 Description: 266 251 (RO) The number of targets across which this decoder's host ··· 275 260 276 261 What: /sys/bus/cxl/devices/decoderX.Y/interleave_granularity 277 262 Date: May, 2022 278 278 - KernelVersion: v5.20 263 263 + KernelVersion: v6.0 279 264 Contact: linux-cxl@vger.kernel.org 280 265 Description: 281 266 (RO) The number of consecutive bytes of host physical address ··· 285 270 interleave_granularity). 286 271 287 272 288 288 - What: /sys/bus/cxl/devices/decoderX.Y/create_pmem_region 289 289 - Date: May, 2022 290 290 - KernelVersion: v5.20 273 273 + What: /sys/bus/cxl/devices/decoderX.Y/create_{pmem,ram}_region 274 274 + Date: May, 2022, January, 2023 275 275 + KernelVersion: v6.0 (pmem), v6.3 (ram) 291 276 Contact: linux-cxl@vger.kernel.org 292 277 Description: 293 278 (RW) Write a string in the form 'regionZ' to start the process 294 294 - of defining a new persistent memory region (interleave-set) 295 295 - within the decode range bounded by root decoder 'decoderX.Y'. 296 296 - The value written must match the current value returned from 297 297 - reading this attribute. An atomic compare exchange operation is 298 298 - done on write to assign the requested id to a region and 299 299 - allocate the region-id for the next creation attempt. EBUSY is 300 300 - returned if the region name written does not match the current 301 301 - cached value. 279 279 + of defining a new persistent, or volatile memory region 280 280 + (interleave-set) within the decode range bounded by root decoder 281 281 + 'decoderX.Y'. The value written must match the current value 282 282 + returned from reading this attribute. An atomic compare exchange 283 283 + operation is done on write to assign the requested id to a 284 284 + region and allocate the region-id for the next creation attempt. 285 285 + EBUSY is returned if the region name written does not match the 286 286 + current cached value. 302 287 303 288 304 289 What: /sys/bus/cxl/devices/decoderX.Y/delete_region 305 290 Date: May, 2022 306 306 - KernelVersion: v5.20 291 291 + KernelVersion: v6.0 307 292 Contact: linux-cxl@vger.kernel.org 308 293 Description: 309 294 (WO) Write a string in the form 'regionZ' to delete that region, ··· 312 297 313 298 What: /sys/bus/cxl/devices/regionZ/uuid 314 299 Date: May, 2022 315 315 - KernelVersion: v5.20 300 300 + KernelVersion: v6.0 316 301 Contact: linux-cxl@vger.kernel.org 317 302 Description: 318 303 (RW) Write a unique identifier for the region. This field must 319 304 be set for persistent regions and it must not conflict with the 320 320 - UUID of another region. 305 305 + UUID of another region. For volatile ram regions this 306 306 + attribute is a read-only empty string. 321 307 322 308 323 309 What: /sys/bus/cxl/devices/regionZ/interleave_granularity 324 310 Date: May, 2022 325 325 - KernelVersion: v5.20 311 311 + KernelVersion: v6.0 326 312 Contact: linux-cxl@vger.kernel.org 327 313 Description: 328 314 (RW) Set the number of consecutive bytes each device in the ··· 334 318 335 319 What: /sys/bus/cxl/devices/regionZ/interleave_ways 336 320 Date: May, 2022 337 337 - KernelVersion: v5.20 321 321 + KernelVersion: v6.0 338 322 Contact: linux-cxl@vger.kernel.org 339 323 Description: 340 324 (RW) Configures the number of devices participating in the ··· 344 328 345 329 What: /sys/bus/cxl/devices/regionZ/size 346 330 Date: May, 2022 347 347 - KernelVersion: v5.20 331 331 + KernelVersion: v6.0 348 332 Contact: linux-cxl@vger.kernel.org 349 333 Description: 350 334 (RW) System physical address space to be consumed by the region. ··· 359 343 results in the same address being allocated. 360 344 361 345 346 346 + What: /sys/bus/cxl/devices/regionZ/mode 347 347 + Date: January, 2023 348 348 + KernelVersion: v6.3 349 349 + Contact: linux-cxl@vger.kernel.org 350 350 + Description: 351 351 + (RO) The mode of a region is established at region creation time 352 352 + and dictates the mode of the endpoint decoder that comprise the 353 353 + region. For more details on the possible modes see 354 354 + /sys/bus/cxl/devices/decoderX.Y/mode 355 355 + 356 356 + 362 357 What: /sys/bus/cxl/devices/regionZ/resource 363 358 Date: May, 2022 364 364 - KernelVersion: v5.20 359 359 + KernelVersion: v6.0 365 360 Contact: linux-cxl@vger.kernel.org 366 361 Description: 367 362 (RO) A region is a contiguous partition of a CXL root decoder ··· 384 357 385 358 What: /sys/bus/cxl/devices/regionZ/target[0..N] 386 359 Date: May, 2022 387 387 - KernelVersion: v5.20 360 360 + KernelVersion: v6.0 388 361 Contact: linux-cxl@vger.kernel.org 389 362 Description: 390 363 (RW) Write an endpoint decoder object name to 'targetX' where X ··· 403 376 404 377 What: /sys/bus/cxl/devices/regionZ/commit 405 378 Date: May, 2022 406 406 - KernelVersion: v5.20 379 379 + KernelVersion: v6.0 407 380 Contact: linux-cxl@vger.kernel.org 408 381 Description: 409 382 (RW) Write a boolean 'true' string value to this attribute to

+1

MAINTAINERS

reviewed

··· 5912 5912 M: Vishal Verma <vishal.l.verma@intel.com> 5913 5913 M: Dave Jiang <dave.jiang@intel.com> 5914 5914 L: nvdimm@lists.linux.dev 5915 5915 + L: linux-cxl@vger.kernel.org 5915 5916 S: Supported 5916 5917 F: drivers/dax/ 5917 5918

+1 -1

drivers/Makefile

reviewed

··· 71 71 obj-$(CONFIG_PARPORT) += parport/ 72 72 obj-y += base/ block/ misc/ mfd/ nfc/ 73 73 obj-$(CONFIG_LIBNVDIMM) += nvdimm/ 74 74 - obj-$(CONFIG_DAX) += dax/ 74 74 + obj-y += dax/ 75 75 obj-$(CONFIG_DMA_SHARED_BUFFER) += dma-buf/ 76 76 obj-$(CONFIG_NUBUS) += nubus/ 77 77 obj-y += cxl/

+2 -2

drivers/acpi/numa/hmat.c

reviewed

··· 718 718 for (res = target->memregions.child; res; res = res->sibling) { 719 719 int target_nid = pxm_to_node(target->memory_pxm); 720 720 721 721 - hmem_register_device(target_nid, res); 721 721 + hmem_register_resource(target_nid, res); 722 722 } 723 723 } 724 724 ··· 869 869 acpi_put_table(tbl); 870 870 return 0; 871 871 } 872 872 - device_initcall(hmat_init); 872 872 + subsys_initcall(hmat_init);

+3

drivers/acpi/pci_root.c

reviewed

··· 1047 1047 if (!(root->osc_control_set & OSC_PCI_EXPRESS_DPC_CONTROL)) 1048 1048 host_bridge->native_dpc = 0; 1049 1049 1050 1050 + if (!(root->osc_ext_control_set & OSC_CXL_ERROR_REPORTING_CONTROL)) 1051 1051 + host_bridge->native_cxl_error = 0; 1052 1052 + 1050 1053 /* 1051 1054 * Evaluate the "PCI Boot Configuration" _DSM Function. If it 1052 1055 * exists and returns 0, we must preserve any PCI resource

+12 -2

drivers/cxl/Kconfig

reviewed

··· 104 104 depends on SUSPEND && CXL_MEM 105 105 106 106 config CXL_REGION 107 107 - bool 107 107 + bool "CXL: Region Support" 108 108 default CXL_BUS 109 109 # For MAX_PHYSMEM_BITS 110 110 depends on SPARSEMEM 111 111 select MEMREGION 112 112 select GET_FREE_REGION 113 113 + help 114 114 + Enable the CXL core to enumerate and provision CXL regions. A CXL 115 115 + region is defined by one or more CXL expanders that decode a given 116 116 + system-physical address range. For CXL regions established by 117 117 + platform-firmware this option enables memory error handling to 118 118 + identify the devices participating in a given interleaved memory 119 119 + range. Otherwise, platform-firmware managed CXL is enabled by being 120 120 + placed in the system address map and does not need a driver. 121 121 + 122 122 + If unsure say 'y' 113 123 114 124 config CXL_REGION_INVALIDATION_TEST 115 125 bool "CXL: Region Cache Management Bypass (TEST)" 116 126 depends on CXL_REGION 117 127 help 118 128 CXL Region management and security operations potentially invalidate 119 119 - the content of CPU caches without notifiying those caches to 129 129 + the content of CPU caches without notifying those caches to 120 130 invalidate the affected cachelines. The CXL Region driver attempts 121 131 to invalidate caches when those events occur. If that invalidation 122 132 fails the region will fail to enable. Reasons for cache

+3 -2

drivers/cxl/acpi.c

reviewed

··· 19 19 20 20 /* 21 21 * Find a targets entry (n) in the host bridge interleave list. 22 22 - * CXL Specfication 3.0 Table 9-22 22 22 + * CXL Specification 3.0 Table 9-22 23 23 */ 24 24 static int cxl_xor_calc_n(u64 hpa, struct cxl_cxims_data *cximsd, int iw, 25 25 int ig) ··· 731 731 cxl_bus_drain(); 732 732 } 733 733 734 734 - module_init(cxl_acpi_init); 734 734 + /* load before dax_hmem sees 'Soft Reserved' CXL ranges */ 735 735 + subsys_initcall(cxl_acpi_init); 735 736 module_exit(cxl_acpi_exit); 736 737 MODULE_LICENSE("GPL v2"); 737 738 MODULE_IMPORT_NS(CXL);

+3

drivers/cxl/core/Makefile

reviewed

··· 3 3 obj-$(CONFIG_CXL_SUSPEND) += suspend.o 4 4 5 5 ccflags-y += -I$(srctree)/drivers/cxl 6 6 + CFLAGS_trace.o = -DTRACE_INCLUDE_PATH=. -I$(src) 7 7 + 6 8 cxl_core-y := port.o 7 9 cxl_core-y += pmem.o 8 10 cxl_core-y += regs.o ··· 12 10 cxl_core-y += mbox.o 13 11 cxl_core-y += pci.o 14 12 cxl_core-y += hdm.o 13 13 + cxl_core-$(CONFIG_TRACING) += trace.o 15 14 cxl_core-$(CONFIG_CXL_REGION) += region.o

+4 -3

drivers/cxl/core/core.h

reviewed

··· 11 11 12 12 #ifdef CONFIG_CXL_REGION 13 13 extern struct device_attribute dev_attr_create_pmem_region; 14 14 + extern struct device_attribute dev_attr_create_ram_region; 14 15 extern struct device_attribute dev_attr_delete_region; 15 16 extern struct device_attribute dev_attr_region; 16 17 extern const struct device_type cxl_pmem_region_type; 18 18 + extern const struct device_type cxl_dax_region_type; 17 19 extern const struct device_type cxl_region_type; 18 20 void cxl_decoder_kill_region(struct cxl_endpoint_decoder *cxled); 19 21 #define CXL_REGION_ATTR(x) (&dev_attr_##x.attr) 20 22 #define CXL_REGION_TYPE(x) (&cxl_region_type) 21 23 #define SET_CXL_REGION_ATTR(x) (&dev_attr_##x.attr), 22 24 #define CXL_PMEM_REGION_TYPE(x) (&cxl_pmem_region_type) 25 25 + #define CXL_DAX_REGION_TYPE(x) (&cxl_dax_region_type) 23 26 int cxl_region_init(void); 24 27 void cxl_region_exit(void); 25 28 #else ··· 40 37 #define CXL_REGION_TYPE(x) NULL 41 38 #define SET_CXL_REGION_ATTR(x) 42 39 #define CXL_PMEM_REGION_TYPE(x) NULL 40 40 + #define CXL_DAX_REGION_TYPE(x) NULL 43 41 #endif 44 42 45 43 struct cxl_send_command; ··· 59 55 resource_size_t cxl_dpa_size(struct cxl_endpoint_decoder *cxled); 60 56 resource_size_t cxl_dpa_resource_start(struct cxl_endpoint_decoder *cxled); 61 57 extern struct rw_semaphore cxl_dpa_rwsem; 62 62 - 63 63 - bool is_switch_decoder(struct device *dev); 64 64 - struct cxl_switch_decoder *to_cxl_switch_decoder(struct device *dev); 65 58 66 59 int cxl_memdev_init(void); 67 60 void cxl_memdev_exit(void);

+129 -15

drivers/cxl/core/hdm.c

reviewed

··· 101 101 BIT(CXL_CM_CAP_CAP_ID_HDM)); 102 102 } 103 103 104 104 + static struct cxl_hdm *devm_cxl_setup_emulated_hdm(struct cxl_port *port, 105 105 + struct cxl_endpoint_dvsec_info *info) 106 106 + { 107 107 + struct device *dev = &port->dev; 108 108 + struct cxl_hdm *cxlhdm; 109 109 + 110 110 + if (!info->mem_enabled) 111 111 + return ERR_PTR(-ENODEV); 112 112 + 113 113 + cxlhdm = devm_kzalloc(dev, sizeof(*cxlhdm), GFP_KERNEL); 114 114 + if (!cxlhdm) 115 115 + return ERR_PTR(-ENOMEM); 116 116 + 117 117 + cxlhdm->port = port; 118 118 + cxlhdm->decoder_count = info->ranges; 119 119 + cxlhdm->target_count = info->ranges; 120 120 + dev_set_drvdata(&port->dev, cxlhdm); 121 121 + 122 122 + return cxlhdm; 123 123 + } 124 124 + 104 125 /** 105 126 * devm_cxl_setup_hdm - map HDM decoder component registers 106 127 * @port: cxl_port to map 128 128 + * @info: cached DVSEC range register info 107 129 */ 108 108 - struct cxl_hdm *devm_cxl_setup_hdm(struct cxl_port *port) 130 130 + struct cxl_hdm *devm_cxl_setup_hdm(struct cxl_port *port, 131 131 + struct cxl_endpoint_dvsec_info *info) 109 132 { 110 133 struct device *dev = &port->dev; 111 134 struct cxl_hdm *cxlhdm; ··· 142 119 cxlhdm->port = port; 143 120 crb = ioremap(port->component_reg_phys, CXL_COMPONENT_REG_BLOCK_SIZE); 144 121 if (!crb) { 122 122 + if (info && info->mem_enabled) 123 123 + return devm_cxl_setup_emulated_hdm(port, info); 124 124 + 145 125 dev_err(dev, "No component registers mapped\n"); 146 126 return ERR_PTR(-ENXIO); 147 127 } ··· 305 279 return 0; 306 280 } 307 281 308 308 - static int devm_cxl_dpa_reserve(struct cxl_endpoint_decoder *cxled, 282 282 + int devm_cxl_dpa_reserve(struct cxl_endpoint_decoder *cxled, 309 283 resource_size_t base, resource_size_t len, 310 284 resource_size_t skipped) 311 285 { ··· 321 295 322 296 return devm_add_action_or_reset(&port->dev, cxl_dpa_release, cxled); 323 297 } 298 298 + EXPORT_SYMBOL_NS_GPL(devm_cxl_dpa_reserve, CXL); 324 299 325 300 resource_size_t cxl_dpa_size(struct cxl_endpoint_decoder *cxled) 326 301 { ··· 703 676 port->commit_end--; 704 677 cxld->flags &= ~CXL_DECODER_F_ENABLE; 705 678 679 679 + /* Userspace is now responsible for reconfiguring this decoder */ 680 680 + if (is_endpoint_decoder(&cxld->dev)) { 681 681 + struct cxl_endpoint_decoder *cxled; 682 682 + 683 683 + cxled = to_cxl_endpoint_decoder(&cxld->dev); 684 684 + cxled->state = CXL_DECODER_STATE_MANUAL; 685 685 + } 686 686 + 706 687 return 0; 688 688 + } 689 689 + 690 690 + static int cxl_setup_hdm_decoder_from_dvsec(struct cxl_port *port, 691 691 + struct cxl_decoder *cxld, int which, 692 692 + struct cxl_endpoint_dvsec_info *info) 693 693 + { 694 694 + if (!is_cxl_endpoint(port)) 695 695 + return -EOPNOTSUPP; 696 696 + 697 697 + if (!range_len(&info->dvsec_range[which])) 698 698 + return -ENOENT; 699 699 + 700 700 + cxld->target_type = CXL_DECODER_EXPANDER; 701 701 + cxld->commit = NULL; 702 702 + cxld->reset = NULL; 703 703 + cxld->hpa_range = info->dvsec_range[which]; 704 704 + 705 705 + /* 706 706 + * Set the emulated decoder as locked pending additional support to 707 707 + * change the range registers at run time. 708 708 + */ 709 709 + cxld->flags |= CXL_DECODER_F_ENABLE | CXL_DECODER_F_LOCK; 710 710 + port->commit_end = cxld->id; 711 711 + 712 712 + return 0; 713 713 + } 714 714 + 715 715 + static bool should_emulate_decoders(struct cxl_port *port) 716 716 + { 717 717 + struct cxl_hdm *cxlhdm = dev_get_drvdata(&port->dev); 718 718 + void __iomem *hdm = cxlhdm->regs.hdm_decoder; 719 719 + u32 ctrl; 720 720 + int i; 721 721 + 722 722 + if (!is_cxl_endpoint(cxlhdm->port)) 723 723 + return false; 724 724 + 725 725 + if (!hdm) 726 726 + return true; 727 727 + 728 728 + /* 729 729 + * If any decoders are committed already, there should not be any 730 730 + * emulated DVSEC decoders. 731 731 + */ 732 732 + for (i = 0; i < cxlhdm->decoder_count; i++) { 733 733 + ctrl = readl(hdm + CXL_HDM_DECODER0_CTRL_OFFSET(i)); 734 734 + if (FIELD_GET(CXL_HDM_DECODER0_CTRL_COMMITTED, ctrl)) 735 735 + return false; 736 736 + } 737 737 + 738 738 + return true; 707 739 } 708 740 709 741 static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, 710 742 int *target_map, void __iomem *hdm, int which, 711 711 - u64 *dpa_base) 743 743 + u64 *dpa_base, struct cxl_endpoint_dvsec_info *info) 712 744 { 713 745 struct cxl_endpoint_decoder *cxled = NULL; 714 746 u64 size, base, skip, dpa_size; ··· 779 693 u64 value; 780 694 unsigned char target_id[8]; 781 695 } target_list; 696 696 + 697 697 + if (should_emulate_decoders(port)) 698 698 + return cxl_setup_hdm_decoder_from_dvsec(port, cxld, which, info); 782 699 783 700 if (is_endpoint_decoder(&cxld->dev)) 784 701 cxled = to_cxl_endpoint_decoder(&cxld->dev); ··· 805 716 .start = base, 806 717 .end = base + size - 1, 807 718 }; 719 719 + 720 720 + if (cxled && !committed && range_len(&info->dvsec_range[which])) 721 721 + return cxl_setup_hdm_decoder_from_dvsec(port, cxld, which, info); 808 722 809 723 /* decoders are enabled if committed */ 810 724 if (committed) { ··· 875 783 return rc; 876 784 } 877 785 *dpa_base += dpa_size + skip; 786 786 + 787 787 + cxled->state = CXL_DECODER_STATE_AUTO; 788 788 + 878 789 return 0; 879 790 } 880 791 881 881 - /** 882 882 - * devm_cxl_enumerate_decoders - add decoder objects per HDM register set 883 883 - * @cxlhdm: Structure to populate with HDM capabilities 884 884 - */ 885 885 - int devm_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm) 792 792 + static void cxl_settle_decoders(struct cxl_hdm *cxlhdm) 886 793 { 887 794 void __iomem *hdm = cxlhdm->regs.hdm_decoder; 888 888 - struct cxl_port *port = cxlhdm->port; 889 889 - int i, committed; 890 890 - u64 dpa_base = 0; 795 795 + int committed, i; 891 796 u32 ctrl; 797 797 + 798 798 + if (!hdm) 799 799 + return; 892 800 893 801 /* 894 802 * Since the register resource was recently claimed via request_region() ··· 906 814 /* ensure that future checks of committed can be trusted */ 907 815 if (committed != cxlhdm->decoder_count) 908 816 msleep(20); 817 817 + } 818 818 + 819 819 + /** 820 820 + * devm_cxl_enumerate_decoders - add decoder objects per HDM register set 821 821 + * @cxlhdm: Structure to populate with HDM capabilities 822 822 + * @info: cached DVSEC range register info 823 823 + */ 824 824 + int devm_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm, 825 825 + struct cxl_endpoint_dvsec_info *info) 826 826 + { 827 827 + void __iomem *hdm = cxlhdm->regs.hdm_decoder; 828 828 + struct cxl_port *port = cxlhdm->port; 829 829 + int i; 830 830 + u64 dpa_base = 0; 831 831 + 832 832 + cxl_settle_decoders(cxlhdm); 909 833 910 834 for (i = 0; i < cxlhdm->decoder_count; i++) { 911 835 int target_map[CXL_DECODER_MAX_INTERLEAVE] = { 0 }; ··· 934 826 cxled = cxl_endpoint_decoder_alloc(port); 935 827 if (IS_ERR(cxled)) { 936 828 dev_warn(&port->dev, 937 937 - "Failed to allocate the decoder\n"); 829 829 + "Failed to allocate decoder%d.%d\n", 830 830 + port->id, i); 938 831 return PTR_ERR(cxled); 939 832 } 940 833 cxld = &cxled->cxld; ··· 945 836 cxlsd = cxl_switch_decoder_alloc(port, target_count); 946 837 if (IS_ERR(cxlsd)) { 947 838 dev_warn(&port->dev, 948 948 - "Failed to allocate the decoder\n"); 839 839 + "Failed to allocate decoder%d.%d\n", 840 840 + port->id, i); 949 841 return PTR_ERR(cxlsd); 950 842 } 951 843 cxld = &cxlsd->cxld; 952 844 } 953 845 954 954 - rc = init_hdm_decoder(port, cxld, target_map, hdm, i, &dpa_base); 846 846 + rc = init_hdm_decoder(port, cxld, target_map, hdm, i, 847 847 + &dpa_base, info); 955 848 if (rc) { 849 849 + dev_warn(&port->dev, 850 850 + "Failed to initialize decoder%d.%d\n", 851 851 + port->id, i); 956 852 put_device(&cxld->dev); 957 853 return rc; 958 854 } 959 855 rc = add_hdm_decoder(port, cxld, target_map); 960 856 if (rc) { 961 857 dev_warn(&port->dev, 962 962 - "Failed to add decoder to port\n"); 858 858 + "Failed to add decoder%d.%d\n", port->id, i); 963 859 return rc; 964 860 } 965 861 }

+257 -6

drivers/cxl/core/mbox.c

reviewed

··· 3 3 #include <linux/io-64-nonatomic-lo-hi.h> 4 4 #include <linux/security.h> 5 5 #include <linux/debugfs.h> 6 6 + #include <linux/ktime.h> 6 7 #include <linux/mutex.h> 7 8 #include <cxlmem.h> 8 9 #include <cxl.h> 9 10 10 11 #include "core.h" 12 12 + #include "trace.h" 11 13 12 14 static bool cxl_raw_allow_all; 13 15 ··· 172 170 out_size = mbox_cmd->size_out; 173 171 min_out = mbox_cmd->min_out; 174 172 rc = cxlds->mbox_send(cxlds, mbox_cmd); 173 173 + /* 174 174 + * EIO is reserved for a payload size mismatch and mbox_send() 175 175 + * may not return this error. 176 176 + */ 177 177 + if (WARN_ONCE(rc == -EIO, "Bad return code: -EIO")) 178 178 + return -ENXIO; 175 179 if (rc) 176 180 return rc; 177 181 ··· 453 445 * structures. 454 446 */ 455 447 cxl_for_each_cmd(cmd) { 456 456 - const struct cxl_command_info *info = &cmd->info; 448 448 + struct cxl_command_info info = cmd->info; 457 449 458 458 - if (copy_to_user(&q->commands[j++], info, sizeof(*info))) 450 450 + if (test_bit(info.id, cxlmd->cxlds->enabled_cmds)) 451 451 + info.flags |= CXL_MEM_COMMAND_FLAG_ENABLED; 452 452 + if (test_bit(info.id, cxlmd->cxlds->exclusive_cmds)) 453 453 + info.flags |= CXL_MEM_COMMAND_FLAG_EXCLUSIVE; 454 454 + 455 455 + if (copy_to_user(&q->commands[j++], &info, sizeof(info))) 459 456 return -EFAULT; 460 457 461 458 if (j == n_commands) ··· 563 550 return 0; 564 551 } 565 552 566 566 - static int cxl_xfer_log(struct cxl_dev_state *cxlds, uuid_t *uuid, u32 size, u8 *out) 553 553 + static int cxl_xfer_log(struct cxl_dev_state *cxlds, uuid_t *uuid, u32 *size, u8 *out) 567 554 { 568 568 - u32 remaining = size; 555 555 + u32 remaining = *size; 569 556 u32 offset = 0; 570 557 571 558 while (remaining) { ··· 589 576 }; 590 577 591 578 rc = cxl_internal_send_cmd(cxlds, &mbox_cmd); 579 579 + 580 580 + /* 581 581 + * The output payload length that indicates the number 582 582 + * of valid bytes can be smaller than the Log buffer 583 583 + * size. 584 584 + */ 585 585 + if (rc == -EIO && mbox_cmd.size_out < xfer_size) { 586 586 + offset += mbox_cmd.size_out; 587 587 + break; 588 588 + } 589 589 + 592 590 if (rc < 0) 593 591 return rc; 594 592 ··· 607 583 remaining -= xfer_size; 608 584 offset += xfer_size; 609 585 } 586 586 + 587 587 + *size = offset; 610 588 611 589 return 0; 612 590 } ··· 636 610 637 611 if (!cmd) { 638 612 dev_dbg(cxlds->dev, 639 639 - "Opcode 0x%04x unsupported by driver", opcode); 613 613 + "Opcode 0x%04x unsupported by driver\n", opcode); 640 614 continue; 641 615 } 642 616 643 617 set_bit(cmd->info.id, cxlds->enabled_cmds); 618 618 + dev_dbg(cxlds->dev, "Opcode 0x%04x enabled\n", opcode); 644 619 } 645 620 } 646 621 ··· 721 694 goto out; 722 695 } 723 696 724 724 - rc = cxl_xfer_log(cxlds, &uuid, size, log); 697 697 + rc = cxl_xfer_log(cxlds, &uuid, &size, log); 725 698 if (rc) { 726 699 kvfree(log); 727 700 goto out; ··· 743 716 return rc; 744 717 } 745 718 EXPORT_SYMBOL_NS_GPL(cxl_enumerate_cmds, CXL); 719 719 + 720 720 + /* 721 721 + * General Media Event Record 722 722 + * CXL rev 3.0 Section 8.2.9.2.1.1; Table 8-43 723 723 + */ 724 724 + static const uuid_t gen_media_event_uuid = 725 725 + UUID_INIT(0xfbcd0a77, 0xc260, 0x417f, 726 726 + 0x85, 0xa9, 0x08, 0x8b, 0x16, 0x21, 0xeb, 0xa6); 727 727 + 728 728 + /* 729 729 + * DRAM Event Record 730 730 + * CXL rev 3.0 section 8.2.9.2.1.2; Table 8-44 731 731 + */ 732 732 + static const uuid_t dram_event_uuid = 733 733 + UUID_INIT(0x601dcbb3, 0x9c06, 0x4eab, 734 734 + 0xb8, 0xaf, 0x4e, 0x9b, 0xfb, 0x5c, 0x96, 0x24); 735 735 + 736 736 + /* 737 737 + * Memory Module Event Record 738 738 + * CXL rev 3.0 section 8.2.9.2.1.3; Table 8-45 739 739 + */ 740 740 + static const uuid_t mem_mod_event_uuid = 741 741 + UUID_INIT(0xfe927475, 0xdd59, 0x4339, 742 742 + 0xa5, 0x86, 0x79, 0xba, 0xb1, 0x13, 0xb7, 0x74); 743 743 + 744 744 + static void cxl_event_trace_record(const struct cxl_memdev *cxlmd, 745 745 + enum cxl_event_log_type type, 746 746 + struct cxl_event_record_raw *record) 747 747 + { 748 748 + uuid_t *id = &record->hdr.id; 749 749 + 750 750 + if (uuid_equal(id, &gen_media_event_uuid)) { 751 751 + struct cxl_event_gen_media *rec = 752 752 + (struct cxl_event_gen_media *)record; 753 753 + 754 754 + trace_cxl_general_media(cxlmd, type, rec); 755 755 + } else if (uuid_equal(id, &dram_event_uuid)) { 756 756 + struct cxl_event_dram *rec = (struct cxl_event_dram *)record; 757 757 + 758 758 + trace_cxl_dram(cxlmd, type, rec); 759 759 + } else if (uuid_equal(id, &mem_mod_event_uuid)) { 760 760 + struct cxl_event_mem_module *rec = 761 761 + (struct cxl_event_mem_module *)record; 762 762 + 763 763 + trace_cxl_memory_module(cxlmd, type, rec); 764 764 + } else { 765 765 + /* For unknown record types print just the header */ 766 766 + trace_cxl_generic_event(cxlmd, type, record); 767 767 + } 768 768 + } 769 769 + 770 770 + static int cxl_clear_event_record(struct cxl_dev_state *cxlds, 771 771 + enum cxl_event_log_type log, 772 772 + struct cxl_get_event_payload *get_pl) 773 773 + { 774 774 + struct cxl_mbox_clear_event_payload *payload; 775 775 + u16 total = le16_to_cpu(get_pl->record_count); 776 776 + u8 max_handles = CXL_CLEAR_EVENT_MAX_HANDLES; 777 777 + size_t pl_size = struct_size(payload, handles, max_handles); 778 778 + struct cxl_mbox_cmd mbox_cmd; 779 779 + u16 cnt; 780 780 + int rc = 0; 781 781 + int i; 782 782 + 783 783 + /* Payload size may limit the max handles */ 784 784 + if (pl_size > cxlds->payload_size) { 785 785 + max_handles = (cxlds->payload_size - sizeof(*payload)) / 786 786 + sizeof(__le16); 787 787 + pl_size = struct_size(payload, handles, max_handles); 788 788 + } 789 789 + 790 790 + payload = kvzalloc(pl_size, GFP_KERNEL); 791 791 + if (!payload) 792 792 + return -ENOMEM; 793 793 + 794 794 + *payload = (struct cxl_mbox_clear_event_payload) { 795 795 + .event_log = log, 796 796 + }; 797 797 + 798 798 + mbox_cmd = (struct cxl_mbox_cmd) { 799 799 + .opcode = CXL_MBOX_OP_CLEAR_EVENT_RECORD, 800 800 + .payload_in = payload, 801 801 + .size_in = pl_size, 802 802 + }; 803 803 + 804 804 + /* 805 805 + * Clear Event Records uses u8 for the handle cnt while Get Event 806 806 + * Record can return up to 0xffff records. 807 807 + */ 808 808 + i = 0; 809 809 + for (cnt = 0; cnt < total; cnt++) { 810 810 + payload->handles[i++] = get_pl->records[cnt].hdr.handle; 811 811 + dev_dbg(cxlds->dev, "Event log '%d': Clearing %u\n", 812 812 + log, le16_to_cpu(payload->handles[i])); 813 813 + 814 814 + if (i == max_handles) { 815 815 + payload->nr_recs = i; 816 816 + rc = cxl_internal_send_cmd(cxlds, &mbox_cmd); 817 817 + if (rc) 818 818 + goto free_pl; 819 819 + i = 0; 820 820 + } 821 821 + } 822 822 + 823 823 + /* Clear what is left if any */ 824 824 + if (i) { 825 825 + payload->nr_recs = i; 826 826 + mbox_cmd.size_in = struct_size(payload, handles, i); 827 827 + rc = cxl_internal_send_cmd(cxlds, &mbox_cmd); 828 828 + if (rc) 829 829 + goto free_pl; 830 830 + } 831 831 + 832 832 + free_pl: 833 833 + kvfree(payload); 834 834 + return rc; 835 835 + } 836 836 + 837 837 + static void cxl_mem_get_records_log(struct cxl_dev_state *cxlds, 838 838 + enum cxl_event_log_type type) 839 839 + { 840 840 + struct cxl_get_event_payload *payload; 841 841 + struct cxl_mbox_cmd mbox_cmd; 842 842 + u8 log_type = type; 843 843 + u16 nr_rec; 844 844 + 845 845 + mutex_lock(&cxlds->event.log_lock); 846 846 + payload = cxlds->event.buf; 847 847 + 848 848 + mbox_cmd = (struct cxl_mbox_cmd) { 849 849 + .opcode = CXL_MBOX_OP_GET_EVENT_RECORD, 850 850 + .payload_in = &log_type, 851 851 + .size_in = sizeof(log_type), 852 852 + .payload_out = payload, 853 853 + .size_out = cxlds->payload_size, 854 854 + .min_out = struct_size(payload, records, 0), 855 855 + }; 856 856 + 857 857 + do { 858 858 + int rc, i; 859 859 + 860 860 + rc = cxl_internal_send_cmd(cxlds, &mbox_cmd); 861 861 + if (rc) { 862 862 + dev_err_ratelimited(cxlds->dev, 863 863 + "Event log '%d': Failed to query event records : %d", 864 864 + type, rc); 865 865 + break; 866 866 + } 867 867 + 868 868 + nr_rec = le16_to_cpu(payload->record_count); 869 869 + if (!nr_rec) 870 870 + break; 871 871 + 872 872 + for (i = 0; i < nr_rec; i++) 873 873 + cxl_event_trace_record(cxlds->cxlmd, type, 874 874 + &payload->records[i]); 875 875 + 876 876 + if (payload->flags & CXL_GET_EVENT_FLAG_OVERFLOW) 877 877 + trace_cxl_overflow(cxlds->cxlmd, type, payload); 878 878 + 879 879 + rc = cxl_clear_event_record(cxlds, type, payload); 880 880 + if (rc) { 881 881 + dev_err_ratelimited(cxlds->dev, 882 882 + "Event log '%d': Failed to clear events : %d", 883 883 + type, rc); 884 884 + break; 885 885 + } 886 886 + } while (nr_rec); 887 887 + 888 888 + mutex_unlock(&cxlds->event.log_lock); 889 889 + } 890 890 + 891 891 + /** 892 892 + * cxl_mem_get_event_records - Get Event Records from the device 893 893 + * @cxlds: The device data for the operation 894 894 + * @status: Event Status register value identifying which events are available. 895 895 + * 896 896 + * Retrieve all event records available on the device, report them as trace 897 897 + * events, and clear them. 898 898 + * 899 899 + * See CXL rev 3.0 @8.2.9.2.2 Get Event Records 900 900 + * See CXL rev 3.0 @8.2.9.2.3 Clear Event Records 901 901 + */ 902 902 + void cxl_mem_get_event_records(struct cxl_dev_state *cxlds, u32 status) 903 903 + { 904 904 + dev_dbg(cxlds->dev, "Reading event logs: %x\n", status); 905 905 + 906 906 + if (status & CXLDEV_EVENT_STATUS_FATAL) 907 907 + cxl_mem_get_records_log(cxlds, CXL_EVENT_TYPE_FATAL); 908 908 + if (status & CXLDEV_EVENT_STATUS_FAIL) 909 909 + cxl_mem_get_records_log(cxlds, CXL_EVENT_TYPE_FAIL); 910 910 + if (status & CXLDEV_EVENT_STATUS_WARN) 911 911 + cxl_mem_get_records_log(cxlds, CXL_EVENT_TYPE_WARN); 912 912 + if (status & CXLDEV_EVENT_STATUS_INFO) 913 913 + cxl_mem_get_records_log(cxlds, CXL_EVENT_TYPE_INFO); 914 914 + } 915 915 + EXPORT_SYMBOL_NS_GPL(cxl_mem_get_event_records, CXL); 746 916 747 917 /** 748 918 * cxl_mem_get_partition_info - Get partition info ··· 1081 857 } 1082 858 EXPORT_SYMBOL_NS_GPL(cxl_mem_create_range_info, CXL); 1083 859 860 860 + int cxl_set_timestamp(struct cxl_dev_state *cxlds) 861 861 + { 862 862 + struct cxl_mbox_cmd mbox_cmd; 863 863 + struct cxl_mbox_set_timestamp_in pi; 864 864 + int rc; 865 865 + 866 866 + pi.timestamp = cpu_to_le64(ktime_get_real_ns()); 867 867 + mbox_cmd = (struct cxl_mbox_cmd) { 868 868 + .opcode = CXL_MBOX_OP_SET_TIMESTAMP, 869 869 + .size_in = sizeof(pi), 870 870 + .payload_in = &pi, 871 871 + }; 872 872 + 873 873 + rc = cxl_internal_send_cmd(cxlds, &mbox_cmd); 874 874 + /* 875 875 + * Command is optional. Devices may have another way of providing 876 876 + * a timestamp, or may return all 0s in timestamp fields. 877 877 + * Don't report an error if this command isn't supported 878 878 + */ 879 879 + if (rc && (mbox_cmd.return_code != CXL_MBOX_CMD_RC_UNSUPPORTED)) 880 880 + return rc; 881 881 + 882 882 + return 0; 883 883 + } 884 884 + EXPORT_SYMBOL_NS_GPL(cxl_set_timestamp, CXL); 885 885 + 1084 886 struct cxl_dev_state *cxl_dev_state_create(struct device *dev) 1085 887 { 1086 888 struct cxl_dev_state *cxlds; ··· 1118 868 } 1119 869 1120 870 mutex_init(&cxlds->mbox_mutex); 871 871 + mutex_init(&cxlds->event.log_lock); 1121 872 cxlds->dev = dev; 1122 873 1123 874 return cxlds;

+2 -1

drivers/cxl/core/memdev.c

reviewed

··· 242 242 if (!cxlmd) 243 243 return ERR_PTR(-ENOMEM); 244 244 245 245 - rc = ida_alloc_range(&cxl_memdev_ida, 0, CXL_MEM_MAX_DEVS, GFP_KERNEL); 245 245 + rc = ida_alloc_max(&cxl_memdev_ida, CXL_MEM_MAX_DEVS - 1, GFP_KERNEL); 246 246 if (rc < 0) 247 247 goto err; 248 248 cxlmd->id = rc; 249 249 + cxlmd->depth = -1; 249 250 250 251 dev = &cxlmd->dev; 251 252 device_initialize(dev);

+236 -144

drivers/cxl/core/pci.c

reviewed

··· 9 9 #include <cxlmem.h> 10 10 #include <cxl.h> 11 11 #include "core.h" 12 12 + #include "trace.h" 12 13 13 14 /** 14 15 * DOC: cxl core pci ··· 142 141 } 143 142 EXPORT_SYMBOL_NS_GPL(cxl_await_media_ready, CXL); 144 143 145 145 - static int wait_for_valid(struct cxl_dev_state *cxlds) 144 144 + static int wait_for_valid(struct pci_dev *pdev, int d) 146 145 { 147 147 - struct pci_dev *pdev = to_pci_dev(cxlds->dev); 148 148 - int d = cxlds->cxl_dvsec, rc; 149 146 u32 val; 147 147 + int rc; 150 148 151 149 /* 152 150 * Memory_Info_Valid: When set, indicates that the CXL Range 1 Size high ··· 213 213 return devm_add_action_or_reset(host, clear_mem_enable, cxlds); 214 214 } 215 215 216 216 - static bool range_contains(struct range *r1, struct range *r2) 217 217 - { 218 218 - return r1->start <= r2->start && r1->end >= r2->end; 219 219 - } 220 220 - 221 216 /* require dvsec ranges to be covered by a locked platform window */ 222 217 static int dvsec_range_allowed(struct device *dev, void *arg) 223 218 { ··· 224 229 225 230 cxld = to_cxl_decoder(dev); 226 231 227 227 - if (!(cxld->flags & CXL_DECODER_F_LOCK)) 228 228 - return 0; 229 232 if (!(cxld->flags & CXL_DECODER_F_RAM)) 230 233 return 0; 231 234 ··· 253 260 return devm_add_action_or_reset(host, disable_hdm, cxlhdm); 254 261 } 255 262 256 256 - static bool __cxl_hdm_decode_init(struct cxl_dev_state *cxlds, 257 257 - struct cxl_hdm *cxlhdm, 258 258 - struct cxl_endpoint_dvsec_info *info) 263 263 + int cxl_dvsec_rr_decode(struct device *dev, int d, 264 264 + struct cxl_endpoint_dvsec_info *info) 265 265 + { 266 266 + struct pci_dev *pdev = to_pci_dev(dev); 267 267 + int hdm_count, rc, i, ranges = 0; 268 268 + u16 cap, ctrl; 269 269 + 270 270 + if (!d) { 271 271 + dev_dbg(dev, "No DVSEC Capability\n"); 272 272 + return -ENXIO; 273 273 + } 274 274 + 275 275 + rc = pci_read_config_word(pdev, d + CXL_DVSEC_CAP_OFFSET, &cap); 276 276 + if (rc) 277 277 + return rc; 278 278 + 279 279 + rc = pci_read_config_word(pdev, d + CXL_DVSEC_CTRL_OFFSET, &ctrl); 280 280 + if (rc) 281 281 + return rc; 282 282 + 283 283 + if (!(cap & CXL_DVSEC_MEM_CAPABLE)) { 284 284 + dev_dbg(dev, "Not MEM Capable\n"); 285 285 + return -ENXIO; 286 286 + } 287 287 + 288 288 + /* 289 289 + * It is not allowed by spec for MEM.capable to be set and have 0 legacy 290 290 + * HDM decoders (values > 2 are also undefined as of CXL 2.0). As this 291 291 + * driver is for a spec defined class code which must be CXL.mem 292 292 + * capable, there is no point in continuing to enable CXL.mem. 293 293 + */ 294 294 + hdm_count = FIELD_GET(CXL_DVSEC_HDM_COUNT_MASK, cap); 295 295 + if (!hdm_count || hdm_count > 2) 296 296 + return -EINVAL; 297 297 + 298 298 + rc = wait_for_valid(pdev, d); 299 299 + if (rc) { 300 300 + dev_dbg(dev, "Failure awaiting MEM_INFO_VALID (%d)\n", rc); 301 301 + return rc; 302 302 + } 303 303 + 304 304 + /* 305 305 + * The current DVSEC values are moot if the memory capability is 306 306 + * disabled, and they will remain moot after the HDM Decoder 307 307 + * capability is enabled. 308 308 + */ 309 309 + info->mem_enabled = FIELD_GET(CXL_DVSEC_MEM_ENABLE, ctrl); 310 310 + if (!info->mem_enabled) 311 311 + return 0; 312 312 + 313 313 + for (i = 0; i < hdm_count; i++) { 314 314 + u64 base, size; 315 315 + u32 temp; 316 316 + 317 317 + rc = pci_read_config_dword( 318 318 + pdev, d + CXL_DVSEC_RANGE_SIZE_HIGH(i), &temp); 319 319 + if (rc) 320 320 + return rc; 321 321 + 322 322 + size = (u64)temp << 32; 323 323 + 324 324 + rc = pci_read_config_dword( 325 325 + pdev, d + CXL_DVSEC_RANGE_SIZE_LOW(i), &temp); 326 326 + if (rc) 327 327 + return rc; 328 328 + 329 329 + size |= temp & CXL_DVSEC_MEM_SIZE_LOW_MASK; 330 330 + if (!size) { 331 331 + info->dvsec_range[i] = (struct range) { 332 332 + .start = 0, 333 333 + .end = CXL_RESOURCE_NONE, 334 334 + }; 335 335 + continue; 336 336 + } 337 337 + 338 338 + rc = pci_read_config_dword( 339 339 + pdev, d + CXL_DVSEC_RANGE_BASE_HIGH(i), &temp); 340 340 + if (rc) 341 341 + return rc; 342 342 + 343 343 + base = (u64)temp << 32; 344 344 + 345 345 + rc = pci_read_config_dword( 346 346 + pdev, d + CXL_DVSEC_RANGE_BASE_LOW(i), &temp); 347 347 + if (rc) 348 348 + return rc; 349 349 + 350 350 + base |= temp & CXL_DVSEC_MEM_BASE_LOW_MASK; 351 351 + 352 352 + info->dvsec_range[i] = (struct range) { 353 353 + .start = base, 354 354 + .end = base + size - 1 355 355 + }; 356 356 + 357 357 + ranges++; 358 358 + } 359 359 + 360 360 + info->ranges = ranges; 361 361 + 362 362 + return 0; 363 363 + } 364 364 + EXPORT_SYMBOL_NS_GPL(cxl_dvsec_rr_decode, CXL); 365 365 + 366 366 + /** 367 367 + * cxl_hdm_decode_init() - Setup HDM decoding for the endpoint 368 368 + * @cxlds: Device state 369 369 + * @cxlhdm: Mapped HDM decoder Capability 370 370 + * @info: Cached DVSEC range registers info 371 371 + * 372 372 + * Try to enable the endpoint's HDM Decoder Capability 373 373 + */ 374 374 + int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm, 375 375 + struct cxl_endpoint_dvsec_info *info) 259 376 { 260 377 void __iomem *hdm = cxlhdm->regs.hdm_decoder; 261 378 struct cxl_port *port = cxlhdm->port; 262 379 struct device *dev = cxlds->dev; 263 380 struct cxl_port *root; 264 381 int i, rc, allowed; 265 265 - u32 global_ctrl; 382 382 + u32 global_ctrl = 0; 266 383 267 267 - global_ctrl = readl(hdm + CXL_HDM_DECODER_CTRL_OFFSET); 384 384 + if (hdm) 385 385 + global_ctrl = readl(hdm + CXL_HDM_DECODER_CTRL_OFFSET); 268 386 269 387 /* 270 388 * If the HDM Decoder Capability is already enabled then assume 271 389 * that some other agent like platform firmware set it up. 272 390 */ 273 273 - if (global_ctrl & CXL_HDM_DECODER_ENABLE) { 274 274 - rc = devm_cxl_enable_mem(&port->dev, cxlds); 275 275 - if (rc) 276 276 - return false; 277 277 - return true; 278 278 - } 391 391 + if (global_ctrl & CXL_HDM_DECODER_ENABLE || (!hdm && info->mem_enabled)) 392 392 + return devm_cxl_enable_mem(&port->dev, cxlds); 393 393 + else if (!hdm) 394 394 + return -ENODEV; 279 395 280 396 root = to_cxl_port(port->dev.parent); 281 397 while (!is_cxl_root(root) && is_cxl_port(root->dev.parent)) 282 398 root = to_cxl_port(root->dev.parent); 283 399 if (!is_cxl_root(root)) { 284 400 dev_err(dev, "Failed to acquire root port for HDM enable\n"); 285 285 - return false; 401 401 + return -ENODEV; 286 402 } 287 403 288 404 for (i = 0, allowed = 0; info->mem_enabled && i < info->ranges; i++) { ··· 423 321 * Decoder Capability Enable. 424 322 */ 425 323 if (info->mem_enabled) 426 426 - return false; 324 324 + return 0; 427 325 428 326 rc = devm_cxl_enable_hdm(&port->dev, cxlhdm); 429 327 if (rc) 430 430 - return false; 431 431 - 432 432 - rc = devm_cxl_enable_mem(&port->dev, cxlds); 433 433 - if (rc) 434 434 - return false; 435 435 - 436 436 - return true; 437 437 - } 438 438 - 439 439 - /** 440 440 - * cxl_hdm_decode_init() - Setup HDM decoding for the endpoint 441 441 - * @cxlds: Device state 442 442 - * @cxlhdm: Mapped HDM decoder Capability 443 443 - * 444 444 - * Try to enable the endpoint's HDM Decoder Capability 445 445 - */ 446 446 - int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm) 447 447 - { 448 448 - struct pci_dev *pdev = to_pci_dev(cxlds->dev); 449 449 - struct cxl_endpoint_dvsec_info info = { 0 }; 450 450 - int hdm_count, rc, i, ranges = 0; 451 451 - struct device *dev = &pdev->dev; 452 452 - int d = cxlds->cxl_dvsec; 453 453 - u16 cap, ctrl; 454 454 - 455 455 - if (!d) { 456 456 - dev_dbg(dev, "No DVSEC Capability\n"); 457 457 - return -ENXIO; 458 458 - } 459 459 - 460 460 - rc = pci_read_config_word(pdev, d + CXL_DVSEC_CAP_OFFSET, &cap); 461 461 - if (rc) 462 328 return rc; 463 329 464 464 - rc = pci_read_config_word(pdev, d + CXL_DVSEC_CTRL_OFFSET, &ctrl); 465 465 - if (rc) 466 466 - return rc; 467 467 - 468 468 - if (!(cap & CXL_DVSEC_MEM_CAPABLE)) { 469 469 - dev_dbg(dev, "Not MEM Capable\n"); 470 470 - return -ENXIO; 471 471 - } 472 472 - 473 473 - /* 474 474 - * It is not allowed by spec for MEM.capable to be set and have 0 legacy 475 475 - * HDM decoders (values > 2 are also undefined as of CXL 2.0). As this 476 476 - * driver is for a spec defined class code which must be CXL.mem 477 477 - * capable, there is no point in continuing to enable CXL.mem. 478 478 - */ 479 479 - hdm_count = FIELD_GET(CXL_DVSEC_HDM_COUNT_MASK, cap); 480 480 - if (!hdm_count || hdm_count > 2) 481 481 - return -EINVAL; 482 482 - 483 483 - rc = wait_for_valid(cxlds); 484 484 - if (rc) { 485 485 - dev_dbg(dev, "Failure awaiting MEM_INFO_VALID (%d)\n", rc); 486 486 - return rc; 487 487 - } 488 488 - 489 489 - /* 490 490 - * The current DVSEC values are moot if the memory capability is 491 491 - * disabled, and they will remain moot after the HDM Decoder 492 492 - * capability is enabled. 493 493 - */ 494 494 - info.mem_enabled = FIELD_GET(CXL_DVSEC_MEM_ENABLE, ctrl); 495 495 - if (!info.mem_enabled) 496 496 - goto hdm_init; 497 497 - 498 498 - for (i = 0; i < hdm_count; i++) { 499 499 - u64 base, size; 500 500 - u32 temp; 501 501 - 502 502 - rc = pci_read_config_dword( 503 503 - pdev, d + CXL_DVSEC_RANGE_SIZE_HIGH(i), &temp); 504 504 - if (rc) 505 505 - return rc; 506 506 - 507 507 - size = (u64)temp << 32; 508 508 - 509 509 - rc = pci_read_config_dword( 510 510 - pdev, d + CXL_DVSEC_RANGE_SIZE_LOW(i), &temp); 511 511 - if (rc) 512 512 - return rc; 513 513 - 514 514 - size |= temp & CXL_DVSEC_MEM_SIZE_LOW_MASK; 515 515 - 516 516 - rc = pci_read_config_dword( 517 517 - pdev, d + CXL_DVSEC_RANGE_BASE_HIGH(i), &temp); 518 518 - if (rc) 519 519 - return rc; 520 520 - 521 521 - base = (u64)temp << 32; 522 522 - 523 523 - rc = pci_read_config_dword( 524 524 - pdev, d + CXL_DVSEC_RANGE_BASE_LOW(i), &temp); 525 525 - if (rc) 526 526 - return rc; 527 527 - 528 528 - base |= temp & CXL_DVSEC_MEM_BASE_LOW_MASK; 529 529 - 530 530 - info.dvsec_range[i] = (struct range) { 531 531 - .start = base, 532 532 - .end = base + size - 1 533 533 - }; 534 534 - 535 535 - if (size) 536 536 - ranges++; 537 537 - } 538 538 - 539 539 - info.ranges = ranges; 540 540 - 541 541 - /* 542 542 - * If DVSEC ranges are being used instead of HDM decoder registers there 543 543 - * is no use in trying to manage those. 544 544 - */ 545 545 - hdm_init: 546 546 - if (!__cxl_hdm_decode_init(cxlds, cxlhdm, &info)) { 547 547 - dev_err(dev, 548 548 - "Legacy range registers configuration prevents HDM operation.\n"); 549 549 - return -EBUSY; 550 550 - } 551 551 - 552 552 - return 0; 330 330 + return devm_cxl_enable_mem(&port->dev, cxlds); 553 331 } 554 332 EXPORT_SYMBOL_NS_GPL(cxl_hdm_decode_init, CXL); 555 333 ··· 604 622 } 605 623 } 606 624 EXPORT_SYMBOL_NS_GPL(read_cdat_data, CXL); 625 625 + 626 626 + void cxl_cor_error_detected(struct pci_dev *pdev) 627 627 + { 628 628 + struct cxl_dev_state *cxlds = pci_get_drvdata(pdev); 629 629 + void __iomem *addr; 630 630 + u32 status; 631 631 + 632 632 + if (!cxlds->regs.ras) 633 633 + return; 634 634 + 635 635 + addr = cxlds->regs.ras + CXL_RAS_CORRECTABLE_STATUS_OFFSET; 636 636 + status = readl(addr); 637 637 + if (status & CXL_RAS_CORRECTABLE_STATUS_MASK) { 638 638 + writel(status & CXL_RAS_CORRECTABLE_STATUS_MASK, addr); 639 639 + trace_cxl_aer_correctable_error(cxlds->cxlmd, status); 640 640 + } 641 641 + } 642 642 + EXPORT_SYMBOL_NS_GPL(cxl_cor_error_detected, CXL); 643 643 + 644 644 + /* CXL spec rev3.0 8.2.4.16.1 */ 645 645 + static void header_log_copy(struct cxl_dev_state *cxlds, u32 *log) 646 646 + { 647 647 + void __iomem *addr; 648 648 + u32 *log_addr; 649 649 + int i, log_u32_size = CXL_HEADERLOG_SIZE / sizeof(u32); 650 650 + 651 651 + addr = cxlds->regs.ras + CXL_RAS_HEADER_LOG_OFFSET; 652 652 + log_addr = log; 653 653 + 654 654 + for (i = 0; i < log_u32_size; i++) { 655 655 + *log_addr = readl(addr); 656 656 + log_addr++; 657 657 + addr += sizeof(u32); 658 658 + } 659 659 + } 660 660 + 661 661 + /* 662 662 + * Log the state of the RAS status registers and prepare them to log the 663 663 + * next error status. Return 1 if reset needed. 664 664 + */ 665 665 + static bool cxl_report_and_clear(struct cxl_dev_state *cxlds) 666 666 + { 667 667 + u32 hl[CXL_HEADERLOG_SIZE_U32]; 668 668 + void __iomem *addr; 669 669 + u32 status; 670 670 + u32 fe; 671 671 + 672 672 + if (!cxlds->regs.ras) 673 673 + return false; 674 674 + 675 675 + addr = cxlds->regs.ras + CXL_RAS_UNCORRECTABLE_STATUS_OFFSET; 676 676 + status = readl(addr); 677 677 + if (!(status & CXL_RAS_UNCORRECTABLE_STATUS_MASK)) 678 678 + return false; 679 679 + 680 680 + /* If multiple errors, log header points to first error from ctrl reg */ 681 681 + if (hweight32(status) > 1) { 682 682 + void __iomem *rcc_addr = 683 683 + cxlds->regs.ras + CXL_RAS_CAP_CONTROL_OFFSET; 684 684 + 685 685 + fe = BIT(FIELD_GET(CXL_RAS_CAP_CONTROL_FE_MASK, 686 686 + readl(rcc_addr))); 687 687 + } else { 688 688 + fe = status; 689 689 + } 690 690 + 691 691 + header_log_copy(cxlds, hl); 692 692 + trace_cxl_aer_uncorrectable_error(cxlds->cxlmd, status, fe, hl); 693 693 + writel(status & CXL_RAS_UNCORRECTABLE_STATUS_MASK, addr); 694 694 + 695 695 + return true; 696 696 + } 697 697 + 698 698 + pci_ers_result_t cxl_error_detected(struct pci_dev *pdev, 699 699 + pci_channel_state_t state) 700 700 + { 701 701 + struct cxl_dev_state *cxlds = pci_get_drvdata(pdev); 702 702 + struct cxl_memdev *cxlmd = cxlds->cxlmd; 703 703 + struct device *dev = &cxlmd->dev; 704 704 + bool ue; 705 705 + 706 706 + /* 707 707 + * A frozen channel indicates an impending reset which is fatal to 708 708 + * CXL.mem operation, and will likely crash the system. On the off 709 709 + * chance the situation is recoverable dump the status of the RAS 710 710 + * capability registers and bounce the active state of the memdev. 711 711 + */ 712 712 + ue = cxl_report_and_clear(cxlds); 713 713 + 714 714 + switch (state) { 715 715 + case pci_channel_io_normal: 716 716 + if (ue) { 717 717 + device_release_driver(dev); 718 718 + return PCI_ERS_RESULT_NEED_RESET; 719 719 + } 720 720 + return PCI_ERS_RESULT_CAN_RECOVER; 721 721 + case pci_channel_io_frozen: 722 722 + dev_warn(&pdev->dev, 723 723 + "%s: frozen state error detected, disable CXL.mem\n", 724 724 + dev_name(dev)); 725 725 + device_release_driver(dev); 726 726 + return PCI_ERS_RESULT_NEED_RESET; 727 727 + case pci_channel_io_perm_failure: 728 728 + dev_warn(&pdev->dev, 729 729 + "failure state error detected, request disconnect\n"); 730 730 + return PCI_ERS_RESULT_DISCONNECT; 731 731 + } 732 732 + return PCI_ERS_RESULT_NEED_RESET; 733 733 + } 734 734 + EXPORT_SYMBOL_NS_GPL(cxl_error_detected, CXL);

+83 -40

drivers/cxl/core/port.c

reviewed

··· 46 46 return CXL_DEVICE_NVDIMM; 47 47 if (dev->type == CXL_PMEM_REGION_TYPE()) 48 48 return CXL_DEVICE_PMEM_REGION; 49 49 + if (dev->type == CXL_DAX_REGION_TYPE()) 50 50 + return CXL_DEVICE_DAX_REGION; 49 51 if (is_cxl_port(dev)) { 50 52 if (is_cxl_root(to_cxl_port(dev))) 51 53 return CXL_DEVICE_ROOT; ··· 182 180 { 183 181 struct cxl_endpoint_decoder *cxled = to_cxl_endpoint_decoder(dev); 184 182 185 185 - switch (cxled->mode) { 186 186 - case CXL_DECODER_RAM: 187 187 - return sysfs_emit(buf, "ram\n"); 188 188 - case CXL_DECODER_PMEM: 189 189 - return sysfs_emit(buf, "pmem\n"); 190 190 - case CXL_DECODER_NONE: 191 191 - return sysfs_emit(buf, "none\n"); 192 192 - case CXL_DECODER_MIXED: 193 193 - default: 194 194 - return sysfs_emit(buf, "mixed\n"); 195 195 - } 183 183 + return sysfs_emit(buf, "%s\n", cxl_decoder_mode_name(cxled->mode)); 196 184 } 197 185 198 186 static ssize_t mode_store(struct device *dev, struct device_attribute *attr, ··· 296 304 &dev_attr_cap_type3.attr, 297 305 &dev_attr_target_list.attr, 298 306 SET_CXL_REGION_ATTR(create_pmem_region) 307 307 + SET_CXL_REGION_ATTR(create_ram_region) 299 308 SET_CXL_REGION_ATTR(delete_region) 300 309 NULL, 301 310 }; ··· 304 311 static bool can_create_pmem(struct cxl_root_decoder *cxlrd) 305 312 { 306 313 unsigned long flags = CXL_DECODER_F_TYPE3 | CXL_DECODER_F_PMEM; 314 314 + 315 315 + return (cxlrd->cxlsd.cxld.flags & flags) == flags; 316 316 + } 317 317 + 318 318 + static bool can_create_ram(struct cxl_root_decoder *cxlrd) 319 319 + { 320 320 + unsigned long flags = CXL_DECODER_F_TYPE3 | CXL_DECODER_F_RAM; 307 321 308 322 return (cxlrd->cxlsd.cxld.flags & flags) == flags; 309 323 } ··· 323 323 if (a == CXL_REGION_ATTR(create_pmem_region) && !can_create_pmem(cxlrd)) 324 324 return 0; 325 325 326 326 - if (a == CXL_REGION_ATTR(delete_region) && !can_create_pmem(cxlrd)) 326 326 + if (a == CXL_REGION_ATTR(create_ram_region) && !can_create_ram(cxlrd)) 327 327 + return 0; 328 328 + 329 329 + if (a == CXL_REGION_ATTR(delete_region) && 330 330 + !(can_create_pmem(cxlrd) || can_create_ram(cxlrd))) 327 331 return 0; 328 332 329 333 return a->mode; ··· 448 444 { 449 445 return dev->type == &cxl_decoder_endpoint_type; 450 446 } 447 447 + EXPORT_SYMBOL_NS_GPL(is_endpoint_decoder, CXL); 451 448 452 449 bool is_root_decoder(struct device *dev) 453 450 { ··· 460 455 { 461 456 return is_root_decoder(dev) || dev->type == &cxl_decoder_switch_type; 462 457 } 458 458 + EXPORT_SYMBOL_NS_GPL(is_switch_decoder, CXL); 463 459 464 460 struct cxl_decoder *to_cxl_decoder(struct device *dev) 465 461 { ··· 488 482 return NULL; 489 483 return container_of(dev, struct cxl_switch_decoder, cxld.dev); 490 484 } 485 485 + EXPORT_SYMBOL_NS_GPL(to_cxl_switch_decoder, CXL); 491 486 492 487 static void cxl_ep_release(struct cxl_ep *ep) 493 488 { ··· 588 581 if (rc) 589 582 return rc; 590 583 return devm_add_action_or_reset(host, cxl_unlink_uport, port); 584 584 + } 585 585 + 586 586 + static void cxl_unlink_parent_dport(void *_port) 587 587 + { 588 588 + struct cxl_port *port = _port; 589 589 + 590 590 + sysfs_remove_link(&port->dev.kobj, "parent_dport"); 591 591 + } 592 592 + 593 593 + static int devm_cxl_link_parent_dport(struct device *host, 594 594 + struct cxl_port *port, 595 595 + struct cxl_dport *parent_dport) 596 596 + { 597 597 + int rc; 598 598 + 599 599 + if (!parent_dport) 600 600 + return 0; 601 601 + 602 602 + rc = sysfs_create_link(&port->dev.kobj, &parent_dport->dport->kobj, 603 603 + "parent_dport"); 604 604 + if (rc) 605 605 + return rc; 606 606 + return devm_add_action_or_reset(host, cxl_unlink_parent_dport, port); 591 607 } 592 608 593 609 static struct lock_class_key cxl_port_key; ··· 719 689 return ERR_PTR(rc); 720 690 721 691 rc = devm_cxl_link_uport(host, port); 692 692 + if (rc) 693 693 + return ERR_PTR(rc); 694 694 + 695 695 + rc = devm_cxl_link_parent_dport(host, port, parent_dport); 722 696 if (rc) 723 697 return ERR_PTR(rc); 724 698 ··· 1171 1137 } 1172 1138 1173 1139 /* 1174 1174 - * All users of grandparent() are using it to walk PCIe-like swich port 1140 1140 + * All users of grandparent() are using it to walk PCIe-like switch port 1175 1141 * hierarchy. A PCIe switch is comprised of a bridge device representing the 1176 1142 * upstream switch port and N bridges representing downstream switch ports. When 1177 1143 * bridges stack the grand-parent of a downstream switch port is another ··· 1198 1164 1199 1165 device_lock(parent); 1200 1166 if (parent->driver && !endpoint->dead) { 1167 1167 + devm_release_action(parent, cxl_unlink_parent_dport, endpoint); 1201 1168 devm_release_action(parent, cxl_unlink_uport, endpoint); 1202 1169 devm_release_action(parent, unregister_port, endpoint); 1203 1170 } ··· 1214 1179 1215 1180 get_device(&endpoint->dev); 1216 1181 dev_set_drvdata(dev, endpoint); 1182 1182 + cxlmd->depth = endpoint->depth; 1217 1183 return devm_add_action_or_reset(dev, delete_endpoint, cxlmd); 1218 1184 } 1219 1185 EXPORT_SYMBOL_NS_GPL(cxl_endpoint_autoremove, CXL); ··· 1230 1194 */ 1231 1195 static void delete_switch_port(struct cxl_port *port) 1232 1196 { 1197 1197 + devm_release_action(port->dev.parent, cxl_unlink_parent_dport, port); 1233 1198 devm_release_action(port->dev.parent, cxl_unlink_uport, port); 1234 1199 devm_release_action(port->dev.parent, unregister_port, port); 1235 1200 } ··· 1249 1212 } 1250 1213 } 1251 1214 1215 1215 + struct detach_ctx { 1216 1216 + struct cxl_memdev *cxlmd; 1217 1217 + int depth; 1218 1218 + }; 1219 1219 + 1220 1220 + static int port_has_memdev(struct device *dev, const void *data) 1221 1221 + { 1222 1222 + const struct detach_ctx *ctx = data; 1223 1223 + struct cxl_port *port; 1224 1224 + 1225 1225 + if (!is_cxl_port(dev)) 1226 1226 + return 0; 1227 1227 + 1228 1228 + port = to_cxl_port(dev); 1229 1229 + if (port->depth != ctx->depth) 1230 1230 + return 0; 1231 1231 + 1232 1232 + return !!cxl_ep_load(port, ctx->cxlmd); 1233 1233 + } 1234 1234 + 1252 1235 static void cxl_detach_ep(void *data) 1253 1236 { 1254 1237 struct cxl_memdev *cxlmd = data; 1255 1255 - struct device *iter; 1256 1238 1257 1257 - for (iter = &cxlmd->dev; iter; iter = grandparent(iter)) { 1258 1258 - struct device *dport_dev = grandparent(iter); 1239 1239 + for (int i = cxlmd->depth - 1; i >= 1; i--) { 1259 1240 struct cxl_port *port, *parent_port; 1241 1241 + struct detach_ctx ctx = { 1242 1242 + .cxlmd = cxlmd, 1243 1243 + .depth = i, 1244 1244 + }; 1245 1245 + struct device *dev; 1260 1246 struct cxl_ep *ep; 1261 1247 bool died = false; 1262 1248 1263 1263 - if (!dport_dev) 1264 1264 - break; 1265 1265 - 1266 1266 - port = find_cxl_port(dport_dev, NULL); 1267 1267 - if (!port) 1249 1249 + dev = bus_find_device(&cxl_bus_type, NULL, &ctx, 1250 1250 + port_has_memdev); 1251 1251 + if (!dev) 1268 1252 continue; 1269 1269 - 1270 1270 - if (is_cxl_root(port)) { 1271 1271 - put_device(&port->dev); 1272 1272 - continue; 1273 1273 - } 1253 1253 + port = to_cxl_port(dev); 1274 1254 1275 1255 parent_port = to_cxl_port(port->dev.parent); 1276 1256 device_lock(&parent_port->dev); 1277 1277 - if (!parent_port->dev.driver) { 1278 1278 - /* 1279 1279 - * The bottom-up race to delete the port lost to a 1280 1280 - * top-down port disable, give up here, because the 1281 1281 - * parent_port ->remove() will have cleaned up all 1282 1282 - * descendants. 1283 1283 - */ 1284 1284 - device_unlock(&parent_port->dev); 1285 1285 - put_device(&port->dev); 1286 1286 - continue; 1287 1287 - } 1288 1288 - 1289 1257 device_lock(&port->dev); 1290 1258 ep = cxl_ep_load(port, cxlmd); 1291 1259 dev_dbg(&cxlmd->dev, "disconnect %s from %s\n", 1292 1260 ep ? dev_name(ep->ep) : "", dev_name(&port->dev)); 1293 1261 cxl_ep_remove(port, ep); 1294 1262 if (ep && !port->dead && xa_empty(&port->endpoints) && 1295 1295 - !is_cxl_root(parent_port)) { 1263 1263 + !is_cxl_root(parent_port) && parent_port->dev.driver) { 1296 1264 /* 1297 1265 * This was the last ep attached to a dynamically 1298 1266 * enumerated port. Block new cxl_add_ep() and garbage ··· 1633 1591 } 1634 1592 1635 1593 cxlrd->calc_hb = calc_hb; 1594 1594 + mutex_init(&cxlrd->range_lock); 1636 1595 1637 1596 cxld = &cxlsd->cxld; 1638 1597 cxld->dev.type = &cxl_decoder_root_type; ··· 2017 1974 debugfs_remove_recursive(cxl_debugfs); 2018 1975 } 2019 1976 2020 2020 - module_init(cxl_core_init); 1977 1977 + subsys_initcall(cxl_core_init); 2021 1978 module_exit(cxl_core_exit); 2022 1979 MODULE_LICENSE("GPL v2");

+796 -84

drivers/cxl/core/region.c

reviewed

··· 6 6 #include <linux/module.h> 7 7 #include <linux/slab.h> 8 8 #include <linux/uuid.h> 9 9 + #include <linux/sort.h> 9 10 #include <linux/idr.h> 10 11 #include <cxlmem.h> 11 12 #include <cxl.h> ··· 46 45 rc = down_read_interruptible(&cxl_region_rwsem); 47 46 if (rc) 48 47 return rc; 49 49 - rc = sysfs_emit(buf, "%pUb\n", &p->uuid); 48 48 + if (cxlr->mode != CXL_DECODER_PMEM) 49 49 + rc = sysfs_emit(buf, "\n"); 50 50 + else 51 51 + rc = sysfs_emit(buf, "%pUb\n", &p->uuid); 50 52 up_read(&cxl_region_rwsem); 51 53 52 54 return rc; ··· 161 157 return 0; 162 158 } 163 159 160 160 + static int commit_decoder(struct cxl_decoder *cxld) 161 161 + { 162 162 + struct cxl_switch_decoder *cxlsd = NULL; 163 163 + 164 164 + if (cxld->commit) 165 165 + return cxld->commit(cxld); 166 166 + 167 167 + if (is_switch_decoder(&cxld->dev)) 168 168 + cxlsd = to_cxl_switch_decoder(&cxld->dev); 169 169 + 170 170 + if (dev_WARN_ONCE(&cxld->dev, !cxlsd || cxlsd->nr_targets > 1, 171 171 + "->commit() is required\n")) 172 172 + return -ENXIO; 173 173 + return 0; 174 174 + } 175 175 + 164 176 static int cxl_region_decode_commit(struct cxl_region *cxlr) 165 177 { 166 178 struct cxl_region_params *p = &cxlr->params; ··· 195 175 iter = to_cxl_port(iter->dev.parent)) { 196 176 cxl_rr = cxl_rr_load(iter, cxlr); 197 177 cxld = cxl_rr->decoder; 198 198 - if (cxld->commit) 199 199 - rc = cxld->commit(cxld); 178 178 + rc = commit_decoder(cxld); 200 179 if (rc) 201 180 break; 202 181 } ··· 306 287 struct device *dev = kobj_to_dev(kobj); 307 288 struct cxl_region *cxlr = to_cxl_region(dev); 308 289 290 290 + /* 291 291 + * Support tooling that expects to find a 'uuid' attribute for all 292 292 + * regions regardless of mode. 293 293 + */ 309 294 if (a == &dev_attr_uuid.attr && cxlr->mode != CXL_DECODER_PMEM) 310 310 - return 0; 295 295 + return 0444; 311 296 return a->mode; 312 297 } 313 298 ··· 424 401 * When the host-bridge is interleaved, disallow region granularity != 425 402 * root granularity. Regions with a granularity less than the root 426 403 * interleave result in needing multiple endpoints to support a single 427 427 - * slot in the interleave (possible to suport in the future). Regions 404 404 + * slot in the interleave (possible to support in the future). Regions 428 405 * with a granularity greater than the root interleave result in invalid 429 406 * DPA translations (invalid to support). 430 407 */ ··· 467 444 return rc; 468 445 } 469 446 static DEVICE_ATTR_RO(resource); 447 447 + 448 448 + static ssize_t mode_show(struct device *dev, struct device_attribute *attr, 449 449 + char *buf) 450 450 + { 451 451 + struct cxl_region *cxlr = to_cxl_region(dev); 452 452 + 453 453 + return sysfs_emit(buf, "%s\n", cxl_decoder_mode_name(cxlr->mode)); 454 454 + } 455 455 + static DEVICE_ATTR_RO(mode); 470 456 471 457 static int alloc_hpa(struct cxl_region *cxlr, resource_size_t size) 472 458 { ··· 527 495 if (device_is_registered(&cxlr->dev)) 528 496 lockdep_assert_held_write(&cxl_region_rwsem); 529 497 if (p->res) { 530 530 - remove_resource(p->res); 498 498 + /* 499 499 + * Autodiscovered regions may not have been able to insert their 500 500 + * resource. 501 501 + */ 502 502 + if (p->res->parent) 503 503 + remove_resource(p->res); 531 504 kfree(p->res); 532 505 p->res = NULL; 533 506 } ··· 609 572 &dev_attr_interleave_granularity.attr, 610 573 &dev_attr_resource.attr, 611 574 &dev_attr_size.attr, 575 575 + &dev_attr_mode.attr, 612 576 NULL, 613 577 }; 614 578 ··· 1113 1075 return rc; 1114 1076 } 1115 1077 1116 1116 - cxld->interleave_ways = iw; 1117 1117 - cxld->interleave_granularity = ig; 1118 1118 - cxld->hpa_range = (struct range) { 1119 1119 - .start = p->res->start, 1120 1120 - .end = p->res->end, 1121 1121 - }; 1078 1078 + if (test_bit(CXL_REGION_F_AUTO, &cxlr->flags)) { 1079 1079 + if (cxld->interleave_ways != iw || 1080 1080 + cxld->interleave_granularity != ig || 1081 1081 + cxld->hpa_range.start != p->res->start || 1082 1082 + cxld->hpa_range.end != p->res->end || 1083 1083 + ((cxld->flags & CXL_DECODER_F_ENABLE) == 0)) { 1084 1084 + dev_err(&cxlr->dev, 1085 1085 + "%s:%s %s expected iw: %d ig: %d %pr\n", 1086 1086 + dev_name(port->uport), dev_name(&port->dev), 1087 1087 + __func__, iw, ig, p->res); 1088 1088 + dev_err(&cxlr->dev, 1089 1089 + "%s:%s %s got iw: %d ig: %d state: %s %#llx:%#llx\n", 1090 1090 + dev_name(port->uport), dev_name(&port->dev), 1091 1091 + __func__, cxld->interleave_ways, 1092 1092 + cxld->interleave_granularity, 1093 1093 + (cxld->flags & CXL_DECODER_F_ENABLE) ? 1094 1094 + "enabled" : 1095 1095 + "disabled", 1096 1096 + cxld->hpa_range.start, cxld->hpa_range.end); 1097 1097 + return -ENXIO; 1098 1098 + } 1099 1099 + } else { 1100 1100 + cxld->interleave_ways = iw; 1101 1101 + cxld->interleave_granularity = ig; 1102 1102 + cxld->hpa_range = (struct range) { 1103 1103 + .start = p->res->start, 1104 1104 + .end = p->res->end, 1105 1105 + }; 1106 1106 + } 1122 1107 dev_dbg(&cxlr->dev, "%s:%s iw: %d ig: %d\n", dev_name(port->uport), 1123 1108 dev_name(&port->dev), iw, ig); 1124 1109 add_target: ··· 1152 1091 dev_name(&cxlmd->dev), dev_name(&cxled->cxld.dev), pos); 1153 1092 return -ENXIO; 1154 1093 } 1155 1155 - cxlsd->target[cxl_rr->nr_targets_set] = ep->dport; 1094 1094 + if (test_bit(CXL_REGION_F_AUTO, &cxlr->flags)) { 1095 1095 + if (cxlsd->target[cxl_rr->nr_targets_set] != ep->dport) { 1096 1096 + dev_dbg(&cxlr->dev, "%s:%s: %s expected %s at %d\n", 1097 1097 + dev_name(port->uport), dev_name(&port->dev), 1098 1098 + dev_name(&cxlsd->cxld.dev), 1099 1099 + dev_name(ep->dport->dport), 1100 1100 + cxl_rr->nr_targets_set); 1101 1101 + return -ENXIO; 1102 1102 + } 1103 1103 + } else 1104 1104 + cxlsd->target[cxl_rr->nr_targets_set] = ep->dport; 1156 1105 inc = 1; 1157 1106 out_target_set: 1158 1107 cxl_rr->nr_targets_set += inc; ··· 1204 1133 struct cxl_ep *ep; 1205 1134 int i; 1206 1135 1136 1136 + /* 1137 1137 + * In the auto-discovery case skip automatic teardown since the 1138 1138 + * address space is already active 1139 1139 + */ 1140 1140 + if (test_bit(CXL_REGION_F_AUTO, &cxlr->flags)) 1141 1141 + return; 1142 1142 + 1207 1143 for (i = 0; i < p->nr_targets; i++) { 1208 1144 cxled = p->targets[i]; 1209 1145 cxlmd = cxled_to_memdev(cxled); ··· 1243 1165 iter = to_cxl_port(iter->dev.parent); 1244 1166 1245 1167 /* 1246 1246 - * Descend the topology tree programming targets while 1247 1247 - * looking for conflicts. 1168 1168 + * Descend the topology tree programming / validating 1169 1169 + * targets while looking for conflicts. 1248 1170 */ 1249 1171 for (ep = cxl_ep_load(iter, cxlmd); iter; 1250 1172 iter = ep->next, ep = cxl_ep_load(iter, cxlmd)) { ··· 1259 1181 return 0; 1260 1182 } 1261 1183 1262 1262 - static int cxl_region_attach(struct cxl_region *cxlr, 1263 1263 - struct cxl_endpoint_decoder *cxled, int pos) 1184 1184 + static int cxl_region_validate_position(struct cxl_region *cxlr, 1185 1185 + struct cxl_endpoint_decoder *cxled, 1186 1186 + int pos) 1264 1187 { 1265 1265 - struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(cxlr->dev.parent); 1266 1188 struct cxl_memdev *cxlmd = cxled_to_memdev(cxled); 1267 1267 - struct cxl_port *ep_port, *root_port, *iter; 1268 1189 struct cxl_region_params *p = &cxlr->params; 1269 1269 - struct cxl_dport *dport; 1270 1270 - int i, rc = -ENXIO; 1271 1271 - 1272 1272 - if (cxled->mode == CXL_DECODER_DEAD) { 1273 1273 - dev_dbg(&cxlr->dev, "%s dead\n", dev_name(&cxled->cxld.dev)); 1274 1274 - return -ENODEV; 1275 1275 - } 1276 1276 - 1277 1277 - /* all full of members, or interleave config not established? */ 1278 1278 - if (p->state > CXL_CONFIG_INTERLEAVE_ACTIVE) { 1279 1279 - dev_dbg(&cxlr->dev, "region already active\n"); 1280 1280 - return -EBUSY; 1281 1281 - } else if (p->state < CXL_CONFIG_INTERLEAVE_ACTIVE) { 1282 1282 - dev_dbg(&cxlr->dev, "interleave config missing\n"); 1283 1283 - return -ENXIO; 1284 1284 - } 1190 1190 + int i; 1285 1191 1286 1192 if (pos < 0 || pos >= p->interleave_ways) { 1287 1193 dev_dbg(&cxlr->dev, "position %d out of range %d\n", pos, ··· 1304 1242 } 1305 1243 } 1306 1244 1245 1245 + return 0; 1246 1246 + } 1247 1247 + 1248 1248 + static int cxl_region_attach_position(struct cxl_region *cxlr, 1249 1249 + struct cxl_root_decoder *cxlrd, 1250 1250 + struct cxl_endpoint_decoder *cxled, 1251 1251 + const struct cxl_dport *dport, int pos) 1252 1252 + { 1253 1253 + struct cxl_memdev *cxlmd = cxled_to_memdev(cxled); 1254 1254 + struct cxl_port *iter; 1255 1255 + int rc; 1256 1256 + 1257 1257 + if (cxlrd->calc_hb(cxlrd, pos) != dport) { 1258 1258 + dev_dbg(&cxlr->dev, "%s:%s invalid target position for %s\n", 1259 1259 + dev_name(&cxlmd->dev), dev_name(&cxled->cxld.dev), 1260 1260 + dev_name(&cxlrd->cxlsd.cxld.dev)); 1261 1261 + return -ENXIO; 1262 1262 + } 1263 1263 + 1264 1264 + for (iter = cxled_to_port(cxled); !is_cxl_root(iter); 1265 1265 + iter = to_cxl_port(iter->dev.parent)) { 1266 1266 + rc = cxl_port_attach_region(iter, cxlr, cxled, pos); 1267 1267 + if (rc) 1268 1268 + goto err; 1269 1269 + } 1270 1270 + 1271 1271 + return 0; 1272 1272 + 1273 1273 + err: 1274 1274 + for (iter = cxled_to_port(cxled); !is_cxl_root(iter); 1275 1275 + iter = to_cxl_port(iter->dev.parent)) 1276 1276 + cxl_port_detach_region(iter, cxlr, cxled); 1277 1277 + return rc; 1278 1278 + } 1279 1279 + 1280 1280 + static int cxl_region_attach_auto(struct cxl_region *cxlr, 1281 1281 + struct cxl_endpoint_decoder *cxled, int pos) 1282 1282 + { 1283 1283 + struct cxl_region_params *p = &cxlr->params; 1284 1284 + 1285 1285 + if (cxled->state != CXL_DECODER_STATE_AUTO) { 1286 1286 + dev_err(&cxlr->dev, 1287 1287 + "%s: unable to add decoder to autodetected region\n", 1288 1288 + dev_name(&cxled->cxld.dev)); 1289 1289 + return -EINVAL; 1290 1290 + } 1291 1291 + 1292 1292 + if (pos >= 0) { 1293 1293 + dev_dbg(&cxlr->dev, "%s: expected auto position, not %d\n", 1294 1294 + dev_name(&cxled->cxld.dev), pos); 1295 1295 + return -EINVAL; 1296 1296 + } 1297 1297 + 1298 1298 + if (p->nr_targets >= p->interleave_ways) { 1299 1299 + dev_err(&cxlr->dev, "%s: no more target slots available\n", 1300 1300 + dev_name(&cxled->cxld.dev)); 1301 1301 + return -ENXIO; 1302 1302 + } 1303 1303 + 1304 1304 + /* 1305 1305 + * Temporarily record the endpoint decoder into the target array. Yes, 1306 1306 + * this means that userspace can view devices in the wrong position 1307 1307 + * before the region activates, and must be careful to understand when 1308 1308 + * it might be racing region autodiscovery. 1309 1309 + */ 1310 1310 + pos = p->nr_targets; 1311 1311 + p->targets[pos] = cxled; 1312 1312 + cxled->pos = pos; 1313 1313 + p->nr_targets++; 1314 1314 + 1315 1315 + return 0; 1316 1316 + } 1317 1317 + 1318 1318 + static struct cxl_port *next_port(struct cxl_port *port) 1319 1319 + { 1320 1320 + if (!port->parent_dport) 1321 1321 + return NULL; 1322 1322 + return port->parent_dport->port; 1323 1323 + } 1324 1324 + 1325 1325 + static int decoder_match_range(struct device *dev, void *data) 1326 1326 + { 1327 1327 + struct cxl_endpoint_decoder *cxled = data; 1328 1328 + struct cxl_switch_decoder *cxlsd; 1329 1329 + 1330 1330 + if (!is_switch_decoder(dev)) 1331 1331 + return 0; 1332 1332 + 1333 1333 + cxlsd = to_cxl_switch_decoder(dev); 1334 1334 + return range_contains(&cxlsd->cxld.hpa_range, &cxled->cxld.hpa_range); 1335 1335 + } 1336 1336 + 1337 1337 + static void find_positions(const struct cxl_switch_decoder *cxlsd, 1338 1338 + const struct cxl_port *iter_a, 1339 1339 + const struct cxl_port *iter_b, int *a_pos, 1340 1340 + int *b_pos) 1341 1341 + { 1342 1342 + int i; 1343 1343 + 1344 1344 + for (i = 0, *a_pos = -1, *b_pos = -1; i < cxlsd->nr_targets; i++) { 1345 1345 + if (cxlsd->target[i] == iter_a->parent_dport) 1346 1346 + *a_pos = i; 1347 1347 + else if (cxlsd->target[i] == iter_b->parent_dport) 1348 1348 + *b_pos = i; 1349 1349 + if (*a_pos >= 0 && *b_pos >= 0) 1350 1350 + break; 1351 1351 + } 1352 1352 + } 1353 1353 + 1354 1354 + static int cmp_decode_pos(const void *a, const void *b) 1355 1355 + { 1356 1356 + struct cxl_endpoint_decoder *cxled_a = *(typeof(cxled_a) *)a; 1357 1357 + struct cxl_endpoint_decoder *cxled_b = *(typeof(cxled_b) *)b; 1358 1358 + struct cxl_memdev *cxlmd_a = cxled_to_memdev(cxled_a); 1359 1359 + struct cxl_memdev *cxlmd_b = cxled_to_memdev(cxled_b); 1360 1360 + struct cxl_port *port_a = cxled_to_port(cxled_a); 1361 1361 + struct cxl_port *port_b = cxled_to_port(cxled_b); 1362 1362 + struct cxl_port *iter_a, *iter_b, *port = NULL; 1363 1363 + struct cxl_switch_decoder *cxlsd; 1364 1364 + struct device *dev; 1365 1365 + int a_pos, b_pos; 1366 1366 + unsigned int seq; 1367 1367 + 1368 1368 + /* Exit early if any prior sorting failed */ 1369 1369 + if (cxled_a->pos < 0 || cxled_b->pos < 0) 1370 1370 + return 0; 1371 1371 + 1372 1372 + /* 1373 1373 + * Walk up the hierarchy to find a shared port, find the decoder that 1374 1374 + * maps the range, compare the relative position of those dport 1375 1375 + * mappings. 1376 1376 + */ 1377 1377 + for (iter_a = port_a; iter_a; iter_a = next_port(iter_a)) { 1378 1378 + struct cxl_port *next_a, *next_b; 1379 1379 + 1380 1380 + next_a = next_port(iter_a); 1381 1381 + if (!next_a) 1382 1382 + break; 1383 1383 + 1384 1384 + for (iter_b = port_b; iter_b; iter_b = next_port(iter_b)) { 1385 1385 + next_b = next_port(iter_b); 1386 1386 + if (next_a != next_b) 1387 1387 + continue; 1388 1388 + port = next_a; 1389 1389 + break; 1390 1390 + } 1391 1391 + 1392 1392 + if (port) 1393 1393 + break; 1394 1394 + } 1395 1395 + 1396 1396 + if (!port) { 1397 1397 + dev_err(cxlmd_a->dev.parent, 1398 1398 + "failed to find shared port with %s\n", 1399 1399 + dev_name(cxlmd_b->dev.parent)); 1400 1400 + goto err; 1401 1401 + } 1402 1402 + 1403 1403 + dev = device_find_child(&port->dev, cxled_a, decoder_match_range); 1404 1404 + if (!dev) { 1405 1405 + struct range *range = &cxled_a->cxld.hpa_range; 1406 1406 + 1407 1407 + dev_err(port->uport, 1408 1408 + "failed to find decoder that maps %#llx-%#llx\n", 1409 1409 + range->start, range->end); 1410 1410 + goto err; 1411 1411 + } 1412 1412 + 1413 1413 + cxlsd = to_cxl_switch_decoder(dev); 1414 1414 + do { 1415 1415 + seq = read_seqbegin(&cxlsd->target_lock); 1416 1416 + find_positions(cxlsd, iter_a, iter_b, &a_pos, &b_pos); 1417 1417 + } while (read_seqretry(&cxlsd->target_lock, seq)); 1418 1418 + 1419 1419 + put_device(dev); 1420 1420 + 1421 1421 + if (a_pos < 0 || b_pos < 0) { 1422 1422 + dev_err(port->uport, 1423 1423 + "failed to find shared decoder for %s and %s\n", 1424 1424 + dev_name(cxlmd_a->dev.parent), 1425 1425 + dev_name(cxlmd_b->dev.parent)); 1426 1426 + goto err; 1427 1427 + } 1428 1428 + 1429 1429 + dev_dbg(port->uport, "%s comes %s %s\n", dev_name(cxlmd_a->dev.parent), 1430 1430 + a_pos - b_pos < 0 ? "before" : "after", 1431 1431 + dev_name(cxlmd_b->dev.parent)); 1432 1432 + 1433 1433 + return a_pos - b_pos; 1434 1434 + err: 1435 1435 + cxled_a->pos = -1; 1436 1436 + return 0; 1437 1437 + } 1438 1438 + 1439 1439 + static int cxl_region_sort_targets(struct cxl_region *cxlr) 1440 1440 + { 1441 1441 + struct cxl_region_params *p = &cxlr->params; 1442 1442 + int i, rc = 0; 1443 1443 + 1444 1444 + sort(p->targets, p->nr_targets, sizeof(p->targets[0]), cmp_decode_pos, 1445 1445 + NULL); 1446 1446 + 1447 1447 + for (i = 0; i < p->nr_targets; i++) { 1448 1448 + struct cxl_endpoint_decoder *cxled = p->targets[i]; 1449 1449 + 1450 1450 + /* 1451 1451 + * Record that sorting failed, but still continue to restore 1452 1452 + * cxled->pos with its ->targets[] position so that follow-on 1453 1453 + * code paths can reliably do p->targets[cxled->pos] to 1454 1454 + * self-reference their entry. 1455 1455 + */ 1456 1456 + if (cxled->pos < 0) 1457 1457 + rc = -ENXIO; 1458 1458 + cxled->pos = i; 1459 1459 + } 1460 1460 + 1461 1461 + dev_dbg(&cxlr->dev, "region sort %s\n", rc ? "failed" : "successful"); 1462 1462 + return rc; 1463 1463 + } 1464 1464 + 1465 1465 + static int cxl_region_attach(struct cxl_region *cxlr, 1466 1466 + struct cxl_endpoint_decoder *cxled, int pos) 1467 1467 + { 1468 1468 + struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(cxlr->dev.parent); 1469 1469 + struct cxl_memdev *cxlmd = cxled_to_memdev(cxled); 1470 1470 + struct cxl_region_params *p = &cxlr->params; 1471 1471 + struct cxl_port *ep_port, *root_port; 1472 1472 + struct cxl_dport *dport; 1473 1473 + int rc = -ENXIO; 1474 1474 + 1475 1475 + if (cxled->mode != cxlr->mode) { 1476 1476 + dev_dbg(&cxlr->dev, "%s region mode: %d mismatch: %d\n", 1477 1477 + dev_name(&cxled->cxld.dev), cxlr->mode, cxled->mode); 1478 1478 + return -EINVAL; 1479 1479 + } 1480 1480 + 1481 1481 + if (cxled->mode == CXL_DECODER_DEAD) { 1482 1482 + dev_dbg(&cxlr->dev, "%s dead\n", dev_name(&cxled->cxld.dev)); 1483 1483 + return -ENODEV; 1484 1484 + } 1485 1485 + 1486 1486 + /* all full of members, or interleave config not established? */ 1487 1487 + if (p->state > CXL_CONFIG_INTERLEAVE_ACTIVE) { 1488 1488 + dev_dbg(&cxlr->dev, "region already active\n"); 1489 1489 + return -EBUSY; 1490 1490 + } else if (p->state < CXL_CONFIG_INTERLEAVE_ACTIVE) { 1491 1491 + dev_dbg(&cxlr->dev, "interleave config missing\n"); 1492 1492 + return -ENXIO; 1493 1493 + } 1494 1494 + 1307 1495 ep_port = cxled_to_port(cxled); 1308 1496 root_port = cxlrd_to_port(cxlrd); 1309 1497 dport = cxl_find_dport_by_dev(root_port, ep_port->host_bridge); ··· 1561 1249 dev_dbg(&cxlr->dev, "%s:%s invalid target for %s\n", 1562 1250 dev_name(&cxlmd->dev), dev_name(&cxled->cxld.dev), 1563 1251 dev_name(cxlr->dev.parent)); 1564 1564 - return -ENXIO; 1565 1565 - } 1566 1566 - 1567 1567 - if (cxlrd->calc_hb(cxlrd, pos) != dport) { 1568 1568 - dev_dbg(&cxlr->dev, "%s:%s invalid target position for %s\n", 1569 1569 - dev_name(&cxlmd->dev), dev_name(&cxled->cxld.dev), 1570 1570 - dev_name(&cxlrd->cxlsd.cxld.dev)); 1571 1252 return -ENXIO; 1572 1253 } 1573 1254 ··· 1587 1282 return -EINVAL; 1588 1283 } 1589 1284 1590 1590 - for (iter = ep_port; !is_cxl_root(iter); 1591 1591 - iter = to_cxl_port(iter->dev.parent)) { 1592 1592 - rc = cxl_port_attach_region(iter, cxlr, cxled, pos); 1285 1285 + if (test_bit(CXL_REGION_F_AUTO, &cxlr->flags)) { 1286 1286 + int i; 1287 1287 + 1288 1288 + rc = cxl_region_attach_auto(cxlr, cxled, pos); 1593 1289 if (rc) 1594 1594 - goto err; 1290 1290 + return rc; 1291 1291 + 1292 1292 + /* await more targets to arrive... */ 1293 1293 + if (p->nr_targets < p->interleave_ways) 1294 1294 + return 0; 1295 1295 + 1296 1296 + /* 1297 1297 + * All targets are here, which implies all PCI enumeration that 1298 1298 + * affects this region has been completed. Walk the topology to 1299 1299 + * sort the devices into their relative region decode position. 1300 1300 + */ 1301 1301 + rc = cxl_region_sort_targets(cxlr); 1302 1302 + if (rc) 1303 1303 + return rc; 1304 1304 + 1305 1305 + for (i = 0; i < p->nr_targets; i++) { 1306 1306 + cxled = p->targets[i]; 1307 1307 + ep_port = cxled_to_port(cxled); 1308 1308 + dport = cxl_find_dport_by_dev(root_port, 1309 1309 + ep_port->host_bridge); 1310 1310 + rc = cxl_region_attach_position(cxlr, cxlrd, cxled, 1311 1311 + dport, i); 1312 1312 + if (rc) 1313 1313 + return rc; 1314 1314 + } 1315 1315 + 1316 1316 + rc = cxl_region_setup_targets(cxlr); 1317 1317 + if (rc) 1318 1318 + return rc; 1319 1319 + 1320 1320 + /* 1321 1321 + * If target setup succeeds in the autodiscovery case 1322 1322 + * then the region is already committed. 1323 1323 + */ 1324 1324 + p->state = CXL_CONFIG_COMMIT; 1325 1325 + 1326 1326 + return 0; 1595 1327 } 1328 1328 + 1329 1329 + rc = cxl_region_validate_position(cxlr, cxled, pos); 1330 1330 + if (rc) 1331 1331 + return rc; 1332 1332 + 1333 1333 + rc = cxl_region_attach_position(cxlr, cxlrd, cxled, dport, pos); 1334 1334 + if (rc) 1335 1335 + return rc; 1596 1336 1597 1337 p->targets[pos] = cxled; 1598 1338 cxled->pos = pos; ··· 1661 1311 1662 1312 err_decrement: 1663 1313 p->nr_targets--; 1664 1664 - err: 1665 1665 - for (iter = ep_port; !is_cxl_root(iter); 1666 1666 - iter = to_cxl_port(iter->dev.parent)) 1667 1667 - cxl_port_detach_region(iter, cxlr, cxled); 1314 1314 + cxled->pos = -1; 1315 1315 + p->targets[pos] = NULL; 1668 1316 return rc; 1669 1317 } 1670 1318 ··· 1734 1386 up_write(&cxl_region_rwsem); 1735 1387 } 1736 1388 1737 1737 - static int attach_target(struct cxl_region *cxlr, const char *decoder, int pos) 1389 1389 + static int attach_target(struct cxl_region *cxlr, 1390 1390 + struct cxl_endpoint_decoder *cxled, int pos, 1391 1391 + unsigned int state) 1738 1392 { 1739 1739 - struct device *dev; 1740 1740 - int rc; 1393 1393 + int rc = 0; 1741 1394 1742 1742 - dev = bus_find_device_by_name(&cxl_bus_type, NULL, decoder); 1743 1743 - if (!dev) 1744 1744 - return -ENODEV; 1745 1745 - 1746 1746 - if (!is_endpoint_decoder(dev)) { 1747 1747 - put_device(dev); 1748 1748 - return -EINVAL; 1749 1749 - } 1750 1750 - 1751 1751 - rc = down_write_killable(&cxl_region_rwsem); 1395 1395 + if (state == TASK_INTERRUPTIBLE) 1396 1396 + rc = down_write_killable(&cxl_region_rwsem); 1397 1397 + else 1398 1398 + down_write(&cxl_region_rwsem); 1752 1399 if (rc) 1753 1753 - goto out; 1400 1400 + return rc; 1401 1401 + 1754 1402 down_read(&cxl_dpa_rwsem); 1755 1755 - rc = cxl_region_attach(cxlr, to_cxl_endpoint_decoder(dev), pos); 1403 1403 + rc = cxl_region_attach(cxlr, cxled, pos); 1756 1404 if (rc == 0) 1757 1405 set_bit(CXL_REGION_F_INCOHERENT, &cxlr->flags); 1758 1406 up_read(&cxl_dpa_rwsem); 1759 1407 up_write(&cxl_region_rwsem); 1760 1760 - out: 1761 1761 - put_device(dev); 1762 1408 return rc; 1763 1409 } 1764 1410 ··· 1790 1448 1791 1449 if (sysfs_streq(buf, "\n")) 1792 1450 rc = detach_target(cxlr, pos); 1793 1793 - else 1794 1794 - rc = attach_target(cxlr, buf, pos); 1451 1451 + else { 1452 1452 + struct device *dev; 1453 1453 + 1454 1454 + dev = bus_find_device_by_name(&cxl_bus_type, NULL, buf); 1455 1455 + if (!dev) 1456 1456 + return -ENODEV; 1457 1457 + 1458 1458 + if (!is_endpoint_decoder(dev)) { 1459 1459 + rc = -EINVAL; 1460 1460 + goto out; 1461 1461 + } 1462 1462 + 1463 1463 + rc = attach_target(cxlr, to_cxl_endpoint_decoder(dev), pos, 1464 1464 + TASK_INTERRUPTIBLE); 1465 1465 + out: 1466 1466 + put_device(dev); 1467 1467 + } 1795 1468 1796 1469 if (rc < 0) 1797 1470 return rc; ··· 2010 1653 struct device *dev; 2011 1654 int rc; 2012 1655 1656 1656 + switch (mode) { 1657 1657 + case CXL_DECODER_RAM: 1658 1658 + case CXL_DECODER_PMEM: 1659 1659 + break; 1660 1660 + default: 1661 1661 + dev_err(&cxlrd->cxlsd.cxld.dev, "unsupported mode %d\n", mode); 1662 1662 + return ERR_PTR(-EINVAL); 1663 1663 + } 1664 1664 + 2013 1665 cxlr = cxl_region_alloc(cxlrd, id); 2014 1666 if (IS_ERR(cxlr)) 2015 1667 return cxlr; ··· 2047 1681 return ERR_PTR(rc); 2048 1682 } 2049 1683 1684 1684 + static ssize_t __create_region_show(struct cxl_root_decoder *cxlrd, char *buf) 1685 1685 + { 1686 1686 + return sysfs_emit(buf, "region%u\n", atomic_read(&cxlrd->region_id)); 1687 1687 + } 1688 1688 + 2050 1689 static ssize_t create_pmem_region_show(struct device *dev, 2051 1690 struct device_attribute *attr, char *buf) 2052 1691 { 2053 2053 - struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(dev); 1692 1692 + return __create_region_show(to_cxl_root_decoder(dev), buf); 1693 1693 + } 2054 1694 2055 2055 - return sysfs_emit(buf, "region%u\n", atomic_read(&cxlrd->region_id)); 1695 1695 + static ssize_t create_ram_region_show(struct device *dev, 1696 1696 + struct device_attribute *attr, char *buf) 1697 1697 + { 1698 1698 + return __create_region_show(to_cxl_root_decoder(dev), buf); 1699 1699 + } 1700 1700 + 1701 1701 + static struct cxl_region *__create_region(struct cxl_root_decoder *cxlrd, 1702 1702 + enum cxl_decoder_mode mode, int id) 1703 1703 + { 1704 1704 + int rc; 1705 1705 + 1706 1706 + rc = memregion_alloc(GFP_KERNEL); 1707 1707 + if (rc < 0) 1708 1708 + return ERR_PTR(rc); 1709 1709 + 1710 1710 + if (atomic_cmpxchg(&cxlrd->region_id, id, rc) != id) { 1711 1711 + memregion_free(rc); 1712 1712 + return ERR_PTR(-EBUSY); 1713 1713 + } 1714 1714 + 1715 1715 + return devm_cxl_add_region(cxlrd, id, mode, CXL_DECODER_EXPANDER); 2056 1716 } 2057 1717 2058 1718 static ssize_t create_pmem_region_store(struct device *dev, ··· 2087 1695 { 2088 1696 struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(dev); 2089 1697 struct cxl_region *cxlr; 2090 2090 - int id, rc; 1698 1698 + int rc, id; 2091 1699 2092 1700 rc = sscanf(buf, "region%d\n", &id); 2093 1701 if (rc != 1) 2094 1702 return -EINVAL; 2095 1703 2096 2096 - rc = memregion_alloc(GFP_KERNEL); 2097 2097 - if (rc < 0) 2098 2098 - return rc; 2099 2099 - 2100 2100 - if (atomic_cmpxchg(&cxlrd->region_id, id, rc) != id) { 2101 2101 - memregion_free(rc); 2102 2102 - return -EBUSY; 2103 2103 - } 2104 2104 - 2105 2105 - cxlr = devm_cxl_add_region(cxlrd, id, CXL_DECODER_PMEM, 2106 2106 - CXL_DECODER_EXPANDER); 1704 1704 + cxlr = __create_region(cxlrd, CXL_DECODER_PMEM, id); 2107 1705 if (IS_ERR(cxlr)) 2108 1706 return PTR_ERR(cxlr); 2109 1707 2110 1708 return len; 2111 1709 } 2112 1710 DEVICE_ATTR_RW(create_pmem_region); 1711 1711 + 1712 1712 + static ssize_t create_ram_region_store(struct device *dev, 1713 1713 + struct device_attribute *attr, 1714 1714 + const char *buf, size_t len) 1715 1715 + { 1716 1716 + struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(dev); 1717 1717 + struct cxl_region *cxlr; 1718 1718 + int rc, id; 1719 1719 + 1720 1720 + rc = sscanf(buf, "region%d\n", &id); 1721 1721 + if (rc != 1) 1722 1722 + return -EINVAL; 1723 1723 + 1724 1724 + cxlr = __create_region(cxlrd, CXL_DECODER_RAM, id); 1725 1725 + if (IS_ERR(cxlr)) 1726 1726 + return PTR_ERR(cxlr); 1727 1727 + 1728 1728 + return len; 1729 1729 + } 1730 1730 + DEVICE_ATTR_RW(create_ram_region); 2113 1731 2114 1732 static ssize_t region_show(struct device *dev, struct device_attribute *attr, 2115 1733 char *buf) ··· 2280 1878 return cxlr_pmem; 2281 1879 } 2282 1880 1881 1881 + static void cxl_dax_region_release(struct device *dev) 1882 1882 + { 1883 1883 + struct cxl_dax_region *cxlr_dax = to_cxl_dax_region(dev); 1884 1884 + 1885 1885 + kfree(cxlr_dax); 1886 1886 + } 1887 1887 + 1888 1888 + static const struct attribute_group *cxl_dax_region_attribute_groups[] = { 1889 1889 + &cxl_base_attribute_group, 1890 1890 + NULL, 1891 1891 + }; 1892 1892 + 1893 1893 + const struct device_type cxl_dax_region_type = { 1894 1894 + .name = "cxl_dax_region", 1895 1895 + .release = cxl_dax_region_release, 1896 1896 + .groups = cxl_dax_region_attribute_groups, 1897 1897 + }; 1898 1898 + 1899 1899 + static bool is_cxl_dax_region(struct device *dev) 1900 1900 + { 1901 1901 + return dev->type == &cxl_dax_region_type; 1902 1902 + } 1903 1903 + 1904 1904 + struct cxl_dax_region *to_cxl_dax_region(struct device *dev) 1905 1905 + { 1906 1906 + if (dev_WARN_ONCE(dev, !is_cxl_dax_region(dev), 1907 1907 + "not a cxl_dax_region device\n")) 1908 1908 + return NULL; 1909 1909 + return container_of(dev, struct cxl_dax_region, dev); 1910 1910 + } 1911 1911 + EXPORT_SYMBOL_NS_GPL(to_cxl_dax_region, CXL); 1912 1912 + 1913 1913 + static struct lock_class_key cxl_dax_region_key; 1914 1914 + 1915 1915 + static struct cxl_dax_region *cxl_dax_region_alloc(struct cxl_region *cxlr) 1916 1916 + { 1917 1917 + struct cxl_region_params *p = &cxlr->params; 1918 1918 + struct cxl_dax_region *cxlr_dax; 1919 1919 + struct device *dev; 1920 1920 + 1921 1921 + down_read(&cxl_region_rwsem); 1922 1922 + if (p->state != CXL_CONFIG_COMMIT) { 1923 1923 + cxlr_dax = ERR_PTR(-ENXIO); 1924 1924 + goto out; 1925 1925 + } 1926 1926 + 1927 1927 + cxlr_dax = kzalloc(sizeof(*cxlr_dax), GFP_KERNEL); 1928 1928 + if (!cxlr_dax) { 1929 1929 + cxlr_dax = ERR_PTR(-ENOMEM); 1930 1930 + goto out; 1931 1931 + } 1932 1932 + 1933 1933 + cxlr_dax->hpa_range.start = p->res->start; 1934 1934 + cxlr_dax->hpa_range.end = p->res->end; 1935 1935 + 1936 1936 + dev = &cxlr_dax->dev; 1937 1937 + cxlr_dax->cxlr = cxlr; 1938 1938 + device_initialize(dev); 1939 1939 + lockdep_set_class(&dev->mutex, &cxl_dax_region_key); 1940 1940 + device_set_pm_not_required(dev); 1941 1941 + dev->parent = &cxlr->dev; 1942 1942 + dev->bus = &cxl_bus_type; 1943 1943 + dev->type = &cxl_dax_region_type; 1944 1944 + out: 1945 1945 + up_read(&cxl_region_rwsem); 1946 1946 + 1947 1947 + return cxlr_dax; 1948 1948 + } 1949 1949 + 2283 1950 static void cxlr_pmem_unregister(void *_cxlr_pmem) 2284 1951 { 2285 1952 struct cxl_pmem_region *cxlr_pmem = _cxlr_pmem; ··· 2433 1962 return rc; 2434 1963 } 2435 1964 1965 1965 + static void cxlr_dax_unregister(void *_cxlr_dax) 1966 1966 + { 1967 1967 + struct cxl_dax_region *cxlr_dax = _cxlr_dax; 1968 1968 + 1969 1969 + device_unregister(&cxlr_dax->dev); 1970 1970 + } 1971 1971 + 1972 1972 + static int devm_cxl_add_dax_region(struct cxl_region *cxlr) 1973 1973 + { 1974 1974 + struct cxl_dax_region *cxlr_dax; 1975 1975 + struct device *dev; 1976 1976 + int rc; 1977 1977 + 1978 1978 + cxlr_dax = cxl_dax_region_alloc(cxlr); 1979 1979 + if (IS_ERR(cxlr_dax)) 1980 1980 + return PTR_ERR(cxlr_dax); 1981 1981 + 1982 1982 + dev = &cxlr_dax->dev; 1983 1983 + rc = dev_set_name(dev, "dax_region%d", cxlr->id); 1984 1984 + if (rc) 1985 1985 + goto err; 1986 1986 + 1987 1987 + rc = device_add(dev); 1988 1988 + if (rc) 1989 1989 + goto err; 1990 1990 + 1991 1991 + dev_dbg(&cxlr->dev, "%s: register %s\n", dev_name(dev->parent), 1992 1992 + dev_name(dev)); 1993 1993 + 1994 1994 + return devm_add_action_or_reset(&cxlr->dev, cxlr_dax_unregister, 1995 1995 + cxlr_dax); 1996 1996 + err: 1997 1997 + put_device(dev); 1998 1998 + return rc; 1999 1999 + } 2000 2000 + 2001 2001 + static int match_decoder_by_range(struct device *dev, void *data) 2002 2002 + { 2003 2003 + struct range *r1, *r2 = data; 2004 2004 + struct cxl_root_decoder *cxlrd; 2005 2005 + 2006 2006 + if (!is_root_decoder(dev)) 2007 2007 + return 0; 2008 2008 + 2009 2009 + cxlrd = to_cxl_root_decoder(dev); 2010 2010 + r1 = &cxlrd->cxlsd.cxld.hpa_range; 2011 2011 + return range_contains(r1, r2); 2012 2012 + } 2013 2013 + 2014 2014 + static int match_region_by_range(struct device *dev, void *data) 2015 2015 + { 2016 2016 + struct cxl_region_params *p; 2017 2017 + struct cxl_region *cxlr; 2018 2018 + struct range *r = data; 2019 2019 + int rc = 0; 2020 2020 + 2021 2021 + if (!is_cxl_region(dev)) 2022 2022 + return 0; 2023 2023 + 2024 2024 + cxlr = to_cxl_region(dev); 2025 2025 + p = &cxlr->params; 2026 2026 + 2027 2027 + down_read(&cxl_region_rwsem); 2028 2028 + if (p->res && p->res->start == r->start && p->res->end == r->end) 2029 2029 + rc = 1; 2030 2030 + up_read(&cxl_region_rwsem); 2031 2031 + 2032 2032 + return rc; 2033 2033 + } 2034 2034 + 2035 2035 + /* Establish an empty region covering the given HPA range */ 2036 2036 + static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd, 2037 2037 + struct cxl_endpoint_decoder *cxled) 2038 2038 + { 2039 2039 + struct cxl_memdev *cxlmd = cxled_to_memdev(cxled); 2040 2040 + struct cxl_port *port = cxlrd_to_port(cxlrd); 2041 2041 + struct range *hpa = &cxled->cxld.hpa_range; 2042 2042 + struct cxl_region_params *p; 2043 2043 + struct cxl_region *cxlr; 2044 2044 + struct resource *res; 2045 2045 + int rc; 2046 2046 + 2047 2047 + do { 2048 2048 + cxlr = __create_region(cxlrd, cxled->mode, 2049 2049 + atomic_read(&cxlrd->region_id)); 2050 2050 + } while (IS_ERR(cxlr) && PTR_ERR(cxlr) == -EBUSY); 2051 2051 + 2052 2052 + if (IS_ERR(cxlr)) { 2053 2053 + dev_err(cxlmd->dev.parent, 2054 2054 + "%s:%s: %s failed assign region: %ld\n", 2055 2055 + dev_name(&cxlmd->dev), dev_name(&cxled->cxld.dev), 2056 2056 + __func__, PTR_ERR(cxlr)); 2057 2057 + return cxlr; 2058 2058 + } 2059 2059 + 2060 2060 + down_write(&cxl_region_rwsem); 2061 2061 + p = &cxlr->params; 2062 2062 + if (p->state >= CXL_CONFIG_INTERLEAVE_ACTIVE) { 2063 2063 + dev_err(cxlmd->dev.parent, 2064 2064 + "%s:%s: %s autodiscovery interrupted\n", 2065 2065 + dev_name(&cxlmd->dev), dev_name(&cxled->cxld.dev), 2066 2066 + __func__); 2067 2067 + rc = -EBUSY; 2068 2068 + goto err; 2069 2069 + } 2070 2070 + 2071 2071 + set_bit(CXL_REGION_F_AUTO, &cxlr->flags); 2072 2072 + 2073 2073 + res = kmalloc(sizeof(*res), GFP_KERNEL); 2074 2074 + if (!res) { 2075 2075 + rc = -ENOMEM; 2076 2076 + goto err; 2077 2077 + } 2078 2078 + 2079 2079 + *res = DEFINE_RES_MEM_NAMED(hpa->start, range_len(hpa), 2080 2080 + dev_name(&cxlr->dev)); 2081 2081 + rc = insert_resource(cxlrd->res, res); 2082 2082 + if (rc) { 2083 2083 + /* 2084 2084 + * Platform-firmware may not have split resources like "System 2085 2085 + * RAM" on CXL window boundaries see cxl_region_iomem_release() 2086 2086 + */ 2087 2087 + dev_warn(cxlmd->dev.parent, 2088 2088 + "%s:%s: %s %s cannot insert resource\n", 2089 2089 + dev_name(&cxlmd->dev), dev_name(&cxled->cxld.dev), 2090 2090 + __func__, dev_name(&cxlr->dev)); 2091 2091 + } 2092 2092 + 2093 2093 + p->res = res; 2094 2094 + p->interleave_ways = cxled->cxld.interleave_ways; 2095 2095 + p->interleave_granularity = cxled->cxld.interleave_granularity; 2096 2096 + p->state = CXL_CONFIG_INTERLEAVE_ACTIVE; 2097 2097 + 2098 2098 + rc = sysfs_update_group(&cxlr->dev.kobj, get_cxl_region_target_group()); 2099 2099 + if (rc) 2100 2100 + goto err; 2101 2101 + 2102 2102 + dev_dbg(cxlmd->dev.parent, "%s:%s: %s %s res: %pr iw: %d ig: %d\n", 2103 2103 + dev_name(&cxlmd->dev), dev_name(&cxled->cxld.dev), __func__, 2104 2104 + dev_name(&cxlr->dev), p->res, p->interleave_ways, 2105 2105 + p->interleave_granularity); 2106 2106 + 2107 2107 + /* ...to match put_device() in cxl_add_to_region() */ 2108 2108 + get_device(&cxlr->dev); 2109 2109 + up_write(&cxl_region_rwsem); 2110 2110 + 2111 2111 + return cxlr; 2112 2112 + 2113 2113 + err: 2114 2114 + up_write(&cxl_region_rwsem); 2115 2115 + devm_release_action(port->uport, unregister_region, cxlr); 2116 2116 + return ERR_PTR(rc); 2117 2117 + } 2118 2118 + 2119 2119 + int cxl_add_to_region(struct cxl_port *root, struct cxl_endpoint_decoder *cxled) 2120 2120 + { 2121 2121 + struct cxl_memdev *cxlmd = cxled_to_memdev(cxled); 2122 2122 + struct range *hpa = &cxled->cxld.hpa_range; 2123 2123 + struct cxl_decoder *cxld = &cxled->cxld; 2124 2124 + struct device *cxlrd_dev, *region_dev; 2125 2125 + struct cxl_root_decoder *cxlrd; 2126 2126 + struct cxl_region_params *p; 2127 2127 + struct cxl_region *cxlr; 2128 2128 + bool attach = false; 2129 2129 + int rc; 2130 2130 + 2131 2131 + cxlrd_dev = device_find_child(&root->dev, &cxld->hpa_range, 2132 2132 + match_decoder_by_range); 2133 2133 + if (!cxlrd_dev) { 2134 2134 + dev_err(cxlmd->dev.parent, 2135 2135 + "%s:%s no CXL window for range %#llx:%#llx\n", 2136 2136 + dev_name(&cxlmd->dev), dev_name(&cxld->dev), 2137 2137 + cxld->hpa_range.start, cxld->hpa_range.end); 2138 2138 + return -ENXIO; 2139 2139 + } 2140 2140 + 2141 2141 + cxlrd = to_cxl_root_decoder(cxlrd_dev); 2142 2142 + 2143 2143 + /* 2144 2144 + * Ensure that if multiple threads race to construct_region() for @hpa 2145 2145 + * one does the construction and the others add to that. 2146 2146 + */ 2147 2147 + mutex_lock(&cxlrd->range_lock); 2148 2148 + region_dev = device_find_child(&cxlrd->cxlsd.cxld.dev, hpa, 2149 2149 + match_region_by_range); 2150 2150 + if (!region_dev) { 2151 2151 + cxlr = construct_region(cxlrd, cxled); 2152 2152 + region_dev = &cxlr->dev; 2153 2153 + } else 2154 2154 + cxlr = to_cxl_region(region_dev); 2155 2155 + mutex_unlock(&cxlrd->range_lock); 2156 2156 + 2157 2157 + rc = PTR_ERR_OR_ZERO(cxlr); 2158 2158 + if (rc) 2159 2159 + goto out; 2160 2160 + 2161 2161 + attach_target(cxlr, cxled, -1, TASK_UNINTERRUPTIBLE); 2162 2162 + 2163 2163 + down_read(&cxl_region_rwsem); 2164 2164 + p = &cxlr->params; 2165 2165 + attach = p->state == CXL_CONFIG_COMMIT; 2166 2166 + up_read(&cxl_region_rwsem); 2167 2167 + 2168 2168 + if (attach) { 2169 2169 + /* 2170 2170 + * If device_attach() fails the range may still be active via 2171 2171 + * the platform-firmware memory map, otherwise the driver for 2172 2172 + * regions is local to this file, so driver matching can't fail. 2173 2173 + */ 2174 2174 + if (device_attach(&cxlr->dev) < 0) 2175 2175 + dev_err(&cxlr->dev, "failed to enable, range: %pr\n", 2176 2176 + p->res); 2177 2177 + } 2178 2178 + 2179 2179 + put_device(region_dev); 2180 2180 + out: 2181 2181 + put_device(cxlrd_dev); 2182 2182 + return rc; 2183 2183 + } 2184 2184 + EXPORT_SYMBOL_NS_GPL(cxl_add_to_region, CXL); 2185 2185 + 2436 2186 static int cxl_region_invalidate_memregion(struct cxl_region *cxlr) 2437 2187 { 2438 2188 if (!test_bit(CXL_REGION_F_INCOHERENT, &cxlr->flags)) ··· 2661 1969 2662 1970 if (!cpu_cache_has_invalidate_memregion()) { 2663 1971 if (IS_ENABLED(CONFIG_CXL_REGION_INVALIDATION_TEST)) { 2664 2664 - dev_warn( 1972 1972 + dev_warn_once( 2665 1973 &cxlr->dev, 2666 1974 "Bypassing cpu_cache_invalidate_memregion() for testing!\n"); 2667 1975 clear_bit(CXL_REGION_F_INCOHERENT, &cxlr->flags); ··· 2676 1984 cpu_cache_invalidate_memregion(IORES_DESC_CXL); 2677 1985 clear_bit(CXL_REGION_F_INCOHERENT, &cxlr->flags); 2678 1986 return 0; 1987 1987 + } 1988 1988 + 1989 1989 + static int is_system_ram(struct resource *res, void *arg) 1990 1990 + { 1991 1991 + struct cxl_region *cxlr = arg; 1992 1992 + struct cxl_region_params *p = &cxlr->params; 1993 1993 + 1994 1994 + dev_dbg(&cxlr->dev, "%pr has System RAM: %pr\n", p->res, res); 1995 1995 + return 1; 2679 1996 } 2680 1997 2681 1998 static int cxl_region_probe(struct device *dev) ··· 2720 2019 switch (cxlr->mode) { 2721 2020 case CXL_DECODER_PMEM: 2722 2021 return devm_cxl_add_pmem_region(cxlr); 2022 2022 + case CXL_DECODER_RAM: 2023 2023 + /* 2024 2024 + * The region can not be manged by CXL if any portion of 2025 2025 + * it is already online as 'System RAM' 2026 2026 + */ 2027 2027 + if (walk_iomem_res_desc(IORES_DESC_NONE, 2028 2028 + IORESOURCE_SYSTEM_RAM | IORESOURCE_BUSY, 2029 2029 + p->res->start, p->res->end, cxlr, 2030 2030 + is_system_ram) > 0) 2031 2031 + return 0; 2032 2032 + return devm_cxl_add_dax_region(cxlr); 2723 2033 default: 2724 2034 dev_dbg(&cxlr->dev, "unsupported region mode: %d\n", 2725 2035 cxlr->mode);

+5

drivers/cxl/core/trace.c

reviewed

··· 1 1 + // SPDX-License-Identifier: GPL-2.0-only 2 2 + /* Copyright(c) 2022 Intel Corporation. All rights reserved. */ 3 3 + 4 4 + #define CREATE_TRACE_POINTS 5 5 + #include "trace.h"

+606

drivers/cxl/core/trace.h

reviewed

··· 1 1 + // SPDX-License-Identifier: GPL-2.0 2 2 + /* Copyright(c) 2022 Intel Corporation. All rights reserved. */ 3 3 + #undef TRACE_SYSTEM 4 4 + #define TRACE_SYSTEM cxl 5 5 + 6 6 + #if !defined(_CXL_EVENTS_H) || defined(TRACE_HEADER_MULTI_READ) 7 7 + #define _CXL_EVENTS_H 8 8 + 9 9 + #include <linux/tracepoint.h> 10 10 + #include <asm-generic/unaligned.h> 11 11 + 12 12 + #include <cxl.h> 13 13 + #include <cxlmem.h> 14 14 + 15 15 + #define CXL_RAS_UC_CACHE_DATA_PARITY BIT(0) 16 16 + #define CXL_RAS_UC_CACHE_ADDR_PARITY BIT(1) 17 17 + #define CXL_RAS_UC_CACHE_BE_PARITY BIT(2) 18 18 + #define CXL_RAS_UC_CACHE_DATA_ECC BIT(3) 19 19 + #define CXL_RAS_UC_MEM_DATA_PARITY BIT(4) 20 20 + #define CXL_RAS_UC_MEM_ADDR_PARITY BIT(5) 21 21 + #define CXL_RAS_UC_MEM_BE_PARITY BIT(6) 22 22 + #define CXL_RAS_UC_MEM_DATA_ECC BIT(7) 23 23 + #define CXL_RAS_UC_REINIT_THRESH BIT(8) 24 24 + #define CXL_RAS_UC_RSVD_ENCODE BIT(9) 25 25 + #define CXL_RAS_UC_POISON BIT(10) 26 26 + #define CXL_RAS_UC_RECV_OVERFLOW BIT(11) 27 27 + #define CXL_RAS_UC_INTERNAL_ERR BIT(14) 28 28 + #define CXL_RAS_UC_IDE_TX_ERR BIT(15) 29 29 + #define CXL_RAS_UC_IDE_RX_ERR BIT(16) 30 30 + 31 31 + #define show_uc_errs(status) __print_flags(status, " | ", \ 32 32 + { CXL_RAS_UC_CACHE_DATA_PARITY, "Cache Data Parity Error" }, \ 33 33 + { CXL_RAS_UC_CACHE_ADDR_PARITY, "Cache Address Parity Error" }, \ 34 34 + { CXL_RAS_UC_CACHE_BE_PARITY, "Cache Byte Enable Parity Error" }, \ 35 35 + { CXL_RAS_UC_CACHE_DATA_ECC, "Cache Data ECC Error" }, \ 36 36 + { CXL_RAS_UC_MEM_DATA_PARITY, "Memory Data Parity Error" }, \ 37 37 + { CXL_RAS_UC_MEM_ADDR_PARITY, "Memory Address Parity Error" }, \ 38 38 + { CXL_RAS_UC_MEM_BE_PARITY, "Memory Byte Enable Parity Error" }, \ 39 39 + { CXL_RAS_UC_MEM_DATA_ECC, "Memory Data ECC Error" }, \ 40 40 + { CXL_RAS_UC_REINIT_THRESH, "REINIT Threshold Hit" }, \ 41 41 + { CXL_RAS_UC_RSVD_ENCODE, "Received Unrecognized Encoding" }, \ 42 42 + { CXL_RAS_UC_POISON, "Received Poison From Peer" }, \ 43 43 + { CXL_RAS_UC_RECV_OVERFLOW, "Receiver Overflow" }, \ 44 44 + { CXL_RAS_UC_INTERNAL_ERR, "Component Specific Error" }, \ 45 45 + { CXL_RAS_UC_IDE_TX_ERR, "IDE Tx Error" }, \ 46 46 + { CXL_RAS_UC_IDE_RX_ERR, "IDE Rx Error" } \ 47 47 + ) 48 48 + 49 49 + TRACE_EVENT(cxl_aer_uncorrectable_error, 50 50 + TP_PROTO(const struct cxl_memdev *cxlmd, u32 status, u32 fe, u32 *hl), 51 51 + TP_ARGS(cxlmd, status, fe, hl), 52 52 + TP_STRUCT__entry( 53 53 + __string(memdev, dev_name(&cxlmd->dev)) 54 54 + __string(host, dev_name(cxlmd->dev.parent)) 55 55 + __field(u64, serial) 56 56 + __field(u32, status) 57 57 + __field(u32, first_error) 58 58 + __array(u32, header_log, CXL_HEADERLOG_SIZE_U32) 59 59 + ), 60 60 + TP_fast_assign( 61 61 + __assign_str(memdev, dev_name(&cxlmd->dev)); 62 62 + __assign_str(host, dev_name(cxlmd->dev.parent)); 63 63 + __entry->serial = cxlmd->cxlds->serial; 64 64 + __entry->status = status; 65 65 + __entry->first_error = fe; 66 66 + /* 67 67 + * Embed the 512B headerlog data for user app retrieval and 68 68 + * parsing, but no need to print this in the trace buffer. 69 69 + */ 70 70 + memcpy(__entry->header_log, hl, CXL_HEADERLOG_SIZE); 71 71 + ), 72 72 + TP_printk("memdev=%s host=%s serial=%lld: status: '%s' first_error: '%s'", 73 73 + __get_str(memdev), __get_str(host), __entry->serial, 74 74 + show_uc_errs(__entry->status), 75 75 + show_uc_errs(__entry->first_error) 76 76 + ) 77 77 + ); 78 78 + 79 79 + #define CXL_RAS_CE_CACHE_DATA_ECC BIT(0) 80 80 + #define CXL_RAS_CE_MEM_DATA_ECC BIT(1) 81 81 + #define CXL_RAS_CE_CRC_THRESH BIT(2) 82 82 + #define CLX_RAS_CE_RETRY_THRESH BIT(3) 83 83 + #define CXL_RAS_CE_CACHE_POISON BIT(4) 84 84 + #define CXL_RAS_CE_MEM_POISON BIT(5) 85 85 + #define CXL_RAS_CE_PHYS_LAYER_ERR BIT(6) 86 86 + 87 87 + #define show_ce_errs(status) __print_flags(status, " | ", \ 88 88 + { CXL_RAS_CE_CACHE_DATA_ECC, "Cache Data ECC Error" }, \ 89 89 + { CXL_RAS_CE_MEM_DATA_ECC, "Memory Data ECC Error" }, \ 90 90 + { CXL_RAS_CE_CRC_THRESH, "CRC Threshold Hit" }, \ 91 91 + { CLX_RAS_CE_RETRY_THRESH, "Retry Threshold" }, \ 92 92 + { CXL_RAS_CE_CACHE_POISON, "Received Cache Poison From Peer" }, \ 93 93 + { CXL_RAS_CE_MEM_POISON, "Received Memory Poison From Peer" }, \ 94 94 + { CXL_RAS_CE_PHYS_LAYER_ERR, "Received Error From Physical Layer" } \ 95 95 + ) 96 96 + 97 97 + TRACE_EVENT(cxl_aer_correctable_error, 98 98 + TP_PROTO(const struct cxl_memdev *cxlmd, u32 status), 99 99 + TP_ARGS(cxlmd, status), 100 100 + TP_STRUCT__entry( 101 101 + __string(memdev, dev_name(&cxlmd->dev)) 102 102 + __string(host, dev_name(cxlmd->dev.parent)) 103 103 + __field(u64, serial) 104 104 + __field(u32, status) 105 105 + ), 106 106 + TP_fast_assign( 107 107 + __assign_str(memdev, dev_name(&cxlmd->dev)); 108 108 + __assign_str(host, dev_name(cxlmd->dev.parent)); 109 109 + __entry->serial = cxlmd->cxlds->serial; 110 110 + __entry->status = status; 111 111 + ), 112 112 + TP_printk("memdev=%s host=%s serial=%lld: status: '%s'", 113 113 + __get_str(memdev), __get_str(host), __entry->serial, 114 114 + show_ce_errs(__entry->status) 115 115 + ) 116 116 + ); 117 117 + 118 118 + #define cxl_event_log_type_str(type) \ 119 119 + __print_symbolic(type, \ 120 120 + { CXL_EVENT_TYPE_INFO, "Informational" }, \ 121 121 + { CXL_EVENT_TYPE_WARN, "Warning" }, \ 122 122 + { CXL_EVENT_TYPE_FAIL, "Failure" }, \ 123 123 + { CXL_EVENT_TYPE_FATAL, "Fatal" }) 124 124 + 125 125 + TRACE_EVENT(cxl_overflow, 126 126 + 127 127 + TP_PROTO(const struct cxl_memdev *cxlmd, enum cxl_event_log_type log, 128 128 + struct cxl_get_event_payload *payload), 129 129 + 130 130 + TP_ARGS(cxlmd, log, payload), 131 131 + 132 132 + TP_STRUCT__entry( 133 133 + __string(memdev, dev_name(&cxlmd->dev)) 134 134 + __string(host, dev_name(cxlmd->dev.parent)) 135 135 + __field(int, log) 136 136 + __field(u64, serial) 137 137 + __field(u64, first_ts) 138 138 + __field(u64, last_ts) 139 139 + __field(u16, count) 140 140 + ), 141 141 + 142 142 + TP_fast_assign( 143 143 + __assign_str(memdev, dev_name(&cxlmd->dev)); 144 144 + __assign_str(host, dev_name(cxlmd->dev.parent)); 145 145 + __entry->serial = cxlmd->cxlds->serial; 146 146 + __entry->log = log; 147 147 + __entry->count = le16_to_cpu(payload->overflow_err_count); 148 148 + __entry->first_ts = le64_to_cpu(payload->first_overflow_timestamp); 149 149 + __entry->last_ts = le64_to_cpu(payload->last_overflow_timestamp); 150 150 + ), 151 151 + 152 152 + TP_printk("memdev=%s host=%s serial=%lld: log=%s : %u records from %llu to %llu", 153 153 + __get_str(memdev), __get_str(host), __entry->serial, 154 154 + cxl_event_log_type_str(__entry->log), __entry->count, 155 155 + __entry->first_ts, __entry->last_ts) 156 156 + 157 157 + ); 158 158 + 159 159 + /* 160 160 + * Common Event Record Format 161 161 + * CXL 3.0 section 8.2.9.2.1; Table 8-42 162 162 + */ 163 163 + #define CXL_EVENT_RECORD_FLAG_PERMANENT BIT(2) 164 164 + #define CXL_EVENT_RECORD_FLAG_MAINT_NEEDED BIT(3) 165 165 + #define CXL_EVENT_RECORD_FLAG_PERF_DEGRADED BIT(4) 166 166 + #define CXL_EVENT_RECORD_FLAG_HW_REPLACE BIT(5) 167 167 + #define show_hdr_flags(flags) __print_flags(flags, " | ", \ 168 168 + { CXL_EVENT_RECORD_FLAG_PERMANENT, "PERMANENT_CONDITION" }, \ 169 169 + { CXL_EVENT_RECORD_FLAG_MAINT_NEEDED, "MAINTENANCE_NEEDED" }, \ 170 170 + { CXL_EVENT_RECORD_FLAG_PERF_DEGRADED, "PERFORMANCE_DEGRADED" }, \ 171 171 + { CXL_EVENT_RECORD_FLAG_HW_REPLACE, "HARDWARE_REPLACEMENT_NEEDED" } \ 172 172 + ) 173 173 + 174 174 + /* 175 175 + * Define macros for the common header of each CXL event. 176 176 + * 177 177 + * Tracepoints using these macros must do 3 things: 178 178 + * 179 179 + * 1) Add CXL_EVT_TP_entry to TP_STRUCT__entry 180 180 + * 2) Use CXL_EVT_TP_fast_assign within TP_fast_assign; 181 181 + * pass the dev, log, and CXL event header 182 182 + * 3) Use CXL_EVT_TP_printk() instead of TP_printk() 183 183 + * 184 184 + * See the generic_event tracepoint as an example. 185 185 + */ 186 186 + #define CXL_EVT_TP_entry \ 187 187 + __string(memdev, dev_name(&cxlmd->dev)) \ 188 188 + __string(host, dev_name(cxlmd->dev.parent)) \ 189 189 + __field(int, log) \ 190 190 + __field_struct(uuid_t, hdr_uuid) \ 191 191 + __field(u64, serial) \ 192 192 + __field(u32, hdr_flags) \ 193 193 + __field(u16, hdr_handle) \ 194 194 + __field(u16, hdr_related_handle) \ 195 195 + __field(u64, hdr_timestamp) \ 196 196 + __field(u8, hdr_length) \ 197 197 + __field(u8, hdr_maint_op_class) 198 198 + 199 199 + #define CXL_EVT_TP_fast_assign(cxlmd, l, hdr) \ 200 200 + __assign_str(memdev, dev_name(&(cxlmd)->dev)); \ 201 201 + __assign_str(host, dev_name((cxlmd)->dev.parent)); \ 202 202 + __entry->log = (l); \ 203 203 + __entry->serial = (cxlmd)->cxlds->serial; \ 204 204 + memcpy(&__entry->hdr_uuid, &(hdr).id, sizeof(uuid_t)); \ 205 205 + __entry->hdr_length = (hdr).length; \ 206 206 + __entry->hdr_flags = get_unaligned_le24((hdr).flags); \ 207 207 + __entry->hdr_handle = le16_to_cpu((hdr).handle); \ 208 208 + __entry->hdr_related_handle = le16_to_cpu((hdr).related_handle); \ 209 209 + __entry->hdr_timestamp = le64_to_cpu((hdr).timestamp); \ 210 210 + __entry->hdr_maint_op_class = (hdr).maint_op_class 211 211 + 212 212 + #define CXL_EVT_TP_printk(fmt, ...) \ 213 213 + TP_printk("memdev=%s host=%s serial=%lld log=%s : time=%llu uuid=%pUb " \ 214 214 + "len=%d flags='%s' handle=%x related_handle=%x " \ 215 215 + "maint_op_class=%u : " fmt, \ 216 216 + __get_str(memdev), __get_str(host), __entry->serial, \ 217 217 + cxl_event_log_type_str(__entry->log), \ 218 218 + __entry->hdr_timestamp, &__entry->hdr_uuid, __entry->hdr_length,\ 219 219 + show_hdr_flags(__entry->hdr_flags), __entry->hdr_handle, \ 220 220 + __entry->hdr_related_handle, __entry->hdr_maint_op_class, \ 221 221 + ##__VA_ARGS__) 222 222 + 223 223 + TRACE_EVENT(cxl_generic_event, 224 224 + 225 225 + TP_PROTO(const struct cxl_memdev *cxlmd, enum cxl_event_log_type log, 226 226 + struct cxl_event_record_raw *rec), 227 227 + 228 228 + TP_ARGS(cxlmd, log, rec), 229 229 + 230 230 + TP_STRUCT__entry( 231 231 + CXL_EVT_TP_entry 232 232 + __array(u8, data, CXL_EVENT_RECORD_DATA_LENGTH) 233 233 + ), 234 234 + 235 235 + TP_fast_assign( 236 236 + CXL_EVT_TP_fast_assign(cxlmd, log, rec->hdr); 237 237 + memcpy(__entry->data, &rec->data, CXL_EVENT_RECORD_DATA_LENGTH); 238 238 + ), 239 239 + 240 240 + CXL_EVT_TP_printk("%s", 241 241 + __print_hex(__entry->data, CXL_EVENT_RECORD_DATA_LENGTH)) 242 242 + ); 243 243 + 244 244 + /* 245 245 + * Physical Address field masks 246 246 + * 247 247 + * General Media Event Record 248 248 + * CXL rev 3.0 Section 8.2.9.2.1.1; Table 8-43 249 249 + * 250 250 + * DRAM Event Record 251 251 + * CXL rev 3.0 section 8.2.9.2.1.2; Table 8-44 252 252 + */ 253 253 + #define CXL_DPA_FLAGS_MASK 0x3F 254 254 + #define CXL_DPA_MASK (~CXL_DPA_FLAGS_MASK) 255 255 + 256 256 + #define CXL_DPA_VOLATILE BIT(0) 257 257 + #define CXL_DPA_NOT_REPAIRABLE BIT(1) 258 258 + #define show_dpa_flags(flags) __print_flags(flags, "|", \ 259 259 + { CXL_DPA_VOLATILE, "VOLATILE" }, \ 260 260 + { CXL_DPA_NOT_REPAIRABLE, "NOT_REPAIRABLE" } \ 261 261 + ) 262 262 + 263 263 + /* 264 264 + * General Media Event Record - GMER 265 265 + * CXL rev 3.0 Section 8.2.9.2.1.1; Table 8-43 266 266 + */ 267 267 + #define CXL_GMER_EVT_DESC_UNCORECTABLE_EVENT BIT(0) 268 268 + #define CXL_GMER_EVT_DESC_THRESHOLD_EVENT BIT(1) 269 269 + #define CXL_GMER_EVT_DESC_POISON_LIST_OVERFLOW BIT(2) 270 270 + #define show_event_desc_flags(flags) __print_flags(flags, "|", \ 271 271 + { CXL_GMER_EVT_DESC_UNCORECTABLE_EVENT, "UNCORRECTABLE_EVENT" }, \ 272 272 + { CXL_GMER_EVT_DESC_THRESHOLD_EVENT, "THRESHOLD_EVENT" }, \ 273 273 + { CXL_GMER_EVT_DESC_POISON_LIST_OVERFLOW, "POISON_LIST_OVERFLOW" } \ 274 274 + ) 275 275 + 276 276 + #define CXL_GMER_MEM_EVT_TYPE_ECC_ERROR 0x00 277 277 + #define CXL_GMER_MEM_EVT_TYPE_INV_ADDR 0x01 278 278 + #define CXL_GMER_MEM_EVT_TYPE_DATA_PATH_ERROR 0x02 279 279 + #define show_mem_event_type(type) __print_symbolic(type, \ 280 280 + { CXL_GMER_MEM_EVT_TYPE_ECC_ERROR, "ECC Error" }, \ 281 281 + { CXL_GMER_MEM_EVT_TYPE_INV_ADDR, "Invalid Address" }, \ 282 282 + { CXL_GMER_MEM_EVT_TYPE_DATA_PATH_ERROR, "Data Path Error" } \ 283 283 + ) 284 284 + 285 285 + #define CXL_GMER_TRANS_UNKNOWN 0x00 286 286 + #define CXL_GMER_TRANS_HOST_READ 0x01 287 287 + #define CXL_GMER_TRANS_HOST_WRITE 0x02 288 288 + #define CXL_GMER_TRANS_HOST_SCAN_MEDIA 0x03 289 289 + #define CXL_GMER_TRANS_HOST_INJECT_POISON 0x04 290 290 + #define CXL_GMER_TRANS_INTERNAL_MEDIA_SCRUB 0x05 291 291 + #define CXL_GMER_TRANS_INTERNAL_MEDIA_MANAGEMENT 0x06 292 292 + #define show_trans_type(type) __print_symbolic(type, \ 293 293 + { CXL_GMER_TRANS_UNKNOWN, "Unknown" }, \ 294 294 + { CXL_GMER_TRANS_HOST_READ, "Host Read" }, \ 295 295 + { CXL_GMER_TRANS_HOST_WRITE, "Host Write" }, \ 296 296 + { CXL_GMER_TRANS_HOST_SCAN_MEDIA, "Host Scan Media" }, \ 297 297 + { CXL_GMER_TRANS_HOST_INJECT_POISON, "Host Inject Poison" }, \ 298 298 + { CXL_GMER_TRANS_INTERNAL_MEDIA_SCRUB, "Internal Media Scrub" }, \ 299 299 + { CXL_GMER_TRANS_INTERNAL_MEDIA_MANAGEMENT, "Internal Media Management" } \ 300 300 + ) 301 301 + 302 302 + #define CXL_GMER_VALID_CHANNEL BIT(0) 303 303 + #define CXL_GMER_VALID_RANK BIT(1) 304 304 + #define CXL_GMER_VALID_DEVICE BIT(2) 305 305 + #define CXL_GMER_VALID_COMPONENT BIT(3) 306 306 + #define show_valid_flags(flags) __print_flags(flags, "|", \ 307 307 + { CXL_GMER_VALID_CHANNEL, "CHANNEL" }, \ 308 308 + { CXL_GMER_VALID_RANK, "RANK" }, \ 309 309 + { CXL_GMER_VALID_DEVICE, "DEVICE" }, \ 310 310 + { CXL_GMER_VALID_COMPONENT, "COMPONENT" } \ 311 311 + ) 312 312 + 313 313 + TRACE_EVENT(cxl_general_media, 314 314 + 315 315 + TP_PROTO(const struct cxl_memdev *cxlmd, enum cxl_event_log_type log, 316 316 + struct cxl_event_gen_media *rec), 317 317 + 318 318 + TP_ARGS(cxlmd, log, rec), 319 319 + 320 320 + TP_STRUCT__entry( 321 321 + CXL_EVT_TP_entry 322 322 + /* General Media */ 323 323 + __field(u64, dpa) 324 324 + __field(u8, descriptor) 325 325 + __field(u8, type) 326 326 + __field(u8, transaction_type) 327 327 + __field(u8, channel) 328 328 + __field(u32, device) 329 329 + __array(u8, comp_id, CXL_EVENT_GEN_MED_COMP_ID_SIZE) 330 330 + __field(u16, validity_flags) 331 331 + /* Following are out of order to pack trace record */ 332 332 + __field(u8, rank) 333 333 + __field(u8, dpa_flags) 334 334 + ), 335 335 + 336 336 + TP_fast_assign( 337 337 + CXL_EVT_TP_fast_assign(cxlmd, log, rec->hdr); 338 338 + 339 339 + /* General Media */ 340 340 + __entry->dpa = le64_to_cpu(rec->phys_addr); 341 341 + __entry->dpa_flags = __entry->dpa & CXL_DPA_FLAGS_MASK; 342 342 + /* Mask after flags have been parsed */ 343 343 + __entry->dpa &= CXL_DPA_MASK; 344 344 + __entry->descriptor = rec->descriptor; 345 345 + __entry->type = rec->type; 346 346 + __entry->transaction_type = rec->transaction_type; 347 347 + __entry->channel = rec->channel; 348 348 + __entry->rank = rec->rank; 349 349 + __entry->device = get_unaligned_le24(rec->device); 350 350 + memcpy(__entry->comp_id, &rec->component_id, 351 351 + CXL_EVENT_GEN_MED_COMP_ID_SIZE); 352 352 + __entry->validity_flags = get_unaligned_le16(&rec->validity_flags); 353 353 + ), 354 354 + 355 355 + CXL_EVT_TP_printk("dpa=%llx dpa_flags='%s' " \ 356 356 + "descriptor='%s' type='%s' transaction_type='%s' channel=%u rank=%u " \ 357 357 + "device=%x comp_id=%s validity_flags='%s'", 358 358 + __entry->dpa, show_dpa_flags(__entry->dpa_flags), 359 359 + show_event_desc_flags(__entry->descriptor), 360 360 + show_mem_event_type(__entry->type), 361 361 + show_trans_type(__entry->transaction_type), 362 362 + __entry->channel, __entry->rank, __entry->device, 363 363 + __print_hex(__entry->comp_id, CXL_EVENT_GEN_MED_COMP_ID_SIZE), 364 364 + show_valid_flags(__entry->validity_flags) 365 365 + ) 366 366 + ); 367 367 + 368 368 + /* 369 369 + * DRAM Event Record - DER 370 370 + * 371 371 + * CXL rev 3.0 section 8.2.9.2.1.2; Table 8-44 372 372 + */ 373 373 + /* 374 374 + * DRAM Event Record defines many fields the same as the General Media Event 375 375 + * Record. Reuse those definitions as appropriate. 376 376 + */ 377 377 + #define CXL_DER_VALID_CHANNEL BIT(0) 378 378 + #define CXL_DER_VALID_RANK BIT(1) 379 379 + #define CXL_DER_VALID_NIBBLE BIT(2) 380 380 + #define CXL_DER_VALID_BANK_GROUP BIT(3) 381 381 + #define CXL_DER_VALID_BANK BIT(4) 382 382 + #define CXL_DER_VALID_ROW BIT(5) 383 383 + #define CXL_DER_VALID_COLUMN BIT(6) 384 384 + #define CXL_DER_VALID_CORRECTION_MASK BIT(7) 385 385 + #define show_dram_valid_flags(flags) __print_flags(flags, "|", \ 386 386 + { CXL_DER_VALID_CHANNEL, "CHANNEL" }, \ 387 387 + { CXL_DER_VALID_RANK, "RANK" }, \ 388 388 + { CXL_DER_VALID_NIBBLE, "NIBBLE" }, \ 389 389 + { CXL_DER_VALID_BANK_GROUP, "BANK GROUP" }, \ 390 390 + { CXL_DER_VALID_BANK, "BANK" }, \ 391 391 + { CXL_DER_VALID_ROW, "ROW" }, \ 392 392 + { CXL_DER_VALID_COLUMN, "COLUMN" }, \ 393 393 + { CXL_DER_VALID_CORRECTION_MASK, "CORRECTION MASK" } \ 394 394 + ) 395 395 + 396 396 + TRACE_EVENT(cxl_dram, 397 397 + 398 398 + TP_PROTO(const struct cxl_memdev *cxlmd, enum cxl_event_log_type log, 399 399 + struct cxl_event_dram *rec), 400 400 + 401 401 + TP_ARGS(cxlmd, log, rec), 402 402 + 403 403 + TP_STRUCT__entry( 404 404 + CXL_EVT_TP_entry 405 405 + /* DRAM */ 406 406 + __field(u64, dpa) 407 407 + __field(u8, descriptor) 408 408 + __field(u8, type) 409 409 + __field(u8, transaction_type) 410 410 + __field(u8, channel) 411 411 + __field(u16, validity_flags) 412 412 + __field(u16, column) /* Out of order to pack trace record */ 413 413 + __field(u32, nibble_mask) 414 414 + __field(u32, row) 415 415 + __array(u8, cor_mask, CXL_EVENT_DER_CORRECTION_MASK_SIZE) 416 416 + __field(u8, rank) /* Out of order to pack trace record */ 417 417 + __field(u8, bank_group) /* Out of order to pack trace record */ 418 418 + __field(u8, bank) /* Out of order to pack trace record */ 419 419 + __field(u8, dpa_flags) /* Out of order to pack trace record */ 420 420 + ), 421 421 + 422 422 + TP_fast_assign( 423 423 + CXL_EVT_TP_fast_assign(cxlmd, log, rec->hdr); 424 424 + 425 425 + /* DRAM */ 426 426 + __entry->dpa = le64_to_cpu(rec->phys_addr); 427 427 + __entry->dpa_flags = __entry->dpa & CXL_DPA_FLAGS_MASK; 428 428 + __entry->dpa &= CXL_DPA_MASK; 429 429 + __entry->descriptor = rec->descriptor; 430 430 + __entry->type = rec->type; 431 431 + __entry->transaction_type = rec->transaction_type; 432 432 + __entry->validity_flags = get_unaligned_le16(rec->validity_flags); 433 433 + __entry->channel = rec->channel; 434 434 + __entry->rank = rec->rank; 435 435 + __entry->nibble_mask = get_unaligned_le24(rec->nibble_mask); 436 436 + __entry->bank_group = rec->bank_group; 437 437 + __entry->bank = rec->bank; 438 438 + __entry->row = get_unaligned_le24(rec->row); 439 439 + __entry->column = get_unaligned_le16(rec->column); 440 440 + memcpy(__entry->cor_mask, &rec->correction_mask, 441 441 + CXL_EVENT_DER_CORRECTION_MASK_SIZE); 442 442 + ), 443 443 + 444 444 + CXL_EVT_TP_printk("dpa=%llx dpa_flags='%s' descriptor='%s' type='%s' " \ 445 445 + "transaction_type='%s' channel=%u rank=%u nibble_mask=%x " \ 446 446 + "bank_group=%u bank=%u row=%u column=%u cor_mask=%s " \ 447 447 + "validity_flags='%s'", 448 448 + __entry->dpa, show_dpa_flags(__entry->dpa_flags), 449 449 + show_event_desc_flags(__entry->descriptor), 450 450 + show_mem_event_type(__entry->type), 451 451 + show_trans_type(__entry->transaction_type), 452 452 + __entry->channel, __entry->rank, __entry->nibble_mask, 453 453 + __entry->bank_group, __entry->bank, 454 454 + __entry->row, __entry->column, 455 455 + __print_hex(__entry->cor_mask, CXL_EVENT_DER_CORRECTION_MASK_SIZE), 456 456 + show_dram_valid_flags(__entry->validity_flags) 457 457 + ) 458 458 + ); 459 459 + 460 460 + /* 461 461 + * Memory Module Event Record - MMER 462 462 + * 463 463 + * CXL res 3.0 section 8.2.9.2.1.3; Table 8-45 464 464 + */ 465 465 + #define CXL_MMER_HEALTH_STATUS_CHANGE 0x00 466 466 + #define CXL_MMER_MEDIA_STATUS_CHANGE 0x01 467 467 + #define CXL_MMER_LIFE_USED_CHANGE 0x02 468 468 + #define CXL_MMER_TEMP_CHANGE 0x03 469 469 + #define CXL_MMER_DATA_PATH_ERROR 0x04 470 470 + #define CXL_MMER_LSA_ERROR 0x05 471 471 + #define show_dev_evt_type(type) __print_symbolic(type, \ 472 472 + { CXL_MMER_HEALTH_STATUS_CHANGE, "Health Status Change" }, \ 473 473 + { CXL_MMER_MEDIA_STATUS_CHANGE, "Media Status Change" }, \ 474 474 + { CXL_MMER_LIFE_USED_CHANGE, "Life Used Change" }, \ 475 475 + { CXL_MMER_TEMP_CHANGE, "Temperature Change" }, \ 476 476 + { CXL_MMER_DATA_PATH_ERROR, "Data Path Error" }, \ 477 477 + { CXL_MMER_LSA_ERROR, "LSA Error" } \ 478 478 + ) 479 479 + 480 480 + /* 481 481 + * Device Health Information - DHI 482 482 + * 483 483 + * CXL res 3.0 section 8.2.9.8.3.1; Table 8-100 484 484 + */ 485 485 + #define CXL_DHI_HS_MAINTENANCE_NEEDED BIT(0) 486 486 + #define CXL_DHI_HS_PERFORMANCE_DEGRADED BIT(1) 487 487 + #define CXL_DHI_HS_HW_REPLACEMENT_NEEDED BIT(2) 488 488 + #define show_health_status_flags(flags) __print_flags(flags, "|", \ 489 489 + { CXL_DHI_HS_MAINTENANCE_NEEDED, "MAINTENANCE_NEEDED" }, \ 490 490 + { CXL_DHI_HS_PERFORMANCE_DEGRADED, "PERFORMANCE_DEGRADED" }, \ 491 491 + { CXL_DHI_HS_HW_REPLACEMENT_NEEDED, "REPLACEMENT_NEEDED" } \ 492 492 + ) 493 493 + 494 494 + #define CXL_DHI_MS_NORMAL 0x00 495 495 + #define CXL_DHI_MS_NOT_READY 0x01 496 496 + #define CXL_DHI_MS_WRITE_PERSISTENCY_LOST 0x02 497 497 + #define CXL_DHI_MS_ALL_DATA_LOST 0x03 498 498 + #define CXL_DHI_MS_WRITE_PERSISTENCY_LOSS_EVENT_POWER_LOSS 0x04 499 499 + #define CXL_DHI_MS_WRITE_PERSISTENCY_LOSS_EVENT_SHUTDOWN 0x05 500 500 + #define CXL_DHI_MS_WRITE_PERSISTENCY_LOSS_IMMINENT 0x06 501 501 + #define CXL_DHI_MS_WRITE_ALL_DATA_LOSS_EVENT_POWER_LOSS 0x07 502 502 + #define CXL_DHI_MS_WRITE_ALL_DATA_LOSS_EVENT_SHUTDOWN 0x08 503 503 + #define CXL_DHI_MS_WRITE_ALL_DATA_LOSS_IMMINENT 0x09 504 504 + #define show_media_status(ms) __print_symbolic(ms, \ 505 505 + { CXL_DHI_MS_NORMAL, \ 506 506 + "Normal" }, \ 507 507 + { CXL_DHI_MS_NOT_READY, \ 508 508 + "Not Ready" }, \ 509 509 + { CXL_DHI_MS_WRITE_PERSISTENCY_LOST, \ 510 510 + "Write Persistency Lost" }, \ 511 511 + { CXL_DHI_MS_ALL_DATA_LOST, \ 512 512 + "All Data Lost" }, \ 513 513 + { CXL_DHI_MS_WRITE_PERSISTENCY_LOSS_EVENT_POWER_LOSS, \ 514 514 + "Write Persistency Loss in the Event of Power Loss" }, \ 515 515 + { CXL_DHI_MS_WRITE_PERSISTENCY_LOSS_EVENT_SHUTDOWN, \ 516 516 + "Write Persistency Loss in Event of Shutdown" }, \ 517 517 + { CXL_DHI_MS_WRITE_PERSISTENCY_LOSS_IMMINENT, \ 518 518 + "Write Persistency Loss Imminent" }, \ 519 519 + { CXL_DHI_MS_WRITE_ALL_DATA_LOSS_EVENT_POWER_LOSS, \ 520 520 + "All Data Loss in Event of Power Loss" }, \ 521 521 + { CXL_DHI_MS_WRITE_ALL_DATA_LOSS_EVENT_SHUTDOWN, \ 522 522 + "All Data loss in the Event of Shutdown" }, \ 523 523 + { CXL_DHI_MS_WRITE_ALL_DATA_LOSS_IMMINENT, \ 524 524 + "All Data Loss Imminent" } \ 525 525 + ) 526 526 + 527 527 + #define CXL_DHI_AS_NORMAL 0x0 528 528 + #define CXL_DHI_AS_WARNING 0x1 529 529 + #define CXL_DHI_AS_CRITICAL 0x2 530 530 + #define show_two_bit_status(as) __print_symbolic(as, \ 531 531 + { CXL_DHI_AS_NORMAL, "Normal" }, \ 532 532 + { CXL_DHI_AS_WARNING, "Warning" }, \ 533 533 + { CXL_DHI_AS_CRITICAL, "Critical" } \ 534 534 + ) 535 535 + #define show_one_bit_status(as) __print_symbolic(as, \ 536 536 + { CXL_DHI_AS_NORMAL, "Normal" }, \ 537 537 + { CXL_DHI_AS_WARNING, "Warning" } \ 538 538 + ) 539 539 + 540 540 + #define CXL_DHI_AS_LIFE_USED(as) (as & 0x3) 541 541 + #define CXL_DHI_AS_DEV_TEMP(as) ((as & 0xC) >> 2) 542 542 + #define CXL_DHI_AS_COR_VOL_ERR_CNT(as) ((as & 0x10) >> 4) 543 543 + #define CXL_DHI_AS_COR_PER_ERR_CNT(as) ((as & 0x20) >> 5) 544 544 + 545 545 + TRACE_EVENT(cxl_memory_module, 546 546 + 547 547 + TP_PROTO(const struct cxl_memdev *cxlmd, enum cxl_event_log_type log, 548 548 + struct cxl_event_mem_module *rec), 549 549 + 550 550 + TP_ARGS(cxlmd, log, rec), 551 551 + 552 552 + TP_STRUCT__entry( 553 553 + CXL_EVT_TP_entry 554 554 + 555 555 + /* Memory Module Event */ 556 556 + __field(u8, event_type) 557 557 + 558 558 + /* Device Health Info */ 559 559 + __field(u8, health_status) 560 560 + __field(u8, media_status) 561 561 + __field(u8, life_used) 562 562 + __field(u32, dirty_shutdown_cnt) 563 563 + __field(u32, cor_vol_err_cnt) 564 564 + __field(u32, cor_per_err_cnt) 565 565 + __field(s16, device_temp) 566 566 + __field(u8, add_status) 567 567 + ), 568 568 + 569 569 + TP_fast_assign( 570 570 + CXL_EVT_TP_fast_assign(cxlmd, log, rec->hdr); 571 571 + 572 572 + /* Memory Module Event */ 573 573 + __entry->event_type = rec->event_type; 574 574 + 575 575 + /* Device Health Info */ 576 576 + __entry->health_status = rec->info.health_status; 577 577 + __entry->media_status = rec->info.media_status; 578 578 + __entry->life_used = rec->info.life_used; 579 579 + __entry->dirty_shutdown_cnt = get_unaligned_le32(rec->info.dirty_shutdown_cnt); 580 580 + __entry->cor_vol_err_cnt = get_unaligned_le32(rec->info.cor_vol_err_cnt); 581 581 + __entry->cor_per_err_cnt = get_unaligned_le32(rec->info.cor_per_err_cnt); 582 582 + __entry->device_temp = get_unaligned_le16(rec->info.device_temp); 583 583 + __entry->add_status = rec->info.add_status; 584 584 + ), 585 585 + 586 586 + CXL_EVT_TP_printk("event_type='%s' health_status='%s' media_status='%s' " \ 587 587 + "as_life_used=%s as_dev_temp=%s as_cor_vol_err_cnt=%s " \ 588 588 + "as_cor_per_err_cnt=%s life_used=%u device_temp=%d " \ 589 589 + "dirty_shutdown_cnt=%u cor_vol_err_cnt=%u cor_per_err_cnt=%u", 590 590 + show_dev_evt_type(__entry->event_type), 591 591 + show_health_status_flags(__entry->health_status), 592 592 + show_media_status(__entry->media_status), 593 593 + show_two_bit_status(CXL_DHI_AS_LIFE_USED(__entry->add_status)), 594 594 + show_two_bit_status(CXL_DHI_AS_DEV_TEMP(__entry->add_status)), 595 595 + show_one_bit_status(CXL_DHI_AS_COR_VOL_ERR_CNT(__entry->add_status)), 596 596 + show_one_bit_status(CXL_DHI_AS_COR_PER_ERR_CNT(__entry->add_status)), 597 597 + __entry->life_used, __entry->device_temp, 598 598 + __entry->dirty_shutdown_cnt, __entry->cor_vol_err_cnt, 599 599 + __entry->cor_per_err_cnt 600 600 + ) 601 601 + ); 602 602 + 603 603 + #endif /* _CXL_EVENTS_H */ 604 604 + 605 605 + #define TRACE_INCLUDE_FILE trace 606 606 + #include <trace/define_trace.h>

+94 -2

drivers/cxl/cxl.h

reviewed

··· 130 130 #define CXL_RAS_UNCORRECTABLE_STATUS_MASK (GENMASK(16, 14) | GENMASK(11, 0)) 131 131 #define CXL_RAS_UNCORRECTABLE_MASK_OFFSET 0x4 132 132 #define CXL_RAS_UNCORRECTABLE_MASK_MASK (GENMASK(16, 14) | GENMASK(11, 0)) 133 133 + #define CXL_RAS_UNCORRECTABLE_MASK_F256B_MASK BIT(8) 133 134 #define CXL_RAS_UNCORRECTABLE_SEVERITY_OFFSET 0x8 134 135 #define CXL_RAS_UNCORRECTABLE_SEVERITY_MASK (GENMASK(16, 14) | GENMASK(11, 0)) 135 136 #define CXL_RAS_CORRECTABLE_STATUS_OFFSET 0xC ··· 141 140 #define CXL_RAS_CAP_CONTROL_FE_MASK GENMASK(5, 0) 142 141 #define CXL_RAS_HEADER_LOG_OFFSET 0x18 143 142 #define CXL_RAS_CAPABILITY_LENGTH 0x58 143 143 + #define CXL_HEADERLOG_SIZE SZ_512 144 144 + #define CXL_HEADERLOG_SIZE_U32 SZ_512 / sizeof(u32) 144 145 145 146 /* CXL 2.0 8.2.8.1 Device Capabilities Array Register */ 146 147 #define CXLDEV_CAP_ARRAY_OFFSET 0x0 ··· 156 153 #define CXLDEV_CAP_CAP_ID_PRIMARY_MAILBOX 0x2 157 154 #define CXLDEV_CAP_CAP_ID_SECONDARY_MAILBOX 0x3 158 155 #define CXLDEV_CAP_CAP_ID_MEMDEV 0x4000 156 156 + 157 157 + /* CXL 3.0 8.2.8.3.1 Event Status Register */ 158 158 + #define CXLDEV_DEV_EVENT_STATUS_OFFSET 0x00 159 159 + #define CXLDEV_EVENT_STATUS_INFO BIT(0) 160 160 + #define CXLDEV_EVENT_STATUS_WARN BIT(1) 161 161 + #define CXLDEV_EVENT_STATUS_FAIL BIT(2) 162 162 + #define CXLDEV_EVENT_STATUS_FATAL BIT(3) 163 163 + 164 164 + #define CXLDEV_EVENT_STATUS_ALL (CXLDEV_EVENT_STATUS_INFO | \ 165 165 + CXLDEV_EVENT_STATUS_WARN | \ 166 166 + CXLDEV_EVENT_STATUS_FAIL | \ 167 167 + CXLDEV_EVENT_STATUS_FATAL) 168 168 + 169 169 + /* CXL rev 3.0 section 8.2.9.2.4; Table 8-52 */ 170 170 + #define CXLDEV_EVENT_INT_MODE_MASK GENMASK(1, 0) 171 171 + #define CXLDEV_EVENT_INT_MSGNUM_MASK GENMASK(7, 4) 159 172 160 173 /* CXL 2.0 8.2.8.4 Mailbox Registers */ 161 174 #define CXLDEV_MBOX_CAPS_OFFSET 0x00 ··· 278 259 * cxl_decoder flags that define the type of memory / devices this 279 260 * decoder supports as well as configuration lock status See "CXL 2.0 280 261 * 8.2.5.12.7 CXL HDM Decoder 0 Control Register" for details. 262 262 + * Additionally indicate whether decoder settings were autodetected, 263 263 + * user customized. 281 264 */ 282 265 #define CXL_DECODER_F_RAM BIT(0) 283 266 #define CXL_DECODER_F_PMEM BIT(1) ··· 339 318 CXL_DECODER_DEAD, 340 319 }; 341 320 321 321 + static inline const char *cxl_decoder_mode_name(enum cxl_decoder_mode mode) 322 322 + { 323 323 + static const char * const names[] = { 324 324 + [CXL_DECODER_NONE] = "none", 325 325 + [CXL_DECODER_RAM] = "ram", 326 326 + [CXL_DECODER_PMEM] = "pmem", 327 327 + [CXL_DECODER_MIXED] = "mixed", 328 328 + }; 329 329 + 330 330 + if (mode >= CXL_DECODER_NONE && mode <= CXL_DECODER_MIXED) 331 331 + return names[mode]; 332 332 + return "mixed"; 333 333 + } 334 334 + 335 335 + /* 336 336 + * Track whether this decoder is reserved for region autodiscovery, or 337 337 + * free for userspace provisioning. 338 338 + */ 339 339 + enum cxl_decoder_state { 340 340 + CXL_DECODER_STATE_MANUAL, 341 341 + CXL_DECODER_STATE_AUTO, 342 342 + }; 343 343 + 342 344 /** 343 345 * struct cxl_endpoint_decoder - Endpoint / SPA to DPA decoder 344 346 * @cxld: base cxl_decoder_object 345 347 * @dpa_res: actively claimed DPA span of this decoder 346 348 * @skip: offset into @dpa_res where @cxld.hpa_range maps 347 349 * @mode: which memory type / access-mode-partition this decoder targets 350 350 + * @state: autodiscovery state 348 351 * @pos: interleave position in @cxld.region 349 352 */ 350 353 struct cxl_endpoint_decoder { ··· 376 331 struct resource *dpa_res; 377 332 resource_size_t skip; 378 333 enum cxl_decoder_mode mode; 334 334 + enum cxl_decoder_state state; 379 335 int pos; 380 336 }; 381 337 ··· 410 364 * @region_id: region id for next region provisioning event 411 365 * @calc_hb: which host bridge covers the n'th position by granularity 412 366 * @platform_data: platform specific configuration data 367 367 + * @range_lock: sync region autodiscovery by address range 413 368 * @cxlsd: base cxl switch decoder 414 369 */ 415 370 struct cxl_root_decoder { ··· 418 371 atomic_t region_id; 419 372 cxl_calc_hb_fn calc_hb; 420 373 void *platform_data; 374 374 + struct mutex range_lock; 421 375 struct cxl_switch_decoder cxlsd; 422 376 }; 423 377 ··· 467 419 * CPU cache state at region activation time. 468 420 */ 469 421 #define CXL_REGION_F_INCOHERENT 0 422 422 + 423 423 + /* 424 424 + * Indicate whether this region has been assembled by autodetection or 425 425 + * userspace assembly. Prevent endpoint decoders outside of automatic 426 426 + * detection from being added to the region. 427 427 + */ 428 428 + #define CXL_REGION_F_AUTO 1 470 429 471 430 /** 472 431 * struct cxl_region - CXL region ··· 528 473 struct range hpa_range; 529 474 int nr_mappings; 530 475 struct cxl_pmem_region_mapping mapping[]; 476 476 + }; 477 477 + 478 478 + struct cxl_dax_region { 479 479 + struct device dev; 480 480 + struct cxl_region *cxlr; 481 481 + struct range hpa_range; 531 482 }; 532 483 533 484 /** ··· 676 615 677 616 struct cxl_decoder *to_cxl_decoder(struct device *dev); 678 617 struct cxl_root_decoder *to_cxl_root_decoder(struct device *dev); 618 618 + struct cxl_switch_decoder *to_cxl_switch_decoder(struct device *dev); 679 619 struct cxl_endpoint_decoder *to_cxl_endpoint_decoder(struct device *dev); 680 620 bool is_root_decoder(struct device *dev); 621 621 + bool is_switch_decoder(struct device *dev); 681 622 bool is_endpoint_decoder(struct device *dev); 682 623 struct cxl_root_decoder *cxl_root_decoder_alloc(struct cxl_port *port, 683 624 unsigned int nr_targets, ··· 693 630 int cxl_decoder_autoremove(struct device *host, struct cxl_decoder *cxld); 694 631 int cxl_endpoint_autoremove(struct cxl_memdev *cxlmd, struct cxl_port *endpoint); 695 632 633 633 + /** 634 634 + * struct cxl_endpoint_dvsec_info - Cached DVSEC info 635 635 + * @mem_enabled: cached value of mem_enabled in the DVSEC, PCIE_DEVICE 636 636 + * @ranges: Number of active HDM ranges this device uses. 637 637 + * @dvsec_range: cached attributes of the ranges in the DVSEC, PCIE_DEVICE 638 638 + */ 639 639 + struct cxl_endpoint_dvsec_info { 640 640 + bool mem_enabled; 641 641 + int ranges; 642 642 + struct range dvsec_range[2]; 643 643 + }; 644 644 + 696 645 struct cxl_hdm; 697 697 - struct cxl_hdm *devm_cxl_setup_hdm(struct cxl_port *port); 698 698 - int devm_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm); 646 646 + struct cxl_hdm *devm_cxl_setup_hdm(struct cxl_port *port, 647 647 + struct cxl_endpoint_dvsec_info *info); 648 648 + int devm_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm, 649 649 + struct cxl_endpoint_dvsec_info *info); 699 650 int devm_cxl_add_passthrough_decoder(struct cxl_port *port); 651 651 + int cxl_dvsec_rr_decode(struct device *dev, int dvsec, 652 652 + struct cxl_endpoint_dvsec_info *info); 700 653 701 654 bool is_cxl_region(struct device *dev); 702 655 ··· 746 667 #define CXL_DEVICE_MEMORY_EXPANDER 5 747 668 #define CXL_DEVICE_REGION 6 748 669 #define CXL_DEVICE_PMEM_REGION 7 670 670 + #define CXL_DEVICE_DAX_REGION 8 749 671 750 672 #define MODULE_ALIAS_CXL(type) MODULE_ALIAS("cxl:t" __stringify(type) "*") 751 673 #define CXL_MODALIAS_FMT "cxl:t%d" ··· 763 683 #ifdef CONFIG_CXL_REGION 764 684 bool is_cxl_pmem_region(struct device *dev); 765 685 struct cxl_pmem_region *to_cxl_pmem_region(struct device *dev); 686 686 + int cxl_add_to_region(struct cxl_port *root, 687 687 + struct cxl_endpoint_decoder *cxled); 688 688 + struct cxl_dax_region *to_cxl_dax_region(struct device *dev); 766 689 #else 767 690 static inline bool is_cxl_pmem_region(struct device *dev) 768 691 { 769 692 return false; 770 693 } 771 694 static inline struct cxl_pmem_region *to_cxl_pmem_region(struct device *dev) 695 695 + { 696 696 + return NULL; 697 697 + } 698 698 + static inline int cxl_add_to_region(struct cxl_port *root, 699 699 + struct cxl_endpoint_decoder *cxled) 700 700 + { 701 701 + return 0; 702 702 + } 703 703 + static inline struct cxl_dax_region *to_cxl_dax_region(struct device *dev) 772 704 { 773 705 return NULL; 774 706 }

+177 -9

drivers/cxl/cxlmem.h

reviewed

··· 4 4 #define __CXL_MEM_H__ 5 5 #include <uapi/linux/cxl_mem.h> 6 6 #include <linux/cdev.h> 7 7 + #include <linux/uuid.h> 7 8 #include "cxl.h" 8 9 9 10 /* CXL 2.0 8.2.8.5.1.1 Memory Device Status Register */ ··· 39 38 * @cxl_nvb: coordinate removal of @cxl_nvd if present 40 39 * @cxl_nvd: optional bridge to an nvdimm if the device supports pmem 41 40 * @id: id number of this memdev instance. 41 41 + * @depth: endpoint port depth 42 42 */ 43 43 struct cxl_memdev { 44 44 struct device dev; ··· 49 47 struct cxl_nvdimm_bridge *cxl_nvb; 50 48 struct cxl_nvdimm *cxl_nvd; 51 49 int id; 50 50 + int depth; 52 51 }; 53 52 54 53 static inline struct cxl_memdev *to_cxl_memdev(struct device *dev) ··· 82 79 } 83 80 84 81 struct cxl_memdev *devm_cxl_add_memdev(struct cxl_dev_state *cxlds); 82 82 + int devm_cxl_dpa_reserve(struct cxl_endpoint_decoder *cxled, 83 83 + resource_size_t base, resource_size_t len, 84 84 + resource_size_t skipped); 85 85 86 86 static inline struct cxl_ep *cxl_ep_load(struct cxl_port *port, 87 87 struct cxl_memdev *cxlmd) ··· 188 182 #define CXL_CAPACITY_MULTIPLIER SZ_256M 189 183 190 184 /** 191 191 - * struct cxl_endpoint_dvsec_info - Cached DVSEC info 192 192 - * @mem_enabled: cached value of mem_enabled in the DVSEC, PCIE_DEVICE 193 193 - * @ranges: Number of active HDM ranges this device uses. 194 194 - * @dvsec_range: cached attributes of the ranges in the DVSEC, PCIE_DEVICE 185 185 + * Event Interrupt Policy 186 186 + * 187 187 + * CXL rev 3.0 section 8.2.9.2.4; Table 8-52 195 188 */ 196 196 - struct cxl_endpoint_dvsec_info { 197 197 - bool mem_enabled; 198 198 - int ranges; 199 199 - struct range dvsec_range[2]; 189 189 + enum cxl_event_int_mode { 190 190 + CXL_INT_NONE = 0x00, 191 191 + CXL_INT_MSI_MSIX = 0x01, 192 192 + CXL_INT_FW = 0x02 193 193 + }; 194 194 + struct cxl_event_interrupt_policy { 195 195 + u8 info_settings; 196 196 + u8 warn_settings; 197 197 + u8 failure_settings; 198 198 + u8 fatal_settings; 199 199 + } __packed; 200 200 + 201 201 + /** 202 202 + * struct cxl_event_state - Event log driver state 203 203 + * 204 204 + * @event_buf: Buffer to receive event data 205 205 + * @event_log_lock: Serialize event_buf and log use 206 206 + */ 207 207 + struct cxl_event_state { 208 208 + struct cxl_get_event_payload *buf; 209 209 + struct mutex log_lock; 200 210 }; 201 211 202 212 /** ··· 250 228 * @info: Cached DVSEC information about the device. 251 229 * @serial: PCIe Device Serial Number 252 230 * @doe_mbs: PCI DOE mailbox array 231 231 + * @event: event log driver state 253 232 * @mbox_send: @dev specific transport for transmitting mailbox commands 254 233 * 255 234 * See section 8.2.9.5.2 Capacity Configuration and Label Storage for ··· 289 266 290 267 struct xarray doe_mbs; 291 268 269 269 + struct cxl_event_state event; 270 270 + 292 271 int (*mbox_send)(struct cxl_dev_state *cxlds, struct cxl_mbox_cmd *cmd); 293 272 }; 294 273 295 274 enum cxl_opcode { 296 275 CXL_MBOX_OP_INVALID = 0x0000, 297 276 CXL_MBOX_OP_RAW = CXL_MBOX_OP_INVALID, 277 277 + CXL_MBOX_OP_GET_EVENT_RECORD = 0x0100, 278 278 + CXL_MBOX_OP_CLEAR_EVENT_RECORD = 0x0101, 279 279 + CXL_MBOX_OP_GET_EVT_INT_POLICY = 0x0102, 280 280 + CXL_MBOX_OP_SET_EVT_INT_POLICY = 0x0103, 298 281 CXL_MBOX_OP_GET_FW_INFO = 0x0200, 299 282 CXL_MBOX_OP_ACTIVATE_FW = 0x0202, 283 283 + CXL_MBOX_OP_SET_TIMESTAMP = 0x0301, 300 284 CXL_MBOX_OP_GET_SUPPORTED_LOGS = 0x0400, 301 285 CXL_MBOX_OP_GET_LOG = 0x0401, 302 286 CXL_MBOX_OP_IDENTIFY = 0x4000, ··· 377 347 u8 qos_telemetry_caps; 378 348 } __packed; 379 349 350 350 + /* 351 351 + * Common Event Record Format 352 352 + * CXL rev 3.0 section 8.2.9.2.1; Table 8-42 353 353 + */ 354 354 + struct cxl_event_record_hdr { 355 355 + uuid_t id; 356 356 + u8 length; 357 357 + u8 flags[3]; 358 358 + __le16 handle; 359 359 + __le16 related_handle; 360 360 + __le64 timestamp; 361 361 + u8 maint_op_class; 362 362 + u8 reserved[15]; 363 363 + } __packed; 364 364 + 365 365 + #define CXL_EVENT_RECORD_DATA_LENGTH 0x50 366 366 + struct cxl_event_record_raw { 367 367 + struct cxl_event_record_hdr hdr; 368 368 + u8 data[CXL_EVENT_RECORD_DATA_LENGTH]; 369 369 + } __packed; 370 370 + 371 371 + /* 372 372 + * Get Event Records output payload 373 373 + * CXL rev 3.0 section 8.2.9.2.2; Table 8-50 374 374 + */ 375 375 + #define CXL_GET_EVENT_FLAG_OVERFLOW BIT(0) 376 376 + #define CXL_GET_EVENT_FLAG_MORE_RECORDS BIT(1) 377 377 + struct cxl_get_event_payload { 378 378 + u8 flags; 379 379 + u8 reserved1; 380 380 + __le16 overflow_err_count; 381 381 + __le64 first_overflow_timestamp; 382 382 + __le64 last_overflow_timestamp; 383 383 + __le16 record_count; 384 384 + u8 reserved2[10]; 385 385 + struct cxl_event_record_raw records[]; 386 386 + } __packed; 387 387 + 388 388 + /* 389 389 + * CXL rev 3.0 section 8.2.9.2.2; Table 8-49 390 390 + */ 391 391 + enum cxl_event_log_type { 392 392 + CXL_EVENT_TYPE_INFO = 0x00, 393 393 + CXL_EVENT_TYPE_WARN, 394 394 + CXL_EVENT_TYPE_FAIL, 395 395 + CXL_EVENT_TYPE_FATAL, 396 396 + CXL_EVENT_TYPE_MAX 397 397 + }; 398 398 + 399 399 + /* 400 400 + * Clear Event Records input payload 401 401 + * CXL rev 3.0 section 8.2.9.2.3; Table 8-51 402 402 + */ 403 403 + struct cxl_mbox_clear_event_payload { 404 404 + u8 event_log; /* enum cxl_event_log_type */ 405 405 + u8 clear_flags; 406 406 + u8 nr_recs; 407 407 + u8 reserved[3]; 408 408 + __le16 handles[]; 409 409 + } __packed; 410 410 + #define CXL_CLEAR_EVENT_MAX_HANDLES U8_MAX 411 411 + 412 412 + /* 413 413 + * General Media Event Record 414 414 + * CXL rev 3.0 Section 8.2.9.2.1.1; Table 8-43 415 415 + */ 416 416 + #define CXL_EVENT_GEN_MED_COMP_ID_SIZE 0x10 417 417 + struct cxl_event_gen_media { 418 418 + struct cxl_event_record_hdr hdr; 419 419 + __le64 phys_addr; 420 420 + u8 descriptor; 421 421 + u8 type; 422 422 + u8 transaction_type; 423 423 + u8 validity_flags[2]; 424 424 + u8 channel; 425 425 + u8 rank; 426 426 + u8 device[3]; 427 427 + u8 component_id[CXL_EVENT_GEN_MED_COMP_ID_SIZE]; 428 428 + u8 reserved[46]; 429 429 + } __packed; 430 430 + 431 431 + /* 432 432 + * DRAM Event Record - DER 433 433 + * CXL rev 3.0 section 8.2.9.2.1.2; Table 3-44 434 434 + */ 435 435 + #define CXL_EVENT_DER_CORRECTION_MASK_SIZE 0x20 436 436 + struct cxl_event_dram { 437 437 + struct cxl_event_record_hdr hdr; 438 438 + __le64 phys_addr; 439 439 + u8 descriptor; 440 440 + u8 type; 441 441 + u8 transaction_type; 442 442 + u8 validity_flags[2]; 443 443 + u8 channel; 444 444 + u8 rank; 445 445 + u8 nibble_mask[3]; 446 446 + u8 bank_group; 447 447 + u8 bank; 448 448 + u8 row[3]; 449 449 + u8 column[2]; 450 450 + u8 correction_mask[CXL_EVENT_DER_CORRECTION_MASK_SIZE]; 451 451 + u8 reserved[0x17]; 452 452 + } __packed; 453 453 + 454 454 + /* 455 455 + * Get Health Info Record 456 456 + * CXL rev 3.0 section 8.2.9.8.3.1; Table 8-100 457 457 + */ 458 458 + struct cxl_get_health_info { 459 459 + u8 health_status; 460 460 + u8 media_status; 461 461 + u8 add_status; 462 462 + u8 life_used; 463 463 + u8 device_temp[2]; 464 464 + u8 dirty_shutdown_cnt[4]; 465 465 + u8 cor_vol_err_cnt[4]; 466 466 + u8 cor_per_err_cnt[4]; 467 467 + } __packed; 468 468 + 469 469 + /* 470 470 + * Memory Module Event Record 471 471 + * CXL rev 3.0 section 8.2.9.2.1.3; Table 8-45 472 472 + */ 473 473 + struct cxl_event_mem_module { 474 474 + struct cxl_event_record_hdr hdr; 475 475 + u8 event_type; 476 476 + struct cxl_get_health_info info; 477 477 + u8 reserved[0x3d]; 478 478 + } __packed; 479 479 + 380 480 struct cxl_mbox_get_partition_info { 381 481 __le64 active_volatile_cap; 382 482 __le64 active_persistent_cap; ··· 532 372 533 373 #define CXL_SET_PARTITION_IMMEDIATE_FLAG BIT(0) 534 374 375 375 + /* Set Timestamp CXL 3.0 Spec 8.2.9.4.2 */ 376 376 + struct cxl_mbox_set_timestamp_in { 377 377 + __le64 timestamp; 378 378 + 379 379 + } __packed; 380 380 + 535 381 /** 536 382 * struct cxl_mem_command - Driver representation of a memory device command 537 383 * @info: Command information as it exists for the UAPI ··· 559 393 struct cxl_command_info info; 560 394 enum cxl_opcode opcode; 561 395 u32 flags; 562 562 - #define CXL_CMD_FLAG_NONE 0 563 396 #define CXL_CMD_FLAG_FORCE_ENABLE BIT(0) 564 397 }; 565 398 ··· 606 441 struct cxl_dev_state *cxl_dev_state_create(struct device *dev); 607 442 void set_exclusive_cxl_commands(struct cxl_dev_state *cxlds, unsigned long *cmds); 608 443 void clear_exclusive_cxl_commands(struct cxl_dev_state *cxlds, unsigned long *cmds); 444 444 + void cxl_mem_get_event_records(struct cxl_dev_state *cxlds, u32 status); 445 445 + int cxl_set_timestamp(struct cxl_dev_state *cxlds); 446 446 + 609 447 #ifdef CONFIG_CXL_SUSPEND 610 448 void cxl_mem_active_inc(void); 611 449 void cxl_mem_active_dec(void);

+11 -1

drivers/cxl/cxlpci.h

reviewed

··· 53 53 #define CXL_DVSEC_REG_LOCATOR_BLOCK_ID_MASK GENMASK(15, 8) 54 54 #define CXL_DVSEC_REG_LOCATOR_BLOCK_OFF_LOW_MASK GENMASK(31, 16) 55 55 56 56 + /* 57 57 + * NOTE: Currently all the functions which are enabled for CXL require their 58 58 + * vectors to be in the first 16. Use this as the default max. 59 59 + */ 60 60 + #define CXL_PCI_DEFAULT_MAX_VECTORS 16 61 61 + 56 62 /* Register Block Identifier (RBI) */ 57 63 enum cxl_regloc_type { 58 64 CXL_REGLOC_RBI_EMPTY = 0, ··· 70 64 71 65 int devm_cxl_port_enumerate_dports(struct cxl_port *port); 72 66 struct cxl_dev_state; 73 73 - int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm); 67 67 + int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm, 68 68 + struct cxl_endpoint_dvsec_info *info); 74 69 void read_cdat_data(struct cxl_port *port); 70 70 + void cxl_cor_error_detected(struct pci_dev *pdev); 71 71 + pci_ers_result_t cxl_error_detected(struct pci_dev *pdev, 72 72 + pci_channel_state_t state); 75 73 #endif /* __CXL_PCI_H__ */

+302 -123

drivers/cxl/pci.c

reviewed

··· 14 14 #include "cxlmem.h" 15 15 #include "cxlpci.h" 16 16 #include "cxl.h" 17 17 - #define CREATE_TRACE_POINTS 18 18 - #include <trace/events/cxl.h> 19 17 20 18 /** 21 19 * DOC: cxl pci ··· 160 162 writeq(cmd_reg, cxlds->regs.mbox + CXLDEV_MBOX_CMD_OFFSET); 161 163 162 164 /* #4 */ 163 163 - dev_dbg(dev, "Sending command\n"); 165 165 + dev_dbg(dev, "Sending command: 0x%04x\n", mbox_cmd->opcode); 164 166 writel(CXLDEV_MBOX_CTRL_DOORBELL, 165 167 cxlds->regs.mbox + CXLDEV_MBOX_CTRL_OFFSET); 166 168 ··· 412 414 return pci_pcie_type(pdev) == PCI_EXP_TYPE_RC_END; 413 415 } 414 416 415 415 - static void disable_aer(void *pdev) 417 417 + /* 418 418 + * CXL v3.0 6.2.3 Table 6-4 419 419 + * The table indicates that if PCIe Flit Mode is set, then CXL is in 256B flits 420 420 + * mode, otherwise it's 68B flits mode. 421 421 + */ 422 422 + static bool cxl_pci_flit_256(struct pci_dev *pdev) 416 423 { 417 417 - pci_disable_pcie_error_reporting(pdev); 424 424 + u16 lnksta2; 425 425 + 426 426 + pcie_capability_read_word(pdev, PCI_EXP_LNKSTA2, &lnksta2); 427 427 + return lnksta2 & PCI_EXP_LNKSTA2_FLIT; 428 428 + } 429 429 + 430 430 + static int cxl_pci_ras_unmask(struct pci_dev *pdev) 431 431 + { 432 432 + struct pci_host_bridge *host_bridge = pci_find_host_bridge(pdev->bus); 433 433 + struct cxl_dev_state *cxlds = pci_get_drvdata(pdev); 434 434 + void __iomem *addr; 435 435 + u32 orig_val, val, mask; 436 436 + u16 cap; 437 437 + int rc; 438 438 + 439 439 + if (!cxlds->regs.ras) { 440 440 + dev_dbg(&pdev->dev, "No RAS registers.\n"); 441 441 + return 0; 442 442 + } 443 443 + 444 444 + /* BIOS has CXL error control */ 445 445 + if (!host_bridge->native_cxl_error) 446 446 + return -ENXIO; 447 447 + 448 448 + rc = pcie_capability_read_word(pdev, PCI_EXP_DEVCTL, &cap); 449 449 + if (rc) 450 450 + return rc; 451 451 + 452 452 + if (cap & PCI_EXP_DEVCTL_URRE) { 453 453 + addr = cxlds->regs.ras + CXL_RAS_UNCORRECTABLE_MASK_OFFSET; 454 454 + orig_val = readl(addr); 455 455 + 456 456 + mask = CXL_RAS_UNCORRECTABLE_MASK_MASK; 457 457 + if (!cxl_pci_flit_256(pdev)) 458 458 + mask &= ~CXL_RAS_UNCORRECTABLE_MASK_F256B_MASK; 459 459 + val = orig_val & ~mask; 460 460 + writel(val, addr); 461 461 + dev_dbg(&pdev->dev, 462 462 + "Uncorrectable RAS Errors Mask: %#x -> %#x\n", 463 463 + orig_val, val); 464 464 + } 465 465 + 466 466 + if (cap & PCI_EXP_DEVCTL_CERE) { 467 467 + addr = cxlds->regs.ras + CXL_RAS_CORRECTABLE_MASK_OFFSET; 468 468 + orig_val = readl(addr); 469 469 + val = orig_val & ~CXL_RAS_CORRECTABLE_MASK_MASK; 470 470 + writel(val, addr); 471 471 + dev_dbg(&pdev->dev, "Correctable RAS Errors Mask: %#x -> %#x\n", 472 472 + orig_val, val); 473 473 + } 474 474 + 475 475 + return 0; 476 476 + } 477 477 + 478 478 + static void free_event_buf(void *buf) 479 479 + { 480 480 + kvfree(buf); 481 481 + } 482 482 + 483 483 + /* 484 484 + * There is a single buffer for reading event logs from the mailbox. All logs 485 485 + * share this buffer protected by the cxlds->event_log_lock. 486 486 + */ 487 487 + static int cxl_mem_alloc_event_buf(struct cxl_dev_state *cxlds) 488 488 + { 489 489 + struct cxl_get_event_payload *buf; 490 490 + 491 491 + buf = kvmalloc(cxlds->payload_size, GFP_KERNEL); 492 492 + if (!buf) 493 493 + return -ENOMEM; 494 494 + cxlds->event.buf = buf; 495 495 + 496 496 + return devm_add_action_or_reset(cxlds->dev, free_event_buf, buf); 497 497 + } 498 498 + 499 499 + static int cxl_alloc_irq_vectors(struct pci_dev *pdev) 500 500 + { 501 501 + int nvecs; 502 502 + 503 503 + /* 504 504 + * Per CXL 3.0 3.1.1 CXL.io Endpoint a function on a CXL device must 505 505 + * not generate INTx messages if that function participates in 506 506 + * CXL.cache or CXL.mem. 507 507 + * 508 508 + * Additionally pci_alloc_irq_vectors() handles calling 509 509 + * pci_free_irq_vectors() automatically despite not being called 510 510 + * pcim_*. See pci_setup_msi_context(). 511 511 + */ 512 512 + nvecs = pci_alloc_irq_vectors(pdev, 1, CXL_PCI_DEFAULT_MAX_VECTORS, 513 513 + PCI_IRQ_MSIX | PCI_IRQ_MSI); 514 514 + if (nvecs < 1) { 515 515 + dev_dbg(&pdev->dev, "Failed to alloc irq vectors: %d\n", nvecs); 516 516 + return -ENXIO; 517 517 + } 518 518 + return 0; 519 519 + } 520 520 + 521 521 + struct cxl_dev_id { 522 522 + struct cxl_dev_state *cxlds; 523 523 + }; 524 524 + 525 525 + static irqreturn_t cxl_event_thread(int irq, void *id) 526 526 + { 527 527 + struct cxl_dev_id *dev_id = id; 528 528 + struct cxl_dev_state *cxlds = dev_id->cxlds; 529 529 + u32 status; 530 530 + 531 531 + do { 532 532 + /* 533 533 + * CXL 3.0 8.2.8.3.1: The lower 32 bits are the status; 534 534 + * ignore the reserved upper 32 bits 535 535 + */ 536 536 + status = readl(cxlds->regs.status + CXLDEV_DEV_EVENT_STATUS_OFFSET); 537 537 + /* Ignore logs unknown to the driver */ 538 538 + status &= CXLDEV_EVENT_STATUS_ALL; 539 539 + if (!status) 540 540 + break; 541 541 + cxl_mem_get_event_records(cxlds, status); 542 542 + cond_resched(); 543 543 + } while (status); 544 544 + 545 545 + return IRQ_HANDLED; 546 546 + } 547 547 + 548 548 + static int cxl_event_req_irq(struct cxl_dev_state *cxlds, u8 setting) 549 549 + { 550 550 + struct device *dev = cxlds->dev; 551 551 + struct pci_dev *pdev = to_pci_dev(dev); 552 552 + struct cxl_dev_id *dev_id; 553 553 + int irq; 554 554 + 555 555 + if (FIELD_GET(CXLDEV_EVENT_INT_MODE_MASK, setting) != CXL_INT_MSI_MSIX) 556 556 + return -ENXIO; 557 557 + 558 558 + /* dev_id must be globally unique and must contain the cxlds */ 559 559 + dev_id = devm_kzalloc(dev, sizeof(*dev_id), GFP_KERNEL); 560 560 + if (!dev_id) 561 561 + return -ENOMEM; 562 562 + dev_id->cxlds = cxlds; 563 563 + 564 564 + irq = pci_irq_vector(pdev, 565 565 + FIELD_GET(CXLDEV_EVENT_INT_MSGNUM_MASK, setting)); 566 566 + if (irq < 0) 567 567 + return irq; 568 568 + 569 569 + return devm_request_threaded_irq(dev, irq, NULL, cxl_event_thread, 570 570 + IRQF_SHARED | IRQF_ONESHOT, NULL, 571 571 + dev_id); 572 572 + } 573 573 + 574 574 + static int cxl_event_get_int_policy(struct cxl_dev_state *cxlds, 575 575 + struct cxl_event_interrupt_policy *policy) 576 576 + { 577 577 + struct cxl_mbox_cmd mbox_cmd = { 578 578 + .opcode = CXL_MBOX_OP_GET_EVT_INT_POLICY, 579 579 + .payload_out = policy, 580 580 + .size_out = sizeof(*policy), 581 581 + }; 582 582 + int rc; 583 583 + 584 584 + rc = cxl_internal_send_cmd(cxlds, &mbox_cmd); 585 585 + if (rc < 0) 586 586 + dev_err(cxlds->dev, "Failed to get event interrupt policy : %d", 587 587 + rc); 588 588 + 589 589 + return rc; 590 590 + } 591 591 + 592 592 + static int cxl_event_config_msgnums(struct cxl_dev_state *cxlds, 593 593 + struct cxl_event_interrupt_policy *policy) 594 594 + { 595 595 + struct cxl_mbox_cmd mbox_cmd; 596 596 + int rc; 597 597 + 598 598 + *policy = (struct cxl_event_interrupt_policy) { 599 599 + .info_settings = CXL_INT_MSI_MSIX, 600 600 + .warn_settings = CXL_INT_MSI_MSIX, 601 601 + .failure_settings = CXL_INT_MSI_MSIX, 602 602 + .fatal_settings = CXL_INT_MSI_MSIX, 603 603 + }; 604 604 + 605 605 + mbox_cmd = (struct cxl_mbox_cmd) { 606 606 + .opcode = CXL_MBOX_OP_SET_EVT_INT_POLICY, 607 607 + .payload_in = policy, 608 608 + .size_in = sizeof(*policy), 609 609 + }; 610 610 + 611 611 + rc = cxl_internal_send_cmd(cxlds, &mbox_cmd); 612 612 + if (rc < 0) { 613 613 + dev_err(cxlds->dev, "Failed to set event interrupt policy : %d", 614 614 + rc); 615 615 + return rc; 616 616 + } 617 617 + 618 618 + /* Retrieve final interrupt settings */ 619 619 + return cxl_event_get_int_policy(cxlds, policy); 620 620 + } 621 621 + 622 622 + static int cxl_event_irqsetup(struct cxl_dev_state *cxlds) 623 623 + { 624 624 + struct cxl_event_interrupt_policy policy; 625 625 + int rc; 626 626 + 627 627 + rc = cxl_event_config_msgnums(cxlds, &policy); 628 628 + if (rc) 629 629 + return rc; 630 630 + 631 631 + rc = cxl_event_req_irq(cxlds, policy.info_settings); 632 632 + if (rc) { 633 633 + dev_err(cxlds->dev, "Failed to get interrupt for event Info log\n"); 634 634 + return rc; 635 635 + } 636 636 + 637 637 + rc = cxl_event_req_irq(cxlds, policy.warn_settings); 638 638 + if (rc) { 639 639 + dev_err(cxlds->dev, "Failed to get interrupt for event Warn log\n"); 640 640 + return rc; 641 641 + } 642 642 + 643 643 + rc = cxl_event_req_irq(cxlds, policy.failure_settings); 644 644 + if (rc) { 645 645 + dev_err(cxlds->dev, "Failed to get interrupt for event Failure log\n"); 646 646 + return rc; 647 647 + } 648 648 + 649 649 + rc = cxl_event_req_irq(cxlds, policy.fatal_settings); 650 650 + if (rc) { 651 651 + dev_err(cxlds->dev, "Failed to get interrupt for event Fatal log\n"); 652 652 + return rc; 653 653 + } 654 654 + 655 655 + return 0; 656 656 + } 657 657 + 658 658 + static bool cxl_event_int_is_fw(u8 setting) 659 659 + { 660 660 + u8 mode = FIELD_GET(CXLDEV_EVENT_INT_MODE_MASK, setting); 661 661 + 662 662 + return mode == CXL_INT_FW; 663 663 + } 664 664 + 665 665 + static int cxl_event_config(struct pci_host_bridge *host_bridge, 666 666 + struct cxl_dev_state *cxlds) 667 667 + { 668 668 + struct cxl_event_interrupt_policy policy; 669 669 + int rc; 670 670 + 671 671 + /* 672 672 + * When BIOS maintains CXL error reporting control, it will process 673 673 + * event records. Only one agent can do so. 674 674 + */ 675 675 + if (!host_bridge->native_cxl_error) 676 676 + return 0; 677 677 + 678 678 + rc = cxl_mem_alloc_event_buf(cxlds); 679 679 + if (rc) 680 680 + return rc; 681 681 + 682 682 + rc = cxl_event_get_int_policy(cxlds, &policy); 683 683 + if (rc) 684 684 + return rc; 685 685 + 686 686 + if (cxl_event_int_is_fw(policy.info_settings) || 687 687 + cxl_event_int_is_fw(policy.warn_settings) || 688 688 + cxl_event_int_is_fw(policy.failure_settings) || 689 689 + cxl_event_int_is_fw(policy.fatal_settings)) { 690 690 + dev_err(cxlds->dev, "FW still in control of Event Logs despite _OSC settings\n"); 691 691 + return -EBUSY; 692 692 + } 693 693 + 694 694 + rc = cxl_event_irqsetup(cxlds); 695 695 + if (rc) 696 696 + return rc; 697 697 + 698 698 + cxl_mem_get_event_records(cxlds, CXLDEV_EVENT_STATUS_ALL); 699 699 + 700 700 + return 0; 418 701 } 419 702 420 703 static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) 421 704 { 705 705 + struct pci_host_bridge *host_bridge = pci_find_host_bridge(pdev->bus); 422 706 struct cxl_register_map map; 423 707 struct cxl_memdev *cxlmd; 424 708 struct cxl_dev_state *cxlds; ··· 716 436 rc = pcim_enable_device(pdev); 717 437 if (rc) 718 438 return rc; 439 439 + pci_set_master(pdev); 719 440 720 441 cxlds = cxl_dev_state_create(&pdev->dev); 721 442 if (IS_ERR(cxlds)) ··· 765 484 if (rc) 766 485 return rc; 767 486 487 487 + rc = cxl_set_timestamp(cxlds); 488 488 + if (rc) 489 489 + return rc; 490 490 + 768 491 rc = cxl_dev_state_identify(cxlds); 769 492 if (rc) 770 493 return rc; ··· 777 492 if (rc) 778 493 return rc; 779 494 495 495 + rc = cxl_alloc_irq_vectors(pdev); 496 496 + if (rc) 497 497 + return rc; 498 498 + 780 499 cxlmd = devm_cxl_add_memdev(cxlds); 781 500 if (IS_ERR(cxlmd)) 782 501 return PTR_ERR(cxlmd); 783 502 784 784 - if (cxlds->regs.ras) { 785 785 - pci_enable_pcie_error_reporting(pdev); 786 786 - rc = devm_add_action_or_reset(&pdev->dev, disable_aer, pdev); 787 787 - if (rc) 788 788 - return rc; 789 789 - } 503 503 + rc = cxl_event_config(host_bridge, cxlds); 504 504 + if (rc) 505 505 + return rc; 506 506 + 507 507 + rc = cxl_pci_ras_unmask(pdev); 508 508 + if (rc) 509 509 + dev_dbg(&pdev->dev, "No RAS reporting unmasked\n"); 510 510 + 790 511 pci_save_state(pdev); 791 512 792 513 return rc; ··· 804 513 { /* terminate list */ }, 805 514 }; 806 515 MODULE_DEVICE_TABLE(pci, cxl_mem_pci_tbl); 807 807 - 808 808 - /* CXL spec rev3.0 8.2.4.16.1 */ 809 809 - static void header_log_copy(struct cxl_dev_state *cxlds, u32 *log) 810 810 - { 811 811 - void __iomem *addr; 812 812 - u32 *log_addr; 813 813 - int i, log_u32_size = CXL_HEADERLOG_SIZE / sizeof(u32); 814 814 - 815 815 - addr = cxlds->regs.ras + CXL_RAS_HEADER_LOG_OFFSET; 816 816 - log_addr = log; 817 817 - 818 818 - for (i = 0; i < log_u32_size; i++) { 819 819 - *log_addr = readl(addr); 820 820 - log_addr++; 821 821 - addr += sizeof(u32); 822 822 - } 823 823 - } 824 824 - 825 825 - /* 826 826 - * Log the state of the RAS status registers and prepare them to log the 827 827 - * next error status. Return 1 if reset needed. 828 828 - */ 829 829 - static bool cxl_report_and_clear(struct cxl_dev_state *cxlds) 830 830 - { 831 831 - struct cxl_memdev *cxlmd = cxlds->cxlmd; 832 832 - struct device *dev = &cxlmd->dev; 833 833 - u32 hl[CXL_HEADERLOG_SIZE_U32]; 834 834 - void __iomem *addr; 835 835 - u32 status; 836 836 - u32 fe; 837 837 - 838 838 - if (!cxlds->regs.ras) 839 839 - return false; 840 840 - 841 841 - addr = cxlds->regs.ras + CXL_RAS_UNCORRECTABLE_STATUS_OFFSET; 842 842 - status = readl(addr); 843 843 - if (!(status & CXL_RAS_UNCORRECTABLE_STATUS_MASK)) 844 844 - return false; 845 845 - 846 846 - /* If multiple errors, log header points to first error from ctrl reg */ 847 847 - if (hweight32(status) > 1) { 848 848 - void __iomem *rcc_addr = 849 849 - cxlds->regs.ras + CXL_RAS_CAP_CONTROL_OFFSET; 850 850 - 851 851 - fe = BIT(FIELD_GET(CXL_RAS_CAP_CONTROL_FE_MASK, 852 852 - readl(rcc_addr))); 853 853 - } else { 854 854 - fe = status; 855 855 - } 856 856 - 857 857 - header_log_copy(cxlds, hl); 858 858 - trace_cxl_aer_uncorrectable_error(dev, status, fe, hl); 859 859 - writel(status & CXL_RAS_UNCORRECTABLE_STATUS_MASK, addr); 860 860 - 861 861 - return true; 862 862 - } 863 863 - 864 864 - static pci_ers_result_t cxl_error_detected(struct pci_dev *pdev, 865 865 - pci_channel_state_t state) 866 866 - { 867 867 - struct cxl_dev_state *cxlds = pci_get_drvdata(pdev); 868 868 - struct cxl_memdev *cxlmd = cxlds->cxlmd; 869 869 - struct device *dev = &cxlmd->dev; 870 870 - bool ue; 871 871 - 872 872 - /* 873 873 - * A frozen channel indicates an impending reset which is fatal to 874 874 - * CXL.mem operation, and will likely crash the system. On the off 875 875 - * chance the situation is recoverable dump the status of the RAS 876 876 - * capability registers and bounce the active state of the memdev. 877 877 - */ 878 878 - ue = cxl_report_and_clear(cxlds); 879 879 - 880 880 - switch (state) { 881 881 - case pci_channel_io_normal: 882 882 - if (ue) { 883 883 - device_release_driver(dev); 884 884 - return PCI_ERS_RESULT_NEED_RESET; 885 885 - } 886 886 - return PCI_ERS_RESULT_CAN_RECOVER; 887 887 - case pci_channel_io_frozen: 888 888 - dev_warn(&pdev->dev, 889 889 - "%s: frozen state error detected, disable CXL.mem\n", 890 890 - dev_name(dev)); 891 891 - device_release_driver(dev); 892 892 - return PCI_ERS_RESULT_NEED_RESET; 893 893 - case pci_channel_io_perm_failure: 894 894 - dev_warn(&pdev->dev, 895 895 - "failure state error detected, request disconnect\n"); 896 896 - return PCI_ERS_RESULT_DISCONNECT; 897 897 - } 898 898 - return PCI_ERS_RESULT_NEED_RESET; 899 899 - } 900 516 901 517 static pci_ers_result_t cxl_slot_reset(struct pci_dev *pdev) 902 518 { ··· 827 629 828 630 dev_info(&pdev->dev, "%s: error resume %s\n", dev_name(dev), 829 631 dev->driver ? "successful" : "failed"); 830 830 - } 831 831 - 832 832 - static void cxl_cor_error_detected(struct pci_dev *pdev) 833 833 - { 834 834 - struct cxl_dev_state *cxlds = pci_get_drvdata(pdev); 835 835 - struct cxl_memdev *cxlmd = cxlds->cxlmd; 836 836 - struct device *dev = &cxlmd->dev; 837 837 - void __iomem *addr; 838 838 - u32 status; 839 839 - 840 840 - if (!cxlds->regs.ras) 841 841 - return; 842 842 - 843 843 - addr = cxlds->regs.ras + CXL_RAS_CORRECTABLE_STATUS_OFFSET; 844 844 - status = readl(addr); 845 845 - if (status & CXL_RAS_CORRECTABLE_STATUS_MASK) { 846 846 - writel(status & CXL_RAS_CORRECTABLE_STATUS_MASK, addr); 847 847 - trace_cxl_aer_correctable_error(dev, status); 848 848 - } 849 632 } 850 633 851 634 static const struct pci_error_handlers cxl_error_handlers = {

+1

drivers/cxl/pmem.c

reviewed

··· 76 76 return rc; 77 77 78 78 set_bit(NDD_LABELING, &flags); 79 79 + set_bit(NDD_REGISTER_SYNC, &flags); 79 80 set_bit(ND_CMD_GET_CONFIG_SIZE, &cmd_mask); 80 81 set_bit(ND_CMD_GET_CONFIG_DATA, &cmd_mask); 81 82 set_bit(ND_CMD_SET_CONFIG_DATA, &cmd_mask);

+89 -30

drivers/cxl/port.c

reviewed

··· 30 30 schedule_cxl_memdev_detach(cxlmd); 31 31 } 32 32 33 33 - static int cxl_port_probe(struct device *dev) 33 33 + static int discover_region(struct device *dev, void *root) 34 34 { 35 35 - struct cxl_port *port = to_cxl_port(dev); 35 35 + struct cxl_endpoint_decoder *cxled; 36 36 + int rc; 37 37 + 38 38 + if (!is_endpoint_decoder(dev)) 39 39 + return 0; 40 40 + 41 41 + cxled = to_cxl_endpoint_decoder(dev); 42 42 + if ((cxled->cxld.flags & CXL_DECODER_F_ENABLE) == 0) 43 43 + return 0; 44 44 + 45 45 + if (cxled->state != CXL_DECODER_STATE_AUTO) 46 46 + return 0; 47 47 + 48 48 + /* 49 49 + * Region enumeration is opportunistic, if this add-event fails, 50 50 + * continue to the next endpoint decoder. 51 51 + */ 52 52 + rc = cxl_add_to_region(root, cxled); 53 53 + if (rc) 54 54 + dev_dbg(dev, "failed to add to region: %#llx-%#llx\n", 55 55 + cxled->cxld.hpa_range.start, cxled->cxld.hpa_range.end); 56 56 + 57 57 + return 0; 58 58 + } 59 59 + 60 60 + static int cxl_switch_port_probe(struct cxl_port *port) 61 61 + { 36 62 struct cxl_hdm *cxlhdm; 37 63 int rc; 38 64 65 65 + rc = devm_cxl_port_enumerate_dports(port); 66 66 + if (rc < 0) 67 67 + return rc; 39 68 40 40 - if (!is_cxl_endpoint(port)) { 41 41 - rc = devm_cxl_port_enumerate_dports(port); 42 42 - if (rc < 0) 43 43 - return rc; 44 44 - if (rc == 1) 45 45 - return devm_cxl_add_passthrough_decoder(port); 46 46 - } 69 69 + if (rc == 1) 70 70 + return devm_cxl_add_passthrough_decoder(port); 47 71 48 48 - cxlhdm = devm_cxl_setup_hdm(port); 72 72 + cxlhdm = devm_cxl_setup_hdm(port, NULL); 49 73 if (IS_ERR(cxlhdm)) 50 74 return PTR_ERR(cxlhdm); 51 75 52 52 - if (is_cxl_endpoint(port)) { 53 53 - struct cxl_memdev *cxlmd = to_cxl_memdev(port->uport); 54 54 - struct cxl_dev_state *cxlds = cxlmd->cxlds; 76 76 + return devm_cxl_enumerate_decoders(cxlhdm, NULL); 77 77 + } 55 78 56 56 - /* Cache the data early to ensure is_visible() works */ 57 57 - read_cdat_data(port); 79 79 + static int cxl_endpoint_port_probe(struct cxl_port *port) 80 80 + { 81 81 + struct cxl_memdev *cxlmd = to_cxl_memdev(port->uport); 82 82 + struct cxl_endpoint_dvsec_info info = { 0 }; 83 83 + struct cxl_dev_state *cxlds = cxlmd->cxlds; 84 84 + struct cxl_hdm *cxlhdm; 85 85 + struct cxl_port *root; 86 86 + int rc; 58 87 59 59 - get_device(&cxlmd->dev); 60 60 - rc = devm_add_action_or_reset(dev, schedule_detach, cxlmd); 61 61 - if (rc) 62 62 - return rc; 88 88 + rc = cxl_dvsec_rr_decode(cxlds->dev, cxlds->cxl_dvsec, &info); 89 89 + if (rc < 0) 90 90 + return rc; 63 91 64 64 - rc = cxl_hdm_decode_init(cxlds, cxlhdm); 65 65 - if (rc) 66 66 - return rc; 92 92 + cxlhdm = devm_cxl_setup_hdm(port, &info); 93 93 + if (IS_ERR(cxlhdm)) 94 94 + return PTR_ERR(cxlhdm); 67 95 68 68 - rc = cxl_await_media_ready(cxlds); 69 69 - if (rc) { 70 70 - dev_err(dev, "Media not active (%d)\n", rc); 71 71 - return rc; 72 72 - } 73 73 - } 96 96 + /* Cache the data early to ensure is_visible() works */ 97 97 + read_cdat_data(port); 74 98 75 75 - rc = devm_cxl_enumerate_decoders(cxlhdm); 99 99 + get_device(&cxlmd->dev); 100 100 + rc = devm_add_action_or_reset(&port->dev, schedule_detach, cxlmd); 101 101 + if (rc) 102 102 + return rc; 103 103 + 104 104 + rc = cxl_hdm_decode_init(cxlds, cxlhdm, &info); 105 105 + if (rc) 106 106 + return rc; 107 107 + 108 108 + rc = cxl_await_media_ready(cxlds); 76 109 if (rc) { 77 77 - dev_err(dev, "Couldn't enumerate decoders (%d)\n", rc); 110 110 + dev_err(&port->dev, "Media not active (%d)\n", rc); 78 111 return rc; 79 112 } 80 113 114 114 + rc = devm_cxl_enumerate_decoders(cxlhdm, &info); 115 115 + if (rc) 116 116 + return rc; 117 117 + 118 118 + /* 119 119 + * This can't fail in practice as CXL root exit unregisters all 120 120 + * descendant ports and that in turn synchronizes with cxl_port_probe() 121 121 + */ 122 122 + root = find_cxl_root(&cxlmd->dev); 123 123 + 124 124 + /* 125 125 + * Now that all endpoint decoders are successfully enumerated, try to 126 126 + * assemble regions from committed decoders 127 127 + */ 128 128 + device_for_each_child(&port->dev, root, discover_region); 129 129 + put_device(&root->dev); 130 130 + 81 131 return 0; 132 132 + } 133 133 + 134 134 + static int cxl_port_probe(struct device *dev) 135 135 + { 136 136 + struct cxl_port *port = to_cxl_port(dev); 137 137 + 138 138 + if (is_cxl_endpoint(port)) 139 139 + return cxl_endpoint_port_probe(port); 140 140 + return cxl_switch_port_probe(port); 82 141 } 83 142 84 143 static ssize_t CDAT_read(struct file *filp, struct kobject *kobj,

+15 -2

drivers/dax/Kconfig

reviewed

··· 44 44 45 45 Say M if unsure. 46 46 47 47 + config DEV_DAX_CXL 48 48 + tristate "CXL DAX: direct access to CXL RAM regions" 49 49 + depends on CXL_BUS && CXL_REGION && DEV_DAX 50 50 + default CXL_REGION && DEV_DAX 51 51 + help 52 52 + CXL RAM regions are either mapped by platform-firmware 53 53 + and published in the initial system-memory map as "System RAM", mapped 54 54 + by platform-firmware as "Soft Reserved", or dynamically provisioned 55 55 + after boot by the CXL driver. In the latter two cases a device-dax 56 56 + instance is created to access that unmapped-by-default address range. 57 57 + Per usual it can remain as dedicated access via a device interface, or 58 58 + converted to "System RAM" via the dax_kmem facility. 59 59 + 47 60 config DEV_DAX_HMEM_DEVICES 48 48 - depends on DEV_DAX_HMEM && DAX=y 61 61 + depends on DEV_DAX_HMEM && DAX 49 62 def_bool y 50 63 51 64 config DEV_DAX_KMEM 52 52 - tristate "KMEM DAX: volatile-use of persistent memory" 65 65 + tristate "KMEM DAX: map dax-devices as System-RAM" 53 66 default DEV_DAX 54 67 depends on DEV_DAX 55 68 depends on MEMORY_HOTPLUG # for add_memory() and friends

+2

drivers/dax/Makefile

reviewed

··· 3 3 obj-$(CONFIG_DEV_DAX) += device_dax.o 4 4 obj-$(CONFIG_DEV_DAX_KMEM) += kmem.o 5 5 obj-$(CONFIG_DEV_DAX_PMEM) += dax_pmem.o 6 6 + obj-$(CONFIG_DEV_DAX_CXL) += dax_cxl.o 6 7 7 8 dax-y := super.o 8 9 dax-y += bus.o 9 10 device_dax-y := device.o 10 11 dax_pmem-y := pmem.o 12 12 + dax_cxl-y := cxl.o 11 13 12 14 obj-y += hmem/

+23 -32

drivers/dax/bus.c

reviewed

··· 56 56 return match; 57 57 } 58 58 59 59 + static int dax_match_type(struct dax_device_driver *dax_drv, struct device *dev) 60 60 + { 61 61 + enum dax_driver_type type = DAXDRV_DEVICE_TYPE; 62 62 + struct dev_dax *dev_dax = to_dev_dax(dev); 63 63 + 64 64 + if (dev_dax->region->res.flags & IORESOURCE_DAX_KMEM) 65 65 + type = DAXDRV_KMEM_TYPE; 66 66 + 67 67 + if (dax_drv->type == type) 68 68 + return 1; 69 69 + 70 70 + /* default to device mode if dax_kmem is disabled */ 71 71 + if (dax_drv->type == DAXDRV_DEVICE_TYPE && 72 72 + !IS_ENABLED(CONFIG_DEV_DAX_KMEM)) 73 73 + return 1; 74 74 + 75 75 + return 0; 76 76 + } 77 77 + 59 78 enum id_action { 60 79 ID_REMOVE, 61 80 ID_ADD, ··· 235 216 { 236 217 struct dax_device_driver *dax_drv = to_dax_drv(drv); 237 218 238 238 - /* 239 239 - * All but the 'device-dax' driver, which has 'match_always' 240 240 - * set, requires an exact id match. 241 241 - */ 242 242 - if (dax_drv->match_always) 219 219 + if (dax_match_id(dax_drv, dev)) 243 220 return 1; 244 244 - 245 245 - return dax_match_id(dax_drv, dev); 221 221 + return dax_match_type(dax_drv, dev); 246 222 } 247 223 248 224 /* ··· 441 427 dev_dbg(dev, "%s\n", __func__); 442 428 443 429 kill_dev_dax(dev_dax); 444 444 - free_dev_dax_ranges(dev_dax); 445 430 device_del(dev); 431 431 + free_dev_dax_ranges(dev_dax); 446 432 put_device(dev); 447 433 } 448 434 ··· 1427 1413 } 1428 1414 EXPORT_SYMBOL_GPL(devm_create_dev_dax); 1429 1415 1430 1430 - static int match_always_count; 1431 1431 - 1432 1416 int __dax_driver_register(struct dax_device_driver *dax_drv, 1433 1417 struct module *module, const char *mod_name) 1434 1418 { 1435 1419 struct device_driver *drv = &dax_drv->drv; 1436 1436 - int rc = 0; 1437 1420 1438 1421 /* 1439 1422 * dax_bus_probe() calls dax_drv->probe() unconditionally. ··· 1445 1434 drv->mod_name = mod_name; 1446 1435 drv->bus = &dax_bus_type; 1447 1436 1448 1448 - /* there can only be one default driver */ 1449 1449 - mutex_lock(&dax_bus_lock); 1450 1450 - match_always_count += dax_drv->match_always; 1451 1451 - if (match_always_count > 1) { 1452 1452 - match_always_count--; 1453 1453 - WARN_ON(1); 1454 1454 - rc = -EINVAL; 1455 1455 - } 1456 1456 - mutex_unlock(&dax_bus_lock); 1457 1457 - if (rc) 1458 1458 - return rc; 1459 1459 - 1460 1460 - rc = driver_register(drv); 1461 1461 - if (rc && dax_drv->match_always) { 1462 1462 - mutex_lock(&dax_bus_lock); 1463 1463 - match_always_count -= dax_drv->match_always; 1464 1464 - mutex_unlock(&dax_bus_lock); 1465 1465 - } 1466 1466 - 1467 1467 - return rc; 1437 1437 + return driver_register(drv); 1468 1438 } 1469 1439 EXPORT_SYMBOL_GPL(__dax_driver_register); 1470 1440 ··· 1455 1463 struct dax_id *dax_id, *_id; 1456 1464 1457 1465 mutex_lock(&dax_bus_lock); 1458 1458 - match_always_count -= dax_drv->match_always; 1459 1466 list_for_each_entry_safe(dax_id, _id, &dax_drv->ids, list) { 1460 1467 list_del(&dax_id->list); 1461 1468 kfree(dax_id);

+10 -2

drivers/dax/bus.h

reviewed

··· 11 11 struct dax_region; 12 12 void dax_region_put(struct dax_region *dax_region); 13 13 14 14 - #define IORESOURCE_DAX_STATIC (1UL << 0) 14 14 + /* dax bus specific ioresource flags */ 15 15 + #define IORESOURCE_DAX_STATIC BIT(0) 16 16 + #define IORESOURCE_DAX_KMEM BIT(1) 17 17 + 15 18 struct dax_region *alloc_dax_region(struct device *parent, int region_id, 16 19 struct range *range, int target_node, unsigned int align, 17 20 unsigned long flags); ··· 28 25 29 26 struct dev_dax *devm_create_dev_dax(struct dev_dax_data *data); 30 27 28 28 + enum dax_driver_type { 29 29 + DAXDRV_KMEM_TYPE, 30 30 + DAXDRV_DEVICE_TYPE, 31 31 + }; 32 32 + 31 33 struct dax_device_driver { 32 34 struct device_driver drv; 33 35 struct list_head ids; 34 34 - int match_always; 36 36 + enum dax_driver_type type; 35 37 int (*probe)(struct dev_dax *dev); 36 38 void (*remove)(struct dev_dax *dev); 37 39 };

+53

drivers/dax/cxl.c

reviewed

··· 1 1 + // SPDX-License-Identifier: GPL-2.0-only 2 2 + /* Copyright(c) 2023 Intel Corporation. All rights reserved. */ 3 3 + #include <linux/module.h> 4 4 + #include <linux/dax.h> 5 5 + 6 6 + #include "../cxl/cxl.h" 7 7 + #include "bus.h" 8 8 + 9 9 + static int cxl_dax_region_probe(struct device *dev) 10 10 + { 11 11 + struct cxl_dax_region *cxlr_dax = to_cxl_dax_region(dev); 12 12 + int nid = phys_to_target_node(cxlr_dax->hpa_range.start); 13 13 + struct cxl_region *cxlr = cxlr_dax->cxlr; 14 14 + struct dax_region *dax_region; 15 15 + struct dev_dax_data data; 16 16 + struct dev_dax *dev_dax; 17 17 + 18 18 + if (nid == NUMA_NO_NODE) 19 19 + nid = memory_add_physaddr_to_nid(cxlr_dax->hpa_range.start); 20 20 + 21 21 + dax_region = alloc_dax_region(dev, cxlr->id, &cxlr_dax->hpa_range, nid, 22 22 + PMD_SIZE, IORESOURCE_DAX_KMEM); 23 23 + if (!dax_region) 24 24 + return -ENOMEM; 25 25 + 26 26 + data = (struct dev_dax_data) { 27 27 + .dax_region = dax_region, 28 28 + .id = -1, 29 29 + .size = range_len(&cxlr_dax->hpa_range), 30 30 + }; 31 31 + dev_dax = devm_create_dev_dax(&data); 32 32 + if (IS_ERR(dev_dax)) 33 33 + return PTR_ERR(dev_dax); 34 34 + 35 35 + /* child dev_dax instances now own the lifetime of the dax_region */ 36 36 + dax_region_put(dax_region); 37 37 + return 0; 38 38 + } 39 39 + 40 40 + static struct cxl_driver cxl_dax_region_driver = { 41 41 + .name = "cxl_dax_region", 42 42 + .probe = cxl_dax_region_probe, 43 43 + .id = CXL_DEVICE_DAX_REGION, 44 44 + .drv = { 45 45 + .suppress_bind_attrs = true, 46 46 + }, 47 47 + }; 48 48 + 49 49 + module_cxl_driver(cxl_dax_region_driver); 50 50 + MODULE_ALIAS_CXL(CXL_DEVICE_DAX_REGION); 51 51 + MODULE_LICENSE("GPL"); 52 52 + MODULE_AUTHOR("Intel Corporation"); 53 53 + MODULE_IMPORT_NS(CXL);

+1 -2

drivers/dax/device.c

reviewed

··· 475 475 476 476 static struct dax_device_driver device_dax_driver = { 477 477 .probe = dev_dax_probe, 478 478 - /* all probe actions are unwound by devm, so .remove isn't necessary */ 479 479 - .match_always = 1, 478 478 + .type = DAXDRV_DEVICE_TYPE, 480 479 }; 481 480 482 481 static int __init dax_init(void)

+2 -1

drivers/dax/hmem/Makefile

reviewed

··· 1 1 # SPDX-License-Identifier: GPL-2.0 2 2 - obj-$(CONFIG_DEV_DAX_HMEM) += dax_hmem.o 2 2 + # device_hmem.o deliberately precedes dax_hmem.o for initcall ordering 3 3 obj-$(CONFIG_DEV_DAX_HMEM_DEVICES) += device_hmem.o 4 4 + obj-$(CONFIG_DEV_DAX_HMEM) += dax_hmem.o 4 5 5 6 device_hmem-y := device.o 6 7 dax_hmem-y := hmem.o

+47 -59

drivers/dax/hmem/device.c

reviewed

··· 8 8 static bool nohmem; 9 9 module_param_named(disable, nohmem, bool, 0444); 10 10 11 11 + static bool platform_initialized; 12 12 + static DEFINE_MUTEX(hmem_resource_lock); 11 13 static struct resource hmem_active = { 12 14 .name = "HMEM devices", 13 15 .start = 0, ··· 17 15 .flags = IORESOURCE_MEM, 18 16 }; 19 17 20 20 - void hmem_register_device(int target_nid, struct resource *r) 18 18 + int walk_hmem_resources(struct device *host, walk_hmem_fn fn) 21 19 { 22 22 - /* define a clean / non-busy resource for the platform device */ 23 23 - struct resource res = { 24 24 - .start = r->start, 25 25 - .end = r->end, 26 26 - .flags = IORESOURCE_MEM, 27 27 - .desc = IORES_DESC_SOFT_RESERVED, 28 28 - }; 20 20 + struct resource *res; 21 21 + int rc = 0; 22 22 + 23 23 + mutex_lock(&hmem_resource_lock); 24 24 + for (res = hmem_active.child; res; res = res->sibling) { 25 25 + rc = fn(host, (int) res->desc, res); 26 26 + if (rc) 27 27 + break; 28 28 + } 29 29 + mutex_unlock(&hmem_resource_lock); 30 30 + return rc; 31 31 + } 32 32 + EXPORT_SYMBOL_GPL(walk_hmem_resources); 33 33 + 34 34 + static void __hmem_register_resource(int target_nid, struct resource *res) 35 35 + { 29 36 struct platform_device *pdev; 30 30 - struct memregion_info info; 31 31 - int rc, id; 37 37 + struct resource *new; 38 38 + int rc; 32 39 33 33 - if (nohmem) 34 34 - return; 35 35 - 36 36 - rc = region_intersects(res.start, resource_size(&res), IORESOURCE_MEM, 37 37 - IORES_DESC_SOFT_RESERVED); 38 38 - if (rc != REGION_INTERSECTS) 39 39 - return; 40 40 - 41 41 - id = memregion_alloc(GFP_KERNEL); 42 42 - if (id < 0) { 43 43 - pr_err("memregion allocation failure for %pr\n", &res); 40 40 + new = __request_region(&hmem_active, res->start, resource_size(res), "", 41 41 + 0); 42 42 + if (!new) { 43 43 + pr_debug("hmem range %pr already active\n", res); 44 44 return; 45 45 } 46 46 47 47 - pdev = platform_device_alloc("hmem", id); 47 47 + new->desc = target_nid; 48 48 + 49 49 + if (platform_initialized) 50 50 + return; 51 51 + 52 52 + pdev = platform_device_alloc("hmem_platform", 0); 48 53 if (!pdev) { 49 49 - pr_err("hmem device allocation failure for %pr\n", &res); 50 50 - goto out_pdev; 51 51 - } 52 52 - 53 53 - if (!__request_region(&hmem_active, res.start, resource_size(&res), 54 54 - dev_name(&pdev->dev), 0)) { 55 55 - dev_dbg(&pdev->dev, "hmem range %pr already active\n", &res); 56 56 - goto out_active; 57 57 - } 58 58 - 59 59 - pdev->dev.numa_node = numa_map_to_online_node(target_nid); 60 60 - info = (struct memregion_info) { 61 61 - .target_node = target_nid, 62 62 - }; 63 63 - rc = platform_device_add_data(pdev, &info, sizeof(info)); 64 64 - if (rc < 0) { 65 65 - pr_err("hmem memregion_info allocation failure for %pr\n", &res); 66 66 - goto out_resource; 67 67 - } 68 68 - 69 69 - rc = platform_device_add_resources(pdev, &res, 1); 70 70 - if (rc < 0) { 71 71 - pr_err("hmem resource allocation failure for %pr\n", &res); 72 72 - goto out_resource; 54 54 + pr_err_once("failed to register device-dax hmem_platform device\n"); 55 55 + return; 73 56 } 74 57 75 58 rc = platform_device_add(pdev); 76 76 - if (rc < 0) { 77 77 - dev_err(&pdev->dev, "device add failed for %pr\n", &res); 78 78 - goto out_resource; 79 79 - } 59 59 + if (rc) 60 60 + platform_device_put(pdev); 61 61 + else 62 62 + platform_initialized = true; 63 63 + } 80 64 81 81 - return; 65 65 + void hmem_register_resource(int target_nid, struct resource *res) 66 66 + { 67 67 + if (nohmem) 68 68 + return; 82 69 83 83 - out_resource: 84 84 - __release_region(&hmem_active, res.start, resource_size(&res)); 85 85 - out_active: 86 86 - platform_device_put(pdev); 87 87 - out_pdev: 88 88 - memregion_free(id); 70 70 + mutex_lock(&hmem_resource_lock); 71 71 + __hmem_register_resource(target_nid, res); 72 72 + mutex_unlock(&hmem_resource_lock); 89 73 } 90 74 91 75 static __init int hmem_register_one(struct resource *res, void *data) 92 76 { 93 93 - hmem_register_device(phys_to_target_node(res->start), res); 77 77 + hmem_register_resource(phys_to_target_node(res->start), res); 94 78 95 79 return 0; 96 80 } ··· 92 104 * As this is a fallback for address ranges unclaimed by the ACPI HMAT 93 105 * parsing it must be at an initcall level greater than hmat_init(). 94 106 */ 95 95 - late_initcall(hmem_init); 107 107 + device_initcall(hmem_init);

+130 -18

drivers/dax/hmem/hmem.c

reviewed

··· 3 3 #include <linux/memregion.h> 4 4 #include <linux/module.h> 5 5 #include <linux/pfn_t.h> 6 6 + #include <linux/dax.h> 6 7 #include "../bus.h" 7 8 8 9 static bool region_idle; ··· 11 10 12 11 static int dax_hmem_probe(struct platform_device *pdev) 13 12 { 13 13 + unsigned long flags = IORESOURCE_DAX_KMEM; 14 14 struct device *dev = &pdev->dev; 15 15 struct dax_region *dax_region; 16 16 struct memregion_info *mri; 17 17 struct dev_dax_data data; 18 18 struct dev_dax *dev_dax; 19 19 - struct resource *res; 20 20 - struct range range; 21 19 22 22 - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); 23 23 - if (!res) 24 24 - return -ENOMEM; 20 20 + /* 21 21 + * @region_idle == true indicates that an administrative agent 22 22 + * wants to manipulate the range partitioning before the devices 23 23 + * are created, so do not send them to the dax_kmem driver by 24 24 + * default. 25 25 + */ 26 26 + if (region_idle) 27 27 + flags = 0; 25 28 26 29 mri = dev->platform_data; 27 27 - range.start = res->start; 28 28 - range.end = res->end; 29 29 - dax_region = alloc_dax_region(dev, pdev->id, &range, mri->target_node, 30 30 - PMD_SIZE, 0); 30 30 + dax_region = alloc_dax_region(dev, pdev->id, &mri->range, 31 31 + mri->target_node, PMD_SIZE, flags); 31 32 if (!dax_region) 32 33 return -ENOMEM; 33 34 34 35 data = (struct dev_dax_data) { 35 36 .dax_region = dax_region, 36 37 .id = -1, 37 37 - .size = region_idle ? 0 : resource_size(res), 38 38 + .size = region_idle ? 0 : range_len(&mri->range), 38 39 }; 39 40 dev_dax = devm_create_dev_dax(&data); 40 41 if (IS_ERR(dev_dax)) ··· 47 44 return 0; 48 45 } 49 46 50 50 - static int dax_hmem_remove(struct platform_device *pdev) 51 51 - { 52 52 - /* devm handles teardown */ 53 53 - return 0; 54 54 - } 55 55 - 56 47 static struct platform_driver dax_hmem_driver = { 57 48 .probe = dax_hmem_probe, 58 58 - .remove = dax_hmem_remove, 59 49 .driver = { 60 50 .name = "hmem", 61 51 }, 62 52 }; 63 53 64 64 - module_platform_driver(dax_hmem_driver); 54 54 + static void release_memregion(void *data) 55 55 + { 56 56 + memregion_free((long) data); 57 57 + } 58 58 + 59 59 + static void release_hmem(void *pdev) 60 60 + { 61 61 + platform_device_unregister(pdev); 62 62 + } 63 63 + 64 64 + static int hmem_register_device(struct device *host, int target_nid, 65 65 + const struct resource *res) 66 66 + { 67 67 + struct platform_device *pdev; 68 68 + struct memregion_info info; 69 69 + long id; 70 70 + int rc; 71 71 + 72 72 + if (IS_ENABLED(CONFIG_CXL_REGION) && 73 73 + region_intersects(res->start, resource_size(res), IORESOURCE_MEM, 74 74 + IORES_DESC_CXL) != REGION_DISJOINT) { 75 75 + dev_dbg(host, "deferring range to CXL: %pr\n", res); 76 76 + return 0; 77 77 + } 78 78 + 79 79 + rc = region_intersects(res->start, resource_size(res), IORESOURCE_MEM, 80 80 + IORES_DESC_SOFT_RESERVED); 81 81 + if (rc != REGION_INTERSECTS) 82 82 + return 0; 83 83 + 84 84 + id = memregion_alloc(GFP_KERNEL); 85 85 + if (id < 0) { 86 86 + dev_err(host, "memregion allocation failure for %pr\n", res); 87 87 + return -ENOMEM; 88 88 + } 89 89 + rc = devm_add_action_or_reset(host, release_memregion, (void *) id); 90 90 + if (rc) 91 91 + return rc; 92 92 + 93 93 + pdev = platform_device_alloc("hmem", id); 94 94 + if (!pdev) { 95 95 + dev_err(host, "device allocation failure for %pr\n", res); 96 96 + return -ENOMEM; 97 97 + } 98 98 + 99 99 + pdev->dev.numa_node = numa_map_to_online_node(target_nid); 100 100 + info = (struct memregion_info) { 101 101 + .target_node = target_nid, 102 102 + .range = { 103 103 + .start = res->start, 104 104 + .end = res->end, 105 105 + }, 106 106 + }; 107 107 + rc = platform_device_add_data(pdev, &info, sizeof(info)); 108 108 + if (rc < 0) { 109 109 + dev_err(host, "memregion_info allocation failure for %pr\n", 110 110 + res); 111 111 + goto out_put; 112 112 + } 113 113 + 114 114 + rc = platform_device_add(pdev); 115 115 + if (rc < 0) { 116 116 + dev_err(host, "%s add failed for %pr\n", dev_name(&pdev->dev), 117 117 + res); 118 118 + goto out_put; 119 119 + } 120 120 + 121 121 + return devm_add_action_or_reset(host, release_hmem, pdev); 122 122 + 123 123 + out_put: 124 124 + platform_device_put(pdev); 125 125 + return rc; 126 126 + } 127 127 + 128 128 + static int dax_hmem_platform_probe(struct platform_device *pdev) 129 129 + { 130 130 + return walk_hmem_resources(&pdev->dev, hmem_register_device); 131 131 + } 132 132 + 133 133 + static struct platform_driver dax_hmem_platform_driver = { 134 134 + .probe = dax_hmem_platform_probe, 135 135 + .driver = { 136 136 + .name = "hmem_platform", 137 137 + }, 138 138 + }; 139 139 + 140 140 + static __init int dax_hmem_init(void) 141 141 + { 142 142 + int rc; 143 143 + 144 144 + rc = platform_driver_register(&dax_hmem_platform_driver); 145 145 + if (rc) 146 146 + return rc; 147 147 + 148 148 + rc = platform_driver_register(&dax_hmem_driver); 149 149 + if (rc) 150 150 + platform_driver_unregister(&dax_hmem_platform_driver); 151 151 + 152 152 + return rc; 153 153 + } 154 154 + 155 155 + static __exit void dax_hmem_exit(void) 156 156 + { 157 157 + platform_driver_unregister(&dax_hmem_driver); 158 158 + platform_driver_unregister(&dax_hmem_platform_driver); 159 159 + } 160 160 + 161 161 + module_init(dax_hmem_init); 162 162 + module_exit(dax_hmem_exit); 163 163 + 164 164 + /* Allow for CXL to define its own dax regions */ 165 165 + #if IS_ENABLED(CONFIG_CXL_REGION) 166 166 + #if IS_MODULE(CONFIG_CXL_ACPI) 167 167 + MODULE_SOFTDEP("pre: cxl_acpi"); 168 168 + #endif 169 169 + #endif 65 170 66 171 MODULE_ALIAS("platform:hmem*"); 172 172 + MODULE_ALIAS("platform:hmem_platform*"); 67 173 MODULE_LICENSE("GPL v2"); 68 174 MODULE_AUTHOR("Intel Corporation");

+3 -2

drivers/dax/kmem.c

reviewed

··· 146 146 if (rc) { 147 147 dev_warn(dev, "mapping%d: %#llx-%#llx memory add failed\n", 148 148 i, range.start, range.end); 149 149 - release_resource(res); 149 149 + remove_resource(res); 150 150 kfree(res); 151 151 data->res[i] = NULL; 152 152 if (mapped) ··· 195 195 196 196 rc = remove_memory(range.start, range_len(&range)); 197 197 if (rc == 0) { 198 198 - release_resource(data->res[i]); 198 198 + remove_resource(data->res[i]); 199 199 kfree(data->res[i]); 200 200 data->res[i] = NULL; 201 201 success++; ··· 239 239 static struct dax_device_driver device_dax_kmem_driver = { 240 240 .probe = dev_dax_kmem_probe, 241 241 .remove = dev_dax_kmem_remove, 242 242 + .type = DAXDRV_KMEM_TYPE, 242 243 }; 243 244 244 245 static int __init dax_kmem_init(void)

+16 -3

drivers/nvdimm/bus.c

reviewed

··· 508 508 put_device(dev); 509 509 } 510 510 511 511 - void nd_device_register(struct device *dev) 511 511 + static void __nd_device_register(struct device *dev, bool sync) 512 512 { 513 513 if (!dev) 514 514 return; ··· 531 531 } 532 532 get_device(dev); 533 533 534 534 - async_schedule_dev_domain(nd_async_device_register, dev, 535 535 - &nd_async_domain); 534 534 + if (sync) 535 535 + nd_async_device_register(dev, 0); 536 536 + else 537 537 + async_schedule_dev_domain(nd_async_device_register, dev, 538 538 + &nd_async_domain); 539 539 + } 540 540 + 541 541 + void nd_device_register(struct device *dev) 542 542 + { 543 543 + __nd_device_register(dev, false); 536 544 } 537 545 EXPORT_SYMBOL(nd_device_register); 546 546 + 547 547 + void nd_device_register_sync(struct device *dev) 548 548 + { 549 549 + __nd_device_register(dev, true); 550 550 + } 538 551 539 552 void nd_device_unregister(struct device *dev, enum nd_async_mode mode) 540 553 {

+4 -1

drivers/nvdimm/dimm_devs.c

reviewed

··· 624 624 nvdimm->sec.ext_flags = nvdimm_security_flags(nvdimm, NVDIMM_MASTER); 625 625 device_initialize(dev); 626 626 lockdep_set_class(&dev->mutex, &nvdimm_key); 627 627 - nd_device_register(dev); 627 627 + if (test_bit(NDD_REGISTER_SYNC, &flags)) 628 628 + nd_device_register_sync(dev); 629 629 + else 630 630 + nd_device_register(dev); 628 631 629 632 return nvdimm; 630 633 }

+1

drivers/nvdimm/nd-core.h

reviewed

··· 107 107 void nvdimm_bus_destroy_ndctl(struct nvdimm_bus *nvdimm_bus); 108 108 void nd_synchronize(void); 109 109 void nd_device_register(struct device *dev); 110 110 + void nd_device_register_sync(struct device *dev); 110 111 struct nd_label_id; 111 112 char *nd_label_gen_id(struct nd_label_id *label_id, const uuid_t *uuid, 112 113 u32 flags);

+1

drivers/pci/probe.c

reviewed

··· 596 596 bridge->native_ltr = 1; 597 597 bridge->native_dpc = 1; 598 598 bridge->domain_nr = PCI_DOMAIN_NR_NOT_SET; 599 599 + bridge->native_cxl_error = 1; 599 600 600 601 device_initialize(&bridge->dev); 601 602 }

+5 -2

include/linux/dax.h

reviewed

··· 262 262 } 263 263 264 264 #ifdef CONFIG_DEV_DAX_HMEM_DEVICES 265 265 - void hmem_register_device(int target_nid, struct resource *r); 265 265 + void hmem_register_resource(int target_nid, struct resource *r); 266 266 #else 267 267 - static inline void hmem_register_device(int target_nid, struct resource *r) 267 267 + static inline void hmem_register_resource(int target_nid, struct resource *r) 268 268 { 269 269 } 270 270 #endif 271 271 272 272 + typedef int (*walk_hmem_fn)(struct device *dev, int target_nid, 273 273 + const struct resource *res); 274 274 + int walk_hmem_resources(struct device *dev, walk_hmem_fn fn); 272 275 #endif

+3

include/linux/libnvdimm.h

reviewed

··· 41 41 */ 42 42 NDD_INCOHERENT = 7, 43 43 44 44 + /* dimm provider wants synchronous registration by __nvdimm_create() */ 45 45 + NDD_REGISTER_SYNC = 8, 46 46 + 44 47 /* need to set a limit somewhere, but yes, this is likely overkill */ 45 48 ND_IOCTL_MAX_BUFLEN = SZ_4M, 46 49 ND_CMD_MAX_ELEM = 5,

+2

include/linux/memregion.h

reviewed

··· 3 3 #define _MEMREGION_H_ 4 4 #include <linux/types.h> 5 5 #include <linux/errno.h> 6 6 + #include <linux/range.h> 6 7 #include <linux/bug.h> 7 8 8 9 struct memregion_info { 9 10 int target_node; 11 11 + struct range range; 10 12 }; 11 13 12 14 #ifdef CONFIG_MEMREGION

+1

include/linux/pci.h

reviewed

··· 579 579 unsigned int native_pme:1; /* OS may use PCIe PME */ 580 580 unsigned int native_ltr:1; /* OS may use PCIe LTR */ 581 581 unsigned int native_dpc:1; /* OS may use PCIe DPC */ 582 582 + unsigned int native_cxl_error:1; /* OS may use CXL RAS/Events */ 582 583 unsigned int preserve_config:1; /* Preserve FW resource setup */ 583 584 unsigned int size_windows:1; /* Enable root bus sizing */ 584 585 unsigned int msi_domain:1; /* Bridge wants MSI domain */

+5

include/linux/range.h

reviewed

··· 13 13 return range->end - range->start + 1; 14 14 } 15 15 16 16 + static inline bool range_contains(struct range *r1, struct range *r2) 17 17 + { 18 18 + return r1->start <= r2->start && r1->end >= r2->end; 19 19 + } 20 20 + 16 21 int add_range(struct range *range, int az, int nr_range, 17 22 u64 start, u64 end); 18 23

-112

include/trace/events/cxl.h

reviewed

··· 1 1 - /* SPDX-License-Identifier: GPL-2.0 */ 2 2 - #undef TRACE_SYSTEM 3 3 - #define TRACE_SYSTEM cxl 4 4 - 5 5 - #if !defined(_CXL_EVENTS_H) || defined(TRACE_HEADER_MULTI_READ) 6 6 - #define _CXL_EVENTS_H 7 7 - 8 8 - #include <linux/tracepoint.h> 9 9 - 10 10 - #define CXL_HEADERLOG_SIZE SZ_512 11 11 - #define CXL_HEADERLOG_SIZE_U32 SZ_512 / sizeof(u32) 12 12 - 13 13 - #define CXL_RAS_UC_CACHE_DATA_PARITY BIT(0) 14 14 - #define CXL_RAS_UC_CACHE_ADDR_PARITY BIT(1) 15 15 - #define CXL_RAS_UC_CACHE_BE_PARITY BIT(2) 16 16 - #define CXL_RAS_UC_CACHE_DATA_ECC BIT(3) 17 17 - #define CXL_RAS_UC_MEM_DATA_PARITY BIT(4) 18 18 - #define CXL_RAS_UC_MEM_ADDR_PARITY BIT(5) 19 19 - #define CXL_RAS_UC_MEM_BE_PARITY BIT(6) 20 20 - #define CXL_RAS_UC_MEM_DATA_ECC BIT(7) 21 21 - #define CXL_RAS_UC_REINIT_THRESH BIT(8) 22 22 - #define CXL_RAS_UC_RSVD_ENCODE BIT(9) 23 23 - #define CXL_RAS_UC_POISON BIT(10) 24 24 - #define CXL_RAS_UC_RECV_OVERFLOW BIT(11) 25 25 - #define CXL_RAS_UC_INTERNAL_ERR BIT(14) 26 26 - #define CXL_RAS_UC_IDE_TX_ERR BIT(15) 27 27 - #define CXL_RAS_UC_IDE_RX_ERR BIT(16) 28 28 - 29 29 - #define show_uc_errs(status) __print_flags(status, " | ", \ 30 30 - { CXL_RAS_UC_CACHE_DATA_PARITY, "Cache Data Parity Error" }, \ 31 31 - { CXL_RAS_UC_CACHE_ADDR_PARITY, "Cache Address Parity Error" }, \ 32 32 - { CXL_RAS_UC_CACHE_BE_PARITY, "Cache Byte Enable Parity Error" }, \ 33 33 - { CXL_RAS_UC_CACHE_DATA_ECC, "Cache Data ECC Error" }, \ 34 34 - { CXL_RAS_UC_MEM_DATA_PARITY, "Memory Data Parity Error" }, \ 35 35 - { CXL_RAS_UC_MEM_ADDR_PARITY, "Memory Address Parity Error" }, \ 36 36 - { CXL_RAS_UC_MEM_BE_PARITY, "Memory Byte Enable Parity Error" }, \ 37 37 - { CXL_RAS_UC_MEM_DATA_ECC, "Memory Data ECC Error" }, \ 38 38 - { CXL_RAS_UC_REINIT_THRESH, "REINIT Threshold Hit" }, \ 39 39 - { CXL_RAS_UC_RSVD_ENCODE, "Received Unrecognized Encoding" }, \ 40 40 - { CXL_RAS_UC_POISON, "Received Poison From Peer" }, \ 41 41 - { CXL_RAS_UC_RECV_OVERFLOW, "Receiver Overflow" }, \ 42 42 - { CXL_RAS_UC_INTERNAL_ERR, "Component Specific Error" }, \ 43 43 - { CXL_RAS_UC_IDE_TX_ERR, "IDE Tx Error" }, \ 44 44 - { CXL_RAS_UC_IDE_RX_ERR, "IDE Rx Error" } \ 45 45 - ) 46 46 - 47 47 - TRACE_EVENT(cxl_aer_uncorrectable_error, 48 48 - TP_PROTO(const struct device *dev, u32 status, u32 fe, u32 *hl), 49 49 - TP_ARGS(dev, status, fe, hl), 50 50 - TP_STRUCT__entry( 51 51 - __string(dev_name, dev_name(dev)) 52 52 - __field(u32, status) 53 53 - __field(u32, first_error) 54 54 - __array(u32, header_log, CXL_HEADERLOG_SIZE_U32) 55 55 - ), 56 56 - TP_fast_assign( 57 57 - __assign_str(dev_name, dev_name(dev)); 58 58 - __entry->status = status; 59 59 - __entry->first_error = fe; 60 60 - /* 61 61 - * Embed the 512B headerlog data for user app retrieval and 62 62 - * parsing, but no need to print this in the trace buffer. 63 63 - */ 64 64 - memcpy(__entry->header_log, hl, CXL_HEADERLOG_SIZE); 65 65 - ), 66 66 - TP_printk("%s: status: '%s' first_error: '%s'", 67 67 - __get_str(dev_name), 68 68 - show_uc_errs(__entry->status), 69 69 - show_uc_errs(__entry->first_error) 70 70 - ) 71 71 - ); 72 72 - 73 73 - #define CXL_RAS_CE_CACHE_DATA_ECC BIT(0) 74 74 - #define CXL_RAS_CE_MEM_DATA_ECC BIT(1) 75 75 - #define CXL_RAS_CE_CRC_THRESH BIT(2) 76 76 - #define CLX_RAS_CE_RETRY_THRESH BIT(3) 77 77 - #define CXL_RAS_CE_CACHE_POISON BIT(4) 78 78 - #define CXL_RAS_CE_MEM_POISON BIT(5) 79 79 - #define CXL_RAS_CE_PHYS_LAYER_ERR BIT(6) 80 80 - 81 81 - #define show_ce_errs(status) __print_flags(status, " | ", \ 82 82 - { CXL_RAS_CE_CACHE_DATA_ECC, "Cache Data ECC Error" }, \ 83 83 - { CXL_RAS_CE_MEM_DATA_ECC, "Memory Data ECC Error" }, \ 84 84 - { CXL_RAS_CE_CRC_THRESH, "CRC Threshold Hit" }, \ 85 85 - { CLX_RAS_CE_RETRY_THRESH, "Retry Threshold" }, \ 86 86 - { CXL_RAS_CE_CACHE_POISON, "Received Cache Poison From Peer" }, \ 87 87 - { CXL_RAS_CE_MEM_POISON, "Received Memory Poison From Peer" }, \ 88 88 - { CXL_RAS_CE_PHYS_LAYER_ERR, "Received Error From Physical Layer" } \ 89 89 - ) 90 90 - 91 91 - TRACE_EVENT(cxl_aer_correctable_error, 92 92 - TP_PROTO(const struct device *dev, u32 status), 93 93 - TP_ARGS(dev, status), 94 94 - TP_STRUCT__entry( 95 95 - __string(dev_name, dev_name(dev)) 96 96 - __field(u32, status) 97 97 - ), 98 98 - TP_fast_assign( 99 99 - __assign_str(dev_name, dev_name(dev)); 100 100 - __entry->status = status; 101 101 - ), 102 102 - TP_printk("%s: status: '%s'", 103 103 - __get_str(dev_name), show_ce_errs(__entry->status) 104 104 - ) 105 105 - ); 106 106 - 107 107 - #endif /* _CXL_EVENTS_H */ 108 108 - 109 109 - /* This part must be outside protection */ 110 110 - #undef TRACE_INCLUDE_FILE 111 111 - #define TRACE_INCLUDE_FILE cxl 112 112 - #include <trace/define_trace.h>

+25 -5

include/uapi/linux/cxl_mem.h

reviewed

··· 11 11 /** 12 12 * DOC: UAPI 13 13 * 14 14 - * Not all of all commands that the driver supports are always available for use 15 15 - * by userspace. Userspace must check the results from the QUERY command in 16 16 - * order to determine the live set of commands. 14 14 + * Not all of the commands that the driver supports are available for use by 15 15 + * userspace at all times. Userspace can check the result of the QUERY command 16 16 + * to determine the live set of commands. Alternatively, it can issue the 17 17 + * command and check for failure. 17 18 */ 18 19 19 20 #define CXL_MEM_QUERY_COMMANDS _IOR(0xCE, 1, struct cxl_mem_query_commands) 20 21 #define CXL_MEM_SEND_COMMAND _IOWR(0xCE, 2, struct cxl_send_command) 21 22 23 23 + /* 24 24 + * NOTE: New defines must be added to the end of the list to preserve 25 25 + * compatibility because this enum is exported to user space. 26 26 + */ 22 27 #define CXL_CMDS \ 23 28 ___C(INVALID, "Invalid Command"), \ 24 29 ___C(IDENTIFY, "Identify Command"), \ ··· 73 68 * struct cxl_command_info - Command information returned from a query. 74 69 * @id: ID number for the command. 75 70 * @flags: Flags that specify command behavior. 71 71 + * 72 72 + * CXL_MEM_COMMAND_FLAG_USER_ENABLED 73 73 + * 74 74 + * The given command id is supported by the driver and is supported by 75 75 + * a related opcode on the device. 76 76 + * 77 77 + * CXL_MEM_COMMAND_FLAG_EXCLUSIVE 78 78 + * 79 79 + * Requests with the given command id will terminate with EBUSY as the 80 80 + * kernel actively owns management of the given resource. For example, 81 81 + * the label-storage-area can not be written while the kernel is 82 82 + * actively managing that space. 83 83 + * 76 84 * @size_in: Expected input size, or ~0 if variable length. 77 85 * @size_out: Expected output size, or ~0 if variable length. 78 86 * ··· 95 77 * bytes of output. 96 78 * 97 79 * - @id = 10 98 98 - * - @flags = 0 80 80 + * - @flags = CXL_MEM_COMMAND_FLAG_ENABLED 99 81 * - @size_in = ~0 100 82 * - @size_out = 0 101 83 * ··· 105 87 __u32 id; 106 88 107 89 __u32 flags; 108 108 - #define CXL_MEM_COMMAND_FLAG_MASK GENMASK(0, 0) 90 90 + #define CXL_MEM_COMMAND_FLAG_MASK GENMASK(1, 0) 91 91 + #define CXL_MEM_COMMAND_FLAG_ENABLED BIT(0) 92 92 + #define CXL_MEM_COMMAND_FLAG_EXCLUSIVE BIT(1) 109 93 110 94 __u32 size_in; 111 95 __u32 size_out;

+1

include/uapi/linux/pci_regs.h

reviewed

··· 693 693 #define PCI_EXP_LNKCTL2_TX_MARGIN 0x0380 /* Transmit Margin */ 694 694 #define PCI_EXP_LNKCTL2_HASD 0x0020 /* HW Autonomous Speed Disable */ 695 695 #define PCI_EXP_LNKSTA2 0x32 /* Link Status 2 */ 696 696 + #define PCI_EXP_LNKSTA2_FLIT 0x0400 /* Flit Mode Status */ 696 697 #define PCI_CAP_EXP_ENDPOINT_SIZEOF_V2 0x32 /* end of v2 EPs w/ link */ 697 698 #define PCI_EXP_SLTCAP2 0x34 /* Slot Capabilities 2 */ 698 699 #define PCI_EXP_SLTCAP2_IBPD 0x00000001 /* In-band PD Disable Supported */

-14

kernel/resource.c

reviewed

··· 1343 1343 continue; 1344 1344 } 1345 1345 1346 1346 - /* 1347 1347 - * All memory regions added from memory-hotplug path have the 1348 1348 - * flag IORESOURCE_SYSTEM_RAM. If the resource does not have 1349 1349 - * this flag, we know that we are dealing with a resource coming 1350 1350 - * from HMM/devm. HMM/devm use another mechanism to add/release 1351 1351 - * a resource. This goes via devm_request_mem_region and 1352 1352 - * devm_release_mem_region. 1353 1353 - * HMM/devm take care to release their resources when they want, 1354 1354 - * so if we are dealing with them, let us just back off here. 1355 1355 - */ 1356 1356 - if (!(res->flags & IORESOURCE_SYSRAM)) { 1357 1357 - break; 1358 1358 - } 1359 1359 - 1360 1346 if (!(res->flags & IORESOURCE_MEM)) 1361 1347 break; 1362 1348

+3 -3

lib/stackinit_kunit.c

reviewed

··· 31 31 static void *fill_start, *target_start; 32 32 static size_t fill_size, target_size; 33 33 34 34 - static bool range_contains(char *haystack_start, size_t haystack_size, 35 35 - char *needle_start, size_t needle_size) 34 34 + static bool stackinit_range_contains(char *haystack_start, size_t haystack_size, 35 35 + char *needle_start, size_t needle_size) 36 36 { 37 37 if (needle_start >= haystack_start && 38 38 needle_start + needle_size <= haystack_start + haystack_size) ··· 175 175 \ 176 176 /* Validate that compiler lined up fill and target. */ \ 177 177 KUNIT_ASSERT_TRUE_MSG(test, \ 178 178 - range_contains(fill_start, fill_size, \ 178 178 + stackinit_range_contains(fill_start, fill_size, \ 179 179 target_start, target_size), \ 180 180 "stack fill missed target!? " \ 181 181 "(fill %zu wide, target offset by %d)\n", \

+9

tools/testing/cxl/Kbuild

reviewed

··· 10 10 ldflags-y += --wrap=devm_cxl_enumerate_decoders 11 11 ldflags-y += --wrap=cxl_await_media_ready 12 12 ldflags-y += --wrap=cxl_hdm_decode_init 13 13 + ldflags-y += --wrap=cxl_dvsec_rr_decode 13 14 ldflags-y += --wrap=cxl_rcrb_to_component 14 15 15 16 DRIVERS := ../../../drivers ··· 18 17 CXL_CORE_SRC := $(DRIVERS)/cxl/core 19 18 ccflags-y := -I$(srctree)/drivers/cxl/ 20 19 ccflags-y += -D__mock=__weak 20 20 + ccflags-y += -DTRACE_INCLUDE_PATH=$(CXL_CORE_SRC) -I$(srctree)/drivers/cxl/core/ 21 21 22 22 obj-m += cxl_acpi.o 23 23 24 24 cxl_acpi-y := $(CXL_SRC)/acpi.o 25 25 cxl_acpi-y += mock_acpi.o 26 26 cxl_acpi-y += config_check.o 27 27 + cxl_acpi-y += cxl_acpi_test.o 27 28 28 29 obj-m += cxl_pmem.o 29 30 30 31 cxl_pmem-y := $(CXL_SRC)/pmem.o 31 32 cxl_pmem-y += $(CXL_SRC)/security.o 32 33 cxl_pmem-y += config_check.o 34 34 + cxl_pmem-y += cxl_pmem_test.o 33 35 34 36 obj-m += cxl_port.o 35 37 36 38 cxl_port-y := $(CXL_SRC)/port.o 37 39 cxl_port-y += config_check.o 40 40 + cxl_port-y += cxl_port_test.o 41 41 + 38 42 39 43 obj-m += cxl_mem.o 40 44 41 45 cxl_mem-y := $(CXL_SRC)/mem.o 42 46 cxl_mem-y += config_check.o 47 47 + cxl_mem-y += cxl_mem_test.o 43 48 44 49 obj-m += cxl_core.o 45 50 ··· 56 49 cxl_core-y += $(CXL_CORE_SRC)/mbox.o 57 50 cxl_core-y += $(CXL_CORE_SRC)/pci.o 58 51 cxl_core-y += $(CXL_CORE_SRC)/hdm.o 52 52 + cxl_core-$(CONFIG_TRACING) += $(CXL_CORE_SRC)/trace.o 59 53 cxl_core-$(CONFIG_CXL_REGION) += $(CXL_CORE_SRC)/region.o 60 54 cxl_core-y += config_check.o 55 55 + cxl_core-y += cxl_core_test.o 61 56 62 57 obj-m += test/

+1

tools/testing/cxl/config_check.c

reviewed

··· 7 7 * These kconfig symbols must be set to "m" for cxl_test to load 8 8 * and operate. 9 9 */ 10 10 + BUILD_BUG_ON(!IS_ENABLED(CONFIG_64BIT)); 10 11 BUILD_BUG_ON(!IS_MODULE(CONFIG_CXL_BUS)); 11 12 BUILD_BUG_ON(!IS_MODULE(CONFIG_CXL_ACPI)); 12 13 BUILD_BUG_ON(!IS_MODULE(CONFIG_CXL_PMEM));

+6

tools/testing/cxl/cxl_acpi_test.c

reviewed

··· 1 1 + // SPDX-License-Identifier: GPL-2.0 2 2 + /* Copyright(c) 2022 Intel Corporation. All rights reserved. */ 3 3 + 4 4 + #include "watermark.h" 5 5 + 6 6 + cxl_test_watermark(cxl_acpi);

+6

tools/testing/cxl/cxl_core_test.c

reviewed

··· 1 1 + // SPDX-License-Identifier: GPL-2.0 2 2 + /* Copyright(c) 2022 Intel Corporation. All rights reserved. */ 3 3 + 4 4 + #include "watermark.h" 5 5 + 6 6 + cxl_test_watermark(cxl_core);

+6

tools/testing/cxl/cxl_mem_test.c

reviewed

··· 1 1 + // SPDX-License-Identifier: GPL-2.0 2 2 + /* Copyright(c) 2022 Intel Corporation. All rights reserved. */ 3 3 + 4 4 + #include "watermark.h" 5 5 + 6 6 + cxl_test_watermark(cxl_mem);

+6

tools/testing/cxl/cxl_pmem_test.c

reviewed

··· 1 1 + // SPDX-License-Identifier: GPL-2.0 2 2 + /* Copyright(c) 2022 Intel Corporation. All rights reserved. */ 3 3 + 4 4 + #include "watermark.h" 5 5 + 6 6 + cxl_test_watermark(cxl_pmem);

+6

tools/testing/cxl/cxl_port_test.c

reviewed

··· 1 1 + // SPDX-License-Identifier: GPL-2.0 2 2 + /* Copyright(c) 2022 Intel Corporation. All rights reserved. */ 3 3 + 4 4 + #include "watermark.h" 5 5 + 6 6 + cxl_test_watermark(cxl_port);

+1 -1

tools/testing/cxl/test/Kbuild

reviewed

··· 1 1 # SPDX-License-Identifier: GPL-2.0 2 2 - ccflags-y := -I$(srctree)/drivers/cxl/ 2 2 + ccflags-y := -I$(srctree)/drivers/cxl/ -I$(srctree)/drivers/cxl/core 3 3 4 4 obj-m += cxl_test.o 5 5 obj-m += cxl_mock.o

+150 -15

tools/testing/cxl/test/cxl.c

reviewed

··· 9 9 #include <linux/pci.h> 10 10 #include <linux/mm.h> 11 11 #include <cxlmem.h> 12 12 + 13 13 + #include "../watermark.h" 12 14 #include "mock.h" 13 15 14 16 static int interleave_arithmetic; ··· 620 618 return &mock_pci_root[host_bridge_index(adev)]; 621 619 } 622 620 623 623 - static struct cxl_hdm *mock_cxl_setup_hdm(struct cxl_port *port) 621 621 + static struct cxl_hdm *mock_cxl_setup_hdm(struct cxl_port *port, 622 622 + struct cxl_endpoint_dvsec_info *info) 624 623 { 625 624 struct cxl_hdm *cxlhdm = devm_kzalloc(&port->dev, sizeof(*cxlhdm), GFP_KERNEL); 626 625 ··· 704 701 return 0; 705 702 } 706 703 707 707 - static int mock_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm) 704 704 + static void default_mock_decoder(struct cxl_decoder *cxld) 705 705 + { 706 706 + cxld->hpa_range = (struct range){ 707 707 + .start = 0, 708 708 + .end = -1, 709 709 + }; 710 710 + 711 711 + cxld->interleave_ways = 1; 712 712 + cxld->interleave_granularity = 256; 713 713 + cxld->target_type = CXL_DECODER_EXPANDER; 714 714 + cxld->commit = mock_decoder_commit; 715 715 + cxld->reset = mock_decoder_reset; 716 716 + } 717 717 + 718 718 + static int first_decoder(struct device *dev, void *data) 719 719 + { 720 720 + struct cxl_decoder *cxld; 721 721 + 722 722 + if (!is_switch_decoder(dev)) 723 723 + return 0; 724 724 + cxld = to_cxl_decoder(dev); 725 725 + if (cxld->id == 0) 726 726 + return 1; 727 727 + return 0; 728 728 + } 729 729 + 730 730 + static void mock_init_hdm_decoder(struct cxl_decoder *cxld) 731 731 + { 732 732 + struct acpi_cedt_cfmws *window = mock_cfmws[0]; 733 733 + struct platform_device *pdev = NULL; 734 734 + struct cxl_endpoint_decoder *cxled; 735 735 + struct cxl_switch_decoder *cxlsd; 736 736 + struct cxl_port *port, *iter; 737 737 + const int size = SZ_512M; 738 738 + struct cxl_memdev *cxlmd; 739 739 + struct cxl_dport *dport; 740 740 + struct device *dev; 741 741 + bool hb0 = false; 742 742 + u64 base; 743 743 + int i; 744 744 + 745 745 + if (is_endpoint_decoder(&cxld->dev)) { 746 746 + cxled = to_cxl_endpoint_decoder(&cxld->dev); 747 747 + cxlmd = cxled_to_memdev(cxled); 748 748 + WARN_ON(!dev_is_platform(cxlmd->dev.parent)); 749 749 + pdev = to_platform_device(cxlmd->dev.parent); 750 750 + 751 751 + /* check is endpoint is attach to host-bridge0 */ 752 752 + port = cxled_to_port(cxled); 753 753 + do { 754 754 + if (port->uport == &cxl_host_bridge[0]->dev) { 755 755 + hb0 = true; 756 756 + break; 757 757 + } 758 758 + if (is_cxl_port(port->dev.parent)) 759 759 + port = to_cxl_port(port->dev.parent); 760 760 + else 761 761 + port = NULL; 762 762 + } while (port); 763 763 + port = cxled_to_port(cxled); 764 764 + } 765 765 + 766 766 + /* 767 767 + * The first decoder on the first 2 devices on the first switch 768 768 + * attached to host-bridge0 mock a fake / static RAM region. All 769 769 + * other decoders are default disabled. Given the round robin 770 770 + * assignment those devices are named cxl_mem.0, and cxl_mem.4. 771 771 + * 772 772 + * See 'cxl list -BMPu -m cxl_mem.0,cxl_mem.4' 773 773 + */ 774 774 + if (!hb0 || pdev->id % 4 || pdev->id > 4 || cxld->id > 0) { 775 775 + default_mock_decoder(cxld); 776 776 + return; 777 777 + } 778 778 + 779 779 + base = window->base_hpa; 780 780 + cxld->hpa_range = (struct range) { 781 781 + .start = base, 782 782 + .end = base + size - 1, 783 783 + }; 784 784 + 785 785 + cxld->interleave_ways = 2; 786 786 + eig_to_granularity(window->granularity, &cxld->interleave_granularity); 787 787 + cxld->target_type = CXL_DECODER_EXPANDER; 788 788 + cxld->flags = CXL_DECODER_F_ENABLE; 789 789 + cxled->state = CXL_DECODER_STATE_AUTO; 790 790 + port->commit_end = cxld->id; 791 791 + devm_cxl_dpa_reserve(cxled, 0, size / cxld->interleave_ways, 0); 792 792 + cxld->commit = mock_decoder_commit; 793 793 + cxld->reset = mock_decoder_reset; 794 794 + 795 795 + /* 796 796 + * Now that endpoint decoder is set up, walk up the hierarchy 797 797 + * and setup the switch and root port decoders targeting @cxlmd. 798 798 + */ 799 799 + iter = port; 800 800 + for (i = 0; i < 2; i++) { 801 801 + dport = iter->parent_dport; 802 802 + iter = dport->port; 803 803 + dev = device_find_child(&iter->dev, NULL, first_decoder); 804 804 + /* 805 805 + * Ancestor ports are guaranteed to be enumerated before 806 806 + * @port, and all ports have at least one decoder. 807 807 + */ 808 808 + if (WARN_ON(!dev)) 809 809 + continue; 810 810 + cxlsd = to_cxl_switch_decoder(dev); 811 811 + if (i == 0) { 812 812 + /* put cxl_mem.4 second in the decode order */ 813 813 + if (pdev->id == 4) 814 814 + cxlsd->target[1] = dport; 815 815 + else 816 816 + cxlsd->target[0] = dport; 817 817 + } else 818 818 + cxlsd->target[0] = dport; 819 819 + cxld = &cxlsd->cxld; 820 820 + cxld->target_type = CXL_DECODER_EXPANDER; 821 821 + cxld->flags = CXL_DECODER_F_ENABLE; 822 822 + iter->commit_end = 0; 823 823 + /* 824 824 + * Switch targets 2 endpoints, while host bridge targets 825 825 + * one root port 826 826 + */ 827 827 + if (i == 0) 828 828 + cxld->interleave_ways = 2; 829 829 + else 830 830 + cxld->interleave_ways = 1; 831 831 + cxld->interleave_granularity = 256; 832 832 + cxld->hpa_range = (struct range) { 833 833 + .start = base, 834 834 + .end = base + size - 1, 835 835 + }; 836 836 + put_device(dev); 837 837 + } 838 838 + } 839 839 + 840 840 + static int mock_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm, 841 841 + struct cxl_endpoint_dvsec_info *info) 708 842 { 709 843 struct cxl_port *port = cxlhdm->port; 710 844 struct cxl_port *parent_port = to_cxl_port(port->dev.parent); ··· 886 746 cxld = &cxled->cxld; 887 747 } 888 748 889 889 - cxld->hpa_range = (struct range) { 890 890 - .start = 0, 891 891 - .end = -1, 892 892 - }; 893 893 - 894 894 - cxld->interleave_ways = min_not_zero(target_count, 1); 895 895 - cxld->interleave_granularity = SZ_4K; 896 896 - cxld->target_type = CXL_DECODER_EXPANDER; 897 897 - cxld->commit = mock_decoder_commit; 898 898 - cxld->reset = mock_decoder_reset; 749 749 + mock_init_hdm_decoder(cxld); 899 750 900 751 if (target_count) { 901 752 rc = device_for_each_child(port->uport, &ctx, ··· 1250 1119 { 1251 1120 int rc, i; 1252 1121 1122 1122 + cxl_acpi_test(); 1123 1123 + cxl_core_test(); 1124 1124 + cxl_mem_test(); 1125 1125 + cxl_pmem_test(); 1126 1126 + cxl_port_test(); 1127 1127 + 1253 1128 register_cxl_mock_ops(&cxl_mock_ops); 1254 1129 1255 1130 cxl_mock_pool = gen_pool_create(ilog2(SZ_2M), NUMA_NO_NODE); ··· 1272 1135 if (interleave_arithmetic == 1) { 1273 1136 cfmws_start = CFMWS_XOR_ARRAY_START; 1274 1137 cfmws_end = CFMWS_XOR_ARRAY_END; 1275 1275 - dev_dbg(NULL, "cxl_test loading xor math option\n"); 1276 1138 } else { 1277 1139 cfmws_start = CFMWS_MOD_ARRAY_START; 1278 1140 cfmws_end = CFMWS_MOD_ARRAY_END; 1279 1279 - dev_dbg(NULL, "cxl_test loading modulo math option\n"); 1280 1141 } 1281 1142 1282 1143 rc = populate_cedt(); ··· 1461 1326 unregister_cxl_mock_ops(&cxl_mock_ops); 1462 1327 } 1463 1328 1464 1464 - module_param(interleave_arithmetic, int, 0000); 1329 1329 + module_param(interleave_arithmetic, int, 0444); 1465 1330 MODULE_PARM_DESC(interleave_arithmetic, "Modulo:0, XOR:1"); 1466 1331 module_init(cxl_test_init); 1467 1332 module_exit(cxl_test_exit);

+353 -1

tools/testing/cxl/test/mem.c

reviewed

··· 9 9 #include <linux/bits.h> 10 10 #include <cxlmem.h> 11 11 12 12 + #include "trace.h" 13 13 + 12 14 #define LSA_SIZE SZ_128K 13 15 #define DEV_SIZE SZ_2G 14 16 #define EFFECT(x) (1U << x) ··· 69 67 70 68 #define PASS_TRY_LIMIT 3 71 69 70 70 + #define CXL_TEST_EVENT_CNT_MAX 15 71 71 + 72 72 + /* Set a number of events to return at a time for simulation. */ 73 73 + #define CXL_TEST_EVENT_CNT 3 74 74 + 75 75 + struct mock_event_log { 76 76 + u16 clear_idx; 77 77 + u16 cur_idx; 78 78 + u16 nr_events; 79 79 + u16 nr_overflow; 80 80 + u16 overflow_reset; 81 81 + struct cxl_event_record_raw *events[CXL_TEST_EVENT_CNT_MAX]; 82 82 + }; 83 83 + 84 84 + struct mock_event_store { 85 85 + struct cxl_dev_state *cxlds; 86 86 + struct mock_event_log mock_logs[CXL_EVENT_TYPE_MAX]; 87 87 + u32 ev_status; 88 88 + }; 89 89 + 72 90 struct cxl_mockmem_data { 73 91 void *lsa; 74 92 u32 security_state; ··· 96 74 u8 master_pass[NVDIMM_PASSPHRASE_LEN]; 97 75 int user_limit; 98 76 int master_limit; 99 99 - 77 77 + struct mock_event_store mes; 78 78 + u8 event_buf[SZ_4K]; 100 79 }; 80 80 + 81 81 + static struct mock_event_log *event_find_log(struct device *dev, int log_type) 82 82 + { 83 83 + struct cxl_mockmem_data *mdata = dev_get_drvdata(dev); 84 84 + 85 85 + if (log_type >= CXL_EVENT_TYPE_MAX) 86 86 + return NULL; 87 87 + return &mdata->mes.mock_logs[log_type]; 88 88 + } 89 89 + 90 90 + static struct cxl_event_record_raw *event_get_current(struct mock_event_log *log) 91 91 + { 92 92 + return log->events[log->cur_idx]; 93 93 + } 94 94 + 95 95 + static void event_reset_log(struct mock_event_log *log) 96 96 + { 97 97 + log->cur_idx = 0; 98 98 + log->clear_idx = 0; 99 99 + log->nr_overflow = log->overflow_reset; 100 100 + } 101 101 + 102 102 + /* Handle can never be 0 use 1 based indexing for handle */ 103 103 + static u16 event_get_clear_handle(struct mock_event_log *log) 104 104 + { 105 105 + return log->clear_idx + 1; 106 106 + } 107 107 + 108 108 + /* Handle can never be 0 use 1 based indexing for handle */ 109 109 + static __le16 event_get_cur_event_handle(struct mock_event_log *log) 110 110 + { 111 111 + u16 cur_handle = log->cur_idx + 1; 112 112 + 113 113 + return cpu_to_le16(cur_handle); 114 114 + } 115 115 + 116 116 + static bool event_log_empty(struct mock_event_log *log) 117 117 + { 118 118 + return log->cur_idx == log->nr_events; 119 119 + } 120 120 + 121 121 + static void mes_add_event(struct mock_event_store *mes, 122 122 + enum cxl_event_log_type log_type, 123 123 + struct cxl_event_record_raw *event) 124 124 + { 125 125 + struct mock_event_log *log; 126 126 + 127 127 + if (WARN_ON(log_type >= CXL_EVENT_TYPE_MAX)) 128 128 + return; 129 129 + 130 130 + log = &mes->mock_logs[log_type]; 131 131 + 132 132 + if ((log->nr_events + 1) > CXL_TEST_EVENT_CNT_MAX) { 133 133 + log->nr_overflow++; 134 134 + log->overflow_reset = log->nr_overflow; 135 135 + return; 136 136 + } 137 137 + 138 138 + log->events[log->nr_events] = event; 139 139 + log->nr_events++; 140 140 + } 141 141 + 142 142 + static int mock_get_event(struct cxl_dev_state *cxlds, 143 143 + struct cxl_mbox_cmd *cmd) 144 144 + { 145 145 + struct cxl_get_event_payload *pl; 146 146 + struct mock_event_log *log; 147 147 + u16 nr_overflow; 148 148 + u8 log_type; 149 149 + int i; 150 150 + 151 151 + if (cmd->size_in != sizeof(log_type)) 152 152 + return -EINVAL; 153 153 + 154 154 + if (cmd->size_out < struct_size(pl, records, CXL_TEST_EVENT_CNT)) 155 155 + return -EINVAL; 156 156 + 157 157 + log_type = *((u8 *)cmd->payload_in); 158 158 + if (log_type >= CXL_EVENT_TYPE_MAX) 159 159 + return -EINVAL; 160 160 + 161 161 + memset(cmd->payload_out, 0, cmd->size_out); 162 162 + 163 163 + log = event_find_log(cxlds->dev, log_type); 164 164 + if (!log || event_log_empty(log)) 165 165 + return 0; 166 166 + 167 167 + pl = cmd->payload_out; 168 168 + 169 169 + for (i = 0; i < CXL_TEST_EVENT_CNT && !event_log_empty(log); i++) { 170 170 + memcpy(&pl->records[i], event_get_current(log), 171 171 + sizeof(pl->records[i])); 172 172 + pl->records[i].hdr.handle = event_get_cur_event_handle(log); 173 173 + log->cur_idx++; 174 174 + } 175 175 + 176 176 + pl->record_count = cpu_to_le16(i); 177 177 + if (!event_log_empty(log)) 178 178 + pl->flags |= CXL_GET_EVENT_FLAG_MORE_RECORDS; 179 179 + 180 180 + if (log->nr_overflow) { 181 181 + u64 ns; 182 182 + 183 183 + pl->flags |= CXL_GET_EVENT_FLAG_OVERFLOW; 184 184 + pl->overflow_err_count = cpu_to_le16(nr_overflow); 185 185 + ns = ktime_get_real_ns(); 186 186 + ns -= 5000000000; /* 5s ago */ 187 187 + pl->first_overflow_timestamp = cpu_to_le64(ns); 188 188 + ns = ktime_get_real_ns(); 189 189 + ns -= 1000000000; /* 1s ago */ 190 190 + pl->last_overflow_timestamp = cpu_to_le64(ns); 191 191 + } 192 192 + 193 193 + return 0; 194 194 + } 195 195 + 196 196 + static int mock_clear_event(struct cxl_dev_state *cxlds, 197 197 + struct cxl_mbox_cmd *cmd) 198 198 + { 199 199 + struct cxl_mbox_clear_event_payload *pl = cmd->payload_in; 200 200 + struct mock_event_log *log; 201 201 + u8 log_type = pl->event_log; 202 202 + u16 handle; 203 203 + int nr; 204 204 + 205 205 + if (log_type >= CXL_EVENT_TYPE_MAX) 206 206 + return -EINVAL; 207 207 + 208 208 + log = event_find_log(cxlds->dev, log_type); 209 209 + if (!log) 210 210 + return 0; /* No mock data in this log */ 211 211 + 212 212 + /* 213 213 + * This check is technically not invalid per the specification AFAICS. 214 214 + * (The host could 'guess' handles and clear them in order). 215 215 + * However, this is not good behavior for the host so test it. 216 216 + */ 217 217 + if (log->clear_idx + pl->nr_recs > log->cur_idx) { 218 218 + dev_err(cxlds->dev, 219 219 + "Attempting to clear more events than returned!\n"); 220 220 + return -EINVAL; 221 221 + } 222 222 + 223 223 + /* Check handle order prior to clearing events */ 224 224 + for (nr = 0, handle = event_get_clear_handle(log); 225 225 + nr < pl->nr_recs; 226 226 + nr++, handle++) { 227 227 + if (handle != le16_to_cpu(pl->handles[nr])) { 228 228 + dev_err(cxlds->dev, "Clearing events out of order\n"); 229 229 + return -EINVAL; 230 230 + } 231 231 + } 232 232 + 233 233 + if (log->nr_overflow) 234 234 + log->nr_overflow = 0; 235 235 + 236 236 + /* Clear events */ 237 237 + log->clear_idx += pl->nr_recs; 238 238 + return 0; 239 239 + } 240 240 + 241 241 + static void cxl_mock_event_trigger(struct device *dev) 242 242 + { 243 243 + struct cxl_mockmem_data *mdata = dev_get_drvdata(dev); 244 244 + struct mock_event_store *mes = &mdata->mes; 245 245 + int i; 246 246 + 247 247 + for (i = CXL_EVENT_TYPE_INFO; i < CXL_EVENT_TYPE_MAX; i++) { 248 248 + struct mock_event_log *log; 249 249 + 250 250 + log = event_find_log(dev, i); 251 251 + if (log) 252 252 + event_reset_log(log); 253 253 + } 254 254 + 255 255 + cxl_mem_get_event_records(mes->cxlds, mes->ev_status); 256 256 + } 257 257 + 258 258 + struct cxl_event_record_raw maint_needed = { 259 259 + .hdr = { 260 260 + .id = UUID_INIT(0xBA5EBA11, 0xABCD, 0xEFEB, 261 261 + 0xa5, 0x5a, 0xa5, 0x5a, 0xa5, 0xa5, 0x5a, 0xa5), 262 262 + .length = sizeof(struct cxl_event_record_raw), 263 263 + .flags[0] = CXL_EVENT_RECORD_FLAG_MAINT_NEEDED, 264 264 + /* .handle = Set dynamically */ 265 265 + .related_handle = cpu_to_le16(0xa5b6), 266 266 + }, 267 267 + .data = { 0xDE, 0xAD, 0xBE, 0xEF }, 268 268 + }; 269 269 + 270 270 + struct cxl_event_record_raw hardware_replace = { 271 271 + .hdr = { 272 272 + .id = UUID_INIT(0xABCDEFEB, 0xBA11, 0xBA5E, 273 273 + 0xa5, 0x5a, 0xa5, 0x5a, 0xa5, 0xa5, 0x5a, 0xa5), 274 274 + .length = sizeof(struct cxl_event_record_raw), 275 275 + .flags[0] = CXL_EVENT_RECORD_FLAG_HW_REPLACE, 276 276 + /* .handle = Set dynamically */ 277 277 + .related_handle = cpu_to_le16(0xb6a5), 278 278 + }, 279 279 + .data = { 0xDE, 0xAD, 0xBE, 0xEF }, 280 280 + }; 281 281 + 282 282 + struct cxl_event_gen_media gen_media = { 283 283 + .hdr = { 284 284 + .id = UUID_INIT(0xfbcd0a77, 0xc260, 0x417f, 285 285 + 0x85, 0xa9, 0x08, 0x8b, 0x16, 0x21, 0xeb, 0xa6), 286 286 + .length = sizeof(struct cxl_event_gen_media), 287 287 + .flags[0] = CXL_EVENT_RECORD_FLAG_PERMANENT, 288 288 + /* .handle = Set dynamically */ 289 289 + .related_handle = cpu_to_le16(0), 290 290 + }, 291 291 + .phys_addr = cpu_to_le64(0x2000), 292 292 + .descriptor = CXL_GMER_EVT_DESC_UNCORECTABLE_EVENT, 293 293 + .type = CXL_GMER_MEM_EVT_TYPE_DATA_PATH_ERROR, 294 294 + .transaction_type = CXL_GMER_TRANS_HOST_WRITE, 295 295 + /* .validity_flags = <set below> */ 296 296 + .channel = 1, 297 297 + .rank = 30 298 298 + }; 299 299 + 300 300 + struct cxl_event_dram dram = { 301 301 + .hdr = { 302 302 + .id = UUID_INIT(0x601dcbb3, 0x9c06, 0x4eab, 303 303 + 0xb8, 0xaf, 0x4e, 0x9b, 0xfb, 0x5c, 0x96, 0x24), 304 304 + .length = sizeof(struct cxl_event_dram), 305 305 + .flags[0] = CXL_EVENT_RECORD_FLAG_PERF_DEGRADED, 306 306 + /* .handle = Set dynamically */ 307 307 + .related_handle = cpu_to_le16(0), 308 308 + }, 309 309 + .phys_addr = cpu_to_le64(0x8000), 310 310 + .descriptor = CXL_GMER_EVT_DESC_THRESHOLD_EVENT, 311 311 + .type = CXL_GMER_MEM_EVT_TYPE_INV_ADDR, 312 312 + .transaction_type = CXL_GMER_TRANS_INTERNAL_MEDIA_SCRUB, 313 313 + /* .validity_flags = <set below> */ 314 314 + .channel = 1, 315 315 + .bank_group = 5, 316 316 + .bank = 2, 317 317 + .column = {0xDE, 0xAD}, 318 318 + }; 319 319 + 320 320 + struct cxl_event_mem_module mem_module = { 321 321 + .hdr = { 322 322 + .id = UUID_INIT(0xfe927475, 0xdd59, 0x4339, 323 323 + 0xa5, 0x86, 0x79, 0xba, 0xb1, 0x13, 0xb7, 0x74), 324 324 + .length = sizeof(struct cxl_event_mem_module), 325 325 + /* .handle = Set dynamically */ 326 326 + .related_handle = cpu_to_le16(0), 327 327 + }, 328 328 + .event_type = CXL_MMER_TEMP_CHANGE, 329 329 + .info = { 330 330 + .health_status = CXL_DHI_HS_PERFORMANCE_DEGRADED, 331 331 + .media_status = CXL_DHI_MS_ALL_DATA_LOST, 332 332 + .add_status = (CXL_DHI_AS_CRITICAL << 2) | 333 333 + (CXL_DHI_AS_WARNING << 4) | 334 334 + (CXL_DHI_AS_WARNING << 5), 335 335 + .device_temp = { 0xDE, 0xAD}, 336 336 + .dirty_shutdown_cnt = { 0xde, 0xad, 0xbe, 0xef }, 337 337 + .cor_vol_err_cnt = { 0xde, 0xad, 0xbe, 0xef }, 338 338 + .cor_per_err_cnt = { 0xde, 0xad, 0xbe, 0xef }, 339 339 + } 340 340 + }; 341 341 + 342 342 + static void cxl_mock_add_event_logs(struct mock_event_store *mes) 343 343 + { 344 344 + put_unaligned_le16(CXL_GMER_VALID_CHANNEL | CXL_GMER_VALID_RANK, 345 345 + &gen_media.validity_flags); 346 346 + 347 347 + put_unaligned_le16(CXL_DER_VALID_CHANNEL | CXL_DER_VALID_BANK_GROUP | 348 348 + CXL_DER_VALID_BANK | CXL_DER_VALID_COLUMN, 349 349 + &dram.validity_flags); 350 350 + 351 351 + mes_add_event(mes, CXL_EVENT_TYPE_INFO, &maint_needed); 352 352 + mes_add_event(mes, CXL_EVENT_TYPE_INFO, 353 353 + (struct cxl_event_record_raw *)&gen_media); 354 354 + mes_add_event(mes, CXL_EVENT_TYPE_INFO, 355 355 + (struct cxl_event_record_raw *)&mem_module); 356 356 + mes->ev_status |= CXLDEV_EVENT_STATUS_INFO; 357 357 + 358 358 + mes_add_event(mes, CXL_EVENT_TYPE_FAIL, &maint_needed); 359 359 + mes_add_event(mes, CXL_EVENT_TYPE_FAIL, &hardware_replace); 360 360 + mes_add_event(mes, CXL_EVENT_TYPE_FAIL, 361 361 + (struct cxl_event_record_raw *)&dram); 362 362 + mes_add_event(mes, CXL_EVENT_TYPE_FAIL, 363 363 + (struct cxl_event_record_raw *)&gen_media); 364 364 + mes_add_event(mes, CXL_EVENT_TYPE_FAIL, 365 365 + (struct cxl_event_record_raw *)&mem_module); 366 366 + mes_add_event(mes, CXL_EVENT_TYPE_FAIL, &hardware_replace); 367 367 + mes_add_event(mes, CXL_EVENT_TYPE_FAIL, 368 368 + (struct cxl_event_record_raw *)&dram); 369 369 + /* Overflow this log */ 370 370 + mes_add_event(mes, CXL_EVENT_TYPE_FAIL, &hardware_replace); 371 371 + mes_add_event(mes, CXL_EVENT_TYPE_FAIL, &hardware_replace); 372 372 + mes_add_event(mes, CXL_EVENT_TYPE_FAIL, &hardware_replace); 373 373 + mes_add_event(mes, CXL_EVENT_TYPE_FAIL, &hardware_replace); 374 374 + mes_add_event(mes, CXL_EVENT_TYPE_FAIL, &hardware_replace); 375 375 + mes_add_event(mes, CXL_EVENT_TYPE_FAIL, &hardware_replace); 376 376 + mes_add_event(mes, CXL_EVENT_TYPE_FAIL, &hardware_replace); 377 377 + mes_add_event(mes, CXL_EVENT_TYPE_FAIL, &hardware_replace); 378 378 + mes_add_event(mes, CXL_EVENT_TYPE_FAIL, &hardware_replace); 379 379 + mes_add_event(mes, CXL_EVENT_TYPE_FAIL, &hardware_replace); 380 380 + mes->ev_status |= CXLDEV_EVENT_STATUS_FAIL; 381 381 + 382 382 + mes_add_event(mes, CXL_EVENT_TYPE_FATAL, &hardware_replace); 383 383 + mes_add_event(mes, CXL_EVENT_TYPE_FATAL, 384 384 + (struct cxl_event_record_raw *)&dram); 385 385 + mes->ev_status |= CXLDEV_EVENT_STATUS_FATAL; 386 386 + } 101 387 102 388 static int mock_gsl(struct cxl_mbox_cmd *cmd) 103 389 { ··· 912 582 case CXL_MBOX_OP_GET_PARTITION_INFO: 913 583 rc = mock_partition_info(cxlds, cmd); 914 584 break; 585 585 + case CXL_MBOX_OP_GET_EVENT_RECORD: 586 586 + rc = mock_get_event(cxlds, cmd); 587 587 + break; 588 588 + case CXL_MBOX_OP_CLEAR_EVENT_RECORD: 589 589 + rc = mock_clear_event(cxlds, cmd); 590 590 + break; 915 591 case CXL_MBOX_OP_SET_LSA: 916 592 rc = mock_set_lsa(cxlds, cmd); 917 593 break; ··· 964 628 return !!id->driver_data; 965 629 } 966 630 631 631 + static ssize_t event_trigger_store(struct device *dev, 632 632 + struct device_attribute *attr, 633 633 + const char *buf, size_t count) 634 634 + { 635 635 + cxl_mock_event_trigger(dev); 636 636 + return count; 637 637 + } 638 638 + static DEVICE_ATTR_WO(event_trigger); 639 639 + 967 640 static int cxl_mock_mem_probe(struct platform_device *pdev) 968 641 { 969 642 struct device *dev = &pdev->dev; ··· 1000 655 cxlds->serial = pdev->id; 1001 656 cxlds->mbox_send = cxl_mock_mbox_send; 1002 657 cxlds->payload_size = SZ_4K; 658 658 + cxlds->event.buf = (struct cxl_get_event_payload *) mdata->event_buf; 1003 659 if (is_rcd(pdev)) { 1004 660 cxlds->rcd = true; 1005 661 cxlds->component_reg_phys = CXL_RESOURCE_NONE; ··· 1018 672 if (rc) 1019 673 return rc; 1020 674 675 675 + mdata->mes.cxlds = cxlds; 676 676 + cxl_mock_add_event_logs(&mdata->mes); 677 677 + 1021 678 cxlmd = devm_cxl_add_memdev(cxlds); 1022 679 if (IS_ERR(cxlmd)) 1023 680 return PTR_ERR(cxlmd); 681 681 + 682 682 + cxl_mem_get_event_records(cxlds, CXLDEV_EVENT_STATUS_ALL); 1024 683 1025 684 return 0; 1026 685 } ··· 1065 714 1066 715 static struct attribute *cxl_mock_mem_attrs[] = { 1067 716 &dev_attr_security_lock.attr, 717 717 + &dev_attr_event_trigger.attr, 1068 718 NULL 1069 719 }; 1070 720 ATTRIBUTE_GROUPS(cxl_mock_mem);

+28 -8

tools/testing/cxl/test/mock.c

reviewed

··· 131 131 } 132 132 EXPORT_SYMBOL_GPL(__wrap_nvdimm_bus_register); 133 133 134 134 - struct cxl_hdm *__wrap_devm_cxl_setup_hdm(struct cxl_port *port) 134 134 + struct cxl_hdm *__wrap_devm_cxl_setup_hdm(struct cxl_port *port, 135 135 + struct cxl_endpoint_dvsec_info *info) 136 136 + 135 137 { 136 138 int index; 137 139 struct cxl_hdm *cxlhdm; 138 140 struct cxl_mock_ops *ops = get_cxl_mock_ops(&index); 139 141 140 142 if (ops && ops->is_mock_port(port->uport)) 141 141 - cxlhdm = ops->devm_cxl_setup_hdm(port); 143 143 + cxlhdm = ops->devm_cxl_setup_hdm(port, info); 142 144 else 143 143 - cxlhdm = devm_cxl_setup_hdm(port); 145 145 + cxlhdm = devm_cxl_setup_hdm(port, info); 144 146 put_cxl_mock_ops(index); 145 147 146 148 return cxlhdm; ··· 164 162 } 165 163 EXPORT_SYMBOL_NS_GPL(__wrap_devm_cxl_add_passthrough_decoder, CXL); 166 164 167 167 - int __wrap_devm_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm) 165 165 + int __wrap_devm_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm, 166 166 + struct cxl_endpoint_dvsec_info *info) 168 167 { 169 168 int rc, index; 170 169 struct cxl_port *port = cxlhdm->port; 171 170 struct cxl_mock_ops *ops = get_cxl_mock_ops(&index); 172 171 173 172 if (ops && ops->is_mock_port(port->uport)) 174 174 - rc = ops->devm_cxl_enumerate_decoders(cxlhdm); 173 173 + rc = ops->devm_cxl_enumerate_decoders(cxlhdm, info); 175 174 else 176 176 - rc = devm_cxl_enumerate_decoders(cxlhdm); 175 175 + rc = devm_cxl_enumerate_decoders(cxlhdm, info); 177 176 put_cxl_mock_ops(index); 178 177 179 178 return rc; ··· 212 209 EXPORT_SYMBOL_NS_GPL(__wrap_cxl_await_media_ready, CXL); 213 210 214 211 int __wrap_cxl_hdm_decode_init(struct cxl_dev_state *cxlds, 215 215 - struct cxl_hdm *cxlhdm) 212 212 + struct cxl_hdm *cxlhdm, 213 213 + struct cxl_endpoint_dvsec_info *info) 216 214 { 217 215 int rc = 0, index; 218 216 struct cxl_mock_ops *ops = get_cxl_mock_ops(&index); ··· 221 217 if (ops && ops->is_mock_dev(cxlds->dev)) 222 218 rc = 0; 223 219 else 224 224 - rc = cxl_hdm_decode_init(cxlds, cxlhdm); 220 220 + rc = cxl_hdm_decode_init(cxlds, cxlhdm, info); 225 221 put_cxl_mock_ops(index); 226 222 227 223 return rc; 228 224 } 229 225 EXPORT_SYMBOL_NS_GPL(__wrap_cxl_hdm_decode_init, CXL); 226 226 + 227 227 + int __wrap_cxl_dvsec_rr_decode(struct device *dev, int dvsec, 228 228 + struct cxl_endpoint_dvsec_info *info) 229 229 + { 230 230 + int rc = 0, index; 231 231 + struct cxl_mock_ops *ops = get_cxl_mock_ops(&index); 232 232 + 233 233 + if (ops && ops->is_mock_dev(dev)) 234 234 + rc = 0; 235 235 + else 236 236 + rc = cxl_dvsec_rr_decode(dev, dvsec, info); 237 237 + put_cxl_mock_ops(index); 238 238 + 239 239 + return rc; 240 240 + } 241 241 + EXPORT_SYMBOL_NS_GPL(__wrap_cxl_dvsec_rr_decode, CXL); 230 242 231 243 resource_size_t __wrap_cxl_rcrb_to_component(struct device *dev, 232 244 resource_size_t rcrb,

+4 -2

tools/testing/cxl/test/mock.h

reviewed

··· 23 23 bool (*is_mock_port)(struct device *dev); 24 24 bool (*is_mock_dev)(struct device *dev); 25 25 int (*devm_cxl_port_enumerate_dports)(struct cxl_port *port); 26 26 - struct cxl_hdm *(*devm_cxl_setup_hdm)(struct cxl_port *port); 26 26 + struct cxl_hdm *(*devm_cxl_setup_hdm)( 27 27 + struct cxl_port *port, struct cxl_endpoint_dvsec_info *info); 27 28 int (*devm_cxl_add_passthrough_decoder)(struct cxl_port *port); 28 28 - int (*devm_cxl_enumerate_decoders)(struct cxl_hdm *hdm); 29 29 + int (*devm_cxl_enumerate_decoders)( 30 30 + struct cxl_hdm *hdm, struct cxl_endpoint_dvsec_info *info); 29 31 }; 30 32 31 33 void register_cxl_mock_ops(struct cxl_mock_ops *ops);

+25

tools/testing/cxl/watermark.h

reviewed

··· 1 1 + // SPDX-License-Identifier: GPL-2.0 2 2 + /* Copyright(c) 2022 Intel Corporation. All rights reserved. */ 3 3 + #ifndef _TEST_CXL_WATERMARK_H_ 4 4 + #define _TEST_CXL_WATERMARK_H_ 5 5 + #include <linux/module.h> 6 6 + #include <linux/printk.h> 7 7 + 8 8 + int cxl_acpi_test(void); 9 9 + int cxl_core_test(void); 10 10 + int cxl_mem_test(void); 11 11 + int cxl_pmem_test(void); 12 12 + int cxl_port_test(void); 13 13 + 14 14 + /* 15 15 + * dummy routine for cxl_test to validate it is linking to the properly 16 16 + * mocked module and not the standard one from the base tree. 17 17 + */ 18 18 + #define cxl_test_watermark(x) \ 19 19 + int x##_test(void) \ 20 20 + { \ 21 21 + pr_debug("%s for cxl_test\n", KBUILD_MODNAME); \ 22 22 + return 0; \ 23 23 + } \ 24 24 + EXPORT_SYMBOL(x##_test) 25 25 + #endif /* _TEST_CXL_WATERMARK_H_ */