+127

Documentation/ntb.txt

reviewed

··· 1 1 + # NTB Drivers 2 2 + 3 3 + NTB (Non-Transparent Bridge) is a type of PCI-Express bridge chip that connects 4 4 + the separate memory systems of two computers to the same PCI-Express fabric. 5 5 + Existing NTB hardware supports a common feature set, including scratchpad 6 6 + registers, doorbell registers, and memory translation windows. Scratchpad 7 7 + registers are read-and-writable registers that are accessible from either side 8 8 + of the device, so that peers can exchange a small amount of information at a 9 9 + fixed address. Doorbell registers provide a way for peers to send interrupt 10 10 + events. Memory windows allow translated read and write access to the peer 11 11 + memory. 12 12 + 13 13 + ## NTB Core Driver (ntb) 14 14 + 15 15 + The NTB core driver defines an api wrapping the common feature set, and allows 16 16 + clients interested in NTB features to discover NTB the devices supported by 17 17 + hardware drivers. The term "client" is used here to mean an upper layer 18 18 + component making use of the NTB api. The term "driver," or "hardware driver," 19 19 + is used here to mean a driver for a specific vendor and model of NTB hardware. 20 20 + 21 21 + ## NTB Client Drivers 22 22 + 23 23 + NTB client drivers should register with the NTB core driver. After 24 24 + registering, the client probe and remove functions will be called appropriately 25 25 + as ntb hardware, or hardware drivers, are inserted and removed. The 26 26 + registration uses the Linux Device framework, so it should feel familiar to 27 27 + anyone who has written a pci driver. 28 28 + 29 29 + ### NTB Transport Client (ntb\_transport) and NTB Netdev (ntb\_netdev) 30 30 + 31 31 + The primary client for NTB is the Transport client, used in tandem with NTB 32 32 + Netdev. These drivers function together to create a logical link to the peer, 33 33 + across the ntb, to exchange packets of network data. The Transport client 34 34 + establishes a logical link to the peer, and creates queue pairs to exchange 35 35 + messages and data. The NTB Netdev then creates an ethernet device using a 36 36 + Transport queue pair. Network data is copied between socket buffers and the 37 37 + Transport queue pair buffer. The Transport client may be used for other things 38 38 + besides Netdev, however no other applications have yet been written. 39 39 + 40 40 + ### NTB Ping Pong Test Client (ntb\_pingpong) 41 41 + 42 42 + The Ping Pong test client serves as a demonstration to exercise the doorbell 43 43 + and scratchpad registers of NTB hardware, and as an example simple NTB client. 44 44 + Ping Pong enables the link when started, waits for the NTB link to come up, and 45 45 + then proceeds to read and write the doorbell scratchpad registers of the NTB. 46 46 + The peers interrupt each other using a bit mask of doorbell bits, which is 47 47 + shifted by one in each round, to test the behavior of multiple doorbell bits 48 48 + and interrupt vectors. The Ping Pong driver also reads the first local 49 49 + scratchpad, and writes the value plus one to the first peer scratchpad, each 50 50 + round before writing the peer doorbell register. 51 51 + 52 52 + Module Parameters: 53 53 + 54 54 + * unsafe - Some hardware has known issues with scratchpad and doorbell 55 55 + registers. By default, Ping Pong will not attempt to exercise such 56 56 + hardware. You may override this behavior at your own risk by setting 57 57 + unsafe=1. 58 58 + * delay\_ms - Specify the delay between receiving a doorbell 59 59 + interrupt event and setting the peer doorbell register for the next 60 60 + round. 61 61 + * init\_db - Specify the doorbell bits to start new series of rounds. A new 62 62 + series begins once all the doorbell bits have been shifted out of 63 63 + range. 64 64 + * dyndbg - It is suggested to specify dyndbg=+p when loading this module, and 65 65 + then to observe debugging output on the console. 66 66 + 67 67 + ### NTB Tool Test Client (ntb\_tool) 68 68 + 69 69 + The Tool test client serves for debugging, primarily, ntb hardware and drivers. 70 70 + The Tool provides access through debugfs for reading, setting, and clearing the 71 71 + NTB doorbell, and reading and writing scratchpads. 72 72 + 73 73 + The Tool does not currently have any module parameters. 74 74 + 75 75 + Debugfs Files: 76 76 + 77 77 + * *debugfs*/ntb\_tool/*hw*/ - A directory in debugfs will be created for each 78 78 + NTB device probed by the tool. This directory is shortened to *hw* 79 79 + below. 80 80 + * *hw*/db - This file is used to read, set, and clear the local doorbell. Not 81 81 + all operations may be supported by all hardware. To read the doorbell, 82 82 + read the file. To set the doorbell, write `s` followed by the bits to 83 83 + set (eg: `echo 's 0x0101' > db`). To clear the doorbell, write `c` 84 84 + followed by the bits to clear. 85 85 + * *hw*/mask - This file is used to read, set, and clear the local doorbell mask. 86 86 + See *db* for details. 87 87 + * *hw*/peer\_db - This file is used to read, set, and clear the peer doorbell. 88 88 + See *db* for details. 89 89 + * *hw*/peer\_mask - This file is used to read, set, and clear the peer doorbell 90 90 + mask. See *db* for details. 91 91 + * *hw*/spad - This file is used to read and write local scratchpads. To read 92 92 + the values of all scratchpads, read the file. To write values, write a 93 93 + series of pairs of scratchpad number and value 94 94 + (eg: `echo '4 0x123 7 0xabc' > spad` 95 95 + # to set scratchpads `4` and `7` to `0x123` and `0xabc`, respectively). 96 96 + * *hw*/peer\_spad - This file is used to read and write peer scratchpads. See 97 97 + *spad* for details. 98 98 + 99 99 + ## NTB Hardware Drivers 100 100 + 101 101 + NTB hardware drivers should register devices with the NTB core driver. After 102 102 + registering, clients probe and remove functions will be called. 103 103 + 104 104 + ### NTB Intel Hardware Driver (ntb\_hw\_intel) 105 105 + 106 106 + The Intel hardware driver supports NTB on Xeon and Atom CPUs. 107 107 + 108 108 + Module Parameters: 109 109 + 110 110 + * b2b\_mw\_idx - If the peer ntb is to be accessed via a memory window, then use 111 111 + this memory window to access the peer ntb. A value of zero or positive 112 112 + starts from the first mw idx, and a negative value starts from the last 113 113 + mw idx. Both sides MUST set the same value here! The default value is 114 114 + `-1`. 115 115 + * b2b\_mw\_share - If the peer ntb is to be accessed via a memory window, and if 116 116 + the memory window is large enough, still allow the client to use the 117 117 + second half of the memory window for address translation to the peer. 118 118 + * xeon\_b2b\_usd\_bar2\_addr64 - If using B2B topology on Xeon hardware, use 119 119 + this 64 bit address on the bus between the NTB devices for the window 120 120 + at BAR2, on the upstream side of the link. 121 121 + * xeon\_b2b\_usd\_bar4\_addr64 - See *xeon\_b2b\_bar2\_addr64*. 122 122 + * xeon\_b2b\_usd\_bar4\_addr32 - See *xeon\_b2b\_bar2\_addr64*. 123 123 + * xeon\_b2b\_usd\_bar5\_addr32 - See *xeon\_b2b\_bar2\_addr64*. 124 124 + * xeon\_b2b\_dsd\_bar2\_addr64 - See *xeon\_b2b\_bar2\_addr64*. 125 125 + * xeon\_b2b\_dsd\_bar4\_addr64 - See *xeon\_b2b\_bar2\_addr64*. 126 126 + * xeon\_b2b\_dsd\_bar4\_addr32 - See *xeon\_b2b\_bar2\_addr64*. 127 127 + * xeon\_b2b\_dsd\_bar5\_addr32 - See *xeon\_b2b\_bar2\_addr64*.

+11 -1

MAINTAINERS

reviewed

··· 7212 7212 F: drivers/power/isp1704_charger.c 7213 7213 F: drivers/power/rx51_battery.c 7214 7214 7215 7215 - NTB DRIVER 7215 7215 + NTB DRIVER CORE 7216 7216 M: Jon Mason <jdmason@kudzu.us> 7217 7217 M: Dave Jiang <dave.jiang@intel.com> 7218 7218 + M: Allen Hubbe <Allen.Hubbe@emc.com> 7218 7219 S: Supported 7219 7220 W: https://github.com/jonmason/ntb/wiki 7220 7221 T: git git://github.com/jonmason/ntb.git 7221 7222 F: drivers/ntb/ 7222 7223 F: drivers/net/ntb_netdev.c 7223 7224 F: include/linux/ntb.h 7225 7225 + F: include/linux/ntb_transport.h 7226 7226 + 7227 7227 + NTB INTEL DRIVER 7228 7228 + M: Jon Mason <jdmason@kudzu.us> 7229 7229 + M: Dave Jiang <dave.jiang@intel.com> 7230 7230 + S: Supported 7231 7231 + W: https://github.com/jonmason/ntb/wiki 7232 7232 + T: git git://github.com/jonmason/ntb.git 7233 7233 + F: drivers/ntb/hw/intel/ 7224 7234 7225 7235 NTFS FILESYSTEM 7226 7236 M: Anton Altaparmakov <anton@tuxera.com>

+2 -2

drivers/net/Kconfig

reviewed

··· 217 217 def_bool NETPOLL 218 218 219 219 config NTB_NETDEV 220 220 - tristate "Virtual Ethernet over NTB" 221 221 - depends on NTB 220 220 + tristate "Virtual Ethernet over NTB Transport" 221 221 + depends on NTB_TRANSPORT 222 222 223 223 config RIONET 224 224 tristate "RapidIO Ethernet over messaging driver support"

+32 -26

drivers/net/ntb_netdev.c

reviewed

··· 5 5 * GPL LICENSE SUMMARY 6 6 * 7 7 * Copyright(c) 2012 Intel Corporation. All rights reserved. 8 8 + * Copyright (C) 2015 EMC Corporation. All Rights Reserved. 8 9 * 9 10 * This program is free software; you can redistribute it and/or modify 10 11 * it under the terms of version 2 of the GNU General Public License as ··· 14 13 * BSD LICENSE 15 14 * 16 15 * Copyright(c) 2012 Intel Corporation. All rights reserved. 16 16 + * Copyright (C) 2015 EMC Corporation. All Rights Reserved. 17 17 * 18 18 * Redistribution and use in source and binary forms, with or without 19 19 * modification, are permitted provided that the following conditions ··· 42 40 * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 43 41 * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 44 42 * 45 45 - * Intel PCIe NTB Network Linux driver 43 43 + * PCIe NTB Network Linux driver 46 44 * 47 45 * Contact Information: 48 46 * Jon Mason <jon.mason@intel.com> ··· 52 50 #include <linux/module.h> 53 51 #include <linux/pci.h> 54 52 #include <linux/ntb.h> 53 53 + #include <linux/ntb_transport.h> 55 54 56 55 #define NTB_NETDEV_VER "0.7" 57 56 ··· 73 70 74 71 static LIST_HEAD(dev_list); 75 72 76 76 - static void ntb_netdev_event_handler(void *data, int status) 73 73 + static void ntb_netdev_event_handler(void *data, int link_is_up) 77 74 { 78 75 struct net_device *ndev = data; 79 76 struct ntb_netdev *dev = netdev_priv(ndev); 80 77 81 81 - netdev_dbg(ndev, "Event %x, Link %x\n", status, 78 78 + netdev_dbg(ndev, "Event %x, Link %x\n", link_is_up, 82 79 ntb_transport_link_query(dev->qp)); 83 80 84 84 - switch (status) { 85 85 - case NTB_LINK_DOWN: 81 81 + if (link_is_up) { 82 82 + if (ntb_transport_link_query(dev->qp)) 83 83 + netif_carrier_on(ndev); 84 84 + } else { 86 85 netif_carrier_off(ndev); 87 87 - break; 88 88 - case NTB_LINK_UP: 89 89 - if (!ntb_transport_link_query(dev->qp)) 90 90 - return; 91 91 - 92 92 - netif_carrier_on(ndev); 93 93 - break; 94 94 - default: 95 95 - netdev_warn(ndev, "Unsupported event type %d\n", status); 96 86 } 97 87 } 98 88 ··· 155 159 { 156 160 struct ntb_netdev *dev = netdev_priv(ndev); 157 161 int rc; 158 158 - 159 159 - netdev_dbg(ndev, "%s: skb len %d\n", __func__, skb->len); 160 162 161 163 rc = ntb_transport_tx_enqueue(dev->qp, skb, skb->data, skb->len); 162 164 if (rc) ··· 316 322 .event_handler = ntb_netdev_event_handler, 317 323 }; 318 324 319 319 - static int ntb_netdev_probe(struct pci_dev *pdev) 325 325 + static int ntb_netdev_probe(struct device *client_dev) 320 326 { 327 327 + struct ntb_dev *ntb; 321 328 struct net_device *ndev; 329 329 + struct pci_dev *pdev; 322 330 struct ntb_netdev *dev; 323 331 int rc; 324 332 325 325 - ndev = alloc_etherdev(sizeof(struct ntb_netdev)); 333 333 + ntb = dev_ntb(client_dev->parent); 334 334 + pdev = ntb->pdev; 335 335 + if (!pdev) 336 336 + return -ENODEV; 337 337 + 338 338 + ndev = alloc_etherdev(sizeof(*dev)); 326 339 if (!ndev) 327 340 return -ENOMEM; 328 341 329 342 dev = netdev_priv(ndev); 330 343 dev->ndev = ndev; 331 344 dev->pdev = pdev; 332 332 - BUG_ON(!dev->pdev); 333 345 ndev->features = NETIF_F_HIGHDMA; 334 346 335 347 ndev->priv_flags |= IFF_LIVE_ADDR_CHANGE; ··· 349 349 ndev->netdev_ops = &ntb_netdev_ops; 350 350 ndev->ethtool_ops = &ntb_ethtool_ops; 351 351 352 352 - dev->qp = ntb_transport_create_queue(ndev, pdev, &ntb_netdev_handlers); 352 352 + dev->qp = ntb_transport_create_queue(ndev, client_dev, 353 353 + &ntb_netdev_handlers); 353 354 if (!dev->qp) { 354 355 rc = -EIO; 355 356 goto err; ··· 373 372 return rc; 374 373 } 375 374 376 376 - static void ntb_netdev_remove(struct pci_dev *pdev) 375 375 + static void ntb_netdev_remove(struct device *client_dev) 377 376 { 377 377 + struct ntb_dev *ntb; 378 378 struct net_device *ndev; 379 379 + struct pci_dev *pdev; 379 380 struct ntb_netdev *dev; 380 381 bool found = false; 382 382 + 383 383 + ntb = dev_ntb(client_dev->parent); 384 384 + pdev = ntb->pdev; 381 385 382 386 list_for_each_entry(dev, &dev_list, list) { 383 387 if (dev->pdev == pdev) { ··· 402 396 free_netdev(ndev); 403 397 } 404 398 405 405 - static struct ntb_client ntb_netdev_client = { 399 399 + static struct ntb_transport_client ntb_netdev_client = { 406 400 .driver.name = KBUILD_MODNAME, 407 401 .driver.owner = THIS_MODULE, 408 402 .probe = ntb_netdev_probe, ··· 413 407 { 414 408 int rc; 415 409 416 416 - rc = ntb_register_client_dev(KBUILD_MODNAME); 410 410 + rc = ntb_transport_register_client_dev(KBUILD_MODNAME); 417 411 if (rc) 418 412 return rc; 419 419 - return ntb_register_client(&ntb_netdev_client); 413 413 + return ntb_transport_register_client(&ntb_netdev_client); 420 414 } 421 415 module_init(ntb_netdev_init_module); 422 416 423 417 static void __exit ntb_netdev_exit_module(void) 424 418 { 425 425 - ntb_unregister_client(&ntb_netdev_client); 426 426 - ntb_unregister_client_dev(KBUILD_MODNAME); 419 419 + ntb_transport_unregister_client(&ntb_netdev_client); 420 420 + ntb_transport_unregister_client_dev(KBUILD_MODNAME); 427 421 } 428 422 module_exit(ntb_netdev_exit_module);

+26 -11

drivers/ntb/Kconfig

reviewed

··· 1 1 - config NTB 2 2 - tristate "Intel Non-Transparent Bridge support" 3 3 - depends on PCI 4 4 - depends on X86 5 5 - help 6 6 - The PCI-E Non-transparent bridge hardware is a point-to-point PCI-E bus 7 7 - connecting 2 systems. When configured, writes to the device's PCI 8 8 - mapped memory will be mirrored to a buffer on the remote system. The 9 9 - ntb Linux driver uses this point-to-point communication as a method to 10 10 - transfer data from one system to the other. 1 1 + menuconfig NTB 2 2 + tristate "Non-Transparent Bridge support" 3 3 + depends on PCI 4 4 + help 5 5 + The PCI-E Non-transparent bridge hardware is a point-to-point PCI-E bus 6 6 + connecting 2 systems. When configured, writes to the device's PCI 7 7 + mapped memory will be mirrored to a buffer on the remote system. The 8 8 + ntb Linux driver uses this point-to-point communication as a method to 9 9 + transfer data from one system to the other. 11 10 12 12 - If unsure, say N. 11 11 + If unsure, say N. 13 12 13 13 + if NTB 14 14 + 15 15 + source "drivers/ntb/hw/Kconfig" 16 16 + 17 17 + source "drivers/ntb/test/Kconfig" 18 18 + 19 19 + config NTB_TRANSPORT 20 20 + tristate "NTB Transport Client" 21 21 + help 22 22 + This is a transport driver that enables connected systems to exchange 23 23 + messages over the ntb hardware. The transport exposes a queue pair api 24 24 + to client drivers. 25 25 + 26 26 + If unsure, say N. 27 27 + 28 28 + endif # NTB

+2 -3

drivers/ntb/Makefile

reviewed

··· 1 1 - obj-$(CONFIG_NTB) += ntb.o 2 2 - 3 3 - ntb-objs := ntb_hw.o ntb_transport.o 1 1 + obj-$(CONFIG_NTB) += ntb.o hw/ test/ 2 2 + obj-$(CONFIG_NTB_TRANSPORT) += ntb_transport.o

+1

drivers/ntb/hw/Kconfig

reviewed

··· 1 1 + source "drivers/ntb/hw/intel/Kconfig"

+1

drivers/ntb/hw/Makefile

reviewed

··· 1 1 + obj-$(CONFIG_NTB_INTEL) += intel/

+7

drivers/ntb/hw/intel/Kconfig

reviewed

··· 1 1 + config NTB_INTEL 2 2 + tristate "Intel Non-Transparent Bridge support" 3 3 + depends on X86_64 4 4 + help 5 5 + This driver supports Intel NTB on capable Xeon and Atom hardware. 6 6 + 7 7 + If unsure, say N.

+1

drivers/ntb/hw/intel/Makefile

reviewed

··· 1 1 + obj-$(CONFIG_NTB_INTEL) += ntb_hw_intel.o

+2274

drivers/ntb/hw/intel/ntb_hw_intel.c

reviewed

··· 1 1 + /* 2 2 + * This file is provided under a dual BSD/GPLv2 license. When using or 3 3 + * redistributing this file, you may do so under either license. 4 4 + * 5 5 + * GPL LICENSE SUMMARY 6 6 + * 7 7 + * Copyright(c) 2012 Intel Corporation. All rights reserved. 8 8 + * Copyright (C) 2015 EMC Corporation. All Rights Reserved. 9 9 + * 10 10 + * This program is free software; you can redistribute it and/or modify 11 11 + * it under the terms of version 2 of the GNU General Public License as 12 12 + * published by the Free Software Foundation. 13 13 + * 14 14 + * BSD LICENSE 15 15 + * 16 16 + * Copyright(c) 2012 Intel Corporation. All rights reserved. 17 17 + * Copyright (C) 2015 EMC Corporation. All Rights Reserved. 18 18 + * 19 19 + * Redistribution and use in source and binary forms, with or without 20 20 + * modification, are permitted provided that the following conditions 21 21 + * are met: 22 22 + * 23 23 + * * Redistributions of source code must retain the above copyright 24 24 + * notice, this list of conditions and the following disclaimer. 25 25 + * * Redistributions in binary form must reproduce the above copy 26 26 + * notice, this list of conditions and the following disclaimer in 27 27 + * the documentation and/or other materials provided with the 28 28 + * distribution. 29 29 + * * Neither the name of Intel Corporation nor the names of its 30 30 + * contributors may be used to endorse or promote products derived 31 31 + * from this software without specific prior written permission. 32 32 + * 33 33 + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 34 34 + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 35 35 + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 36 36 + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 37 37 + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 38 38 + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 39 39 + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 40 40 + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 41 41 + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 42 42 + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 43 43 + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 44 44 + * 45 45 + * Intel PCIe NTB Linux driver 46 46 + * 47 47 + * Contact Information: 48 48 + * Jon Mason <jon.mason@intel.com> 49 49 + */ 50 50 + 51 51 + #include <linux/debugfs.h> 52 52 + #include <linux/delay.h> 53 53 + #include <linux/init.h> 54 54 + #include <linux/interrupt.h> 55 55 + #include <linux/module.h> 56 56 + #include <linux/pci.h> 57 57 + #include <linux/random.h> 58 58 + #include <linux/slab.h> 59 59 + #include <linux/ntb.h> 60 60 + 61 61 + #include "ntb_hw_intel.h" 62 62 + 63 63 + #define NTB_NAME "ntb_hw_intel" 64 64 + #define NTB_DESC "Intel(R) PCI-E Non-Transparent Bridge Driver" 65 65 + #define NTB_VER "2.0" 66 66 + 67 67 + MODULE_DESCRIPTION(NTB_DESC); 68 68 + MODULE_VERSION(NTB_VER); 69 69 + MODULE_LICENSE("Dual BSD/GPL"); 70 70 + MODULE_AUTHOR("Intel Corporation"); 71 71 + 72 72 + #define bar0_off(base, bar) ((base) + ((bar) << 2)) 73 73 + #define bar2_off(base, bar) bar0_off(base, (bar) - 2) 74 74 + 75 75 + static const struct intel_ntb_reg atom_reg; 76 76 + static const struct intel_ntb_alt_reg atom_pri_reg; 77 77 + static const struct intel_ntb_alt_reg atom_sec_reg; 78 78 + static const struct intel_ntb_alt_reg atom_b2b_reg; 79 79 + static const struct intel_ntb_xlat_reg atom_pri_xlat; 80 80 + static const struct intel_ntb_xlat_reg atom_sec_xlat; 81 81 + static const struct intel_ntb_reg xeon_reg; 82 82 + static const struct intel_ntb_alt_reg xeon_pri_reg; 83 83 + static const struct intel_ntb_alt_reg xeon_sec_reg; 84 84 + static const struct intel_ntb_alt_reg xeon_b2b_reg; 85 85 + static const struct intel_ntb_xlat_reg xeon_pri_xlat; 86 86 + static const struct intel_ntb_xlat_reg xeon_sec_xlat; 87 87 + static struct intel_b2b_addr xeon_b2b_usd_addr; 88 88 + static struct intel_b2b_addr xeon_b2b_dsd_addr; 89 89 + static const struct ntb_dev_ops intel_ntb_ops; 90 90 + 91 91 + static const struct file_operations intel_ntb_debugfs_info; 92 92 + static struct dentry *debugfs_dir; 93 93 + 94 94 + static int b2b_mw_idx = -1; 95 95 + module_param(b2b_mw_idx, int, 0644); 96 96 + MODULE_PARM_DESC(b2b_mw_idx, "Use this mw idx to access the peer ntb. A " 97 97 + "value of zero or positive starts from first mw idx, and a " 98 98 + "negative value starts from last mw idx. Both sides MUST " 99 99 + "set the same value here!"); 100 100 + 101 101 + static unsigned int b2b_mw_share; 102 102 + module_param(b2b_mw_share, uint, 0644); 103 103 + MODULE_PARM_DESC(b2b_mw_share, "If the b2b mw is large enough, configure the " 104 104 + "ntb so that the peer ntb only occupies the first half of " 105 105 + "the mw, so the second half can still be used as a mw. Both " 106 106 + "sides MUST set the same value here!"); 107 107 + 108 108 + module_param_named(xeon_b2b_usd_bar2_addr64, 109 109 + xeon_b2b_usd_addr.bar2_addr64, ullong, 0644); 110 110 + MODULE_PARM_DESC(xeon_b2b_usd_bar2_addr64, 111 111 + "XEON B2B USD BAR 2 64-bit address"); 112 112 + 113 113 + module_param_named(xeon_b2b_usd_bar4_addr64, 114 114 + xeon_b2b_usd_addr.bar4_addr64, ullong, 0644); 115 115 + MODULE_PARM_DESC(xeon_b2b_usd_bar2_addr64, 116 116 + "XEON B2B USD BAR 4 64-bit address"); 117 117 + 118 118 + module_param_named(xeon_b2b_usd_bar4_addr32, 119 119 + xeon_b2b_usd_addr.bar4_addr32, ullong, 0644); 120 120 + MODULE_PARM_DESC(xeon_b2b_usd_bar2_addr64, 121 121 + "XEON B2B USD split-BAR 4 32-bit address"); 122 122 + 123 123 + module_param_named(xeon_b2b_usd_bar5_addr32, 124 124 + xeon_b2b_usd_addr.bar5_addr32, ullong, 0644); 125 125 + MODULE_PARM_DESC(xeon_b2b_usd_bar2_addr64, 126 126 + "XEON B2B USD split-BAR 5 32-bit address"); 127 127 + 128 128 + module_param_named(xeon_b2b_dsd_bar2_addr64, 129 129 + xeon_b2b_dsd_addr.bar2_addr64, ullong, 0644); 130 130 + MODULE_PARM_DESC(xeon_b2b_dsd_bar2_addr64, 131 131 + "XEON B2B DSD BAR 2 64-bit address"); 132 132 + 133 133 + module_param_named(xeon_b2b_dsd_bar4_addr64, 134 134 + xeon_b2b_dsd_addr.bar4_addr64, ullong, 0644); 135 135 + MODULE_PARM_DESC(xeon_b2b_dsd_bar2_addr64, 136 136 + "XEON B2B DSD BAR 4 64-bit address"); 137 137 + 138 138 + module_param_named(xeon_b2b_dsd_bar4_addr32, 139 139 + xeon_b2b_dsd_addr.bar4_addr32, ullong, 0644); 140 140 + MODULE_PARM_DESC(xeon_b2b_dsd_bar2_addr64, 141 141 + "XEON B2B DSD split-BAR 4 32-bit address"); 142 142 + 143 143 + module_param_named(xeon_b2b_dsd_bar5_addr32, 144 144 + xeon_b2b_dsd_addr.bar5_addr32, ullong, 0644); 145 145 + MODULE_PARM_DESC(xeon_b2b_dsd_bar2_addr64, 146 146 + "XEON B2B DSD split-BAR 5 32-bit address"); 147 147 + 148 148 + #ifndef ioread64 149 149 + #ifdef readq 150 150 + #define ioread64 readq 151 151 + #else 152 152 + #define ioread64 _ioread64 153 153 + static inline u64 _ioread64(void __iomem *mmio) 154 154 + { 155 155 + u64 low, high; 156 156 + 157 157 + low = ioread32(mmio); 158 158 + high = ioread32(mmio + sizeof(u32)); 159 159 + return low | (high << 32); 160 160 + } 161 161 + #endif 162 162 + #endif 163 163 + 164 164 + #ifndef iowrite64 165 165 + #ifdef writeq 166 166 + #define iowrite64 writeq 167 167 + #else 168 168 + #define iowrite64 _iowrite64 169 169 + static inline void _iowrite64(u64 val, void __iomem *mmio) 170 170 + { 171 171 + iowrite32(val, mmio); 172 172 + iowrite32(val >> 32, mmio + sizeof(u32)); 173 173 + } 174 174 + #endif 175 175 + #endif 176 176 + 177 177 + static inline int pdev_is_atom(struct pci_dev *pdev) 178 178 + { 179 179 + switch (pdev->device) { 180 180 + case PCI_DEVICE_ID_INTEL_NTB_B2B_BWD: 181 181 + return 1; 182 182 + } 183 183 + return 0; 184 184 + } 185 185 + 186 186 + static inline int pdev_is_xeon(struct pci_dev *pdev) 187 187 + { 188 188 + switch (pdev->device) { 189 189 + case PCI_DEVICE_ID_INTEL_NTB_SS_JSF: 190 190 + case PCI_DEVICE_ID_INTEL_NTB_SS_SNB: 191 191 + case PCI_DEVICE_ID_INTEL_NTB_SS_IVT: 192 192 + case PCI_DEVICE_ID_INTEL_NTB_SS_HSX: 193 193 + case PCI_DEVICE_ID_INTEL_NTB_PS_JSF: 194 194 + case PCI_DEVICE_ID_INTEL_NTB_PS_SNB: 195 195 + case PCI_DEVICE_ID_INTEL_NTB_PS_IVT: 196 196 + case PCI_DEVICE_ID_INTEL_NTB_PS_HSX: 197 197 + case PCI_DEVICE_ID_INTEL_NTB_B2B_JSF: 198 198 + case PCI_DEVICE_ID_INTEL_NTB_B2B_SNB: 199 199 + case PCI_DEVICE_ID_INTEL_NTB_B2B_IVT: 200 200 + case PCI_DEVICE_ID_INTEL_NTB_B2B_HSX: 201 201 + return 1; 202 202 + } 203 203 + return 0; 204 204 + } 205 205 + 206 206 + static inline void ndev_reset_unsafe_flags(struct intel_ntb_dev *ndev) 207 207 + { 208 208 + ndev->unsafe_flags = 0; 209 209 + ndev->unsafe_flags_ignore = 0; 210 210 + 211 211 + /* Only B2B has a workaround to avoid SDOORBELL */ 212 212 + if (ndev->hwerr_flags & NTB_HWERR_SDOORBELL_LOCKUP) 213 213 + if (!ntb_topo_is_b2b(ndev->ntb.topo)) 214 214 + ndev->unsafe_flags |= NTB_UNSAFE_DB; 215 215 + 216 216 + /* No low level workaround to avoid SB01BASE */ 217 217 + if (ndev->hwerr_flags & NTB_HWERR_SB01BASE_LOCKUP) { 218 218 + ndev->unsafe_flags |= NTB_UNSAFE_DB; 219 219 + ndev->unsafe_flags |= NTB_UNSAFE_SPAD; 220 220 + } 221 221 + } 222 222 + 223 223 + static inline int ndev_is_unsafe(struct intel_ntb_dev *ndev, 224 224 + unsigned long flag) 225 225 + { 226 226 + return !!(flag & ndev->unsafe_flags & ~ndev->unsafe_flags_ignore); 227 227 + } 228 228 + 229 229 + static inline int ndev_ignore_unsafe(struct intel_ntb_dev *ndev, 230 230 + unsigned long flag) 231 231 + { 232 232 + flag &= ndev->unsafe_flags; 233 233 + ndev->unsafe_flags_ignore |= flag; 234 234 + 235 235 + return !!flag; 236 236 + } 237 237 + 238 238 + static int ndev_mw_to_bar(struct intel_ntb_dev *ndev, int idx) 239 239 + { 240 240 + if (idx < 0 || idx > ndev->mw_count) 241 241 + return -EINVAL; 242 242 + return ndev->reg->mw_bar[idx]; 243 243 + } 244 244 + 245 245 + static inline int ndev_db_addr(struct intel_ntb_dev *ndev, 246 246 + phys_addr_t *db_addr, resource_size_t *db_size, 247 247 + phys_addr_t reg_addr, unsigned long reg) 248 248 + { 249 249 + if (ndev_is_unsafe(ndev, NTB_UNSAFE_DB)) 250 250 + pr_warn_once("%s: NTB unsafe doorbell access", __func__); 251 251 + 252 252 + if (db_addr) { 253 253 + *db_addr = reg_addr + reg; 254 254 + dev_dbg(ndev_dev(ndev), "Peer db addr %llx\n", *db_addr); 255 255 + } 256 256 + 257 257 + if (db_size) { 258 258 + *db_size = ndev->reg->db_size; 259 259 + dev_dbg(ndev_dev(ndev), "Peer db size %llx\n", *db_size); 260 260 + } 261 261 + 262 262 + return 0; 263 263 + } 264 264 + 265 265 + static inline u64 ndev_db_read(struct intel_ntb_dev *ndev, 266 266 + void __iomem *mmio) 267 267 + { 268 268 + if (ndev_is_unsafe(ndev, NTB_UNSAFE_DB)) 269 269 + pr_warn_once("%s: NTB unsafe doorbell access", __func__); 270 270 + 271 271 + return ndev->reg->db_ioread(mmio); 272 272 + } 273 273 + 274 274 + static inline int ndev_db_write(struct intel_ntb_dev *ndev, u64 db_bits, 275 275 + void __iomem *mmio) 276 276 + { 277 277 + if (ndev_is_unsafe(ndev, NTB_UNSAFE_DB)) 278 278 + pr_warn_once("%s: NTB unsafe doorbell access", __func__); 279 279 + 280 280 + if (db_bits & ~ndev->db_valid_mask) 281 281 + return -EINVAL; 282 282 + 283 283 + ndev->reg->db_iowrite(db_bits, mmio); 284 284 + 285 285 + return 0; 286 286 + } 287 287 + 288 288 + static inline int ndev_db_set_mask(struct intel_ntb_dev *ndev, u64 db_bits, 289 289 + void __iomem *mmio) 290 290 + { 291 291 + unsigned long irqflags; 292 292 + 293 293 + if (ndev_is_unsafe(ndev, NTB_UNSAFE_DB)) 294 294 + pr_warn_once("%s: NTB unsafe doorbell access", __func__); 295 295 + 296 296 + if (db_bits & ~ndev->db_valid_mask) 297 297 + return -EINVAL; 298 298 + 299 299 + spin_lock_irqsave(&ndev->db_mask_lock, irqflags); 300 300 + { 301 301 + ndev->db_mask |= db_bits; 302 302 + ndev->reg->db_iowrite(ndev->db_mask, mmio); 303 303 + } 304 304 + spin_unlock_irqrestore(&ndev->db_mask_lock, irqflags); 305 305 + 306 306 + return 0; 307 307 + } 308 308 + 309 309 + static inline int ndev_db_clear_mask(struct intel_ntb_dev *ndev, u64 db_bits, 310 310 + void __iomem *mmio) 311 311 + { 312 312 + unsigned long irqflags; 313 313 + 314 314 + if (ndev_is_unsafe(ndev, NTB_UNSAFE_DB)) 315 315 + pr_warn_once("%s: NTB unsafe doorbell access", __func__); 316 316 + 317 317 + if (db_bits & ~ndev->db_valid_mask) 318 318 + return -EINVAL; 319 319 + 320 320 + spin_lock_irqsave(&ndev->db_mask_lock, irqflags); 321 321 + { 322 322 + ndev->db_mask &= ~db_bits; 323 323 + ndev->reg->db_iowrite(ndev->db_mask, mmio); 324 324 + } 325 325 + spin_unlock_irqrestore(&ndev->db_mask_lock, irqflags); 326 326 + 327 327 + return 0; 328 328 + } 329 329 + 330 330 + static inline int ndev_vec_mask(struct intel_ntb_dev *ndev, int db_vector) 331 331 + { 332 332 + u64 shift, mask; 333 333 + 334 334 + shift = ndev->db_vec_shift; 335 335 + mask = BIT_ULL(shift) - 1; 336 336 + 337 337 + return mask << (shift * db_vector); 338 338 + } 339 339 + 340 340 + static inline int ndev_spad_addr(struct intel_ntb_dev *ndev, int idx, 341 341 + phys_addr_t *spad_addr, phys_addr_t reg_addr, 342 342 + unsigned long reg) 343 343 + { 344 344 + if (ndev_is_unsafe(ndev, NTB_UNSAFE_SPAD)) 345 345 + pr_warn_once("%s: NTB unsafe scratchpad access", __func__); 346 346 + 347 347 + if (idx < 0 || idx >= ndev->spad_count) 348 348 + return -EINVAL; 349 349 + 350 350 + if (spad_addr) { 351 351 + *spad_addr = reg_addr + reg + (idx << 2); 352 352 + dev_dbg(ndev_dev(ndev), "Peer spad addr %llx\n", *spad_addr); 353 353 + } 354 354 + 355 355 + return 0; 356 356 + } 357 357 + 358 358 + static inline u32 ndev_spad_read(struct intel_ntb_dev *ndev, int idx, 359 359 + void __iomem *mmio) 360 360 + { 361 361 + if (ndev_is_unsafe(ndev, NTB_UNSAFE_SPAD)) 362 362 + pr_warn_once("%s: NTB unsafe scratchpad access", __func__); 363 363 + 364 364 + if (idx < 0 || idx >= ndev->spad_count) 365 365 + return 0; 366 366 + 367 367 + return ioread32(mmio + (idx << 2)); 368 368 + } 369 369 + 370 370 + static inline int ndev_spad_write(struct intel_ntb_dev *ndev, int idx, u32 val, 371 371 + void __iomem *mmio) 372 372 + { 373 373 + if (ndev_is_unsafe(ndev, NTB_UNSAFE_SPAD)) 374 374 + pr_warn_once("%s: NTB unsafe scratchpad access", __func__); 375 375 + 376 376 + if (idx < 0 || idx >= ndev->spad_count) 377 377 + return -EINVAL; 378 378 + 379 379 + iowrite32(val, mmio + (idx << 2)); 380 380 + 381 381 + return 0; 382 382 + } 383 383 + 384 384 + static irqreturn_t ndev_interrupt(struct intel_ntb_dev *ndev, int vec) 385 385 + { 386 386 + u64 vec_mask; 387 387 + 388 388 + vec_mask = ndev_vec_mask(ndev, vec); 389 389 + 390 390 + dev_dbg(ndev_dev(ndev), "vec %d vec_mask %llx\n", vec, vec_mask); 391 391 + 392 392 + ndev->last_ts = jiffies; 393 393 + 394 394 + if (vec_mask & ndev->db_link_mask) { 395 395 + if (ndev->reg->poll_link(ndev)) 396 396 + ntb_link_event(&ndev->ntb); 397 397 + } 398 398 + 399 399 + if (vec_mask & ndev->db_valid_mask) 400 400 + ntb_db_event(&ndev->ntb, vec); 401 401 + 402 402 + return IRQ_HANDLED; 403 403 + } 404 404 + 405 405 + static irqreturn_t ndev_vec_isr(int irq, void *dev) 406 406 + { 407 407 + struct intel_ntb_vec *nvec = dev; 408 408 + 409 409 + return ndev_interrupt(nvec->ndev, nvec->num); 410 410 + } 411 411 + 412 412 + static irqreturn_t ndev_irq_isr(int irq, void *dev) 413 413 + { 414 414 + struct intel_ntb_dev *ndev = dev; 415 415 + 416 416 + return ndev_interrupt(ndev, irq - ndev_pdev(ndev)->irq); 417 417 + } 418 418 + 419 419 + static int ndev_init_isr(struct intel_ntb_dev *ndev, 420 420 + int msix_min, int msix_max, 421 421 + int msix_shift, int total_shift) 422 422 + { 423 423 + struct pci_dev *pdev; 424 424 + int rc, i, msix_count, node; 425 425 + 426 426 + pdev = ndev_pdev(ndev); 427 427 + 428 428 + node = dev_to_node(&pdev->dev); 429 429 + 430 430 + /* Mask all doorbell interrupts */ 431 431 + ndev->db_mask = ndev->db_valid_mask; 432 432 + ndev->reg->db_iowrite(ndev->db_mask, 433 433 + ndev->self_mmio + 434 434 + ndev->self_reg->db_mask); 435 435 + 436 436 + /* Try to set up msix irq */ 437 437 + 438 438 + ndev->vec = kzalloc_node(msix_max * sizeof(*ndev->vec), 439 439 + GFP_KERNEL, node); 440 440 + if (!ndev->vec) 441 441 + goto err_msix_vec_alloc; 442 442 + 443 443 + ndev->msix = kzalloc_node(msix_max * sizeof(*ndev->msix), 444 444 + GFP_KERNEL, node); 445 445 + if (!ndev->msix) 446 446 + goto err_msix_alloc; 447 447 + 448 448 + for (i = 0; i < msix_max; ++i) 449 449 + ndev->msix[i].entry = i; 450 450 + 451 451 + msix_count = pci_enable_msix_range(pdev, ndev->msix, 452 452 + msix_min, msix_max); 453 453 + if (msix_count < 0) 454 454 + goto err_msix_enable; 455 455 + 456 456 + for (i = 0; i < msix_count; ++i) { 457 457 + ndev->vec[i].ndev = ndev; 458 458 + ndev->vec[i].num = i; 459 459 + rc = request_irq(ndev->msix[i].vector, ndev_vec_isr, 0, 460 460 + "ndev_vec_isr", &ndev->vec[i]); 461 461 + if (rc) 462 462 + goto err_msix_request; 463 463 + } 464 464 + 465 465 + dev_dbg(ndev_dev(ndev), "Using msix interrupts\n"); 466 466 + ndev->db_vec_count = msix_count; 467 467 + ndev->db_vec_shift = msix_shift; 468 468 + return 0; 469 469 + 470 470 + err_msix_request: 471 471 + while (i-- > 0) 472 472 + free_irq(ndev->msix[i].vector, ndev); 473 473 + pci_disable_msix(pdev); 474 474 + err_msix_enable: 475 475 + kfree(ndev->msix); 476 476 + err_msix_alloc: 477 477 + kfree(ndev->vec); 478 478 + err_msix_vec_alloc: 479 479 + ndev->msix = NULL; 480 480 + ndev->vec = NULL; 481 481 + 482 482 + /* Try to set up msi irq */ 483 483 + 484 484 + rc = pci_enable_msi(pdev); 485 485 + if (rc) 486 486 + goto err_msi_enable; 487 487 + 488 488 + rc = request_irq(pdev->irq, ndev_irq_isr, 0, 489 489 + "ndev_irq_isr", ndev); 490 490 + if (rc) 491 491 + goto err_msi_request; 492 492 + 493 493 + dev_dbg(ndev_dev(ndev), "Using msi interrupts\n"); 494 494 + ndev->db_vec_count = 1; 495 495 + ndev->db_vec_shift = total_shift; 496 496 + return 0; 497 497 + 498 498 + err_msi_request: 499 499 + pci_disable_msi(pdev); 500 500 + err_msi_enable: 501 501 + 502 502 + /* Try to set up intx irq */ 503 503 + 504 504 + pci_intx(pdev, 1); 505 505 + 506 506 + rc = request_irq(pdev->irq, ndev_irq_isr, IRQF_SHARED, 507 507 + "ndev_irq_isr", ndev); 508 508 + if (rc) 509 509 + goto err_intx_request; 510 510 + 511 511 + dev_dbg(ndev_dev(ndev), "Using intx interrupts\n"); 512 512 + ndev->db_vec_count = 1; 513 513 + ndev->db_vec_shift = total_shift; 514 514 + return 0; 515 515 + 516 516 + err_intx_request: 517 517 + return rc; 518 518 + } 519 519 + 520 520 + static void ndev_deinit_isr(struct intel_ntb_dev *ndev) 521 521 + { 522 522 + struct pci_dev *pdev; 523 523 + int i; 524 524 + 525 525 + pdev = ndev_pdev(ndev); 526 526 + 527 527 + /* Mask all doorbell interrupts */ 528 528 + ndev->db_mask = ndev->db_valid_mask; 529 529 + ndev->reg->db_iowrite(ndev->db_mask, 530 530 + ndev->self_mmio + 531 531 + ndev->self_reg->db_mask); 532 532 + 533 533 + if (ndev->msix) { 534 534 + i = ndev->db_vec_count; 535 535 + while (i--) 536 536 + free_irq(ndev->msix[i].vector, &ndev->vec[i]); 537 537 + pci_disable_msix(pdev); 538 538 + kfree(ndev->msix); 539 539 + kfree(ndev->vec); 540 540 + } else { 541 541 + free_irq(pdev->irq, ndev); 542 542 + if (pci_dev_msi_enabled(pdev)) 543 543 + pci_disable_msi(pdev); 544 544 + } 545 545 + } 546 546 + 547 547 + static ssize_t ndev_debugfs_read(struct file *filp, char __user *ubuf, 548 548 + size_t count, loff_t *offp) 549 549 + { 550 550 + struct intel_ntb_dev *ndev; 551 551 + void __iomem *mmio; 552 552 + char *buf; 553 553 + size_t buf_size; 554 554 + ssize_t ret, off; 555 555 + union { u64 v64; u32 v32; u16 v16; } u; 556 556 + 557 557 + ndev = filp->private_data; 558 558 + mmio = ndev->self_mmio; 559 559 + 560 560 + buf_size = min(count, 0x800ul); 561 561 + 562 562 + buf = kmalloc(buf_size, GFP_KERNEL); 563 563 + if (!buf) 564 564 + return -ENOMEM; 565 565 + 566 566 + off = 0; 567 567 + 568 568 + off += scnprintf(buf + off, buf_size - off, 569 569 + "NTB Device Information:\n"); 570 570 + 571 571 + off += scnprintf(buf + off, buf_size - off, 572 572 + "Connection Topology -\t%s\n", 573 573 + ntb_topo_string(ndev->ntb.topo)); 574 574 + 575 575 + off += scnprintf(buf + off, buf_size - off, 576 576 + "B2B Offset -\t\t%#lx\n", ndev->b2b_off); 577 577 + off += scnprintf(buf + off, buf_size - off, 578 578 + "B2B MW Idx -\t\t%d\n", ndev->b2b_idx); 579 579 + off += scnprintf(buf + off, buf_size - off, 580 580 + "BAR4 Split -\t\t%s\n", 581 581 + ndev->bar4_split ? "yes" : "no"); 582 582 + 583 583 + off += scnprintf(buf + off, buf_size - off, 584 584 + "NTB CTL -\t\t%#06x\n", ndev->ntb_ctl); 585 585 + off += scnprintf(buf + off, buf_size - off, 586 586 + "LNK STA -\t\t%#06x\n", ndev->lnk_sta); 587 587 + 588 588 + if (!ndev->reg->link_is_up(ndev)) { 589 589 + off += scnprintf(buf + off, buf_size - off, 590 590 + "Link Status -\t\tDown\n"); 591 591 + } else { 592 592 + off += scnprintf(buf + off, buf_size - off, 593 593 + "Link Status -\t\tUp\n"); 594 594 + off += scnprintf(buf + off, buf_size - off, 595 595 + "Link Speed -\t\tPCI-E Gen %u\n", 596 596 + NTB_LNK_STA_SPEED(ndev->lnk_sta)); 597 597 + off += scnprintf(buf + off, buf_size - off, 598 598 + "Link Width -\t\tx%u\n", 599 599 + NTB_LNK_STA_WIDTH(ndev->lnk_sta)); 600 600 + } 601 601 + 602 602 + off += scnprintf(buf + off, buf_size - off, 603 603 + "Memory Window Count -\t%u\n", ndev->mw_count); 604 604 + off += scnprintf(buf + off, buf_size - off, 605 605 + "Scratchpad Count -\t%u\n", ndev->spad_count); 606 606 + off += scnprintf(buf + off, buf_size - off, 607 607 + "Doorbell Count -\t%u\n", ndev->db_count); 608 608 + off += scnprintf(buf + off, buf_size - off, 609 609 + "Doorbell Vector Count -\t%u\n", ndev->db_vec_count); 610 610 + off += scnprintf(buf + off, buf_size - off, 611 611 + "Doorbell Vector Shift -\t%u\n", ndev->db_vec_shift); 612 612 + 613 613 + off += scnprintf(buf + off, buf_size - off, 614 614 + "Doorbell Valid Mask -\t%#llx\n", ndev->db_valid_mask); 615 615 + off += scnprintf(buf + off, buf_size - off, 616 616 + "Doorbell Link Mask -\t%#llx\n", ndev->db_link_mask); 617 617 + off += scnprintf(buf + off, buf_size - off, 618 618 + "Doorbell Mask Cached -\t%#llx\n", ndev->db_mask); 619 619 + 620 620 + u.v64 = ndev_db_read(ndev, mmio + ndev->self_reg->db_mask); 621 621 + off += scnprintf(buf + off, buf_size - off, 622 622 + "Doorbell Mask -\t\t%#llx\n", u.v64); 623 623 + 624 624 + u.v64 = ndev_db_read(ndev, mmio + ndev->self_reg->db_bell); 625 625 + off += scnprintf(buf + off, buf_size - off, 626 626 + "Doorbell Bell -\t\t%#llx\n", u.v64); 627 627 + 628 628 + off += scnprintf(buf + off, buf_size - off, 629 629 + "\nNTB Incoming XLAT:\n"); 630 630 + 631 631 + u.v64 = ioread64(mmio + bar2_off(ndev->xlat_reg->bar2_xlat, 2)); 632 632 + off += scnprintf(buf + off, buf_size - off, 633 633 + "XLAT23 -\t\t%#018llx\n", u.v64); 634 634 + 635 635 + if (ndev->bar4_split) { 636 636 + u.v32 = ioread32(mmio + bar2_off(ndev->xlat_reg->bar2_xlat, 4)); 637 637 + off += scnprintf(buf + off, buf_size - off, 638 638 + "XLAT4 -\t\t\t%#06x\n", u.v32); 639 639 + 640 640 + u.v32 = ioread32(mmio + bar2_off(ndev->xlat_reg->bar2_xlat, 5)); 641 641 + off += scnprintf(buf + off, buf_size - off, 642 642 + "XLAT5 -\t\t\t%#06x\n", u.v32); 643 643 + } else { 644 644 + u.v64 = ioread64(mmio + bar2_off(ndev->xlat_reg->bar2_xlat, 4)); 645 645 + off += scnprintf(buf + off, buf_size - off, 646 646 + "XLAT45 -\t\t%#018llx\n", u.v64); 647 647 + } 648 648 + 649 649 + u.v64 = ioread64(mmio + bar2_off(ndev->xlat_reg->bar2_limit, 2)); 650 650 + off += scnprintf(buf + off, buf_size - off, 651 651 + "LMT23 -\t\t\t%#018llx\n", u.v64); 652 652 + 653 653 + if (ndev->bar4_split) { 654 654 + u.v32 = ioread32(mmio + bar2_off(ndev->xlat_reg->bar2_limit, 4)); 655 655 + off += scnprintf(buf + off, buf_size - off, 656 656 + "LMT4 -\t\t\t%#06x\n", u.v32); 657 657 + u.v32 = ioread32(mmio + bar2_off(ndev->xlat_reg->bar2_limit, 5)); 658 658 + off += scnprintf(buf + off, buf_size - off, 659 659 + "LMT5 -\t\t\t%#06x\n", u.v32); 660 660 + } else { 661 661 + u.v64 = ioread64(mmio + bar2_off(ndev->xlat_reg->bar2_limit, 4)); 662 662 + off += scnprintf(buf + off, buf_size - off, 663 663 + "LMT45 -\t\t\t%#018llx\n", u.v64); 664 664 + } 665 665 + 666 666 + if (pdev_is_xeon(ndev->ntb.pdev)) { 667 667 + if (ntb_topo_is_b2b(ndev->ntb.topo)) { 668 668 + off += scnprintf(buf + off, buf_size - off, 669 669 + "\nNTB Outgoing B2B XLAT:\n"); 670 670 + 671 671 + u.v64 = ioread64(mmio + XEON_PBAR23XLAT_OFFSET); 672 672 + off += scnprintf(buf + off, buf_size - off, 673 673 + "B2B XLAT23 -\t\t%#018llx\n", u.v64); 674 674 + 675 675 + if (ndev->bar4_split) { 676 676 + u.v32 = ioread32(mmio + XEON_PBAR4XLAT_OFFSET); 677 677 + off += scnprintf(buf + off, buf_size - off, 678 678 + "B2B XLAT4 -\t\t%#06x\n", 679 679 + u.v32); 680 680 + u.v32 = ioread32(mmio + XEON_PBAR5XLAT_OFFSET); 681 681 + off += scnprintf(buf + off, buf_size - off, 682 682 + "B2B XLAT5 -\t\t%#06x\n", 683 683 + u.v32); 684 684 + } else { 685 685 + u.v64 = ioread64(mmio + XEON_PBAR45XLAT_OFFSET); 686 686 + off += scnprintf(buf + off, buf_size - off, 687 687 + "B2B XLAT45 -\t\t%#018llx\n", 688 688 + u.v64); 689 689 + } 690 690 + 691 691 + u.v64 = ioread64(mmio + XEON_PBAR23LMT_OFFSET); 692 692 + off += scnprintf(buf + off, buf_size - off, 693 693 + "B2B LMT23 -\t\t%#018llx\n", u.v64); 694 694 + 695 695 + if (ndev->bar4_split) { 696 696 + u.v32 = ioread32(mmio + XEON_PBAR4LMT_OFFSET); 697 697 + off += scnprintf(buf + off, buf_size - off, 698 698 + "B2B LMT4 -\t\t%#06x\n", 699 699 + u.v32); 700 700 + u.v32 = ioread32(mmio + XEON_PBAR5LMT_OFFSET); 701 701 + off += scnprintf(buf + off, buf_size - off, 702 702 + "B2B LMT5 -\t\t%#06x\n", 703 703 + u.v32); 704 704 + } else { 705 705 + u.v64 = ioread64(mmio + XEON_PBAR45LMT_OFFSET); 706 706 + off += scnprintf(buf + off, buf_size - off, 707 707 + "B2B LMT45 -\t\t%#018llx\n", 708 708 + u.v64); 709 709 + } 710 710 + 711 711 + off += scnprintf(buf + off, buf_size - off, 712 712 + "\nNTB Secondary BAR:\n"); 713 713 + 714 714 + u.v64 = ioread64(mmio + XEON_SBAR0BASE_OFFSET); 715 715 + off += scnprintf(buf + off, buf_size - off, 716 716 + "SBAR01 -\t\t%#018llx\n", u.v64); 717 717 + 718 718 + u.v64 = ioread64(mmio + XEON_SBAR23BASE_OFFSET); 719 719 + off += scnprintf(buf + off, buf_size - off, 720 720 + "SBAR23 -\t\t%#018llx\n", u.v64); 721 721 + 722 722 + if (ndev->bar4_split) { 723 723 + u.v32 = ioread32(mmio + XEON_SBAR4BASE_OFFSET); 724 724 + off += scnprintf(buf + off, buf_size - off, 725 725 + "SBAR4 -\t\t\t%#06x\n", u.v32); 726 726 + u.v32 = ioread32(mmio + XEON_SBAR5BASE_OFFSET); 727 727 + off += scnprintf(buf + off, buf_size - off, 728 728 + "SBAR5 -\t\t\t%#06x\n", u.v32); 729 729 + } else { 730 730 + u.v64 = ioread64(mmio + XEON_SBAR45BASE_OFFSET); 731 731 + off += scnprintf(buf + off, buf_size - off, 732 732 + "SBAR45 -\t\t%#018llx\n", 733 733 + u.v64); 734 734 + } 735 735 + } 736 736 + 737 737 + off += scnprintf(buf + off, buf_size - off, 738 738 + "\nXEON NTB Statistics:\n"); 739 739 + 740 740 + u.v16 = ioread16(mmio + XEON_USMEMMISS_OFFSET); 741 741 + off += scnprintf(buf + off, buf_size - off, 742 742 + "Upstream Memory Miss -\t%u\n", u.v16); 743 743 + 744 744 + off += scnprintf(buf + off, buf_size - off, 745 745 + "\nXEON NTB Hardware Errors:\n"); 746 746 + 747 747 + if (!pci_read_config_word(ndev->ntb.pdev, 748 748 + XEON_DEVSTS_OFFSET, &u.v16)) 749 749 + off += scnprintf(buf + off, buf_size - off, 750 750 + "DEVSTS -\t\t%#06x\n", u.v16); 751 751 + 752 752 + if (!pci_read_config_word(ndev->ntb.pdev, 753 753 + XEON_LINK_STATUS_OFFSET, &u.v16)) 754 754 + off += scnprintf(buf + off, buf_size - off, 755 755 + "LNKSTS -\t\t%#06x\n", u.v16); 756 756 + 757 757 + if (!pci_read_config_dword(ndev->ntb.pdev, 758 758 + XEON_UNCERRSTS_OFFSET, &u.v32)) 759 759 + off += scnprintf(buf + off, buf_size - off, 760 760 + "UNCERRSTS -\t\t%#06x\n", u.v32); 761 761 + 762 762 + if (!pci_read_config_dword(ndev->ntb.pdev, 763 763 + XEON_CORERRSTS_OFFSET, &u.v32)) 764 764 + off += scnprintf(buf + off, buf_size - off, 765 765 + "CORERRSTS -\t\t%#06x\n", u.v32); 766 766 + } 767 767 + 768 768 + ret = simple_read_from_buffer(ubuf, count, offp, buf, off); 769 769 + kfree(buf); 770 770 + return ret; 771 771 + } 772 772 + 773 773 + static void ndev_init_debugfs(struct intel_ntb_dev *ndev) 774 774 + { 775 775 + if (!debugfs_dir) { 776 776 + ndev->debugfs_dir = NULL; 777 777 + ndev->debugfs_info = NULL; 778 778 + } else { 779 779 + ndev->debugfs_dir = 780 780 + debugfs_create_dir(ndev_name(ndev), debugfs_dir); 781 781 + if (!ndev->debugfs_dir) 782 782 + ndev->debugfs_info = NULL; 783 783 + else 784 784 + ndev->debugfs_info = 785 785 + debugfs_create_file("info", S_IRUSR, 786 786 + ndev->debugfs_dir, ndev, 787 787 + &intel_ntb_debugfs_info); 788 788 + } 789 789 + } 790 790 + 791 791 + static void ndev_deinit_debugfs(struct intel_ntb_dev *ndev) 792 792 + { 793 793 + debugfs_remove_recursive(ndev->debugfs_dir); 794 794 + } 795 795 + 796 796 + static int intel_ntb_mw_count(struct ntb_dev *ntb) 797 797 + { 798 798 + return ntb_ndev(ntb)->mw_count; 799 799 + } 800 800 + 801 801 + static int intel_ntb_mw_get_range(struct ntb_dev *ntb, int idx, 802 802 + phys_addr_t *base, 803 803 + resource_size_t *size, 804 804 + resource_size_t *align, 805 805 + resource_size_t *align_size) 806 806 + { 807 807 + struct intel_ntb_dev *ndev = ntb_ndev(ntb); 808 808 + int bar; 809 809 + 810 810 + if (idx >= ndev->b2b_idx && !ndev->b2b_off) 811 811 + idx += 1; 812 812 + 813 813 + bar = ndev_mw_to_bar(ndev, idx); 814 814 + if (bar < 0) 815 815 + return bar; 816 816 + 817 817 + if (base) 818 818 + *base = pci_resource_start(ndev->ntb.pdev, bar) + 819 819 + (idx == ndev->b2b_idx ? ndev->b2b_off : 0); 820 820 + 821 821 + if (size) 822 822 + *size = pci_resource_len(ndev->ntb.pdev, bar) - 823 823 + (idx == ndev->b2b_idx ? ndev->b2b_off : 0); 824 824 + 825 825 + if (align) 826 826 + *align = pci_resource_len(ndev->ntb.pdev, bar); 827 827 + 828 828 + if (align_size) 829 829 + *align_size = 1; 830 830 + 831 831 + return 0; 832 832 + } 833 833 + 834 834 + static int intel_ntb_mw_set_trans(struct ntb_dev *ntb, int idx, 835 835 + dma_addr_t addr, resource_size_t size) 836 836 + { 837 837 + struct intel_ntb_dev *ndev = ntb_ndev(ntb); 838 838 + unsigned long base_reg, xlat_reg, limit_reg; 839 839 + resource_size_t bar_size, mw_size; 840 840 + void __iomem *mmio; 841 841 + u64 base, limit, reg_val; 842 842 + int bar; 843 843 + 844 844 + if (idx >= ndev->b2b_idx && !ndev->b2b_off) 845 845 + idx += 1; 846 846 + 847 847 + bar = ndev_mw_to_bar(ndev, idx); 848 848 + if (bar < 0) 849 849 + return bar; 850 850 + 851 851 + bar_size = pci_resource_len(ndev->ntb.pdev, bar); 852 852 + 853 853 + if (idx == ndev->b2b_idx) 854 854 + mw_size = bar_size - ndev->b2b_off; 855 855 + else 856 856 + mw_size = bar_size; 857 857 + 858 858 + /* hardware requires that addr is aligned to bar size */ 859 859 + if (addr & (bar_size - 1)) 860 860 + return -EINVAL; 861 861 + 862 862 + /* make sure the range fits in the usable mw size */ 863 863 + if (size > mw_size) 864 864 + return -EINVAL; 865 865 + 866 866 + mmio = ndev->self_mmio; 867 867 + base_reg = bar0_off(ndev->xlat_reg->bar0_base, bar); 868 868 + xlat_reg = bar2_off(ndev->xlat_reg->bar2_xlat, bar); 869 869 + limit_reg = bar2_off(ndev->xlat_reg->bar2_limit, bar); 870 870 + 871 871 + if (bar < 4 || !ndev->bar4_split) { 872 872 + base = ioread64(mmio + base_reg); 873 873 + 874 874 + /* Set the limit if supported, if size is not mw_size */ 875 875 + if (limit_reg && size != mw_size) 876 876 + limit = base + size; 877 877 + else 878 878 + limit = 0; 879 879 + 880 880 + /* set and verify setting the translation address */ 881 881 + iowrite64(addr, mmio + xlat_reg); 882 882 + reg_val = ioread64(mmio + xlat_reg); 883 883 + if (reg_val != addr) { 884 884 + iowrite64(0, mmio + xlat_reg); 885 885 + return -EIO; 886 886 + } 887 887 + 888 888 + /* set and verify setting the limit */ 889 889 + iowrite64(limit, mmio + limit_reg); 890 890 + reg_val = ioread64(mmio + limit_reg); 891 891 + if (reg_val != limit) { 892 892 + iowrite64(base, mmio + limit_reg); 893 893 + iowrite64(0, mmio + xlat_reg); 894 894 + return -EIO; 895 895 + } 896 896 + } else { 897 897 + /* split bar addr range must all be 32 bit */ 898 898 + if (addr & (~0ull << 32)) 899 899 + return -EINVAL; 900 900 + if ((addr + size) & (~0ull << 32)) 901 901 + return -EINVAL; 902 902 + 903 903 + base = ioread32(mmio + base_reg); 904 904 + 905 905 + /* Set the limit if supported, if size is not mw_size */ 906 906 + if (limit_reg && size != mw_size) 907 907 + limit = base + size; 908 908 + else 909 909 + limit = 0; 910 910 + 911 911 + /* set and verify setting the translation address */ 912 912 + iowrite32(addr, mmio + xlat_reg); 913 913 + reg_val = ioread32(mmio + xlat_reg); 914 914 + if (reg_val != addr) { 915 915 + iowrite32(0, mmio + xlat_reg); 916 916 + return -EIO; 917 917 + } 918 918 + 919 919 + /* set and verify setting the limit */ 920 920 + iowrite32(limit, mmio + limit_reg); 921 921 + reg_val = ioread32(mmio + limit_reg); 922 922 + if (reg_val != limit) { 923 923 + iowrite32(base, mmio + limit_reg); 924 924 + iowrite32(0, mmio + xlat_reg); 925 925 + return -EIO; 926 926 + } 927 927 + } 928 928 + 929 929 + return 0; 930 930 + } 931 931 + 932 932 + static int intel_ntb_link_is_up(struct ntb_dev *ntb, 933 933 + enum ntb_speed *speed, 934 934 + enum ntb_width *width) 935 935 + { 936 936 + struct intel_ntb_dev *ndev = ntb_ndev(ntb); 937 937 + 938 938 + if (ndev->reg->link_is_up(ndev)) { 939 939 + if (speed) 940 940 + *speed = NTB_LNK_STA_SPEED(ndev->lnk_sta); 941 941 + if (width) 942 942 + *width = NTB_LNK_STA_WIDTH(ndev->lnk_sta); 943 943 + return 1; 944 944 + } else { 945 945 + /* TODO MAYBE: is it possible to observe the link speed and 946 946 + * width while link is training? */ 947 947 + if (speed) 948 948 + *speed = NTB_SPEED_NONE; 949 949 + if (width) 950 950 + *width = NTB_WIDTH_NONE; 951 951 + return 0; 952 952 + } 953 953 + } 954 954 + 955 955 + static int intel_ntb_link_enable(struct ntb_dev *ntb, 956 956 + enum ntb_speed max_speed, 957 957 + enum ntb_width max_width) 958 958 + { 959 959 + struct intel_ntb_dev *ndev; 960 960 + u32 ntb_ctl; 961 961 + 962 962 + ndev = container_of(ntb, struct intel_ntb_dev, ntb); 963 963 + 964 964 + if (ndev->ntb.topo == NTB_TOPO_SEC) 965 965 + return -EINVAL; 966 966 + 967 967 + dev_dbg(ndev_dev(ndev), 968 968 + "Enabling link with max_speed %d max_width %d\n", 969 969 + max_speed, max_width); 970 970 + if (max_speed != NTB_SPEED_AUTO) 971 971 + dev_dbg(ndev_dev(ndev), "ignoring max_speed %d\n", max_speed); 972 972 + if (max_width != NTB_WIDTH_AUTO) 973 973 + dev_dbg(ndev_dev(ndev), "ignoring max_width %d\n", max_width); 974 974 + 975 975 + ntb_ctl = ioread32(ndev->self_mmio + ndev->reg->ntb_ctl); 976 976 + ntb_ctl &= ~(NTB_CTL_DISABLE | NTB_CTL_CFG_LOCK); 977 977 + ntb_ctl |= NTB_CTL_P2S_BAR2_SNOOP | NTB_CTL_S2P_BAR2_SNOOP; 978 978 + ntb_ctl |= NTB_CTL_P2S_BAR4_SNOOP | NTB_CTL_S2P_BAR4_SNOOP; 979 979 + if (ndev->bar4_split) 980 980 + ntb_ctl |= NTB_CTL_P2S_BAR5_SNOOP | NTB_CTL_S2P_BAR5_SNOOP; 981 981 + iowrite32(ntb_ctl, ndev->self_mmio + ndev->reg->ntb_ctl); 982 982 + 983 983 + return 0; 984 984 + } 985 985 + 986 986 + static int intel_ntb_link_disable(struct ntb_dev *ntb) 987 987 + { 988 988 + struct intel_ntb_dev *ndev; 989 989 + u32 ntb_cntl; 990 990 + 991 991 + ndev = container_of(ntb, struct intel_ntb_dev, ntb); 992 992 + 993 993 + if (ndev->ntb.topo == NTB_TOPO_SEC) 994 994 + return -EINVAL; 995 995 + 996 996 + dev_dbg(ndev_dev(ndev), "Disabling link\n"); 997 997 + 998 998 + /* Bring NTB link down */ 999 999 + ntb_cntl = ioread32(ndev->self_mmio + ndev->reg->ntb_ctl); 1000 1000 + ntb_cntl &= ~(NTB_CTL_P2S_BAR2_SNOOP | NTB_CTL_S2P_BAR2_SNOOP); 1001 1001 + ntb_cntl &= ~(NTB_CTL_P2S_BAR4_SNOOP | NTB_CTL_S2P_BAR4_SNOOP); 1002 1002 + if (ndev->bar4_split) 1003 1003 + ntb_cntl &= ~(NTB_CTL_P2S_BAR5_SNOOP | NTB_CTL_S2P_BAR5_SNOOP); 1004 1004 + ntb_cntl |= NTB_CTL_DISABLE | NTB_CTL_CFG_LOCK; 1005 1005 + iowrite32(ntb_cntl, ndev->self_mmio + ndev->reg->ntb_ctl); 1006 1006 + 1007 1007 + return 0; 1008 1008 + } 1009 1009 + 1010 1010 + static int intel_ntb_db_is_unsafe(struct ntb_dev *ntb) 1011 1011 + { 1012 1012 + return ndev_ignore_unsafe(ntb_ndev(ntb), NTB_UNSAFE_DB); 1013 1013 + } 1014 1014 + 1015 1015 + static u64 intel_ntb_db_valid_mask(struct ntb_dev *ntb) 1016 1016 + { 1017 1017 + return ntb_ndev(ntb)->db_valid_mask; 1018 1018 + } 1019 1019 + 1020 1020 + static int intel_ntb_db_vector_count(struct ntb_dev *ntb) 1021 1021 + { 1022 1022 + struct intel_ntb_dev *ndev; 1023 1023 + 1024 1024 + ndev = container_of(ntb, struct intel_ntb_dev, ntb); 1025 1025 + 1026 1026 + return ndev->db_vec_count; 1027 1027 + } 1028 1028 + 1029 1029 + static u64 intel_ntb_db_vector_mask(struct ntb_dev *ntb, int db_vector) 1030 1030 + { 1031 1031 + struct intel_ntb_dev *ndev = ntb_ndev(ntb); 1032 1032 + 1033 1033 + if (db_vector < 0 || db_vector > ndev->db_vec_count) 1034 1034 + return 0; 1035 1035 + 1036 1036 + return ndev->db_valid_mask & ndev_vec_mask(ndev, db_vector); 1037 1037 + } 1038 1038 + 1039 1039 + static u64 intel_ntb_db_read(struct ntb_dev *ntb) 1040 1040 + { 1041 1041 + struct intel_ntb_dev *ndev = ntb_ndev(ntb); 1042 1042 + 1043 1043 + return ndev_db_read(ndev, 1044 1044 + ndev->self_mmio + 1045 1045 + ndev->self_reg->db_bell); 1046 1046 + } 1047 1047 + 1048 1048 + static int intel_ntb_db_clear(struct ntb_dev *ntb, u64 db_bits) 1049 1049 + { 1050 1050 + struct intel_ntb_dev *ndev = ntb_ndev(ntb); 1051 1051 + 1052 1052 + return ndev_db_write(ndev, db_bits, 1053 1053 + ndev->self_mmio + 1054 1054 + ndev->self_reg->db_bell); 1055 1055 + } 1056 1056 + 1057 1057 + static int intel_ntb_db_set_mask(struct ntb_dev *ntb, u64 db_bits) 1058 1058 + { 1059 1059 + struct intel_ntb_dev *ndev = ntb_ndev(ntb); 1060 1060 + 1061 1061 + return ndev_db_set_mask(ndev, db_bits, 1062 1062 + ndev->self_mmio + 1063 1063 + ndev->self_reg->db_mask); 1064 1064 + } 1065 1065 + 1066 1066 + static int intel_ntb_db_clear_mask(struct ntb_dev *ntb, u64 db_bits) 1067 1067 + { 1068 1068 + struct intel_ntb_dev *ndev = ntb_ndev(ntb); 1069 1069 + 1070 1070 + return ndev_db_clear_mask(ndev, db_bits, 1071 1071 + ndev->self_mmio + 1072 1072 + ndev->self_reg->db_mask); 1073 1073 + } 1074 1074 + 1075 1075 + static int intel_ntb_peer_db_addr(struct ntb_dev *ntb, 1076 1076 + phys_addr_t *db_addr, 1077 1077 + resource_size_t *db_size) 1078 1078 + { 1079 1079 + struct intel_ntb_dev *ndev = ntb_ndev(ntb); 1080 1080 + 1081 1081 + return ndev_db_addr(ndev, db_addr, db_size, ndev->peer_addr, 1082 1082 + ndev->peer_reg->db_bell); 1083 1083 + } 1084 1084 + 1085 1085 + static int intel_ntb_peer_db_set(struct ntb_dev *ntb, u64 db_bits) 1086 1086 + { 1087 1087 + struct intel_ntb_dev *ndev = ntb_ndev(ntb); 1088 1088 + 1089 1089 + return ndev_db_write(ndev, db_bits, 1090 1090 + ndev->peer_mmio + 1091 1091 + ndev->peer_reg->db_bell); 1092 1092 + } 1093 1093 + 1094 1094 + static int intel_ntb_spad_is_unsafe(struct ntb_dev *ntb) 1095 1095 + { 1096 1096 + return ndev_ignore_unsafe(ntb_ndev(ntb), NTB_UNSAFE_SPAD); 1097 1097 + } 1098 1098 + 1099 1099 + static int intel_ntb_spad_count(struct ntb_dev *ntb) 1100 1100 + { 1101 1101 + struct intel_ntb_dev *ndev; 1102 1102 + 1103 1103 + ndev = container_of(ntb, struct intel_ntb_dev, ntb); 1104 1104 + 1105 1105 + return ndev->spad_count; 1106 1106 + } 1107 1107 + 1108 1108 + static u32 intel_ntb_spad_read(struct ntb_dev *ntb, int idx) 1109 1109 + { 1110 1110 + struct intel_ntb_dev *ndev = ntb_ndev(ntb); 1111 1111 + 1112 1112 + return ndev_spad_read(ndev, idx, 1113 1113 + ndev->self_mmio + 1114 1114 + ndev->self_reg->spad); 1115 1115 + } 1116 1116 + 1117 1117 + static int intel_ntb_spad_write(struct ntb_dev *ntb, 1118 1118 + int idx, u32 val) 1119 1119 + { 1120 1120 + struct intel_ntb_dev *ndev = ntb_ndev(ntb); 1121 1121 + 1122 1122 + return ndev_spad_write(ndev, idx, val, 1123 1123 + ndev->self_mmio + 1124 1124 + ndev->self_reg->spad); 1125 1125 + } 1126 1126 + 1127 1127 + static int intel_ntb_peer_spad_addr(struct ntb_dev *ntb, int idx, 1128 1128 + phys_addr_t *spad_addr) 1129 1129 + { 1130 1130 + struct intel_ntb_dev *ndev = ntb_ndev(ntb); 1131 1131 + 1132 1132 + return ndev_spad_addr(ndev, idx, spad_addr, ndev->peer_addr, 1133 1133 + ndev->peer_reg->spad); 1134 1134 + } 1135 1135 + 1136 1136 + static u32 intel_ntb_peer_spad_read(struct ntb_dev *ntb, int idx) 1137 1137 + { 1138 1138 + struct intel_ntb_dev *ndev = ntb_ndev(ntb); 1139 1139 + 1140 1140 + return ndev_spad_read(ndev, idx, 1141 1141 + ndev->peer_mmio + 1142 1142 + ndev->peer_reg->spad); 1143 1143 + } 1144 1144 + 1145 1145 + static int intel_ntb_peer_spad_write(struct ntb_dev *ntb, 1146 1146 + int idx, u32 val) 1147 1147 + { 1148 1148 + struct intel_ntb_dev *ndev = ntb_ndev(ntb); 1149 1149 + 1150 1150 + return ndev_spad_write(ndev, idx, val, 1151 1151 + ndev->peer_mmio + 1152 1152 + ndev->peer_reg->spad); 1153 1153 + } 1154 1154 + 1155 1155 + /* ATOM */ 1156 1156 + 1157 1157 + static u64 atom_db_ioread(void __iomem *mmio) 1158 1158 + { 1159 1159 + return ioread64(mmio); 1160 1160 + } 1161 1161 + 1162 1162 + static void atom_db_iowrite(u64 bits, void __iomem *mmio) 1163 1163 + { 1164 1164 + iowrite64(bits, mmio); 1165 1165 + } 1166 1166 + 1167 1167 + static int atom_poll_link(struct intel_ntb_dev *ndev) 1168 1168 + { 1169 1169 + u32 ntb_ctl; 1170 1170 + 1171 1171 + ntb_ctl = ioread32(ndev->self_mmio + ATOM_NTBCNTL_OFFSET); 1172 1172 + 1173 1173 + if (ntb_ctl == ndev->ntb_ctl) 1174 1174 + return 0; 1175 1175 + 1176 1176 + ndev->ntb_ctl = ntb_ctl; 1177 1177 + 1178 1178 + ndev->lnk_sta = ioread32(ndev->self_mmio + ATOM_LINK_STATUS_OFFSET); 1179 1179 + 1180 1180 + return 1; 1181 1181 + } 1182 1182 + 1183 1183 + static int atom_link_is_up(struct intel_ntb_dev *ndev) 1184 1184 + { 1185 1185 + return ATOM_NTB_CTL_ACTIVE(ndev->ntb_ctl); 1186 1186 + } 1187 1187 + 1188 1188 + static int atom_link_is_err(struct intel_ntb_dev *ndev) 1189 1189 + { 1190 1190 + if (ioread32(ndev->self_mmio + ATOM_LTSSMSTATEJMP_OFFSET) 1191 1191 + & ATOM_LTSSMSTATEJMP_FORCEDETECT) 1192 1192 + return 1; 1193 1193 + 1194 1194 + if (ioread32(ndev->self_mmio + ATOM_IBSTERRRCRVSTS0_OFFSET) 1195 1195 + & ATOM_IBIST_ERR_OFLOW) 1196 1196 + return 1; 1197 1197 + 1198 1198 + return 0; 1199 1199 + } 1200 1200 + 1201 1201 + static inline enum ntb_topo atom_ppd_topo(struct intel_ntb_dev *ndev, u32 ppd) 1202 1202 + { 1203 1203 + switch (ppd & ATOM_PPD_TOPO_MASK) { 1204 1204 + case ATOM_PPD_TOPO_B2B_USD: 1205 1205 + dev_dbg(ndev_dev(ndev), "PPD %d B2B USD\n", ppd); 1206 1206 + return NTB_TOPO_B2B_USD; 1207 1207 + 1208 1208 + case ATOM_PPD_TOPO_B2B_DSD: 1209 1209 + dev_dbg(ndev_dev(ndev), "PPD %d B2B DSD\n", ppd); 1210 1210 + return NTB_TOPO_B2B_DSD; 1211 1211 + 1212 1212 + case ATOM_PPD_TOPO_PRI_USD: 1213 1213 + case ATOM_PPD_TOPO_PRI_DSD: /* accept bogus PRI_DSD */ 1214 1214 + case ATOM_PPD_TOPO_SEC_USD: 1215 1215 + case ATOM_PPD_TOPO_SEC_DSD: /* accept bogus SEC_DSD */ 1216 1216 + dev_dbg(ndev_dev(ndev), "PPD %d non B2B disabled\n", ppd); 1217 1217 + return NTB_TOPO_NONE; 1218 1218 + } 1219 1219 + 1220 1220 + dev_dbg(ndev_dev(ndev), "PPD %d invalid\n", ppd); 1221 1221 + return NTB_TOPO_NONE; 1222 1222 + } 1223 1223 + 1224 1224 + static void atom_link_hb(struct work_struct *work) 1225 1225 + { 1226 1226 + struct intel_ntb_dev *ndev = hb_ndev(work); 1227 1227 + unsigned long poll_ts; 1228 1228 + void __iomem *mmio; 1229 1229 + u32 status32; 1230 1230 + 1231 1231 + poll_ts = ndev->last_ts + ATOM_LINK_HB_TIMEOUT; 1232 1232 + 1233 1233 + /* Delay polling the link status if an interrupt was received, 1234 1234 + * unless the cached link status says the link is down. 1235 1235 + */ 1236 1236 + if (time_after(poll_ts, jiffies) && atom_link_is_up(ndev)) { 1237 1237 + schedule_delayed_work(&ndev->hb_timer, poll_ts - jiffies); 1238 1238 + return; 1239 1239 + } 1240 1240 + 1241 1241 + if (atom_poll_link(ndev)) 1242 1242 + ntb_link_event(&ndev->ntb); 1243 1243 + 1244 1244 + if (atom_link_is_up(ndev) || !atom_link_is_err(ndev)) { 1245 1245 + schedule_delayed_work(&ndev->hb_timer, ATOM_LINK_HB_TIMEOUT); 1246 1246 + return; 1247 1247 + } 1248 1248 + 1249 1249 + /* Link is down with error: recover the link! */ 1250 1250 + 1251 1251 + mmio = ndev->self_mmio; 1252 1252 + 1253 1253 + /* Driver resets the NTB ModPhy lanes - magic! */ 1254 1254 + iowrite8(0xe0, mmio + ATOM_MODPHY_PCSREG6); 1255 1255 + iowrite8(0x40, mmio + ATOM_MODPHY_PCSREG4); 1256 1256 + iowrite8(0x60, mmio + ATOM_MODPHY_PCSREG4); 1257 1257 + iowrite8(0x60, mmio + ATOM_MODPHY_PCSREG6); 1258 1258 + 1259 1259 + /* Driver waits 100ms to allow the NTB ModPhy to settle */ 1260 1260 + msleep(100); 1261 1261 + 1262 1262 + /* Clear AER Errors, write to clear */ 1263 1263 + status32 = ioread32(mmio + ATOM_ERRCORSTS_OFFSET); 1264 1264 + dev_dbg(ndev_dev(ndev), "ERRCORSTS = %x\n", status32); 1265 1265 + status32 &= PCI_ERR_COR_REP_ROLL; 1266 1266 + iowrite32(status32, mmio + ATOM_ERRCORSTS_OFFSET); 1267 1267 + 1268 1268 + /* Clear unexpected electrical idle event in LTSSM, write to clear */ 1269 1269 + status32 = ioread32(mmio + ATOM_LTSSMERRSTS0_OFFSET); 1270 1270 + dev_dbg(ndev_dev(ndev), "LTSSMERRSTS0 = %x\n", status32); 1271 1271 + status32 |= ATOM_LTSSMERRSTS0_UNEXPECTEDEI; 1272 1272 + iowrite32(status32, mmio + ATOM_LTSSMERRSTS0_OFFSET); 1273 1273 + 1274 1274 + /* Clear DeSkew Buffer error, write to clear */ 1275 1275 + status32 = ioread32(mmio + ATOM_DESKEWSTS_OFFSET); 1276 1276 + dev_dbg(ndev_dev(ndev), "DESKEWSTS = %x\n", status32); 1277 1277 + status32 |= ATOM_DESKEWSTS_DBERR; 1278 1278 + iowrite32(status32, mmio + ATOM_DESKEWSTS_OFFSET); 1279 1279 + 1280 1280 + status32 = ioread32(mmio + ATOM_IBSTERRRCRVSTS0_OFFSET); 1281 1281 + dev_dbg(ndev_dev(ndev), "IBSTERRRCRVSTS0 = %x\n", status32); 1282 1282 + status32 &= ATOM_IBIST_ERR_OFLOW; 1283 1283 + iowrite32(status32, mmio + ATOM_IBSTERRRCRVSTS0_OFFSET); 1284 1284 + 1285 1285 + /* Releases the NTB state machine to allow the link to retrain */ 1286 1286 + status32 = ioread32(mmio + ATOM_LTSSMSTATEJMP_OFFSET); 1287 1287 + dev_dbg(ndev_dev(ndev), "LTSSMSTATEJMP = %x\n", status32); 1288 1288 + status32 &= ~ATOM_LTSSMSTATEJMP_FORCEDETECT; 1289 1289 + iowrite32(status32, mmio + ATOM_LTSSMSTATEJMP_OFFSET); 1290 1290 + 1291 1291 + /* There is a potential race between the 2 NTB devices recovering at the 1292 1292 + * same time. If the times are the same, the link will not recover and 1293 1293 + * the driver will be stuck in this loop forever. Add a random interval 1294 1294 + * to the recovery time to prevent this race. 1295 1295 + */ 1296 1296 + schedule_delayed_work(&ndev->hb_timer, ATOM_LINK_RECOVERY_TIME 1297 1297 + + prandom_u32() % ATOM_LINK_RECOVERY_TIME); 1298 1298 + } 1299 1299 + 1300 1300 + static int atom_init_isr(struct intel_ntb_dev *ndev) 1301 1301 + { 1302 1302 + int rc; 1303 1303 + 1304 1304 + rc = ndev_init_isr(ndev, 1, ATOM_DB_MSIX_VECTOR_COUNT, 1305 1305 + ATOM_DB_MSIX_VECTOR_SHIFT, ATOM_DB_TOTAL_SHIFT); 1306 1306 + if (rc) 1307 1307 + return rc; 1308 1308 + 1309 1309 + /* ATOM doesn't have link status interrupt, poll on that platform */ 1310 1310 + ndev->last_ts = jiffies; 1311 1311 + INIT_DELAYED_WORK(&ndev->hb_timer, atom_link_hb); 1312 1312 + schedule_delayed_work(&ndev->hb_timer, ATOM_LINK_HB_TIMEOUT); 1313 1313 + 1314 1314 + return 0; 1315 1315 + } 1316 1316 + 1317 1317 + static void atom_deinit_isr(struct intel_ntb_dev *ndev) 1318 1318 + { 1319 1319 + cancel_delayed_work_sync(&ndev->hb_timer); 1320 1320 + ndev_deinit_isr(ndev); 1321 1321 + } 1322 1322 + 1323 1323 + static int atom_init_ntb(struct intel_ntb_dev *ndev) 1324 1324 + { 1325 1325 + ndev->mw_count = ATOM_MW_COUNT; 1326 1326 + ndev->spad_count = ATOM_SPAD_COUNT; 1327 1327 + ndev->db_count = ATOM_DB_COUNT; 1328 1328 + 1329 1329 + switch (ndev->ntb.topo) { 1330 1330 + case NTB_TOPO_B2B_USD: 1331 1331 + case NTB_TOPO_B2B_DSD: 1332 1332 + ndev->self_reg = &atom_pri_reg; 1333 1333 + ndev->peer_reg = &atom_b2b_reg; 1334 1334 + ndev->xlat_reg = &atom_sec_xlat; 1335 1335 + 1336 1336 + /* Enable Bus Master and Memory Space on the secondary side */ 1337 1337 + iowrite16(PCI_COMMAND_MEMORY | PCI_COMMAND_MASTER, 1338 1338 + ndev->self_mmio + ATOM_SPCICMD_OFFSET); 1339 1339 + 1340 1340 + break; 1341 1341 + 1342 1342 + default: 1343 1343 + return -EINVAL; 1344 1344 + } 1345 1345 + 1346 1346 + ndev->db_valid_mask = BIT_ULL(ndev->db_count) - 1; 1347 1347 + 1348 1348 + return 0; 1349 1349 + } 1350 1350 + 1351 1351 + static int atom_init_dev(struct intel_ntb_dev *ndev) 1352 1352 + { 1353 1353 + u32 ppd; 1354 1354 + int rc; 1355 1355 + 1356 1356 + rc = pci_read_config_dword(ndev->ntb.pdev, ATOM_PPD_OFFSET, &ppd); 1357 1357 + if (rc) 1358 1358 + return -EIO; 1359 1359 + 1360 1360 + ndev->ntb.topo = atom_ppd_topo(ndev, ppd); 1361 1361 + if (ndev->ntb.topo == NTB_TOPO_NONE) 1362 1362 + return -EINVAL; 1363 1363 + 1364 1364 + rc = atom_init_ntb(ndev); 1365 1365 + if (rc) 1366 1366 + return rc; 1367 1367 + 1368 1368 + rc = atom_init_isr(ndev); 1369 1369 + if (rc) 1370 1370 + return rc; 1371 1371 + 1372 1372 + if (ndev->ntb.topo != NTB_TOPO_SEC) { 1373 1373 + /* Initiate PCI-E link training */ 1374 1374 + rc = pci_write_config_dword(ndev->ntb.pdev, ATOM_PPD_OFFSET, 1375 1375 + ppd | ATOM_PPD_INIT_LINK); 1376 1376 + if (rc) 1377 1377 + return rc; 1378 1378 + } 1379 1379 + 1380 1380 + return 0; 1381 1381 + } 1382 1382 + 1383 1383 + static void atom_deinit_dev(struct intel_ntb_dev *ndev) 1384 1384 + { 1385 1385 + atom_deinit_isr(ndev); 1386 1386 + } 1387 1387 + 1388 1388 + /* XEON */ 1389 1389 + 1390 1390 + static u64 xeon_db_ioread(void __iomem *mmio) 1391 1391 + { 1392 1392 + return (u64)ioread16(mmio); 1393 1393 + } 1394 1394 + 1395 1395 + static void xeon_db_iowrite(u64 bits, void __iomem *mmio) 1396 1396 + { 1397 1397 + iowrite16((u16)bits, mmio); 1398 1398 + } 1399 1399 + 1400 1400 + static int xeon_poll_link(struct intel_ntb_dev *ndev) 1401 1401 + { 1402 1402 + u16 reg_val; 1403 1403 + int rc; 1404 1404 + 1405 1405 + ndev->reg->db_iowrite(ndev->db_link_mask, 1406 1406 + ndev->self_mmio + 1407 1407 + ndev->self_reg->db_bell); 1408 1408 + 1409 1409 + rc = pci_read_config_word(ndev->ntb.pdev, 1410 1410 + XEON_LINK_STATUS_OFFSET, &reg_val); 1411 1411 + if (rc) 1412 1412 + return 0; 1413 1413 + 1414 1414 + if (reg_val == ndev->lnk_sta) 1415 1415 + return 0; 1416 1416 + 1417 1417 + ndev->lnk_sta = reg_val; 1418 1418 + 1419 1419 + return 1; 1420 1420 + } 1421 1421 + 1422 1422 + static int xeon_link_is_up(struct intel_ntb_dev *ndev) 1423 1423 + { 1424 1424 + if (ndev->ntb.topo == NTB_TOPO_SEC) 1425 1425 + return 1; 1426 1426 + 1427 1427 + return NTB_LNK_STA_ACTIVE(ndev->lnk_sta); 1428 1428 + } 1429 1429 + 1430 1430 + static inline enum ntb_topo xeon_ppd_topo(struct intel_ntb_dev *ndev, u8 ppd) 1431 1431 + { 1432 1432 + switch (ppd & XEON_PPD_TOPO_MASK) { 1433 1433 + case XEON_PPD_TOPO_B2B_USD: 1434 1434 + return NTB_TOPO_B2B_USD; 1435 1435 + 1436 1436 + case XEON_PPD_TOPO_B2B_DSD: 1437 1437 + return NTB_TOPO_B2B_DSD; 1438 1438 + 1439 1439 + case XEON_PPD_TOPO_PRI_USD: 1440 1440 + case XEON_PPD_TOPO_PRI_DSD: /* accept bogus PRI_DSD */ 1441 1441 + return NTB_TOPO_PRI; 1442 1442 + 1443 1443 + case XEON_PPD_TOPO_SEC_USD: 1444 1444 + case XEON_PPD_TOPO_SEC_DSD: /* accept bogus SEC_DSD */ 1445 1445 + return NTB_TOPO_SEC; 1446 1446 + } 1447 1447 + 1448 1448 + return NTB_TOPO_NONE; 1449 1449 + } 1450 1450 + 1451 1451 + static inline int xeon_ppd_bar4_split(struct intel_ntb_dev *ndev, u8 ppd) 1452 1452 + { 1453 1453 + if (ppd & XEON_PPD_SPLIT_BAR_MASK) { 1454 1454 + dev_dbg(ndev_dev(ndev), "PPD %d split bar\n", ppd); 1455 1455 + return 1; 1456 1456 + } 1457 1457 + return 0; 1458 1458 + } 1459 1459 + 1460 1460 + static int xeon_init_isr(struct intel_ntb_dev *ndev) 1461 1461 + { 1462 1462 + return ndev_init_isr(ndev, XEON_DB_MSIX_VECTOR_COUNT, 1463 1463 + XEON_DB_MSIX_VECTOR_COUNT, 1464 1464 + XEON_DB_MSIX_VECTOR_SHIFT, 1465 1465 + XEON_DB_TOTAL_SHIFT); 1466 1466 + } 1467 1467 + 1468 1468 + static void xeon_deinit_isr(struct intel_ntb_dev *ndev) 1469 1469 + { 1470 1470 + ndev_deinit_isr(ndev); 1471 1471 + } 1472 1472 + 1473 1473 + static int xeon_setup_b2b_mw(struct intel_ntb_dev *ndev, 1474 1474 + const struct intel_b2b_addr *addr, 1475 1475 + const struct intel_b2b_addr *peer_addr) 1476 1476 + { 1477 1477 + struct pci_dev *pdev; 1478 1478 + void __iomem *mmio; 1479 1479 + resource_size_t bar_size; 1480 1480 + phys_addr_t bar_addr; 1481 1481 + int b2b_bar; 1482 1482 + u8 bar_sz; 1483 1483 + 1484 1484 + pdev = ndev_pdev(ndev); 1485 1485 + mmio = ndev->self_mmio; 1486 1486 + 1487 1487 + if (ndev->b2b_idx >= ndev->mw_count) { 1488 1488 + dev_dbg(ndev_dev(ndev), "not using b2b mw\n"); 1489 1489 + b2b_bar = 0; 1490 1490 + ndev->b2b_off = 0; 1491 1491 + } else { 1492 1492 + b2b_bar = ndev_mw_to_bar(ndev, ndev->b2b_idx); 1493 1493 + if (b2b_bar < 0) 1494 1494 + return -EIO; 1495 1495 + 1496 1496 + dev_dbg(ndev_dev(ndev), "using b2b mw bar %d\n", b2b_bar); 1497 1497 + 1498 1498 + bar_size = pci_resource_len(ndev->ntb.pdev, b2b_bar); 1499 1499 + 1500 1500 + dev_dbg(ndev_dev(ndev), "b2b bar size %#llx\n", bar_size); 1501 1501 + 1502 1502 + if (b2b_mw_share && XEON_B2B_MIN_SIZE <= bar_size >> 1) { 1503 1503 + dev_dbg(ndev_dev(ndev), 1504 1504 + "b2b using first half of bar\n"); 1505 1505 + ndev->b2b_off = bar_size >> 1; 1506 1506 + } else if (XEON_B2B_MIN_SIZE <= bar_size) { 1507 1507 + dev_dbg(ndev_dev(ndev), 1508 1508 + "b2b using whole bar\n"); 1509 1509 + ndev->b2b_off = 0; 1510 1510 + --ndev->mw_count; 1511 1511 + } else { 1512 1512 + dev_dbg(ndev_dev(ndev), 1513 1513 + "b2b bar size is too small\n"); 1514 1514 + return -EIO; 1515 1515 + } 1516 1516 + } 1517 1517 + 1518 1518 + /* Reset the secondary bar sizes to match the primary bar sizes, 1519 1519 + * except disable or halve the size of the b2b secondary bar. 1520 1520 + * 1521 1521 + * Note: code for each specific bar size register, because the register 1522 1522 + * offsets are not in a consistent order (bar5sz comes after ppd, odd). 1523 1523 + */ 1524 1524 + pci_read_config_byte(pdev, XEON_PBAR23SZ_OFFSET, &bar_sz); 1525 1525 + dev_dbg(ndev_dev(ndev), "PBAR23SZ %#x\n", bar_sz); 1526 1526 + if (b2b_bar == 2) { 1527 1527 + if (ndev->b2b_off) 1528 1528 + bar_sz -= 1; 1529 1529 + else 1530 1530 + bar_sz = 0; 1531 1531 + } 1532 1532 + pci_write_config_byte(pdev, XEON_SBAR23SZ_OFFSET, bar_sz); 1533 1533 + pci_read_config_byte(pdev, XEON_SBAR23SZ_OFFSET, &bar_sz); 1534 1534 + dev_dbg(ndev_dev(ndev), "SBAR23SZ %#x\n", bar_sz); 1535 1535 + 1536 1536 + if (!ndev->bar4_split) { 1537 1537 + pci_read_config_byte(pdev, XEON_PBAR45SZ_OFFSET, &bar_sz); 1538 1538 + dev_dbg(ndev_dev(ndev), "PBAR45SZ %#x\n", bar_sz); 1539 1539 + if (b2b_bar == 4) { 1540 1540 + if (ndev->b2b_off) 1541 1541 + bar_sz -= 1; 1542 1542 + else 1543 1543 + bar_sz = 0; 1544 1544 + } 1545 1545 + pci_write_config_byte(pdev, XEON_SBAR45SZ_OFFSET, bar_sz); 1546 1546 + pci_read_config_byte(pdev, XEON_SBAR45SZ_OFFSET, &bar_sz); 1547 1547 + dev_dbg(ndev_dev(ndev), "SBAR45SZ %#x\n", bar_sz); 1548 1548 + } else { 1549 1549 + pci_read_config_byte(pdev, XEON_PBAR4SZ_OFFSET, &bar_sz); 1550 1550 + dev_dbg(ndev_dev(ndev), "PBAR4SZ %#x\n", bar_sz); 1551 1551 + if (b2b_bar == 4) { 1552 1552 + if (ndev->b2b_off) 1553 1553 + bar_sz -= 1; 1554 1554 + else 1555 1555 + bar_sz = 0; 1556 1556 + } 1557 1557 + pci_write_config_byte(pdev, XEON_SBAR4SZ_OFFSET, bar_sz); 1558 1558 + pci_read_config_byte(pdev, XEON_SBAR4SZ_OFFSET, &bar_sz); 1559 1559 + dev_dbg(ndev_dev(ndev), "SBAR4SZ %#x\n", bar_sz); 1560 1560 + 1561 1561 + pci_read_config_byte(pdev, XEON_PBAR5SZ_OFFSET, &bar_sz); 1562 1562 + dev_dbg(ndev_dev(ndev), "PBAR5SZ %#x\n", bar_sz); 1563 1563 + if (b2b_bar == 5) { 1564 1564 + if (ndev->b2b_off) 1565 1565 + bar_sz -= 1; 1566 1566 + else 1567 1567 + bar_sz = 0; 1568 1568 + } 1569 1569 + pci_write_config_byte(pdev, XEON_SBAR5SZ_OFFSET, bar_sz); 1570 1570 + pci_read_config_byte(pdev, XEON_SBAR5SZ_OFFSET, &bar_sz); 1571 1571 + dev_dbg(ndev_dev(ndev), "SBAR5SZ %#x\n", bar_sz); 1572 1572 + } 1573 1573 + 1574 1574 + /* SBAR01 hit by first part of the b2b bar */ 1575 1575 + if (b2b_bar == 0) 1576 1576 + bar_addr = addr->bar0_addr; 1577 1577 + else if (b2b_bar == 2) 1578 1578 + bar_addr = addr->bar2_addr64; 1579 1579 + else if (b2b_bar == 4 && !ndev->bar4_split) 1580 1580 + bar_addr = addr->bar4_addr64; 1581 1581 + else if (b2b_bar == 4) 1582 1582 + bar_addr = addr->bar4_addr32; 1583 1583 + else if (b2b_bar == 5) 1584 1584 + bar_addr = addr->bar5_addr32; 1585 1585 + else 1586 1586 + return -EIO; 1587 1587 + 1588 1588 + dev_dbg(ndev_dev(ndev), "SBAR01 %#018llx\n", bar_addr); 1589 1589 + iowrite64(bar_addr, mmio + XEON_SBAR0BASE_OFFSET); 1590 1590 + 1591 1591 + /* Other SBAR are normally hit by the PBAR xlat, except for b2b bar. 1592 1592 + * The b2b bar is either disabled above, or configured half-size, and 1593 1593 + * it starts at the PBAR xlat + offset. 1594 1594 + */ 1595 1595 + 1596 1596 + bar_addr = addr->bar2_addr64 + (b2b_bar == 2 ? ndev->b2b_off : 0); 1597 1597 + iowrite64(bar_addr, mmio + XEON_SBAR23BASE_OFFSET); 1598 1598 + bar_addr = ioread64(mmio + XEON_SBAR23BASE_OFFSET); 1599 1599 + dev_dbg(ndev_dev(ndev), "SBAR23 %#018llx\n", bar_addr); 1600 1600 + 1601 1601 + if (!ndev->bar4_split) { 1602 1602 + bar_addr = addr->bar4_addr64 + 1603 1603 + (b2b_bar == 4 ? ndev->b2b_off : 0); 1604 1604 + iowrite64(bar_addr, mmio + XEON_SBAR45BASE_OFFSET); 1605 1605 + bar_addr = ioread64(mmio + XEON_SBAR45BASE_OFFSET); 1606 1606 + dev_dbg(ndev_dev(ndev), "SBAR45 %#018llx\n", bar_addr); 1607 1607 + } else { 1608 1608 + bar_addr = addr->bar4_addr32 + 1609 1609 + (b2b_bar == 4 ? ndev->b2b_off : 0); 1610 1610 + iowrite32(bar_addr, mmio + XEON_SBAR4BASE_OFFSET); 1611 1611 + bar_addr = ioread32(mmio + XEON_SBAR4BASE_OFFSET); 1612 1612 + dev_dbg(ndev_dev(ndev), "SBAR4 %#010llx\n", bar_addr); 1613 1613 + 1614 1614 + bar_addr = addr->bar5_addr32 + 1615 1615 + (b2b_bar == 5 ? ndev->b2b_off : 0); 1616 1616 + iowrite32(bar_addr, mmio + XEON_SBAR5BASE_OFFSET); 1617 1617 + bar_addr = ioread32(mmio + XEON_SBAR5BASE_OFFSET); 1618 1618 + dev_dbg(ndev_dev(ndev), "SBAR5 %#010llx\n", bar_addr); 1619 1619 + } 1620 1620 + 1621 1621 + /* setup incoming bar limits == base addrs (zero length windows) */ 1622 1622 + 1623 1623 + bar_addr = addr->bar2_addr64 + (b2b_bar == 2 ? ndev->b2b_off : 0); 1624 1624 + iowrite64(bar_addr, mmio + XEON_SBAR23LMT_OFFSET); 1625 1625 + bar_addr = ioread64(mmio + XEON_SBAR23LMT_OFFSET); 1626 1626 + dev_dbg(ndev_dev(ndev), "SBAR23LMT %#018llx\n", bar_addr); 1627 1627 + 1628 1628 + if (!ndev->bar4_split) { 1629 1629 + bar_addr = addr->bar4_addr64 + 1630 1630 + (b2b_bar == 4 ? ndev->b2b_off : 0); 1631 1631 + iowrite64(bar_addr, mmio + XEON_SBAR45LMT_OFFSET); 1632 1632 + bar_addr = ioread64(mmio + XEON_SBAR45LMT_OFFSET); 1633 1633 + dev_dbg(ndev_dev(ndev), "SBAR45LMT %#018llx\n", bar_addr); 1634 1634 + } else { 1635 1635 + bar_addr = addr->bar4_addr32 + 1636 1636 + (b2b_bar == 4 ? ndev->b2b_off : 0); 1637 1637 + iowrite32(bar_addr, mmio + XEON_SBAR4LMT_OFFSET); 1638 1638 + bar_addr = ioread32(mmio + XEON_SBAR4LMT_OFFSET); 1639 1639 + dev_dbg(ndev_dev(ndev), "SBAR4LMT %#010llx\n", bar_addr); 1640 1640 + 1641 1641 + bar_addr = addr->bar5_addr32 + 1642 1642 + (b2b_bar == 5 ? ndev->b2b_off : 0); 1643 1643 + iowrite32(bar_addr, mmio + XEON_SBAR5LMT_OFFSET); 1644 1644 + bar_addr = ioread32(mmio + XEON_SBAR5LMT_OFFSET); 1645 1645 + dev_dbg(ndev_dev(ndev), "SBAR5LMT %#05llx\n", bar_addr); 1646 1646 + } 1647 1647 + 1648 1648 + /* zero incoming translation addrs */ 1649 1649 + iowrite64(0, mmio + XEON_SBAR23XLAT_OFFSET); 1650 1650 + 1651 1651 + if (!ndev->bar4_split) { 1652 1652 + iowrite64(0, mmio + XEON_SBAR45XLAT_OFFSET); 1653 1653 + } else { 1654 1654 + iowrite32(0, mmio + XEON_SBAR4XLAT_OFFSET); 1655 1655 + iowrite32(0, mmio + XEON_SBAR5XLAT_OFFSET); 1656 1656 + } 1657 1657 + 1658 1658 + /* zero outgoing translation limits (whole bar size windows) */ 1659 1659 + iowrite64(0, mmio + XEON_PBAR23LMT_OFFSET); 1660 1660 + if (!ndev->bar4_split) { 1661 1661 + iowrite64(0, mmio + XEON_PBAR45LMT_OFFSET); 1662 1662 + } else { 1663 1663 + iowrite32(0, mmio + XEON_PBAR4LMT_OFFSET); 1664 1664 + iowrite32(0, mmio + XEON_PBAR5LMT_OFFSET); 1665 1665 + } 1666 1666 + 1667 1667 + /* set outgoing translation offsets */ 1668 1668 + bar_addr = peer_addr->bar2_addr64; 1669 1669 + iowrite64(bar_addr, mmio + XEON_PBAR23XLAT_OFFSET); 1670 1670 + bar_addr = ioread64(mmio + XEON_PBAR23XLAT_OFFSET); 1671 1671 + dev_dbg(ndev_dev(ndev), "PBAR23XLAT %#018llx\n", bar_addr); 1672 1672 + 1673 1673 + if (!ndev->bar4_split) { 1674 1674 + bar_addr = peer_addr->bar4_addr64; 1675 1675 + iowrite64(bar_addr, mmio + XEON_PBAR45XLAT_OFFSET); 1676 1676 + bar_addr = ioread64(mmio + XEON_PBAR45XLAT_OFFSET); 1677 1677 + dev_dbg(ndev_dev(ndev), "PBAR45XLAT %#018llx\n", bar_addr); 1678 1678 + } else { 1679 1679 + bar_addr = peer_addr->bar4_addr32; 1680 1680 + iowrite32(bar_addr, mmio + XEON_PBAR4XLAT_OFFSET); 1681 1681 + bar_addr = ioread32(mmio + XEON_PBAR4XLAT_OFFSET); 1682 1682 + dev_dbg(ndev_dev(ndev), "PBAR4XLAT %#010llx\n", bar_addr); 1683 1683 + 1684 1684 + bar_addr = peer_addr->bar5_addr32; 1685 1685 + iowrite32(bar_addr, mmio + XEON_PBAR5XLAT_OFFSET); 1686 1686 + bar_addr = ioread32(mmio + XEON_PBAR5XLAT_OFFSET); 1687 1687 + dev_dbg(ndev_dev(ndev), "PBAR5XLAT %#010llx\n", bar_addr); 1688 1688 + } 1689 1689 + 1690 1690 + /* set the translation offset for b2b registers */ 1691 1691 + if (b2b_bar == 0) 1692 1692 + bar_addr = peer_addr->bar0_addr; 1693 1693 + else if (b2b_bar == 2) 1694 1694 + bar_addr = peer_addr->bar2_addr64; 1695 1695 + else if (b2b_bar == 4 && !ndev->bar4_split) 1696 1696 + bar_addr = peer_addr->bar4_addr64; 1697 1697 + else if (b2b_bar == 4) 1698 1698 + bar_addr = peer_addr->bar4_addr32; 1699 1699 + else if (b2b_bar == 5) 1700 1700 + bar_addr = peer_addr->bar5_addr32; 1701 1701 + else 1702 1702 + return -EIO; 1703 1703 + 1704 1704 + /* B2B_XLAT_OFFSET is 64bit, but can only take 32bit writes */ 1705 1705 + dev_dbg(ndev_dev(ndev), "B2BXLAT %#018llx\n", bar_addr); 1706 1706 + iowrite32(bar_addr, mmio + XEON_B2B_XLAT_OFFSETL); 1707 1707 + iowrite32(bar_addr >> 32, mmio + XEON_B2B_XLAT_OFFSETU); 1708 1708 + 1709 1709 + if (b2b_bar) { 1710 1710 + /* map peer ntb mmio config space registers */ 1711 1711 + ndev->peer_mmio = pci_iomap(pdev, b2b_bar, 1712 1712 + XEON_B2B_MIN_SIZE); 1713 1713 + if (!ndev->peer_mmio) 1714 1714 + return -EIO; 1715 1715 + } 1716 1716 + 1717 1717 + return 0; 1718 1718 + } 1719 1719 + 1720 1720 + static int xeon_init_ntb(struct intel_ntb_dev *ndev) 1721 1721 + { 1722 1722 + int rc; 1723 1723 + u32 ntb_ctl; 1724 1724 + 1725 1725 + if (ndev->bar4_split) 1726 1726 + ndev->mw_count = HSX_SPLIT_BAR_MW_COUNT; 1727 1727 + else 1728 1728 + ndev->mw_count = XEON_MW_COUNT; 1729 1729 + 1730 1730 + ndev->spad_count = XEON_SPAD_COUNT; 1731 1731 + ndev->db_count = XEON_DB_COUNT; 1732 1732 + ndev->db_link_mask = XEON_DB_LINK_BIT; 1733 1733 + 1734 1734 + switch (ndev->ntb.topo) { 1735 1735 + case NTB_TOPO_PRI: 1736 1736 + if (ndev->hwerr_flags & NTB_HWERR_SDOORBELL_LOCKUP) { 1737 1737 + dev_err(ndev_dev(ndev), "NTB Primary config disabled\n"); 1738 1738 + return -EINVAL; 1739 1739 + } 1740 1740 + 1741 1741 + /* enable link to allow secondary side device to appear */ 1742 1742 + ntb_ctl = ioread32(ndev->self_mmio + ndev->reg->ntb_ctl); 1743 1743 + ntb_ctl &= ~NTB_CTL_DISABLE; 1744 1744 + iowrite32(ntb_ctl, ndev->self_mmio + ndev->reg->ntb_ctl); 1745 1745 + 1746 1746 + /* use half the spads for the peer */ 1747 1747 + ndev->spad_count >>= 1; 1748 1748 + ndev->self_reg = &xeon_pri_reg; 1749 1749 + ndev->peer_reg = &xeon_sec_reg; 1750 1750 + ndev->xlat_reg = &xeon_sec_xlat; 1751 1751 + break; 1752 1752 + 1753 1753 + case NTB_TOPO_SEC: 1754 1754 + if (ndev->hwerr_flags & NTB_HWERR_SDOORBELL_LOCKUP) { 1755 1755 + dev_err(ndev_dev(ndev), "NTB Secondary config disabled\n"); 1756 1756 + return -EINVAL; 1757 1757 + } 1758 1758 + /* use half the spads for the peer */ 1759 1759 + ndev->spad_count >>= 1; 1760 1760 + ndev->self_reg = &xeon_sec_reg; 1761 1761 + ndev->peer_reg = &xeon_pri_reg; 1762 1762 + ndev->xlat_reg = &xeon_pri_xlat; 1763 1763 + break; 1764 1764 + 1765 1765 + case NTB_TOPO_B2B_USD: 1766 1766 + case NTB_TOPO_B2B_DSD: 1767 1767 + ndev->self_reg = &xeon_pri_reg; 1768 1768 + ndev->peer_reg = &xeon_b2b_reg; 1769 1769 + ndev->xlat_reg = &xeon_sec_xlat; 1770 1770 + 1771 1771 + if (ndev->hwerr_flags & NTB_HWERR_SDOORBELL_LOCKUP) { 1772 1772 + ndev->peer_reg = &xeon_pri_reg; 1773 1773 + 1774 1774 + if (b2b_mw_idx < 0) 1775 1775 + ndev->b2b_idx = b2b_mw_idx + ndev->mw_count; 1776 1776 + else 1777 1777 + ndev->b2b_idx = b2b_mw_idx; 1778 1778 + 1779 1779 + dev_dbg(ndev_dev(ndev), 1780 1780 + "setting up b2b mw idx %d means %d\n", 1781 1781 + b2b_mw_idx, ndev->b2b_idx); 1782 1782 + 1783 1783 + } else if (ndev->hwerr_flags & NTB_HWERR_B2BDOORBELL_BIT14) { 1784 1784 + dev_warn(ndev_dev(ndev), "Reduce doorbell count by 1\n"); 1785 1785 + ndev->db_count -= 1; 1786 1786 + } 1787 1787 + 1788 1788 + if (ndev->ntb.topo == NTB_TOPO_B2B_USD) { 1789 1789 + rc = xeon_setup_b2b_mw(ndev, 1790 1790 + &xeon_b2b_dsd_addr, 1791 1791 + &xeon_b2b_usd_addr); 1792 1792 + } else { 1793 1793 + rc = xeon_setup_b2b_mw(ndev, 1794 1794 + &xeon_b2b_usd_addr, 1795 1795 + &xeon_b2b_dsd_addr); 1796 1796 + } 1797 1797 + if (rc) 1798 1798 + return rc; 1799 1799 + 1800 1800 + /* Enable Bus Master and Memory Space on the secondary side */ 1801 1801 + iowrite16(PCI_COMMAND_MEMORY | PCI_COMMAND_MASTER, 1802 1802 + ndev->self_mmio + XEON_SPCICMD_OFFSET); 1803 1803 + 1804 1804 + break; 1805 1805 + 1806 1806 + default: 1807 1807 + return -EINVAL; 1808 1808 + } 1809 1809 + 1810 1810 + ndev->db_valid_mask = BIT_ULL(ndev->db_count) - 1; 1811 1811 + 1812 1812 + ndev->reg->db_iowrite(ndev->db_valid_mask, 1813 1813 + ndev->self_mmio + 1814 1814 + ndev->self_reg->db_mask); 1815 1815 + 1816 1816 + return 0; 1817 1817 + } 1818 1818 + 1819 1819 + static int xeon_init_dev(struct intel_ntb_dev *ndev) 1820 1820 + { 1821 1821 + struct pci_dev *pdev; 1822 1822 + u8 ppd; 1823 1823 + int rc, mem; 1824 1824 + 1825 1825 + pdev = ndev_pdev(ndev); 1826 1826 + 1827 1827 + switch (pdev->device) { 1828 1828 + /* There is a Xeon hardware errata related to writes to SDOORBELL or 1829 1829 + * B2BDOORBELL in conjunction with inbound access to NTB MMIO Space, 1830 1830 + * which may hang the system. To workaround this use the second memory 1831 1831 + * window to access the interrupt and scratch pad registers on the 1832 1832 + * remote system. 1833 1833 + */ 1834 1834 + case PCI_DEVICE_ID_INTEL_NTB_SS_JSF: 1835 1835 + case PCI_DEVICE_ID_INTEL_NTB_PS_JSF: 1836 1836 + case PCI_DEVICE_ID_INTEL_NTB_B2B_JSF: 1837 1837 + case PCI_DEVICE_ID_INTEL_NTB_SS_SNB: 1838 1838 + case PCI_DEVICE_ID_INTEL_NTB_PS_SNB: 1839 1839 + case PCI_DEVICE_ID_INTEL_NTB_B2B_SNB: 1840 1840 + case PCI_DEVICE_ID_INTEL_NTB_SS_IVT: 1841 1841 + case PCI_DEVICE_ID_INTEL_NTB_PS_IVT: 1842 1842 + case PCI_DEVICE_ID_INTEL_NTB_B2B_IVT: 1843 1843 + case PCI_DEVICE_ID_INTEL_NTB_SS_HSX: 1844 1844 + case PCI_DEVICE_ID_INTEL_NTB_PS_HSX: 1845 1845 + case PCI_DEVICE_ID_INTEL_NTB_B2B_HSX: 1846 1846 + ndev->hwerr_flags |= NTB_HWERR_SDOORBELL_LOCKUP; 1847 1847 + break; 1848 1848 + } 1849 1849 + 1850 1850 + switch (pdev->device) { 1851 1851 + /* There is a hardware errata related to accessing any register in 1852 1852 + * SB01BASE in the presence of bidirectional traffic crossing the NTB. 1853 1853 + */ 1854 1854 + case PCI_DEVICE_ID_INTEL_NTB_SS_IVT: 1855 1855 + case PCI_DEVICE_ID_INTEL_NTB_PS_IVT: 1856 1856 + case PCI_DEVICE_ID_INTEL_NTB_B2B_IVT: 1857 1857 + case PCI_DEVICE_ID_INTEL_NTB_SS_HSX: 1858 1858 + case PCI_DEVICE_ID_INTEL_NTB_PS_HSX: 1859 1859 + case PCI_DEVICE_ID_INTEL_NTB_B2B_HSX: 1860 1860 + ndev->hwerr_flags |= NTB_HWERR_SB01BASE_LOCKUP; 1861 1861 + break; 1862 1862 + } 1863 1863 + 1864 1864 + switch (pdev->device) { 1865 1865 + /* HW Errata on bit 14 of b2bdoorbell register. Writes will not be 1866 1866 + * mirrored to the remote system. Shrink the number of bits by one, 1867 1867 + * since bit 14 is the last bit. 1868 1868 + */ 1869 1869 + case PCI_DEVICE_ID_INTEL_NTB_SS_JSF: 1870 1870 + case PCI_DEVICE_ID_INTEL_NTB_PS_JSF: 1871 1871 + case PCI_DEVICE_ID_INTEL_NTB_B2B_JSF: 1872 1872 + case PCI_DEVICE_ID_INTEL_NTB_SS_SNB: 1873 1873 + case PCI_DEVICE_ID_INTEL_NTB_PS_SNB: 1874 1874 + case PCI_DEVICE_ID_INTEL_NTB_B2B_SNB: 1875 1875 + case PCI_DEVICE_ID_INTEL_NTB_SS_IVT: 1876 1876 + case PCI_DEVICE_ID_INTEL_NTB_PS_IVT: 1877 1877 + case PCI_DEVICE_ID_INTEL_NTB_B2B_IVT: 1878 1878 + case PCI_DEVICE_ID_INTEL_NTB_SS_HSX: 1879 1879 + case PCI_DEVICE_ID_INTEL_NTB_PS_HSX: 1880 1880 + case PCI_DEVICE_ID_INTEL_NTB_B2B_HSX: 1881 1881 + ndev->hwerr_flags |= NTB_HWERR_B2BDOORBELL_BIT14; 1882 1882 + break; 1883 1883 + } 1884 1884 + 1885 1885 + ndev->reg = &xeon_reg; 1886 1886 + 1887 1887 + rc = pci_read_config_byte(pdev, XEON_PPD_OFFSET, &ppd); 1888 1888 + if (rc) 1889 1889 + return -EIO; 1890 1890 + 1891 1891 + ndev->ntb.topo = xeon_ppd_topo(ndev, ppd); 1892 1892 + dev_dbg(ndev_dev(ndev), "ppd %#x topo %s\n", ppd, 1893 1893 + ntb_topo_string(ndev->ntb.topo)); 1894 1894 + if (ndev->ntb.topo == NTB_TOPO_NONE) 1895 1895 + return -EINVAL; 1896 1896 + 1897 1897 + if (ndev->ntb.topo != NTB_TOPO_SEC) { 1898 1898 + ndev->bar4_split = xeon_ppd_bar4_split(ndev, ppd); 1899 1899 + dev_dbg(ndev_dev(ndev), "ppd %#x bar4_split %d\n", 1900 1900 + ppd, ndev->bar4_split); 1901 1901 + } else { 1902 1902 + /* This is a way for transparent BAR to figure out if we are 1903 1903 + * doing split BAR or not. There is no way for the hw on the 1904 1904 + * transparent side to know and set the PPD. 1905 1905 + */ 1906 1906 + mem = pci_select_bars(pdev, IORESOURCE_MEM); 1907 1907 + ndev->bar4_split = hweight32(mem) == 1908 1908 + HSX_SPLIT_BAR_MW_COUNT + 1; 1909 1909 + dev_dbg(ndev_dev(ndev), "mem %#x bar4_split %d\n", 1910 1910 + mem, ndev->bar4_split); 1911 1911 + } 1912 1912 + 1913 1913 + rc = xeon_init_ntb(ndev); 1914 1914 + if (rc) 1915 1915 + return rc; 1916 1916 + 1917 1917 + return xeon_init_isr(ndev); 1918 1918 + } 1919 1919 + 1920 1920 + static void xeon_deinit_dev(struct intel_ntb_dev *ndev) 1921 1921 + { 1922 1922 + xeon_deinit_isr(ndev); 1923 1923 + } 1924 1924 + 1925 1925 + static int intel_ntb_init_pci(struct intel_ntb_dev *ndev, struct pci_dev *pdev) 1926 1926 + { 1927 1927 + int rc; 1928 1928 + 1929 1929 + pci_set_drvdata(pdev, ndev); 1930 1930 + 1931 1931 + rc = pci_enable_device(pdev); 1932 1932 + if (rc) 1933 1933 + goto err_pci_enable; 1934 1934 + 1935 1935 + rc = pci_request_regions(pdev, NTB_NAME); 1936 1936 + if (rc) 1937 1937 + goto err_pci_regions; 1938 1938 + 1939 1939 + pci_set_master(pdev); 1940 1940 + 1941 1941 + rc = pci_set_dma_mask(pdev, DMA_BIT_MASK(64)); 1942 1942 + if (rc) { 1943 1943 + rc = pci_set_dma_mask(pdev, DMA_BIT_MASK(32)); 1944 1944 + if (rc) 1945 1945 + goto err_dma_mask; 1946 1946 + dev_warn(ndev_dev(ndev), "Cannot DMA highmem\n"); 1947 1947 + } 1948 1948 + 1949 1949 + rc = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(64)); 1950 1950 + if (rc) { 1951 1951 + rc = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(32)); 1952 1952 + if (rc) 1953 1953 + goto err_dma_mask; 1954 1954 + dev_warn(ndev_dev(ndev), "Cannot DMA consistent highmem\n"); 1955 1955 + } 1956 1956 + 1957 1957 + ndev->self_mmio = pci_iomap(pdev, 0, 0); 1958 1958 + if (!ndev->self_mmio) { 1959 1959 + rc = -EIO; 1960 1960 + goto err_mmio; 1961 1961 + } 1962 1962 + ndev->peer_mmio = ndev->self_mmio; 1963 1963 + 1964 1964 + return 0; 1965 1965 + 1966 1966 + err_mmio: 1967 1967 + err_dma_mask: 1968 1968 + pci_clear_master(pdev); 1969 1969 + pci_release_regions(pdev); 1970 1970 + err_pci_regions: 1971 1971 + pci_disable_device(pdev); 1972 1972 + err_pci_enable: 1973 1973 + pci_set_drvdata(pdev, NULL); 1974 1974 + return rc; 1975 1975 + } 1976 1976 + 1977 1977 + static void intel_ntb_deinit_pci(struct intel_ntb_dev *ndev) 1978 1978 + { 1979 1979 + struct pci_dev *pdev = ndev_pdev(ndev); 1980 1980 + 1981 1981 + if (ndev->peer_mmio && ndev->peer_mmio != ndev->self_mmio) 1982 1982 + pci_iounmap(pdev, ndev->peer_mmio); 1983 1983 + pci_iounmap(pdev, ndev->self_mmio); 1984 1984 + 1985 1985 + pci_clear_master(pdev); 1986 1986 + pci_release_regions(pdev); 1987 1987 + pci_disable_device(pdev); 1988 1988 + pci_set_drvdata(pdev, NULL); 1989 1989 + } 1990 1990 + 1991 1991 + static inline void ndev_init_struct(struct intel_ntb_dev *ndev, 1992 1992 + struct pci_dev *pdev) 1993 1993 + { 1994 1994 + ndev->ntb.pdev = pdev; 1995 1995 + ndev->ntb.topo = NTB_TOPO_NONE; 1996 1996 + ndev->ntb.ops = &intel_ntb_ops; 1997 1997 + 1998 1998 + ndev->b2b_off = 0; 1999 1999 + ndev->b2b_idx = INT_MAX; 2000 2000 + 2001 2001 + ndev->bar4_split = 0; 2002 2002 + 2003 2003 + ndev->mw_count = 0; 2004 2004 + ndev->spad_count = 0; 2005 2005 + ndev->db_count = 0; 2006 2006 + ndev->db_vec_count = 0; 2007 2007 + ndev->db_vec_shift = 0; 2008 2008 + 2009 2009 + ndev->ntb_ctl = 0; 2010 2010 + ndev->lnk_sta = 0; 2011 2011 + 2012 2012 + ndev->db_valid_mask = 0; 2013 2013 + ndev->db_link_mask = 0; 2014 2014 + ndev->db_mask = 0; 2015 2015 + 2016 2016 + spin_lock_init(&ndev->db_mask_lock); 2017 2017 + } 2018 2018 + 2019 2019 + static int intel_ntb_pci_probe(struct pci_dev *pdev, 2020 2020 + const struct pci_device_id *id) 2021 2021 + { 2022 2022 + struct intel_ntb_dev *ndev; 2023 2023 + int rc, node; 2024 2024 + 2025 2025 + node = dev_to_node(&pdev->dev); 2026 2026 + 2027 2027 + if (pdev_is_atom(pdev)) { 2028 2028 + ndev = kzalloc_node(sizeof(*ndev), GFP_KERNEL, node); 2029 2029 + if (!ndev) { 2030 2030 + rc = -ENOMEM; 2031 2031 + goto err_ndev; 2032 2032 + } 2033 2033 + 2034 2034 + ndev_init_struct(ndev, pdev); 2035 2035 + 2036 2036 + rc = intel_ntb_init_pci(ndev, pdev); 2037 2037 + if (rc) 2038 2038 + goto err_init_pci; 2039 2039 + 2040 2040 + rc = atom_init_dev(ndev); 2041 2041 + if (rc) 2042 2042 + goto err_init_dev; 2043 2043 + 2044 2044 + } else if (pdev_is_xeon(pdev)) { 2045 2045 + ndev = kzalloc_node(sizeof(*ndev), GFP_KERNEL, node); 2046 2046 + if (!ndev) { 2047 2047 + rc = -ENOMEM; 2048 2048 + goto err_ndev; 2049 2049 + } 2050 2050 + 2051 2051 + ndev_init_struct(ndev, pdev); 2052 2052 + 2053 2053 + rc = intel_ntb_init_pci(ndev, pdev); 2054 2054 + if (rc) 2055 2055 + goto err_init_pci; 2056 2056 + 2057 2057 + rc = xeon_init_dev(ndev); 2058 2058 + if (rc) 2059 2059 + goto err_init_dev; 2060 2060 + 2061 2061 + } else { 2062 2062 + rc = -EINVAL; 2063 2063 + goto err_ndev; 2064 2064 + } 2065 2065 + 2066 2066 + ndev_reset_unsafe_flags(ndev); 2067 2067 + 2068 2068 + ndev->reg->poll_link(ndev); 2069 2069 + 2070 2070 + ndev_init_debugfs(ndev); 2071 2071 + 2072 2072 + rc = ntb_register_device(&ndev->ntb); 2073 2073 + if (rc) 2074 2074 + goto err_register; 2075 2075 + 2076 2076 + dev_info(&pdev->dev, "NTB device registered.\n"); 2077 2077 + 2078 2078 + return 0; 2079 2079 + 2080 2080 + err_register: 2081 2081 + ndev_deinit_debugfs(ndev); 2082 2082 + if (pdev_is_atom(pdev)) 2083 2083 + atom_deinit_dev(ndev); 2084 2084 + else if (pdev_is_xeon(pdev)) 2085 2085 + xeon_deinit_dev(ndev); 2086 2086 + err_init_dev: 2087 2087 + intel_ntb_deinit_pci(ndev); 2088 2088 + err_init_pci: 2089 2089 + kfree(ndev); 2090 2090 + err_ndev: 2091 2091 + return rc; 2092 2092 + } 2093 2093 + 2094 2094 + static void intel_ntb_pci_remove(struct pci_dev *pdev) 2095 2095 + { 2096 2096 + struct intel_ntb_dev *ndev = pci_get_drvdata(pdev); 2097 2097 + 2098 2098 + ntb_unregister_device(&ndev->ntb); 2099 2099 + ndev_deinit_debugfs(ndev); 2100 2100 + if (pdev_is_atom(pdev)) 2101 2101 + atom_deinit_dev(ndev); 2102 2102 + else if (pdev_is_xeon(pdev)) 2103 2103 + xeon_deinit_dev(ndev); 2104 2104 + intel_ntb_deinit_pci(ndev); 2105 2105 + kfree(ndev); 2106 2106 + } 2107 2107 + 2108 2108 + static const struct intel_ntb_reg atom_reg = { 2109 2109 + .poll_link = atom_poll_link, 2110 2110 + .link_is_up = atom_link_is_up, 2111 2111 + .db_ioread = atom_db_ioread, 2112 2112 + .db_iowrite = atom_db_iowrite, 2113 2113 + .db_size = sizeof(u64), 2114 2114 + .ntb_ctl = ATOM_NTBCNTL_OFFSET, 2115 2115 + .mw_bar = {2, 4}, 2116 2116 + }; 2117 2117 + 2118 2118 + static const struct intel_ntb_alt_reg atom_pri_reg = { 2119 2119 + .db_bell = ATOM_PDOORBELL_OFFSET, 2120 2120 + .db_mask = ATOM_PDBMSK_OFFSET, 2121 2121 + .spad = ATOM_SPAD_OFFSET, 2122 2122 + }; 2123 2123 + 2124 2124 + static const struct intel_ntb_alt_reg atom_b2b_reg = { 2125 2125 + .db_bell = ATOM_B2B_DOORBELL_OFFSET, 2126 2126 + .spad = ATOM_B2B_SPAD_OFFSET, 2127 2127 + }; 2128 2128 + 2129 2129 + static const struct intel_ntb_xlat_reg atom_sec_xlat = { 2130 2130 + /* FIXME : .bar0_base = ATOM_SBAR0BASE_OFFSET, */ 2131 2131 + /* FIXME : .bar2_limit = ATOM_SBAR2LMT_OFFSET, */ 2132 2132 + .bar2_xlat = ATOM_SBAR2XLAT_OFFSET, 2133 2133 + }; 2134 2134 + 2135 2135 + static const struct intel_ntb_reg xeon_reg = { 2136 2136 + .poll_link = xeon_poll_link, 2137 2137 + .link_is_up = xeon_link_is_up, 2138 2138 + .db_ioread = xeon_db_ioread, 2139 2139 + .db_iowrite = xeon_db_iowrite, 2140 2140 + .db_size = sizeof(u32), 2141 2141 + .ntb_ctl = XEON_NTBCNTL_OFFSET, 2142 2142 + .mw_bar = {2, 4, 5}, 2143 2143 + }; 2144 2144 + 2145 2145 + static const struct intel_ntb_alt_reg xeon_pri_reg = { 2146 2146 + .db_bell = XEON_PDOORBELL_OFFSET, 2147 2147 + .db_mask = XEON_PDBMSK_OFFSET, 2148 2148 + .spad = XEON_SPAD_OFFSET, 2149 2149 + }; 2150 2150 + 2151 2151 + static const struct intel_ntb_alt_reg xeon_sec_reg = { 2152 2152 + .db_bell = XEON_SDOORBELL_OFFSET, 2153 2153 + .db_mask = XEON_SDBMSK_OFFSET, 2154 2154 + /* second half of the scratchpads */ 2155 2155 + .spad = XEON_SPAD_OFFSET + (XEON_SPAD_COUNT << 1), 2156 2156 + }; 2157 2157 + 2158 2158 + static const struct intel_ntb_alt_reg xeon_b2b_reg = { 2159 2159 + .db_bell = XEON_B2B_DOORBELL_OFFSET, 2160 2160 + .spad = XEON_B2B_SPAD_OFFSET, 2161 2161 + }; 2162 2162 + 2163 2163 + static const struct intel_ntb_xlat_reg xeon_pri_xlat = { 2164 2164 + /* Note: no primary .bar0_base visible to the secondary side. 2165 2165 + * 2166 2166 + * The secondary side cannot get the base address stored in primary 2167 2167 + * bars. The base address is necessary to set the limit register to 2168 2168 + * any value other than zero, or unlimited. 2169 2169 + * 2170 2170 + * WITHOUT THE BASE ADDRESS, THE SECONDARY SIDE CANNOT DISABLE the 2171 2171 + * window by setting the limit equal to base, nor can it limit the size 2172 2172 + * of the memory window by setting the limit to base + size. 2173 2173 + */ 2174 2174 + .bar2_limit = XEON_PBAR23LMT_OFFSET, 2175 2175 + .bar2_xlat = XEON_PBAR23XLAT_OFFSET, 2176 2176 + }; 2177 2177 + 2178 2178 + static const struct intel_ntb_xlat_reg xeon_sec_xlat = { 2179 2179 + .bar0_base = XEON_SBAR0BASE_OFFSET, 2180 2180 + .bar2_limit = XEON_SBAR23LMT_OFFSET, 2181 2181 + .bar2_xlat = XEON_SBAR23XLAT_OFFSET, 2182 2182 + }; 2183 2183 + 2184 2184 + static struct intel_b2b_addr xeon_b2b_usd_addr = { 2185 2185 + .bar2_addr64 = XEON_B2B_BAR2_USD_ADDR64, 2186 2186 + .bar4_addr64 = XEON_B2B_BAR4_USD_ADDR64, 2187 2187 + .bar4_addr32 = XEON_B2B_BAR4_USD_ADDR32, 2188 2188 + .bar5_addr32 = XEON_B2B_BAR5_USD_ADDR32, 2189 2189 + }; 2190 2190 + 2191 2191 + static struct intel_b2b_addr xeon_b2b_dsd_addr = { 2192 2192 + .bar2_addr64 = XEON_B2B_BAR2_DSD_ADDR64, 2193 2193 + .bar4_addr64 = XEON_B2B_BAR4_DSD_ADDR64, 2194 2194 + .bar4_addr32 = XEON_B2B_BAR4_DSD_ADDR32, 2195 2195 + .bar5_addr32 = XEON_B2B_BAR5_DSD_ADDR32, 2196 2196 + }; 2197 2197 + 2198 2198 + /* operations for primary side of local ntb */ 2199 2199 + static const struct ntb_dev_ops intel_ntb_ops = { 2200 2200 + .mw_count = intel_ntb_mw_count, 2201 2201 + .mw_get_range = intel_ntb_mw_get_range, 2202 2202 + .mw_set_trans = intel_ntb_mw_set_trans, 2203 2203 + .link_is_up = intel_ntb_link_is_up, 2204 2204 + .link_enable = intel_ntb_link_enable, 2205 2205 + .link_disable = intel_ntb_link_disable, 2206 2206 + .db_is_unsafe = intel_ntb_db_is_unsafe, 2207 2207 + .db_valid_mask = intel_ntb_db_valid_mask, 2208 2208 + .db_vector_count = intel_ntb_db_vector_count, 2209 2209 + .db_vector_mask = intel_ntb_db_vector_mask, 2210 2210 + .db_read = intel_ntb_db_read, 2211 2211 + .db_clear = intel_ntb_db_clear, 2212 2212 + .db_set_mask = intel_ntb_db_set_mask, 2213 2213 + .db_clear_mask = intel_ntb_db_clear_mask, 2214 2214 + .peer_db_addr = intel_ntb_peer_db_addr, 2215 2215 + .peer_db_set = intel_ntb_peer_db_set, 2216 2216 + .spad_is_unsafe = intel_ntb_spad_is_unsafe, 2217 2217 + .spad_count = intel_ntb_spad_count, 2218 2218 + .spad_read = intel_ntb_spad_read, 2219 2219 + .spad_write = intel_ntb_spad_write, 2220 2220 + .peer_spad_addr = intel_ntb_peer_spad_addr, 2221 2221 + .peer_spad_read = intel_ntb_peer_spad_read, 2222 2222 + .peer_spad_write = intel_ntb_peer_spad_write, 2223 2223 + }; 2224 2224 + 2225 2225 + static const struct file_operations intel_ntb_debugfs_info = { 2226 2226 + .owner = THIS_MODULE, 2227 2227 + .open = simple_open, 2228 2228 + .read = ndev_debugfs_read, 2229 2229 + }; 2230 2230 + 2231 2231 + static const struct pci_device_id intel_ntb_pci_tbl[] = { 2232 2232 + {PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_NTB_B2B_BWD)}, 2233 2233 + {PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_NTB_B2B_JSF)}, 2234 2234 + {PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_NTB_B2B_SNB)}, 2235 2235 + {PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_NTB_B2B_IVT)}, 2236 2236 + {PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_NTB_B2B_HSX)}, 2237 2237 + {PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_NTB_PS_JSF)}, 2238 2238 + {PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_NTB_PS_SNB)}, 2239 2239 + {PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_NTB_PS_IVT)}, 2240 2240 + {PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_NTB_PS_HSX)}, 2241 2241 + {PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_NTB_SS_JSF)}, 2242 2242 + {PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_NTB_SS_SNB)}, 2243 2243 + {PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_NTB_SS_IVT)}, 2244 2244 + {PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_NTB_SS_HSX)}, 2245 2245 + {0} 2246 2246 + }; 2247 2247 + MODULE_DEVICE_TABLE(pci, intel_ntb_pci_tbl); 2248 2248 + 2249 2249 + static struct pci_driver intel_ntb_pci_driver = { 2250 2250 + .name = KBUILD_MODNAME, 2251 2251 + .id_table = intel_ntb_pci_tbl, 2252 2252 + .probe = intel_ntb_pci_probe, 2253 2253 + .remove = intel_ntb_pci_remove, 2254 2254 + }; 2255 2255 + 2256 2256 + static int __init intel_ntb_pci_driver_init(void) 2257 2257 + { 2258 2258 + pr_info("%s %s\n", NTB_DESC, NTB_VER); 2259 2259 + 2260 2260 + if (debugfs_initialized()) 2261 2261 + debugfs_dir = debugfs_create_dir(KBUILD_MODNAME, NULL); 2262 2262 + 2263 2263 + return pci_register_driver(&intel_ntb_pci_driver); 2264 2264 + } 2265 2265 + module_init(intel_ntb_pci_driver_init); 2266 2266 + 2267 2267 + static void __exit intel_ntb_pci_driver_exit(void) 2268 2268 + { 2269 2269 + pci_unregister_driver(&intel_ntb_pci_driver); 2270 2270 + 2271 2271 + debugfs_remove_recursive(debugfs_dir); 2272 2272 + } 2273 2273 + module_exit(intel_ntb_pci_driver_exit); 2274 2274 +

+342

drivers/ntb/hw/intel/ntb_hw_intel.h

reviewed

··· 1 1 + /* 2 2 + * This file is provided under a dual BSD/GPLv2 license. When using or 3 3 + * redistributing this file, you may do so under either license. 4 4 + * 5 5 + * GPL LICENSE SUMMARY 6 6 + * 7 7 + * Copyright(c) 2012 Intel Corporation. All rights reserved. 8 8 + * Copyright (C) 2015 EMC Corporation. All Rights Reserved. 9 9 + * 10 10 + * This program is free software; you can redistribute it and/or modify 11 11 + * it under the terms of version 2 of the GNU General Public License as 12 12 + * published by the Free Software Foundation. 13 13 + * 14 14 + * BSD LICENSE 15 15 + * 16 16 + * Copyright(c) 2012 Intel Corporation. All rights reserved. 17 17 + * Copyright (C) 2015 EMC Corporation. All Rights Reserved. 18 18 + * 19 19 + * Redistribution and use in source and binary forms, with or without 20 20 + * modification, are permitted provided that the following conditions 21 21 + * are met: 22 22 + * 23 23 + * * Redistributions of source code must retain the above copyright 24 24 + * notice, this list of conditions and the following disclaimer. 25 25 + * * Redistributions in binary form must reproduce the above copy 26 26 + * notice, this list of conditions and the following disclaimer in 27 27 + * the documentation and/or other materials provided with the 28 28 + * distribution. 29 29 + * * Neither the name of Intel Corporation nor the names of its 30 30 + * contributors may be used to endorse or promote products derived 31 31 + * from this software without specific prior written permission. 32 32 + * 33 33 + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 34 34 + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 35 35 + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 36 36 + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 37 37 + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 38 38 + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 39 39 + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 40 40 + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 41 41 + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 42 42 + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 43 43 + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 44 44 + * 45 45 + * Intel PCIe NTB Linux driver 46 46 + * 47 47 + * Contact Information: 48 48 + * Jon Mason <jon.mason@intel.com> 49 49 + */ 50 50 + 51 51 + #ifndef NTB_HW_INTEL_H 52 52 + #define NTB_HW_INTEL_H 53 53 + 54 54 + #include <linux/ntb.h> 55 55 + #include <linux/pci.h> 56 56 + 57 57 + #define PCI_DEVICE_ID_INTEL_NTB_B2B_JSF 0x3725 58 58 + #define PCI_DEVICE_ID_INTEL_NTB_PS_JSF 0x3726 59 59 + #define PCI_DEVICE_ID_INTEL_NTB_SS_JSF 0x3727 60 60 + #define PCI_DEVICE_ID_INTEL_NTB_B2B_SNB 0x3C0D 61 61 + #define PCI_DEVICE_ID_INTEL_NTB_PS_SNB 0x3C0E 62 62 + #define PCI_DEVICE_ID_INTEL_NTB_SS_SNB 0x3C0F 63 63 + #define PCI_DEVICE_ID_INTEL_NTB_B2B_IVT 0x0E0D 64 64 + #define PCI_DEVICE_ID_INTEL_NTB_PS_IVT 0x0E0E 65 65 + #define PCI_DEVICE_ID_INTEL_NTB_SS_IVT 0x0E0F 66 66 + #define PCI_DEVICE_ID_INTEL_NTB_B2B_HSX 0x2F0D 67 67 + #define PCI_DEVICE_ID_INTEL_NTB_PS_HSX 0x2F0E 68 68 + #define PCI_DEVICE_ID_INTEL_NTB_SS_HSX 0x2F0F 69 69 + #define PCI_DEVICE_ID_INTEL_NTB_B2B_BWD 0x0C4E 70 70 + 71 71 + /* Intel Xeon hardware */ 72 72 + 73 73 + #define XEON_PBAR23LMT_OFFSET 0x0000 74 74 + #define XEON_PBAR45LMT_OFFSET 0x0008 75 75 + #define XEON_PBAR4LMT_OFFSET 0x0008 76 76 + #define XEON_PBAR5LMT_OFFSET 0x000c 77 77 + #define XEON_PBAR23XLAT_OFFSET 0x0010 78 78 + #define XEON_PBAR45XLAT_OFFSET 0x0018 79 79 + #define XEON_PBAR4XLAT_OFFSET 0x0018 80 80 + #define XEON_PBAR5XLAT_OFFSET 0x001c 81 81 + #define XEON_SBAR23LMT_OFFSET 0x0020 82 82 + #define XEON_SBAR45LMT_OFFSET 0x0028 83 83 + #define XEON_SBAR4LMT_OFFSET 0x0028 84 84 + #define XEON_SBAR5LMT_OFFSET 0x002c 85 85 + #define XEON_SBAR23XLAT_OFFSET 0x0030 86 86 + #define XEON_SBAR45XLAT_OFFSET 0x0038 87 87 + #define XEON_SBAR4XLAT_OFFSET 0x0038 88 88 + #define XEON_SBAR5XLAT_OFFSET 0x003c 89 89 + #define XEON_SBAR0BASE_OFFSET 0x0040 90 90 + #define XEON_SBAR23BASE_OFFSET 0x0048 91 91 + #define XEON_SBAR45BASE_OFFSET 0x0050 92 92 + #define XEON_SBAR4BASE_OFFSET 0x0050 93 93 + #define XEON_SBAR5BASE_OFFSET 0x0054 94 94 + #define XEON_SBDF_OFFSET 0x005c 95 95 + #define XEON_NTBCNTL_OFFSET 0x0058 96 96 + #define XEON_PDOORBELL_OFFSET 0x0060 97 97 + #define XEON_PDBMSK_OFFSET 0x0062 98 98 + #define XEON_SDOORBELL_OFFSET 0x0064 99 99 + #define XEON_SDBMSK_OFFSET 0x0066 100 100 + #define XEON_USMEMMISS_OFFSET 0x0070 101 101 + #define XEON_SPAD_OFFSET 0x0080 102 102 + #define XEON_PBAR23SZ_OFFSET 0x00d0 103 103 + #define XEON_PBAR45SZ_OFFSET 0x00d1 104 104 + #define XEON_PBAR4SZ_OFFSET 0x00d1 105 105 + #define XEON_SBAR23SZ_OFFSET 0x00d2 106 106 + #define XEON_SBAR45SZ_OFFSET 0x00d3 107 107 + #define XEON_SBAR4SZ_OFFSET 0x00d3 108 108 + #define XEON_PPD_OFFSET 0x00d4 109 109 + #define XEON_PBAR5SZ_OFFSET 0x00d5 110 110 + #define XEON_SBAR5SZ_OFFSET 0x00d6 111 111 + #define XEON_WCCNTRL_OFFSET 0x00e0 112 112 + #define XEON_UNCERRSTS_OFFSET 0x014c 113 113 + #define XEON_CORERRSTS_OFFSET 0x0158 114 114 + #define XEON_LINK_STATUS_OFFSET 0x01a2 115 115 + #define XEON_SPCICMD_OFFSET 0x0504 116 116 + #define XEON_DEVCTRL_OFFSET 0x0598 117 117 + #define XEON_DEVSTS_OFFSET 0x059a 118 118 + #define XEON_SLINK_STATUS_OFFSET 0x05a2 119 119 + #define XEON_B2B_SPAD_OFFSET 0x0100 120 120 + #define XEON_B2B_DOORBELL_OFFSET 0x0140 121 121 + #define XEON_B2B_XLAT_OFFSETL 0x0144 122 122 + #define XEON_B2B_XLAT_OFFSETU 0x0148 123 123 + #define XEON_PPD_CONN_MASK 0x03 124 124 + #define XEON_PPD_CONN_TRANSPARENT 0x00 125 125 + #define XEON_PPD_CONN_B2B 0x01 126 126 + #define XEON_PPD_CONN_RP 0x02 127 127 + #define XEON_PPD_DEV_MASK 0x10 128 128 + #define XEON_PPD_DEV_USD 0x00 129 129 + #define XEON_PPD_DEV_DSD 0x10 130 130 + #define XEON_PPD_SPLIT_BAR_MASK 0x40 131 131 + 132 132 + #define XEON_PPD_TOPO_MASK (XEON_PPD_CONN_MASK | XEON_PPD_DEV_MASK) 133 133 + #define XEON_PPD_TOPO_PRI_USD (XEON_PPD_CONN_RP | XEON_PPD_DEV_USD) 134 134 + #define XEON_PPD_TOPO_PRI_DSD (XEON_PPD_CONN_RP | XEON_PPD_DEV_DSD) 135 135 + #define XEON_PPD_TOPO_SEC_USD (XEON_PPD_CONN_TRANSPARENT | XEON_PPD_DEV_USD) 136 136 + #define XEON_PPD_TOPO_SEC_DSD (XEON_PPD_CONN_TRANSPARENT | XEON_PPD_DEV_DSD) 137 137 + #define XEON_PPD_TOPO_B2B_USD (XEON_PPD_CONN_B2B | XEON_PPD_DEV_USD) 138 138 + #define XEON_PPD_TOPO_B2B_DSD (XEON_PPD_CONN_B2B | XEON_PPD_DEV_DSD) 139 139 + 140 140 + #define XEON_MW_COUNT 2 141 141 + #define HSX_SPLIT_BAR_MW_COUNT 3 142 142 + #define XEON_DB_COUNT 15 143 143 + #define XEON_DB_LINK 15 144 144 + #define XEON_DB_LINK_BIT BIT_ULL(XEON_DB_LINK) 145 145 + #define XEON_DB_MSIX_VECTOR_COUNT 4 146 146 + #define XEON_DB_MSIX_VECTOR_SHIFT 5 147 147 + #define XEON_DB_TOTAL_SHIFT 16 148 148 + #define XEON_SPAD_COUNT 16 149 149 + 150 150 + /* Intel Atom hardware */ 151 151 + 152 152 + #define ATOM_SBAR2XLAT_OFFSET 0x0008 153 153 + #define ATOM_PDOORBELL_OFFSET 0x0020 154 154 + #define ATOM_PDBMSK_OFFSET 0x0028 155 155 + #define ATOM_NTBCNTL_OFFSET 0x0060 156 156 + #define ATOM_SPAD_OFFSET 0x0080 157 157 + #define ATOM_PPD_OFFSET 0x00d4 158 158 + #define ATOM_PBAR2XLAT_OFFSET 0x8008 159 159 + #define ATOM_B2B_DOORBELL_OFFSET 0x8020 160 160 + #define ATOM_B2B_SPAD_OFFSET 0x8080 161 161 + #define ATOM_SPCICMD_OFFSET 0xb004 162 162 + #define ATOM_LINK_STATUS_OFFSET 0xb052 163 163 + #define ATOM_ERRCORSTS_OFFSET 0xb110 164 164 + #define ATOM_IP_BASE 0xc000 165 165 + #define ATOM_DESKEWSTS_OFFSET (ATOM_IP_BASE + 0x3024) 166 166 + #define ATOM_LTSSMERRSTS0_OFFSET (ATOM_IP_BASE + 0x3180) 167 167 + #define ATOM_LTSSMSTATEJMP_OFFSET (ATOM_IP_BASE + 0x3040) 168 168 + #define ATOM_IBSTERRRCRVSTS0_OFFSET (ATOM_IP_BASE + 0x3324) 169 169 + #define ATOM_MODPHY_PCSREG4 0x1c004 170 170 + #define ATOM_MODPHY_PCSREG6 0x1c006 171 171 + 172 172 + #define ATOM_PPD_INIT_LINK 0x0008 173 173 + #define ATOM_PPD_CONN_MASK 0x0300 174 174 + #define ATOM_PPD_CONN_TRANSPARENT 0x0000 175 175 + #define ATOM_PPD_CONN_B2B 0x0100 176 176 + #define ATOM_PPD_CONN_RP 0x0200 177 177 + #define ATOM_PPD_DEV_MASK 0x1000 178 178 + #define ATOM_PPD_DEV_USD 0x0000 179 179 + #define ATOM_PPD_DEV_DSD 0x1000 180 180 + #define ATOM_PPD_TOPO_MASK (ATOM_PPD_CONN_MASK | ATOM_PPD_DEV_MASK) 181 181 + #define ATOM_PPD_TOPO_PRI_USD (ATOM_PPD_CONN_TRANSPARENT | ATOM_PPD_DEV_USD) 182 182 + #define ATOM_PPD_TOPO_PRI_DSD (ATOM_PPD_CONN_TRANSPARENT | ATOM_PPD_DEV_DSD) 183 183 + #define ATOM_PPD_TOPO_SEC_USD (ATOM_PPD_CONN_RP | ATOM_PPD_DEV_USD) 184 184 + #define ATOM_PPD_TOPO_SEC_DSD (ATOM_PPD_CONN_RP | ATOM_PPD_DEV_DSD) 185 185 + #define ATOM_PPD_TOPO_B2B_USD (ATOM_PPD_CONN_B2B | ATOM_PPD_DEV_USD) 186 186 + #define ATOM_PPD_TOPO_B2B_DSD (ATOM_PPD_CONN_B2B | ATOM_PPD_DEV_DSD) 187 187 + 188 188 + #define ATOM_MW_COUNT 2 189 189 + #define ATOM_DB_COUNT 34 190 190 + #define ATOM_DB_VALID_MASK (BIT_ULL(ATOM_DB_COUNT) - 1) 191 191 + #define ATOM_DB_MSIX_VECTOR_COUNT 34 192 192 + #define ATOM_DB_MSIX_VECTOR_SHIFT 1 193 193 + #define ATOM_DB_TOTAL_SHIFT 34 194 194 + #define ATOM_SPAD_COUNT 16 195 195 + 196 196 + #define ATOM_NTB_CTL_DOWN_BIT BIT(16) 197 197 + #define ATOM_NTB_CTL_ACTIVE(x) !(x & ATOM_NTB_CTL_DOWN_BIT) 198 198 + 199 199 + #define ATOM_DESKEWSTS_DBERR BIT(15) 200 200 + #define ATOM_LTSSMERRSTS0_UNEXPECTEDEI BIT(20) 201 201 + #define ATOM_LTSSMSTATEJMP_FORCEDETECT BIT(2) 202 202 + #define ATOM_IBIST_ERR_OFLOW 0x7FFF7FFF 203 203 + 204 204 + #define ATOM_LINK_HB_TIMEOUT msecs_to_jiffies(1000) 205 205 + #define ATOM_LINK_RECOVERY_TIME msecs_to_jiffies(500) 206 206 + 207 207 + /* Ntb control and link status */ 208 208 + 209 209 + #define NTB_CTL_CFG_LOCK BIT(0) 210 210 + #define NTB_CTL_DISABLE BIT(1) 211 211 + #define NTB_CTL_S2P_BAR2_SNOOP BIT(2) 212 212 + #define NTB_CTL_P2S_BAR2_SNOOP BIT(4) 213 213 + #define NTB_CTL_S2P_BAR4_SNOOP BIT(6) 214 214 + #define NTB_CTL_P2S_BAR4_SNOOP BIT(8) 215 215 + #define NTB_CTL_S2P_BAR5_SNOOP BIT(12) 216 216 + #define NTB_CTL_P2S_BAR5_SNOOP BIT(14) 217 217 + 218 218 + #define NTB_LNK_STA_ACTIVE_BIT 0x2000 219 219 + #define NTB_LNK_STA_SPEED_MASK 0x000f 220 220 + #define NTB_LNK_STA_WIDTH_MASK 0x03f0 221 221 + #define NTB_LNK_STA_ACTIVE(x) (!!((x) & NTB_LNK_STA_ACTIVE_BIT)) 222 222 + #define NTB_LNK_STA_SPEED(x) ((x) & NTB_LNK_STA_SPEED_MASK) 223 223 + #define NTB_LNK_STA_WIDTH(x) (((x) & NTB_LNK_STA_WIDTH_MASK) >> 4) 224 224 + 225 225 + /* Use the following addresses for translation between b2b ntb devices in case 226 226 + * the hardware default values are not reliable. */ 227 227 + #define XEON_B2B_BAR0_USD_ADDR 0x1000000000000000ull 228 228 + #define XEON_B2B_BAR2_USD_ADDR64 0x2000000000000000ull 229 229 + #define XEON_B2B_BAR4_USD_ADDR64 0x4000000000000000ull 230 230 + #define XEON_B2B_BAR4_USD_ADDR32 0x20000000u 231 231 + #define XEON_B2B_BAR5_USD_ADDR32 0x40000000u 232 232 + #define XEON_B2B_BAR0_DSD_ADDR 0x9000000000000000ull 233 233 + #define XEON_B2B_BAR2_DSD_ADDR64 0xa000000000000000ull 234 234 + #define XEON_B2B_BAR4_DSD_ADDR64 0xc000000000000000ull 235 235 + #define XEON_B2B_BAR4_DSD_ADDR32 0xa0000000u 236 236 + #define XEON_B2B_BAR5_DSD_ADDR32 0xc0000000u 237 237 + 238 238 + /* The peer ntb secondary config space is 32KB fixed size */ 239 239 + #define XEON_B2B_MIN_SIZE 0x8000 240 240 + 241 241 + /* flags to indicate hardware errata */ 242 242 + #define NTB_HWERR_SDOORBELL_LOCKUP BIT_ULL(0) 243 243 + #define NTB_HWERR_SB01BASE_LOCKUP BIT_ULL(1) 244 244 + #define NTB_HWERR_B2BDOORBELL_BIT14 BIT_ULL(2) 245 245 + 246 246 + /* flags to indicate unsafe api */ 247 247 + #define NTB_UNSAFE_DB BIT_ULL(0) 248 248 + #define NTB_UNSAFE_SPAD BIT_ULL(1) 249 249 + 250 250 + struct intel_ntb_dev; 251 251 + 252 252 + struct intel_ntb_reg { 253 253 + int (*poll_link)(struct intel_ntb_dev *ndev); 254 254 + int (*link_is_up)(struct intel_ntb_dev *ndev); 255 255 + u64 (*db_ioread)(void __iomem *mmio); 256 256 + void (*db_iowrite)(u64 db_bits, void __iomem *mmio); 257 257 + unsigned long ntb_ctl; 258 258 + resource_size_t db_size; 259 259 + int mw_bar[]; 260 260 + }; 261 261 + 262 262 + struct intel_ntb_alt_reg { 263 263 + unsigned long db_bell; 264 264 + unsigned long db_mask; 265 265 + unsigned long spad; 266 266 + }; 267 267 + 268 268 + struct intel_ntb_xlat_reg { 269 269 + unsigned long bar0_base; 270 270 + unsigned long bar2_xlat; 271 271 + unsigned long bar2_limit; 272 272 + }; 273 273 + 274 274 + struct intel_b2b_addr { 275 275 + phys_addr_t bar0_addr; 276 276 + phys_addr_t bar2_addr64; 277 277 + phys_addr_t bar4_addr64; 278 278 + phys_addr_t bar4_addr32; 279 279 + phys_addr_t bar5_addr32; 280 280 + }; 281 281 + 282 282 + struct intel_ntb_vec { 283 283 + struct intel_ntb_dev *ndev; 284 284 + int num; 285 285 + }; 286 286 + 287 287 + struct intel_ntb_dev { 288 288 + struct ntb_dev ntb; 289 289 + 290 290 + /* offset of peer bar0 in b2b bar */ 291 291 + unsigned long b2b_off; 292 292 + /* mw idx used to access peer bar0 */ 293 293 + unsigned int b2b_idx; 294 294 + 295 295 + /* BAR45 is split into BAR4 and BAR5 */ 296 296 + bool bar4_split; 297 297 + 298 298 + u32 ntb_ctl; 299 299 + u32 lnk_sta; 300 300 + 301 301 + unsigned char mw_count; 302 302 + unsigned char spad_count; 303 303 + unsigned char db_count; 304 304 + unsigned char db_vec_count; 305 305 + unsigned char db_vec_shift; 306 306 + 307 307 + u64 db_valid_mask; 308 308 + u64 db_link_mask; 309 309 + u64 db_mask; 310 310 + 311 311 + /* synchronize rmw access of db_mask and hw reg */ 312 312 + spinlock_t db_mask_lock; 313 313 + 314 314 + struct msix_entry *msix; 315 315 + struct intel_ntb_vec *vec; 316 316 + 317 317 + const struct intel_ntb_reg *reg; 318 318 + const struct intel_ntb_alt_reg *self_reg; 319 319 + const struct intel_ntb_alt_reg *peer_reg; 320 320 + const struct intel_ntb_xlat_reg *xlat_reg; 321 321 + void __iomem *self_mmio; 322 322 + void __iomem *peer_mmio; 323 323 + phys_addr_t peer_addr; 324 324 + 325 325 + unsigned long last_ts; 326 326 + struct delayed_work hb_timer; 327 327 + 328 328 + unsigned long hwerr_flags; 329 329 + unsigned long unsafe_flags; 330 330 + unsigned long unsafe_flags_ignore; 331 331 + 332 332 + struct dentry *debugfs_dir; 333 333 + struct dentry *debugfs_info; 334 334 + }; 335 335 + 336 336 + #define ndev_pdev(ndev) ((ndev)->ntb.pdev) 337 337 + #define ndev_name(ndev) pci_name(ndev_pdev(ndev)) 338 338 + #define ndev_dev(ndev) (&ndev_pdev(ndev)->dev) 339 339 + #define ntb_ndev(ntb) container_of(ntb, struct intel_ntb_dev, ntb) 340 340 + #define hb_ndev(work) container_of(work, struct intel_ntb_dev, hb_timer.work) 341 341 + 342 342 + #endif

+251

drivers/ntb/ntb.c

reviewed

··· 1 1 + /* 2 2 + * This file is provided under a dual BSD/GPLv2 license. When using or 3 3 + * redistributing this file, you may do so under either license. 4 4 + * 5 5 + * GPL LICENSE SUMMARY 6 6 + * 7 7 + * Copyright (C) 2015 EMC Corporation. All Rights Reserved. 8 8 + * 9 9 + * This program is free software; you can redistribute it and/or modify 10 10 + * it under the terms of version 2 of the GNU General Public License as 11 11 + * published by the Free Software Foundation. 12 12 + * 13 13 + * This program is distributed in the hope that it will be useful, but 14 14 + * WITHOUT ANY WARRANTY; without even the implied warranty of 15 15 + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 16 16 + * General Public License for more details. 17 17 + * 18 18 + * BSD LICENSE 19 19 + * 20 20 + * Copyright (C) 2015 EMC Corporation. All Rights Reserved. 21 21 + * 22 22 + * Redistribution and use in source and binary forms, with or without 23 23 + * modification, are permitted provided that the following conditions 24 24 + * are met: 25 25 + * 26 26 + * * Redistributions of source code must retain the above copyright 27 27 + * notice, this list of conditions and the following disclaimer. 28 28 + * * Redistributions in binary form must reproduce the above copy 29 29 + * notice, this list of conditions and the following disclaimer in 30 30 + * the documentation and/or other materials provided with the 31 31 + * distribution. 32 32 + * * Neither the name of Intel Corporation nor the names of its 33 33 + * contributors may be used to endorse or promote products derived 34 34 + * from this software without specific prior written permission. 35 35 + * 36 36 + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 37 37 + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 38 38 + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 39 39 + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 40 40 + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 41 41 + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 42 42 + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 43 43 + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 44 44 + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 45 45 + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 46 46 + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 47 47 + * 48 48 + * PCIe NTB Linux driver 49 49 + * 50 50 + * Contact Information: 51 51 + * Allen Hubbe <Allen.Hubbe@emc.com> 52 52 + */ 53 53 + 54 54 + #include <linux/device.h> 55 55 + #include <linux/kernel.h> 56 56 + #include <linux/module.h> 57 57 + 58 58 + #include <linux/ntb.h> 59 59 + #include <linux/pci.h> 60 60 + 61 61 + #define DRIVER_NAME "ntb" 62 62 + #define DRIVER_DESCRIPTION "PCIe NTB Driver Framework" 63 63 + 64 64 + #define DRIVER_LICENSE "Dual BSD/GPL" 65 65 + #define DRIVER_VERSION "1.0" 66 66 + #define DRIVER_RELDATE "24 March 2015" 67 67 + #define DRIVER_AUTHOR "Allen Hubbe <Allen.Hubbe@emc.com>" 68 68 + 69 69 + MODULE_LICENSE(DRIVER_LICENSE); 70 70 + MODULE_VERSION(DRIVER_VERSION); 71 71 + MODULE_AUTHOR(DRIVER_AUTHOR); 72 72 + MODULE_DESCRIPTION(DRIVER_DESCRIPTION); 73 73 + 74 74 + static struct bus_type ntb_bus; 75 75 + static void ntb_dev_release(struct device *dev); 76 76 + 77 77 + int __ntb_register_client(struct ntb_client *client, struct module *mod, 78 78 + const char *mod_name) 79 79 + { 80 80 + if (!client) 81 81 + return -EINVAL; 82 82 + if (!ntb_client_ops_is_valid(&client->ops)) 83 83 + return -EINVAL; 84 84 + 85 85 + memset(&client->drv, 0, sizeof(client->drv)); 86 86 + client->drv.bus = &ntb_bus; 87 87 + client->drv.name = mod_name; 88 88 + client->drv.owner = mod; 89 89 + 90 90 + return driver_register(&client->drv); 91 91 + } 92 92 + EXPORT_SYMBOL(__ntb_register_client); 93 93 + 94 94 + void ntb_unregister_client(struct ntb_client *client) 95 95 + { 96 96 + driver_unregister(&client->drv); 97 97 + } 98 98 + EXPORT_SYMBOL(ntb_unregister_client); 99 99 + 100 100 + int ntb_register_device(struct ntb_dev *ntb) 101 101 + { 102 102 + if (!ntb) 103 103 + return -EINVAL; 104 104 + if (!ntb->pdev) 105 105 + return -EINVAL; 106 106 + if (!ntb->ops) 107 107 + return -EINVAL; 108 108 + if (!ntb_dev_ops_is_valid(ntb->ops)) 109 109 + return -EINVAL; 110 110 + 111 111 + init_completion(&ntb->released); 112 112 + 113 113 + memset(&ntb->dev, 0, sizeof(ntb->dev)); 114 114 + ntb->dev.bus = &ntb_bus; 115 115 + ntb->dev.parent = &ntb->pdev->dev; 116 116 + ntb->dev.release = ntb_dev_release; 117 117 + dev_set_name(&ntb->dev, pci_name(ntb->pdev)); 118 118 + 119 119 + ntb->ctx = NULL; 120 120 + ntb->ctx_ops = NULL; 121 121 + spin_lock_init(&ntb->ctx_lock); 122 122 + 123 123 + return device_register(&ntb->dev); 124 124 + } 125 125 + EXPORT_SYMBOL(ntb_register_device); 126 126 + 127 127 + void ntb_unregister_device(struct ntb_dev *ntb) 128 128 + { 129 129 + device_unregister(&ntb->dev); 130 130 + wait_for_completion(&ntb->released); 131 131 + } 132 132 + EXPORT_SYMBOL(ntb_unregister_device); 133 133 + 134 134 + int ntb_set_ctx(struct ntb_dev *ntb, void *ctx, 135 135 + const struct ntb_ctx_ops *ctx_ops) 136 136 + { 137 137 + unsigned long irqflags; 138 138 + 139 139 + if (!ntb_ctx_ops_is_valid(ctx_ops)) 140 140 + return -EINVAL; 141 141 + if (ntb->ctx_ops) 142 142 + return -EINVAL; 143 143 + 144 144 + spin_lock_irqsave(&ntb->ctx_lock, irqflags); 145 145 + { 146 146 + ntb->ctx = ctx; 147 147 + ntb->ctx_ops = ctx_ops; 148 148 + } 149 149 + spin_unlock_irqrestore(&ntb->ctx_lock, irqflags); 150 150 + 151 151 + return 0; 152 152 + } 153 153 + EXPORT_SYMBOL(ntb_set_ctx); 154 154 + 155 155 + void ntb_clear_ctx(struct ntb_dev *ntb) 156 156 + { 157 157 + unsigned long irqflags; 158 158 + 159 159 + spin_lock_irqsave(&ntb->ctx_lock, irqflags); 160 160 + { 161 161 + ntb->ctx_ops = NULL; 162 162 + ntb->ctx = NULL; 163 163 + } 164 164 + spin_unlock_irqrestore(&ntb->ctx_lock, irqflags); 165 165 + } 166 166 + EXPORT_SYMBOL(ntb_clear_ctx); 167 167 + 168 168 + void ntb_link_event(struct ntb_dev *ntb) 169 169 + { 170 170 + unsigned long irqflags; 171 171 + 172 172 + spin_lock_irqsave(&ntb->ctx_lock, irqflags); 173 173 + { 174 174 + if (ntb->ctx_ops && ntb->ctx_ops->link_event) 175 175 + ntb->ctx_ops->link_event(ntb->ctx); 176 176 + } 177 177 + spin_unlock_irqrestore(&ntb->ctx_lock, irqflags); 178 178 + } 179 179 + EXPORT_SYMBOL(ntb_link_event); 180 180 + 181 181 + void ntb_db_event(struct ntb_dev *ntb, int vector) 182 182 + { 183 183 + unsigned long irqflags; 184 184 + 185 185 + spin_lock_irqsave(&ntb->ctx_lock, irqflags); 186 186 + { 187 187 + if (ntb->ctx_ops && ntb->ctx_ops->db_event) 188 188 + ntb->ctx_ops->db_event(ntb->ctx, vector); 189 189 + } 190 190 + spin_unlock_irqrestore(&ntb->ctx_lock, irqflags); 191 191 + } 192 192 + EXPORT_SYMBOL(ntb_db_event); 193 193 + 194 194 + static int ntb_probe(struct device *dev) 195 195 + { 196 196 + struct ntb_dev *ntb; 197 197 + struct ntb_client *client; 198 198 + int rc; 199 199 + 200 200 + get_device(dev); 201 201 + ntb = dev_ntb(dev); 202 202 + client = drv_ntb_client(dev->driver); 203 203 + 204 204 + rc = client->ops.probe(client, ntb); 205 205 + if (rc) 206 206 + put_device(dev); 207 207 + 208 208 + return rc; 209 209 + } 210 210 + 211 211 + static int ntb_remove(struct device *dev) 212 212 + { 213 213 + struct ntb_dev *ntb; 214 214 + struct ntb_client *client; 215 215 + 216 216 + if (dev->driver) { 217 217 + ntb = dev_ntb(dev); 218 218 + client = drv_ntb_client(dev->driver); 219 219 + 220 220 + client->ops.remove(client, ntb); 221 221 + put_device(dev); 222 222 + } 223 223 + 224 224 + return 0; 225 225 + } 226 226 + 227 227 + static void ntb_dev_release(struct device *dev) 228 228 + { 229 229 + struct ntb_dev *ntb = dev_ntb(dev); 230 230 + 231 231 + complete(&ntb->released); 232 232 + } 233 233 + 234 234 + static struct bus_type ntb_bus = { 235 235 + .name = "ntb", 236 236 + .probe = ntb_probe, 237 237 + .remove = ntb_remove, 238 238 + }; 239 239 + 240 240 + static int __init ntb_driver_init(void) 241 241 + { 242 242 + return bus_register(&ntb_bus); 243 243 + } 244 244 + module_init(ntb_driver_init); 245 245 + 246 246 + static void __exit ntb_driver_exit(void) 247 247 + { 248 248 + bus_unregister(&ntb_bus); 249 249 + } 250 250 + module_exit(ntb_driver_exit); 251 251 +

-1895

drivers/ntb/ntb_hw.c

reviewed

··· 1 1 - /* 2 2 - * This file is provided under a dual BSD/GPLv2 license. When using or 3 3 - * redistributing this file, you may do so under either license. 4 4 - * 5 5 - * GPL LICENSE SUMMARY 6 6 - * 7 7 - * Copyright(c) 2012 Intel Corporation. All rights reserved. 8 8 - * 9 9 - * This program is free software; you can redistribute it and/or modify 10 10 - * it under the terms of version 2 of the GNU General Public License as 11 11 - * published by the Free Software Foundation. 12 12 - * 13 13 - * BSD LICENSE 14 14 - * 15 15 - * Copyright(c) 2012 Intel Corporation. All rights reserved. 16 16 - * 17 17 - * Redistribution and use in source and binary forms, with or without 18 18 - * modification, are permitted provided that the following conditions 19 19 - * are met: 20 20 - * 21 21 - * * Redistributions of source code must retain the above copyright 22 22 - * notice, this list of conditions and the following disclaimer. 23 23 - * * Redistributions in binary form must reproduce the above copy 24 24 - * notice, this list of conditions and the following disclaimer in 25 25 - * the documentation and/or other materials provided with the 26 26 - * distribution. 27 27 - * * Neither the name of Intel Corporation nor the names of its 28 28 - * contributors may be used to endorse or promote products derived 29 29 - * from this software without specific prior written permission. 30 30 - * 31 31 - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 32 32 - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 33 33 - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 34 34 - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 35 35 - * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 36 36 - * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 37 37 - * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 38 38 - * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 39 39 - * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 40 40 - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 41 41 - * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 42 42 - * 43 43 - * Intel PCIe NTB Linux driver 44 44 - * 45 45 - * Contact Information: 46 46 - * Jon Mason <jon.mason@intel.com> 47 47 - */ 48 48 - #include <linux/debugfs.h> 49 49 - #include <linux/delay.h> 50 50 - #include <linux/init.h> 51 51 - #include <linux/interrupt.h> 52 52 - #include <linux/module.h> 53 53 - #include <linux/pci.h> 54 54 - #include <linux/random.h> 55 55 - #include <linux/slab.h> 56 56 - #include "ntb_hw.h" 57 57 - #include "ntb_regs.h" 58 58 - 59 59 - #define NTB_NAME "Intel(R) PCI-E Non-Transparent Bridge Driver" 60 60 - #define NTB_VER "1.0" 61 61 - 62 62 - MODULE_DESCRIPTION(NTB_NAME); 63 63 - MODULE_VERSION(NTB_VER); 64 64 - MODULE_LICENSE("Dual BSD/GPL"); 65 65 - MODULE_AUTHOR("Intel Corporation"); 66 66 - 67 67 - enum { 68 68 - NTB_CONN_TRANSPARENT = 0, 69 69 - NTB_CONN_B2B, 70 70 - NTB_CONN_RP, 71 71 - }; 72 72 - 73 73 - enum { 74 74 - NTB_DEV_USD = 0, 75 75 - NTB_DEV_DSD, 76 76 - }; 77 77 - 78 78 - enum { 79 79 - SNB_HW = 0, 80 80 - BWD_HW, 81 81 - }; 82 82 - 83 83 - static struct dentry *debugfs_dir; 84 84 - 85 85 - #define BWD_LINK_RECOVERY_TIME 500 86 86 - 87 87 - /* Translate memory window 0,1,2 to BAR 2,4,5 */ 88 88 - #define MW_TO_BAR(mw) (mw == 0 ? 2 : (mw == 1 ? 4 : 5)) 89 89 - 90 90 - static const struct pci_device_id ntb_pci_tbl[] = { 91 91 - {PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_NTB_B2B_BWD)}, 92 92 - {PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_NTB_B2B_JSF)}, 93 93 - {PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_NTB_B2B_SNB)}, 94 94 - {PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_NTB_B2B_IVT)}, 95 95 - {PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_NTB_B2B_HSX)}, 96 96 - {PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_NTB_PS_JSF)}, 97 97 - {PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_NTB_PS_SNB)}, 98 98 - {PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_NTB_PS_IVT)}, 99 99 - {PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_NTB_PS_HSX)}, 100 100 - {PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_NTB_SS_JSF)}, 101 101 - {PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_NTB_SS_SNB)}, 102 102 - {PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_NTB_SS_IVT)}, 103 103 - {PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_NTB_SS_HSX)}, 104 104 - {0} 105 105 - }; 106 106 - MODULE_DEVICE_TABLE(pci, ntb_pci_tbl); 107 107 - 108 108 - static int is_ntb_xeon(struct ntb_device *ndev) 109 109 - { 110 110 - switch (ndev->pdev->device) { 111 111 - case PCI_DEVICE_ID_INTEL_NTB_SS_JSF: 112 112 - case PCI_DEVICE_ID_INTEL_NTB_SS_SNB: 113 113 - case PCI_DEVICE_ID_INTEL_NTB_SS_IVT: 114 114 - case PCI_DEVICE_ID_INTEL_NTB_SS_HSX: 115 115 - case PCI_DEVICE_ID_INTEL_NTB_PS_JSF: 116 116 - case PCI_DEVICE_ID_INTEL_NTB_PS_SNB: 117 117 - case PCI_DEVICE_ID_INTEL_NTB_PS_IVT: 118 118 - case PCI_DEVICE_ID_INTEL_NTB_PS_HSX: 119 119 - case PCI_DEVICE_ID_INTEL_NTB_B2B_JSF: 120 120 - case PCI_DEVICE_ID_INTEL_NTB_B2B_SNB: 121 121 - case PCI_DEVICE_ID_INTEL_NTB_B2B_IVT: 122 122 - case PCI_DEVICE_ID_INTEL_NTB_B2B_HSX: 123 123 - return 1; 124 124 - default: 125 125 - return 0; 126 126 - } 127 127 - 128 128 - return 0; 129 129 - } 130 130 - 131 131 - static int is_ntb_atom(struct ntb_device *ndev) 132 132 - { 133 133 - switch (ndev->pdev->device) { 134 134 - case PCI_DEVICE_ID_INTEL_NTB_B2B_BWD: 135 135 - return 1; 136 136 - default: 137 137 - return 0; 138 138 - } 139 139 - 140 140 - return 0; 141 141 - } 142 142 - 143 143 - static void ntb_set_errata_flags(struct ntb_device *ndev) 144 144 - { 145 145 - switch (ndev->pdev->device) { 146 146 - /* 147 147 - * this workaround applies to all platform up to IvyBridge 148 148 - * Haswell has splitbar support and use a different workaround 149 149 - */ 150 150 - case PCI_DEVICE_ID_INTEL_NTB_SS_JSF: 151 151 - case PCI_DEVICE_ID_INTEL_NTB_SS_SNB: 152 152 - case PCI_DEVICE_ID_INTEL_NTB_SS_IVT: 153 153 - case PCI_DEVICE_ID_INTEL_NTB_SS_HSX: 154 154 - case PCI_DEVICE_ID_INTEL_NTB_PS_JSF: 155 155 - case PCI_DEVICE_ID_INTEL_NTB_PS_SNB: 156 156 - case PCI_DEVICE_ID_INTEL_NTB_PS_IVT: 157 157 - case PCI_DEVICE_ID_INTEL_NTB_PS_HSX: 158 158 - case PCI_DEVICE_ID_INTEL_NTB_B2B_JSF: 159 159 - case PCI_DEVICE_ID_INTEL_NTB_B2B_SNB: 160 160 - case PCI_DEVICE_ID_INTEL_NTB_B2B_IVT: 161 161 - case PCI_DEVICE_ID_INTEL_NTB_B2B_HSX: 162 162 - ndev->wa_flags |= WA_SNB_ERR; 163 163 - break; 164 164 - } 165 165 - } 166 166 - 167 167 - /** 168 168 - * ntb_register_event_callback() - register event callback 169 169 - * @ndev: pointer to ntb_device instance 170 170 - * @func: callback function to register 171 171 - * 172 172 - * This function registers a callback for any HW driver events such as link 173 173 - * up/down, power management notices and etc. 174 174 - * 175 175 - * RETURNS: An appropriate -ERRNO error value on error, or zero for success. 176 176 - */ 177 177 - int ntb_register_event_callback(struct ntb_device *ndev, 178 178 - void (*func)(void *handle, 179 179 - enum ntb_hw_event event)) 180 180 - { 181 181 - if (ndev->event_cb) 182 182 - return -EINVAL; 183 183 - 184 184 - ndev->event_cb = func; 185 185 - 186 186 - return 0; 187 187 - } 188 188 - 189 189 - /** 190 190 - * ntb_unregister_event_callback() - unregisters the event callback 191 191 - * @ndev: pointer to ntb_device instance 192 192 - * 193 193 - * This function unregisters the existing callback from transport 194 194 - */ 195 195 - void ntb_unregister_event_callback(struct ntb_device *ndev) 196 196 - { 197 197 - ndev->event_cb = NULL; 198 198 - } 199 199 - 200 200 - static void ntb_irq_work(unsigned long data) 201 201 - { 202 202 - struct ntb_db_cb *db_cb = (struct ntb_db_cb *)data; 203 203 - int rc; 204 204 - 205 205 - rc = db_cb->callback(db_cb->data, db_cb->db_num); 206 206 - if (rc) 207 207 - tasklet_schedule(&db_cb->irq_work); 208 208 - else { 209 209 - struct ntb_device *ndev = db_cb->ndev; 210 210 - unsigned long mask; 211 211 - 212 212 - mask = readw(ndev->reg_ofs.ldb_mask); 213 213 - clear_bit(db_cb->db_num * ndev->bits_per_vector, &mask); 214 214 - writew(mask, ndev->reg_ofs.ldb_mask); 215 215 - } 216 216 - } 217 217 - 218 218 - /** 219 219 - * ntb_register_db_callback() - register a callback for doorbell interrupt 220 220 - * @ndev: pointer to ntb_device instance 221 221 - * @idx: doorbell index to register callback, zero based 222 222 - * @data: pointer to be returned to caller with every callback 223 223 - * @func: callback function to register 224 224 - * 225 225 - * This function registers a callback function for the doorbell interrupt 226 226 - * on the primary side. The function will unmask the doorbell as well to 227 227 - * allow interrupt. 228 228 - * 229 229 - * RETURNS: An appropriate -ERRNO error value on error, or zero for success. 230 230 - */ 231 231 - int ntb_register_db_callback(struct ntb_device *ndev, unsigned int idx, 232 232 - void *data, int (*func)(void *data, int db_num)) 233 233 - { 234 234 - unsigned long mask; 235 235 - 236 236 - if (idx >= ndev->max_cbs || ndev->db_cb[idx].callback) { 237 237 - dev_warn(&ndev->pdev->dev, "Invalid Index.\n"); 238 238 - return -EINVAL; 239 239 - } 240 240 - 241 241 - ndev->db_cb[idx].callback = func; 242 242 - ndev->db_cb[idx].data = data; 243 243 - ndev->db_cb[idx].ndev = ndev; 244 244 - 245 245 - tasklet_init(&ndev->db_cb[idx].irq_work, ntb_irq_work, 246 246 - (unsigned long) &ndev->db_cb[idx]); 247 247 - 248 248 - /* unmask interrupt */ 249 249 - mask = readw(ndev->reg_ofs.ldb_mask); 250 250 - clear_bit(idx * ndev->bits_per_vector, &mask); 251 251 - writew(mask, ndev->reg_ofs.ldb_mask); 252 252 - 253 253 - return 0; 254 254 - } 255 255 - 256 256 - /** 257 257 - * ntb_unregister_db_callback() - unregister a callback for doorbell interrupt 258 258 - * @ndev: pointer to ntb_device instance 259 259 - * @idx: doorbell index to register callback, zero based 260 260 - * 261 261 - * This function unregisters a callback function for the doorbell interrupt 262 262 - * on the primary side. The function will also mask the said doorbell. 263 263 - */ 264 264 - void ntb_unregister_db_callback(struct ntb_device *ndev, unsigned int idx) 265 265 - { 266 266 - unsigned long mask; 267 267 - 268 268 - if (idx >= ndev->max_cbs || !ndev->db_cb[idx].callback) 269 269 - return; 270 270 - 271 271 - mask = readw(ndev->reg_ofs.ldb_mask); 272 272 - set_bit(idx * ndev->bits_per_vector, &mask); 273 273 - writew(mask, ndev->reg_ofs.ldb_mask); 274 274 - 275 275 - tasklet_disable(&ndev->db_cb[idx].irq_work); 276 276 - 277 277 - ndev->db_cb[idx].callback = NULL; 278 278 - } 279 279 - 280 280 - /** 281 281 - * ntb_find_transport() - find the transport pointer 282 282 - * @transport: pointer to pci device 283 283 - * 284 284 - * Given the pci device pointer, return the transport pointer passed in when 285 285 - * the transport attached when it was inited. 286 286 - * 287 287 - * RETURNS: pointer to transport. 288 288 - */ 289 289 - void *ntb_find_transport(struct pci_dev *pdev) 290 290 - { 291 291 - struct ntb_device *ndev = pci_get_drvdata(pdev); 292 292 - return ndev->ntb_transport; 293 293 - } 294 294 - 295 295 - /** 296 296 - * ntb_register_transport() - Register NTB transport with NTB HW driver 297 297 - * @transport: transport identifier 298 298 - * 299 299 - * This function allows a transport to reserve the hardware driver for 300 300 - * NTB usage. 301 301 - * 302 302 - * RETURNS: pointer to ntb_device, NULL on error. 303 303 - */ 304 304 - struct ntb_device *ntb_register_transport(struct pci_dev *pdev, void *transport) 305 305 - { 306 306 - struct ntb_device *ndev = pci_get_drvdata(pdev); 307 307 - 308 308 - if (ndev->ntb_transport) 309 309 - return NULL; 310 310 - 311 311 - ndev->ntb_transport = transport; 312 312 - return ndev; 313 313 - } 314 314 - 315 315 - /** 316 316 - * ntb_unregister_transport() - Unregister the transport with the NTB HW driver 317 317 - * @ndev - ntb_device of the transport to be freed 318 318 - * 319 319 - * This function unregisters the transport from the HW driver and performs any 320 320 - * necessary cleanups. 321 321 - */ 322 322 - void ntb_unregister_transport(struct ntb_device *ndev) 323 323 - { 324 324 - int i; 325 325 - 326 326 - if (!ndev->ntb_transport) 327 327 - return; 328 328 - 329 329 - for (i = 0; i < ndev->max_cbs; i++) 330 330 - ntb_unregister_db_callback(ndev, i); 331 331 - 332 332 - ntb_unregister_event_callback(ndev); 333 333 - ndev->ntb_transport = NULL; 334 334 - } 335 335 - 336 336 - /** 337 337 - * ntb_write_local_spad() - write to the secondary scratchpad register 338 338 - * @ndev: pointer to ntb_device instance 339 339 - * @idx: index to the scratchpad register, 0 based 340 340 - * @val: the data value to put into the register 341 341 - * 342 342 - * This function allows writing of a 32bit value to the indexed scratchpad 343 343 - * register. This writes over the data mirrored to the local scratchpad register 344 344 - * by the remote system. 345 345 - * 346 346 - * RETURNS: An appropriate -ERRNO error value on error, or zero for success. 347 347 - */ 348 348 - int ntb_write_local_spad(struct ntb_device *ndev, unsigned int idx, u32 val) 349 349 - { 350 350 - if (idx >= ndev->limits.max_spads) 351 351 - return -EINVAL; 352 352 - 353 353 - dev_dbg(&ndev->pdev->dev, "Writing %x to local scratch pad index %d\n", 354 354 - val, idx); 355 355 - writel(val, ndev->reg_ofs.spad_read + idx * 4); 356 356 - 357 357 - return 0; 358 358 - } 359 359 - 360 360 - /** 361 361 - * ntb_read_local_spad() - read from the primary scratchpad register 362 362 - * @ndev: pointer to ntb_device instance 363 363 - * @idx: index to scratchpad register, 0 based 364 364 - * @val: pointer to 32bit integer for storing the register value 365 365 - * 366 366 - * This function allows reading of the 32bit scratchpad register on 367 367 - * the primary (internal) side. This allows the local system to read data 368 368 - * written and mirrored to the scratchpad register by the remote system. 369 369 - * 370 370 - * RETURNS: An appropriate -ERRNO error value on error, or zero for success. 371 371 - */ 372 372 - int ntb_read_local_spad(struct ntb_device *ndev, unsigned int idx, u32 *val) 373 373 - { 374 374 - if (idx >= ndev->limits.max_spads) 375 375 - return -EINVAL; 376 376 - 377 377 - *val = readl(ndev->reg_ofs.spad_write + idx * 4); 378 378 - dev_dbg(&ndev->pdev->dev, 379 379 - "Reading %x from local scratch pad index %d\n", *val, idx); 380 380 - 381 381 - return 0; 382 382 - } 383 383 - 384 384 - /** 385 385 - * ntb_write_remote_spad() - write to the secondary scratchpad register 386 386 - * @ndev: pointer to ntb_device instance 387 387 - * @idx: index to the scratchpad register, 0 based 388 388 - * @val: the data value to put into the register 389 389 - * 390 390 - * This function allows writing of a 32bit value to the indexed scratchpad 391 391 - * register. The register resides on the secondary (external) side. This allows 392 392 - * the local system to write data to be mirrored to the remote systems 393 393 - * scratchpad register. 394 394 - * 395 395 - * RETURNS: An appropriate -ERRNO error value on error, or zero for success. 396 396 - */ 397 397 - int ntb_write_remote_spad(struct ntb_device *ndev, unsigned int idx, u32 val) 398 398 - { 399 399 - if (idx >= ndev->limits.max_spads) 400 400 - return -EINVAL; 401 401 - 402 402 - dev_dbg(&ndev->pdev->dev, "Writing %x to remote scratch pad index %d\n", 403 403 - val, idx); 404 404 - writel(val, ndev->reg_ofs.spad_write + idx * 4); 405 405 - 406 406 - return 0; 407 407 - } 408 408 - 409 409 - /** 410 410 - * ntb_read_remote_spad() - read from the primary scratchpad register 411 411 - * @ndev: pointer to ntb_device instance 412 412 - * @idx: index to scratchpad register, 0 based 413 413 - * @val: pointer to 32bit integer for storing the register value 414 414 - * 415 415 - * This function allows reading of the 32bit scratchpad register on 416 416 - * the primary (internal) side. This alloows the local system to read the data 417 417 - * it wrote to be mirrored on the remote system. 418 418 - * 419 419 - * RETURNS: An appropriate -ERRNO error value on error, or zero for success. 420 420 - */ 421 421 - int ntb_read_remote_spad(struct ntb_device *ndev, unsigned int idx, u32 *val) 422 422 - { 423 423 - if (idx >= ndev->limits.max_spads) 424 424 - return -EINVAL; 425 425 - 426 426 - *val = readl(ndev->reg_ofs.spad_read + idx * 4); 427 427 - dev_dbg(&ndev->pdev->dev, 428 428 - "Reading %x from remote scratch pad index %d\n", *val, idx); 429 429 - 430 430 - return 0; 431 431 - } 432 432 - 433 433 - /** 434 434 - * ntb_get_mw_base() - get addr for the NTB memory window 435 435 - * @ndev: pointer to ntb_device instance 436 436 - * @mw: memory window number 437 437 - * 438 438 - * This function provides the base address of the memory window specified. 439 439 - * 440 440 - * RETURNS: address, or NULL on error. 441 441 - */ 442 442 - resource_size_t ntb_get_mw_base(struct ntb_device *ndev, unsigned int mw) 443 443 - { 444 444 - if (mw >= ntb_max_mw(ndev)) 445 445 - return 0; 446 446 - 447 447 - return pci_resource_start(ndev->pdev, MW_TO_BAR(mw)); 448 448 - } 449 449 - 450 450 - /** 451 451 - * ntb_get_mw_vbase() - get virtual addr for the NTB memory window 452 452 - * @ndev: pointer to ntb_device instance 453 453 - * @mw: memory window number 454 454 - * 455 455 - * This function provides the base virtual address of the memory window 456 456 - * specified. 457 457 - * 458 458 - * RETURNS: pointer to virtual address, or NULL on error. 459 459 - */ 460 460 - void __iomem *ntb_get_mw_vbase(struct ntb_device *ndev, unsigned int mw) 461 461 - { 462 462 - if (mw >= ntb_max_mw(ndev)) 463 463 - return NULL; 464 464 - 465 465 - return ndev->mw[mw].vbase; 466 466 - } 467 467 - 468 468 - /** 469 469 - * ntb_get_mw_size() - return size of NTB memory window 470 470 - * @ndev: pointer to ntb_device instance 471 471 - * @mw: memory window number 472 472 - * 473 473 - * This function provides the physical size of the memory window specified 474 474 - * 475 475 - * RETURNS: the size of the memory window or zero on error 476 476 - */ 477 477 - u64 ntb_get_mw_size(struct ntb_device *ndev, unsigned int mw) 478 478 - { 479 479 - if (mw >= ntb_max_mw(ndev)) 480 480 - return 0; 481 481 - 482 482 - return ndev->mw[mw].bar_sz; 483 483 - } 484 484 - 485 485 - /** 486 486 - * ntb_set_mw_addr - set the memory window address 487 487 - * @ndev: pointer to ntb_device instance 488 488 - * @mw: memory window number 489 489 - * @addr: base address for data 490 490 - * 491 491 - * This function sets the base physical address of the memory window. This 492 492 - * memory address is where data from the remote system will be transfered into 493 493 - * or out of depending on how the transport is configured. 494 494 - */ 495 495 - void ntb_set_mw_addr(struct ntb_device *ndev, unsigned int mw, u64 addr) 496 496 - { 497 497 - if (mw >= ntb_max_mw(ndev)) 498 498 - return; 499 499 - 500 500 - dev_dbg(&ndev->pdev->dev, "Writing addr %Lx to BAR %d\n", addr, 501 501 - MW_TO_BAR(mw)); 502 502 - 503 503 - ndev->mw[mw].phys_addr = addr; 504 504 - 505 505 - switch (MW_TO_BAR(mw)) { 506 506 - case NTB_BAR_23: 507 507 - writeq(addr, ndev->reg_ofs.bar2_xlat); 508 508 - break; 509 509 - case NTB_BAR_4: 510 510 - if (ndev->split_bar) 511 511 - writel(addr, ndev->reg_ofs.bar4_xlat); 512 512 - else 513 513 - writeq(addr, ndev->reg_ofs.bar4_xlat); 514 514 - break; 515 515 - case NTB_BAR_5: 516 516 - writel(addr, ndev->reg_ofs.bar5_xlat); 517 517 - break; 518 518 - } 519 519 - } 520 520 - 521 521 - /** 522 522 - * ntb_ring_doorbell() - Set the doorbell on the secondary/external side 523 523 - * @ndev: pointer to ntb_device instance 524 524 - * @db: doorbell to ring 525 525 - * 526 526 - * This function allows triggering of a doorbell on the secondary/external 527 527 - * side that will initiate an interrupt on the remote host 528 528 - * 529 529 - * RETURNS: An appropriate -ERRNO error value on error, or zero for success. 530 530 - */ 531 531 - void ntb_ring_doorbell(struct ntb_device *ndev, unsigned int db) 532 532 - { 533 533 - dev_dbg(&ndev->pdev->dev, "%s: ringing doorbell %d\n", __func__, db); 534 534 - 535 535 - if (ndev->hw_type == BWD_HW) 536 536 - writeq((u64) 1 << db, ndev->reg_ofs.rdb); 537 537 - else 538 538 - writew(((1 << ndev->bits_per_vector) - 1) << 539 539 - (db * ndev->bits_per_vector), ndev->reg_ofs.rdb); 540 540 - } 541 541 - 542 542 - static void bwd_recover_link(struct ntb_device *ndev) 543 543 - { 544 544 - u32 status; 545 545 - 546 546 - /* Driver resets the NTB ModPhy lanes - magic! */ 547 547 - writeb(0xe0, ndev->reg_base + BWD_MODPHY_PCSREG6); 548 548 - writeb(0x40, ndev->reg_base + BWD_MODPHY_PCSREG4); 549 549 - writeb(0x60, ndev->reg_base + BWD_MODPHY_PCSREG4); 550 550 - writeb(0x60, ndev->reg_base + BWD_MODPHY_PCSREG6); 551 551 - 552 552 - /* Driver waits 100ms to allow the NTB ModPhy to settle */ 553 553 - msleep(100); 554 554 - 555 555 - /* Clear AER Errors, write to clear */ 556 556 - status = readl(ndev->reg_base + BWD_ERRCORSTS_OFFSET); 557 557 - dev_dbg(&ndev->pdev->dev, "ERRCORSTS = %x\n", status); 558 558 - status &= PCI_ERR_COR_REP_ROLL; 559 559 - writel(status, ndev->reg_base + BWD_ERRCORSTS_OFFSET); 560 560 - 561 561 - /* Clear unexpected electrical idle event in LTSSM, write to clear */ 562 562 - status = readl(ndev->reg_base + BWD_LTSSMERRSTS0_OFFSET); 563 563 - dev_dbg(&ndev->pdev->dev, "LTSSMERRSTS0 = %x\n", status); 564 564 - status |= BWD_LTSSMERRSTS0_UNEXPECTEDEI; 565 565 - writel(status, ndev->reg_base + BWD_LTSSMERRSTS0_OFFSET); 566 566 - 567 567 - /* Clear DeSkew Buffer error, write to clear */ 568 568 - status = readl(ndev->reg_base + BWD_DESKEWSTS_OFFSET); 569 569 - dev_dbg(&ndev->pdev->dev, "DESKEWSTS = %x\n", status); 570 570 - status |= BWD_DESKEWSTS_DBERR; 571 571 - writel(status, ndev->reg_base + BWD_DESKEWSTS_OFFSET); 572 572 - 573 573 - status = readl(ndev->reg_base + BWD_IBSTERRRCRVSTS0_OFFSET); 574 574 - dev_dbg(&ndev->pdev->dev, "IBSTERRRCRVSTS0 = %x\n", status); 575 575 - status &= BWD_IBIST_ERR_OFLOW; 576 576 - writel(status, ndev->reg_base + BWD_IBSTERRRCRVSTS0_OFFSET); 577 577 - 578 578 - /* Releases the NTB state machine to allow the link to retrain */ 579 579 - status = readl(ndev->reg_base + BWD_LTSSMSTATEJMP_OFFSET); 580 580 - dev_dbg(&ndev->pdev->dev, "LTSSMSTATEJMP = %x\n", status); 581 581 - status &= ~BWD_LTSSMSTATEJMP_FORCEDETECT; 582 582 - writel(status, ndev->reg_base + BWD_LTSSMSTATEJMP_OFFSET); 583 583 - } 584 584 - 585 585 - static void ntb_link_event(struct ntb_device *ndev, int link_state) 586 586 - { 587 587 - unsigned int event; 588 588 - 589 589 - if (ndev->link_status == link_state) 590 590 - return; 591 591 - 592 592 - if (link_state == NTB_LINK_UP) { 593 593 - u16 status; 594 594 - 595 595 - dev_info(&ndev->pdev->dev, "Link Up\n"); 596 596 - ndev->link_status = NTB_LINK_UP; 597 597 - event = NTB_EVENT_HW_LINK_UP; 598 598 - 599 599 - if (is_ntb_atom(ndev) || 600 600 - ndev->conn_type == NTB_CONN_TRANSPARENT) 601 601 - status = readw(ndev->reg_ofs.lnk_stat); 602 602 - else { 603 603 - int rc = pci_read_config_word(ndev->pdev, 604 604 - SNB_LINK_STATUS_OFFSET, 605 605 - &status); 606 606 - if (rc) 607 607 - return; 608 608 - } 609 609 - 610 610 - ndev->link_width = (status & NTB_LINK_WIDTH_MASK) >> 4; 611 611 - ndev->link_speed = (status & NTB_LINK_SPEED_MASK); 612 612 - dev_info(&ndev->pdev->dev, "Link Width %d, Link Speed %d\n", 613 613 - ndev->link_width, ndev->link_speed); 614 614 - } else { 615 615 - dev_info(&ndev->pdev->dev, "Link Down\n"); 616 616 - ndev->link_status = NTB_LINK_DOWN; 617 617 - event = NTB_EVENT_HW_LINK_DOWN; 618 618 - /* Don't modify link width/speed, we need it in link recovery */ 619 619 - } 620 620 - 621 621 - /* notify the upper layer if we have an event change */ 622 622 - if (ndev->event_cb) 623 623 - ndev->event_cb(ndev->ntb_transport, event); 624 624 - } 625 625 - 626 626 - static int ntb_link_status(struct ntb_device *ndev) 627 627 - { 628 628 - int link_state; 629 629 - 630 630 - if (is_ntb_atom(ndev)) { 631 631 - u32 ntb_cntl; 632 632 - 633 633 - ntb_cntl = readl(ndev->reg_ofs.lnk_cntl); 634 634 - if (ntb_cntl & BWD_CNTL_LINK_DOWN) 635 635 - link_state = NTB_LINK_DOWN; 636 636 - else 637 637 - link_state = NTB_LINK_UP; 638 638 - } else { 639 639 - u16 status; 640 640 - int rc; 641 641 - 642 642 - rc = pci_read_config_word(ndev->pdev, SNB_LINK_STATUS_OFFSET, 643 643 - &status); 644 644 - if (rc) 645 645 - return rc; 646 646 - 647 647 - if (status & NTB_LINK_STATUS_ACTIVE) 648 648 - link_state = NTB_LINK_UP; 649 649 - else 650 650 - link_state = NTB_LINK_DOWN; 651 651 - } 652 652 - 653 653 - ntb_link_event(ndev, link_state); 654 654 - 655 655 - return 0; 656 656 - } 657 657 - 658 658 - static void bwd_link_recovery(struct work_struct *work) 659 659 - { 660 660 - struct ntb_device *ndev = container_of(work, struct ntb_device, 661 661 - lr_timer.work); 662 662 - u32 status32; 663 663 - 664 664 - bwd_recover_link(ndev); 665 665 - /* There is a potential race between the 2 NTB devices recovering at the 666 666 - * same time. If the times are the same, the link will not recover and 667 667 - * the driver will be stuck in this loop forever. Add a random interval 668 668 - * to the recovery time to prevent this race. 669 669 - */ 670 670 - msleep(BWD_LINK_RECOVERY_TIME + prandom_u32() % BWD_LINK_RECOVERY_TIME); 671 671 - 672 672 - status32 = readl(ndev->reg_base + BWD_LTSSMSTATEJMP_OFFSET); 673 673 - if (status32 & BWD_LTSSMSTATEJMP_FORCEDETECT) 674 674 - goto retry; 675 675 - 676 676 - status32 = readl(ndev->reg_base + BWD_IBSTERRRCRVSTS0_OFFSET); 677 677 - if (status32 & BWD_IBIST_ERR_OFLOW) 678 678 - goto retry; 679 679 - 680 680 - status32 = readl(ndev->reg_ofs.lnk_cntl); 681 681 - if (!(status32 & BWD_CNTL_LINK_DOWN)) { 682 682 - unsigned char speed, width; 683 683 - u16 status16; 684 684 - 685 685 - status16 = readw(ndev->reg_ofs.lnk_stat); 686 686 - width = (status16 & NTB_LINK_WIDTH_MASK) >> 4; 687 687 - speed = (status16 & NTB_LINK_SPEED_MASK); 688 688 - if (ndev->link_width != width || ndev->link_speed != speed) 689 689 - goto retry; 690 690 - } 691 691 - 692 692 - schedule_delayed_work(&ndev->hb_timer, NTB_HB_TIMEOUT); 693 693 - return; 694 694 - 695 695 - retry: 696 696 - schedule_delayed_work(&ndev->lr_timer, NTB_HB_TIMEOUT); 697 697 - } 698 698 - 699 699 - /* BWD doesn't have link status interrupt, poll on that platform */ 700 700 - static void bwd_link_poll(struct work_struct *work) 701 701 - { 702 702 - struct ntb_device *ndev = container_of(work, struct ntb_device, 703 703 - hb_timer.work); 704 704 - unsigned long ts = jiffies; 705 705 - 706 706 - /* If we haven't gotten an interrupt in a while, check the BWD link 707 707 - * status bit 708 708 - */ 709 709 - if (ts > ndev->last_ts + NTB_HB_TIMEOUT) { 710 710 - int rc = ntb_link_status(ndev); 711 711 - if (rc) 712 712 - dev_err(&ndev->pdev->dev, 713 713 - "Error determining link status\n"); 714 714 - 715 715 - /* Check to see if a link error is the cause of the link down */ 716 716 - if (ndev->link_status == NTB_LINK_DOWN) { 717 717 - u32 status32 = readl(ndev->reg_base + 718 718 - BWD_LTSSMSTATEJMP_OFFSET); 719 719 - if (status32 & BWD_LTSSMSTATEJMP_FORCEDETECT) { 720 720 - schedule_delayed_work(&ndev->lr_timer, 0); 721 721 - return; 722 722 - } 723 723 - } 724 724 - } 725 725 - 726 726 - schedule_delayed_work(&ndev->hb_timer, NTB_HB_TIMEOUT); 727 727 - } 728 728 - 729 729 - static int ntb_xeon_setup(struct ntb_device *ndev) 730 730 - { 731 731 - switch (ndev->conn_type) { 732 732 - case NTB_CONN_B2B: 733 733 - ndev->reg_ofs.ldb = ndev->reg_base + SNB_PDOORBELL_OFFSET; 734 734 - ndev->reg_ofs.ldb_mask = ndev->reg_base + SNB_PDBMSK_OFFSET; 735 735 - ndev->reg_ofs.spad_read = ndev->reg_base + SNB_SPAD_OFFSET; 736 736 - ndev->reg_ofs.bar2_xlat = ndev->reg_base + SNB_SBAR2XLAT_OFFSET; 737 737 - ndev->reg_ofs.bar4_xlat = ndev->reg_base + SNB_SBAR4XLAT_OFFSET; 738 738 - if (ndev->split_bar) 739 739 - ndev->reg_ofs.bar5_xlat = 740 740 - ndev->reg_base + SNB_SBAR5XLAT_OFFSET; 741 741 - ndev->limits.max_spads = SNB_MAX_B2B_SPADS; 742 742 - 743 743 - /* There is a Xeon hardware errata related to writes to 744 744 - * SDOORBELL or B2BDOORBELL in conjunction with inbound access 745 745 - * to NTB MMIO Space, which may hang the system. To workaround 746 746 - * this use the second memory window to access the interrupt and 747 747 - * scratch pad registers on the remote system. 748 748 - */ 749 749 - if (ndev->wa_flags & WA_SNB_ERR) { 750 750 - if (!ndev->mw[ndev->limits.max_mw - 1].bar_sz) 751 751 - return -EINVAL; 752 752 - 753 753 - ndev->limits.max_db_bits = SNB_MAX_DB_BITS; 754 754 - ndev->reg_ofs.spad_write = 755 755 - ndev->mw[ndev->limits.max_mw - 1].vbase + 756 756 - SNB_SPAD_OFFSET; 757 757 - ndev->reg_ofs.rdb = 758 758 - ndev->mw[ndev->limits.max_mw - 1].vbase + 759 759 - SNB_PDOORBELL_OFFSET; 760 760 - 761 761 - /* Set the Limit register to 4k, the minimum size, to 762 762 - * prevent an illegal access 763 763 - */ 764 764 - writeq(ndev->mw[1].bar_sz + 0x1000, ndev->reg_base + 765 765 - SNB_PBAR4LMT_OFFSET); 766 766 - /* HW errata on the Limit registers. They can only be 767 767 - * written when the base register is 4GB aligned and 768 768 - * < 32bit. This should already be the case based on 769 769 - * the driver defaults, but write the Limit registers 770 770 - * first just in case. 771 771 - */ 772 772 - 773 773 - ndev->limits.max_mw = SNB_ERRATA_MAX_MW; 774 774 - } else { 775 775 - /* HW Errata on bit 14 of b2bdoorbell register. Writes 776 776 - * will not be mirrored to the remote system. Shrink 777 777 - * the number of bits by one, since bit 14 is the last 778 778 - * bit. 779 779 - */ 780 780 - ndev->limits.max_db_bits = SNB_MAX_DB_BITS - 1; 781 781 - ndev->reg_ofs.spad_write = ndev->reg_base + 782 782 - SNB_B2B_SPAD_OFFSET; 783 783 - ndev->reg_ofs.rdb = ndev->reg_base + 784 784 - SNB_B2B_DOORBELL_OFFSET; 785 785 - 786 786 - /* Disable the Limit register, just incase it is set to 787 787 - * something silly. A 64bit write should handle it 788 788 - * regardless of whether it has a split BAR or not. 789 789 - */ 790 790 - writeq(0, ndev->reg_base + SNB_PBAR4LMT_OFFSET); 791 791 - /* HW errata on the Limit registers. They can only be 792 792 - * written when the base register is 4GB aligned and 793 793 - * < 32bit. This should already be the case based on 794 794 - * the driver defaults, but write the Limit registers 795 795 - * first just in case. 796 796 - */ 797 797 - if (ndev->split_bar) 798 798 - ndev->limits.max_mw = HSX_SPLITBAR_MAX_MW; 799 799 - else 800 800 - ndev->limits.max_mw = SNB_MAX_MW; 801 801 - } 802 802 - 803 803 - /* The Xeon errata workaround requires setting SBAR Base 804 804 - * addresses to known values, so that the PBAR XLAT can be 805 805 - * pointed at SBAR0 of the remote system. 806 806 - */ 807 807 - if (ndev->dev_type == NTB_DEV_USD) { 808 808 - writeq(SNB_MBAR23_DSD_ADDR, ndev->reg_base + 809 809 - SNB_PBAR2XLAT_OFFSET); 810 810 - if (ndev->wa_flags & WA_SNB_ERR) 811 811 - writeq(SNB_MBAR01_DSD_ADDR, ndev->reg_base + 812 812 - SNB_PBAR4XLAT_OFFSET); 813 813 - else { 814 814 - if (ndev->split_bar) { 815 815 - writel(SNB_MBAR4_DSD_ADDR, 816 816 - ndev->reg_base + 817 817 - SNB_PBAR4XLAT_OFFSET); 818 818 - writel(SNB_MBAR5_DSD_ADDR, 819 819 - ndev->reg_base + 820 820 - SNB_PBAR5XLAT_OFFSET); 821 821 - } else 822 822 - writeq(SNB_MBAR4_DSD_ADDR, 823 823 - ndev->reg_base + 824 824 - SNB_PBAR4XLAT_OFFSET); 825 825 - 826 826 - /* B2B_XLAT_OFFSET is a 64bit register, but can 827 827 - * only take 32bit writes 828 828 - */ 829 829 - writel(SNB_MBAR01_DSD_ADDR & 0xffffffff, 830 830 - ndev->reg_base + SNB_B2B_XLAT_OFFSETL); 831 831 - writel(SNB_MBAR01_DSD_ADDR >> 32, 832 832 - ndev->reg_base + SNB_B2B_XLAT_OFFSETU); 833 833 - } 834 834 - 835 835 - writeq(SNB_MBAR01_USD_ADDR, ndev->reg_base + 836 836 - SNB_SBAR0BASE_OFFSET); 837 837 - writeq(SNB_MBAR23_USD_ADDR, ndev->reg_base + 838 838 - SNB_SBAR2BASE_OFFSET); 839 839 - if (ndev->split_bar) { 840 840 - writel(SNB_MBAR4_USD_ADDR, ndev->reg_base + 841 841 - SNB_SBAR4BASE_OFFSET); 842 842 - writel(SNB_MBAR5_USD_ADDR, ndev->reg_base + 843 843 - SNB_SBAR5BASE_OFFSET); 844 844 - } else 845 845 - writeq(SNB_MBAR4_USD_ADDR, ndev->reg_base + 846 846 - SNB_SBAR4BASE_OFFSET); 847 847 - } else { 848 848 - writeq(SNB_MBAR23_USD_ADDR, ndev->reg_base + 849 849 - SNB_PBAR2XLAT_OFFSET); 850 850 - if (ndev->wa_flags & WA_SNB_ERR) 851 851 - writeq(SNB_MBAR01_USD_ADDR, ndev->reg_base + 852 852 - SNB_PBAR4XLAT_OFFSET); 853 853 - else { 854 854 - if (ndev->split_bar) { 855 855 - writel(SNB_MBAR4_USD_ADDR, 856 856 - ndev->reg_base + 857 857 - SNB_PBAR4XLAT_OFFSET); 858 858 - writel(SNB_MBAR5_USD_ADDR, 859 859 - ndev->reg_base + 860 860 - SNB_PBAR5XLAT_OFFSET); 861 861 - } else 862 862 - writeq(SNB_MBAR4_USD_ADDR, 863 863 - ndev->reg_base + 864 864 - SNB_PBAR4XLAT_OFFSET); 865 865 - 866 866 - /* 867 867 - * B2B_XLAT_OFFSET is a 64bit register, but can 868 868 - * only take 32bit writes 869 869 - */ 870 870 - writel(SNB_MBAR01_USD_ADDR & 0xffffffff, 871 871 - ndev->reg_base + SNB_B2B_XLAT_OFFSETL); 872 872 - writel(SNB_MBAR01_USD_ADDR >> 32, 873 873 - ndev->reg_base + SNB_B2B_XLAT_OFFSETU); 874 874 - } 875 875 - writeq(SNB_MBAR01_DSD_ADDR, ndev->reg_base + 876 876 - SNB_SBAR0BASE_OFFSET); 877 877 - writeq(SNB_MBAR23_DSD_ADDR, ndev->reg_base + 878 878 - SNB_SBAR2BASE_OFFSET); 879 879 - if (ndev->split_bar) { 880 880 - writel(SNB_MBAR4_DSD_ADDR, ndev->reg_base + 881 881 - SNB_SBAR4BASE_OFFSET); 882 882 - writel(SNB_MBAR5_DSD_ADDR, ndev->reg_base + 883 883 - SNB_SBAR5BASE_OFFSET); 884 884 - } else 885 885 - writeq(SNB_MBAR4_DSD_ADDR, ndev->reg_base + 886 886 - SNB_SBAR4BASE_OFFSET); 887 887 - 888 888 - } 889 889 - break; 890 890 - case NTB_CONN_RP: 891 891 - if (ndev->wa_flags & WA_SNB_ERR) { 892 892 - dev_err(&ndev->pdev->dev, 893 893 - "NTB-RP disabled due to hardware errata.\n"); 894 894 - return -EINVAL; 895 895 - } 896 896 - 897 897 - /* Scratch pads need to have exclusive access from the primary 898 898 - * or secondary side. Halve the num spads so that each side can 899 899 - * have an equal amount. 900 900 - */ 901 901 - ndev->limits.max_spads = SNB_MAX_COMPAT_SPADS / 2; 902 902 - ndev->limits.max_db_bits = SNB_MAX_DB_BITS; 903 903 - /* Note: The SDOORBELL is the cause of the errata. You REALLY 904 904 - * don't want to touch it. 905 905 - */ 906 906 - ndev->reg_ofs.rdb = ndev->reg_base + SNB_SDOORBELL_OFFSET; 907 907 - ndev->reg_ofs.ldb = ndev->reg_base + SNB_PDOORBELL_OFFSET; 908 908 - ndev->reg_ofs.ldb_mask = ndev->reg_base + SNB_PDBMSK_OFFSET; 909 909 - /* Offset the start of the spads to correspond to whether it is 910 910 - * primary or secondary 911 911 - */ 912 912 - ndev->reg_ofs.spad_write = ndev->reg_base + SNB_SPAD_OFFSET + 913 913 - ndev->limits.max_spads * 4; 914 914 - ndev->reg_ofs.spad_read = ndev->reg_base + SNB_SPAD_OFFSET; 915 915 - ndev->reg_ofs.bar2_xlat = ndev->reg_base + SNB_SBAR2XLAT_OFFSET; 916 916 - ndev->reg_ofs.bar4_xlat = ndev->reg_base + SNB_SBAR4XLAT_OFFSET; 917 917 - if (ndev->split_bar) { 918 918 - ndev->reg_ofs.bar5_xlat = 919 919 - ndev->reg_base + SNB_SBAR5XLAT_OFFSET; 920 920 - ndev->limits.max_mw = HSX_SPLITBAR_MAX_MW; 921 921 - } else 922 922 - ndev->limits.max_mw = SNB_MAX_MW; 923 923 - break; 924 924 - case NTB_CONN_TRANSPARENT: 925 925 - if (ndev->wa_flags & WA_SNB_ERR) { 926 926 - dev_err(&ndev->pdev->dev, 927 927 - "NTB-TRANSPARENT disabled due to hardware errata.\n"); 928 928 - return -EINVAL; 929 929 - } 930 930 - 931 931 - /* Scratch pads need to have exclusive access from the primary 932 932 - * or secondary side. Halve the num spads so that each side can 933 933 - * have an equal amount. 934 934 - */ 935 935 - ndev->limits.max_spads = SNB_MAX_COMPAT_SPADS / 2; 936 936 - ndev->limits.max_db_bits = SNB_MAX_DB_BITS; 937 937 - ndev->reg_ofs.rdb = ndev->reg_base + SNB_PDOORBELL_OFFSET; 938 938 - ndev->reg_ofs.ldb = ndev->reg_base + SNB_SDOORBELL_OFFSET; 939 939 - ndev->reg_ofs.ldb_mask = ndev->reg_base + SNB_SDBMSK_OFFSET; 940 940 - ndev->reg_ofs.spad_write = ndev->reg_base + SNB_SPAD_OFFSET; 941 941 - /* Offset the start of the spads to correspond to whether it is 942 942 - * primary or secondary 943 943 - */ 944 944 - ndev->reg_ofs.spad_read = ndev->reg_base + SNB_SPAD_OFFSET + 945 945 - ndev->limits.max_spads * 4; 946 946 - ndev->reg_ofs.bar2_xlat = ndev->reg_base + SNB_PBAR2XLAT_OFFSET; 947 947 - ndev->reg_ofs.bar4_xlat = ndev->reg_base + SNB_PBAR4XLAT_OFFSET; 948 948 - 949 949 - if (ndev->split_bar) { 950 950 - ndev->reg_ofs.bar5_xlat = 951 951 - ndev->reg_base + SNB_PBAR5XLAT_OFFSET; 952 952 - ndev->limits.max_mw = HSX_SPLITBAR_MAX_MW; 953 953 - } else 954 954 - ndev->limits.max_mw = SNB_MAX_MW; 955 955 - break; 956 956 - default: 957 957 - /* 958 958 - * we should never hit this. the detect function should've 959 959 - * take cared of everything. 960 960 - */ 961 961 - return -EINVAL; 962 962 - } 963 963 - 964 964 - ndev->reg_ofs.lnk_cntl = ndev->reg_base + SNB_NTBCNTL_OFFSET; 965 965 - ndev->reg_ofs.lnk_stat = ndev->reg_base + SNB_SLINK_STATUS_OFFSET; 966 966 - ndev->reg_ofs.spci_cmd = ndev->reg_base + SNB_PCICMD_OFFSET; 967 967 - 968 968 - ndev->limits.msix_cnt = SNB_MSIX_CNT; 969 969 - ndev->bits_per_vector = SNB_DB_BITS_PER_VEC; 970 970 - 971 971 - return 0; 972 972 - } 973 973 - 974 974 - static int ntb_bwd_setup(struct ntb_device *ndev) 975 975 - { 976 976 - int rc; 977 977 - u32 val; 978 978 - 979 979 - ndev->hw_type = BWD_HW; 980 980 - 981 981 - rc = pci_read_config_dword(ndev->pdev, NTB_PPD_OFFSET, &val); 982 982 - if (rc) 983 983 - return rc; 984 984 - 985 985 - switch ((val & BWD_PPD_CONN_TYPE) >> 8) { 986 986 - case NTB_CONN_B2B: 987 987 - ndev->conn_type = NTB_CONN_B2B; 988 988 - break; 989 989 - case NTB_CONN_RP: 990 990 - default: 991 991 - dev_err(&ndev->pdev->dev, "Unsupported NTB configuration\n"); 992 992 - return -EINVAL; 993 993 - } 994 994 - 995 995 - if (val & BWD_PPD_DEV_TYPE) 996 996 - ndev->dev_type = NTB_DEV_DSD; 997 997 - else 998 998 - ndev->dev_type = NTB_DEV_USD; 999 999 - 1000 1000 - /* Initiate PCI-E link training */ 1001 1001 - rc = pci_write_config_dword(ndev->pdev, NTB_PPD_OFFSET, 1002 1002 - val | BWD_PPD_INIT_LINK); 1003 1003 - if (rc) 1004 1004 - return rc; 1005 1005 - 1006 1006 - ndev->reg_ofs.ldb = ndev->reg_base + BWD_PDOORBELL_OFFSET; 1007 1007 - ndev->reg_ofs.ldb_mask = ndev->reg_base + BWD_PDBMSK_OFFSET; 1008 1008 - ndev->reg_ofs.rdb = ndev->reg_base + BWD_B2B_DOORBELL_OFFSET; 1009 1009 - ndev->reg_ofs.bar2_xlat = ndev->reg_base + BWD_SBAR2XLAT_OFFSET; 1010 1010 - ndev->reg_ofs.bar4_xlat = ndev->reg_base + BWD_SBAR4XLAT_OFFSET; 1011 1011 - ndev->reg_ofs.lnk_cntl = ndev->reg_base + BWD_NTBCNTL_OFFSET; 1012 1012 - ndev->reg_ofs.lnk_stat = ndev->reg_base + BWD_LINK_STATUS_OFFSET; 1013 1013 - ndev->reg_ofs.spad_read = ndev->reg_base + BWD_SPAD_OFFSET; 1014 1014 - ndev->reg_ofs.spad_write = ndev->reg_base + BWD_B2B_SPAD_OFFSET; 1015 1015 - ndev->reg_ofs.spci_cmd = ndev->reg_base + BWD_PCICMD_OFFSET; 1016 1016 - ndev->limits.max_mw = BWD_MAX_MW; 1017 1017 - ndev->limits.max_spads = BWD_MAX_SPADS; 1018 1018 - ndev->limits.max_db_bits = BWD_MAX_DB_BITS; 1019 1019 - ndev->limits.msix_cnt = BWD_MSIX_CNT; 1020 1020 - ndev->bits_per_vector = BWD_DB_BITS_PER_VEC; 1021 1021 - 1022 1022 - /* Since bwd doesn't have a link interrupt, setup a poll timer */ 1023 1023 - INIT_DELAYED_WORK(&ndev->hb_timer, bwd_link_poll); 1024 1024 - INIT_DELAYED_WORK(&ndev->lr_timer, bwd_link_recovery); 1025 1025 - schedule_delayed_work(&ndev->hb_timer, NTB_HB_TIMEOUT); 1026 1026 - 1027 1027 - return 0; 1028 1028 - } 1029 1029 - 1030 1030 - static int ntb_device_setup(struct ntb_device *ndev) 1031 1031 - { 1032 1032 - int rc; 1033 1033 - 1034 1034 - if (is_ntb_xeon(ndev)) 1035 1035 - rc = ntb_xeon_setup(ndev); 1036 1036 - else if (is_ntb_atom(ndev)) 1037 1037 - rc = ntb_bwd_setup(ndev); 1038 1038 - else 1039 1039 - rc = -ENODEV; 1040 1040 - 1041 1041 - if (rc) 1042 1042 - return rc; 1043 1043 - 1044 1044 - if (ndev->conn_type == NTB_CONN_B2B) 1045 1045 - /* Enable Bus Master and Memory Space on the secondary side */ 1046 1046 - writew(PCI_COMMAND_MEMORY | PCI_COMMAND_MASTER, 1047 1047 - ndev->reg_ofs.spci_cmd); 1048 1048 - 1049 1049 - return 0; 1050 1050 - } 1051 1051 - 1052 1052 - static void ntb_device_free(struct ntb_device *ndev) 1053 1053 - { 1054 1054 - if (is_ntb_atom(ndev)) { 1055 1055 - cancel_delayed_work_sync(&ndev->hb_timer); 1056 1056 - cancel_delayed_work_sync(&ndev->lr_timer); 1057 1057 - } 1058 1058 - } 1059 1059 - 1060 1060 - static irqreturn_t bwd_callback_msix_irq(int irq, void *data) 1061 1061 - { 1062 1062 - struct ntb_db_cb *db_cb = data; 1063 1063 - struct ntb_device *ndev = db_cb->ndev; 1064 1064 - unsigned long mask; 1065 1065 - 1066 1066 - dev_dbg(&ndev->pdev->dev, "MSI-X irq %d received for DB %d\n", irq, 1067 1067 - db_cb->db_num); 1068 1068 - 1069 1069 - mask = readw(ndev->reg_ofs.ldb_mask); 1070 1070 - set_bit(db_cb->db_num * ndev->bits_per_vector, &mask); 1071 1071 - writew(mask, ndev->reg_ofs.ldb_mask); 1072 1072 - 1073 1073 - tasklet_schedule(&db_cb->irq_work); 1074 1074 - 1075 1075 - /* No need to check for the specific HB irq, any interrupt means 1076 1076 - * we're connected. 1077 1077 - */ 1078 1078 - ndev->last_ts = jiffies; 1079 1079 - 1080 1080 - writeq((u64) 1 << db_cb->db_num, ndev->reg_ofs.ldb); 1081 1081 - 1082 1082 - return IRQ_HANDLED; 1083 1083 - } 1084 1084 - 1085 1085 - static irqreturn_t xeon_callback_msix_irq(int irq, void *data) 1086 1086 - { 1087 1087 - struct ntb_db_cb *db_cb = data; 1088 1088 - struct ntb_device *ndev = db_cb->ndev; 1089 1089 - unsigned long mask; 1090 1090 - 1091 1091 - dev_dbg(&ndev->pdev->dev, "MSI-X irq %d received for DB %d\n", irq, 1092 1092 - db_cb->db_num); 1093 1093 - 1094 1094 - mask = readw(ndev->reg_ofs.ldb_mask); 1095 1095 - set_bit(db_cb->db_num * ndev->bits_per_vector, &mask); 1096 1096 - writew(mask, ndev->reg_ofs.ldb_mask); 1097 1097 - 1098 1098 - tasklet_schedule(&db_cb->irq_work); 1099 1099 - 1100 1100 - /* On Sandybridge, there are 16 bits in the interrupt register 1101 1101 - * but only 4 vectors. So, 5 bits are assigned to the first 3 1102 1102 - * vectors, with the 4th having a single bit for link 1103 1103 - * interrupts. 1104 1104 - */ 1105 1105 - writew(((1 << ndev->bits_per_vector) - 1) << 1106 1106 - (db_cb->db_num * ndev->bits_per_vector), ndev->reg_ofs.ldb); 1107 1107 - 1108 1108 - return IRQ_HANDLED; 1109 1109 - } 1110 1110 - 1111 1111 - /* Since we do not have a HW doorbell in BWD, this is only used in JF/JT */ 1112 1112 - static irqreturn_t xeon_event_msix_irq(int irq, void *dev) 1113 1113 - { 1114 1114 - struct ntb_device *ndev = dev; 1115 1115 - int rc; 1116 1116 - 1117 1117 - dev_dbg(&ndev->pdev->dev, "MSI-X irq %d received for Events\n", irq); 1118 1118 - 1119 1119 - rc = ntb_link_status(ndev); 1120 1120 - if (rc) 1121 1121 - dev_err(&ndev->pdev->dev, "Error determining link status\n"); 1122 1122 - 1123 1123 - /* bit 15 is always the link bit */ 1124 1124 - writew(1 << SNB_LINK_DB, ndev->reg_ofs.ldb); 1125 1125 - 1126 1126 - return IRQ_HANDLED; 1127 1127 - } 1128 1128 - 1129 1129 - static irqreturn_t ntb_interrupt(int irq, void *dev) 1130 1130 - { 1131 1131 - struct ntb_device *ndev = dev; 1132 1132 - unsigned int i = 0; 1133 1133 - 1134 1134 - if (is_ntb_atom(ndev)) { 1135 1135 - u64 ldb = readq(ndev->reg_ofs.ldb); 1136 1136 - 1137 1137 - dev_dbg(&ndev->pdev->dev, "irq %d - ldb = %Lx\n", irq, ldb); 1138 1138 - 1139 1139 - while (ldb) { 1140 1140 - i = __ffs(ldb); 1141 1141 - ldb &= ldb - 1; 1142 1142 - bwd_callback_msix_irq(irq, &ndev->db_cb[i]); 1143 1143 - } 1144 1144 - } else { 1145 1145 - u16 ldb = readw(ndev->reg_ofs.ldb); 1146 1146 - 1147 1147 - dev_dbg(&ndev->pdev->dev, "irq %d - ldb = %x\n", irq, ldb); 1148 1148 - 1149 1149 - if (ldb & SNB_DB_HW_LINK) { 1150 1150 - xeon_event_msix_irq(irq, dev); 1151 1151 - ldb &= ~SNB_DB_HW_LINK; 1152 1152 - } 1153 1153 - 1154 1154 - while (ldb) { 1155 1155 - i = __ffs(ldb); 1156 1156 - ldb &= ldb - 1; 1157 1157 - xeon_callback_msix_irq(irq, &ndev->db_cb[i]); 1158 1158 - } 1159 1159 - } 1160 1160 - 1161 1161 - return IRQ_HANDLED; 1162 1162 - } 1163 1163 - 1164 1164 - static int ntb_setup_snb_msix(struct ntb_device *ndev, int msix_entries) 1165 1165 - { 1166 1166 - struct pci_dev *pdev = ndev->pdev; 1167 1167 - struct msix_entry *msix; 1168 1168 - int rc, i; 1169 1169 - 1170 1170 - if (msix_entries < ndev->limits.msix_cnt) 1171 1171 - return -ENOSPC; 1172 1172 - 1173 1173 - rc = pci_enable_msix_exact(pdev, ndev->msix_entries, msix_entries); 1174 1174 - if (rc < 0) 1175 1175 - return rc; 1176 1176 - 1177 1177 - for (i = 0; i < msix_entries; i++) { 1178 1178 - msix = &ndev->msix_entries[i]; 1179 1179 - WARN_ON(!msix->vector); 1180 1180 - 1181 1181 - if (i == msix_entries - 1) { 1182 1182 - rc = request_irq(msix->vector, 1183 1183 - xeon_event_msix_irq, 0, 1184 1184 - "ntb-event-msix", ndev); 1185 1185 - if (rc) 1186 1186 - goto err; 1187 1187 - } else { 1188 1188 - rc = request_irq(msix->vector, 1189 1189 - xeon_callback_msix_irq, 0, 1190 1190 - "ntb-callback-msix", 1191 1191 - &ndev->db_cb[i]); 1192 1192 - if (rc) 1193 1193 - goto err; 1194 1194 - } 1195 1195 - } 1196 1196 - 1197 1197 - ndev->num_msix = msix_entries; 1198 1198 - ndev->max_cbs = msix_entries - 1; 1199 1199 - 1200 1200 - return 0; 1201 1201 - 1202 1202 - err: 1203 1203 - while (--i >= 0) { 1204 1204 - /* Code never reaches here for entry nr 'ndev->num_msix - 1' */ 1205 1205 - msix = &ndev->msix_entries[i]; 1206 1206 - free_irq(msix->vector, &ndev->db_cb[i]); 1207 1207 - } 1208 1208 - 1209 1209 - pci_disable_msix(pdev); 1210 1210 - ndev->num_msix = 0; 1211 1211 - 1212 1212 - return rc; 1213 1213 - } 1214 1214 - 1215 1215 - static int ntb_setup_bwd_msix(struct ntb_device *ndev, int msix_entries) 1216 1216 - { 1217 1217 - struct pci_dev *pdev = ndev->pdev; 1218 1218 - struct msix_entry *msix; 1219 1219 - int rc, i; 1220 1220 - 1221 1221 - msix_entries = pci_enable_msix_range(pdev, ndev->msix_entries, 1222 1222 - 1, msix_entries); 1223 1223 - if (msix_entries < 0) 1224 1224 - return msix_entries; 1225 1225 - 1226 1226 - for (i = 0; i < msix_entries; i++) { 1227 1227 - msix = &ndev->msix_entries[i]; 1228 1228 - WARN_ON(!msix->vector); 1229 1229 - 1230 1230 - rc = request_irq(msix->vector, bwd_callback_msix_irq, 0, 1231 1231 - "ntb-callback-msix", &ndev->db_cb[i]); 1232 1232 - if (rc) 1233 1233 - goto err; 1234 1234 - } 1235 1235 - 1236 1236 - ndev->num_msix = msix_entries; 1237 1237 - ndev->max_cbs = msix_entries; 1238 1238 - 1239 1239 - return 0; 1240 1240 - 1241 1241 - err: 1242 1242 - while (--i >= 0) 1243 1243 - free_irq(msix->vector, &ndev->db_cb[i]); 1244 1244 - 1245 1245 - pci_disable_msix(pdev); 1246 1246 - ndev->num_msix = 0; 1247 1247 - 1248 1248 - return rc; 1249 1249 - } 1250 1250 - 1251 1251 - static int ntb_setup_msix(struct ntb_device *ndev) 1252 1252 - { 1253 1253 - struct pci_dev *pdev = ndev->pdev; 1254 1254 - int msix_entries; 1255 1255 - int rc, i; 1256 1256 - 1257 1257 - msix_entries = pci_msix_vec_count(pdev); 1258 1258 - if (msix_entries < 0) { 1259 1259 - rc = msix_entries; 1260 1260 - goto err; 1261 1261 - } else if (msix_entries > ndev->limits.msix_cnt) { 1262 1262 - rc = -EINVAL; 1263 1263 - goto err; 1264 1264 - } 1265 1265 - 1266 1266 - ndev->msix_entries = kmalloc(sizeof(struct msix_entry) * msix_entries, 1267 1267 - GFP_KERNEL); 1268 1268 - if (!ndev->msix_entries) { 1269 1269 - rc = -ENOMEM; 1270 1270 - goto err; 1271 1271 - } 1272 1272 - 1273 1273 - for (i = 0; i < msix_entries; i++) 1274 1274 - ndev->msix_entries[i].entry = i; 1275 1275 - 1276 1276 - if (is_ntb_atom(ndev)) 1277 1277 - rc = ntb_setup_bwd_msix(ndev, msix_entries); 1278 1278 - else 1279 1279 - rc = ntb_setup_snb_msix(ndev, msix_entries); 1280 1280 - if (rc) 1281 1281 - goto err1; 1282 1282 - 1283 1283 - return 0; 1284 1284 - 1285 1285 - err1: 1286 1286 - kfree(ndev->msix_entries); 1287 1287 - err: 1288 1288 - dev_err(&pdev->dev, "Error allocating MSI-X interrupt\n"); 1289 1289 - return rc; 1290 1290 - } 1291 1291 - 1292 1292 - static int ntb_setup_msi(struct ntb_device *ndev) 1293 1293 - { 1294 1294 - struct pci_dev *pdev = ndev->pdev; 1295 1295 - int rc; 1296 1296 - 1297 1297 - rc = pci_enable_msi(pdev); 1298 1298 - if (rc) 1299 1299 - return rc; 1300 1300 - 1301 1301 - rc = request_irq(pdev->irq, ntb_interrupt, 0, "ntb-msi", ndev); 1302 1302 - if (rc) { 1303 1303 - pci_disable_msi(pdev); 1304 1304 - dev_err(&pdev->dev, "Error allocating MSI interrupt\n"); 1305 1305 - return rc; 1306 1306 - } 1307 1307 - 1308 1308 - return 0; 1309 1309 - } 1310 1310 - 1311 1311 - static int ntb_setup_intx(struct ntb_device *ndev) 1312 1312 - { 1313 1313 - struct pci_dev *pdev = ndev->pdev; 1314 1314 - int rc; 1315 1315 - 1316 1316 - /* Verify intx is enabled */ 1317 1317 - pci_intx(pdev, 1); 1318 1318 - 1319 1319 - rc = request_irq(pdev->irq, ntb_interrupt, IRQF_SHARED, "ntb-intx", 1320 1320 - ndev); 1321 1321 - if (rc) 1322 1322 - return rc; 1323 1323 - 1324 1324 - return 0; 1325 1325 - } 1326 1326 - 1327 1327 - static int ntb_setup_interrupts(struct ntb_device *ndev) 1328 1328 - { 1329 1329 - int rc; 1330 1330 - 1331 1331 - /* On BWD, disable all interrupts. On SNB, disable all but Link 1332 1332 - * Interrupt. The rest will be unmasked as callbacks are registered. 1333 1333 - */ 1334 1334 - if (is_ntb_atom(ndev)) 1335 1335 - writeq(~0, ndev->reg_ofs.ldb_mask); 1336 1336 - else { 1337 1337 - u16 var = 1 << SNB_LINK_DB; 1338 1338 - writew(~var, ndev->reg_ofs.ldb_mask); 1339 1339 - } 1340 1340 - 1341 1341 - rc = ntb_setup_msix(ndev); 1342 1342 - if (!rc) 1343 1343 - goto done; 1344 1344 - 1345 1345 - ndev->bits_per_vector = 1; 1346 1346 - ndev->max_cbs = ndev->limits.max_db_bits; 1347 1347 - 1348 1348 - rc = ntb_setup_msi(ndev); 1349 1349 - if (!rc) 1350 1350 - goto done; 1351 1351 - 1352 1352 - rc = ntb_setup_intx(ndev); 1353 1353 - if (rc) { 1354 1354 - dev_err(&ndev->pdev->dev, "no usable interrupts\n"); 1355 1355 - return rc; 1356 1356 - } 1357 1357 - 1358 1358 - done: 1359 1359 - return 0; 1360 1360 - } 1361 1361 - 1362 1362 - static void ntb_free_interrupts(struct ntb_device *ndev) 1363 1363 - { 1364 1364 - struct pci_dev *pdev = ndev->pdev; 1365 1365 - 1366 1366 - /* mask interrupts */ 1367 1367 - if (is_ntb_atom(ndev)) 1368 1368 - writeq(~0, ndev->reg_ofs.ldb_mask); 1369 1369 - else 1370 1370 - writew(~0, ndev->reg_ofs.ldb_mask); 1371 1371 - 1372 1372 - if (ndev->num_msix) { 1373 1373 - struct msix_entry *msix; 1374 1374 - u32 i; 1375 1375 - 1376 1376 - for (i = 0; i < ndev->num_msix; i++) { 1377 1377 - msix = &ndev->msix_entries[i]; 1378 1378 - if (is_ntb_xeon(ndev) && i == ndev->num_msix - 1) 1379 1379 - free_irq(msix->vector, ndev); 1380 1380 - else 1381 1381 - free_irq(msix->vector, &ndev->db_cb[i]); 1382 1382 - } 1383 1383 - pci_disable_msix(pdev); 1384 1384 - kfree(ndev->msix_entries); 1385 1385 - } else { 1386 1386 - free_irq(pdev->irq, ndev); 1387 1387 - 1388 1388 - if (pci_dev_msi_enabled(pdev)) 1389 1389 - pci_disable_msi(pdev); 1390 1390 - } 1391 1391 - } 1392 1392 - 1393 1393 - static int ntb_create_callbacks(struct ntb_device *ndev) 1394 1394 - { 1395 1395 - int i; 1396 1396 - 1397 1397 - /* Chicken-egg issue. We won't know how many callbacks are necessary 1398 1398 - * until we see how many MSI-X vectors we get, but these pointers need 1399 1399 - * to be passed into the MSI-X register function. So, we allocate the 1400 1400 - * max, knowing that they might not all be used, to work around this. 1401 1401 - */ 1402 1402 - ndev->db_cb = kcalloc(ndev->limits.max_db_bits, 1403 1403 - sizeof(struct ntb_db_cb), 1404 1404 - GFP_KERNEL); 1405 1405 - if (!ndev->db_cb) 1406 1406 - return -ENOMEM; 1407 1407 - 1408 1408 - for (i = 0; i < ndev->limits.max_db_bits; i++) { 1409 1409 - ndev->db_cb[i].db_num = i; 1410 1410 - ndev->db_cb[i].ndev = ndev; 1411 1411 - } 1412 1412 - 1413 1413 - return 0; 1414 1414 - } 1415 1415 - 1416 1416 - static void ntb_free_callbacks(struct ntb_device *ndev) 1417 1417 - { 1418 1418 - int i; 1419 1419 - 1420 1420 - for (i = 0; i < ndev->limits.max_db_bits; i++) 1421 1421 - ntb_unregister_db_callback(ndev, i); 1422 1422 - 1423 1423 - kfree(ndev->db_cb); 1424 1424 - } 1425 1425 - 1426 1426 - static ssize_t ntb_debugfs_read(struct file *filp, char __user *ubuf, 1427 1427 - size_t count, loff_t *offp) 1428 1428 - { 1429 1429 - struct ntb_device *ndev; 1430 1430 - char *buf; 1431 1431 - ssize_t ret, offset, out_count; 1432 1432 - 1433 1433 - out_count = 500; 1434 1434 - 1435 1435 - buf = kmalloc(out_count, GFP_KERNEL); 1436 1436 - if (!buf) 1437 1437 - return -ENOMEM; 1438 1438 - 1439 1439 - ndev = filp->private_data; 1440 1440 - offset = 0; 1441 1441 - offset += snprintf(buf + offset, out_count - offset, 1442 1442 - "NTB Device Information:\n"); 1443 1443 - offset += snprintf(buf + offset, out_count - offset, 1444 1444 - "Connection Type - \t\t%s\n", 1445 1445 - ndev->conn_type == NTB_CONN_TRANSPARENT ? 1446 1446 - "Transparent" : (ndev->conn_type == NTB_CONN_B2B) ? 1447 1447 - "Back to back" : "Root Port"); 1448 1448 - offset += snprintf(buf + offset, out_count - offset, 1449 1449 - "Device Type - \t\t\t%s\n", 1450 1450 - ndev->dev_type == NTB_DEV_USD ? 1451 1451 - "DSD/USP" : "USD/DSP"); 1452 1452 - offset += snprintf(buf + offset, out_count - offset, 1453 1453 - "Max Number of Callbacks - \t%u\n", 1454 1454 - ntb_max_cbs(ndev)); 1455 1455 - offset += snprintf(buf + offset, out_count - offset, 1456 1456 - "Link Status - \t\t\t%s\n", 1457 1457 - ntb_hw_link_status(ndev) ? "Up" : "Down"); 1458 1458 - if (ntb_hw_link_status(ndev)) { 1459 1459 - offset += snprintf(buf + offset, out_count - offset, 1460 1460 - "Link Speed - \t\t\tPCI-E Gen %u\n", 1461 1461 - ndev->link_speed); 1462 1462 - offset += snprintf(buf + offset, out_count - offset, 1463 1463 - "Link Width - \t\t\tx%u\n", 1464 1464 - ndev->link_width); 1465 1465 - } 1466 1466 - 1467 1467 - if (is_ntb_xeon(ndev)) { 1468 1468 - u32 status32; 1469 1469 - u16 status16; 1470 1470 - int rc; 1471 1471 - 1472 1472 - offset += snprintf(buf + offset, out_count - offset, 1473 1473 - "\nNTB Device Statistics:\n"); 1474 1474 - offset += snprintf(buf + offset, out_count - offset, 1475 1475 - "Upstream Memory Miss - \t%u\n", 1476 1476 - readw(ndev->reg_base + 1477 1477 - SNB_USMEMMISS_OFFSET)); 1478 1478 - 1479 1479 - offset += snprintf(buf + offset, out_count - offset, 1480 1480 - "\nNTB Hardware Errors:\n"); 1481 1481 - 1482 1482 - rc = pci_read_config_word(ndev->pdev, SNB_DEVSTS_OFFSET, 1483 1483 - &status16); 1484 1484 - if (!rc) 1485 1485 - offset += snprintf(buf + offset, out_count - offset, 1486 1486 - "DEVSTS - \t%#06x\n", status16); 1487 1487 - 1488 1488 - rc = pci_read_config_word(ndev->pdev, SNB_LINK_STATUS_OFFSET, 1489 1489 - &status16); 1490 1490 - if (!rc) 1491 1491 - offset += snprintf(buf + offset, out_count - offset, 1492 1492 - "LNKSTS - \t%#06x\n", status16); 1493 1493 - 1494 1494 - rc = pci_read_config_dword(ndev->pdev, SNB_UNCERRSTS_OFFSET, 1495 1495 - &status32); 1496 1496 - if (!rc) 1497 1497 - offset += snprintf(buf + offset, out_count - offset, 1498 1498 - "UNCERRSTS - \t%#010x\n", status32); 1499 1499 - 1500 1500 - rc = pci_read_config_dword(ndev->pdev, SNB_CORERRSTS_OFFSET, 1501 1501 - &status32); 1502 1502 - if (!rc) 1503 1503 - offset += snprintf(buf + offset, out_count - offset, 1504 1504 - "CORERRSTS - \t%#010x\n", status32); 1505 1505 - } 1506 1506 - 1507 1507 - if (offset > out_count) 1508 1508 - offset = out_count; 1509 1509 - 1510 1510 - ret = simple_read_from_buffer(ubuf, count, offp, buf, offset); 1511 1511 - kfree(buf); 1512 1512 - return ret; 1513 1513 - } 1514 1514 - 1515 1515 - static const struct file_operations ntb_debugfs_info = { 1516 1516 - .owner = THIS_MODULE, 1517 1517 - .open = simple_open, 1518 1518 - .read = ntb_debugfs_read, 1519 1519 - }; 1520 1520 - 1521 1521 - static void ntb_setup_debugfs(struct ntb_device *ndev) 1522 1522 - { 1523 1523 - if (!debugfs_initialized()) 1524 1524 - return; 1525 1525 - 1526 1526 - if (!debugfs_dir) 1527 1527 - debugfs_dir = debugfs_create_dir(KBUILD_MODNAME, NULL); 1528 1528 - 1529 1529 - ndev->debugfs_dir = debugfs_create_dir(pci_name(ndev->pdev), 1530 1530 - debugfs_dir); 1531 1531 - if (ndev->debugfs_dir) 1532 1532 - ndev->debugfs_info = debugfs_create_file("info", S_IRUSR, 1533 1533 - ndev->debugfs_dir, 1534 1534 - ndev, 1535 1535 - &ntb_debugfs_info); 1536 1536 - } 1537 1537 - 1538 1538 - static void ntb_free_debugfs(struct ntb_device *ndev) 1539 1539 - { 1540 1540 - debugfs_remove_recursive(ndev->debugfs_dir); 1541 1541 - 1542 1542 - if (debugfs_dir && simple_empty(debugfs_dir)) { 1543 1543 - debugfs_remove_recursive(debugfs_dir); 1544 1544 - debugfs_dir = NULL; 1545 1545 - } 1546 1546 - } 1547 1547 - 1548 1548 - static void ntb_hw_link_up(struct ntb_device *ndev) 1549 1549 - { 1550 1550 - if (ndev->conn_type == NTB_CONN_TRANSPARENT) 1551 1551 - ntb_link_event(ndev, NTB_LINK_UP); 1552 1552 - else { 1553 1553 - u32 ntb_cntl; 1554 1554 - 1555 1555 - /* Let's bring the NTB link up */ 1556 1556 - ntb_cntl = readl(ndev->reg_ofs.lnk_cntl); 1557 1557 - ntb_cntl &= ~(NTB_CNTL_LINK_DISABLE | NTB_CNTL_CFG_LOCK); 1558 1558 - ntb_cntl |= NTB_CNTL_P2S_BAR23_SNOOP | NTB_CNTL_S2P_BAR23_SNOOP; 1559 1559 - ntb_cntl |= NTB_CNTL_P2S_BAR4_SNOOP | NTB_CNTL_S2P_BAR4_SNOOP; 1560 1560 - if (ndev->split_bar) 1561 1561 - ntb_cntl |= NTB_CNTL_P2S_BAR5_SNOOP | 1562 1562 - NTB_CNTL_S2P_BAR5_SNOOP; 1563 1563 - 1564 1564 - writel(ntb_cntl, ndev->reg_ofs.lnk_cntl); 1565 1565 - } 1566 1566 - } 1567 1567 - 1568 1568 - static void ntb_hw_link_down(struct ntb_device *ndev) 1569 1569 - { 1570 1570 - u32 ntb_cntl; 1571 1571 - 1572 1572 - if (ndev->conn_type == NTB_CONN_TRANSPARENT) { 1573 1573 - ntb_link_event(ndev, NTB_LINK_DOWN); 1574 1574 - return; 1575 1575 - } 1576 1576 - 1577 1577 - /* Bring NTB link down */ 1578 1578 - ntb_cntl = readl(ndev->reg_ofs.lnk_cntl); 1579 1579 - ntb_cntl &= ~(NTB_CNTL_P2S_BAR23_SNOOP | NTB_CNTL_S2P_BAR23_SNOOP); 1580 1580 - ntb_cntl &= ~(NTB_CNTL_P2S_BAR4_SNOOP | NTB_CNTL_S2P_BAR4_SNOOP); 1581 1581 - if (ndev->split_bar) 1582 1582 - ntb_cntl &= ~(NTB_CNTL_P2S_BAR5_SNOOP | 1583 1583 - NTB_CNTL_S2P_BAR5_SNOOP); 1584 1584 - ntb_cntl |= NTB_CNTL_LINK_DISABLE | NTB_CNTL_CFG_LOCK; 1585 1585 - writel(ntb_cntl, ndev->reg_ofs.lnk_cntl); 1586 1586 - } 1587 1587 - 1588 1588 - static void ntb_max_mw_detect(struct ntb_device *ndev) 1589 1589 - { 1590 1590 - if (ndev->split_bar) 1591 1591 - ndev->limits.max_mw = HSX_SPLITBAR_MAX_MW; 1592 1592 - else 1593 1593 - ndev->limits.max_mw = SNB_MAX_MW; 1594 1594 - } 1595 1595 - 1596 1596 - static int ntb_xeon_detect(struct ntb_device *ndev) 1597 1597 - { 1598 1598 - int rc, bars_mask; 1599 1599 - u32 bars; 1600 1600 - u8 ppd; 1601 1601 - 1602 1602 - ndev->hw_type = SNB_HW; 1603 1603 - 1604 1604 - rc = pci_read_config_byte(ndev->pdev, NTB_PPD_OFFSET, &ppd); 1605 1605 - if (rc) 1606 1606 - return -EIO; 1607 1607 - 1608 1608 - if (ppd & SNB_PPD_DEV_TYPE) 1609 1609 - ndev->dev_type = NTB_DEV_USD; 1610 1610 - else 1611 1611 - ndev->dev_type = NTB_DEV_DSD; 1612 1612 - 1613 1613 - ndev->split_bar = (ppd & SNB_PPD_SPLIT_BAR) ? 1 : 0; 1614 1614 - 1615 1615 - switch (ppd & SNB_PPD_CONN_TYPE) { 1616 1616 - case NTB_CONN_B2B: 1617 1617 - dev_info(&ndev->pdev->dev, "Conn Type = B2B\n"); 1618 1618 - ndev->conn_type = NTB_CONN_B2B; 1619 1619 - break; 1620 1620 - case NTB_CONN_RP: 1621 1621 - dev_info(&ndev->pdev->dev, "Conn Type = RP\n"); 1622 1622 - ndev->conn_type = NTB_CONN_RP; 1623 1623 - break; 1624 1624 - case NTB_CONN_TRANSPARENT: 1625 1625 - dev_info(&ndev->pdev->dev, "Conn Type = TRANSPARENT\n"); 1626 1626 - ndev->conn_type = NTB_CONN_TRANSPARENT; 1627 1627 - /* 1628 1628 - * This mode is default to USD/DSP. HW does not report 1629 1629 - * properly in transparent mode as it has no knowledge of 1630 1630 - * NTB. We will just force correct here. 1631 1631 - */ 1632 1632 - ndev->dev_type = NTB_DEV_USD; 1633 1633 - 1634 1634 - /* 1635 1635 - * This is a way for transparent BAR to figure out if we 1636 1636 - * are doing split BAR or not. There is no way for the hw 1637 1637 - * on the transparent side to know and set the PPD. 1638 1638 - */ 1639 1639 - bars_mask = pci_select_bars(ndev->pdev, IORESOURCE_MEM); 1640 1640 - bars = hweight32(bars_mask); 1641 1641 - if (bars == (HSX_SPLITBAR_MAX_MW + 1)) 1642 1642 - ndev->split_bar = 1; 1643 1643 - 1644 1644 - break; 1645 1645 - default: 1646 1646 - dev_err(&ndev->pdev->dev, "Unknown PPD %x\n", ppd); 1647 1647 - return -ENODEV; 1648 1648 - } 1649 1649 - 1650 1650 - ntb_max_mw_detect(ndev); 1651 1651 - 1652 1652 - return 0; 1653 1653 - } 1654 1654 - 1655 1655 - static int ntb_atom_detect(struct ntb_device *ndev) 1656 1656 - { 1657 1657 - int rc; 1658 1658 - u32 ppd; 1659 1659 - 1660 1660 - ndev->hw_type = BWD_HW; 1661 1661 - ndev->limits.max_mw = BWD_MAX_MW; 1662 1662 - 1663 1663 - rc = pci_read_config_dword(ndev->pdev, NTB_PPD_OFFSET, &ppd); 1664 1664 - if (rc) 1665 1665 - return rc; 1666 1666 - 1667 1667 - switch ((ppd & BWD_PPD_CONN_TYPE) >> 8) { 1668 1668 - case NTB_CONN_B2B: 1669 1669 - dev_info(&ndev->pdev->dev, "Conn Type = B2B\n"); 1670 1670 - ndev->conn_type = NTB_CONN_B2B; 1671 1671 - break; 1672 1672 - case NTB_CONN_RP: 1673 1673 - default: 1674 1674 - dev_err(&ndev->pdev->dev, "Unsupported NTB configuration\n"); 1675 1675 - return -EINVAL; 1676 1676 - } 1677 1677 - 1678 1678 - if (ppd & BWD_PPD_DEV_TYPE) 1679 1679 - ndev->dev_type = NTB_DEV_DSD; 1680 1680 - else 1681 1681 - ndev->dev_type = NTB_DEV_USD; 1682 1682 - 1683 1683 - return 0; 1684 1684 - } 1685 1685 - 1686 1686 - static int ntb_device_detect(struct ntb_device *ndev) 1687 1687 - { 1688 1688 - int rc; 1689 1689 - 1690 1690 - if (is_ntb_xeon(ndev)) 1691 1691 - rc = ntb_xeon_detect(ndev); 1692 1692 - else if (is_ntb_atom(ndev)) 1693 1693 - rc = ntb_atom_detect(ndev); 1694 1694 - else 1695 1695 - rc = -ENODEV; 1696 1696 - 1697 1697 - dev_info(&ndev->pdev->dev, "Device Type = %s\n", 1698 1698 - ndev->dev_type == NTB_DEV_USD ? "USD/DSP" : "DSD/USP"); 1699 1699 - 1700 1700 - return 0; 1701 1701 - } 1702 1702 - 1703 1703 - static int ntb_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) 1704 1704 - { 1705 1705 - struct ntb_device *ndev; 1706 1706 - int rc, i; 1707 1707 - 1708 1708 - ndev = kzalloc(sizeof(struct ntb_device), GFP_KERNEL); 1709 1709 - if (!ndev) 1710 1710 - return -ENOMEM; 1711 1711 - 1712 1712 - ndev->pdev = pdev; 1713 1713 - 1714 1714 - ntb_set_errata_flags(ndev); 1715 1715 - 1716 1716 - ndev->link_status = NTB_LINK_DOWN; 1717 1717 - pci_set_drvdata(pdev, ndev); 1718 1718 - ntb_setup_debugfs(ndev); 1719 1719 - 1720 1720 - rc = pci_enable_device(pdev); 1721 1721 - if (rc) 1722 1722 - goto err; 1723 1723 - 1724 1724 - pci_set_master(ndev->pdev); 1725 1725 - 1726 1726 - rc = ntb_device_detect(ndev); 1727 1727 - if (rc) 1728 1728 - goto err; 1729 1729 - 1730 1730 - ndev->mw = kcalloc(ndev->limits.max_mw, sizeof(struct ntb_mw), 1731 1731 - GFP_KERNEL); 1732 1732 - if (!ndev->mw) { 1733 1733 - rc = -ENOMEM; 1734 1734 - goto err1; 1735 1735 - } 1736 1736 - 1737 1737 - if (ndev->split_bar) 1738 1738 - rc = pci_request_selected_regions(pdev, NTB_SPLITBAR_MASK, 1739 1739 - KBUILD_MODNAME); 1740 1740 - else 1741 1741 - rc = pci_request_selected_regions(pdev, NTB_BAR_MASK, 1742 1742 - KBUILD_MODNAME); 1743 1743 - 1744 1744 - if (rc) 1745 1745 - goto err2; 1746 1746 - 1747 1747 - ndev->reg_base = pci_ioremap_bar(pdev, NTB_BAR_MMIO); 1748 1748 - if (!ndev->reg_base) { 1749 1749 - dev_warn(&pdev->dev, "Cannot remap BAR 0\n"); 1750 1750 - rc = -EIO; 1751 1751 - goto err3; 1752 1752 - } 1753 1753 - 1754 1754 - for (i = 0; i < ndev->limits.max_mw; i++) { 1755 1755 - ndev->mw[i].bar_sz = pci_resource_len(pdev, MW_TO_BAR(i)); 1756 1756 - 1757 1757 - /* 1758 1758 - * with the errata we need to steal last of the memory 1759 1759 - * windows for workarounds and they point to MMIO registers. 1760 1760 - */ 1761 1761 - if ((ndev->wa_flags & WA_SNB_ERR) && 1762 1762 - (i == (ndev->limits.max_mw - 1))) { 1763 1763 - ndev->mw[i].vbase = 1764 1764 - ioremap_nocache(pci_resource_start(pdev, 1765 1765 - MW_TO_BAR(i)), 1766 1766 - ndev->mw[i].bar_sz); 1767 1767 - } else { 1768 1768 - ndev->mw[i].vbase = 1769 1769 - ioremap_wc(pci_resource_start(pdev, 1770 1770 - MW_TO_BAR(i)), 1771 1771 - ndev->mw[i].bar_sz); 1772 1772 - } 1773 1773 - 1774 1774 - dev_info(&pdev->dev, "MW %d size %llu\n", i, 1775 1775 - (unsigned long long) ndev->mw[i].bar_sz); 1776 1776 - if (!ndev->mw[i].vbase) { 1777 1777 - dev_warn(&pdev->dev, "Cannot remap BAR %d\n", 1778 1778 - MW_TO_BAR(i)); 1779 1779 - rc = -EIO; 1780 1780 - goto err4; 1781 1781 - } 1782 1782 - } 1783 1783 - 1784 1784 - rc = pci_set_dma_mask(pdev, DMA_BIT_MASK(64)); 1785 1785 - if (rc) { 1786 1786 - rc = pci_set_dma_mask(pdev, DMA_BIT_MASK(32)); 1787 1787 - if (rc) 1788 1788 - goto err4; 1789 1789 - 1790 1790 - dev_warn(&pdev->dev, "Cannot DMA highmem\n"); 1791 1791 - } 1792 1792 - 1793 1793 - rc = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(64)); 1794 1794 - if (rc) { 1795 1795 - rc = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(32)); 1796 1796 - if (rc) 1797 1797 - goto err4; 1798 1798 - 1799 1799 - dev_warn(&pdev->dev, "Cannot DMA consistent highmem\n"); 1800 1800 - } 1801 1801 - 1802 1802 - rc = ntb_device_setup(ndev); 1803 1803 - if (rc) 1804 1804 - goto err4; 1805 1805 - 1806 1806 - rc = ntb_create_callbacks(ndev); 1807 1807 - if (rc) 1808 1808 - goto err5; 1809 1809 - 1810 1810 - rc = ntb_setup_interrupts(ndev); 1811 1811 - if (rc) 1812 1812 - goto err6; 1813 1813 - 1814 1814 - /* The scratchpad registers keep the values between rmmod/insmod, 1815 1815 - * blast them now 1816 1816 - */ 1817 1817 - for (i = 0; i < ndev->limits.max_spads; i++) { 1818 1818 - ntb_write_local_spad(ndev, i, 0); 1819 1819 - ntb_write_remote_spad(ndev, i, 0); 1820 1820 - } 1821 1821 - 1822 1822 - rc = ntb_transport_init(pdev); 1823 1823 - if (rc) 1824 1824 - goto err7; 1825 1825 - 1826 1826 - ntb_hw_link_up(ndev); 1827 1827 - 1828 1828 - return 0; 1829 1829 - 1830 1830 - err7: 1831 1831 - ntb_free_interrupts(ndev); 1832 1832 - err6: 1833 1833 - ntb_free_callbacks(ndev); 1834 1834 - err5: 1835 1835 - ntb_device_free(ndev); 1836 1836 - err4: 1837 1837 - for (i--; i >= 0; i--) 1838 1838 - iounmap(ndev->mw[i].vbase); 1839 1839 - iounmap(ndev->reg_base); 1840 1840 - err3: 1841 1841 - if (ndev->split_bar) 1842 1842 - pci_release_selected_regions(pdev, NTB_SPLITBAR_MASK); 1843 1843 - else 1844 1844 - pci_release_selected_regions(pdev, NTB_BAR_MASK); 1845 1845 - err2: 1846 1846 - kfree(ndev->mw); 1847 1847 - err1: 1848 1848 - pci_disable_device(pdev); 1849 1849 - err: 1850 1850 - ntb_free_debugfs(ndev); 1851 1851 - kfree(ndev); 1852 1852 - 1853 1853 - dev_err(&pdev->dev, "Error loading %s module\n", KBUILD_MODNAME); 1854 1854 - return rc; 1855 1855 - } 1856 1856 - 1857 1857 - static void ntb_pci_remove(struct pci_dev *pdev) 1858 1858 - { 1859 1859 - struct ntb_device *ndev = pci_get_drvdata(pdev); 1860 1860 - int i; 1861 1861 - 1862 1862 - ntb_hw_link_down(ndev); 1863 1863 - 1864 1864 - ntb_transport_free(ndev->ntb_transport); 1865 1865 - 1866 1866 - ntb_free_interrupts(ndev); 1867 1867 - ntb_free_callbacks(ndev); 1868 1868 - ntb_device_free(ndev); 1869 1869 - 1870 1870 - /* need to reset max_mw limits so we can unmap properly */ 1871 1871 - if (ndev->hw_type == SNB_HW) 1872 1872 - ntb_max_mw_detect(ndev); 1873 1873 - 1874 1874 - for (i = 0; i < ndev->limits.max_mw; i++) 1875 1875 - iounmap(ndev->mw[i].vbase); 1876 1876 - 1877 1877 - kfree(ndev->mw); 1878 1878 - iounmap(ndev->reg_base); 1879 1879 - if (ndev->split_bar) 1880 1880 - pci_release_selected_regions(pdev, NTB_SPLITBAR_MASK); 1881 1881 - else 1882 1882 - pci_release_selected_regions(pdev, NTB_BAR_MASK); 1883 1883 - pci_disable_device(pdev); 1884 1884 - ntb_free_debugfs(ndev); 1885 1885 - kfree(ndev); 1886 1886 - } 1887 1887 - 1888 1888 - static struct pci_driver ntb_pci_driver = { 1889 1889 - .name = KBUILD_MODNAME, 1890 1890 - .id_table = ntb_pci_tbl, 1891 1891 - .probe = ntb_pci_probe, 1892 1892 - .remove = ntb_pci_remove, 1893 1893 - }; 1894 1894 - 1895 1895 - module_pci_driver(ntb_pci_driver);

-256

drivers/ntb/ntb_hw.h

reviewed

··· 1 1 - /* 2 2 - * This file is provided under a dual BSD/GPLv2 license. When using or 3 3 - * redistributing this file, you may do so under either license. 4 4 - * 5 5 - * GPL LICENSE SUMMARY 6 6 - * 7 7 - * Copyright(c) 2012 Intel Corporation. All rights reserved. 8 8 - * 9 9 - * This program is free software; you can redistribute it and/or modify 10 10 - * it under the terms of version 2 of the GNU General Public License as 11 11 - * published by the Free Software Foundation. 12 12 - * 13 13 - * BSD LICENSE 14 14 - * 15 15 - * Copyright(c) 2012 Intel Corporation. All rights reserved. 16 16 - * 17 17 - * Redistribution and use in source and binary forms, with or without 18 18 - * modification, are permitted provided that the following conditions 19 19 - * are met: 20 20 - * 21 21 - * * Redistributions of source code must retain the above copyright 22 22 - * notice, this list of conditions and the following disclaimer. 23 23 - * * Redistributions in binary form must reproduce the above copy 24 24 - * notice, this list of conditions and the following disclaimer in 25 25 - * the documentation and/or other materials provided with the 26 26 - * distribution. 27 27 - * * Neither the name of Intel Corporation nor the names of its 28 28 - * contributors may be used to endorse or promote products derived 29 29 - * from this software without specific prior written permission. 30 30 - * 31 31 - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 32 32 - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 33 33 - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 34 34 - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 35 35 - * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 36 36 - * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 37 37 - * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 38 38 - * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 39 39 - * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 40 40 - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 41 41 - * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 42 42 - * 43 43 - * Intel PCIe NTB Linux driver 44 44 - * 45 45 - * Contact Information: 46 46 - * Jon Mason <jon.mason@intel.com> 47 47 - */ 48 48 - #include <linux/ntb.h> 49 49 - 50 50 - #define PCI_DEVICE_ID_INTEL_NTB_B2B_JSF 0x3725 51 51 - #define PCI_DEVICE_ID_INTEL_NTB_PS_JSF 0x3726 52 52 - #define PCI_DEVICE_ID_INTEL_NTB_SS_JSF 0x3727 53 53 - #define PCI_DEVICE_ID_INTEL_NTB_B2B_SNB 0x3C0D 54 54 - #define PCI_DEVICE_ID_INTEL_NTB_PS_SNB 0x3C0E 55 55 - #define PCI_DEVICE_ID_INTEL_NTB_SS_SNB 0x3C0F 56 56 - #define PCI_DEVICE_ID_INTEL_NTB_B2B_IVT 0x0E0D 57 57 - #define PCI_DEVICE_ID_INTEL_NTB_PS_IVT 0x0E0E 58 58 - #define PCI_DEVICE_ID_INTEL_NTB_SS_IVT 0x0E0F 59 59 - #define PCI_DEVICE_ID_INTEL_NTB_B2B_HSX 0x2F0D 60 60 - #define PCI_DEVICE_ID_INTEL_NTB_PS_HSX 0x2F0E 61 61 - #define PCI_DEVICE_ID_INTEL_NTB_SS_HSX 0x2F0F 62 62 - #define PCI_DEVICE_ID_INTEL_NTB_B2B_BWD 0x0C4E 63 63 - 64 64 - #ifndef readq 65 65 - static inline u64 readq(void __iomem *addr) 66 66 - { 67 67 - return readl(addr) | (((u64) readl(addr + 4)) << 32LL); 68 68 - } 69 69 - #endif 70 70 - 71 71 - #ifndef writeq 72 72 - static inline void writeq(u64 val, void __iomem *addr) 73 73 - { 74 74 - writel(val & 0xffffffff, addr); 75 75 - writel(val >> 32, addr + 4); 76 76 - } 77 77 - #endif 78 78 - 79 79 - #define NTB_BAR_MMIO 0 80 80 - #define NTB_BAR_23 2 81 81 - #define NTB_BAR_4 4 82 82 - #define NTB_BAR_5 5 83 83 - 84 84 - #define NTB_BAR_MASK ((1 << NTB_BAR_MMIO) | (1 << NTB_BAR_23) |\ 85 85 - (1 << NTB_BAR_4)) 86 86 - #define NTB_SPLITBAR_MASK ((1 << NTB_BAR_MMIO) | (1 << NTB_BAR_23) |\ 87 87 - (1 << NTB_BAR_4) | (1 << NTB_BAR_5)) 88 88 - 89 89 - #define NTB_HB_TIMEOUT msecs_to_jiffies(1000) 90 90 - 91 91 - enum ntb_hw_event { 92 92 - NTB_EVENT_SW_EVENT0 = 0, 93 93 - NTB_EVENT_SW_EVENT1, 94 94 - NTB_EVENT_SW_EVENT2, 95 95 - NTB_EVENT_HW_ERROR, 96 96 - NTB_EVENT_HW_LINK_UP, 97 97 - NTB_EVENT_HW_LINK_DOWN, 98 98 - }; 99 99 - 100 100 - struct ntb_mw { 101 101 - dma_addr_t phys_addr; 102 102 - void __iomem *vbase; 103 103 - resource_size_t bar_sz; 104 104 - }; 105 105 - 106 106 - struct ntb_db_cb { 107 107 - int (*callback)(void *data, int db_num); 108 108 - unsigned int db_num; 109 109 - void *data; 110 110 - struct ntb_device *ndev; 111 111 - struct tasklet_struct irq_work; 112 112 - }; 113 113 - 114 114 - #define WA_SNB_ERR 0x00000001 115 115 - 116 116 - struct ntb_device { 117 117 - struct pci_dev *pdev; 118 118 - struct msix_entry *msix_entries; 119 119 - void __iomem *reg_base; 120 120 - struct ntb_mw *mw; 121 121 - struct { 122 122 - unsigned char max_mw; 123 123 - unsigned char max_spads; 124 124 - unsigned char max_db_bits; 125 125 - unsigned char msix_cnt; 126 126 - } limits; 127 127 - struct { 128 128 - void __iomem *ldb; 129 129 - void __iomem *ldb_mask; 130 130 - void __iomem *rdb; 131 131 - void __iomem *bar2_xlat; 132 132 - void __iomem *bar4_xlat; 133 133 - void __iomem *bar5_xlat; 134 134 - void __iomem *spad_write; 135 135 - void __iomem *spad_read; 136 136 - void __iomem *lnk_cntl; 137 137 - void __iomem *lnk_stat; 138 138 - void __iomem *spci_cmd; 139 139 - } reg_ofs; 140 140 - struct ntb_transport *ntb_transport; 141 141 - void (*event_cb)(void *handle, enum ntb_hw_event event); 142 142 - 143 143 - struct ntb_db_cb *db_cb; 144 144 - unsigned char hw_type; 145 145 - unsigned char conn_type; 146 146 - unsigned char dev_type; 147 147 - unsigned char num_msix; 148 148 - unsigned char bits_per_vector; 149 149 - unsigned char max_cbs; 150 150 - unsigned char link_width; 151 151 - unsigned char link_speed; 152 152 - unsigned char link_status; 153 153 - unsigned char split_bar; 154 154 - 155 155 - struct delayed_work hb_timer; 156 156 - unsigned long last_ts; 157 157 - 158 158 - struct delayed_work lr_timer; 159 159 - 160 160 - struct dentry *debugfs_dir; 161 161 - struct dentry *debugfs_info; 162 162 - 163 163 - unsigned int wa_flags; 164 164 - }; 165 165 - 166 166 - /** 167 167 - * ntb_max_cbs() - return the max callbacks 168 168 - * @ndev: pointer to ntb_device instance 169 169 - * 170 170 - * Given the ntb pointer, return the maximum number of callbacks 171 171 - * 172 172 - * RETURNS: the maximum number of callbacks 173 173 - */ 174 174 - static inline unsigned char ntb_max_cbs(struct ntb_device *ndev) 175 175 - { 176 176 - return ndev->max_cbs; 177 177 - } 178 178 - 179 179 - /** 180 180 - * ntb_max_mw() - return the max number of memory windows 181 181 - * @ndev: pointer to ntb_device instance 182 182 - * 183 183 - * Given the ntb pointer, return the maximum number of memory windows 184 184 - * 185 185 - * RETURNS: the maximum number of memory windows 186 186 - */ 187 187 - static inline unsigned char ntb_max_mw(struct ntb_device *ndev) 188 188 - { 189 189 - return ndev->limits.max_mw; 190 190 - } 191 191 - 192 192 - /** 193 193 - * ntb_hw_link_status() - return the hardware link status 194 194 - * @ndev: pointer to ntb_device instance 195 195 - * 196 196 - * Returns true if the hardware is connected to the remote system 197 197 - * 198 198 - * RETURNS: true or false based on the hardware link state 199 199 - */ 200 200 - static inline bool ntb_hw_link_status(struct ntb_device *ndev) 201 201 - { 202 202 - return ndev->link_status == NTB_LINK_UP; 203 203 - } 204 204 - 205 205 - /** 206 206 - * ntb_query_pdev() - return the pci_dev pointer 207 207 - * @ndev: pointer to ntb_device instance 208 208 - * 209 209 - * Given the ntb pointer, return the pci_dev pointer for the NTB hardware device 210 210 - * 211 211 - * RETURNS: a pointer to the ntb pci_dev 212 212 - */ 213 213 - static inline struct pci_dev *ntb_query_pdev(struct ntb_device *ndev) 214 214 - { 215 215 - return ndev->pdev; 216 216 - } 217 217 - 218 218 - /** 219 219 - * ntb_query_debugfs() - return the debugfs pointer 220 220 - * @ndev: pointer to ntb_device instance 221 221 - * 222 222 - * Given the ntb pointer, return the debugfs directory pointer for the NTB 223 223 - * hardware device 224 224 - * 225 225 - * RETURNS: a pointer to the debugfs directory 226 226 - */ 227 227 - static inline struct dentry *ntb_query_debugfs(struct ntb_device *ndev) 228 228 - { 229 229 - return ndev->debugfs_dir; 230 230 - } 231 231 - 232 232 - struct ntb_device *ntb_register_transport(struct pci_dev *pdev, 233 233 - void *transport); 234 234 - void ntb_unregister_transport(struct ntb_device *ndev); 235 235 - void ntb_set_mw_addr(struct ntb_device *ndev, unsigned int mw, u64 addr); 236 236 - int ntb_register_db_callback(struct ntb_device *ndev, unsigned int idx, 237 237 - void *data, int (*db_cb_func)(void *data, 238 238 - int db_num)); 239 239 - void ntb_unregister_db_callback(struct ntb_device *ndev, unsigned int idx); 240 240 - int ntb_register_event_callback(struct ntb_device *ndev, 241 241 - void (*event_cb_func)(void *handle, 242 242 - enum ntb_hw_event event)); 243 243 - void ntb_unregister_event_callback(struct ntb_device *ndev); 244 244 - int ntb_get_max_spads(struct ntb_device *ndev); 245 245 - int ntb_write_local_spad(struct ntb_device *ndev, unsigned int idx, u32 val); 246 246 - int ntb_read_local_spad(struct ntb_device *ndev, unsigned int idx, u32 *val); 247 247 - int ntb_write_remote_spad(struct ntb_device *ndev, unsigned int idx, u32 val); 248 248 - int ntb_read_remote_spad(struct ntb_device *ndev, unsigned int idx, u32 *val); 249 249 - resource_size_t ntb_get_mw_base(struct ntb_device *ndev, unsigned int mw); 250 250 - void __iomem *ntb_get_mw_vbase(struct ntb_device *ndev, unsigned int mw); 251 251 - u64 ntb_get_mw_size(struct ntb_device *ndev, unsigned int mw); 252 252 - void ntb_ring_doorbell(struct ntb_device *ndev, unsigned int idx); 253 253 - void *ntb_find_transport(struct pci_dev *pdev); 254 254 - 255 255 - int ntb_transport_init(struct pci_dev *pdev); 256 256 - void ntb_transport_free(void *transport);

-177

drivers/ntb/ntb_regs.h

reviewed

··· 1 1 - /* 2 2 - * This file is provided under a dual BSD/GPLv2 license. When using or 3 3 - * redistributing this file, you may do so under either license. 4 4 - * 5 5 - * GPL LICENSE SUMMARY 6 6 - * 7 7 - * Copyright(c) 2012 Intel Corporation. All rights reserved. 8 8 - * 9 9 - * This program is free software; you can redistribute it and/or modify 10 10 - * it under the terms of version 2 of the GNU General Public License as 11 11 - * published by the Free Software Foundation. 12 12 - * 13 13 - * BSD LICENSE 14 14 - * 15 15 - * Copyright(c) 2012 Intel Corporation. All rights reserved. 16 16 - * 17 17 - * Redistribution and use in source and binary forms, with or without 18 18 - * modification, are permitted provided that the following conditions 19 19 - * are met: 20 20 - * 21 21 - * * Redistributions of source code must retain the above copyright 22 22 - * notice, this list of conditions and the following disclaimer. 23 23 - * * Redistributions in binary form must reproduce the above copy 24 24 - * notice, this list of conditions and the following disclaimer in 25 25 - * the documentation and/or other materials provided with the 26 26 - * distribution. 27 27 - * * Neither the name of Intel Corporation nor the names of its 28 28 - * contributors may be used to endorse or promote products derived 29 29 - * from this software without specific prior written permission. 30 30 - * 31 31 - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 32 32 - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 33 33 - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 34 34 - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 35 35 - * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 36 36 - * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 37 37 - * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 38 38 - * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 39 39 - * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 40 40 - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 41 41 - * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 42 42 - * 43 43 - * Intel PCIe NTB Linux driver 44 44 - * 45 45 - * Contact Information: 46 46 - * Jon Mason <jon.mason@intel.com> 47 47 - */ 48 48 - 49 49 - #define NTB_LINK_STATUS_ACTIVE 0x2000 50 50 - #define NTB_LINK_SPEED_MASK 0x000f 51 51 - #define NTB_LINK_WIDTH_MASK 0x03f0 52 52 - 53 53 - #define SNB_MSIX_CNT 4 54 54 - #define SNB_MAX_B2B_SPADS 16 55 55 - #define SNB_MAX_COMPAT_SPADS 16 56 56 - /* Reserve the uppermost bit for link interrupt */ 57 57 - #define SNB_MAX_DB_BITS 15 58 58 - #define SNB_LINK_DB 15 59 59 - #define SNB_DB_BITS_PER_VEC 5 60 60 - #define HSX_SPLITBAR_MAX_MW 3 61 61 - #define SNB_MAX_MW 2 62 62 - #define SNB_ERRATA_MAX_MW 1 63 63 - 64 64 - #define SNB_DB_HW_LINK 0x8000 65 65 - 66 66 - #define SNB_UNCERRSTS_OFFSET 0x014C 67 67 - #define SNB_CORERRSTS_OFFSET 0x0158 68 68 - #define SNB_LINK_STATUS_OFFSET 0x01A2 69 69 - #define SNB_PCICMD_OFFSET 0x0504 70 70 - #define SNB_DEVCTRL_OFFSET 0x0598 71 71 - #define SNB_DEVSTS_OFFSET 0x059A 72 72 - #define SNB_SLINK_STATUS_OFFSET 0x05A2 73 73 - 74 74 - #define SNB_PBAR2LMT_OFFSET 0x0000 75 75 - #define SNB_PBAR4LMT_OFFSET 0x0008 76 76 - #define SNB_PBAR5LMT_OFFSET 0x000C 77 77 - #define SNB_PBAR2XLAT_OFFSET 0x0010 78 78 - #define SNB_PBAR4XLAT_OFFSET 0x0018 79 79 - #define SNB_PBAR5XLAT_OFFSET 0x001C 80 80 - #define SNB_SBAR2LMT_OFFSET 0x0020 81 81 - #define SNB_SBAR4LMT_OFFSET 0x0028 82 82 - #define SNB_SBAR5LMT_OFFSET 0x002C 83 83 - #define SNB_SBAR2XLAT_OFFSET 0x0030 84 84 - #define SNB_SBAR4XLAT_OFFSET 0x0038 85 85 - #define SNB_SBAR5XLAT_OFFSET 0x003C 86 86 - #define SNB_SBAR0BASE_OFFSET 0x0040 87 87 - #define SNB_SBAR2BASE_OFFSET 0x0048 88 88 - #define SNB_SBAR4BASE_OFFSET 0x0050 89 89 - #define SNB_SBAR5BASE_OFFSET 0x0054 90 90 - #define SNB_NTBCNTL_OFFSET 0x0058 91 91 - #define SNB_SBDF_OFFSET 0x005C 92 92 - #define SNB_PDOORBELL_OFFSET 0x0060 93 93 - #define SNB_PDBMSK_OFFSET 0x0062 94 94 - #define SNB_SDOORBELL_OFFSET 0x0064 95 95 - #define SNB_SDBMSK_OFFSET 0x0066 96 96 - #define SNB_USMEMMISS_OFFSET 0x0070 97 97 - #define SNB_SPAD_OFFSET 0x0080 98 98 - #define SNB_SPADSEMA4_OFFSET 0x00c0 99 99 - #define SNB_WCCNTRL_OFFSET 0x00e0 100 100 - #define SNB_B2B_SPAD_OFFSET 0x0100 101 101 - #define SNB_B2B_DOORBELL_OFFSET 0x0140 102 102 - #define SNB_B2B_XLAT_OFFSETL 0x0144 103 103 - #define SNB_B2B_XLAT_OFFSETU 0x0148 104 104 - 105 105 - /* 106 106 - * The addresses are setup so the 32bit BARs can function. Thus 107 107 - * the addresses are all in 32bit space 108 108 - */ 109 109 - #define SNB_MBAR01_USD_ADDR 0x000000002100000CULL 110 110 - #define SNB_MBAR23_USD_ADDR 0x000000004100000CULL 111 111 - #define SNB_MBAR4_USD_ADDR 0x000000008100000CULL 112 112 - #define SNB_MBAR5_USD_ADDR 0x00000000A100000CULL 113 113 - #define SNB_MBAR01_DSD_ADDR 0x000000002000000CULL 114 114 - #define SNB_MBAR23_DSD_ADDR 0x000000004000000CULL 115 115 - #define SNB_MBAR4_DSD_ADDR 0x000000008000000CULL 116 116 - #define SNB_MBAR5_DSD_ADDR 0x00000000A000000CULL 117 117 - 118 118 - #define BWD_MSIX_CNT 34 119 119 - #define BWD_MAX_SPADS 16 120 120 - #define BWD_MAX_DB_BITS 34 121 121 - #define BWD_DB_BITS_PER_VEC 1 122 122 - #define BWD_MAX_MW 2 123 123 - 124 124 - #define BWD_PCICMD_OFFSET 0xb004 125 125 - #define BWD_MBAR23_OFFSET 0xb018 126 126 - #define BWD_MBAR45_OFFSET 0xb020 127 127 - #define BWD_DEVCTRL_OFFSET 0xb048 128 128 - #define BWD_LINK_STATUS_OFFSET 0xb052 129 129 - #define BWD_ERRCORSTS_OFFSET 0xb110 130 130 - 131 131 - #define BWD_SBAR2XLAT_OFFSET 0x0008 132 132 - #define BWD_SBAR4XLAT_OFFSET 0x0010 133 133 - #define BWD_PDOORBELL_OFFSET 0x0020 134 134 - #define BWD_PDBMSK_OFFSET 0x0028 135 135 - #define BWD_NTBCNTL_OFFSET 0x0060 136 136 - #define BWD_EBDF_OFFSET 0x0064 137 137 - #define BWD_SPAD_OFFSET 0x0080 138 138 - #define BWD_SPADSEMA_OFFSET 0x00c0 139 139 - #define BWD_STKYSPAD_OFFSET 0x00c4 140 140 - #define BWD_PBAR2XLAT_OFFSET 0x8008 141 141 - #define BWD_PBAR4XLAT_OFFSET 0x8010 142 142 - #define BWD_B2B_DOORBELL_OFFSET 0x8020 143 143 - #define BWD_B2B_SPAD_OFFSET 0x8080 144 144 - #define BWD_B2B_SPADSEMA_OFFSET 0x80c0 145 145 - #define BWD_B2B_STKYSPAD_OFFSET 0x80c4 146 146 - 147 147 - #define BWD_MODPHY_PCSREG4 0x1c004 148 148 - #define BWD_MODPHY_PCSREG6 0x1c006 149 149 - 150 150 - #define BWD_IP_BASE 0xC000 151 151 - #define BWD_DESKEWSTS_OFFSET (BWD_IP_BASE + 0x3024) 152 152 - #define BWD_LTSSMERRSTS0_OFFSET (BWD_IP_BASE + 0x3180) 153 153 - #define BWD_LTSSMSTATEJMP_OFFSET (BWD_IP_BASE + 0x3040) 154 154 - #define BWD_IBSTERRRCRVSTS0_OFFSET (BWD_IP_BASE + 0x3324) 155 155 - 156 156 - #define BWD_DESKEWSTS_DBERR (1 << 15) 157 157 - #define BWD_LTSSMERRSTS0_UNEXPECTEDEI (1 << 20) 158 158 - #define BWD_LTSSMSTATEJMP_FORCEDETECT (1 << 2) 159 159 - #define BWD_IBIST_ERR_OFLOW 0x7FFF7FFF 160 160 - 161 161 - #define NTB_CNTL_CFG_LOCK (1 << 0) 162 162 - #define NTB_CNTL_LINK_DISABLE (1 << 1) 163 163 - #define NTB_CNTL_S2P_BAR23_SNOOP (1 << 2) 164 164 - #define NTB_CNTL_P2S_BAR23_SNOOP (1 << 4) 165 165 - #define NTB_CNTL_S2P_BAR4_SNOOP (1 << 6) 166 166 - #define NTB_CNTL_P2S_BAR4_SNOOP (1 << 8) 167 167 - #define NTB_CNTL_S2P_BAR5_SNOOP (1 << 12) 168 168 - #define NTB_CNTL_P2S_BAR5_SNOOP (1 << 14) 169 169 - #define BWD_CNTL_LINK_DOWN (1 << 16) 170 170 - 171 171 - #define NTB_PPD_OFFSET 0x00D4 172 172 - #define SNB_PPD_CONN_TYPE 0x0003 173 173 - #define SNB_PPD_DEV_TYPE 0x0010 174 174 - #define SNB_PPD_SPLIT_BAR (1 << 6) 175 175 - #define BWD_PPD_INIT_LINK 0x0008 176 176 - #define BWD_PPD_CONN_TYPE 0x0300 177 177 - #define BWD_PPD_DEV_TYPE 0x1000

+624 -409

drivers/ntb/ntb_transport.c

reviewed

··· 5 5 * GPL LICENSE SUMMARY 6 6 * 7 7 * Copyright(c) 2012 Intel Corporation. All rights reserved. 8 8 + * Copyright (C) 2015 EMC Corporation. All Rights Reserved. 8 9 * 9 10 * This program is free software; you can redistribute it and/or modify 10 11 * it under the terms of version 2 of the GNU General Public License as ··· 14 13 * BSD LICENSE 15 14 * 16 15 * Copyright(c) 2012 Intel Corporation. All rights reserved. 16 16 + * Copyright (C) 2015 EMC Corporation. All Rights Reserved. 17 17 * 18 18 * Redistribution and use in source and binary forms, with or without 19 19 * modification, are permitted provided that the following conditions ··· 42 40 * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 43 41 * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 44 42 * 45 45 - * Intel PCIe NTB Linux driver 43 43 + * PCIe NTB Transport Linux driver 46 44 * 47 45 * Contact Information: 48 46 * Jon Mason <jon.mason@intel.com> ··· 58 56 #include <linux/pci.h> 59 57 #include <linux/slab.h> 60 58 #include <linux/types.h> 61 61 - #include "ntb_hw.h" 59 59 + #include <linux/uaccess.h> 60 60 + #include "linux/ntb.h" 61 61 + #include "linux/ntb_transport.h" 62 62 63 63 - #define NTB_TRANSPORT_VERSION 3 63 63 + #define NTB_TRANSPORT_VERSION 4 64 64 + #define NTB_TRANSPORT_VER "4" 65 65 + #define NTB_TRANSPORT_NAME "ntb_transport" 66 66 + #define NTB_TRANSPORT_DESC "Software Queue-Pair Transport over NTB" 64 67 65 65 - static unsigned int transport_mtu = 0x401E; 68 68 + MODULE_DESCRIPTION(NTB_TRANSPORT_DESC); 69 69 + MODULE_VERSION(NTB_TRANSPORT_VER); 70 70 + MODULE_LICENSE("Dual BSD/GPL"); 71 71 + MODULE_AUTHOR("Intel Corporation"); 72 72 + 73 73 + static unsigned long max_mw_size; 74 74 + module_param(max_mw_size, ulong, 0644); 75 75 + MODULE_PARM_DESC(max_mw_size, "Limit size of large memory windows"); 76 76 + 77 77 + static unsigned int transport_mtu = 0x10000; 66 78 module_param(transport_mtu, uint, 0644); 67 79 MODULE_PARM_DESC(transport_mtu, "Maximum size of NTB transport packets"); 68 80 ··· 88 72 module_param(copy_bytes, uint, 0644); 89 73 MODULE_PARM_DESC(copy_bytes, "Threshold under which NTB will use the CPU to copy instead of DMA"); 90 74 75 75 + static bool use_dma; 76 76 + module_param(use_dma, bool, 0644); 77 77 + MODULE_PARM_DESC(use_dma, "Use DMA engine to perform large data copy"); 78 78 + 79 79 + static struct dentry *nt_debugfs_dir; 80 80 + 91 81 struct ntb_queue_entry { 92 82 /* ntb_queue list reference */ 93 83 struct list_head entry; 94 94 - /* pointers to data to be transfered */ 84 84 + /* pointers to data to be transferred */ 95 85 void *cb_data; 96 86 void *buf; 97 87 unsigned int len; ··· 116 94 }; 117 95 118 96 struct ntb_transport_qp { 119 119 - struct ntb_transport *transport; 120 120 - struct ntb_device *ndev; 97 97 + struct ntb_transport_ctx *transport; 98 98 + struct ntb_dev *ndev; 121 99 void *cb_data; 122 100 struct dma_chan *dma_chan; 123 101 124 102 bool client_ready; 125 125 - bool qp_link; 103 103 + bool link_is_up; 104 104 + 126 105 u8 qp_num; /* Only 64 QP's are allowed. 0-63 */ 106 106 + u64 qp_bit; 127 107 128 108 struct ntb_rx_info __iomem *rx_info; 129 109 struct ntb_rx_info *remote_rx_info; ··· 151 127 unsigned int rx_max_entry; 152 128 unsigned int rx_max_frame; 153 129 dma_cookie_t last_cookie; 130 130 + struct tasklet_struct rxc_db_work; 154 131 155 132 void (*event_handler)(void *data, int status); 156 133 struct delayed_work link_work; ··· 178 153 }; 179 154 180 155 struct ntb_transport_mw { 181 181 - size_t size; 156 156 + phys_addr_t phys_addr; 157 157 + resource_size_t phys_size; 158 158 + resource_size_t xlat_align; 159 159 + resource_size_t xlat_align_size; 160 160 + void __iomem *vbase; 161 161 + size_t xlat_size; 162 162 + size_t buff_size; 182 163 void *virt_addr; 183 164 dma_addr_t dma_addr; 184 165 }; 185 166 186 167 struct ntb_transport_client_dev { 187 168 struct list_head entry; 169 169 + struct ntb_transport_ctx *nt; 188 170 struct device dev; 189 171 }; 190 172 191 191 - struct ntb_transport { 173 173 + struct ntb_transport_ctx { 192 174 struct list_head entry; 193 175 struct list_head client_devs; 194 176 195 195 - struct ntb_device *ndev; 196 196 - struct ntb_transport_mw *mw; 197 197 - struct ntb_transport_qp *qps; 198 198 - unsigned int max_qps; 199 199 - unsigned long qp_bitmap; 200 200 - bool transport_link; 177 177 + struct ntb_dev *ndev; 178 178 + 179 179 + struct ntb_transport_mw *mw_vec; 180 180 + struct ntb_transport_qp *qp_vec; 181 181 + unsigned int mw_count; 182 182 + unsigned int qp_count; 183 183 + u64 qp_bitmap; 184 184 + u64 qp_bitmap_free; 185 185 + 186 186 + bool link_is_up; 201 187 struct delayed_work link_work; 202 188 struct work_struct link_cleanup; 203 189 }; 204 190 205 191 enum { 206 206 - DESC_DONE_FLAG = 1 << 0, 207 207 - LINK_DOWN_FLAG = 1 << 1, 192 192 + DESC_DONE_FLAG = BIT(0), 193 193 + LINK_DOWN_FLAG = BIT(1), 208 194 }; 209 195 210 196 struct ntb_payload_header { ··· 236 200 MAX_SPAD, 237 201 }; 238 202 239 239 - #define QP_TO_MW(ndev, qp) ((qp) % ntb_max_mw(ndev)) 203 203 + #define dev_client_dev(__dev) \ 204 204 + container_of((__dev), struct ntb_transport_client_dev, dev) 205 205 + 206 206 + #define drv_client(__drv) \ 207 207 + container_of((__drv), struct ntb_transport_client, driver) 208 208 + 209 209 + #define QP_TO_MW(nt, qp) ((qp) % nt->mw_count) 240 210 #define NTB_QP_DEF_NUM_ENTRIES 100 241 211 #define NTB_LINK_DOWN_TIMEOUT 10 242 212 243 243 - static int ntb_match_bus(struct device *dev, struct device_driver *drv) 213 213 + static void ntb_transport_rxc_db(unsigned long data); 214 214 + static const struct ntb_ctx_ops ntb_transport_ops; 215 215 + static struct ntb_client ntb_transport_client; 216 216 + 217 217 + static int ntb_transport_bus_match(struct device *dev, 218 218 + struct device_driver *drv) 244 219 { 245 220 return !strncmp(dev_name(dev), drv->name, strlen(drv->name)); 246 221 } 247 222 248 248 - static int ntb_client_probe(struct device *dev) 223 223 + static int ntb_transport_bus_probe(struct device *dev) 249 224 { 250 250 - const struct ntb_client *drv = container_of(dev->driver, 251 251 - struct ntb_client, driver); 252 252 - struct pci_dev *pdev = container_of(dev->parent, struct pci_dev, dev); 225 225 + const struct ntb_transport_client *client; 253 226 int rc = -EINVAL; 254 227 255 228 get_device(dev); 256 256 - if (drv && drv->probe) 257 257 - rc = drv->probe(pdev); 229 229 + 230 230 + client = drv_client(dev->driver); 231 231 + rc = client->probe(dev); 258 232 if (rc) 259 233 put_device(dev); 260 234 261 235 return rc; 262 236 } 263 237 264 264 - static int ntb_client_remove(struct device *dev) 238 238 + static int ntb_transport_bus_remove(struct device *dev) 265 239 { 266 266 - const struct ntb_client *drv = container_of(dev->driver, 267 267 - struct ntb_client, driver); 268 268 - struct pci_dev *pdev = container_of(dev->parent, struct pci_dev, dev); 240 240 + const struct ntb_transport_client *client; 269 241 270 270 - if (drv && drv->remove) 271 271 - drv->remove(pdev); 242 242 + client = drv_client(dev->driver); 243 243 + client->remove(dev); 272 244 273 245 put_device(dev); 274 246 275 247 return 0; 276 248 } 277 249 278 278 - static struct bus_type ntb_bus_type = { 279 279 - .name = "ntb_bus", 280 280 - .match = ntb_match_bus, 281 281 - .probe = ntb_client_probe, 282 282 - .remove = ntb_client_remove, 250 250 + static struct bus_type ntb_transport_bus = { 251 251 + .name = "ntb_transport", 252 252 + .match = ntb_transport_bus_match, 253 253 + .probe = ntb_transport_bus_probe, 254 254 + .remove = ntb_transport_bus_remove, 283 255 }; 284 256 285 257 static LIST_HEAD(ntb_transport_list); 286 258 287 287 - static int ntb_bus_init(struct ntb_transport *nt) 259 259 + static int ntb_bus_init(struct ntb_transport_ctx *nt) 288 260 { 289 289 - if (list_empty(&ntb_transport_list)) { 290 290 - int rc = bus_register(&ntb_bus_type); 291 291 - if (rc) 292 292 - return rc; 293 293 - } 294 294 - 295 261 list_add(&nt->entry, &ntb_transport_list); 296 296 - 297 262 return 0; 298 263 } 299 264 300 300 - static void ntb_bus_remove(struct ntb_transport *nt) 265 265 + static void ntb_bus_remove(struct ntb_transport_ctx *nt) 301 266 { 302 267 struct ntb_transport_client_dev *client_dev, *cd; 303 268 ··· 310 273 } 311 274 312 275 list_del(&nt->entry); 313 313 - 314 314 - if (list_empty(&ntb_transport_list)) 315 315 - bus_unregister(&ntb_bus_type); 316 276 } 317 277 318 318 - static void ntb_client_release(struct device *dev) 278 278 + static void ntb_transport_client_release(struct device *dev) 319 279 { 320 280 struct ntb_transport_client_dev *client_dev; 321 321 - client_dev = container_of(dev, struct ntb_transport_client_dev, dev); 322 281 282 282 + client_dev = dev_client_dev(dev); 323 283 kfree(client_dev); 324 284 } 325 285 326 286 /** 327 327 - * ntb_unregister_client_dev - Unregister NTB client device 287 287 + * ntb_transport_unregister_client_dev - Unregister NTB client device 328 288 * @device_name: Name of NTB client device 329 289 * 330 290 * Unregister an NTB client device with the NTB transport layer 331 291 */ 332 332 - void ntb_unregister_client_dev(char *device_name) 292 292 + void ntb_transport_unregister_client_dev(char *device_name) 333 293 { 334 294 struct ntb_transport_client_dev *client, *cd; 335 335 - struct ntb_transport *nt; 295 295 + struct ntb_transport_ctx *nt; 336 296 337 297 list_for_each_entry(nt, &ntb_transport_list, entry) 338 298 list_for_each_entry_safe(client, cd, &nt->client_devs, entry) ··· 339 305 device_unregister(&client->dev); 340 306 } 341 307 } 342 342 - EXPORT_SYMBOL_GPL(ntb_unregister_client_dev); 308 308 + EXPORT_SYMBOL_GPL(ntb_transport_unregister_client_dev); 343 309 344 310 /** 345 345 - * ntb_register_client_dev - Register NTB client device 311 311 + * ntb_transport_register_client_dev - Register NTB client device 346 312 * @device_name: Name of NTB client device 347 313 * 348 314 * Register an NTB client device with the NTB transport layer 349 315 */ 350 350 - int ntb_register_client_dev(char *device_name) 316 316 + int ntb_transport_register_client_dev(char *device_name) 351 317 { 352 318 struct ntb_transport_client_dev *client_dev; 353 353 - struct ntb_transport *nt; 319 319 + struct ntb_transport_ctx *nt; 320 320 + int node; 354 321 int rc, i = 0; 355 322 356 323 if (list_empty(&ntb_transport_list)) ··· 360 325 list_for_each_entry(nt, &ntb_transport_list, entry) { 361 326 struct device *dev; 362 327 363 363 - client_dev = kzalloc(sizeof(struct ntb_transport_client_dev), 364 364 - GFP_KERNEL); 328 328 + node = dev_to_node(&nt->ndev->dev); 329 329 + 330 330 + client_dev = kzalloc_node(sizeof(*client_dev), 331 331 + GFP_KERNEL, node); 365 332 if (!client_dev) { 366 333 rc = -ENOMEM; 367 334 goto err; ··· 373 336 374 337 /* setup and register client devices */ 375 338 dev_set_name(dev, "%s%d", device_name, i); 376 376 - dev->bus = &ntb_bus_type; 377 377 - dev->release = ntb_client_release; 378 378 - dev->parent = &ntb_query_pdev(nt->ndev)->dev; 339 339 + dev->bus = &ntb_transport_bus; 340 340 + dev->release = ntb_transport_client_release; 341 341 + dev->parent = &nt->ndev->dev; 379 342 380 343 rc = device_register(dev); 381 344 if (rc) { ··· 390 353 return 0; 391 354 392 355 err: 393 393 - ntb_unregister_client_dev(device_name); 356 356 + ntb_transport_unregister_client_dev(device_name); 394 357 395 358 return rc; 396 359 } 397 397 - EXPORT_SYMBOL_GPL(ntb_register_client_dev); 360 360 + EXPORT_SYMBOL_GPL(ntb_transport_register_client_dev); 398 361 399 362 /** 400 400 - * ntb_register_client - Register NTB client driver 363 363 + * ntb_transport_register_client - Register NTB client driver 401 364 * @drv: NTB client driver to be registered 402 365 * 403 366 * Register an NTB client driver with the NTB transport layer 404 367 * 405 368 * RETURNS: An appropriate -ERRNO error value on error, or zero for success. 406 369 */ 407 407 - int ntb_register_client(struct ntb_client *drv) 370 370 + int ntb_transport_register_client(struct ntb_transport_client *drv) 408 371 { 409 409 - drv->driver.bus = &ntb_bus_type; 372 372 + drv->driver.bus = &ntb_transport_bus; 410 373 411 374 if (list_empty(&ntb_transport_list)) 412 375 return -ENODEV; 413 376 414 377 return driver_register(&drv->driver); 415 378 } 416 416 - EXPORT_SYMBOL_GPL(ntb_register_client); 379 379 + EXPORT_SYMBOL_GPL(ntb_transport_register_client); 417 380 418 381 /** 419 419 - * ntb_unregister_client - Unregister NTB client driver 382 382 + * ntb_transport_unregister_client - Unregister NTB client driver 420 383 * @drv: NTB client driver to be unregistered 421 384 * 422 385 * Unregister an NTB client driver with the NTB transport layer 423 386 * 424 387 * RETURNS: An appropriate -ERRNO error value on error, or zero for success. 425 388 */ 426 426 - void ntb_unregister_client(struct ntb_client *drv) 389 389 + void ntb_transport_unregister_client(struct ntb_transport_client *drv) 427 390 { 428 391 driver_unregister(&drv->driver); 429 392 } 430 430 - EXPORT_SYMBOL_GPL(ntb_unregister_client); 393 393 + EXPORT_SYMBOL_GPL(ntb_transport_unregister_client); 431 394 432 395 static ssize_t debugfs_read(struct file *filp, char __user *ubuf, size_t count, 433 396 loff_t *offp) ··· 489 452 "tx_max_entry - \t%u\n", qp->tx_max_entry); 490 453 491 454 out_offset += snprintf(buf + out_offset, out_count - out_offset, 492 492 - "\nQP Link %s\n", (qp->qp_link == NTB_LINK_UP) ? 493 493 - "Up" : "Down"); 455 455 + "\nQP Link %s\n", 456 456 + qp->link_is_up ? "Up" : "Down"); 494 457 if (out_offset > out_count) 495 458 out_offset = out_count; 496 459 ··· 534 497 return entry; 535 498 } 536 499 537 537 - static void ntb_transport_setup_qp_mw(struct ntb_transport *nt, 538 538 - unsigned int qp_num) 500 500 + static int ntb_transport_setup_qp_mw(struct ntb_transport_ctx *nt, 501 501 + unsigned int qp_num) 539 502 { 540 540 - struct ntb_transport_qp *qp = &nt->qps[qp_num]; 503 503 + struct ntb_transport_qp *qp = &nt->qp_vec[qp_num]; 504 504 + struct ntb_transport_mw *mw; 541 505 unsigned int rx_size, num_qps_mw; 542 542 - u8 mw_num, mw_max; 506 506 + unsigned int mw_num, mw_count, qp_count; 543 507 unsigned int i; 544 508 545 545 - mw_max = ntb_max_mw(nt->ndev); 546 546 - mw_num = QP_TO_MW(nt->ndev, qp_num); 509 509 + mw_count = nt->mw_count; 510 510 + qp_count = nt->qp_count; 547 511 548 548 - WARN_ON(nt->mw[mw_num].virt_addr == NULL); 512 512 + mw_num = QP_TO_MW(nt, qp_num); 513 513 + mw = &nt->mw_vec[mw_num]; 549 514 550 550 - if (nt->max_qps % mw_max && mw_num + 1 < nt->max_qps / mw_max) 551 551 - num_qps_mw = nt->max_qps / mw_max + 1; 515 515 + if (!mw->virt_addr) 516 516 + return -ENOMEM; 517 517 + 518 518 + if (qp_count % mw_count && mw_num + 1 < qp_count / mw_count) 519 519 + num_qps_mw = qp_count / mw_count + 1; 552 520 else 553 553 - num_qps_mw = nt->max_qps / mw_max; 521 521 + num_qps_mw = qp_count / mw_count; 554 522 555 555 - rx_size = (unsigned int) nt->mw[mw_num].size / num_qps_mw; 556 556 - qp->rx_buff = nt->mw[mw_num].virt_addr + qp_num / mw_max * rx_size; 523 523 + rx_size = (unsigned int)mw->xlat_size / num_qps_mw; 524 524 + qp->rx_buff = mw->virt_addr + rx_size * qp_num / mw_count; 557 525 rx_size -= sizeof(struct ntb_rx_info); 558 526 559 527 qp->remote_rx_info = qp->rx_buff + rx_size; ··· 572 530 573 531 /* setup the hdr offsets with 0's */ 574 532 for (i = 0; i < qp->rx_max_entry; i++) { 575 575 - void *offset = qp->rx_buff + qp->rx_max_frame * (i + 1) - 576 576 - sizeof(struct ntb_payload_header); 533 533 + void *offset = (qp->rx_buff + qp->rx_max_frame * (i + 1) - 534 534 + sizeof(struct ntb_payload_header)); 577 535 memset(offset, 0, sizeof(struct ntb_payload_header)); 578 536 } 579 537 580 538 qp->rx_pkts = 0; 581 539 qp->tx_pkts = 0; 582 540 qp->tx_index = 0; 541 541 + 542 542 + return 0; 583 543 } 584 544 585 585 - static void ntb_free_mw(struct ntb_transport *nt, int num_mw) 545 545 + static void ntb_free_mw(struct ntb_transport_ctx *nt, int num_mw) 586 546 { 587 587 - struct ntb_transport_mw *mw = &nt->mw[num_mw]; 588 588 - struct pci_dev *pdev = ntb_query_pdev(nt->ndev); 547 547 + struct ntb_transport_mw *mw = &nt->mw_vec[num_mw]; 548 548 + struct pci_dev *pdev = nt->ndev->pdev; 589 549 590 550 if (!mw->virt_addr) 591 551 return; 592 552 593 593 - dma_free_coherent(&pdev->dev, mw->size, mw->virt_addr, mw->dma_addr); 553 553 + ntb_mw_clear_trans(nt->ndev, num_mw); 554 554 + dma_free_coherent(&pdev->dev, mw->buff_size, 555 555 + mw->virt_addr, mw->dma_addr); 556 556 + mw->xlat_size = 0; 557 557 + mw->buff_size = 0; 594 558 mw->virt_addr = NULL; 595 559 } 596 560 597 597 - static int ntb_set_mw(struct ntb_transport *nt, int num_mw, unsigned int size) 561 561 + static int ntb_set_mw(struct ntb_transport_ctx *nt, int num_mw, 562 562 + unsigned int size) 598 563 { 599 599 - struct ntb_transport_mw *mw = &nt->mw[num_mw]; 600 600 - struct pci_dev *pdev = ntb_query_pdev(nt->ndev); 564 564 + struct ntb_transport_mw *mw = &nt->mw_vec[num_mw]; 565 565 + struct pci_dev *pdev = nt->ndev->pdev; 566 566 + unsigned int xlat_size, buff_size; 567 567 + int rc; 568 568 + 569 569 + xlat_size = round_up(size, mw->xlat_align_size); 570 570 + buff_size = round_up(size, mw->xlat_align); 601 571 602 572 /* No need to re-setup */ 603 603 - if (mw->size == ALIGN(size, 4096)) 573 573 + if (mw->xlat_size == xlat_size) 604 574 return 0; 605 575 606 606 - if (mw->size != 0) 576 576 + if (mw->buff_size) 607 577 ntb_free_mw(nt, num_mw); 608 578 609 609 - /* Alloc memory for receiving data. Must be 4k aligned */ 610 610 - mw->size = ALIGN(size, 4096); 579 579 + /* Alloc memory for receiving data. Must be aligned */ 580 580 + mw->xlat_size = xlat_size; 581 581 + mw->buff_size = buff_size; 611 582 612 612 - mw->virt_addr = dma_alloc_coherent(&pdev->dev, mw->size, &mw->dma_addr, 613 613 - GFP_KERNEL); 583 583 + mw->virt_addr = dma_alloc_coherent(&pdev->dev, buff_size, 584 584 + &mw->dma_addr, GFP_KERNEL); 614 585 if (!mw->virt_addr) { 615 615 - mw->size = 0; 616 616 - dev_err(&pdev->dev, "Unable to allocate MW buffer of size %d\n", 617 617 - (int) mw->size); 586 586 + mw->xlat_size = 0; 587 587 + mw->buff_size = 0; 588 588 + dev_err(&pdev->dev, "Unable to alloc MW buff of size %d\n", 589 589 + buff_size); 618 590 return -ENOMEM; 619 591 } 620 592 ··· 638 582 * is a requirement of the hardware. It is recommended to setup CMA 639 583 * for BAR sizes equal or greater than 4MB. 640 584 */ 641 641 - if (!IS_ALIGNED(mw->dma_addr, mw->size)) { 642 642 - dev_err(&pdev->dev, "DMA memory %pad not aligned to BAR size\n", 585 585 + if (!IS_ALIGNED(mw->dma_addr, mw->xlat_align)) { 586 586 + dev_err(&pdev->dev, "DMA memory %pad is not aligned\n", 643 587 &mw->dma_addr); 644 588 ntb_free_mw(nt, num_mw); 645 589 return -ENOMEM; 646 590 } 647 591 648 592 /* Notify HW the memory location of the receive buffer */ 649 649 - ntb_set_mw_addr(nt->ndev, num_mw, mw->dma_addr); 593 593 + rc = ntb_mw_set_trans(nt->ndev, num_mw, mw->dma_addr, mw->xlat_size); 594 594 + if (rc) { 595 595 + dev_err(&pdev->dev, "Unable to set mw%d translation", num_mw); 596 596 + ntb_free_mw(nt, num_mw); 597 597 + return -EIO; 598 598 + } 650 599 651 600 return 0; 652 601 } 653 602 603 603 + static void ntb_qp_link_down_reset(struct ntb_transport_qp *qp) 604 604 + { 605 605 + qp->link_is_up = false; 606 606 + 607 607 + qp->tx_index = 0; 608 608 + qp->rx_index = 0; 609 609 + qp->rx_bytes = 0; 610 610 + qp->rx_pkts = 0; 611 611 + qp->rx_ring_empty = 0; 612 612 + qp->rx_err_no_buf = 0; 613 613 + qp->rx_err_oflow = 0; 614 614 + qp->rx_err_ver = 0; 615 615 + qp->rx_memcpy = 0; 616 616 + qp->rx_async = 0; 617 617 + qp->tx_bytes = 0; 618 618 + qp->tx_pkts = 0; 619 619 + qp->tx_ring_full = 0; 620 620 + qp->tx_err_no_buf = 0; 621 621 + qp->tx_memcpy = 0; 622 622 + qp->tx_async = 0; 623 623 + } 624 624 + 654 625 static void ntb_qp_link_cleanup(struct ntb_transport_qp *qp) 655 626 { 656 656 - struct ntb_transport *nt = qp->transport; 657 657 - struct pci_dev *pdev = ntb_query_pdev(nt->ndev); 627 627 + struct ntb_transport_ctx *nt = qp->transport; 628 628 + struct pci_dev *pdev = nt->ndev->pdev; 658 629 659 659 - if (qp->qp_link == NTB_LINK_DOWN) { 660 660 - cancel_delayed_work_sync(&qp->link_work); 661 661 - return; 662 662 - } 630 630 + dev_info(&pdev->dev, "qp %d: Link Cleanup\n", qp->qp_num); 631 631 + 632 632 + cancel_delayed_work_sync(&qp->link_work); 633 633 + ntb_qp_link_down_reset(qp); 663 634 664 635 if (qp->event_handler) 665 665 - qp->event_handler(qp->cb_data, NTB_LINK_DOWN); 666 666 - 667 667 - dev_info(&pdev->dev, "qp %d: Link Down\n", qp->qp_num); 668 668 - qp->qp_link = NTB_LINK_DOWN; 636 636 + qp->event_handler(qp->cb_data, qp->link_is_up); 669 637 } 670 638 671 639 static void ntb_qp_link_cleanup_work(struct work_struct *work) ··· 697 617 struct ntb_transport_qp *qp = container_of(work, 698 618 struct ntb_transport_qp, 699 619 link_cleanup); 700 700 - struct ntb_transport *nt = qp->transport; 620 620 + struct ntb_transport_ctx *nt = qp->transport; 701 621 702 622 ntb_qp_link_cleanup(qp); 703 623 704 704 - if (nt->transport_link == NTB_LINK_UP) 624 624 + if (nt->link_is_up) 705 625 schedule_delayed_work(&qp->link_work, 706 626 msecs_to_jiffies(NTB_LINK_DOWN_TIMEOUT)); 707 627 } ··· 711 631 schedule_work(&qp->link_cleanup); 712 632 } 713 633 714 714 - static void ntb_transport_link_cleanup(struct ntb_transport *nt) 634 634 + static void ntb_transport_link_cleanup(struct ntb_transport_ctx *nt) 715 635 { 636 636 + struct ntb_transport_qp *qp; 637 637 + u64 qp_bitmap_alloc; 716 638 int i; 717 639 718 718 - /* Pass along the info to any clients */ 719 719 - for (i = 0; i < nt->max_qps; i++) 720 720 - if (!test_bit(i, &nt->qp_bitmap)) 721 721 - ntb_qp_link_cleanup(&nt->qps[i]); 640 640 + qp_bitmap_alloc = nt->qp_bitmap & ~nt->qp_bitmap_free; 722 641 723 723 - if (nt->transport_link == NTB_LINK_DOWN) 642 642 + /* Pass along the info to any clients */ 643 643 + for (i = 0; i < nt->qp_count; i++) 644 644 + if (qp_bitmap_alloc & BIT_ULL(i)) { 645 645 + qp = &nt->qp_vec[i]; 646 646 + ntb_qp_link_cleanup(qp); 647 647 + cancel_work_sync(&qp->link_cleanup); 648 648 + cancel_delayed_work_sync(&qp->link_work); 649 649 + } 650 650 + 651 651 + if (!nt->link_is_up) 724 652 cancel_delayed_work_sync(&nt->link_work); 725 725 - else 726 726 - nt->transport_link = NTB_LINK_DOWN; 727 653 728 654 /* The scratchpad registers keep the values if the remote side 729 655 * goes down, blast them now to give them a sane value the next 730 656 * time they are accessed 731 657 */ 732 658 for (i = 0; i < MAX_SPAD; i++) 733 733 - ntb_write_local_spad(nt->ndev, i, 0); 659 659 + ntb_spad_write(nt->ndev, i, 0); 734 660 } 735 661 736 662 static void ntb_transport_link_cleanup_work(struct work_struct *work) 737 663 { 738 738 - struct ntb_transport *nt = container_of(work, struct ntb_transport, 739 739 - link_cleanup); 664 664 + struct ntb_transport_ctx *nt = 665 665 + container_of(work, struct ntb_transport_ctx, link_cleanup); 740 666 741 667 ntb_transport_link_cleanup(nt); 742 668 } 743 669 744 744 - static void ntb_transport_event_callback(void *data, enum ntb_hw_event event) 670 670 + static void ntb_transport_event_callback(void *data) 745 671 { 746 746 - struct ntb_transport *nt = data; 672 672 + struct ntb_transport_ctx *nt = data; 747 673 748 748 - switch (event) { 749 749 - case NTB_EVENT_HW_LINK_UP: 674 674 + if (ntb_link_is_up(nt->ndev, NULL, NULL) == 1) 750 675 schedule_delayed_work(&nt->link_work, 0); 751 751 - break; 752 752 - case NTB_EVENT_HW_LINK_DOWN: 676 676 + else 753 677 schedule_work(&nt->link_cleanup); 754 754 - break; 755 755 - default: 756 756 - BUG(); 757 757 - } 758 678 } 759 679 760 680 static void ntb_transport_link_work(struct work_struct *work) 761 681 { 762 762 - struct ntb_transport *nt = container_of(work, struct ntb_transport, 763 763 - link_work.work); 764 764 - struct ntb_device *ndev = nt->ndev; 765 765 - struct pci_dev *pdev = ntb_query_pdev(ndev); 682 682 + struct ntb_transport_ctx *nt = 683 683 + container_of(work, struct ntb_transport_ctx, link_work.work); 684 684 + struct ntb_dev *ndev = nt->ndev; 685 685 + struct pci_dev *pdev = ndev->pdev; 686 686 + resource_size_t size; 766 687 u32 val; 767 767 - int rc, i; 688 688 + int rc, i, spad; 768 689 769 690 /* send the local info, in the opposite order of the way we read it */ 770 770 - for (i = 0; i < ntb_max_mw(ndev); i++) { 771 771 - rc = ntb_write_remote_spad(ndev, MW0_SZ_HIGH + (i * 2), 772 772 - ntb_get_mw_size(ndev, i) >> 32); 773 773 - if (rc) { 774 774 - dev_err(&pdev->dev, "Error writing %u to remote spad %d\n", 775 775 - (u32)(ntb_get_mw_size(ndev, i) >> 32), 776 776 - MW0_SZ_HIGH + (i * 2)); 777 777 - goto out; 778 778 - } 691 691 + for (i = 0; i < nt->mw_count; i++) { 692 692 + size = nt->mw_vec[i].phys_size; 779 693 780 780 - rc = ntb_write_remote_spad(ndev, MW0_SZ_LOW + (i * 2), 781 781 - (u32) ntb_get_mw_size(ndev, i)); 782 782 - if (rc) { 783 783 - dev_err(&pdev->dev, "Error writing %u to remote spad %d\n", 784 784 - (u32) ntb_get_mw_size(ndev, i), 785 785 - MW0_SZ_LOW + (i * 2)); 786 786 - goto out; 787 787 - } 694 694 + if (max_mw_size && size > max_mw_size) 695 695 + size = max_mw_size; 696 696 + 697 697 + spad = MW0_SZ_HIGH + (i * 2); 698 698 + ntb_peer_spad_write(ndev, spad, (u32)(size >> 32)); 699 699 + 700 700 + spad = MW0_SZ_LOW + (i * 2); 701 701 + ntb_peer_spad_write(ndev, spad, (u32)size); 788 702 } 789 703 790 790 - rc = ntb_write_remote_spad(ndev, NUM_MWS, ntb_max_mw(ndev)); 791 791 - if (rc) { 792 792 - dev_err(&pdev->dev, "Error writing %x to remote spad %d\n", 793 793 - ntb_max_mw(ndev), NUM_MWS); 794 794 - goto out; 795 795 - } 704 704 + ntb_peer_spad_write(ndev, NUM_MWS, nt->mw_count); 796 705 797 797 - rc = ntb_write_remote_spad(ndev, NUM_QPS, nt->max_qps); 798 798 - if (rc) { 799 799 - dev_err(&pdev->dev, "Error writing %x to remote spad %d\n", 800 800 - nt->max_qps, NUM_QPS); 801 801 - goto out; 802 802 - } 706 706 + ntb_peer_spad_write(ndev, NUM_QPS, nt->qp_count); 803 707 804 804 - rc = ntb_write_remote_spad(ndev, VERSION, NTB_TRANSPORT_VERSION); 805 805 - if (rc) { 806 806 - dev_err(&pdev->dev, "Error writing %x to remote spad %d\n", 807 807 - NTB_TRANSPORT_VERSION, VERSION); 808 808 - goto out; 809 809 - } 708 708 + ntb_peer_spad_write(ndev, VERSION, NTB_TRANSPORT_VERSION); 810 709 811 710 /* Query the remote side for its info */ 812 812 - rc = ntb_read_remote_spad(ndev, VERSION, &val); 813 813 - if (rc) { 814 814 - dev_err(&pdev->dev, "Error reading remote spad %d\n", VERSION); 815 815 - goto out; 816 816 - } 817 817 - 711 711 + val = ntb_spad_read(ndev, VERSION); 712 712 + dev_dbg(&pdev->dev, "Remote version = %d\n", val); 818 713 if (val != NTB_TRANSPORT_VERSION) 819 714 goto out; 820 820 - dev_dbg(&pdev->dev, "Remote version = %d\n", val); 821 715 822 822 - rc = ntb_read_remote_spad(ndev, NUM_QPS, &val); 823 823 - if (rc) { 824 824 - dev_err(&pdev->dev, "Error reading remote spad %d\n", NUM_QPS); 825 825 - goto out; 826 826 - } 827 827 - 828 828 - if (val != nt->max_qps) 829 829 - goto out; 716 716 + val = ntb_spad_read(ndev, NUM_QPS); 830 717 dev_dbg(&pdev->dev, "Remote max number of qps = %d\n", val); 831 831 - 832 832 - rc = ntb_read_remote_spad(ndev, NUM_MWS, &val); 833 833 - if (rc) { 834 834 - dev_err(&pdev->dev, "Error reading remote spad %d\n", NUM_MWS); 718 718 + if (val != nt->qp_count) 835 719 goto out; 836 836 - } 837 720 838 838 - if (val != ntb_max_mw(ndev)) 839 839 - goto out; 721 721 + val = ntb_spad_read(ndev, NUM_MWS); 840 722 dev_dbg(&pdev->dev, "Remote number of mws = %d\n", val); 723 723 + if (val != nt->mw_count) 724 724 + goto out; 841 725 842 842 - for (i = 0; i < ntb_max_mw(ndev); i++) { 726 726 + for (i = 0; i < nt->mw_count; i++) { 843 727 u64 val64; 844 728 845 845 - rc = ntb_read_remote_spad(ndev, MW0_SZ_HIGH + (i * 2), &val); 846 846 - if (rc) { 847 847 - dev_err(&pdev->dev, "Error reading remote spad %d\n", 848 848 - MW0_SZ_HIGH + (i * 2)); 849 849 - goto out1; 850 850 - } 729 729 + val = ntb_spad_read(ndev, MW0_SZ_HIGH + (i * 2)); 730 730 + val64 = (u64)val << 32; 851 731 852 852 - val64 = (u64) val << 32; 853 853 - 854 854 - rc = ntb_read_remote_spad(ndev, MW0_SZ_LOW + (i * 2), &val); 855 855 - if (rc) { 856 856 - dev_err(&pdev->dev, "Error reading remote spad %d\n", 857 857 - MW0_SZ_LOW + (i * 2)); 858 858 - goto out1; 859 859 - } 860 860 - 732 732 + val = ntb_spad_read(ndev, MW0_SZ_LOW + (i * 2)); 861 733 val64 |= val; 862 734 863 863 - dev_dbg(&pdev->dev, "Remote MW%d size = %llu\n", i, val64); 735 735 + dev_dbg(&pdev->dev, "Remote MW%d size = %#llx\n", i, val64); 864 736 865 737 rc = ntb_set_mw(nt, i, val64); 866 738 if (rc) 867 739 goto out1; 868 740 } 869 741 870 870 - nt->transport_link = NTB_LINK_UP; 742 742 + nt->link_is_up = true; 871 743 872 872 - for (i = 0; i < nt->max_qps; i++) { 873 873 - struct ntb_transport_qp *qp = &nt->qps[i]; 744 744 + for (i = 0; i < nt->qp_count; i++) { 745 745 + struct ntb_transport_qp *qp = &nt->qp_vec[i]; 874 746 875 747 ntb_transport_setup_qp_mw(nt, i); 876 748 877 877 - if (qp->client_ready == NTB_LINK_UP) 749 749 + if (qp->client_ready) 878 750 schedule_delayed_work(&qp->link_work, 0); 879 751 } 880 752 881 753 return; 882 754 883 755 out1: 884 884 - for (i = 0; i < ntb_max_mw(ndev); i++) 756 756 + for (i = 0; i < nt->mw_count; i++) 885 757 ntb_free_mw(nt, i); 886 758 out: 887 887 - if (ntb_hw_link_status(ndev)) 759 759 + if (ntb_link_is_up(ndev, NULL, NULL) == 1) 888 760 schedule_delayed_work(&nt->link_work, 889 761 msecs_to_jiffies(NTB_LINK_DOWN_TIMEOUT)); 890 762 } ··· 846 814 struct ntb_transport_qp *qp = container_of(work, 847 815 struct ntb_transport_qp, 848 816 link_work.work); 849 849 - struct pci_dev *pdev = ntb_query_pdev(qp->ndev); 850 850 - struct ntb_transport *nt = qp->transport; 851 851 - int rc, val; 817 817 + struct pci_dev *pdev = qp->ndev->pdev; 818 818 + struct ntb_transport_ctx *nt = qp->transport; 819 819 + int val; 852 820 853 853 - WARN_ON(nt->transport_link != NTB_LINK_UP); 821 821 + WARN_ON(!nt->link_is_up); 854 822 855 855 - rc = ntb_read_local_spad(nt->ndev, QP_LINKS, &val); 856 856 - if (rc) { 857 857 - dev_err(&pdev->dev, "Error reading spad %d\n", QP_LINKS); 858 858 - return; 859 859 - } 823 823 + val = ntb_spad_read(nt->ndev, QP_LINKS); 860 824 861 861 - rc = ntb_write_remote_spad(nt->ndev, QP_LINKS, val | 1 << qp->qp_num); 862 862 - if (rc) 863 863 - dev_err(&pdev->dev, "Error writing %x to remote spad %d\n", 864 864 - val | 1 << qp->qp_num, QP_LINKS); 825 825 + ntb_peer_spad_write(nt->ndev, QP_LINKS, val | BIT(qp->qp_num)); 865 826 866 827 /* query remote spad for qp ready bits */ 867 867 - rc = ntb_read_remote_spad(nt->ndev, QP_LINKS, &val); 868 868 - if (rc) 869 869 - dev_err(&pdev->dev, "Error reading remote spad %d\n", QP_LINKS); 870 870 - 871 871 - dev_dbg(&pdev->dev, "Remote QP link status = %x\n", val); 828 828 + ntb_peer_spad_read(nt->ndev, QP_LINKS); 829 829 + dev_dbg_ratelimited(&pdev->dev, "Remote QP link status = %x\n", val); 872 830 873 831 /* See if the remote side is up */ 874 874 - if (1 << qp->qp_num & val) { 875 875 - qp->qp_link = NTB_LINK_UP; 876 876 - 832 832 + if (val & BIT(qp->qp_num)) { 877 833 dev_info(&pdev->dev, "qp %d: Link Up\n", qp->qp_num); 834 834 + qp->link_is_up = true; 835 835 + 878 836 if (qp->event_handler) 879 879 - qp->event_handler(qp->cb_data, NTB_LINK_UP); 880 880 - } else if (nt->transport_link == NTB_LINK_UP) 837 837 + qp->event_handler(qp->cb_data, qp->link_is_up); 838 838 + } else if (nt->link_is_up) 881 839 schedule_delayed_work(&qp->link_work, 882 840 msecs_to_jiffies(NTB_LINK_DOWN_TIMEOUT)); 883 841 } 884 842 885 885 - static int ntb_transport_init_queue(struct ntb_transport *nt, 843 843 + static int ntb_transport_init_queue(struct ntb_transport_ctx *nt, 886 844 unsigned int qp_num) 887 845 { 888 846 struct ntb_transport_qp *qp; 847 847 + struct ntb_transport_mw *mw; 848 848 + phys_addr_t mw_base; 849 849 + resource_size_t mw_size; 889 850 unsigned int num_qps_mw, tx_size; 890 890 - u8 mw_num, mw_max; 851 851 + unsigned int mw_num, mw_count, qp_count; 891 852 u64 qp_offset; 892 853 893 893 - mw_max = ntb_max_mw(nt->ndev); 894 894 - mw_num = QP_TO_MW(nt->ndev, qp_num); 854 854 + mw_count = nt->mw_count; 855 855 + qp_count = nt->qp_count; 895 856 896 896 - qp = &nt->qps[qp_num]; 857 857 + mw_num = QP_TO_MW(nt, qp_num); 858 858 + mw = &nt->mw_vec[mw_num]; 859 859 + 860 860 + qp = &nt->qp_vec[qp_num]; 897 861 qp->qp_num = qp_num; 898 862 qp->transport = nt; 899 863 qp->ndev = nt->ndev; 900 900 - qp->qp_link = NTB_LINK_DOWN; 901 901 - qp->client_ready = NTB_LINK_DOWN; 864 864 + qp->client_ready = false; 902 865 qp->event_handler = NULL; 866 866 + ntb_qp_link_down_reset(qp); 903 867 904 904 - if (nt->max_qps % mw_max && mw_num + 1 < nt->max_qps / mw_max) 905 905 - num_qps_mw = nt->max_qps / mw_max + 1; 868 868 + if (qp_count % mw_count && mw_num + 1 < qp_count / mw_count) 869 869 + num_qps_mw = qp_count / mw_count + 1; 906 870 else 907 907 - num_qps_mw = nt->max_qps / mw_max; 871 871 + num_qps_mw = qp_count / mw_count; 908 872 909 909 - tx_size = (unsigned int) ntb_get_mw_size(qp->ndev, mw_num) / num_qps_mw; 910 910 - qp_offset = qp_num / mw_max * tx_size; 911 911 - qp->tx_mw = ntb_get_mw_vbase(nt->ndev, mw_num) + qp_offset; 873 873 + mw_base = nt->mw_vec[mw_num].phys_addr; 874 874 + mw_size = nt->mw_vec[mw_num].phys_size; 875 875 + 876 876 + tx_size = (unsigned int)mw_size / num_qps_mw; 877 877 + qp_offset = tx_size * qp_num / mw_count; 878 878 + 879 879 + qp->tx_mw = nt->mw_vec[mw_num].vbase + qp_offset; 912 880 if (!qp->tx_mw) 913 881 return -EINVAL; 914 882 915 915 - qp->tx_mw_phys = ntb_get_mw_base(qp->ndev, mw_num) + qp_offset; 883 883 + qp->tx_mw_phys = mw_base + qp_offset; 916 884 if (!qp->tx_mw_phys) 917 885 return -EINVAL; 918 886 ··· 923 891 qp->tx_max_frame = min(transport_mtu, tx_size / 2); 924 892 qp->tx_max_entry = tx_size / qp->tx_max_frame; 925 893 926 926 - if (ntb_query_debugfs(nt->ndev)) { 894 894 + if (nt_debugfs_dir) { 927 895 char debugfs_name[4]; 928 896 929 897 snprintf(debugfs_name, 4, "qp%d", qp_num); 930 898 qp->debugfs_dir = debugfs_create_dir(debugfs_name, 931 931 - ntb_query_debugfs(nt->ndev)); 899 899 + nt_debugfs_dir); 932 900 933 901 qp->debugfs_stats = debugfs_create_file("stats", S_IRUSR, 934 902 qp->debugfs_dir, qp, 935 903 &ntb_qp_debugfs_stats); 904 904 + } else { 905 905 + qp->debugfs_dir = NULL; 906 906 + qp->debugfs_stats = NULL; 936 907 } 937 908 938 909 INIT_DELAYED_WORK(&qp->link_work, ntb_qp_link_work); ··· 949 914 INIT_LIST_HEAD(&qp->rx_free_q); 950 915 INIT_LIST_HEAD(&qp->tx_free_q); 951 916 917 917 + tasklet_init(&qp->rxc_db_work, ntb_transport_rxc_db, 918 918 + (unsigned long)qp); 919 919 + 952 920 return 0; 953 921 } 954 922 955 955 - int ntb_transport_init(struct pci_dev *pdev) 923 923 + static int ntb_transport_probe(struct ntb_client *self, struct ntb_dev *ndev) 956 924 { 957 957 - struct ntb_transport *nt; 925 925 + struct ntb_transport_ctx *nt; 926 926 + struct ntb_transport_mw *mw; 927 927 + unsigned int mw_count, qp_count; 928 928 + u64 qp_bitmap; 929 929 + int node; 958 930 int rc, i; 959 931 960 960 - nt = kzalloc(sizeof(struct ntb_transport), GFP_KERNEL); 932 932 + if (ntb_db_is_unsafe(ndev)) 933 933 + dev_dbg(&ndev->dev, 934 934 + "doorbell is unsafe, proceed anyway...\n"); 935 935 + if (ntb_spad_is_unsafe(ndev)) 936 936 + dev_dbg(&ndev->dev, 937 937 + "scratchpad is unsafe, proceed anyway...\n"); 938 938 + 939 939 + node = dev_to_node(&ndev->dev); 940 940 + 941 941 + nt = kzalloc_node(sizeof(*nt), GFP_KERNEL, node); 961 942 if (!nt) 962 943 return -ENOMEM; 963 944 964 964 - nt->ndev = ntb_register_transport(pdev, nt); 965 965 - if (!nt->ndev) { 966 966 - rc = -EIO; 945 945 + nt->ndev = ndev; 946 946 + 947 947 + mw_count = ntb_mw_count(ndev); 948 948 + 949 949 + nt->mw_count = mw_count; 950 950 + 951 951 + nt->mw_vec = kzalloc_node(mw_count * sizeof(*nt->mw_vec), 952 952 + GFP_KERNEL, node); 953 953 + if (!nt->mw_vec) { 954 954 + rc = -ENOMEM; 967 955 goto err; 968 956 } 969 957 970 970 - nt->mw = kcalloc(ntb_max_mw(nt->ndev), sizeof(struct ntb_transport_mw), 971 971 - GFP_KERNEL); 972 972 - if (!nt->mw) { 973 973 - rc = -ENOMEM; 974 974 - goto err1; 958 958 + for (i = 0; i < mw_count; i++) { 959 959 + mw = &nt->mw_vec[i]; 960 960 + 961 961 + rc = ntb_mw_get_range(ndev, i, &mw->phys_addr, &mw->phys_size, 962 962 + &mw->xlat_align, &mw->xlat_align_size); 963 963 + if (rc) 964 964 + goto err1; 965 965 + 966 966 + mw->vbase = ioremap_wc(mw->phys_addr, mw->phys_size); 967 967 + if (!mw->vbase) { 968 968 + rc = -ENOMEM; 969 969 + goto err1; 970 970 + } 971 971 + 972 972 + mw->buff_size = 0; 973 973 + mw->xlat_size = 0; 974 974 + mw->virt_addr = NULL; 975 975 + mw->dma_addr = 0; 975 976 } 976 977 977 977 - if (max_num_clients) 978 978 - nt->max_qps = min(ntb_max_cbs(nt->ndev), max_num_clients); 979 979 - else 980 980 - nt->max_qps = min(ntb_max_cbs(nt->ndev), ntb_max_mw(nt->ndev)); 978 978 + qp_bitmap = ntb_db_valid_mask(ndev); 981 979 982 982 - nt->qps = kcalloc(nt->max_qps, sizeof(struct ntb_transport_qp), 983 983 - GFP_KERNEL); 984 984 - if (!nt->qps) { 980 980 + qp_count = ilog2(qp_bitmap); 981 981 + if (max_num_clients && max_num_clients < qp_count) 982 982 + qp_count = max_num_clients; 983 983 + else if (mw_count < qp_count) 984 984 + qp_count = mw_count; 985 985 + 986 986 + qp_bitmap &= BIT_ULL(qp_count) - 1; 987 987 + 988 988 + nt->qp_count = qp_count; 989 989 + nt->qp_bitmap = qp_bitmap; 990 990 + nt->qp_bitmap_free = qp_bitmap; 991 991 + 992 992 + nt->qp_vec = kzalloc_node(qp_count * sizeof(*nt->qp_vec), 993 993 + GFP_KERNEL, node); 994 994 + if (!nt->qp_vec) { 985 995 rc = -ENOMEM; 986 996 goto err2; 987 997 } 988 998 989 989 - nt->qp_bitmap = ((u64) 1 << nt->max_qps) - 1; 990 990 - 991 991 - for (i = 0; i < nt->max_qps; i++) { 999 999 + for (i = 0; i < qp_count; i++) { 992 1000 rc = ntb_transport_init_queue(nt, i); 993 1001 if (rc) 994 1002 goto err3; ··· 1040 962 INIT_DELAYED_WORK(&nt->link_work, ntb_transport_link_work); 1041 963 INIT_WORK(&nt->link_cleanup, ntb_transport_link_cleanup_work); 1042 964 1043 1043 - rc = ntb_register_event_callback(nt->ndev, 1044 1044 - ntb_transport_event_callback); 965 965 + rc = ntb_set_ctx(ndev, nt, &ntb_transport_ops); 1045 966 if (rc) 1046 967 goto err3; 1047 968 ··· 1049 972 if (rc) 1050 973 goto err4; 1051 974 1052 1052 - if (ntb_hw_link_status(nt->ndev)) 1053 1053 - schedule_delayed_work(&nt->link_work, 0); 975 975 + nt->link_is_up = false; 976 976 + ntb_link_enable(ndev, NTB_SPEED_AUTO, NTB_WIDTH_AUTO); 977 977 + ntb_link_event(ndev); 1054 978 1055 979 return 0; 1056 980 1057 981 err4: 1058 1058 - ntb_unregister_event_callback(nt->ndev); 982 982 + ntb_clear_ctx(ndev); 1059 983 err3: 1060 1060 - kfree(nt->qps); 984 984 + kfree(nt->qp_vec); 1061 985 err2: 1062 1062 - kfree(nt->mw); 986 986 + kfree(nt->mw_vec); 1063 987 err1: 1064 1064 - ntb_unregister_transport(nt->ndev); 988 988 + while (i--) { 989 989 + mw = &nt->mw_vec[i]; 990 990 + iounmap(mw->vbase); 991 991 + } 1065 992 err: 1066 993 kfree(nt); 1067 994 return rc; 1068 995 } 1069 996 1070 1070 - void ntb_transport_free(void *transport) 997 997 + static void ntb_transport_free(struct ntb_client *self, struct ntb_dev *ndev) 1071 998 { 1072 1072 - struct ntb_transport *nt = transport; 1073 1073 - struct ntb_device *ndev = nt->ndev; 999 999 + struct ntb_transport_ctx *nt = ndev->ctx; 1000 1000 + struct ntb_transport_qp *qp; 1001 1001 + u64 qp_bitmap_alloc; 1074 1002 int i; 1075 1003 1076 1004 ntb_transport_link_cleanup(nt); 1005 1005 + cancel_work_sync(&nt->link_cleanup); 1006 1006 + cancel_delayed_work_sync(&nt->link_work); 1007 1007 + 1008 1008 + qp_bitmap_alloc = nt->qp_bitmap & ~nt->qp_bitmap_free; 1077 1009 1078 1010 /* verify that all the qp's are freed */ 1079 1079 - for (i = 0; i < nt->max_qps; i++) { 1080 1080 - if (!test_bit(i, &nt->qp_bitmap)) 1081 1081 - ntb_transport_free_queue(&nt->qps[i]); 1082 1082 - debugfs_remove_recursive(nt->qps[i].debugfs_dir); 1011 1011 + for (i = 0; i < nt->qp_count; i++) { 1012 1012 + qp = &nt->qp_vec[i]; 1013 1013 + if (qp_bitmap_alloc & BIT_ULL(i)) 1014 1014 + ntb_transport_free_queue(qp); 1015 1015 + debugfs_remove_recursive(qp->debugfs_dir); 1083 1016 } 1017 1017 + 1018 1018 + ntb_link_disable(ndev); 1019 1019 + ntb_clear_ctx(ndev); 1084 1020 1085 1021 ntb_bus_remove(nt); 1086 1022 1087 1087 - cancel_delayed_work_sync(&nt->link_work); 1088 1088 - 1089 1089 - ntb_unregister_event_callback(ndev); 1090 1090 - 1091 1091 - for (i = 0; i < ntb_max_mw(ndev); i++) 1023 1023 + for (i = nt->mw_count; i--; ) { 1092 1024 ntb_free_mw(nt, i); 1025 1025 + iounmap(nt->mw_vec[i].vbase); 1026 1026 + } 1093 1027 1094 1094 - kfree(nt->qps); 1095 1095 - kfree(nt->mw); 1096 1096 - ntb_unregister_transport(ndev); 1028 1028 + kfree(nt->qp_vec); 1029 1029 + kfree(nt->mw_vec); 1097 1030 kfree(nt); 1098 1031 } 1099 1032 ··· 1115 1028 unsigned int len = entry->len; 1116 1029 struct ntb_payload_header *hdr = entry->rx_hdr; 1117 1030 1118 1118 - /* Ensure that the data is fully copied out before clearing the flag */ 1119 1119 - wmb(); 1120 1031 hdr->flags = 0; 1121 1032 1122 1033 iowrite32(entry->index, &qp->rx_info->entry); 1123 1034 1124 1035 ntb_list_add(&qp->ntb_rx_free_q_lock, &entry->entry, &qp->rx_free_q); 1125 1036 1126 1126 - if (qp->rx_handler && qp->client_ready == NTB_LINK_UP) 1037 1037 + if (qp->rx_handler && qp->client_ready) 1127 1038 qp->rx_handler(qp, qp->cb_data, cb_data, len); 1128 1039 } 1129 1040 ··· 1131 1046 size_t len = entry->len; 1132 1047 1133 1048 memcpy(buf, offset, len); 1049 1049 + 1050 1050 + /* Ensure that the data is fully copied out before clearing the flag */ 1051 1051 + wmb(); 1134 1052 1135 1053 ntb_rx_copy_callback(entry); 1136 1054 } ··· 1159 1071 goto err_wait; 1160 1072 1161 1073 device = chan->device; 1162 1162 - pay_off = (size_t) offset & ~PAGE_MASK; 1163 1163 - buff_off = (size_t) buf & ~PAGE_MASK; 1074 1074 + pay_off = (size_t)offset & ~PAGE_MASK; 1075 1075 + buff_off = (size_t)buf & ~PAGE_MASK; 1164 1076 1165 1077 if (!is_dma_copy_aligned(device, pay_off, buff_off, len)) 1166 1078 goto err_wait; ··· 1226 1138 struct ntb_payload_header *hdr; 1227 1139 struct ntb_queue_entry *entry; 1228 1140 void *offset; 1141 1141 + int rc; 1229 1142 1230 1143 offset = qp->rx_buff + qp->rx_max_frame * qp->rx_index; 1231 1144 hdr = offset + qp->rx_max_frame - sizeof(struct ntb_payload_header); 1232 1145 1233 1233 - entry = ntb_list_rm(&qp->ntb_rx_pend_q_lock, &qp->rx_pend_q); 1234 1234 - if (!entry) { 1235 1235 - dev_dbg(&ntb_query_pdev(qp->ndev)->dev, 1236 1236 - "no buffer - HDR ver %u, len %d, flags %x\n", 1237 1237 - hdr->ver, hdr->len, hdr->flags); 1238 1238 - qp->rx_err_no_buf++; 1239 1239 - return -ENOMEM; 1240 1240 - } 1146 1146 + dev_dbg(&qp->ndev->pdev->dev, "qp %d: RX ver %u len %d flags %x\n", 1147 1147 + qp->qp_num, hdr->ver, hdr->len, hdr->flags); 1241 1148 1242 1149 if (!(hdr->flags & DESC_DONE_FLAG)) { 1243 1243 - ntb_list_add(&qp->ntb_rx_pend_q_lock, &entry->entry, 1244 1244 - &qp->rx_pend_q); 1150 1150 + dev_dbg(&qp->ndev->pdev->dev, "done flag not set\n"); 1245 1151 qp->rx_ring_empty++; 1246 1152 return -EAGAIN; 1247 1153 } 1248 1154 1249 1249 - if (hdr->ver != (u32) qp->rx_pkts) { 1250 1250 - dev_dbg(&ntb_query_pdev(qp->ndev)->dev, 1251 1251 - "qp %d: version mismatch, expected %llu - got %u\n", 1252 1252 - qp->qp_num, qp->rx_pkts, hdr->ver); 1253 1253 - ntb_list_add(&qp->ntb_rx_pend_q_lock, &entry->entry, 1254 1254 - &qp->rx_pend_q); 1155 1155 + if (hdr->flags & LINK_DOWN_FLAG) { 1156 1156 + dev_dbg(&qp->ndev->pdev->dev, "link down flag set\n"); 1157 1157 + ntb_qp_link_down(qp); 1158 1158 + hdr->flags = 0; 1159 1159 + return -EAGAIN; 1160 1160 + } 1161 1161 + 1162 1162 + if (hdr->ver != (u32)qp->rx_pkts) { 1163 1163 + dev_dbg(&qp->ndev->pdev->dev, 1164 1164 + "version mismatch, expected %llu - got %u\n", 1165 1165 + qp->rx_pkts, hdr->ver); 1255 1166 qp->rx_err_ver++; 1256 1167 return -EIO; 1257 1168 } 1258 1169 1259 1259 - if (hdr->flags & LINK_DOWN_FLAG) { 1260 1260 - ntb_qp_link_down(qp); 1170 1170 + entry = ntb_list_rm(&qp->ntb_rx_pend_q_lock, &qp->rx_pend_q); 1171 1171 + if (!entry) { 1172 1172 + dev_dbg(&qp->ndev->pdev->dev, "no receive buffer\n"); 1173 1173 + qp->rx_err_no_buf++; 1261 1174 1175 1175 + rc = -ENOMEM; 1262 1176 goto err; 1263 1177 } 1264 1178 1265 1265 - dev_dbg(&ntb_query_pdev(qp->ndev)->dev, 1266 1266 - "rx offset %u, ver %u - %d payload received, buf size %d\n", 1179 1179 + if (hdr->len > entry->len) { 1180 1180 + dev_dbg(&qp->ndev->pdev->dev, 1181 1181 + "receive buffer overflow! Wanted %d got %d\n", 1182 1182 + hdr->len, entry->len); 1183 1183 + qp->rx_err_oflow++; 1184 1184 + 1185 1185 + rc = -EIO; 1186 1186 + goto err; 1187 1187 + } 1188 1188 + 1189 1189 + dev_dbg(&qp->ndev->pdev->dev, 1190 1190 + "RX OK index %u ver %u size %d into buf size %d\n", 1267 1191 qp->rx_index, hdr->ver, hdr->len, entry->len); 1268 1192 1269 1193 qp->rx_bytes += hdr->len; 1270 1194 qp->rx_pkts++; 1271 1271 - 1272 1272 - if (hdr->len > entry->len) { 1273 1273 - qp->rx_err_oflow++; 1274 1274 - dev_dbg(&ntb_query_pdev(qp->ndev)->dev, 1275 1275 - "RX overflow! Wanted %d got %d\n", 1276 1276 - hdr->len, entry->len); 1277 1277 - 1278 1278 - goto err; 1279 1279 - } 1280 1195 1281 1196 entry->index = qp->rx_index; 1282 1197 entry->rx_hdr = hdr; 1283 1198 1284 1199 ntb_async_rx(entry, offset, hdr->len); 1285 1200 1286 1286 - out: 1287 1201 qp->rx_index++; 1288 1202 qp->rx_index %= qp->rx_max_entry; 1289 1203 1290 1204 return 0; 1291 1205 1292 1206 err: 1293 1293 - ntb_list_add(&qp->ntb_rx_pend_q_lock, &entry->entry, &qp->rx_pend_q); 1294 1294 - /* Ensure that the data is fully copied out before clearing the flag */ 1295 1295 - wmb(); 1207 1207 + /* FIXME: if this syncrhonous update of the rx_index gets ahead of 1208 1208 + * asyncrhonous ntb_rx_copy_callback of previous entry, there are three 1209 1209 + * scenarios: 1210 1210 + * 1211 1211 + * 1) The peer might miss this update, but observe the update 1212 1212 + * from the memcpy completion callback. In this case, the buffer will 1213 1213 + * not be freed on the peer to be reused for a different packet. The 1214 1214 + * successful rx of a later packet would clear the condition, but the 1215 1215 + * condition could persist if several rx fail in a row. 1216 1216 + * 1217 1217 + * 2) The peer may observe this update before the asyncrhonous copy of 1218 1218 + * prior packets is completed. The peer may overwrite the buffers of 1219 1219 + * the prior packets before they are copied. 1220 1220 + * 1221 1221 + * 3) Both: the peer may observe the update, and then observe the index 1222 1222 + * decrement by the asynchronous completion callback. Who knows what 1223 1223 + * badness that will cause. 1224 1224 + */ 1296 1225 hdr->flags = 0; 1297 1226 iowrite32(qp->rx_index, &qp->rx_info->entry); 1298 1227 1299 1299 - goto out; 1228 1228 + return rc; 1300 1229 } 1301 1230 1302 1302 - static int ntb_transport_rxc_db(void *data, int db_num) 1231 1231 + static void ntb_transport_rxc_db(unsigned long data) 1303 1232 { 1304 1304 - struct ntb_transport_qp *qp = data; 1233 1233 + struct ntb_transport_qp *qp = (void *)data; 1305 1234 int rc, i; 1306 1235 1307 1307 - dev_dbg(&ntb_query_pdev(qp->ndev)->dev, "%s: doorbell %d received\n", 1308 1308 - __func__, db_num); 1236 1236 + dev_dbg(&qp->ndev->pdev->dev, "%s: doorbell %d received\n", 1237 1237 + __func__, qp->qp_num); 1309 1238 1310 1239 /* Limit the number of packets processed in a single interrupt to 1311 1240 * provide fairness to others ··· 1336 1231 if (qp->dma_chan) 1337 1232 dma_async_issue_pending(qp->dma_chan); 1338 1233 1339 1339 - return i; 1234 1234 + if (i == qp->rx_max_entry) { 1235 1235 + /* there is more work to do */ 1236 1236 + tasklet_schedule(&qp->rxc_db_work); 1237 1237 + } else if (ntb_db_read(qp->ndev) & BIT_ULL(qp->qp_num)) { 1238 1238 + /* the doorbell bit is set: clear it */ 1239 1239 + ntb_db_clear(qp->ndev, BIT_ULL(qp->qp_num)); 1240 1240 + /* ntb_db_read ensures ntb_db_clear write is committed */ 1241 1241 + ntb_db_read(qp->ndev); 1242 1242 + 1243 1243 + /* an interrupt may have arrived between finishing 1244 1244 + * ntb_process_rxc and clearing the doorbell bit: 1245 1245 + * there might be some more work to do. 1246 1246 + */ 1247 1247 + tasklet_schedule(&qp->rxc_db_work); 1248 1248 + } 1340 1249 } 1341 1250 1342 1251 static void ntb_tx_copy_callback(void *data) ··· 1359 1240 struct ntb_transport_qp *qp = entry->qp; 1360 1241 struct ntb_payload_header __iomem *hdr = entry->tx_hdr; 1361 1242 1362 1362 - /* Ensure that the data is fully copied out before setting the flags */ 1363 1363 - wmb(); 1364 1243 iowrite32(entry->flags | DESC_DONE_FLAG, &hdr->flags); 1365 1244 1366 1366 - ntb_ring_doorbell(qp->ndev, qp->qp_num); 1245 1245 + ntb_peer_db_set(qp->ndev, BIT_ULL(qp->qp_num)); 1367 1246 1368 1247 /* The entry length can only be zero if the packet is intended to be a 1369 1248 * "link down" or similar. Since no payload is being sent in these ··· 1380 1263 1381 1264 static void ntb_memcpy_tx(struct ntb_queue_entry *entry, void __iomem *offset) 1382 1265 { 1266 1266 + #ifdef ARCH_HAS_NOCACHE_UACCESS 1267 1267 + /* 1268 1268 + * Using non-temporal mov to improve performance on non-cached 1269 1269 + * writes, even though we aren't actually copying from user space. 1270 1270 + */ 1271 1271 + __copy_from_user_inatomic_nocache(offset, entry->buf, entry->len); 1272 1272 + #else 1383 1273 memcpy_toio(offset, entry->buf, entry->len); 1274 1274 + #endif 1275 1275 + 1276 1276 + /* Ensure that the data is fully copied out before setting the flags */ 1277 1277 + wmb(); 1384 1278 1385 1279 ntb_tx_copy_callback(entry); 1386 1280 } ··· 1416 1288 entry->tx_hdr = hdr; 1417 1289 1418 1290 iowrite32(entry->len, &hdr->len); 1419 1419 - iowrite32((u32) qp->tx_pkts, &hdr->ver); 1291 1291 + iowrite32((u32)qp->tx_pkts, &hdr->ver); 1420 1292 1421 1293 if (!chan) 1422 1294 goto err; ··· 1426 1298 1427 1299 device = chan->device; 1428 1300 dest = qp->tx_mw_phys + qp->tx_max_frame * qp->tx_index; 1429 1429 - buff_off = (size_t) buf & ~PAGE_MASK; 1430 1430 - dest_off = (size_t) dest & ~PAGE_MASK; 1301 1301 + buff_off = (size_t)buf & ~PAGE_MASK; 1302 1302 + dest_off = (size_t)dest & ~PAGE_MASK; 1431 1303 1432 1304 if (!is_dma_copy_aligned(device, buff_off, dest_off, len)) 1433 1305 goto err; ··· 1475 1347 static int ntb_process_tx(struct ntb_transport_qp *qp, 1476 1348 struct ntb_queue_entry *entry) 1477 1349 { 1478 1478 - dev_dbg(&ntb_query_pdev(qp->ndev)->dev, "%lld - tx %u, entry len %d flags %x buff %p\n", 1479 1479 - qp->tx_pkts, qp->tx_index, entry->len, entry->flags, 1480 1480 - entry->buf); 1481 1350 if (qp->tx_index == qp->remote_rx_info->entry) { 1482 1351 qp->tx_ring_full++; 1483 1352 return -EAGAIN; ··· 1501 1376 1502 1377 static void ntb_send_link_down(struct ntb_transport_qp *qp) 1503 1378 { 1504 1504 - struct pci_dev *pdev = ntb_query_pdev(qp->ndev); 1379 1379 + struct pci_dev *pdev = qp->ndev->pdev; 1505 1380 struct ntb_queue_entry *entry; 1506 1381 int i, rc; 1507 1382 1508 1508 - if (qp->qp_link == NTB_LINK_DOWN) 1383 1383 + if (!qp->link_is_up) 1509 1384 return; 1510 1385 1511 1511 - qp->qp_link = NTB_LINK_DOWN; 1512 1512 - dev_info(&pdev->dev, "qp %d: Link Down\n", qp->qp_num); 1386 1386 + dev_info(&pdev->dev, "qp %d: Send Link Down\n", qp->qp_num); 1513 1387 1514 1388 for (i = 0; i < NTB_LINK_DOWN_TIMEOUT; i++) { 1515 1389 entry = ntb_list_rm(&qp->ntb_tx_free_q_lock, &qp->tx_free_q); ··· 1529 1405 if (rc) 1530 1406 dev_err(&pdev->dev, "ntb: QP%d unable to send linkdown msg\n", 1531 1407 qp->qp_num); 1408 1408 + 1409 1409 + ntb_qp_link_down_reset(qp); 1410 1410 + } 1411 1411 + 1412 1412 + static bool ntb_dma_filter_fn(struct dma_chan *chan, void *node) 1413 1413 + { 1414 1414 + return dev_to_node(&chan->dev->device) == (int)(unsigned long)node; 1532 1415 } 1533 1416 1534 1417 /** ··· 1553 1422 * RETURNS: pointer to newly created ntb_queue, NULL on error. 1554 1423 */ 1555 1424 struct ntb_transport_qp * 1556 1556 - ntb_transport_create_queue(void *data, struct pci_dev *pdev, 1425 1425 + ntb_transport_create_queue(void *data, struct device *client_dev, 1557 1426 const struct ntb_queue_handlers *handlers) 1558 1427 { 1428 1428 + struct ntb_dev *ndev; 1429 1429 + struct pci_dev *pdev; 1430 1430 + struct ntb_transport_ctx *nt; 1559 1431 struct ntb_queue_entry *entry; 1560 1432 struct ntb_transport_qp *qp; 1561 1561 - struct ntb_transport *nt; 1433 1433 + u64 qp_bit; 1562 1434 unsigned int free_queue; 1563 1563 - int rc, i; 1435 1435 + dma_cap_mask_t dma_mask; 1436 1436 + int node; 1437 1437 + int i; 1564 1438 1565 1565 - nt = ntb_find_transport(pdev); 1566 1566 - if (!nt) 1567 1567 - goto err; 1439 1439 + ndev = dev_ntb(client_dev->parent); 1440 1440 + pdev = ndev->pdev; 1441 1441 + nt = ndev->ctx; 1442 1442 + 1443 1443 + node = dev_to_node(&ndev->dev); 1568 1444 1569 1445 free_queue = ffs(nt->qp_bitmap); 1570 1446 if (!free_queue) ··· 1580 1442 /* decrement free_queue to make it zero based */ 1581 1443 free_queue--; 1582 1444 1583 1583 - clear_bit(free_queue, &nt->qp_bitmap); 1445 1445 + qp = &nt->qp_vec[free_queue]; 1446 1446 + qp_bit = BIT_ULL(qp->qp_num); 1584 1447 1585 1585 - qp = &nt->qps[free_queue]; 1448 1448 + nt->qp_bitmap_free &= ~qp_bit; 1449 1449 + 1586 1450 qp->cb_data = data; 1587 1451 qp->rx_handler = handlers->rx_handler; 1588 1452 qp->tx_handler = handlers->tx_handler; 1589 1453 qp->event_handler = handlers->event_handler; 1590 1454 1591 1591 - dmaengine_get(); 1592 1592 - qp->dma_chan = dma_find_channel(DMA_MEMCPY); 1593 1593 - if (!qp->dma_chan) { 1594 1594 - dmaengine_put(); 1595 1595 - dev_info(&pdev->dev, "Unable to allocate DMA channel, using CPU instead\n"); 1455 1455 + dma_cap_zero(dma_mask); 1456 1456 + dma_cap_set(DMA_MEMCPY, dma_mask); 1457 1457 + 1458 1458 + if (use_dma) { 1459 1459 + qp->dma_chan = dma_request_channel(dma_mask, ntb_dma_filter_fn, 1460 1460 + (void *)(unsigned long)node); 1461 1461 + if (!qp->dma_chan) 1462 1462 + dev_info(&pdev->dev, "Unable to allocate DMA channel\n"); 1463 1463 + } else { 1464 1464 + qp->dma_chan = NULL; 1596 1465 } 1466 1466 + dev_dbg(&pdev->dev, "Using %s memcpy\n", qp->dma_chan ? "DMA" : "CPU"); 1597 1467 1598 1468 for (i = 0; i < NTB_QP_DEF_NUM_ENTRIES; i++) { 1599 1599 - entry = kzalloc(sizeof(struct ntb_queue_entry), GFP_ATOMIC); 1469 1469 + entry = kzalloc_node(sizeof(*entry), GFP_ATOMIC, node); 1600 1470 if (!entry) 1601 1471 goto err1; 1602 1472 ··· 1614 1468 } 1615 1469 1616 1470 for (i = 0; i < NTB_QP_DEF_NUM_ENTRIES; i++) { 1617 1617 - entry = kzalloc(sizeof(struct ntb_queue_entry), GFP_ATOMIC); 1471 1471 + entry = kzalloc_node(sizeof(*entry), GFP_ATOMIC, node); 1618 1472 if (!entry) 1619 1473 goto err2; 1620 1474 ··· 1623 1477 &qp->tx_free_q); 1624 1478 } 1625 1479 1626 1626 - rc = ntb_register_db_callback(qp->ndev, free_queue, qp, 1627 1627 - ntb_transport_rxc_db); 1628 1628 - if (rc) 1629 1629 - goto err2; 1480 1480 + ntb_db_clear(qp->ndev, qp_bit); 1481 1481 + ntb_db_clear_mask(qp->ndev, qp_bit); 1630 1482 1631 1483 dev_info(&pdev->dev, "NTB Transport QP %d created\n", qp->qp_num); 1632 1484 ··· 1637 1493 while ((entry = ntb_list_rm(&qp->ntb_rx_free_q_lock, &qp->rx_free_q))) 1638 1494 kfree(entry); 1639 1495 if (qp->dma_chan) 1640 1640 - dmaengine_put(); 1641 1641 - set_bit(free_queue, &nt->qp_bitmap); 1496 1496 + dma_release_channel(qp->dma_chan); 1497 1497 + nt->qp_bitmap_free |= qp_bit; 1642 1498 err: 1643 1499 return NULL; 1644 1500 } ··· 1652 1508 */ 1653 1509 void ntb_transport_free_queue(struct ntb_transport_qp *qp) 1654 1510 { 1511 1511 + struct ntb_transport_ctx *nt = qp->transport; 1655 1512 struct pci_dev *pdev; 1656 1513 struct ntb_queue_entry *entry; 1514 1514 + u64 qp_bit; 1657 1515 1658 1516 if (!qp) 1659 1517 return; 1660 1518 1661 1661 - pdev = ntb_query_pdev(qp->ndev); 1519 1519 + pdev = qp->ndev->pdev; 1662 1520 1663 1521 if (qp->dma_chan) { 1664 1522 struct dma_chan *chan = qp->dma_chan; ··· 1674 1528 */ 1675 1529 dma_sync_wait(chan, qp->last_cookie); 1676 1530 dmaengine_terminate_all(chan); 1677 1677 - dmaengine_put(); 1531 1531 + dma_release_channel(chan); 1678 1532 } 1679 1533 1680 1680 - ntb_unregister_db_callback(qp->ndev, qp->qp_num); 1534 1534 + qp_bit = BIT_ULL(qp->qp_num); 1535 1535 + 1536 1536 + ntb_db_set_mask(qp->ndev, qp_bit); 1537 1537 + tasklet_disable(&qp->rxc_db_work); 1681 1538 1682 1539 cancel_delayed_work_sync(&qp->link_work); 1540 1540 + 1541 1541 + qp->cb_data = NULL; 1542 1542 + qp->rx_handler = NULL; 1543 1543 + qp->tx_handler = NULL; 1544 1544 + qp->event_handler = NULL; 1683 1545 1684 1546 while ((entry = ntb_list_rm(&qp->ntb_rx_free_q_lock, &qp->rx_free_q))) 1685 1547 kfree(entry); ··· 1700 1546 while ((entry = ntb_list_rm(&qp->ntb_tx_free_q_lock, &qp->tx_free_q))) 1701 1547 kfree(entry); 1702 1548 1703 1703 - set_bit(qp->qp_num, &qp->transport->qp_bitmap); 1549 1549 + nt->qp_bitmap_free |= qp_bit; 1704 1550 1705 1551 dev_info(&pdev->dev, "NTB Transport QP %d freed\n", qp->qp_num); 1706 1552 } ··· 1721 1567 struct ntb_queue_entry *entry; 1722 1568 void *buf; 1723 1569 1724 1724 - if (!qp || qp->client_ready == NTB_LINK_UP) 1570 1570 + if (!qp || qp->client_ready) 1725 1571 return NULL; 1726 1572 1727 1573 entry = ntb_list_rm(&qp->ntb_rx_pend_q_lock, &qp->rx_pend_q); ··· 1790 1636 struct ntb_queue_entry *entry; 1791 1637 int rc; 1792 1638 1793 1793 - if (!qp || qp->qp_link != NTB_LINK_UP || !len) 1639 1639 + if (!qp || !qp->link_is_up || !len) 1794 1640 return -EINVAL; 1795 1641 1796 1642 entry = ntb_list_rm(&qp->ntb_tx_free_q_lock, &qp->tx_free_q); ··· 1824 1670 if (!qp) 1825 1671 return; 1826 1672 1827 1827 - qp->client_ready = NTB_LINK_UP; 1673 1673 + qp->client_ready = true; 1828 1674 1829 1829 - if (qp->transport->transport_link == NTB_LINK_UP) 1675 1675 + if (qp->transport->link_is_up) 1830 1676 schedule_delayed_work(&qp->link_work, 0); 1831 1677 } 1832 1678 EXPORT_SYMBOL_GPL(ntb_transport_link_up); ··· 1842 1688 void ntb_transport_link_down(struct ntb_transport_qp *qp) 1843 1689 { 1844 1690 struct pci_dev *pdev; 1845 1845 - int rc, val; 1691 1691 + int val; 1846 1692 1847 1693 if (!qp) 1848 1694 return; 1849 1695 1850 1850 - pdev = ntb_query_pdev(qp->ndev); 1851 1851 - qp->client_ready = NTB_LINK_DOWN; 1696 1696 + pdev = qp->ndev->pdev; 1697 1697 + qp->client_ready = false; 1852 1698 1853 1853 - rc = ntb_read_local_spad(qp->ndev, QP_LINKS, &val); 1854 1854 - if (rc) { 1855 1855 - dev_err(&pdev->dev, "Error reading spad %d\n", QP_LINKS); 1856 1856 - return; 1857 1857 - } 1699 1699 + val = ntb_spad_read(qp->ndev, QP_LINKS); 1858 1700 1859 1859 - rc = ntb_write_remote_spad(qp->ndev, QP_LINKS, 1860 1860 - val & ~(1 << qp->qp_num)); 1861 1861 - if (rc) 1862 1862 - dev_err(&pdev->dev, "Error writing %x to remote spad %d\n", 1863 1863 - val & ~(1 << qp->qp_num), QP_LINKS); 1701 1701 + ntb_peer_spad_write(qp->ndev, QP_LINKS, 1702 1702 + val & ~BIT(qp->qp_num)); 1864 1703 1865 1865 - if (qp->qp_link == NTB_LINK_UP) 1704 1704 + if (qp->link_is_up) 1866 1705 ntb_send_link_down(qp); 1867 1706 else 1868 1707 cancel_delayed_work_sync(&qp->link_work); ··· 1875 1728 if (!qp) 1876 1729 return false; 1877 1730 1878 1878 - return qp->qp_link == NTB_LINK_UP; 1731 1731 + return qp->link_is_up; 1879 1732 } 1880 1733 EXPORT_SYMBOL_GPL(ntb_transport_link_query); 1881 1734 ··· 1921 1774 return max; 1922 1775 } 1923 1776 EXPORT_SYMBOL_GPL(ntb_transport_max_size); 1777 1777 + 1778 1778 + static void ntb_transport_doorbell_callback(void *data, int vector) 1779 1779 + { 1780 1780 + struct ntb_transport_ctx *nt = data; 1781 1781 + struct ntb_transport_qp *qp; 1782 1782 + u64 db_bits; 1783 1783 + unsigned int qp_num; 1784 1784 + 1785 1785 + db_bits = (nt->qp_bitmap & ~nt->qp_bitmap_free & 1786 1786 + ntb_db_vector_mask(nt->ndev, vector)); 1787 1787 + 1788 1788 + while (db_bits) { 1789 1789 + qp_num = __ffs(db_bits); 1790 1790 + qp = &nt->qp_vec[qp_num]; 1791 1791 + 1792 1792 + tasklet_schedule(&qp->rxc_db_work); 1793 1793 + 1794 1794 + db_bits &= ~BIT_ULL(qp_num); 1795 1795 + } 1796 1796 + } 1797 1797 + 1798 1798 + static const struct ntb_ctx_ops ntb_transport_ops = { 1799 1799 + .link_event = ntb_transport_event_callback, 1800 1800 + .db_event = ntb_transport_doorbell_callback, 1801 1801 + }; 1802 1802 + 1803 1803 + static struct ntb_client ntb_transport_client = { 1804 1804 + .ops = { 1805 1805 + .probe = ntb_transport_probe, 1806 1806 + .remove = ntb_transport_free, 1807 1807 + }, 1808 1808 + }; 1809 1809 + 1810 1810 + static int __init ntb_transport_init(void) 1811 1811 + { 1812 1812 + int rc; 1813 1813 + 1814 1814 + pr_info("%s, version %s\n", NTB_TRANSPORT_DESC, NTB_TRANSPORT_VER); 1815 1815 + 1816 1816 + if (debugfs_initialized()) 1817 1817 + nt_debugfs_dir = debugfs_create_dir(KBUILD_MODNAME, NULL); 1818 1818 + 1819 1819 + rc = bus_register(&ntb_transport_bus); 1820 1820 + if (rc) 1821 1821 + goto err_bus; 1822 1822 + 1823 1823 + rc = ntb_register_client(&ntb_transport_client); 1824 1824 + if (rc) 1825 1825 + goto err_client; 1826 1826 + 1827 1827 + return 0; 1828 1828 + 1829 1829 + err_client: 1830 1830 + bus_unregister(&ntb_transport_bus); 1831 1831 + err_bus: 1832 1832 + debugfs_remove_recursive(nt_debugfs_dir); 1833 1833 + return rc; 1834 1834 + } 1835 1835 + module_init(ntb_transport_init); 1836 1836 + 1837 1837 + static void __exit ntb_transport_exit(void) 1838 1838 + { 1839 1839 + debugfs_remove_recursive(nt_debugfs_dir); 1840 1840 + 1841 1841 + ntb_unregister_client(&ntb_transport_client); 1842 1842 + bus_unregister(&ntb_transport_bus); 1843 1843 + } 1844 1844 + module_exit(ntb_transport_exit);

+19

drivers/ntb/test/Kconfig

reviewed

··· 1 1 + config NTB_PINGPONG 2 2 + tristate "NTB Ping Pong Test Client" 3 3 + help 4 4 + This is a simple ping pong driver that exercises the scratchpads and 5 5 + doorbells of the ntb hardware. This driver may be used to test that 6 6 + your ntb hardware and drivers are functioning at a basic level. 7 7 + 8 8 + If unsure, say N. 9 9 + 10 10 + config NTB_TOOL 11 11 + tristate "NTB Debugging Tool Test Client" 12 12 + help 13 13 + This is a simple debugging driver that enables the doorbell and 14 14 + scratchpad registers to be read and written from the debugfs. This 15 15 + enables more complicated debugging to be scripted from user space. 16 16 + This driver may be used to test that your ntb hardware and drivers are 17 17 + functioning at a basic level. 18 18 + 19 19 + If unsure, say N.

+2

drivers/ntb/test/Makefile

reviewed

··· 1 1 + obj-$(CONFIG_NTB_PINGPONG) += ntb_pingpong.o 2 2 + obj-$(CONFIG_NTB_TOOL) += ntb_tool.o

+250

drivers/ntb/test/ntb_pingpong.c

reviewed

··· 1 1 + /* 2 2 + * This file is provided under a dual BSD/GPLv2 license. When using or 3 3 + * redistributing this file, you may do so under either license. 4 4 + * 5 5 + * GPL LICENSE SUMMARY 6 6 + * 7 7 + * Copyright (C) 2015 EMC Corporation. All Rights Reserved. 8 8 + * 9 9 + * This program is free software; you can redistribute it and/or modify 10 10 + * it under the terms of version 2 of the GNU General Public License as 11 11 + * published by the Free Software Foundation. 12 12 + * 13 13 + * This program is distributed in the hope that it will be useful, but 14 14 + * WITHOUT ANY WARRANTY; without even the implied warranty of 15 15 + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 16 16 + * General Public License for more details. 17 17 + * 18 18 + * BSD LICENSE 19 19 + * 20 20 + * Copyright (C) 2015 EMC Corporation. All Rights Reserved. 21 21 + * 22 22 + * Redistribution and use in source and binary forms, with or without 23 23 + * modification, are permitted provided that the following conditions 24 24 + * are met: 25 25 + * 26 26 + * * Redistributions of source code must retain the above copyright 27 27 + * notice, this list of conditions and the following disclaimer. 28 28 + * * Redistributions in binary form must reproduce the above copy 29 29 + * notice, this list of conditions and the following disclaimer in 30 30 + * the documentation and/or other materials provided with the 31 31 + * distribution. 32 32 + * * Neither the name of Intel Corporation nor the names of its 33 33 + * contributors may be used to endorse or promote products derived 34 34 + * from this software without specific prior written permission. 35 35 + * 36 36 + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 37 37 + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 38 38 + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 39 39 + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 40 40 + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 41 41 + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 42 42 + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 43 43 + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 44 44 + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 45 45 + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 46 46 + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 47 47 + * 48 48 + * PCIe NTB Pingpong Linux driver 49 49 + * 50 50 + * Contact Information: 51 51 + * Allen Hubbe <Allen.Hubbe@emc.com> 52 52 + */ 53 53 + 54 54 + /* Note: load this module with option 'dyndbg=+p' */ 55 55 + 56 56 + #include <linux/init.h> 57 57 + #include <linux/kernel.h> 58 58 + #include <linux/module.h> 59 59 + 60 60 + #include <linux/dma-mapping.h> 61 61 + #include <linux/pci.h> 62 62 + #include <linux/slab.h> 63 63 + #include <linux/spinlock.h> 64 64 + 65 65 + #include <linux/ntb.h> 66 66 + 67 67 + #define DRIVER_NAME "ntb_pingpong" 68 68 + #define DRIVER_DESCRIPTION "PCIe NTB Simple Pingpong Client" 69 69 + 70 70 + #define DRIVER_LICENSE "Dual BSD/GPL" 71 71 + #define DRIVER_VERSION "1.0" 72 72 + #define DRIVER_RELDATE "24 March 2015" 73 73 + #define DRIVER_AUTHOR "Allen Hubbe <Allen.Hubbe@emc.com>" 74 74 + 75 75 + MODULE_LICENSE(DRIVER_LICENSE); 76 76 + MODULE_VERSION(DRIVER_VERSION); 77 77 + MODULE_AUTHOR(DRIVER_AUTHOR); 78 78 + MODULE_DESCRIPTION(DRIVER_DESCRIPTION); 79 79 + 80 80 + static unsigned int unsafe; 81 81 + module_param(unsafe, uint, 0644); 82 82 + MODULE_PARM_DESC(unsafe, "Run even though ntb operations may be unsafe"); 83 83 + 84 84 + static unsigned int delay_ms = 1000; 85 85 + module_param(delay_ms, uint, 0644); 86 86 + MODULE_PARM_DESC(delay_ms, "Milliseconds to delay the response to peer"); 87 87 + 88 88 + static unsigned long db_init = 0x7; 89 89 + module_param(db_init, ulong, 0644); 90 90 + MODULE_PARM_DESC(delay_ms, "Initial doorbell bits to ring on the peer"); 91 91 + 92 92 + struct pp_ctx { 93 93 + struct ntb_dev *ntb; 94 94 + u64 db_bits; 95 95 + /* synchronize access to db_bits by ping and pong */ 96 96 + spinlock_t db_lock; 97 97 + struct timer_list db_timer; 98 98 + unsigned long db_delay; 99 99 + }; 100 100 + 101 101 + static void pp_ping(unsigned long ctx) 102 102 + { 103 103 + struct pp_ctx *pp = (void *)ctx; 104 104 + unsigned long irqflags; 105 105 + u64 db_bits, db_mask; 106 106 + u32 spad_rd, spad_wr; 107 107 + 108 108 + spin_lock_irqsave(&pp->db_lock, irqflags); 109 109 + { 110 110 + db_mask = ntb_db_valid_mask(pp->ntb); 111 111 + db_bits = ntb_db_read(pp->ntb); 112 112 + 113 113 + if (db_bits) { 114 114 + dev_dbg(&pp->ntb->dev, 115 115 + "Masked pongs %#llx\n", 116 116 + db_bits); 117 117 + ntb_db_clear(pp->ntb, db_bits); 118 118 + } 119 119 + 120 120 + db_bits = ((pp->db_bits | db_bits) << 1) & db_mask; 121 121 + 122 122 + if (!db_bits) 123 123 + db_bits = db_init; 124 124 + 125 125 + spad_rd = ntb_spad_read(pp->ntb, 0); 126 126 + spad_wr = spad_rd + 1; 127 127 + 128 128 + dev_dbg(&pp->ntb->dev, 129 129 + "Ping bits %#llx read %#x write %#x\n", 130 130 + db_bits, spad_rd, spad_wr); 131 131 + 132 132 + ntb_peer_spad_write(pp->ntb, 0, spad_wr); 133 133 + ntb_peer_db_set(pp->ntb, db_bits); 134 134 + ntb_db_clear_mask(pp->ntb, db_mask); 135 135 + 136 136 + pp->db_bits = 0; 137 137 + } 138 138 + spin_unlock_irqrestore(&pp->db_lock, irqflags); 139 139 + } 140 140 + 141 141 + static void pp_link_event(void *ctx) 142 142 + { 143 143 + struct pp_ctx *pp = ctx; 144 144 + 145 145 + if (ntb_link_is_up(pp->ntb, NULL, NULL) == 1) { 146 146 + dev_dbg(&pp->ntb->dev, "link is up\n"); 147 147 + pp_ping((unsigned long)pp); 148 148 + } else { 149 149 + dev_dbg(&pp->ntb->dev, "link is down\n"); 150 150 + del_timer(&pp->db_timer); 151 151 + } 152 152 + } 153 153 + 154 154 + static void pp_db_event(void *ctx, int vec) 155 155 + { 156 156 + struct pp_ctx *pp = ctx; 157 157 + u64 db_bits, db_mask; 158 158 + unsigned long irqflags; 159 159 + 160 160 + spin_lock_irqsave(&pp->db_lock, irqflags); 161 161 + { 162 162 + db_mask = ntb_db_vector_mask(pp->ntb, vec); 163 163 + db_bits = db_mask & ntb_db_read(pp->ntb); 164 164 + ntb_db_set_mask(pp->ntb, db_mask); 165 165 + ntb_db_clear(pp->ntb, db_bits); 166 166 + 167 167 + pp->db_bits |= db_bits; 168 168 + 169 169 + mod_timer(&pp->db_timer, jiffies + pp->db_delay); 170 170 + 171 171 + dev_dbg(&pp->ntb->dev, 172 172 + "Pong vec %d bits %#llx\n", 173 173 + vec, db_bits); 174 174 + } 175 175 + spin_unlock_irqrestore(&pp->db_lock, irqflags); 176 176 + } 177 177 + 178 178 + static const struct ntb_ctx_ops pp_ops = { 179 179 + .link_event = pp_link_event, 180 180 + .db_event = pp_db_event, 181 181 + }; 182 182 + 183 183 + static int pp_probe(struct ntb_client *client, 184 184 + struct ntb_dev *ntb) 185 185 + { 186 186 + struct pp_ctx *pp; 187 187 + int rc; 188 188 + 189 189 + if (ntb_db_is_unsafe(ntb)) { 190 190 + dev_dbg(&ntb->dev, "doorbell is unsafe\n"); 191 191 + if (!unsafe) { 192 192 + rc = -EINVAL; 193 193 + goto err_pp; 194 194 + } 195 195 + } 196 196 + 197 197 + if (ntb_spad_is_unsafe(ntb)) { 198 198 + dev_dbg(&ntb->dev, "scratchpad is unsafe\n"); 199 199 + if (!unsafe) { 200 200 + rc = -EINVAL; 201 201 + goto err_pp; 202 202 + } 203 203 + } 204 204 + 205 205 + pp = kmalloc(sizeof(*pp), GFP_KERNEL); 206 206 + if (!pp) { 207 207 + rc = -ENOMEM; 208 208 + goto err_pp; 209 209 + } 210 210 + 211 211 + pp->ntb = ntb; 212 212 + pp->db_bits = 0; 213 213 + spin_lock_init(&pp->db_lock); 214 214 + setup_timer(&pp->db_timer, pp_ping, (unsigned long)pp); 215 215 + pp->db_delay = msecs_to_jiffies(delay_ms); 216 216 + 217 217 + rc = ntb_set_ctx(ntb, pp, &pp_ops); 218 218 + if (rc) 219 219 + goto err_ctx; 220 220 + 221 221 + ntb_link_enable(ntb, NTB_SPEED_AUTO, NTB_WIDTH_AUTO); 222 222 + ntb_link_event(ntb); 223 223 + 224 224 + return 0; 225 225 + 226 226 + err_ctx: 227 227 + kfree(pp); 228 228 + err_pp: 229 229 + return rc; 230 230 + } 231 231 + 232 232 + static void pp_remove(struct ntb_client *client, 233 233 + struct ntb_dev *ntb) 234 234 + { 235 235 + struct pp_ctx *pp = ntb->ctx; 236 236 + 237 237 + ntb_clear_ctx(ntb); 238 238 + del_timer_sync(&pp->db_timer); 239 239 + ntb_link_disable(ntb); 240 240 + 241 241 + kfree(pp); 242 242 + } 243 243 + 244 244 + static struct ntb_client pp_client = { 245 245 + .ops = { 246 246 + .probe = pp_probe, 247 247 + .remove = pp_remove, 248 248 + }, 249 249 + }; 250 250 + module_ntb_client(pp_client);

+556

drivers/ntb/test/ntb_tool.c

reviewed

··· 1 1 + /* 2 2 + * This file is provided under a dual BSD/GPLv2 license. When using or 3 3 + * redistributing this file, you may do so under either license. 4 4 + * 5 5 + * GPL LICENSE SUMMARY 6 6 + * 7 7 + * Copyright (C) 2015 EMC Corporation. All Rights Reserved. 8 8 + * 9 9 + * This program is free software; you can redistribute it and/or modify 10 10 + * it under the terms of version 2 of the GNU General Public License as 11 11 + * published by the Free Software Foundation. 12 12 + * 13 13 + * This program is distributed in the hope that it will be useful, but 14 14 + * WITHOUT ANY WARRANTY; without even the implied warranty of 15 15 + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 16 16 + * General Public License for more details. 17 17 + * 18 18 + * BSD LICENSE 19 19 + * 20 20 + * Copyright (C) 2015 EMC Corporation. All Rights Reserved. 21 21 + * 22 22 + * Redistribution and use in source and binary forms, with or without 23 23 + * modification, are permitted provided that the following conditions 24 24 + * are met: 25 25 + * 26 26 + * * Redistributions of source code must retain the above copyright 27 27 + * notice, this list of conditions and the following disclaimer. 28 28 + * * Redistributions in binary form must reproduce the above copy 29 29 + * notice, this list of conditions and the following disclaimer in 30 30 + * the documentation and/or other materials provided with the 31 31 + * distribution. 32 32 + * * Neither the name of Intel Corporation nor the names of its 33 33 + * contributors may be used to endorse or promote products derived 34 34 + * from this software without specific prior written permission. 35 35 + * 36 36 + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 37 37 + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 38 38 + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 39 39 + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 40 40 + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 41 41 + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 42 42 + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 43 43 + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 44 44 + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 45 45 + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 46 46 + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 47 47 + * 48 48 + * PCIe NTB Debugging Tool Linux driver 49 49 + * 50 50 + * Contact Information: 51 51 + * Allen Hubbe <Allen.Hubbe@emc.com> 52 52 + */ 53 53 + 54 54 + /* 55 55 + * How to use this tool, by example. 56 56 + * 57 57 + * Assuming $DBG_DIR is something like: 58 58 + * '/sys/kernel/debug/ntb_tool/0000:00:03.0' 59 59 + * 60 60 + * Eg: check if clearing the doorbell mask generates an interrupt. 61 61 + * 62 62 + * # Set the doorbell mask 63 63 + * root@self# echo 's 1' > $DBG_DIR/mask 64 64 + * 65 65 + * # Ring the doorbell from the peer 66 66 + * root@peer# echo 's 1' > $DBG_DIR/peer_db 67 67 + * 68 68 + * # Clear the doorbell mask 69 69 + * root@self# echo 'c 1' > $DBG_DIR/mask 70 70 + * 71 71 + * Observe debugging output in dmesg or your console. You should see a 72 72 + * doorbell event triggered by clearing the mask. If not, this may indicate an 73 73 + * issue with the hardware that needs to be worked around in the driver. 74 74 + * 75 75 + * Eg: read and write scratchpad registers 76 76 + * 77 77 + * root@peer# echo '0 0x01010101 1 0x7f7f7f7f' > $DBG_DIR/peer_spad 78 78 + * 79 79 + * root@self# cat $DBG_DIR/spad 80 80 + * 81 81 + * Observe that spad 0 and 1 have the values set by the peer. 82 82 + */ 83 83 + 84 84 + #include <linux/init.h> 85 85 + #include <linux/kernel.h> 86 86 + #include <linux/module.h> 87 87 + 88 88 + #include <linux/debugfs.h> 89 89 + #include <linux/dma-mapping.h> 90 90 + #include <linux/pci.h> 91 91 + #include <linux/slab.h> 92 92 + 93 93 + #include <linux/ntb.h> 94 94 + 95 95 + #define DRIVER_NAME "ntb_tool" 96 96 + #define DRIVER_DESCRIPTION "PCIe NTB Debugging Tool" 97 97 + 98 98 + #define DRIVER_LICENSE "Dual BSD/GPL" 99 99 + #define DRIVER_VERSION "1.0" 100 100 + #define DRIVER_RELDATE "22 April 2015" 101 101 + #define DRIVER_AUTHOR "Allen Hubbe <Allen.Hubbe@emc.com>" 102 102 + 103 103 + MODULE_LICENSE(DRIVER_LICENSE); 104 104 + MODULE_VERSION(DRIVER_VERSION); 105 105 + MODULE_AUTHOR(DRIVER_AUTHOR); 106 106 + MODULE_DESCRIPTION(DRIVER_DESCRIPTION); 107 107 + 108 108 + static struct dentry *tool_dbgfs; 109 109 + 110 110 + struct tool_ctx { 111 111 + struct ntb_dev *ntb; 112 112 + struct dentry *dbgfs; 113 113 + }; 114 114 + 115 115 + #define SPAD_FNAME_SIZE 0x10 116 116 + #define INT_PTR(x) ((void *)(unsigned long)x) 117 117 + #define PTR_INT(x) ((int)(unsigned long)x) 118 118 + 119 119 + #define TOOL_FOPS_RDWR(__name, __read, __write) \ 120 120 + const struct file_operations __name = { \ 121 121 + .owner = THIS_MODULE, \ 122 122 + .open = simple_open, \ 123 123 + .read = __read, \ 124 124 + .write = __write, \ 125 125 + } 126 126 + 127 127 + static void tool_link_event(void *ctx) 128 128 + { 129 129 + struct tool_ctx *tc = ctx; 130 130 + enum ntb_speed speed; 131 131 + enum ntb_width width; 132 132 + int up; 133 133 + 134 134 + up = ntb_link_is_up(tc->ntb, &speed, &width); 135 135 + 136 136 + dev_dbg(&tc->ntb->dev, "link is %s speed %d width %d\n", 137 137 + up ? "up" : "down", speed, width); 138 138 + } 139 139 + 140 140 + static void tool_db_event(void *ctx, int vec) 141 141 + { 142 142 + struct tool_ctx *tc = ctx; 143 143 + u64 db_bits, db_mask; 144 144 + 145 145 + db_mask = ntb_db_vector_mask(tc->ntb, vec); 146 146 + db_bits = ntb_db_read(tc->ntb); 147 147 + 148 148 + dev_dbg(&tc->ntb->dev, "doorbell vec %d mask %#llx bits %#llx\n", 149 149 + vec, db_mask, db_bits); 150 150 + } 151 151 + 152 152 + static const struct ntb_ctx_ops tool_ops = { 153 153 + .link_event = tool_link_event, 154 154 + .db_event = tool_db_event, 155 155 + }; 156 156 + 157 157 + static ssize_t tool_dbfn_read(struct tool_ctx *tc, char __user *ubuf, 158 158 + size_t size, loff_t *offp, 159 159 + u64 (*db_read_fn)(struct ntb_dev *)) 160 160 + { 161 161 + size_t buf_size; 162 162 + char *buf; 163 163 + ssize_t pos, rc; 164 164 + 165 165 + if (!db_read_fn) 166 166 + return -EINVAL; 167 167 + 168 168 + buf_size = min_t(size_t, size, 0x20); 169 169 + 170 170 + buf = kmalloc(buf_size, GFP_KERNEL); 171 171 + if (!buf) 172 172 + return -ENOMEM; 173 173 + 174 174 + pos = scnprintf(buf, buf_size, "%#llx\n", 175 175 + db_read_fn(tc->ntb)); 176 176 + 177 177 + rc = simple_read_from_buffer(ubuf, size, offp, buf, pos); 178 178 + 179 179 + kfree(buf); 180 180 + 181 181 + return rc; 182 182 + } 183 183 + 184 184 + static ssize_t tool_dbfn_write(struct tool_ctx *tc, 185 185 + const char __user *ubuf, 186 186 + size_t size, loff_t *offp, 187 187 + int (*db_set_fn)(struct ntb_dev *, u64), 188 188 + int (*db_clear_fn)(struct ntb_dev *, u64)) 189 189 + { 190 190 + u64 db_bits; 191 191 + char *buf, cmd; 192 192 + ssize_t rc; 193 193 + int n; 194 194 + 195 195 + buf = kmalloc(size + 1, GFP_KERNEL); 196 196 + if (!buf) 197 197 + return -ENOMEM; 198 198 + 199 199 + rc = simple_write_to_buffer(buf, size, offp, ubuf, size); 200 200 + if (rc < 0) { 201 201 + kfree(buf); 202 202 + return rc; 203 203 + } 204 204 + 205 205 + buf[size] = 0; 206 206 + 207 207 + n = sscanf(buf, "%c %lli", &cmd, &db_bits); 208 208 + 209 209 + kfree(buf); 210 210 + 211 211 + if (n != 2) { 212 212 + rc = -EINVAL; 213 213 + } else if (cmd == 's') { 214 214 + if (!db_set_fn) 215 215 + rc = -EINVAL; 216 216 + else 217 217 + rc = db_set_fn(tc->ntb, db_bits); 218 218 + } else if (cmd == 'c') { 219 219 + if (!db_clear_fn) 220 220 + rc = -EINVAL; 221 221 + else 222 222 + rc = db_clear_fn(tc->ntb, db_bits); 223 223 + } else { 224 224 + rc = -EINVAL; 225 225 + } 226 226 + 227 227 + return rc ? : size; 228 228 + } 229 229 + 230 230 + static ssize_t tool_spadfn_read(struct tool_ctx *tc, char __user *ubuf, 231 231 + size_t size, loff_t *offp, 232 232 + u32 (*spad_read_fn)(struct ntb_dev *, int)) 233 233 + { 234 234 + size_t buf_size; 235 235 + char *buf; 236 236 + ssize_t pos, rc; 237 237 + int i, spad_count; 238 238 + 239 239 + if (!spad_read_fn) 240 240 + return -EINVAL; 241 241 + 242 242 + buf_size = min_t(size_t, size, 0x100); 243 243 + 244 244 + buf = kmalloc(buf_size, GFP_KERNEL); 245 245 + if (!buf) 246 246 + return -ENOMEM; 247 247 + 248 248 + pos = 0; 249 249 + 250 250 + spad_count = ntb_spad_count(tc->ntb); 251 251 + for (i = 0; i < spad_count; ++i) { 252 252 + pos += scnprintf(buf + pos, buf_size - pos, "%d\t%#x\n", 253 253 + i, spad_read_fn(tc->ntb, i)); 254 254 + } 255 255 + 256 256 + rc = simple_read_from_buffer(ubuf, size, offp, buf, pos); 257 257 + 258 258 + kfree(buf); 259 259 + 260 260 + return rc; 261 261 + } 262 262 + 263 263 + static ssize_t tool_spadfn_write(struct tool_ctx *tc, 264 264 + const char __user *ubuf, 265 265 + size_t size, loff_t *offp, 266 266 + int (*spad_write_fn)(struct ntb_dev *, 267 267 + int, u32)) 268 268 + { 269 269 + int spad_idx; 270 270 + u32 spad_val; 271 271 + char *buf; 272 272 + int pos, n; 273 273 + ssize_t rc; 274 274 + 275 275 + if (!spad_write_fn) { 276 276 + dev_dbg(&tc->ntb->dev, "no spad write fn\n"); 277 277 + return -EINVAL; 278 278 + } 279 279 + 280 280 + buf = kmalloc(size + 1, GFP_KERNEL); 281 281 + if (!buf) 282 282 + return -ENOMEM; 283 283 + 284 284 + rc = simple_write_to_buffer(buf, size, offp, ubuf, size); 285 285 + if (rc < 0) { 286 286 + kfree(buf); 287 287 + return rc; 288 288 + } 289 289 + 290 290 + buf[size] = 0; 291 291 + 292 292 + n = sscanf(buf, "%d %i%n", &spad_idx, &spad_val, &pos); 293 293 + while (n == 2) { 294 294 + rc = spad_write_fn(tc->ntb, spad_idx, spad_val); 295 295 + if (rc) 296 296 + break; 297 297 + 298 298 + n = sscanf(buf + pos, "%d %i%n", &spad_idx, &spad_val, &pos); 299 299 + } 300 300 + 301 301 + if (n < 0) 302 302 + rc = n; 303 303 + 304 304 + kfree(buf); 305 305 + 306 306 + return rc ? : size; 307 307 + } 308 308 + 309 309 + static ssize_t tool_db_read(struct file *filep, char __user *ubuf, 310 310 + size_t size, loff_t *offp) 311 311 + { 312 312 + struct tool_ctx *tc = filep->private_data; 313 313 + 314 314 + return tool_dbfn_read(tc, ubuf, size, offp, 315 315 + tc->ntb->ops->db_read); 316 316 + } 317 317 + 318 318 + static ssize_t tool_db_write(struct file *filep, const char __user *ubuf, 319 319 + size_t size, loff_t *offp) 320 320 + { 321 321 + struct tool_ctx *tc = filep->private_data; 322 322 + 323 323 + return tool_dbfn_write(tc, ubuf, size, offp, 324 324 + tc->ntb->ops->db_set, 325 325 + tc->ntb->ops->db_clear); 326 326 + } 327 327 + 328 328 + static TOOL_FOPS_RDWR(tool_db_fops, 329 329 + tool_db_read, 330 330 + tool_db_write); 331 331 + 332 332 + static ssize_t tool_mask_read(struct file *filep, char __user *ubuf, 333 333 + size_t size, loff_t *offp) 334 334 + { 335 335 + struct tool_ctx *tc = filep->private_data; 336 336 + 337 337 + return tool_dbfn_read(tc, ubuf, size, offp, 338 338 + tc->ntb->ops->db_read_mask); 339 339 + } 340 340 + 341 341 + static ssize_t tool_mask_write(struct file *filep, const char __user *ubuf, 342 342 + size_t size, loff_t *offp) 343 343 + { 344 344 + struct tool_ctx *tc = filep->private_data; 345 345 + 346 346 + return tool_dbfn_write(tc, ubuf, size, offp, 347 347 + tc->ntb->ops->db_set_mask, 348 348 + tc->ntb->ops->db_clear_mask); 349 349 + } 350 350 + 351 351 + static TOOL_FOPS_RDWR(tool_mask_fops, 352 352 + tool_mask_read, 353 353 + tool_mask_write); 354 354 + 355 355 + static ssize_t tool_peer_db_read(struct file *filep, char __user *ubuf, 356 356 + size_t size, loff_t *offp) 357 357 + { 358 358 + struct tool_ctx *tc = filep->private_data; 359 359 + 360 360 + return tool_dbfn_read(tc, ubuf, size, offp, 361 361 + tc->ntb->ops->peer_db_read); 362 362 + } 363 363 + 364 364 + static ssize_t tool_peer_db_write(struct file *filep, const char __user *ubuf, 365 365 + size_t size, loff_t *offp) 366 366 + { 367 367 + struct tool_ctx *tc = filep->private_data; 368 368 + 369 369 + return tool_dbfn_write(tc, ubuf, size, offp, 370 370 + tc->ntb->ops->peer_db_set, 371 371 + tc->ntb->ops->peer_db_clear); 372 372 + } 373 373 + 374 374 + static TOOL_FOPS_RDWR(tool_peer_db_fops, 375 375 + tool_peer_db_read, 376 376 + tool_peer_db_write); 377 377 + 378 378 + static ssize_t tool_peer_mask_read(struct file *filep, char __user *ubuf, 379 379 + size_t size, loff_t *offp) 380 380 + { 381 381 + struct tool_ctx *tc = filep->private_data; 382 382 + 383 383 + return tool_dbfn_read(tc, ubuf, size, offp, 384 384 + tc->ntb->ops->peer_db_read_mask); 385 385 + } 386 386 + 387 387 + static ssize_t tool_peer_mask_write(struct file *filep, const char __user *ubuf, 388 388 + size_t size, loff_t *offp) 389 389 + { 390 390 + struct tool_ctx *tc = filep->private_data; 391 391 + 392 392 + return tool_dbfn_write(tc, ubuf, size, offp, 393 393 + tc->ntb->ops->peer_db_set_mask, 394 394 + tc->ntb->ops->peer_db_clear_mask); 395 395 + } 396 396 + 397 397 + static TOOL_FOPS_RDWR(tool_peer_mask_fops, 398 398 + tool_peer_mask_read, 399 399 + tool_peer_mask_write); 400 400 + 401 401 + static ssize_t tool_spad_read(struct file *filep, char __user *ubuf, 402 402 + size_t size, loff_t *offp) 403 403 + { 404 404 + struct tool_ctx *tc = filep->private_data; 405 405 + 406 406 + return tool_spadfn_read(tc, ubuf, size, offp, 407 407 + tc->ntb->ops->spad_read); 408 408 + } 409 409 + 410 410 + static ssize_t tool_spad_write(struct file *filep, const char __user *ubuf, 411 411 + size_t size, loff_t *offp) 412 412 + { 413 413 + struct tool_ctx *tc = filep->private_data; 414 414 + 415 415 + return tool_spadfn_write(tc, ubuf, size, offp, 416 416 + tc->ntb->ops->spad_write); 417 417 + } 418 418 + 419 419 + static TOOL_FOPS_RDWR(tool_spad_fops, 420 420 + tool_spad_read, 421 421 + tool_spad_write); 422 422 + 423 423 + static ssize_t tool_peer_spad_read(struct file *filep, char __user *ubuf, 424 424 + size_t size, loff_t *offp) 425 425 + { 426 426 + struct tool_ctx *tc = filep->private_data; 427 427 + 428 428 + return tool_spadfn_read(tc, ubuf, size, offp, 429 429 + tc->ntb->ops->peer_spad_read); 430 430 + } 431 431 + 432 432 + static ssize_t tool_peer_spad_write(struct file *filep, const char __user *ubuf, 433 433 + size_t size, loff_t *offp) 434 434 + { 435 435 + struct tool_ctx *tc = filep->private_data; 436 436 + 437 437 + return tool_spadfn_write(tc, ubuf, size, offp, 438 438 + tc->ntb->ops->peer_spad_write); 439 439 + } 440 440 + 441 441 + static TOOL_FOPS_RDWR(tool_peer_spad_fops, 442 442 + tool_peer_spad_read, 443 443 + tool_peer_spad_write); 444 444 + 445 445 + static void tool_setup_dbgfs(struct tool_ctx *tc) 446 446 + { 447 447 + /* This modules is useless without dbgfs... */ 448 448 + if (!tool_dbgfs) { 449 449 + tc->dbgfs = NULL; 450 450 + return; 451 451 + } 452 452 + 453 453 + tc->dbgfs = debugfs_create_dir(dev_name(&tc->ntb->dev), 454 454 + tool_dbgfs); 455 455 + if (!tc->dbgfs) 456 456 + return; 457 457 + 458 458 + debugfs_create_file("db", S_IRUSR | S_IWUSR, tc->dbgfs, 459 459 + tc, &tool_db_fops); 460 460 + 461 461 + debugfs_create_file("mask", S_IRUSR | S_IWUSR, tc->dbgfs, 462 462 + tc, &tool_mask_fops); 463 463 + 464 464 + debugfs_create_file("peer_db", S_IRUSR | S_IWUSR, tc->dbgfs, 465 465 + tc, &tool_peer_db_fops); 466 466 + 467 467 + debugfs_create_file("peer_mask", S_IRUSR | S_IWUSR, tc->dbgfs, 468 468 + tc, &tool_peer_mask_fops); 469 469 + 470 470 + debugfs_create_file("spad", S_IRUSR | S_IWUSR, tc->dbgfs, 471 471 + tc, &tool_spad_fops); 472 472 + 473 473 + debugfs_create_file("peer_spad", S_IRUSR | S_IWUSR, tc->dbgfs, 474 474 + tc, &tool_peer_spad_fops); 475 475 + } 476 476 + 477 477 + static int tool_probe(struct ntb_client *self, struct ntb_dev *ntb) 478 478 + { 479 479 + struct tool_ctx *tc; 480 480 + int rc; 481 481 + 482 482 + if (ntb_db_is_unsafe(ntb)) 483 483 + dev_dbg(&ntb->dev, "doorbell is unsafe\n"); 484 484 + 485 485 + if (ntb_spad_is_unsafe(ntb)) 486 486 + dev_dbg(&ntb->dev, "scratchpad is unsafe\n"); 487 487 + 488 488 + tc = kmalloc(sizeof(*tc), GFP_KERNEL); 489 489 + if (!tc) { 490 490 + rc = -ENOMEM; 491 491 + goto err_tc; 492 492 + } 493 493 + 494 494 + tc->ntb = ntb; 495 495 + 496 496 + tool_setup_dbgfs(tc); 497 497 + 498 498 + rc = ntb_set_ctx(ntb, tc, &tool_ops); 499 499 + if (rc) 500 500 + goto err_ctx; 501 501 + 502 502 + ntb_link_enable(ntb, NTB_SPEED_AUTO, NTB_WIDTH_AUTO); 503 503 + ntb_link_event(ntb); 504 504 + 505 505 + return 0; 506 506 + 507 507 + err_ctx: 508 508 + debugfs_remove_recursive(tc->dbgfs); 509 509 + kfree(tc); 510 510 + err_tc: 511 511 + return rc; 512 512 + } 513 513 + 514 514 + static void tool_remove(struct ntb_client *self, struct ntb_dev *ntb) 515 515 + { 516 516 + struct tool_ctx *tc = ntb->ctx; 517 517 + 518 518 + ntb_clear_ctx(ntb); 519 519 + ntb_link_disable(ntb); 520 520 + 521 521 + debugfs_remove_recursive(tc->dbgfs); 522 522 + kfree(tc); 523 523 + } 524 524 + 525 525 + static struct ntb_client tool_client = { 526 526 + .ops = { 527 527 + .probe = tool_probe, 528 528 + .remove = tool_remove, 529 529 + }, 530 530 + }; 531 531 + 532 532 + static int __init tool_init(void) 533 533 + { 534 534 + int rc; 535 535 + 536 536 + if (debugfs_initialized()) 537 537 + tool_dbgfs = debugfs_create_dir(KBUILD_MODNAME, NULL); 538 538 + 539 539 + rc = ntb_register_client(&tool_client); 540 540 + if (rc) 541 541 + goto err_client; 542 542 + 543 543 + return 0; 544 544 + 545 545 + err_client: 546 546 + debugfs_remove_recursive(tool_dbgfs); 547 547 + return rc; 548 548 + } 549 549 + module_init(tool_init); 550 550 + 551 551 + static void __exit tool_exit(void) 552 552 + { 553 553 + ntb_unregister_client(&tool_client); 554 554 + debugfs_remove_recursive(tool_dbgfs); 555 555 + } 556 556 + module_exit(tool_exit);

+932 -36

include/linux/ntb.h

reviewed

··· 4 4 * 5 5 * GPL LICENSE SUMMARY 6 6 * 7 7 - * Copyright(c) 2012 Intel Corporation. All rights reserved. 7 7 + * Copyright (C) 2015 EMC Corporation. All Rights Reserved. 8 8 * 9 9 * This program is free software; you can redistribute it and/or modify 10 10 * it under the terms of version 2 of the GNU General Public License as 11 11 * published by the Free Software Foundation. 12 12 * 13 13 + * This program is distributed in the hope that it will be useful, but 14 14 + * WITHOUT ANY WARRANTY; without even the implied warranty of 15 15 + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 16 16 + * General Public License for more details. 17 17 + * 13 18 * BSD LICENSE 14 19 * 15 15 - * Copyright(c) 2012 Intel Corporation. All rights reserved. 20 20 + * Copyright (C) 2015 EMC Corporation. All Rights Reserved. 16 21 * 17 22 * Redistribution and use in source and binary forms, with or without 18 23 * modification, are permitted provided that the following conditions ··· 45 40 * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 46 41 * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 47 42 * 48 48 - * Intel PCIe NTB Linux driver 43 43 + * PCIe NTB Linux driver 49 44 * 50 45 * Contact Information: 51 51 - * Jon Mason <jon.mason@intel.com> 46 46 + * Allen Hubbe <Allen.Hubbe@emc.com> 52 47 */ 53 48 54 54 - struct ntb_transport_qp; 49 49 + #ifndef _NTB_H_ 50 50 + #define _NTB_H_ 55 51 52 52 + #include <linux/completion.h> 53 53 + #include <linux/device.h> 54 54 + 55 55 + struct ntb_client; 56 56 + struct ntb_dev; 57 57 + struct pci_dev; 58 58 + 59 59 + /** 60 60 + * enum ntb_topo - NTB connection topology 61 61 + * @NTB_TOPO_NONE: Topology is unknown or invalid. 62 62 + * @NTB_TOPO_PRI: On primary side of local ntb. 63 63 + * @NTB_TOPO_SEC: On secondary side of remote ntb. 64 64 + * @NTB_TOPO_B2B_USD: On primary side of local ntb upstream of remote ntb. 65 65 + * @NTB_TOPO_B2B_DSD: On primary side of local ntb downstream of remote ntb. 66 66 + */ 67 67 + enum ntb_topo { 68 68 + NTB_TOPO_NONE = -1, 69 69 + NTB_TOPO_PRI, 70 70 + NTB_TOPO_SEC, 71 71 + NTB_TOPO_B2B_USD, 72 72 + NTB_TOPO_B2B_DSD, 73 73 + }; 74 74 + 75 75 + static inline int ntb_topo_is_b2b(enum ntb_topo topo) 76 76 + { 77 77 + switch ((int)topo) { 78 78 + case NTB_TOPO_B2B_USD: 79 79 + case NTB_TOPO_B2B_DSD: 80 80 + return 1; 81 81 + } 82 82 + return 0; 83 83 + } 84 84 + 85 85 + static inline char *ntb_topo_string(enum ntb_topo topo) 86 86 + { 87 87 + switch (topo) { 88 88 + case NTB_TOPO_NONE: return "NTB_TOPO_NONE"; 89 89 + case NTB_TOPO_PRI: return "NTB_TOPO_PRI"; 90 90 + case NTB_TOPO_SEC: return "NTB_TOPO_SEC"; 91 91 + case NTB_TOPO_B2B_USD: return "NTB_TOPO_B2B_USD"; 92 92 + case NTB_TOPO_B2B_DSD: return "NTB_TOPO_B2B_DSD"; 93 93 + } 94 94 + return "NTB_TOPO_INVALID"; 95 95 + } 96 96 + 97 97 + /** 98 98 + * enum ntb_speed - NTB link training speed 99 99 + * @NTB_SPEED_AUTO: Request the max supported speed. 100 100 + * @NTB_SPEED_NONE: Link is not trained to any speed. 101 101 + * @NTB_SPEED_GEN1: Link is trained to gen1 speed. 102 102 + * @NTB_SPEED_GEN2: Link is trained to gen2 speed. 103 103 + * @NTB_SPEED_GEN3: Link is trained to gen3 speed. 104 104 + */ 105 105 + enum ntb_speed { 106 106 + NTB_SPEED_AUTO = -1, 107 107 + NTB_SPEED_NONE = 0, 108 108 + NTB_SPEED_GEN1 = 1, 109 109 + NTB_SPEED_GEN2 = 2, 110 110 + NTB_SPEED_GEN3 = 3, 111 111 + }; 112 112 + 113 113 + /** 114 114 + * enum ntb_width - NTB link training width 115 115 + * @NTB_WIDTH_AUTO: Request the max supported width. 116 116 + * @NTB_WIDTH_NONE: Link is not trained to any width. 117 117 + * @NTB_WIDTH_1: Link is trained to 1 lane width. 118 118 + * @NTB_WIDTH_2: Link is trained to 2 lane width. 119 119 + * @NTB_WIDTH_4: Link is trained to 4 lane width. 120 120 + * @NTB_WIDTH_8: Link is trained to 8 lane width. 121 121 + * @NTB_WIDTH_12: Link is trained to 12 lane width. 122 122 + * @NTB_WIDTH_16: Link is trained to 16 lane width. 123 123 + * @NTB_WIDTH_32: Link is trained to 32 lane width. 124 124 + */ 125 125 + enum ntb_width { 126 126 + NTB_WIDTH_AUTO = -1, 127 127 + NTB_WIDTH_NONE = 0, 128 128 + NTB_WIDTH_1 = 1, 129 129 + NTB_WIDTH_2 = 2, 130 130 + NTB_WIDTH_4 = 4, 131 131 + NTB_WIDTH_8 = 8, 132 132 + NTB_WIDTH_12 = 12, 133 133 + NTB_WIDTH_16 = 16, 134 134 + NTB_WIDTH_32 = 32, 135 135 + }; 136 136 + 137 137 + /** 138 138 + * struct ntb_client_ops - ntb client operations 139 139 + * @probe: Notify client of a new device. 140 140 + * @remove: Notify client to remove a device. 141 141 + */ 142 142 + struct ntb_client_ops { 143 143 + int (*probe)(struct ntb_client *client, struct ntb_dev *ntb); 144 144 + void (*remove)(struct ntb_client *client, struct ntb_dev *ntb); 145 145 + }; 146 146 + 147 147 + static inline int ntb_client_ops_is_valid(const struct ntb_client_ops *ops) 148 148 + { 149 149 + /* commented callbacks are not required: */ 150 150 + return 151 151 + ops->probe && 152 152 + ops->remove && 153 153 + 1; 154 154 + } 155 155 + 156 156 + /** 157 157 + * struct ntb_ctx_ops - ntb driver context operations 158 158 + * @link_event: See ntb_link_event(). 159 159 + * @db_event: See ntb_db_event(). 160 160 + */ 161 161 + struct ntb_ctx_ops { 162 162 + void (*link_event)(void *ctx); 163 163 + void (*db_event)(void *ctx, int db_vector); 164 164 + }; 165 165 + 166 166 + static inline int ntb_ctx_ops_is_valid(const struct ntb_ctx_ops *ops) 167 167 + { 168 168 + /* commented callbacks are not required: */ 169 169 + return 170 170 + /* ops->link_event && */ 171 171 + /* ops->db_event && */ 172 172 + 1; 173 173 + } 174 174 + 175 175 + /** 176 176 + * struct ntb_ctx_ops - ntb device operations 177 177 + * @mw_count: See ntb_mw_count(). 178 178 + * @mw_get_range: See ntb_mw_get_range(). 179 179 + * @mw_set_trans: See ntb_mw_set_trans(). 180 180 + * @mw_clear_trans: See ntb_mw_clear_trans(). 181 181 + * @link_is_up: See ntb_link_is_up(). 182 182 + * @link_enable: See ntb_link_enable(). 183 183 + * @link_disable: See ntb_link_disable(). 184 184 + * @db_is_unsafe: See ntb_db_is_unsafe(). 185 185 + * @db_valid_mask: See ntb_db_valid_mask(). 186 186 + * @db_vector_count: See ntb_db_vector_count(). 187 187 + * @db_vector_mask: See ntb_db_vector_mask(). 188 188 + * @db_read: See ntb_db_read(). 189 189 + * @db_set: See ntb_db_set(). 190 190 + * @db_clear: See ntb_db_clear(). 191 191 + * @db_read_mask: See ntb_db_read_mask(). 192 192 + * @db_set_mask: See ntb_db_set_mask(). 193 193 + * @db_clear_mask: See ntb_db_clear_mask(). 194 194 + * @peer_db_addr: See ntb_peer_db_addr(). 195 195 + * @peer_db_read: See ntb_peer_db_read(). 196 196 + * @peer_db_set: See ntb_peer_db_set(). 197 197 + * @peer_db_clear: See ntb_peer_db_clear(). 198 198 + * @peer_db_read_mask: See ntb_peer_db_read_mask(). 199 199 + * @peer_db_set_mask: See ntb_peer_db_set_mask(). 200 200 + * @peer_db_clear_mask: See ntb_peer_db_clear_mask(). 201 201 + * @spad_is_unsafe: See ntb_spad_is_unsafe(). 202 202 + * @spad_count: See ntb_spad_count(). 203 203 + * @spad_read: See ntb_spad_read(). 204 204 + * @spad_write: See ntb_spad_write(). 205 205 + * @peer_spad_addr: See ntb_peer_spad_addr(). 206 206 + * @peer_spad_read: See ntb_peer_spad_read(). 207 207 + * @peer_spad_write: See ntb_peer_spad_write(). 208 208 + */ 209 209 + struct ntb_dev_ops { 210 210 + int (*mw_count)(struct ntb_dev *ntb); 211 211 + int (*mw_get_range)(struct ntb_dev *ntb, int idx, 212 212 + phys_addr_t *base, resource_size_t *size, 213 213 + resource_size_t *align, resource_size_t *align_size); 214 214 + int (*mw_set_trans)(struct ntb_dev *ntb, int idx, 215 215 + dma_addr_t addr, resource_size_t size); 216 216 + int (*mw_clear_trans)(struct ntb_dev *ntb, int idx); 217 217 + 218 218 + int (*link_is_up)(struct ntb_dev *ntb, 219 219 + enum ntb_speed *speed, enum ntb_width *width); 220 220 + int (*link_enable)(struct ntb_dev *ntb, 221 221 + enum ntb_speed max_speed, enum ntb_width max_width); 222 222 + int (*link_disable)(struct ntb_dev *ntb); 223 223 + 224 224 + int (*db_is_unsafe)(struct ntb_dev *ntb); 225 225 + u64 (*db_valid_mask)(struct ntb_dev *ntb); 226 226 + int (*db_vector_count)(struct ntb_dev *ntb); 227 227 + u64 (*db_vector_mask)(struct ntb_dev *ntb, int db_vector); 228 228 + 229 229 + u64 (*db_read)(struct ntb_dev *ntb); 230 230 + int (*db_set)(struct ntb_dev *ntb, u64 db_bits); 231 231 + int (*db_clear)(struct ntb_dev *ntb, u64 db_bits); 232 232 + 233 233 + u64 (*db_read_mask)(struct ntb_dev *ntb); 234 234 + int (*db_set_mask)(struct ntb_dev *ntb, u64 db_bits); 235 235 + int (*db_clear_mask)(struct ntb_dev *ntb, u64 db_bits); 236 236 + 237 237 + int (*peer_db_addr)(struct ntb_dev *ntb, 238 238 + phys_addr_t *db_addr, resource_size_t *db_size); 239 239 + u64 (*peer_db_read)(struct ntb_dev *ntb); 240 240 + int (*peer_db_set)(struct ntb_dev *ntb, u64 db_bits); 241 241 + int (*peer_db_clear)(struct ntb_dev *ntb, u64 db_bits); 242 242 + 243 243 + u64 (*peer_db_read_mask)(struct ntb_dev *ntb); 244 244 + int (*peer_db_set_mask)(struct ntb_dev *ntb, u64 db_bits); 245 245 + int (*peer_db_clear_mask)(struct ntb_dev *ntb, u64 db_bits); 246 246 + 247 247 + int (*spad_is_unsafe)(struct ntb_dev *ntb); 248 248 + int (*spad_count)(struct ntb_dev *ntb); 249 249 + 250 250 + u32 (*spad_read)(struct ntb_dev *ntb, int idx); 251 251 + int (*spad_write)(struct ntb_dev *ntb, int idx, u32 val); 252 252 + 253 253 + int (*peer_spad_addr)(struct ntb_dev *ntb, int idx, 254 254 + phys_addr_t *spad_addr); 255 255 + u32 (*peer_spad_read)(struct ntb_dev *ntb, int idx); 256 256 + int (*peer_spad_write)(struct ntb_dev *ntb, int idx, u32 val); 257 257 + }; 258 258 + 259 259 + static inline int ntb_dev_ops_is_valid(const struct ntb_dev_ops *ops) 260 260 + { 261 261 + /* commented callbacks are not required: */ 262 262 + return 263 263 + ops->mw_count && 264 264 + ops->mw_get_range && 265 265 + ops->mw_set_trans && 266 266 + /* ops->mw_clear_trans && */ 267 267 + ops->link_is_up && 268 268 + ops->link_enable && 269 269 + ops->link_disable && 270 270 + /* ops->db_is_unsafe && */ 271 271 + ops->db_valid_mask && 272 272 + 273 273 + /* both set, or both unset */ 274 274 + (!ops->db_vector_count == !ops->db_vector_mask) && 275 275 + 276 276 + ops->db_read && 277 277 + /* ops->db_set && */ 278 278 + ops->db_clear && 279 279 + /* ops->db_read_mask && */ 280 280 + ops->db_set_mask && 281 281 + ops->db_clear_mask && 282 282 + ops->peer_db_addr && 283 283 + /* ops->peer_db_read && */ 284 284 + ops->peer_db_set && 285 285 + /* ops->peer_db_clear && */ 286 286 + /* ops->peer_db_read_mask && */ 287 287 + /* ops->peer_db_set_mask && */ 288 288 + /* ops->peer_db_clear_mask && */ 289 289 + /* ops->spad_is_unsafe && */ 290 290 + ops->spad_count && 291 291 + ops->spad_read && 292 292 + ops->spad_write && 293 293 + ops->peer_spad_addr && 294 294 + /* ops->peer_spad_read && */ 295 295 + ops->peer_spad_write && 296 296 + 1; 297 297 + } 298 298 + 299 299 + /** 300 300 + * struct ntb_client - client interested in ntb devices 301 301 + * @drv: Linux driver object. 302 302 + * @ops: See &ntb_client_ops. 303 303 + */ 56 304 struct ntb_client { 57 57 - struct device_driver driver; 58 58 - int (*probe)(struct pci_dev *pdev); 59 59 - void (*remove)(struct pci_dev *pdev); 305 305 + struct device_driver drv; 306 306 + const struct ntb_client_ops ops; 60 307 }; 61 308 62 62 - enum { 63 63 - NTB_LINK_DOWN = 0, 64 64 - NTB_LINK_UP, 309 309 + #define drv_ntb_client(__drv) container_of((__drv), struct ntb_client, drv) 310 310 + 311 311 + /** 312 312 + * struct ntb_device - ntb device 313 313 + * @dev: Linux device object. 314 314 + * @pdev: Pci device entry of the ntb. 315 315 + * @topo: Detected topology of the ntb. 316 316 + * @ops: See &ntb_dev_ops. 317 317 + * @ctx: See &ntb_ctx_ops. 318 318 + * @ctx_ops: See &ntb_ctx_ops. 319 319 + */ 320 320 + struct ntb_dev { 321 321 + struct device dev; 322 322 + struct pci_dev *pdev; 323 323 + enum ntb_topo topo; 324 324 + const struct ntb_dev_ops *ops; 325 325 + void *ctx; 326 326 + const struct ntb_ctx_ops *ctx_ops; 327 327 + 328 328 + /* private: */ 329 329 + 330 330 + /* synchronize setting, clearing, and calling ctx_ops */ 331 331 + spinlock_t ctx_lock; 332 332 + /* block unregister until device is fully released */ 333 333 + struct completion released; 65 334 }; 66 335 67 67 - int ntb_register_client(struct ntb_client *drvr); 68 68 - void ntb_unregister_client(struct ntb_client *drvr); 69 69 - int ntb_register_client_dev(char *device_name); 70 70 - void ntb_unregister_client_dev(char *device_name); 336 336 + #define dev_ntb(__dev) container_of((__dev), struct ntb_dev, dev) 71 337 72 72 - struct ntb_queue_handlers { 73 73 - void (*rx_handler)(struct ntb_transport_qp *qp, void *qp_data, 74 74 - void *data, int len); 75 75 - void (*tx_handler)(struct ntb_transport_qp *qp, void *qp_data, 76 76 - void *data, int len); 77 77 - void (*event_handler)(void *data, int status); 78 78 - }; 338 338 + /** 339 339 + * ntb_register_client() - register a client for interest in ntb devices 340 340 + * @client: Client context. 341 341 + * 342 342 + * The client will be added to the list of clients interested in ntb devices. 343 343 + * The client will be notified of any ntb devices that are not already 344 344 + * associated with a client, or if ntb devices are registered later. 345 345 + * 346 346 + * Return: Zero if the client is registered, otherwise an error number. 347 347 + */ 348 348 + #define ntb_register_client(client) \ 349 349 + __ntb_register_client((client), THIS_MODULE, KBUILD_MODNAME) 79 350 80 80 - unsigned char ntb_transport_qp_num(struct ntb_transport_qp *qp); 81 81 - unsigned int ntb_transport_max_size(struct ntb_transport_qp *qp); 82 82 - struct ntb_transport_qp * 83 83 - ntb_transport_create_queue(void *data, struct pci_dev *pdev, 84 84 - const struct ntb_queue_handlers *handlers); 85 85 - void ntb_transport_free_queue(struct ntb_transport_qp *qp); 86 86 - int ntb_transport_rx_enqueue(struct ntb_transport_qp *qp, void *cb, void *data, 87 87 - unsigned int len); 88 88 - int ntb_transport_tx_enqueue(struct ntb_transport_qp *qp, void *cb, void *data, 89 89 - unsigned int len); 90 90 - void *ntb_transport_rx_remove(struct ntb_transport_qp *qp, unsigned int *len); 91 91 - void ntb_transport_link_up(struct ntb_transport_qp *qp); 92 92 - void ntb_transport_link_down(struct ntb_transport_qp *qp); 93 93 - bool ntb_transport_link_query(struct ntb_transport_qp *qp); 351 351 + int __ntb_register_client(struct ntb_client *client, struct module *mod, 352 352 + const char *mod_name); 353 353 + 354 354 + /** 355 355 + * ntb_unregister_client() - unregister a client for interest in ntb devices 356 356 + * @client: Client context. 357 357 + * 358 358 + * The client will be removed from the list of clients interested in ntb 359 359 + * devices. If any ntb devices are associated with the client, the client will 360 360 + * be notified to remove those devices. 361 361 + */ 362 362 + void ntb_unregister_client(struct ntb_client *client); 363 363 + 364 364 + #define module_ntb_client(__ntb_client) \ 365 365 + module_driver(__ntb_client, ntb_register_client, \ 366 366 + ntb_unregister_client) 367 367 + 368 368 + /** 369 369 + * ntb_register_device() - register a ntb device 370 370 + * @ntb: NTB device context. 371 371 + * 372 372 + * The device will be added to the list of ntb devices. If any clients are 373 373 + * interested in ntb devices, each client will be notified of the ntb device, 374 374 + * until at most one client accepts the device. 375 375 + * 376 376 + * Return: Zero if the device is registered, otherwise an error number. 377 377 + */ 378 378 + int ntb_register_device(struct ntb_dev *ntb); 379 379 + 380 380 + /** 381 381 + * ntb_register_device() - unregister a ntb device 382 382 + * @ntb: NTB device context. 383 383 + * 384 384 + * The device will be removed from the list of ntb devices. If the ntb device 385 385 + * is associated with a client, the client will be notified to remove the 386 386 + * device. 387 387 + */ 388 388 + void ntb_unregister_device(struct ntb_dev *ntb); 389 389 + 390 390 + /** 391 391 + * ntb_set_ctx() - associate a driver context with an ntb device 392 392 + * @ntb: NTB device context. 393 393 + * @ctx: Driver context. 394 394 + * @ctx_ops: Driver context operations. 395 395 + * 396 396 + * Associate a driver context and operations with a ntb device. The context is 397 397 + * provided by the client driver, and the driver may associate a different 398 398 + * context with each ntb device. 399 399 + * 400 400 + * Return: Zero if the context is associated, otherwise an error number. 401 401 + */ 402 402 + int ntb_set_ctx(struct ntb_dev *ntb, void *ctx, 403 403 + const struct ntb_ctx_ops *ctx_ops); 404 404 + 405 405 + /** 406 406 + * ntb_clear_ctx() - disassociate any driver context from an ntb device 407 407 + * @ntb: NTB device context. 408 408 + * 409 409 + * Clear any association that may exist between a driver context and the ntb 410 410 + * device. 411 411 + */ 412 412 + void ntb_clear_ctx(struct ntb_dev *ntb); 413 413 + 414 414 + /** 415 415 + * ntb_link_event() - notify driver context of a change in link status 416 416 + * @ntb: NTB device context. 417 417 + * 418 418 + * Notify the driver context that the link status may have changed. The driver 419 419 + * should call ntb_link_is_up() to get the current status. 420 420 + */ 421 421 + void ntb_link_event(struct ntb_dev *ntb); 422 422 + 423 423 + /** 424 424 + * ntb_db_event() - notify driver context of a doorbell event 425 425 + * @ntb: NTB device context. 426 426 + * @vector: Interrupt vector number. 427 427 + * 428 428 + * Notify the driver context of a doorbell event. If hardware supports 429 429 + * multiple interrupt vectors for doorbells, the vector number indicates which 430 430 + * vector received the interrupt. The vector number is relative to the first 431 431 + * vector used for doorbells, starting at zero, and must be less than 432 432 + ** ntb_db_vector_count(). The driver may call ntb_db_read() to check which 433 433 + * doorbell bits need service, and ntb_db_vector_mask() to determine which of 434 434 + * those bits are associated with the vector number. 435 435 + */ 436 436 + void ntb_db_event(struct ntb_dev *ntb, int vector); 437 437 + 438 438 + /** 439 439 + * ntb_mw_count() - get the number of memory windows 440 440 + * @ntb: NTB device context. 441 441 + * 442 442 + * Hardware and topology may support a different number of memory windows. 443 443 + * 444 444 + * Return: the number of memory windows. 445 445 + */ 446 446 + static inline int ntb_mw_count(struct ntb_dev *ntb) 447 447 + { 448 448 + return ntb->ops->mw_count(ntb); 449 449 + } 450 450 + 451 451 + /** 452 452 + * ntb_mw_get_range() - get the range of a memory window 453 453 + * @ntb: NTB device context. 454 454 + * @idx: Memory window number. 455 455 + * @base: OUT - the base address for mapping the memory window 456 456 + * @size: OUT - the size for mapping the memory window 457 457 + * @align: OUT - the base alignment for translating the memory window 458 458 + * @align_size: OUT - the size alignment for translating the memory window 459 459 + * 460 460 + * Get the range of a memory window. NULL may be given for any output 461 461 + * parameter if the value is not needed. The base and size may be used for 462 462 + * mapping the memory window, to access the peer memory. The alignment and 463 463 + * size may be used for translating the memory window, for the peer to access 464 464 + * memory on the local system. 465 465 + * 466 466 + * Return: Zero on success, otherwise an error number. 467 467 + */ 468 468 + static inline int ntb_mw_get_range(struct ntb_dev *ntb, int idx, 469 469 + phys_addr_t *base, resource_size_t *size, 470 470 + resource_size_t *align, resource_size_t *align_size) 471 471 + { 472 472 + return ntb->ops->mw_get_range(ntb, idx, base, size, 473 473 + align, align_size); 474 474 + } 475 475 + 476 476 + /** 477 477 + * ntb_mw_set_trans() - set the translation of a memory window 478 478 + * @ntb: NTB device context. 479 479 + * @idx: Memory window number. 480 480 + * @addr: The dma address local memory to expose to the peer. 481 481 + * @size: The size of the local memory to expose to the peer. 482 482 + * 483 483 + * Set the translation of a memory window. The peer may access local memory 484 484 + * through the window starting at the address, up to the size. The address 485 485 + * must be aligned to the alignment specified by ntb_mw_get_range(). The size 486 486 + * must be aligned to the size alignment specified by ntb_mw_get_range(). 487 487 + * 488 488 + * Return: Zero on success, otherwise an error number. 489 489 + */ 490 490 + static inline int ntb_mw_set_trans(struct ntb_dev *ntb, int idx, 491 491 + dma_addr_t addr, resource_size_t size) 492 492 + { 493 493 + return ntb->ops->mw_set_trans(ntb, idx, addr, size); 494 494 + } 495 495 + 496 496 + /** 497 497 + * ntb_mw_clear_trans() - clear the translation of a memory window 498 498 + * @ntb: NTB device context. 499 499 + * @idx: Memory window number. 500 500 + * 501 501 + * Clear the translation of a memory window. The peer may no longer access 502 502 + * local memory through the window. 503 503 + * 504 504 + * Return: Zero on success, otherwise an error number. 505 505 + */ 506 506 + static inline int ntb_mw_clear_trans(struct ntb_dev *ntb, int idx) 507 507 + { 508 508 + if (!ntb->ops->mw_clear_trans) 509 509 + return ntb->ops->mw_set_trans(ntb, idx, 0, 0); 510 510 + 511 511 + return ntb->ops->mw_clear_trans(ntb, idx); 512 512 + } 513 513 + 514 514 + /** 515 515 + * ntb_link_is_up() - get the current ntb link state 516 516 + * @ntb: NTB device context. 517 517 + * @speed: OUT - The link speed expressed as PCIe generation number. 518 518 + * @width: OUT - The link width expressed as the number of PCIe lanes. 519 519 + * 520 520 + * Set the translation of a memory window. The peer may access local memory 521 521 + * through the window starting at the address, up to the size. The address 522 522 + * must be aligned to the alignment specified by ntb_mw_get_range(). The size 523 523 + * must be aligned to the size alignment specified by ntb_mw_get_range(). 524 524 + * 525 525 + * Return: One if the link is up, zero if the link is down, otherwise a 526 526 + * negative value indicating the error number. 527 527 + */ 528 528 + static inline int ntb_link_is_up(struct ntb_dev *ntb, 529 529 + enum ntb_speed *speed, enum ntb_width *width) 530 530 + { 531 531 + return ntb->ops->link_is_up(ntb, speed, width); 532 532 + } 533 533 + 534 534 + /** 535 535 + * ntb_link_enable() - enable the link on the secondary side of the ntb 536 536 + * @ntb: NTB device context. 537 537 + * @max_speed: The maximum link speed expressed as PCIe generation number. 538 538 + * @max_width: The maximum link width expressed as the number of PCIe lanes. 539 539 + * 540 540 + * Enable the link on the secondary side of the ntb. This can only be done 541 541 + * from the primary side of the ntb in primary or b2b topology. The ntb device 542 542 + * should train the link to its maximum speed and width, or the requested speed 543 543 + * and width, whichever is smaller, if supported. 544 544 + * 545 545 + * Return: Zero on success, otherwise an error number. 546 546 + */ 547 547 + static inline int ntb_link_enable(struct ntb_dev *ntb, 548 548 + enum ntb_speed max_speed, 549 549 + enum ntb_width max_width) 550 550 + { 551 551 + return ntb->ops->link_enable(ntb, max_speed, max_width); 552 552 + } 553 553 + 554 554 + /** 555 555 + * ntb_link_disable() - disable the link on the secondary side of the ntb 556 556 + * @ntb: NTB device context. 557 557 + * 558 558 + * Disable the link on the secondary side of the ntb. This can only be 559 559 + * done from the primary side of the ntb in primary or b2b topology. The ntb 560 560 + * device should disable the link. Returning from this call must indicate that 561 561 + * a barrier has passed, though with no more writes may pass in either 562 562 + * direction across the link, except if this call returns an error number. 563 563 + * 564 564 + * Return: Zero on success, otherwise an error number. 565 565 + */ 566 566 + static inline int ntb_link_disable(struct ntb_dev *ntb) 567 567 + { 568 568 + return ntb->ops->link_disable(ntb); 569 569 + } 570 570 + 571 571 + /** 572 572 + * ntb_db_is_unsafe() - check if it is safe to use hardware doorbell 573 573 + * @ntb: NTB device context. 574 574 + * 575 575 + * It is possible for some ntb hardware to be affected by errata. Hardware 576 576 + * drivers can advise clients to avoid using doorbells. Clients may ignore 577 577 + * this advice, though caution is recommended. 578 578 + * 579 579 + * Return: Zero if it is safe to use doorbells, or One if it is not safe. 580 580 + */ 581 581 + static inline int ntb_db_is_unsafe(struct ntb_dev *ntb) 582 582 + { 583 583 + if (!ntb->ops->db_is_unsafe) 584 584 + return 0; 585 585 + 586 586 + return ntb->ops->db_is_unsafe(ntb); 587 587 + } 588 588 + 589 589 + /** 590 590 + * ntb_db_valid_mask() - get a mask of doorbell bits supported by the ntb 591 591 + * @ntb: NTB device context. 592 592 + * 593 593 + * Hardware may support different number or arrangement of doorbell bits. 594 594 + * 595 595 + * Return: A mask of doorbell bits supported by the ntb. 596 596 + */ 597 597 + static inline u64 ntb_db_valid_mask(struct ntb_dev *ntb) 598 598 + { 599 599 + return ntb->ops->db_valid_mask(ntb); 600 600 + } 601 601 + 602 602 + /** 603 603 + * ntb_db_vector_count() - get the number of doorbell interrupt vectors 604 604 + * @ntb: NTB device context. 605 605 + * 606 606 + * Hardware may support different number of interrupt vectors. 607 607 + * 608 608 + * Return: The number of doorbell interrupt vectors. 609 609 + */ 610 610 + static inline int ntb_db_vector_count(struct ntb_dev *ntb) 611 611 + { 612 612 + if (!ntb->ops->db_vector_count) 613 613 + return 1; 614 614 + 615 615 + return ntb->ops->db_vector_count(ntb); 616 616 + } 617 617 + 618 618 + /** 619 619 + * ntb_db_vector_mask() - get a mask of doorbell bits serviced by a vector 620 620 + * @ntb: NTB device context. 621 621 + * @vector: Doorbell vector number. 622 622 + * 623 623 + * Each interrupt vector may have a different number or arrangement of bits. 624 624 + * 625 625 + * Return: A mask of doorbell bits serviced by a vector. 626 626 + */ 627 627 + static inline u64 ntb_db_vector_mask(struct ntb_dev *ntb, int vector) 628 628 + { 629 629 + if (!ntb->ops->db_vector_mask) 630 630 + return ntb_db_valid_mask(ntb); 631 631 + 632 632 + return ntb->ops->db_vector_mask(ntb, vector); 633 633 + } 634 634 + 635 635 + /** 636 636 + * ntb_db_read() - read the local doorbell register 637 637 + * @ntb: NTB device context. 638 638 + * 639 639 + * Read the local doorbell register, and return the bits that are set. 640 640 + * 641 641 + * Return: The bits currently set in the local doorbell register. 642 642 + */ 643 643 + static inline u64 ntb_db_read(struct ntb_dev *ntb) 644 644 + { 645 645 + return ntb->ops->db_read(ntb); 646 646 + } 647 647 + 648 648 + /** 649 649 + * ntb_db_set() - set bits in the local doorbell register 650 650 + * @ntb: NTB device context. 651 651 + * @db_bits: Doorbell bits to set. 652 652 + * 653 653 + * Set bits in the local doorbell register, which may generate a local doorbell 654 654 + * interrupt. Bits that were already set must remain set. 655 655 + * 656 656 + * This is unusual, and hardware may not support it. 657 657 + * 658 658 + * Return: Zero on success, otherwise an error number. 659 659 + */ 660 660 + static inline int ntb_db_set(struct ntb_dev *ntb, u64 db_bits) 661 661 + { 662 662 + if (!ntb->ops->db_set) 663 663 + return -EINVAL; 664 664 + 665 665 + return ntb->ops->db_set(ntb, db_bits); 666 666 + } 667 667 + 668 668 + /** 669 669 + * ntb_db_clear() - clear bits in the local doorbell register 670 670 + * @ntb: NTB device context. 671 671 + * @db_bits: Doorbell bits to clear. 672 672 + * 673 673 + * Clear bits in the local doorbell register, arming the bits for the next 674 674 + * doorbell. 675 675 + * 676 676 + * Return: Zero on success, otherwise an error number. 677 677 + */ 678 678 + static inline int ntb_db_clear(struct ntb_dev *ntb, u64 db_bits) 679 679 + { 680 680 + return ntb->ops->db_clear(ntb, db_bits); 681 681 + } 682 682 + 683 683 + /** 684 684 + * ntb_db_read_mask() - read the local doorbell mask 685 685 + * @ntb: NTB device context. 686 686 + * 687 687 + * Read the local doorbell mask register, and return the bits that are set. 688 688 + * 689 689 + * This is unusual, though hardware is likely to support it. 690 690 + * 691 691 + * Return: The bits currently set in the local doorbell mask register. 692 692 + */ 693 693 + static inline u64 ntb_db_read_mask(struct ntb_dev *ntb) 694 694 + { 695 695 + if (!ntb->ops->db_read_mask) 696 696 + return 0; 697 697 + 698 698 + return ntb->ops->db_read_mask(ntb); 699 699 + } 700 700 + 701 701 + /** 702 702 + * ntb_db_set_mask() - set bits in the local doorbell mask 703 703 + * @ntb: NTB device context. 704 704 + * @db_bits: Doorbell mask bits to set. 705 705 + * 706 706 + * Set bits in the local doorbell mask register, preventing doorbell interrupts 707 707 + * from being generated for those doorbell bits. Bits that were already set 708 708 + * must remain set. 709 709 + * 710 710 + * Return: Zero on success, otherwise an error number. 711 711 + */ 712 712 + static inline int ntb_db_set_mask(struct ntb_dev *ntb, u64 db_bits) 713 713 + { 714 714 + return ntb->ops->db_set_mask(ntb, db_bits); 715 715 + } 716 716 + 717 717 + /** 718 718 + * ntb_db_clear_mask() - clear bits in the local doorbell mask 719 719 + * @ntb: NTB device context. 720 720 + * @db_bits: Doorbell bits to clear. 721 721 + * 722 722 + * Clear bits in the local doorbell mask register, allowing doorbell interrupts 723 723 + * from being generated for those doorbell bits. If a doorbell bit is already 724 724 + * set at the time the mask is cleared, and the corresponding mask bit is 725 725 + * changed from set to clear, then the ntb driver must ensure that 726 726 + * ntb_db_event() is called. If the hardware does not generate the interrupt 727 727 + * on clearing the mask bit, then the driver must call ntb_db_event() anyway. 728 728 + * 729 729 + * Return: Zero on success, otherwise an error number. 730 730 + */ 731 731 + static inline int ntb_db_clear_mask(struct ntb_dev *ntb, u64 db_bits) 732 732 + { 733 733 + return ntb->ops->db_clear_mask(ntb, db_bits); 734 734 + } 735 735 + 736 736 + /** 737 737 + * ntb_peer_db_addr() - address and size of the peer doorbell register 738 738 + * @ntb: NTB device context. 739 739 + * @db_addr: OUT - The address of the peer doorbell register. 740 740 + * @db_size: OUT - The number of bytes to write the peer doorbell register. 741 741 + * 742 742 + * Return the address of the peer doorbell register. This may be used, for 743 743 + * example, by drivers that offload memory copy operations to a dma engine. 744 744 + * The drivers may wish to ring the peer doorbell at the completion of memory 745 745 + * copy operations. For efficiency, and to simplify ordering of operations 746 746 + * between the dma memory copies and the ringing doorbell, the driver may 747 747 + * append one additional dma memory copy with the doorbell register as the 748 748 + * destination, after the memory copy operations. 749 749 + * 750 750 + * Return: Zero on success, otherwise an error number. 751 751 + */ 752 752 + static inline int ntb_peer_db_addr(struct ntb_dev *ntb, 753 753 + phys_addr_t *db_addr, 754 754 + resource_size_t *db_size) 755 755 + { 756 756 + return ntb->ops->peer_db_addr(ntb, db_addr, db_size); 757 757 + } 758 758 + 759 759 + /** 760 760 + * ntb_peer_db_read() - read the peer doorbell register 761 761 + * @ntb: NTB device context. 762 762 + * 763 763 + * Read the peer doorbell register, and return the bits that are set. 764 764 + * 765 765 + * This is unusual, and hardware may not support it. 766 766 + * 767 767 + * Return: The bits currently set in the peer doorbell register. 768 768 + */ 769 769 + static inline u64 ntb_peer_db_read(struct ntb_dev *ntb) 770 770 + { 771 771 + if (!ntb->ops->peer_db_read) 772 772 + return 0; 773 773 + 774 774 + return ntb->ops->peer_db_read(ntb); 775 775 + } 776 776 + 777 777 + /** 778 778 + * ntb_peer_db_set() - set bits in the peer doorbell register 779 779 + * @ntb: NTB device context. 780 780 + * @db_bits: Doorbell bits to set. 781 781 + * 782 782 + * Set bits in the peer doorbell register, which may generate a peer doorbell 783 783 + * interrupt. Bits that were already set must remain set. 784 784 + * 785 785 + * Return: Zero on success, otherwise an error number. 786 786 + */ 787 787 + static inline int ntb_peer_db_set(struct ntb_dev *ntb, u64 db_bits) 788 788 + { 789 789 + return ntb->ops->peer_db_set(ntb, db_bits); 790 790 + } 791 791 + 792 792 + /** 793 793 + * ntb_peer_db_clear() - clear bits in the local doorbell register 794 794 + * @ntb: NTB device context. 795 795 + * @db_bits: Doorbell bits to clear. 796 796 + * 797 797 + * Clear bits in the peer doorbell register, arming the bits for the next 798 798 + * doorbell. 799 799 + * 800 800 + * This is unusual, and hardware may not support it. 801 801 + * 802 802 + * Return: Zero on success, otherwise an error number. 803 803 + */ 804 804 + static inline int ntb_peer_db_clear(struct ntb_dev *ntb, u64 db_bits) 805 805 + { 806 806 + if (!ntb->ops->db_clear) 807 807 + return -EINVAL; 808 808 + 809 809 + return ntb->ops->peer_db_clear(ntb, db_bits); 810 810 + } 811 811 + 812 812 + /** 813 813 + * ntb_peer_db_read_mask() - read the peer doorbell mask 814 814 + * @ntb: NTB device context. 815 815 + * 816 816 + * Read the peer doorbell mask register, and return the bits that are set. 817 817 + * 818 818 + * This is unusual, and hardware may not support it. 819 819 + * 820 820 + * Return: The bits currently set in the peer doorbell mask register. 821 821 + */ 822 822 + static inline u64 ntb_peer_db_read_mask(struct ntb_dev *ntb) 823 823 + { 824 824 + if (!ntb->ops->db_read_mask) 825 825 + return 0; 826 826 + 827 827 + return ntb->ops->peer_db_read_mask(ntb); 828 828 + } 829 829 + 830 830 + /** 831 831 + * ntb_peer_db_set_mask() - set bits in the peer doorbell mask 832 832 + * @ntb: NTB device context. 833 833 + * @db_bits: Doorbell mask bits to set. 834 834 + * 835 835 + * Set bits in the peer doorbell mask register, preventing doorbell interrupts 836 836 + * from being generated for those doorbell bits. Bits that were already set 837 837 + * must remain set. 838 838 + * 839 839 + * This is unusual, and hardware may not support it. 840 840 + * 841 841 + * Return: Zero on success, otherwise an error number. 842 842 + */ 843 843 + static inline int ntb_peer_db_set_mask(struct ntb_dev *ntb, u64 db_bits) 844 844 + { 845 845 + if (!ntb->ops->db_set_mask) 846 846 + return -EINVAL; 847 847 + 848 848 + return ntb->ops->peer_db_set_mask(ntb, db_bits); 849 849 + } 850 850 + 851 851 + /** 852 852 + * ntb_peer_db_clear_mask() - clear bits in the peer doorbell mask 853 853 + * @ntb: NTB device context. 854 854 + * @db_bits: Doorbell bits to clear. 855 855 + * 856 856 + * Clear bits in the peer doorbell mask register, allowing doorbell interrupts 857 857 + * from being generated for those doorbell bits. If the hardware does not 858 858 + * generate the interrupt on clearing the mask bit, then the driver should not 859 859 + * implement this function! 860 860 + * 861 861 + * This is unusual, and hardware may not support it. 862 862 + * 863 863 + * Return: Zero on success, otherwise an error number. 864 864 + */ 865 865 + static inline int ntb_peer_db_clear_mask(struct ntb_dev *ntb, u64 db_bits) 866 866 + { 867 867 + if (!ntb->ops->db_clear_mask) 868 868 + return -EINVAL; 869 869 + 870 870 + return ntb->ops->peer_db_clear_mask(ntb, db_bits); 871 871 + } 872 872 + 873 873 + /** 874 874 + * ntb_spad_is_unsafe() - check if it is safe to use the hardware scratchpads 875 875 + * @ntb: NTB device context. 876 876 + * 877 877 + * It is possible for some ntb hardware to be affected by errata. Hardware 878 878 + * drivers can advise clients to avoid using scratchpads. Clients may ignore 879 879 + * this advice, though caution is recommended. 880 880 + * 881 881 + * Return: Zero if it is safe to use scratchpads, or One if it is not safe. 882 882 + */ 883 883 + static inline int ntb_spad_is_unsafe(struct ntb_dev *ntb) 884 884 + { 885 885 + if (!ntb->ops->spad_is_unsafe) 886 886 + return 0; 887 887 + 888 888 + return ntb->ops->spad_is_unsafe(ntb); 889 889 + } 890 890 + 891 891 + /** 892 892 + * ntb_mw_count() - get the number of scratchpads 893 893 + * @ntb: NTB device context. 894 894 + * 895 895 + * Hardware and topology may support a different number of scratchpads. 896 896 + * 897 897 + * Return: the number of scratchpads. 898 898 + */ 899 899 + static inline int ntb_spad_count(struct ntb_dev *ntb) 900 900 + { 901 901 + return ntb->ops->spad_count(ntb); 902 902 + } 903 903 + 904 904 + /** 905 905 + * ntb_spad_read() - read the local scratchpad register 906 906 + * @ntb: NTB device context. 907 907 + * @idx: Scratchpad index. 908 908 + * 909 909 + * Read the local scratchpad register, and return the value. 910 910 + * 911 911 + * Return: The value of the local scratchpad register. 912 912 + */ 913 913 + static inline u32 ntb_spad_read(struct ntb_dev *ntb, int idx) 914 914 + { 915 915 + return ntb->ops->spad_read(ntb, idx); 916 916 + } 917 917 + 918 918 + /** 919 919 + * ntb_spad_write() - write the local scratchpad register 920 920 + * @ntb: NTB device context. 921 921 + * @idx: Scratchpad index. 922 922 + * @val: Scratchpad value. 923 923 + * 924 924 + * Write the value to the local scratchpad register. 925 925 + * 926 926 + * Return: Zero on success, otherwise an error number. 927 927 + */ 928 928 + static inline int ntb_spad_write(struct ntb_dev *ntb, int idx, u32 val) 929 929 + { 930 930 + return ntb->ops->spad_write(ntb, idx, val); 931 931 + } 932 932 + 933 933 + /** 934 934 + * ntb_peer_spad_addr() - address of the peer scratchpad register 935 935 + * @ntb: NTB device context. 936 936 + * @idx: Scratchpad index. 937 937 + * @spad_addr: OUT - The address of the peer scratchpad register. 938 938 + * 939 939 + * Return the address of the peer doorbell register. This may be used, for 940 940 + * example, by drivers that offload memory copy operations to a dma engine. 941 941 + * 942 942 + * Return: Zero on success, otherwise an error number. 943 943 + */ 944 944 + static inline int ntb_peer_spad_addr(struct ntb_dev *ntb, int idx, 945 945 + phys_addr_t *spad_addr) 946 946 + { 947 947 + return ntb->ops->peer_spad_addr(ntb, idx, spad_addr); 948 948 + } 949 949 + 950 950 + /** 951 951 + * ntb_peer_spad_read() - read the peer scratchpad register 952 952 + * @ntb: NTB device context. 953 953 + * @idx: Scratchpad index. 954 954 + * 955 955 + * Read the peer scratchpad register, and return the value. 956 956 + * 957 957 + * Return: The value of the local scratchpad register. 958 958 + */ 959 959 + static inline u32 ntb_peer_spad_read(struct ntb_dev *ntb, int idx) 960 960 + { 961 961 + return ntb->ops->peer_spad_read(ntb, idx); 962 962 + } 963 963 + 964 964 + /** 965 965 + * ntb_peer_spad_write() - write the peer scratchpad register 966 966 + * @ntb: NTB device context. 967 967 + * @idx: Scratchpad index. 968 968 + * @val: Scratchpad value. 969 969 + * 970 970 + * Write the value to the peer scratchpad register. 971 971 + * 972 972 + * Return: Zero on success, otherwise an error number. 973 973 + */ 974 974 + static inline int ntb_peer_spad_write(struct ntb_dev *ntb, int idx, u32 val) 975 975 + { 976 976 + return ntb->ops->peer_spad_write(ntb, idx, val); 977 977 + } 978 978 + 979 979 + #endif

+85

include/linux/ntb_transport.h

reviewed

··· 1 1 + /* 2 2 + * This file is provided under a dual BSD/GPLv2 license. When using or 3 3 + * redistributing this file, you may do so under either license. 4 4 + * 5 5 + * GPL LICENSE SUMMARY 6 6 + * 7 7 + * Copyright(c) 2012 Intel Corporation. All rights reserved. 8 8 + * Copyright (C) 2015 EMC Corporation. All Rights Reserved. 9 9 + * 10 10 + * This program is free software; you can redistribute it and/or modify 11 11 + * it under the terms of version 2 of the GNU General Public License as 12 12 + * published by the Free Software Foundation. 13 13 + * 14 14 + * BSD LICENSE 15 15 + * 16 16 + * Copyright(c) 2012 Intel Corporation. All rights reserved. 17 17 + * Copyright (C) 2015 EMC Corporation. All Rights Reserved. 18 18 + * 19 19 + * Redistribution and use in source and binary forms, with or without 20 20 + * modification, are permitted provided that the following conditions 21 21 + * are met: 22 22 + * 23 23 + * * Redistributions of source code must retain the above copyright 24 24 + * notice, this list of conditions and the following disclaimer. 25 25 + * * Redistributions in binary form must reproduce the above copy 26 26 + * notice, this list of conditions and the following disclaimer in 27 27 + * the documentation and/or other materials provided with the 28 28 + * distribution. 29 29 + * * Neither the name of Intel Corporation nor the names of its 30 30 + * contributors may be used to endorse or promote products derived 31 31 + * from this software without specific prior written permission. 32 32 + * 33 33 + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 34 34 + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 35 35 + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 36 36 + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 37 37 + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 38 38 + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 39 39 + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 40 40 + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 41 41 + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 42 42 + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 43 43 + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 44 44 + * 45 45 + * PCIe NTB Transport Linux driver 46 46 + * 47 47 + * Contact Information: 48 48 + * Jon Mason <jon.mason@intel.com> 49 49 + */ 50 50 + 51 51 + struct ntb_transport_qp; 52 52 + 53 53 + struct ntb_transport_client { 54 54 + struct device_driver driver; 55 55 + int (*probe)(struct device *client_dev); 56 56 + void (*remove)(struct device *client_dev); 57 57 + }; 58 58 + 59 59 + int ntb_transport_register_client(struct ntb_transport_client *drvr); 60 60 + void ntb_transport_unregister_client(struct ntb_transport_client *drvr); 61 61 + int ntb_transport_register_client_dev(char *device_name); 62 62 + void ntb_transport_unregister_client_dev(char *device_name); 63 63 + 64 64 + struct ntb_queue_handlers { 65 65 + void (*rx_handler)(struct ntb_transport_qp *qp, void *qp_data, 66 66 + void *data, int len); 67 67 + void (*tx_handler)(struct ntb_transport_qp *qp, void *qp_data, 68 68 + void *data, int len); 69 69 + void (*event_handler)(void *data, int status); 70 70 + }; 71 71 + 72 72 + unsigned char ntb_transport_qp_num(struct ntb_transport_qp *qp); 73 73 + unsigned int ntb_transport_max_size(struct ntb_transport_qp *qp); 74 74 + struct ntb_transport_qp * 75 75 + ntb_transport_create_queue(void *data, struct device *client_dev, 76 76 + const struct ntb_queue_handlers *handlers); 77 77 + void ntb_transport_free_queue(struct ntb_transport_qp *qp); 78 78 + int ntb_transport_rx_enqueue(struct ntb_transport_qp *qp, void *cb, void *data, 79 79 + unsigned int len); 80 80 + int ntb_transport_tx_enqueue(struct ntb_transport_qp *qp, void *cb, void *data, 81 81 + unsigned int len); 82 82 + void *ntb_transport_rx_remove(struct ntb_transport_qp *qp, unsigned int *len); 83 83 + void ntb_transport_link_up(struct ntb_transport_qp *qp); 84 84 + void ntb_transport_link_down(struct ntb_transport_qp *qp); 85 85 + bool ntb_transport_link_query(struct ntb_transport_qp *qp);