sched/numa: Fix NUMA topology for systems with CPU-less nodes

The NUMA topology parameters (sched_numa_topology_type,
sched_domains_numa_levels, and sched_max_numa_distance, etc.)
identified by scheduler may be wrong for systems with CPU-less nodes.

For example, the ACPI SLIT of a system with CPU-less persistent
memory (Intel Optane DCPMM) nodes is as follows,

[000h 0000 4] Signature : "SLIT" [System Locality Information Table]
[004h 0004 4] Table Length : 0000042C
[008h 0008 1] Revision : 01
[009h 0009 1] Checksum : 59
[00Ah 0010 6] Oem ID : "XXXX"
[010h 0016 8] Oem Table ID : "XXXXXXX"
[018h 0024 4] Oem Revision : 00000001
[01Ch 0028 4] Asl Compiler ID : "INTL"
[020h 0032 4] Asl Compiler Revision : 20091013

[024h 0036 8] Localities : 0000000000000004
[02Ch 0044 4] Locality 0 : 0A 15 11 1C
[030h 0048 4] Locality 1 : 15 0A 1C 11
[034h 0052 4] Locality 2 : 11 1C 0A 1C
[038h 0056 4] Locality 3 : 1C 11 1C 0A

While the `numactl -H` output is as follows,

available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71
node 0 size: 64136 MB
node 0 free: 5981 MB
node 1 cpus: 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
node 1 size: 64466 MB
node 1 free: 10415 MB
node 2 cpus:
node 2 size: 253952 MB
node 2 free: 253920 MB
node 3 cpus:
node 3 size: 253952 MB
node 3 free: 253951 MB
node distances:
node 0 1 2 3
0: 10 21 17 28
1: 21 10 28 17
2: 17 28 10 28
3: 28 17 28 10

In this system, there are only 2 sockets. In each memory controller,
both DRAM and PMEM DIMMs are installed. Although the physical NUMA
topology is simple, the logical NUMA topology becomes a little
complex. Because both the distance(0, 1) and distance (1, 3) are less
than the distance (0, 3), it appears that node 1 sits between node 0
and node 3. And the whole system appears to be a glueless mesh NUMA
topology type. But it's definitely not, there is even no CPU in node 3.

This isn't a practical problem now yet. Because the PMEM nodes (node
2 and node 3 in example system) are offlined by default during system
boot. So init_numa_topology_type() called during system boot will
ignore them and set sched_numa_topology_type to NUMA_DIRECT. And
init_numa_topology_type() is only called at runtime when a CPU of a
never-onlined-before node gets plugged in. And there's no CPU in the
PMEM nodes. But it appears better to fix this to make the code more
robust.

To test the potential problem. We have used a debug patch to call
init_numa_topology_type() when the PMEM node is onlined (in
__set_migration_target_nodes()). With that, the NUMA parameters
identified by scheduler is as follows,

sched_numa_topology_type: NUMA_GLUELESS_MESH
sched_domains_numa_levels: 4
sched_max_numa_distance: 28

To fix the issue, the CPU-less nodes are ignored when the NUMA topology
parameters are identified. Because a node may become CPU-less or not
at run time because of CPU hotplug, the NUMA topology parameters need
to be re-initialized at runtime for CPU hotplug too.

With the patch, the NUMA parameters identified for the example system
above is as follows,

sched_numa_topology_type: NUMA_DIRECT
sched_domains_numa_levels: 2
sched_max_numa_distance: 21

Suggested-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: "Huang, Ying" <ying.huang@intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://lkml.kernel.org/r/20220214121553.582248-1-ying.huang@intel.com

authored by

Huang Ying and committed by

Peter Zijlstra 4 years ago 0fb3978b 1087ad4e

+140 -98

4 changed files

expand all

kernel

sched

core.c

fair.c

sched.h

topology.c

+4 -1

kernel/sched/core.c

··· 9052 9052 set_cpu_active(cpu, true); 9053 9053 9054 9054 if (sched_smp_initialized) { 9055 + sched_update_numa(cpu, true); 9055 9056 sched_domains_numa_masks_set(cpu); 9056 9057 cpuset_cpu_active(); 9057 9058 } ··· 9131 9130 if (!sched_smp_initialized) 9132 9131 return 0; 9133 9132 9133 + sched_update_numa(cpu, false); 9134 9134 ret = cpuset_cpu_inactive(cpu); 9135 9135 if (ret) { 9136 9136 balance_push_set(cpu, false); 9137 9137 set_cpu_active(cpu, true); 9138 + sched_update_numa(cpu, true); 9138 9139 return ret; 9139 9140 } 9140 9141 sched_domains_numa_masks_clear(cpu); ··· 9239 9236 9240 9237 void __init sched_init_smp(void) 9241 9238 { 9242 - sched_init_numa(); 9239 + sched_init_numa(NUMA_NO_NODE); 9243 9240 9244 9241 /* 9245 9242 * There's no userspace yet to cause hotplug operations; hence all the

+8 -7

kernel/sched/fair.c

··· 1259 1259 1260 1260 /* Handle placement on systems where not all nodes are directly connected. */ 1261 1261 static unsigned long score_nearby_nodes(struct task_struct *p, int nid, 1262 - int maxdist, bool task) 1262 + int lim_dist, bool task) 1263 1263 { 1264 1264 unsigned long score = 0; 1265 - int node; 1265 + int node, max_dist; 1266 1266 1267 1267 /* 1268 1268 * All nodes are directly connected, and the same distance ··· 1271 1271 if (sched_numa_topology_type == NUMA_DIRECT) 1272 1272 return 0; 1273 1273 1274 + /* sched_max_numa_distance may be changed in parallel. */ 1275 + max_dist = READ_ONCE(sched_max_numa_distance); 1274 1276 /* 1275 1277 * This code is called for each node, introducing N^2 complexity, 1276 1278 * which should be ok given the number of nodes rarely exceeds 8. ··· 1285 1283 * The furthest away nodes in the system are not interesting 1286 1284 * for placement; nid was already counted. 1287 1285 */ 1288 - if (dist == sched_max_numa_distance || node == nid) 1286 + if (dist >= max_dist || node == nid) 1289 1287 continue; 1290 1288 1291 1289 /* ··· 1295 1293 * "hoplimit", only nodes closer by than "hoplimit" are part 1296 1294 * of each group. Skip other nodes. 1297 1295 */ 1298 - if (sched_numa_topology_type == NUMA_BACKPLANE && 1299 - dist >= maxdist) 1296 + if (sched_numa_topology_type == NUMA_BACKPLANE && dist >= lim_dist) 1300 1297 continue; 1301 1298 1302 1299 /* Add up the faults from nearby nodes. */ ··· 1313 1312 * This seems to result in good task placement. 1314 1313 */ 1315 1314 if (sched_numa_topology_type == NUMA_GLUELESS_MESH) { 1316 - faults *= (sched_max_numa_distance - dist); 1317 - faults /= (sched_max_numa_distance - LOCAL_DISTANCE); 1315 + faults *= (max_dist - dist); 1316 + faults /= (max_dist - LOCAL_DISTANCE); 1318 1317 } 1319 1318 1320 1319 score += faults;

+4 -2

kernel/sched/sched.h

··· 1662 1662 extern enum numa_topology_type sched_numa_topology_type; 1663 1663 extern int sched_max_numa_distance; 1664 1664 extern bool find_numa_distance(int distance); 1665 - extern void sched_init_numa(void); 1665 + extern void sched_init_numa(int offline_node); 1666 + extern void sched_update_numa(int cpu, bool online); 1666 1667 extern void sched_domains_numa_masks_set(unsigned int cpu); 1667 1668 extern void sched_domains_numa_masks_clear(unsigned int cpu); 1668 1669 extern int sched_numa_find_closest(const struct cpumask *cpus, int cpu); 1669 1670 #else 1670 - static inline void sched_init_numa(void) { } 1671 + static inline void sched_init_numa(int offline_node) { } 1672 + static inline void sched_update_numa(int cpu, bool online) { } 1671 1673 static inline void sched_domains_numa_masks_set(unsigned int cpu) { } 1672 1674 static inline void sched_domains_numa_masks_clear(unsigned int cpu) { } 1673 1675 static inline int sched_numa_find_closest(const struct cpumask *cpus, int cpu)

+124 -88

kernel/sched/topology.c

··· 1492 1492 int sched_max_numa_distance; 1493 1493 static int *sched_domains_numa_distance; 1494 1494 static struct cpumask ***sched_domains_numa_masks; 1495 - 1496 - static unsigned long __read_mostly *sched_numa_onlined_nodes; 1497 1495 #endif 1498 1496 1499 1497 /* ··· 1649 1651 1650 1652 static struct sched_domain_topology_level *sched_domain_topology = 1651 1653 default_topology; 1654 + static struct sched_domain_topology_level *sched_domain_topology_saved; 1652 1655 1653 1656 #define for_each_sd_topology(tl) \ 1654 1657 for (tl = sched_domain_topology; tl->mask; tl++) ··· 1660 1661 return; 1661 1662 1662 1663 sched_domain_topology = tl; 1664 + sched_domain_topology_saved = NULL; 1663 1665 } 1664 1666 1665 1667 #ifdef CONFIG_NUMA ··· 1684 1684 1685 1685 for (i = 0; i < nr_node_ids; i++) { 1686 1686 printk(KERN_WARNING " "); 1687 - for (j = 0; j < nr_node_ids; j++) 1688 - printk(KERN_CONT "%02d ", node_distance(i,j)); 1687 + for (j = 0; j < nr_node_ids; j++) { 1688 + if (!node_state(i, N_CPU) || !node_state(j, N_CPU)) 1689 + printk(KERN_CONT "(%02d) ", node_distance(i,j)); 1690 + else 1691 + printk(KERN_CONT " %02d ", node_distance(i,j)); 1692 + } 1689 1693 printk(KERN_CONT "\n"); 1690 1694 } 1691 1695 printk(KERN_WARNING "\n"); ··· 1697 1693 1698 1694 bool find_numa_distance(int distance) 1699 1695 { 1700 - int i; 1696 + bool found = false; 1697 + int i, *distances; 1701 1698 1702 1699 if (distance == node_distance(0, 0)) 1703 1700 return true; 1704 1701 1702 + rcu_read_lock(); 1703 + distances = rcu_dereference(sched_domains_numa_distance); 1704 + if (!distances) 1705 + goto unlock; 1705 1706 for (i = 0; i < sched_domains_numa_levels; i++) { 1706 - if (sched_domains_numa_distance[i] == distance) 1707 - return true; 1707 + if (distances[i] == distance) { 1708 + found = true; 1709 + break; 1710 + } 1708 1711 } 1712 + unlock: 1713 + rcu_read_unlock(); 1709 1714 1710 - return false; 1715 + return found; 1711 1716 } 1717 + 1718 + #define for_each_cpu_node_but(n, nbut) \ 1719 + for_each_node_state(n, N_CPU) \ 1720 + if (n == nbut) \ 1721 + continue; \ 1722 + else 1712 1723 1713 1724 /* 1714 1725 * A system can have three types of NUMA topology: ··· 1744 1725 * there is an intermediary node C, which is < N hops away from both 1745 1726 * nodes A and B, the system is a glueless mesh. 1746 1727 */ 1747 - static void init_numa_topology_type(void) 1728 + static void init_numa_topology_type(int offline_node) 1748 1729 { 1749 1730 int a, b, c, n; 1750 1731 ··· 1755 1736 return; 1756 1737 } 1757 1738 1758 - for_each_online_node(a) { 1759 - for_each_online_node(b) { 1739 + for_each_cpu_node_but(a, offline_node) { 1740 + for_each_cpu_node_but(b, offline_node) { 1760 1741 /* Find two nodes furthest removed from each other. */ 1761 1742 if (node_distance(a, b) < n) 1762 1743 continue; 1763 1744 1764 1745 /* Is there an intermediary node between a and b? */ 1765 - for_each_online_node(c) { 1746 + for_each_cpu_node_but(c, offline_node) { 1766 1747 if (node_distance(a, c) < n && 1767 1748 node_distance(b, c) < n) { 1768 1749 sched_numa_topology_type = ··· 1775 1756 return; 1776 1757 } 1777 1758 } 1759 + 1760 + pr_err("Failed to find a NUMA topology type, defaulting to DIRECT\n"); 1761 + sched_numa_topology_type = NUMA_DIRECT; 1778 1762 } 1779 1763 1780 1764 1781 1765 #define NR_DISTANCE_VALUES (1 << DISTANCE_BITS) 1782 1766 1783 - void sched_init_numa(void) 1767 + void sched_init_numa(int offline_node) 1784 1768 { 1785 1769 struct sched_domain_topology_level *tl; 1786 1770 unsigned long *distance_map; 1787 1771 int nr_levels = 0; 1788 1772 int i, j; 1773 + int *distances; 1774 + struct cpumask ***masks; 1789 1775 1790 1776 /* 1791 1777 * O(nr_nodes^2) deduplicating selection sort -- in order to find the ··· 1801 1777 return; 1802 1778 1803 1779 bitmap_zero(distance_map, NR_DISTANCE_VALUES); 1804 - for (i = 0; i < nr_node_ids; i++) { 1805 - for (j = 0; j < nr_node_ids; j++) { 1780 + for_each_cpu_node_but(i, offline_node) { 1781 + for_each_cpu_node_but(j, offline_node) { 1806 1782 int distance = node_distance(i, j); 1807 1783 1808 1784 if (distance < LOCAL_DISTANCE || distance >= NR_DISTANCE_VALUES) { 1809 1785 sched_numa_warn("Invalid distance value range"); 1786 + bitmap_free(distance_map); 1810 1787 return; 1811 1788 } 1812 1789 ··· 1820 1795 */ 1821 1796 nr_levels = bitmap_weight(distance_map, NR_DISTANCE_VALUES); 1822 1797 1823 - sched_domains_numa_distance = kcalloc(nr_levels, sizeof(int), GFP_KERNEL); 1824 - if (!sched_domains_numa_distance) { 1798 + distances = kcalloc(nr_levels, sizeof(int), GFP_KERNEL); 1799 + if (!distances) { 1825 1800 bitmap_free(distance_map); 1826 1801 return; 1827 1802 } 1828 1803 1829 1804 for (i = 0, j = 0; i < nr_levels; i++, j++) { 1830 1805 j = find_next_bit(distance_map, NR_DISTANCE_VALUES, j); 1831 - sched_domains_numa_distance[i] = j; 1806 + distances[i] = j; 1832 1807 } 1808 + rcu_assign_pointer(sched_domains_numa_distance, distances); 1833 1809 1834 1810 bitmap_free(distance_map); 1835 1811 ··· 1852 1826 */ 1853 1827 sched_domains_numa_levels = 0; 1854 1828 1855 - sched_domains_numa_masks = kzalloc(sizeof(void *) * nr_levels, GFP_KERNEL); 1856 - if (!sched_domains_numa_masks) 1829 + masks = kzalloc(sizeof(void *) * nr_levels, GFP_KERNEL); 1830 + if (!masks) 1857 1831 return; 1858 1832 1859 1833 /* ··· 1861 1835 * CPUs of nodes that are that many hops away from us. 1862 1836 */ 1863 1837 for (i = 0; i < nr_levels; i++) { 1864 - sched_domains_numa_masks[i] = 1865 - kzalloc(nr_node_ids * sizeof(void *), GFP_KERNEL); 1866 - if (!sched_domains_numa_masks[i]) 1838 + masks[i] = kzalloc(nr_node_ids * sizeof(void *), GFP_KERNEL); 1839 + if (!masks[i]) 1867 1840 return; 1868 1841 1869 - for (j = 0; j < nr_node_ids; j++) { 1842 + for_each_cpu_node_but(j, offline_node) { 1870 1843 struct cpumask *mask = kzalloc(cpumask_size(), GFP_KERNEL); 1871 1844 int k; 1872 1845 1873 1846 if (!mask) 1874 1847 return; 1875 1848 1876 - sched_domains_numa_masks[i][j] = mask; 1849 + masks[i][j] = mask; 1877 1850 1878 - for_each_node(k) { 1879 - /* 1880 - * Distance information can be unreliable for 1881 - * offline nodes, defer building the node 1882 - * masks to its bringup. 1883 - * This relies on all unique distance values 1884 - * still being visible at init time. 1885 - */ 1886 - if (!node_online(j)) 1887 - continue; 1888 - 1851 + for_each_cpu_node_but(k, offline_node) { 1889 1852 if (sched_debug() && (node_distance(j, k) != node_distance(k, j))) 1890 1853 sched_numa_warn("Node-distance not symmetric"); 1891 1854 ··· 1885 1870 } 1886 1871 } 1887 1872 } 1873 + rcu_assign_pointer(sched_domains_numa_masks, masks); 1888 1874 1889 1875 /* Compute default topology size */ 1890 1876 for (i = 0; sched_domain_topology[i].mask; i++); ··· 1923 1907 }; 1924 1908 } 1925 1909 1910 + sched_domain_topology_saved = sched_domain_topology; 1926 1911 sched_domain_topology = tl; 1927 1912 1928 1913 sched_domains_numa_levels = nr_levels; 1929 - sched_max_numa_distance = sched_domains_numa_distance[nr_levels - 1]; 1914 + WRITE_ONCE(sched_max_numa_distance, sched_domains_numa_distance[nr_levels - 1]); 1930 1915 1931 - init_numa_topology_type(); 1932 - 1933 - sched_numa_onlined_nodes = bitmap_alloc(nr_node_ids, GFP_KERNEL); 1934 - if (!sched_numa_onlined_nodes) 1935 - return; 1936 - 1937 - bitmap_zero(sched_numa_onlined_nodes, nr_node_ids); 1938 - for_each_online_node(i) 1939 - bitmap_set(sched_numa_onlined_nodes, i, 1); 1916 + init_numa_topology_type(offline_node); 1940 1917 } 1941 1918 1942 - static void __sched_domains_numa_masks_set(unsigned int node) 1943 - { 1944 - int i, j; 1945 1919 1920 + static void sched_reset_numa(void) 1921 + { 1922 + int nr_levels, *distances; 1923 + struct cpumask ***masks; 1924 + 1925 + nr_levels = sched_domains_numa_levels; 1926 + sched_domains_numa_levels = 0; 1927 + sched_max_numa_distance = 0; 1928 + sched_numa_topology_type = NUMA_DIRECT; 1929 + distances = sched_domains_numa_distance; 1930 + rcu_assign_pointer(sched_domains_numa_distance, NULL); 1931 + masks = sched_domains_numa_masks; 1932 + rcu_assign_pointer(sched_domains_numa_masks, NULL); 1933 + if (distances || masks) { 1934 + int i, j; 1935 + 1936 + synchronize_rcu(); 1937 + kfree(distances); 1938 + for (i = 0; i < nr_levels && masks; i++) { 1939 + if (!masks[i]) 1940 + continue; 1941 + for_each_node(j) 1942 + kfree(masks[i][j]); 1943 + kfree(masks[i]); 1944 + } 1945 + kfree(masks); 1946 + } 1947 + if (sched_domain_topology_saved) { 1948 + kfree(sched_domain_topology); 1949 + sched_domain_topology = sched_domain_topology_saved; 1950 + sched_domain_topology_saved = NULL; 1951 + } 1952 + } 1953 + 1954 + /* 1955 + * Call with hotplug lock held 1956 + */ 1957 + void sched_update_numa(int cpu, bool online) 1958 + { 1959 + int node; 1960 + 1961 + node = cpu_to_node(cpu); 1946 1962 /* 1947 - * NUMA masks are not built for offline nodes in sched_init_numa(). 1948 - * Thus, when a CPU of a never-onlined-before node gets plugged in, 1949 - * adding that new CPU to the right NUMA masks is not sufficient: the 1950 - * masks of that CPU's node must also be updated. 1963 + * Scheduler NUMA topology is updated when the first CPU of a 1964 + * node is onlined or the last CPU of a node is offlined. 1951 1965 */ 1952 - if (test_bit(node, sched_numa_onlined_nodes)) 1966 + if (cpumask_weight(cpumask_of_node(node)) != 1) 1953 1967 return; 1954 1968 1955 - bitmap_set(sched_numa_onlined_nodes, node, 1); 1956 - 1957 - for (i = 0; i < sched_domains_numa_levels; i++) { 1958 - for (j = 0; j < nr_node_ids; j++) { 1959 - if (!node_online(j) || node == j) 1960 - continue; 1961 - 1962 - if (node_distance(j, node) > sched_domains_numa_distance[i]) 1963 - continue; 1964 - 1965 - /* Add remote nodes in our masks */ 1966 - cpumask_or(sched_domains_numa_masks[i][node], 1967 - sched_domains_numa_masks[i][node], 1968 - sched_domains_numa_masks[0][j]); 1969 - } 1970 - } 1971 - 1972 - /* 1973 - * A new node has been brought up, potentially changing the topology 1974 - * classification. 1975 - * 1976 - * Note that this is racy vs any use of sched_numa_topology_type :/ 1977 - */ 1978 - init_numa_topology_type(); 1969 + sched_reset_numa(); 1970 + sched_init_numa(online ? NUMA_NO_NODE : node); 1979 1971 } 1980 1972 1981 1973 void sched_domains_numa_masks_set(unsigned int cpu) ··· 1991 1967 int node = cpu_to_node(cpu); 1992 1968 int i, j; 1993 1969 1994 - __sched_domains_numa_masks_set(node); 1995 - 1996 1970 for (i = 0; i < sched_domains_numa_levels; i++) { 1997 1971 for (j = 0; j < nr_node_ids; j++) { 1998 - if (!node_online(j)) 1972 + if (!node_state(j, N_CPU)) 1999 1973 continue; 2000 1974 2001 1975 /* Set ourselves in the remote node's masks */ ··· 2008 1986 int i, j; 2009 1987 2010 1988 for (i = 0; i < sched_domains_numa_levels; i++) { 2011 - for (j = 0; j < nr_node_ids; j++) 2012 - cpumask_clear_cpu(cpu, sched_domains_numa_masks[i][j]); 1989 + for (j = 0; j < nr_node_ids; j++) { 1990 + if (sched_domains_numa_masks[i][j]) 1991 + cpumask_clear_cpu(cpu, sched_domains_numa_masks[i][j]); 1992 + } 2013 1993 } 2014 1994 } 2015 1995 ··· 2025 2001 */ 2026 2002 int sched_numa_find_closest(const struct cpumask *cpus, int cpu) 2027 2003 { 2028 - int i, j = cpu_to_node(cpu); 2004 + int i, j = cpu_to_node(cpu), found = nr_cpu_ids; 2005 + struct cpumask ***masks; 2029 2006 2007 + rcu_read_lock(); 2008 + masks = rcu_dereference(sched_domains_numa_masks); 2009 + if (!masks) 2010 + goto unlock; 2030 2011 for (i = 0; i < sched_domains_numa_levels; i++) { 2031 - cpu = cpumask_any_and(cpus, sched_domains_numa_masks[i][j]); 2032 - if (cpu < nr_cpu_ids) 2033 - return cpu; 2012 + if (!masks[i][j]) 2013 + break; 2014 + cpu = cpumask_any_and(cpus, masks[i][j]); 2015 + if (cpu < nr_cpu_ids) { 2016 + found = cpu; 2017 + break; 2018 + } 2034 2019 } 2035 - return nr_cpu_ids; 2020 + unlock: 2021 + rcu_read_unlock(); 2022 + 2023 + return found; 2036 2024 } 2037 2025 2038 2026 #endif /* CONFIG_NUMA */

Configure Feed

Configure Feed