Zhu, Lipeng
aee79d4e52
fs/address_space: add alignment padding for i_map and i_mmap_rwsem to mitigate a false sharing.
When running UnixBench/Shell Scripts, we observed high false sharing for
accessing i_mmap against i_mmap_rwsem.
UnixBench/Shell Scripts are typical load/execute command test scenarios,
which concurrently launch->execute->exit a lot of shell commands. A lot
of processes invoke vma_interval_tree_remove which touch "i_mmap", the
call stack:
----vma_interval_tree_remove
|----unlink_file_vma
| free_pgtables
| |----exit_mmap
| | mmput
| | |----begin_new_exec
| | | load_elf_binary
| | | bprm_execve
Meanwhile, there are a lot of processes touch 'i_mmap_rwsem' to acquire
the semaphore in order to access 'i_mmap'. In existing 'address_space'
layout, 'i_mmap' and 'i_mmap_rwsem' are in the same cacheline.
The patch places the i_mmap and i_mmap_rwsem in separate cache lines to
avoid this false sharing problem.
With this patch, based on kernel v6.4.0, on Intel Sapphire Rapids
112C/224T platform, the score improves by ~5.3%. And perf c2c tool shows
the false sharing is resolved as expected, the symbol
vma_interval_tree_remove disappeared in cache line 0 after this change.
Baseline:
=================================================
Shared Cache Line Distribution Pareto
=================================================
-------------------------------------------------------------
0 3729 5791 0 0 0xff19b3818445c740
-------------------------------------------------------------
3.27% 3.02% 0.00% 0.00% 0x18 0 1 0xffffffffa194403b 604 483 389 692 203 [k] vma_interval_tree_insert [kernel.kallsyms] vma_interval_tree_insert+75 0 1
4.13% 3.63% 0.00% 0.00% 0x20 0 1 0xffffffffa19440a2 553 413 415 962 215 [k] vma_interval_tree_remove [kernel.kallsyms] vma_interval_tree_remove+18 0 1
2.04% 1.35% 0.00% 0.00% 0x28 0 1 0xffffffffa219a1d6 1210 855 460 1229 222 [k] rwsem_down_write_slowpath [kernel.kallsyms] rwsem_down_write_slowpath+678 0 1
0.62% 1.85% 0.00% 0.00% 0x28 0 1 0xffffffffa219a1bf 762 329 577 527 198 [k] rwsem_down_write_slowpath [kernel.kallsyms] rwsem_down_write_slowpath+655 0 1
0.48% 0.31% 0.00% 0.00% 0x28 0 1 0xffffffffa219a58c 1677 1476 733 1544 224 [k] down_write [kernel.kallsyms] down_write+28 0 1
0.05% 0.07% 0.00% 0.00% 0x28 0 1 0xffffffffa219a21d 1040 819 689 33 27 [k] rwsem_down_write_slowpath [kernel.kallsyms] rwsem_down_write_slowpath+749 0 1
0.00% 0.05% 0.00% 0.00% 0x28 0 1 0xffffffffa17707db 0 1005 786 1373 223 [k] up_write [kernel.kallsyms] up_write+27 0 1
0.00% 0.02% 0.00% 0.00% 0x28 0 1 0xffffffffa219a064 0 233 778 32 30 [k] rwsem_down_write_slowpath [kernel.kallsyms] rwsem_down_write_slowpath+308 0 1
33.82% 34.10% 0.00% 0.00% 0x30 0 1 0xffffffffa1770945 779 495 534 6011 224 [k] rwsem_spin_on_owner [kernel.kallsyms] rwsem_spin_on_owner+53 0 1
17.06% 15.28% 0.00% 0.00% 0x30 0 1 0xffffffffa1770915 593 438 468 2715 224 [k] rwsem_spin_on_owner [kernel.kallsyms] rwsem_spin_on_owner+5 0 1
3.54% 3.52% 0.00% 0.00% 0x30 0 1 0xffffffffa2199f84 881 601 583 1421 223 [k] rwsem_down_write_slowpath [kernel.kallsyms] rwsem_down_write_slowpath+84 0 1
With this change:
-------------------------------------------------------------
0 556 838 0 0 0xff2780d7965d2780
-------------------------------------------------------------
0.18% 0.60% 0.00% 0.00% 0x8 0 1 0xffffffffafff27b8 503 453 569 14 13 [k] do_dentry_open [kernel.kallsyms] do_dentry_open+456 0 1
0.54% 0.12% 0.00% 0.00% 0x8 0 1 0xffffffffaffc51ac 510 199 428 15 12 [k] hugepage_vma_check [kernel.kallsyms] hugepage_vma_check+252 0 1
1.80% 2.15% 0.00% 0.00% 0x18 0 1 0xffffffffb079a1d6 1778 799 343 215 136 [k] rwsem_down_write_slowpath [kernel.kallsyms] rwsem_down_write_slowpath+678 0 1
0.54% 1.31% 0.00% 0.00% 0x18 0 1 0xffffffffb079a1bf 547 296 528 91 71 [k] rwsem_down_write_slowpath [kernel.kallsyms] rwsem_down_write_slowpath+655 0 1
0.72% 0.72% 0.00% 0.00% 0x18 0 1 0xffffffffb079a58c 1479 1534 676 288 163 [k] down_write [kernel.kallsyms] down_write+28 0 1
0.00% 0.12% 0.00% 0.00% 0x18 0 1 0xffffffffafd707db 0 2381 744 282 158 [k] up_write [kernel.kallsyms] up_write+27 0 1
0.00% 0.12% 0.00% 0.00% 0x18 0 1 0xffffffffb079a064 0 239 518 6 6 [k] rwsem_down_write_slowpath [kernel.kallsyms] rwsem_down_write_slowpath+308 0 1
46.58% 47.02% 0.00% 0.00% 0x20 0 1 0xffffffffafd70945 704 403 499 1137 219 [k] rwsem_spin_on_owner [kernel.kallsyms] rwsem_spin_on_owner+53 0 1
23.92% 25.78% 0.00% 0.00% 0x20 0 1 0xffffffffafd70915 558 413 500 542 185 [k] rwsem_spin_on_owner [kernel.kallsyms] rwsem_spin_on_owner+5 0 1
v1->v2: change padding to exchange fields.
Link: https://lkml.kernel.org/r/20230716145653.20122-1-lipeng.zhu@intel.com
Signed-off-by: Lipeng Zhu <lipeng.zhu@intel.com>
Reviewed-by: Tim Chen <tim.c.chen@linux.intel.com>
Cc: Alexander Viro <viro@zeniv.linux.org.uk>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Yu Ma <yu.ma@intel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2023-08-18 10:12:38 -07:00
..
2023-05-31 19:01:32 +01:00
2023-06-16 16:46:30 +02:00
2023-06-22 09:43:40 +02:00
2023-06-16 12:27:04 -06:00
2023-06-26 12:47:20 -07:00
2023-05-24 12:20:45 +05:30
2023-07-04 19:42:27 +01:00
2023-05-31 20:27:24 +01:00
2023-05-30 18:58:43 -05:00
2023-06-29 10:11:10 -07:00
2023-05-20 19:20:30 +02:00
2023-07-03 12:46:47 -07:00
2023-06-26 17:07:53 -07:00
2023-06-19 15:06:07 -04:00
2023-06-05 09:56:36 +01:00
2023-07-03 10:55:04 -07:00
2023-06-29 21:01:17 -07:00
2023-06-28 16:43:10 -07:00
2023-06-12 08:04:04 -06:00
2023-04-22 01:39:40 +02:00
2023-06-22 19:41:01 -07:00
2023-06-27 10:47:08 -04:00
2023-06-28 10:59:38 -07:00
2023-05-19 23:23:52 +05:30
2023-07-06 09:48:31 -07:00
2023-05-08 15:29:12 +02:00
2023-06-28 13:32:47 -07:00
2023-04-05 20:58:32 -07:00
2023-07-04 13:50:38 -07:00
2023-06-29 15:28:33 -07:00
2023-06-21 16:31:44 +05:30
2023-06-29 10:11:10 -07:00
2023-07-01 14:38:25 -07:00
2023-05-30 11:20:02 +02:00
2023-06-22 08:59:56 +02:00
2023-07-03 13:23:10 -07:00
2023-06-26 17:11:53 -07:00
2023-07-06 22:25:06 -07:00
2023-06-12 12:17:22 -05:00
2023-03-24 19:19:19 -04:00
2023-06-21 18:44:56 +02:00
2023-04-16 14:18:10 +02:00
2023-04-06 14:24:04 +02:00
2023-07-01 07:04:29 -04:00
2023-05-26 13:43:52 +02:00
2023-04-08 10:51:23 +01:00
2023-03-24 19:19:19 -04:00
2023-05-22 17:05:20 -04:00
2023-06-05 09:57:21 +02:00
2023-05-17 11:34:55 -04:00
2023-05-17 11:34:55 -04:00
2023-06-16 10:08:09 -06:00
2023-06-22 13:57:41 -07:00
2023-06-30 11:57:07 -07:00
2023-07-05 16:36:12 -06:00
2023-07-20 13:29:25 -06:00
2023-06-26 12:47:20 -07:00
2023-06-14 20:24:03 -06:00
2023-07-08 09:29:31 -07:00
2023-03-25 19:52:52 -07:00
2023-03-25 19:52:51 -07:00
2023-04-22 08:21:05 -07:00
2023-06-13 15:15:08 -07:00
2023-05-23 23:31:42 +02:00
2023-05-31 13:06:57 +02:00
2023-06-09 10:38:43 +01:00
2023-06-12 08:04:04 -06:00
2023-05-19 22:44:14 -07:00
2023-08-18 10:12:31 -07:00
2023-06-19 16:19:20 -07:00
2023-04-14 10:13:39 +01:00
2023-06-12 08:04:05 -06:00
2023-06-28 10:28:11 -07:00
2023-06-26 11:14:18 +02:00
2023-06-26 08:55:22 -07:00
2023-06-19 16:19:28 -07:00
2023-06-22 17:10:09 +02:00
2023-07-04 13:50:38 -07:00
2023-06-26 11:14:18 +02:00
2023-05-17 09:36:46 -07:00
2023-06-05 09:57:20 +02:00
2023-06-05 09:57:20 +02:00
2023-06-15 08:50:23 +01:00
2023-05-25 09:26:19 +02:00
2023-03-24 16:04:21 -07:00
2023-06-26 13:59:56 -07:00
2023-06-15 18:15:25 +02:00
2023-06-26 13:59:56 -07:00
2023-06-05 09:57:20 +02:00
2023-05-08 13:22:33 -10:00
2023-04-17 18:01:23 +02:00
2023-06-23 16:15:36 +08:00
2023-06-26 07:54:23 -06:00
2023-06-09 17:44:16 -07:00
2023-04-18 16:39:34 -07:00
2023-05-29 23:23:04 +09:00
2023-06-12 08:04:05 -06:00
2023-07-04 13:50:38 -07:00
2023-05-09 11:06:45 +02:00
2023-06-16 18:24:13 -04:00
2023-07-03 13:47:06 +02:00
2023-06-19 16:19:23 -07:00
2023-06-19 16:19:20 -07:00
2023-03-28 13:39:02 -07:00
2023-06-05 09:36:38 +02:00
2023-06-30 21:35:52 -07:00
2023-05-19 08:58:11 -06:00
2023-04-13 22:22:10 -07:00
2023-04-18 20:48:59 -07:00
2023-06-15 14:55:15 +02:00
2023-06-22 21:17:10 +09:00
2023-06-22 21:21:06 +09:00
2023-05-22 18:08:37 +02:00
2023-06-30 16:38:27 -07:00
2023-06-09 16:25:23 -07:00
2023-06-15 10:45:17 +02:00
2023-06-26 11:14:18 +02:00
2023-06-06 17:05:21 +02:00
2023-07-04 11:02:34 -07:00
2023-05-16 14:15:45 -07:00
2023-06-06 21:39:55 +09:00
2023-06-19 16:19:05 -07:00
2023-08-18 10:12:38 -07:00
2023-04-26 08:57:41 -07:00
2023-06-19 18:18:04 +02:00
2023-06-14 10:41:07 -07:00
2023-07-28 20:01:36 -04:00
2023-06-06 11:42:51 -07:00
2023-04-01 20:27:55 -06:00
2023-03-28 16:20:16 -07:00
2023-06-09 16:25:24 -07:00
2023-06-13 16:49:23 +02:00
2023-06-19 09:02:51 +02:00
2023-04-08 13:45:38 -07:00
2023-05-26 20:45:39 +02:00
2023-08-18 10:12:29 -07:00
2023-04-04 13:59:10 +02:00
2023-04-05 19:42:56 -07:00
2023-08-18 10:12:26 -07:00
2023-06-08 06:41:17 -07:00
2023-06-09 17:30:47 +02:00
2023-05-01 17:50:18 -07:00
2023-06-08 10:11:17 -04:00
2023-06-20 20:18:39 -07:00
2023-06-22 19:52:09 +02:00
2023-04-21 08:25:49 +01:00
2023-05-19 22:46:37 -07:00
2023-05-10 10:25:55 +01:00
2023-06-28 12:47:30 -07:00
2023-05-24 18:46:20 +02:00
2023-05-18 19:02:23 +03:00
2023-04-05 11:29:59 +03:00
2023-05-30 22:04:46 +03:00
2023-05-16 08:04:55 -06:00
2023-05-25 08:54:06 -06:00
2023-03-28 16:20:16 -07:00
2023-05-18 11:56:18 -07:00
2023-04-21 08:54:47 -07:00
2023-05-22 17:38:45 +02:00
2023-03-31 13:43:32 -03:00
2023-06-05 15:35:27 +02:00
2023-08-18 10:12:35 -07:00
2023-06-16 12:22:35 +01:00
2023-05-24 11:39:44 +02:00
2023-06-26 11:14:18 +02:00
2023-06-26 07:47:11 +02:00
2023-07-08 10:07:14 +01:00
2023-07-10 23:09:21 -04:00
2023-06-05 09:57:20 +02:00
2023-06-08 12:27:20 -07:00
2023-06-09 16:25:19 -07:00
2023-06-09 17:44:17 -07:00
2023-07-03 09:27:12 +01:00
2023-04-08 13:45:38 -07:00
2023-04-08 13:45:38 -07:00
2023-06-08 15:42:02 -07:00
2023-06-29 15:06:32 +01:00
2023-04-18 16:30:09 -07:00
2023-03-24 15:35:51 +01:00
2023-08-18 10:12:10 -07:00
2023-06-16 17:08:01 +02:00
2023-07-01 07:20:04 -04:00
2023-03-27 10:13:28 -04:00
2023-03-23 16:10:59 -07:00
2023-07-03 11:26:05 -07:00
2023-06-30 11:57:07 -07:00
2023-04-27 13:45:31 -07:00
2023-05-18 11:40:32 -07:00
2023-06-27 14:14:30 -07:00
2023-06-27 14:14:30 -07:00
2023-05-18 13:11:09 -04:00
2023-04-24 11:21:50 -07:00
2023-04-18 11:48:55 -05:00
2023-08-18 10:12:21 -07:00
2023-06-05 21:11:06 +02:00
2023-06-09 17:44:22 -07:00
2023-05-29 21:46:53 -07:00
2023-08-18 10:12:07 -07:00
2023-08-18 10:12:15 -07:00
2023-04-18 16:30:11 -07:00
2023-06-19 16:19:04 -07:00
2023-08-18 10:12:11 -07:00
2023-06-19 16:19:12 -07:00
2023-08-18 10:12:16 -07:00
2023-08-18 10:12:36 -07:00
2023-08-18 10:12:24 -07:00
2023-08-18 10:12:35 -07:00
2023-08-18 10:12:25 -07:00
2023-06-09 16:25:31 -07:00
2023-08-18 10:12:12 -07:00
2023-07-03 12:56:23 -07:00
2023-04-13 17:15:50 -07:00
2023-05-26 15:10:18 -07:00
2023-06-28 10:59:38 -07:00
2023-06-20 20:17:10 -07:00
2023-06-20 20:18:39 -07:00
2023-05-24 18:08:49 +02:00
2023-06-26 11:14:18 +02:00
2023-04-24 00:09:20 -05:00
2023-06-18 11:16:00 +01:00
2023-04-03 10:04:27 +01:00
2023-06-24 15:50:13 -07:00
2023-06-24 14:52:28 -07:00
2023-06-26 13:27:51 +02:00
2023-06-08 13:42:33 +02:00
2023-06-12 11:32:44 +01:00
2023-06-19 15:08:12 -04:00
2023-04-11 16:13:21 -04:00
2023-04-11 13:08:26 -04:00
2023-04-11 13:08:26 -04:00
2023-06-19 12:10:48 -04:00
2023-04-26 09:05:00 -04:00
2023-07-25 00:30:02 -05:00
2023-06-19 16:25:29 -07:00
2023-06-22 13:57:41 -07:00
2023-06-07 13:42:02 -07:00
2023-05-22 13:51:56 +02:00
2023-06-12 10:36:55 -07:00
2023-07-10 09:12:31 -07:00
2023-04-05 19:41:11 +02:00
2023-04-05 19:41:13 +02:00
2023-03-23 23:18:58 +01:00
2023-03-23 23:18:58 +01:00
2023-04-13 17:46:35 -05:00
2023-04-27 12:07:50 -07:00
2023-04-13 17:46:34 -05:00
2023-04-27 12:07:50 -07:00
2023-05-18 11:56:19 -07:00
2023-05-26 13:52:19 -07:00
2023-04-05 19:42:54 -07:00
2023-08-18 10:12:25 -07:00
2023-08-18 10:12:29 -07:00
2023-05-29 16:14:28 +01:00
2023-06-19 16:19:00 -07:00
2023-04-05 19:42:46 -07:00
2023-08-18 10:12:13 -07:00
2023-06-23 16:59:30 -07:00
2023-06-09 17:44:15 -07:00
2023-07-03 12:46:47 -07:00
2023-06-30 15:06:45 -07:00
2023-04-18 10:36:58 -07:00
2023-06-23 15:05:28 -05:00
2023-06-23 15:05:28 -05:00
2023-05-31 12:26:53 -05:00
2023-06-08 19:19:50 -07:00
2023-05-28 20:45:46 +02:00
2023-06-05 09:36:39 +02:00
2023-07-04 13:50:38 -07:00
2023-07-01 08:58:41 -07:00
2023-08-18 10:12:28 -07:00
2023-06-02 23:20:31 -07:00
2023-06-27 09:33:54 -07:00
2023-06-30 09:04:01 -07:00
2023-05-17 09:16:59 +02:00
2023-06-07 14:26:09 -06:00
2023-07-24 09:51:51 +02:00
2023-06-22 21:21:06 +09:00
2023-04-21 15:34:33 +02:00
2023-03-29 22:38:56 +02:00
2023-04-11 21:55:35 -04:00
2023-06-26 11:14:18 +02:00
2023-05-17 09:24:49 +02:00
2023-04-20 22:55:35 -04:00
2023-06-15 13:34:30 +02:00
2023-05-17 09:15:40 -05:00
2023-07-10 09:52:30 +02:00
2023-07-10 09:52:30 +02:00
2023-04-20 18:20:05 +08:00
2023-06-20 09:02:33 +01:00
2023-04-13 14:27:53 +02:00
2023-03-24 19:19:19 -04:00
2023-06-19 16:19:04 -07:00
2023-04-18 16:39:33 -07:00
2023-06-05 21:11:03 +02:00
2023-07-04 13:50:38 -07:00
2023-03-28 10:39:29 +02:00
2023-05-23 12:55:12 -07:00
2023-06-05 15:28:42 -07:00
2023-05-12 10:28:01 +09:00
2023-07-12 12:01:16 -07:00
2023-06-05 10:55:20 -06:00
2023-04-13 16:04:28 -07:00
2023-06-26 11:14:18 +02:00
2023-06-23 16:59:28 -07:00
2023-06-28 10:28:11 -07:00
2023-06-06 14:04:01 +01:00
2023-04-21 08:19:30 +01:00
2023-05-18 13:11:09 -04:00
2023-04-19 14:07:13 -06:00
2023-04-18 11:15:24 -07:00
2023-04-25 21:03:14 -04:00
2023-06-05 21:11:03 +02:00
2023-04-23 21:51:25 -07:00
2023-06-21 17:55:53 +02:00
2023-05-31 10:46:59 +01:00
2023-05-18 19:52:31 -07:00
2023-06-11 18:22:47 +02:00
2023-04-18 16:29:55 -07:00
2023-05-17 15:24:33 -07:00
2023-06-10 00:11:41 -07:00
2023-05-23 16:10:11 +02:00
2023-07-04 13:50:38 -07:00
2023-06-05 09:36:39 +02:00
2023-04-28 15:03:43 -07:00
2023-04-03 10:08:43 +01:00
2023-06-24 15:50:13 -07:00
2023-06-26 11:14:18 +02:00
2023-06-28 16:43:10 -07:00
2023-07-04 13:50:38 -07:00
2023-04-04 08:35:34 -07:00
2023-04-05 13:50:37 +00:00
2023-04-14 17:31:23 +02:00
2023-06-20 20:44:38 -07:00
2023-06-12 20:14:22 +03:00
2023-06-12 20:14:21 +03:00
2023-06-05 15:31:12 -07:00
2023-06-28 10:28:11 -07:00
2023-06-23 16:59:29 -07:00
2023-08-18 10:12:16 -07:00
2023-06-07 15:06:34 +02:00
2023-04-16 14:23:07 +02:00
2023-07-06 10:06:04 -07:00
2023-06-18 02:32:54 -07:00
2023-07-20 12:34:18 -07:00
2023-05-16 12:11:00 +02:00
2023-07-24 09:51:31 +02:00
2023-06-09 17:44:16 -07:00
2023-06-16 09:53:27 +03:00
2023-04-05 13:47:43 +00:00
2023-06-09 17:44:17 -07:00
2023-05-16 02:43:12 +03:00
2023-06-06 21:39:55 +09:00
2023-06-06 21:39:55 +09:00
2023-06-06 21:39:55 +09:00
2023-04-27 11:53:57 -07:00
2023-06-28 10:59:38 -07:00
2023-05-31 19:00:26 +01:00
2023-06-28 16:43:10 -07:00
2023-06-08 15:41:34 -07:00
2023-04-27 11:42:11 -07:00
2023-05-23 21:08:33 -04:00
2023-08-18 10:12:16 -07:00
2023-03-23 17:25:46 +01:00
2023-04-21 03:02:35 -04:00
2023-05-23 15:49:03 -06:00
2023-05-02 11:56:43 -07:00
2023-06-27 10:47:08 -04:00
2023-04-21 03:02:35 -04:00
2023-03-29 08:19:38 +01:00
2023-06-27 10:47:09 -04:00
2023-04-05 20:03:01 -07:00
2023-04-05 19:42:57 -07:00
2023-04-05 20:03:01 -07:00
2023-04-21 03:02:34 -04:00
2023-04-24 11:48:33 +02:00
2023-06-06 10:47:04 +02:00
2023-05-09 12:07:25 +02:00
2023-06-23 12:08:14 -07:00
2023-04-17 18:01:22 +02:00
2023-06-19 16:19:27 -07:00