x86/mm: Move flush_tlb_info back to the stack

flush_tlb_info benefits from cacheline alignment, but using
cacheline-aligned stack storage directly can grow stack usage too much on
configurations with large SMP_CACHE_BYTES values. Commit 515ab7c41306
("x86/mm: Align TLB invalidation info") attempted to align stack storage,
and commit 780e0106d468 ("x86/mm/tlb: Revert "x86/mm: Align TLB
invalidation info"") reverted it because using SMP_CACHE_BYTES led to 320
bytes of stack consumption. Commit 3db6d5a5ecaf ("x86/mm/tlb: Remove
'struct flush_tlb_info' from the stack") moved flush_tlb_info to per-CPU
storage, which avoided the stack growth problem while preserving
cacheline alignment. That was a good fit while the callers kept
preemption disabled for the whole flush operation.

However, a single per-CPU flush_tlb_info also requires all flush_tlb*()
operations to keep preemption disabled while the object is in use, so
that it cannot be overwritten by another flush on the same CPU.
flush_tlb*() may send IPIs to remote CPUs and synchronously wait for all
remote CPUs to complete their local TLB flushes. That wait can take tens
of milliseconds when interrupts are disabled on a remote CPU or when a
large number of remote CPUs are involved.

To shorten the CPU-pinned and preemption-disabled section around those
remote TLB flush waits, move flush_tlb_info back to caller-private stack
storage. The caller then does not have to stay on the same CPU until the
remote flush completes.

The type alignment is capped at 64 bytes. This keeps the alignment
benefit for stack objects without reintroducing the old large-cacheline
stack usage problem.

To evaluate the performance impact, use the following script to reproduce
the microbenchmark mentioned in commit 3db6d5a5ecaf ("x86/mm/tlb: Remove
'struct flush_tlb_info' from the stack"). The test environment is an Ice
Lake system (Intel(R) Xeon(R) Platinum 8336C) with 128 CPUs and 2 NUMA
nodes. During the test, the threads were bound to specific CPUs, and both
pti and mitigations were disabled:

    #include <stdio.h>
    #include <stdlib.h>
    #include <pthread.h>
    #include <sys/mman.h>
    #include <sys/time.h>
    #include <unistd.h>

    #define NUM_OPS 1000000
    #define NUM_THREADS 3
    #define NUM_RUNS 5
    #define PAGE_SIZE 4096

    volatile int stop_threads = 0;

    void *busy_wait_thread(void *arg) {
        while (!stop_threads) {
            __asm__ volatile ("nop");
        }
        return NULL;
    }

    long long get_usec() {
        struct timeval tv;
        gettimeofday(&tv, NULL);
        return tv.tv_sec * 1000000LL + tv.tv_usec;
    }

    int main() {
        pthread_t threads[NUM_THREADS];
        char *addr;
        int i, r;
        addr = mmap(NULL, PAGE_SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE
                | MAP_ANONYMOUS, -1, 0);

        if (addr == MAP_FAILED) {
            perror("mmap");
            exit(1);
        }

        for (i = 0; i < NUM_THREADS; i++) {
            if (pthread_create(&threads[i], NULL, busy_wait_thread, NULL))
                exit(1);
        }

        printf("Running benchmark: %d runs, %d ops each, %d background\n"
               "threads\n", NUM_RUNS, NUM_OPS, NUM_THREADS);

        for (r = 0; r < NUM_RUNS; r++) {
            long long start, end;
            start = get_usec();
            for (i = 0; i < NUM_OPS; i++) {
                addr[0] = 1;
                if (madvise(addr, PAGE_SIZE, MADV_DONTNEED)) {
                    perror("madvise");
                    exit(1);
                }
            }
            end = get_usec();
            double duration = (double)(end - start);
            double avg_lat = duration / NUM_OPS;
            printf("Run %d: Total time %.2f us, Avg latency %.4f us/op\n",
                   r + 1, duration, avg_lat);
        }
        stop_threads = 1;
        for (i = 0; i < NUM_THREADS; i++)
            pthread_join(threads[i], NULL);
        munmap(addr, PAGE_SIZE);
        return 0;
    }

                   base   on-stack-aligned  on-stack-not-aligned
                   ----       ---------      -----------
avg (usec/op)     2.5278       2.5261         2.5508
stddev            0.0007       0.0027         0.0023

The benchmark results show that the average latency difference between
the baseline (base) and the properly aligned stack variable
(on-stack-aligned) is within the standard deviation (stddev). This
indicates that the variations are caused by testing noise, and reverting
to a stack variable with proper alignment causes no performance
regression compared to the per-CPU implementation. The unaligned version
(on-stack-not-aligned) shows a minor performance drop. The
CPU-pinned/preemption-disabled section can therefore be shortened without
sacrificing performance.

With caller-private storage there is no shared per-CPU object to protect,
so remove the DEBUG_VM reentrancy counter as well.

Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Tested-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Acked-by: Nadav Amit <nadav.amit@gmail.com>
Link: https://patch.msgid.link/20260709122933.4021501-13-zhouchuyi@bytedance.com

0 files changed, 0 insertions, 0 deletions