rcu-tasks: Fix IRQ read lock/unlock data race

As noted by Marco Elver:

rcu_read_lock_trace()
   ....
t->trc_reader_scp = __srcu_read_lock_fast(&rcu_tasks_trace_srcu_struct);
	<interrupt>
					rcu_read_unlock_trace()
					< ... var decls only ... >
					scp = t->trc_reader_scp;

This constitutes a data race between these two accesses to
t->trc_reader_scp.  If rcu_read_lock_trace() were to tear its store,
this value would be corrupted.

This commit therefore defers the rcu_read_lock_untrace() function's
load from t->trc_reader_scp until after it has verified that this is
the outermost rcu_read_unlock_trace().  With this change, the interrupt
handler increments and decrements t->trc_reader_nesting and does not
access t->trc_reader_scp, thus avoiding the data race.

KCSAN located this issue.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

1 files changed, 6 insertions, 4 deletions

diff --git a/include/linux/rcupdate_trace.h b/include/linux/rcupdate_trace.h
index fd3ddeb6aa3b..70decf877348 100644
--- a/include/linux/rcupdate_trace.h
+++ b/include/linux/rcupdate_trace.h
@@ -126,11 +126,13 @@ static inline void rcu_read_unlock_trace(void)
 	struct srcu_ctr __percpu *scp;
 	struct task_struct *t = current;
 
-	scp = t->trc_reader_scp;
-	barrier();  // scp before nesting to protect against interrupt handler.
 	n = READ_ONCE(t->trc_reader_nesting) - 1;
-	WRITE_ONCE(t->trc_reader_nesting, n);
-	if (!n) {
+	if (n) {
+		WRITE_ONCE(t->trc_reader_nesting, n);
+	} else {
+		scp = t->trc_reader_scp; // Compiler cannot hoist load due to data raciness.
+		barrier();  // scp before nesting to protect against interrupt handler.
+		WRITE_ONCE(t->trc_reader_nesting, n);
 		if (!IS_ENABLED(CONFIG_TASKS_TRACE_RCU_NO_MB))
 			smp_mb(); // Placeholder for more selective ordering
 		__srcu_read_unlock_fast(&rcu_tasks_trace_srcu_struct, scp);