LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Davidlohr Bueso <dave@stgolabs.net>
To: Peter Zijlstra <peterz@infradead.org>, Ingo Molnar <mingo@kernel.org>
Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>,
	Jason Low <jason.low2@hp.com>,
	Michel Lespinasse <walken@google.com>,
	Tim Chen <tim.c.chen@linux.intel.com>,
	linux-kernel@vger.kernel.org, Davidlohr Bueso <dave@stgolabs.net>,
	Davidlohr Bueso <dbueso@suse.de>
Subject: [PATCH 6/6] locking/rwsem: Check for active lock before bailing on spinning
Date: Sun, 25 Jan 2015 23:36:09 -0800	[thread overview]
Message-ID: <1422257769-14083-7-git-send-email-dave@stgolabs.net> (raw)
In-Reply-To: <1422257769-14083-1-git-send-email-dave@stgolabs.net>

37e9562453b (locking/rwsem: Allow conservative optimistic
spinning when readers have lock) forced the default for
optimistic spinning to be disabled if the lock owner was
nil, which makes much sense for readers. However, while
it is not our priority, we can make some optimizations
for write-mostly workloads. We can bail the spinning step
and still be conservative if there are any active tasks,
otherwise there's really no reason not to spin, as the
semaphore is most likely unlocked.

This patch recovers most of a Unixbench 'execl' benchmark
throughput by sleeping less and making better average system
usage:

before:
CPU     %user     %nice   %system   %iowait    %steal     %idle
all      0.60      0.00      8.02      0.00      0.00     91.38

after:
CPU     %user     %nice   %system   %iowait    %steal     %idle
all      1.22      0.00     70.18      0.00      0.00     28.60

Signed-off-by: Davidlohr Bueso <dbueso@suse.de>
---
 kernel/locking/rwsem-xadd.c | 27 +++++++++++++++++----------
 1 file changed, 17 insertions(+), 10 deletions(-)

diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c
index 88b3468..e0e9738 100644
--- a/kernel/locking/rwsem-xadd.c
+++ b/kernel/locking/rwsem-xadd.c
@@ -296,23 +296,30 @@ static inline bool rwsem_try_write_lock_unqueued(struct rw_semaphore *sem)
 static inline bool rwsem_can_spin_on_owner(struct rw_semaphore *sem)
 {
 	struct task_struct *owner;
-	bool on_cpu = false;
+	bool ret = true;
 
 	if (need_resched())
 		return false;
 
 	rcu_read_lock();
 	owner = ACCESS_ONCE(sem->owner);
-	if (owner)
-		on_cpu = owner->on_cpu;
-	rcu_read_unlock();
+	if (!owner) {
+		long count = ACCESS_ONCE(sem->count);
+		/*
+		 * If sem->owner is not set, yet we have just recently entered the
+		 * slowpath with the lock being active, then there is a possibility
+		 * reader(s) may have the lock. To be safe, bail spinning in these
+		 * situations.
+		 */
+		if (count & RWSEM_ACTIVE_MASK)
+			ret = false;
+		goto done;
+	}
 
-	/*
-	 * If sem->owner is not set, yet we have just recently entered the
-	 * slowpath, then there is a possibility reader(s) may have the lock.
-	 * To be safe, avoid spinning in these situations.
-	 */
-	return on_cpu;
+	ret = owner->on_cpu;
+done:
+	rcu_read_unlock();
+	return ret;
 }
 
 static inline bool owner_running(struct rw_semaphore *sem,
-- 
2.1.2


  parent reply	other threads:[~2015-01-26  7:37 UTC|newest]

Thread overview: 26+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2015-01-26  7:36 [PATCH -tip 0/6] rwsem: Fine tuning Davidlohr Bueso
2015-01-26  7:36 ` [PATCH 1/6] locking/rwsem: Use task->state helpers Davidlohr Bueso
2015-02-04 14:38   ` [tip:locking/core] " tip-bot for Davidlohr Bueso
2015-01-26  7:36 ` [PATCH 2/6] locking/rwsem: Document barrier need when waking tasks Davidlohr Bueso
2015-01-27 17:07   ` Peter Zijlstra
2015-01-27 20:30     ` Davidlohr Bueso
2015-01-26  7:36 ` [PATCH 3/6] locking/rwsem: Set lock ownership ASAP Davidlohr Bueso
2015-01-27 17:10   ` Peter Zijlstra
2015-01-27 19:18     ` Davidlohr Bueso
2015-01-26  7:36 ` [PATCH 4/6] locking/rwsem: Avoid deceiving lock spinners Davidlohr Bueso
2015-01-27 17:23   ` Jason Low
2015-01-28  3:54     ` Davidlohr Bueso
2015-01-28 17:01       ` Tim Chen
2015-01-28 21:03       ` Jason Low
2015-01-29  1:10         ` Davidlohr Bueso
2015-01-29 20:13           ` Jason Low
2015-01-29 20:18             ` Jason Low
2015-01-29 23:15               ` Davidlohr Bueso
2015-01-30  1:52                 ` Refactoring mutex spin on owner code Jason Low
2015-01-30  7:14                   ` Davidlohr Bueso
2015-01-30  7:51                     ` Peter Zijlstra
2015-01-26  7:36 ` [PATCH 5/6] locking/rwsem: Optimize slowpath/sleeping Davidlohr Bueso
2015-01-27 17:34   ` Peter Zijlstra
2015-01-27 21:57     ` Davidlohr Bueso
2015-01-26  7:36 ` Davidlohr Bueso [this message]
2015-01-27 18:11   ` [PATCH 6/6] locking/rwsem: Check for active lock before bailing on spinning Jason Low

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1422257769-14083-7-git-send-email-dave@stgolabs.net \
    --to=dave@stgolabs.net \
    --cc=dbueso@suse.de \
    --cc=jason.low2@hp.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@kernel.org \
    --cc=paulmck@linux.vnet.ibm.com \
    --cc=peterz@infradead.org \
    --cc=tim.c.chen@linux.intel.com \
    --cc=walken@google.com \
    --subject='Re: [PATCH 6/6] locking/rwsem: Check for active lock before bailing on spinning' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).