From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932597AbXA1NbO (ORCPT ); Sun, 28 Jan 2007 08:31:14 -0500 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S932573AbXA1NbE (ORCPT ); Sun, 28 Jan 2007 08:31:04 -0500 Received: from mx1.redhat.com ([66.187.233.31]:34738 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932564AbXA1N3a (ORCPT ); Sun, 28 Jan 2007 08:29:30 -0500 Message-Id: <20070128132435.804546000@programming.kicks-ass.net> References: <20070128131343.628722000@programming.kicks-ass.net> User-Agent: quilt/0.45-1 Date: Sun, 28 Jan 2007 14:13:50 +0100 From: Peter Zijlstra To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Andrew Morton , Nick Piggin , Christoph Lameter , Ingo Molnar , Rik van Riel , Peter Zijlstra Subject: [PATCH 07/14] mm: speculative find_get_pages_tag Content-Disposition: inline; filename=mm-lockless-find_get_pages_tag.patch Sender: linux-kernel-owner@vger.kernel.org X-Mailing-List: linux-kernel@vger.kernel.org implement the speculative find_get_pages_tag. Signed-off-by: Peter Zijlstra --- mm/filemap.c | 42 +++++++++++++++++++++++++++++++++--------- 1 file changed, 33 insertions(+), 9 deletions(-) Index: linux-2.6/mm/filemap.c =================================================================== --- linux-2.6.orig/mm/filemap.c 2007-01-22 20:11:07.000000000 +0100 +++ linux-2.6/mm/filemap.c 2007-01-22 20:11:09.000000000 +0100 @@ -849,16 +849,40 @@ unsigned find_get_pages_tag(struct addre { unsigned int i; unsigned int ret; + unsigned int nr_found; - read_lock_irq(&mapping->tree_lock); - /* TODO: implement lookup_tag_slot and make this lockless */ - ret = radix_tree_gang_lookup_tag(&mapping->page_tree, - (void **)pages, *index, nr_pages, tag); - for (i = 0; i < ret; i++) - page_cache_get(pages[i]); - if (ret) - *index = pages[ret - 1]->index + 1; - read_unlock_irq(&mapping->tree_lock); + rcu_read_lock(); +restart: + nr_found = radix_tree_gang_lookup_tag_slot(&mapping->page_tree, + (void ***)pages, *index, nr_pages, tag); + + ret = 0; + for (i = 0; i < nr_found; i++) { + struct page *page; +repeat: + page = radix_tree_deref_slot((void**)pages[i]); + if (unlikely(!page)) + continue; + /* + * this can only trigger if nr_found == 1, making livelocks + * a non issue. + */ + if (unlikely(page == RADIX_TREE_RETRY)) + goto restart; + + if (!page_cache_get_speculative(page)) + goto repeat; + + /* Has the page moved? */ + if (unlikely(page != *((void **)pages[i]))) { + page_cache_release(page); + goto repeat; + } + + pages[ret] = page; + ret++; + } + rcu_read_unlock(); return ret; } --