[patch -mm 1/2] i386: add ptep_test_and_clear_{dirty,young}

[patch -mm 1/2] i386: add ptep_test_and_clear_{dirty,young}

[David Rientjes]

Add ptep_test_and_clear_{dirty,young} to i386.  They advertise that they
have it and there is at least one place where it needs to be called
without the page table lock: to clear the accessed bit on write to
/proc/pid/clear_refs.

ptep_clear_flush_{dirty,young} are updated to use the new functions.  The
overall net effect to current users of ptep_clear_flush_{dirty,young} is
that we introduce an additional branch.

Cc: Hugh Dickins <hugh@veritas.com>
Cc: Ingo Molnar <mingo@redhat.com>
Signed-off-by: David Rientjes <rientjes@google.com>
---
 include/asm-i386/pgtable.h |   25 +++++++++++++++++--------
 1 files changed, 17 insertions(+), 8 deletions(-)

diff --git a/include/asm-i386/pgtable.h b/include/asm-i386/pgtable.h
--- a/include/asm-i386/pgtable.h
+++ b/include/asm-i386/pgtable.h
@@ -283,12 +283,23 @@ do {									\
 	}								\
 } while (0)
 
-/*
- * We don't actually have these, but we want to advertise them so that
- * we can encompass the flush here.
- */
 #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_DIRTY
+static inline int ptep_test_and_clear_dirty(struct vm_area_struct *vma,
+					    unsigned long addr, pte_t *ptep)
+{
+	if (!pte_dirty(*ptep))
+		return 0;
+	return test_and_clear_bit(_PAGE_BIT_DIRTY, &ptep->pte_low);
+}
+
 #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG
+static inline int ptep_test_and_clear_young(struct vm_area_struct *vma,
+					    unsigned long addr, pte_t *ptep)
+{
+	if (!pte_young(*ptep))
+		return 0;
+	return test_and_clear_bit(_PAGE_BIT_ACCESSED, &ptep->pte_low);
+}
 
 /*
  * Rules for using ptep_establish: the pte MUST be a user pte, and
@@ -305,9 +316,8 @@ do {									\
 #define ptep_clear_flush_dirty(vma, address, ptep)			\
 ({									\
 	int __dirty;							\
-	__dirty = pte_dirty(*(ptep));					\
+	__dirty = ptep_test_and_clear_dirty((vma), (address), (ptep));	\
 	if (__dirty) {							\
-		clear_bit(_PAGE_BIT_DIRTY, &(ptep)->pte_low);		\
 		pte_update_defer((vma)->vm_mm, (address), (ptep));	\
 		flush_tlb_page(vma, address);				\
 	}								\
@@ -318,9 +328,8 @@ do {									\
 #define ptep_clear_flush_young(vma, address, ptep)			\
 ({									\
 	int __young;							\
-	__young = pte_young(*(ptep));					\
+	__young = ptep_test_and_clear_young((vma), (address), (ptep));	\
 	if (__young) {							\
-		clear_bit(_PAGE_BIT_ACCESSED, &(ptep)->pte_low);	\
 		pte_update_defer((vma)->vm_mm, (address), (ptep));	\
 		flush_tlb_page(vma, address);				\
 	}								\

[patch -mm 2/2] smaps: use ptep_test_and_clear_young

[David Rientjes]

Use arch-specified ptep_test_and_clear_young() to clear the pte accessed
bits for /proc/pid/clear_refs.  This avoids a race condition if a pte is
modified between pte_mkold() and set_pte_at().

Cc: Hugh Dickins <hugh@veritas.com>
Signed-off-by: David Rientjes <rientjes@google.com>
---
 fs/proc/task_mmu.c |    5 +----
 1 files changed, 1 insertions(+), 4 deletions(-)

diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -277,10 +277,7 @@ static void clear_refs_one_pmd(struct vm_area_struct *vma, pmd_t *pmd,
 			continue;
 
 		/* Clear accessed and referenced bits. */
-		if (pte_young(ptent)) {
-			ptent = pte_mkold(ptent);
-			set_pte_at(vma->vm_mm, addr, pte, ptent);
-		}
+		ptep_test_and_clear_young(vma, addr, pte);
 		ClearPageReferenced(page);
 	}
 	pte_unmap_unlock(pte - 1, ptl);

Re: [patch -mm 1/2] i386: add ptep_test_and_clear_{dirty,young}

[Hugh Dickins]

On Sun, 25 Mar 2007, David Rientjes wrote:
> Add ptep_test_and_clear_{dirty,young} to i386.  They advertise that they
> have it and there is at least one place where it needs to be called
> without the page table lock: to clear the accessed bit on write to

Without the page table lock??

> /proc/pid/clear_refs.
> 
> ptep_clear_flush_{dirty,young} are updated to use the new functions.  The
> overall net effect to current users of ptep_clear_flush_{dirty,young} is
> that we introduce an additional branch.

We need to Cc Zach on this: git blame indicates it was he who replaced
i386's ptep_test_and_clear_{dirty,young} by that "We don't actually
have these" comment - it looks a bit as if what you want to do might
violate the assumptions he wants to make, but I don't grasp it.

Hugh

> 
> Cc: Hugh Dickins <hugh@veritas.com>
> Cc: Ingo Molnar <mingo@redhat.com>
> Signed-off-by: David Rientjes <rientjes@google.com>
> ---
>  include/asm-i386/pgtable.h |   25 +++++++++++++++++--------
>  1 files changed, 17 insertions(+), 8 deletions(-)
> 
> diff --git a/include/asm-i386/pgtable.h b/include/asm-i386/pgtable.h
> --- a/include/asm-i386/pgtable.h
> +++ b/include/asm-i386/pgtable.h
> @@ -283,12 +283,23 @@ do {									\
>  	}								\
>  } while (0)
>  
> -/*
> - * We don't actually have these, but we want to advertise them so that
> - * we can encompass the flush here.
> - */
>  #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_DIRTY
> +static inline int ptep_test_and_clear_dirty(struct vm_area_struct *vma,
> +					    unsigned long addr, pte_t *ptep)
> +{
> +	if (!pte_dirty(*ptep))
> +		return 0;
> +	return test_and_clear_bit(_PAGE_BIT_DIRTY, &ptep->pte_low);
> +}
> +
>  #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG
> +static inline int ptep_test_and_clear_young(struct vm_area_struct *vma,
> +					    unsigned long addr, pte_t *ptep)
> +{
> +	if (!pte_young(*ptep))
> +		return 0;
> +	return test_and_clear_bit(_PAGE_BIT_ACCESSED, &ptep->pte_low);
> +}
>  
>  /*
>   * Rules for using ptep_establish: the pte MUST be a user pte, and
> @@ -305,9 +316,8 @@ do {									\
>  #define ptep_clear_flush_dirty(vma, address, ptep)			\
>  ({									\
>  	int __dirty;							\
> -	__dirty = pte_dirty(*(ptep));					\
> +	__dirty = ptep_test_and_clear_dirty((vma), (address), (ptep));	\
>  	if (__dirty) {							\
> -		clear_bit(_PAGE_BIT_DIRTY, &(ptep)->pte_low);		\
>  		pte_update_defer((vma)->vm_mm, (address), (ptep));	\
>  		flush_tlb_page(vma, address);				\
>  	}								\
> @@ -318,9 +328,8 @@ do {									\
>  #define ptep_clear_flush_young(vma, address, ptep)			\
>  ({									\
>  	int __young;							\
> -	__young = pte_young(*(ptep));					\
> +	__young = ptep_test_and_clear_young((vma), (address), (ptep));	\
>  	if (__young) {							\
> -		clear_bit(_PAGE_BIT_ACCESSED, &(ptep)->pte_low);	\
>  		pte_update_defer((vma)->vm_mm, (address), (ptep));	\
>  		flush_tlb_page(vma, address);				\
>  	}								\

Re: [patch -mm 1/2] i386: add ptep_test_and_clear_{dirty,young}

[Hugh Dickins]

On Sun, 25 Mar 2007, Zachary Amsden wrote:
> 
> If you actually clear the bit, you need to:
> 
> +         pte_update_defer(vma->vm_mm, addr, ptep);
> 
> The reason is, when updating PTEs, the hypervisor must be notified.  Using
> atomic operations to do this is fine for all hypervisors I am aware of.
> However, for hypervisors which shadow page tables, if these PTE modifications
> are not trapped, you need a post-modification call to fulfill the update of
> the shadow page table.

Thanks for the very rapid response.

So, David just needs to move the pte_update_defer out of
ptep_clear_flush_* and into ptep_test_and_clear_*?

That leaves me wondering why you deleted ptep_test_and_clear_*
(while leaving their __HAVE_ARCHes) in the first place?

Hugh

Re: [patch -mm 1/2] i386: add ptep_test_and_clear_{dirty,young}

[David Rientjes]

On Sun, 25 Mar 2007, Zachary Amsden wrote:

> If you actually clear the bit, you need to:
> 
> +         pte_update_defer(vma->vm_mm, addr, ptep);
> 
> The reason is, when updating PTEs, the hypervisor must be notified.  Using
> atomic operations to do this is fine for all hypervisors I am aware of.
> However, for hypervisors which shadow page tables, if these PTE modifications
> are not trapped, you need a post-modification call to fulfill the update of
> the shadow page table.
> 

Then why was ptep_test_and_clear_{dirty,young} ever removed in the first 
place??  To gain the optimization of one fewer branch and introduce a hack 
to advertise it's existance so the generic header file doesn't include its 
own version?

Re: [patch -mm 1/2] i386: add ptep_test_and_clear_{dirty,young}

[Zachary Amsden]

Hugh Dickins wrote:
> On Sun, 25 Mar 2007, David Rientjes wrote:
>   
>> Add ptep_test_and_clear_{dirty,young} to i386.  They advertise that they
>> have it and there is at least one place where it needs to be called
>> without the page table lock: to clear the accessed bit on write to
>>     
>
> Without the page table lock??
>
>   
>> /proc/pid/clear_refs.
>>
>> ptep_clear_flush_{dirty,young} are updated to use the new functions.  The
>> overall net effect to current users of ptep_clear_flush_{dirty,young} is
>> that we introduce an additional branch.
>>     
>
> We need to Cc Zach on this: git blame indicates it was he who replaced
> i386's ptep_test_and_clear_{dirty,young} by that "We don't actually
> have these" comment - it looks a bit as if what you want to do might
> violate the assumptions he wants to make, but I don't grasp it.
>
> Hugh
>
>   
>> Cc: Hugh Dickins <hugh@veritas.com>
>> Cc: Ingo Molnar <mingo@redhat.com>
>> Signed-off-by: David Rientjes <rientjes@google.com>
>> ---
>>  include/asm-i386/pgtable.h |   25 +++++++++++++++++--------
>>  1 files changed, 17 insertions(+), 8 deletions(-)
>>
>> diff --git a/include/asm-i386/pgtable.h b/include/asm-i386/pgtable.h
>> --- a/include/asm-i386/pgtable.h
>> +++ b/include/asm-i386/pgtable.h
>> @@ -283,12 +283,23 @@ do {									\
>>  	}								\
>>  } while (0)
>>  
>> -/*
>> - * We don't actually have these, but we want to advertise them so that
>> - * we can encompass the flush here.
>> - */
>>  #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_DIRTY
>> +static inline int ptep_test_and_clear_dirty(struct vm_area_struct *vma,
>> +					    unsigned long addr, pte_t *ptep)
>> +{
>> +	if (!pte_dirty(*ptep))
>> +		return 0;
>>     
>> +	return test_and_clear_bit(_PAGE_BIT_DIRTY, &ptep->pte_low);
>> +}
>> +
>>     

If you actually clear the bit, you need to:

+         pte_update_defer(vma->vm_mm, addr, ptep);

The reason is, when updating PTEs, the hypervisor must be notified.  
Using atomic operations to do this is fine for all hypervisors I am 
aware of.  However, for hypervisors which shadow page tables, if these 
PTE modifications are not trapped, you need a post-modification call to 
fulfill the update of the shadow page table.

>>  #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG
>> +static inline int ptep_test_and_clear_young(struct vm_area_struct *vma,
>> +					    unsigned long addr, pte_t *ptep)
>> +{
>> +	if (!pte_young(*ptep))
>> +		return 0;
>> +	return test_and_clear_bit(_PAGE_BIT_ACCESSED, &ptep->pte_low);
>> +}
>>     

Same here.

Hugh, thanks for the cc.

Zach

Re: [patch -mm 1/2] i386: add ptep_test_and_clear_{dirty,young}

[Zachary Amsden]

Hugh Dickins wrote:
> On Sun, 25 Mar 2007, Zachary Amsden wrote:
>   
>> If you actually clear the bit, you need to:
>>
>> +         pte_update_defer(vma->vm_mm, addr, ptep);
>>
>> The reason is, when updating PTEs, the hypervisor must be notified.  Using
>> atomic operations to do this is fine for all hypervisors I am aware of.
>> However, for hypervisors which shadow page tables, if these PTE modifications
>> are not trapped, you need a post-modification call to fulfill the update of
>> the shadow page table.
>>     
>
> Thanks for the very rapid response.
>
> So, David just needs to move the pte_update_defer out of
> ptep_clear_flush_* and into ptep_test_and_clear_*?
>   

Yes.

> That leaves me wondering why you deleted ptep_test_and_clear_*
> (while leaving their __HAVE_ARCHes) in the first place?
>   

Because raw use of them in the arch independent MM code would introduce 
exactly this bug on i386, so leaving __HAVE_ARCH but leaving out the 
definition would catch this case.

Zach

Re: [patch -mm 1/2] i386: add ptep_test_and_clear_{dirty,young}

[Zachary Amsden]

David Rientjes wrote:
> On Sun, 25 Mar 2007, Zachary Amsden wrote:
>
>   
>> If you actually clear the bit, you need to:
>>
>> +         pte_update_defer(vma->vm_mm, addr, ptep);
>>
>> The reason is, when updating PTEs, the hypervisor must be notified.  Using
>> atomic operations to do this is fine for all hypervisors I am aware of.
>> However, for hypervisors which shadow page tables, if these PTE modifications
>> are not trapped, you need a post-modification call to fulfill the update of
>> the shadow page table.
>>
>>     
>
> Then why was ptep_test_and_clear_{dirty,young} ever removed in the first 
> place??  To gain the optimization of one fewer branch and introduce a hack 
> to advertise it's existance so the generic header file doesn't include its 
> own version?
>   

Yes, pretty much.

Zach