Re: [kvm-devel] [PATCH] export notifier #1

[Christoph Lameter <clameter@sgi.com>]

On Wed, 23 Jan 2008, Andrea Arcangeli wrote:

> I think it has yet to be demonstrated that doing the invalidate
> _before_ clearing the linux pte is workable at all for
> shadow-pte/RDMA. Infact even doing it _after_ still requires some form
> of serialization but it's less obviously broken and perhaps more
> fixable unlike doing it before that seems hardly fixable given the
> refill event running in the remote node is supposed to wait on a
> bitflag of a page in the master node to return ON. What Christoph
> didn't specify after hinting you have to wait for the PageExported
> bitflag to return on, is that such page may be back in the freelist by

Why would you wait for the PageExported flag to return on? You remove the 
remote mappings, the page lock is being held so no new mappings can occur. 
Then you remove the local mappings. A concurrent remote fault would be 
prevented by the subsystem which could involve waiting on the page lock.

> Until there's some more reasonable theory of how invalidating the
> remote tlbs/ptes _before_ the main linux pte can remotely work, I'm
> "quite" skeptical it's the way to go for the invalidate_page callback.

Not sure that I see the problem if the subsystem prevents new references 
from being established.
 
> Like Avi said, Xen is dealing with the linux pte only, so there's no
> racy smp page fault to serialize against. Perhaps we can add another
> notifier for Xen though.

Well I think we need to come up with a set of notifiers that can cover all 
cases. And so far the export notifiers seem to be the most general. But 
they also require more intelligence in the notifiers to do proper 
serialization and reverse mapping.
 
> But I think it's still not enough for Xen to have a method called
> before the ptep_clear_flush: rmap.c would get confused in
> page_mkclean_one for example. It might be possible that vm_ops is the

Why would it get confused? When we get to the operations in rmap.c we 
just need to be sure that remote references do no longer exist. If the 
operations in rmap.c fail then we can reestablish references on demand.

> right way for you even if it further clutters the VM. Like Avi pointed
> me out once, with our current mmu_notifiers we can export the KVM
> address space with remote dma and keep swapping the whole KVM asset
> just fine despite the triple MMU running the system (qemu using linux
> pte, KVM using spte, quadrics using pcimmu). And the core Linux VM
> code (not some obscure hypervisor) will deal with all aging and VM
> issues like a normal task (especially with my last patch that reflects
> the accessed bitflag in the spte the same way the accessed bitflag is
> reflected for the regular ptes).

The problem for us there is that multiple references may exist remotely. 
So the actual remote reference count needs to be calculated?