LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
* [PATCH] KVM: X86: Don't load/put guest FPU context for sleeping AP
@ 2020-03-10  7:01 Wanpeng Li
  2020-03-10 16:01 ` Sean Christopherson
  0 siblings, 1 reply; 3+ messages in thread
From: Wanpeng Li @ 2020-03-10  7:01 UTC (permalink / raw)
  To: linux-kernel, kvm
  Cc: Paolo Bonzini, Sean Christopherson, Vitaly Kuznetsov, Wanpeng Li,
	Jim Mattson, Joerg Roedel

From: Wanpeng Li <wanpengli@tencent.com>

kvm_load_guest_fpu() and kvm_put_guest_fpu() each consume more than 14us 
observed by ftrace, the qemu userspace FPU is swapped out for the guest 
FPU context for the duration of the KVM_RUN ioctl even if sleeping AP, 
we shouldn't load/put guest FPU context for this case especially for 
serverless scenario which sensitives to boot time.

Signed-off-by: Wanpeng Li <wanpengli@tencent.com>
---
 arch/x86/kvm/x86.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 5de2006..080ffa4 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -8680,7 +8680,6 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
 
 	vcpu_load(vcpu);
 	kvm_sigset_activate(vcpu);
-	kvm_load_guest_fpu(vcpu);
 
 	if (unlikely(vcpu->arch.mp_state == KVM_MP_STATE_UNINITIALIZED)) {
 		if (kvm_run->immediate_exit) {
@@ -8718,12 +8717,14 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
 		}
 	}
 
+	kvm_load_guest_fpu(vcpu);
+
 	if (unlikely(vcpu->arch.complete_userspace_io)) {
 		int (*cui)(struct kvm_vcpu *) = vcpu->arch.complete_userspace_io;
 		vcpu->arch.complete_userspace_io = NULL;
 		r = cui(vcpu);
 		if (r <= 0)
-			goto out;
+			goto out_fpu;
 	} else
 		WARN_ON(vcpu->arch.pio.count || vcpu->mmio_needed);
 
@@ -8732,8 +8733,9 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
 	else
 		r = vcpu_run(vcpu);
 
-out:
+out_fpu:
 	kvm_put_guest_fpu(vcpu);
+out:
 	if (vcpu->run->kvm_valid_regs)
 		store_regs(vcpu);
 	post_kvm_run_save(vcpu);
-- 
2.7.4


^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [PATCH] KVM: X86: Don't load/put guest FPU context for sleeping AP
  2020-03-10  7:01 [PATCH] KVM: X86: Don't load/put guest FPU context for sleeping AP Wanpeng Li
@ 2020-03-10 16:01 ` Sean Christopherson
  2020-03-11  1:12   ` Wanpeng Li
  0 siblings, 1 reply; 3+ messages in thread
From: Sean Christopherson @ 2020-03-10 16:01 UTC (permalink / raw)
  To: Wanpeng Li
  Cc: linux-kernel, kvm, Paolo Bonzini, Vitaly Kuznetsov, Wanpeng Li,
	Jim Mattson, Joerg Roedel

On Tue, Mar 10, 2020 at 03:01:19PM +0800, Wanpeng Li wrote:
> From: Wanpeng Li <wanpengli@tencent.com>
> 
> kvm_load_guest_fpu() and kvm_put_guest_fpu() each consume more than 14us 
> observed by ftrace, the qemu userspace FPU is swapped out for the guest 
> FPU context for the duration of the KVM_RUN ioctl even if sleeping AP, 
> we shouldn't load/put guest FPU context for this case especially for 
> serverless scenario which sensitives to boot time.
> 
> Signed-off-by: Wanpeng Li <wanpengli@tencent.com>
> ---
>  arch/x86/kvm/x86.c | 8 +++++---
>  1 file changed, 5 insertions(+), 3 deletions(-)
> 
> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index 5de2006..080ffa4 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -8680,7 +8680,6 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
>  
>  	vcpu_load(vcpu);
>  	kvm_sigset_activate(vcpu);
> -	kvm_load_guest_fpu(vcpu);
>  
>  	if (unlikely(vcpu->arch.mp_state == KVM_MP_STATE_UNINITIALIZED)) {
>  		if (kvm_run->immediate_exit) {
> @@ -8718,12 +8717,14 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
>  		}
>  	}
>  
> +	kvm_load_guest_fpu(vcpu);

Ugh, so this isn't safe on MPX capable CPUs, kvm_apic_accept_events() can
trigger kvm_vcpu_reset() with @init_event=true and try to unload guest_fpu.

We could hack around that issue, but it'd be ugly, and I'm also concerned
that calling vmx_vcpu_reset() without guest_fpu loaded will be problematic
in the future with all the things that are getting managed by XSAVE.

> +
>  	if (unlikely(vcpu->arch.complete_userspace_io)) {
>  		int (*cui)(struct kvm_vcpu *) = vcpu->arch.complete_userspace_io;
>  		vcpu->arch.complete_userspace_io = NULL;
>  		r = cui(vcpu);
>  		if (r <= 0)
> -			goto out;
> +			goto out_fpu;
>  	} else
>  		WARN_ON(vcpu->arch.pio.count || vcpu->mmio_needed);
>  
> @@ -8732,8 +8733,9 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
>  	else
>  		r = vcpu_run(vcpu);
>  
> -out:
> +out_fpu:
>  	kvm_put_guest_fpu(vcpu);
> +out:
>  	if (vcpu->run->kvm_valid_regs)
>  		store_regs(vcpu);
>  	post_kvm_run_save(vcpu);
> -- 
> 2.7.4
> 

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [PATCH] KVM: X86: Don't load/put guest FPU context for sleeping AP
  2020-03-10 16:01 ` Sean Christopherson
@ 2020-03-11  1:12   ` Wanpeng Li
  0 siblings, 0 replies; 3+ messages in thread
From: Wanpeng Li @ 2020-03-11  1:12 UTC (permalink / raw)
  To: Sean Christopherson
  Cc: LKML, kvm, Paolo Bonzini, Vitaly Kuznetsov, Wanpeng Li,
	Jim Mattson, Joerg Roedel

On Wed, 11 Mar 2020 at 00:01, Sean Christopherson
<sean.j.christopherson@intel.com> wrote:
>
> On Tue, Mar 10, 2020 at 03:01:19PM +0800, Wanpeng Li wrote:
> > From: Wanpeng Li <wanpengli@tencent.com>
> >
> > kvm_load_guest_fpu() and kvm_put_guest_fpu() each consume more than 14us
> > observed by ftrace, the qemu userspace FPU is swapped out for the guest
> > FPU context for the duration of the KVM_RUN ioctl even if sleeping AP,
> > we shouldn't load/put guest FPU context for this case especially for
> > serverless scenario which sensitives to boot time.
> >
> > Signed-off-by: Wanpeng Li <wanpengli@tencent.com>
> > ---
> >  arch/x86/kvm/x86.c | 8 +++++---
> >  1 file changed, 5 insertions(+), 3 deletions(-)
> >
> > diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> > index 5de2006..080ffa4 100644
> > --- a/arch/x86/kvm/x86.c
> > +++ b/arch/x86/kvm/x86.c
> > @@ -8680,7 +8680,6 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
> >
> >       vcpu_load(vcpu);
> >       kvm_sigset_activate(vcpu);
> > -     kvm_load_guest_fpu(vcpu);
> >
> >       if (unlikely(vcpu->arch.mp_state == KVM_MP_STATE_UNINITIALIZED)) {
> >               if (kvm_run->immediate_exit) {
> > @@ -8718,12 +8717,14 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
> >               }
> >       }
> >
> > +     kvm_load_guest_fpu(vcpu);
>
> Ugh, so this isn't safe on MPX capable CPUs, kvm_apic_accept_events() can
> trigger kvm_vcpu_reset() with @init_event=true and try to unload guest_fpu.

Right.

>
> We could hack around that issue, but it'd be ugly, and I'm also concerned
> that calling vmx_vcpu_reset() without guest_fpu loaded will be problematic
> in the future with all the things that are getting managed by XSAVE.
>
> > +
> >       if (unlikely(vcpu->arch.complete_userspace_io)) {
> >               int (*cui)(struct kvm_vcpu *) = vcpu->arch.complete_userspace_io;
> >               vcpu->arch.complete_userspace_io = NULL;
> >               r = cui(vcpu);
> >               if (r <= 0)
> > -                     goto out;
> > +                     goto out_fpu;
> >       } else
> >               WARN_ON(vcpu->arch.pio.count || vcpu->mmio_needed);
> >
> > @@ -8732,8 +8733,9 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
> >       else
> >               r = vcpu_run(vcpu);
> >
> > -out:
> > +out_fpu:
> >       kvm_put_guest_fpu(vcpu);
> > +out:
> >       if (vcpu->run->kvm_valid_regs)
> >               store_regs(vcpu);
> >       post_kvm_run_save(vcpu);
> > --
> > 2.7.4
> >

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2020-03-11  1:12 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-03-10  7:01 [PATCH] KVM: X86: Don't load/put guest FPU context for sleeping AP Wanpeng Li
2020-03-10 16:01 ` Sean Christopherson
2020-03-11  1:12   ` Wanpeng Li

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).