LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: "Raslan, KarimAllah" <karahmed@amazon.de>
To: "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"pbonzini@redhat.com" <pbonzini@redhat.com>
Cc: "jmattson@google.com" <jmattson@google.com>,
	"rkrcmar@redhat.com" <rkrcmar@redhat.com>
Subject: Re: [PATCH 1/2] X86/KVM: Properly restore 'tsc_offset' when running an L2 guest
Date: Thu, 12 Apr 2018 17:21:02 +0000	[thread overview]
Message-ID: <1523553662.32594.29.camel@amazon.de> (raw)
In-Reply-To: <1523552643.32594.18.camel@amazon.de>

On Thu, 2018-04-12 at 17:04 +0000, Raslan, KarimAllah wrote:
> On Thu, 2018-04-12 at 18:35 +0200, Paolo Bonzini wrote:
> > 
> > On 12/04/2018 17:12, KarimAllah Ahmed wrote:
> > > 
> > > 
> > > When the TSC MSR is captured while an L2 guest is running then restored,
> > > the 'tsc_offset' ends up capturing the L02 TSC_OFFSET instead of the L01
> > > TSC_OFFSET. So ensure that this is compensated for when storing the value.
> > > 
> > > Cc: Jim Mattson <jmattson@google.com>
> > > Cc: Paolo Bonzini <pbonzini@redhat.com>
> > > Cc: Radim Krčmář <rkrcmar@redhat.com>
> > > Cc: kvm@vger.kernel.org
> > > Cc: linux-kernel@vger.kernel.org
> > > Signed-off-by: KarimAllah Ahmed <karahmed@amazon.de>
> > > ---
> > >  arch/x86/kvm/vmx.c | 12 +++++++++---
> > >  arch/x86/kvm/x86.c |  1 -
> > >  2 files changed, 9 insertions(+), 4 deletions(-)
> > > 
> > > diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
> > > index cff2f50..2f57571 100644
> > > --- a/arch/x86/kvm/vmx.c
> > > +++ b/arch/x86/kvm/vmx.c
> > > @@ -2900,6 +2900,8 @@ static u64 guest_read_tsc(struct kvm_vcpu *vcpu)
> > >   */
> > >  static void vmx_write_tsc_offset(struct kvm_vcpu *vcpu, u64 offset)
> > >  {
> > > +	u64 l1_tsc_offset = 0;
> > > +
> > >  	if (is_guest_mode(vcpu)) {
> > >  		/*
> > >  		 * We're here if L1 chose not to trap WRMSR to TSC. According
> > > @@ -2908,16 +2910,20 @@ static void vmx_write_tsc_offset(struct kvm_vcpu *vcpu, u64 offset)
> > >  		 * to the newly set TSC to get L2's TSC.
> > >  		 */
> > >  		struct vmcs12 *vmcs12;
> > > +
> > >  		/* recalculate vmcs02.TSC_OFFSET: */
> > >  		vmcs12 = get_vmcs12(vcpu);
> > > -		vmcs_write64(TSC_OFFSET, offset +
> > > -			(nested_cpu_has(vmcs12, CPU_BASED_USE_TSC_OFFSETING) ?
> > > -			 vmcs12->tsc_offset : 0));
> > > +
> > > +		l1_tsc_offset = nested_cpu_has(vmcs12, CPU_BASED_USE_TSC_OFFSETING) ?
> > > +					vmcs12->tsc_offset : 0;
> > > +		vmcs_write64(TSC_OFFSET, offset + l1_tsc_offset);
> > >  	} else {
> > >  		trace_kvm_write_tsc_offset(vcpu->vcpu_id,
> > >  					   vmcs_read64(TSC_OFFSET), offset);
> > >  		vmcs_write64(TSC_OFFSET, offset);
> > >  	}
> > > +
> > > +	vcpu->arch.tsc_offset = offset - l1_tsc_offset;
> > 
> > Using both "offset + l1_tsc_offset" and "offset - l1_tsc_offset" in this 
> > function seems wrong to me: if vcpu->arch.tsc_offset must be "offset - 
> > l1_tsc_offset", then "offset" must be written to TSC_OFFSET.
> 
> Ooops! I forgot to remove the + l1_tsc_offset :D
> 
> > 
> > 
> > I think the bug was introduced by commit 3e3f50262.  Before,
> > vmx_read_tsc_offset returned the L02 offset; now it always contains the
> > L01 offset.  So the right fix is to adjust vcpu->arch.tsc_offset on
> > nested vmentry/vmexit.  If is_guest_mode(vcpu), kvm_read_l1_tsc must use
> > a new kvm_x86_ops callback to subtract the L12 offset from the value it
> > returns.
> 
> ack!

Now looking further at the code, it seems that everywhere in the code
tsc_offset is treated as the L01 TSC_OFFSET.

Like here:

        if (vmcs12->cpu_based_vm_exec_control &
CPU_BASED_USE_TSC_OFFSETING)
                vmcs_write64(TSC_OFFSET,
                        vcpu->arch.tsc_offset + vmcs12->tsc_offset);

and here:

        vmcs_write64(TSC_OFFSET, vcpu->arch.tsc_offset);

and here:

u64 kvm_read_l1_tsc(struct kvm_vcpu *vcpu, u64 host_tsc)
{
        return vcpu->arch.tsc_offset + kvm_scale_tsc(vcpu, host_tsc);
}
EXPORT_SYMBOL_GPL(kvm_read_l1_tsc);

... would not it be simpler and more inline with the current code to
just do what I did above + remove the "+ l1_tsc_offset" + probably
document tsc_offset ?

> 
> > 
> > 
> > Thanks,
> > 
> > Paolo
> > 
> > > 
> > > 
> > >  }
> > >  
> > >  /*
> > > diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> > > index ac42c85..1a2ed92 100644
> > > --- a/arch/x86/kvm/x86.c
> > > +++ b/arch/x86/kvm/x86.c
> > > @@ -1539,7 +1539,6 @@ EXPORT_SYMBOL_GPL(kvm_read_l1_tsc);
> > >  static void kvm_vcpu_write_tsc_offset(struct kvm_vcpu *vcpu, u64 offset)
> > >  {
> > >  	kvm_x86_ops->write_tsc_offset(vcpu, offset);
> > > -	vcpu->arch.tsc_offset = offset;
> > >  }
> > >  
> > >  static inline bool kvm_check_tsc_unstable(void)
> > > 
> > 
> > 
Amazon Development Center Germany GmbH
Berlin - Dresden - Aachen
main office: Krausenstr. 38, 10117 Berlin
Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
Ust-ID: DE289237879
Eingetragen am Amtsgericht Charlottenburg HRB 149173 B

  reply	other threads:[~2018-04-12 17:21 UTC|newest]

Thread overview: 16+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-04-12 15:12 KarimAllah Ahmed
2018-04-12 15:12 ` [PATCH 2/2] kvm: nVMX: Introduce KVM_CAP_STATE KarimAllah Ahmed
2018-04-12 16:39   ` Paolo Bonzini
2018-04-14 15:56   ` Raslan, KarimAllah
2018-04-14 22:31     ` Raslan, KarimAllah
2018-04-16 16:22   ` Jim Mattson
2018-04-16 17:15     ` Raslan, KarimAllah
2018-04-26 22:28       ` Jim Mattson
2018-04-27 10:03         ` Paolo Bonzini
2018-04-27 15:19           ` Jim Mattson
2018-04-28  0:42             ` Paolo Bonzini
2018-04-12 16:35 ` [PATCH 1/2] X86/KVM: Properly restore 'tsc_offset' when running an L2 guest Paolo Bonzini
2018-04-12 17:04   ` Raslan, KarimAllah
2018-04-12 17:21     ` Raslan, KarimAllah [this message]
2018-04-12 20:21       ` Paolo Bonzini
2018-04-12 20:24         ` Raslan, KarimAllah

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1523553662.32594.29.camel@amazon.de \
    --to=karahmed@amazon.de \
    --cc=jmattson@google.com \
    --cc=kvm@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=pbonzini@redhat.com \
    --cc=rkrcmar@redhat.com \
    --subject='Re: [PATCH 1/2] X86/KVM: Properly restore '\''tsc_offset'\'' when running an L2 guest' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).