LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: zhenwei pi <pizhenwei@bytedance.com>
To: Wanpeng Li <kernellwp@gmail.com>, Maxim Levitsky <mlevitsk@redhat.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>,
	Kele Huang <huangkele@bytedance.com>,
	chaiwen.cc@bytedance.com, xieyongji@bytedance.com,
	dengliang.1214@bytedance.com, Wanpeng Li <wanpengli@tencent.com>,
	Sean Christopherson <seanjc@google.com>,
	Vitaly Kuznetsov <vkuznets@redhat.com>,
	Jim Mattson <jmattson@google.com>, Joerg Roedel <joro@8bytes.org>,
	Thomas Gleixner <tglx@linutronix.de>,
	Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
	Dave Hansen <dave.hansen@linux.intel.com>,
	the arch/x86 maintainers <x86@kernel.org>,
	"H. Peter Anvin" <hpa@zytor.com>, kvm <kvm@vger.kernel.org>,
	LKML <linux-kernel@vger.kernel.org>
Subject: Re: Re: [RFC] KVM: x86: SVM: don't expose PV_SEND_IPI feature with AVIC
Date: Tue, 16 Nov 2021 10:56:25 +0800	[thread overview]
Message-ID: <d65fbd73-7612-8348-2fd8-8da0f5e2a3c0@bytedance.com> (raw)
In-Reply-To: <CANRm+Cze_b0PJzOGB4-tPdrz-iHcJj-o7QL1t1Pf1083nJDQKQ@mail.gmail.com>



On 11/16/21 10:48 AM, Wanpeng Li wrote:
> On Mon, 8 Nov 2021 at 22:09, Maxim Levitsky <mlevitsk@redhat.com> wrote:
>>
>> On Mon, 2021-11-08 at 11:30 +0100, Paolo Bonzini wrote:
>>> On 11/8/21 10:59, Kele Huang wrote:
>>>> Currently, AVIC is disabled if x2apic feature is exposed to guest
>>>> or in-kernel PIT is in re-injection mode.
>>>>
>>>> We can enable AVIC with options:
>>>>
>>>>     Kmod args:
>>>>     modprobe kvm_amd avic=1 nested=0 npt=1
>>>>     QEMU args:
>>>>     ... -cpu host,-x2apic -global kvm-pit.lost_tick_policy=discard ...
>>>>
>>>> When LAPIC works in xapic mode, both AVIC and PV_SEND_IPI feature
>>>> can accelerate IPI operations for guest. However, the relationship
>>>> between AVIC and PV_SEND_IPI feature is not sorted out.
>>>>
>>>> In logical, AVIC accelerates most of frequently IPI operations
>>>> without VMM intervention, while the re-hooking of apic->send_IPI_xxx
>>>> from PV_SEND_IPI feature masks out it. People can get confused
>>>> if AVIC is enabled while getting lots of hypercall kvm_exits
>>>> from IPI.
>>>>
>>>> In performance, benchmark tool
>>>> https://lore.kernel.org/kvm/20171219085010.4081-1-ynorov@caviumnetworks.com/
>>>> shows below results:
>>>>
>>>>     Test env:
>>>>     CPU: AMD EPYC 7742 64-Core Processor
>>>>     2 vCPUs pinned 1:1
>>>>     idle=poll
>>>>
>>>>     Test result (average ns per IPI of lots of running):
>>>>     PV_SEND_IPI      : 1860
>>>>     AVIC             : 1390
>>>>
>>>> Besides, disscussions in https://lkml.org/lkml/2021/10/20/423
>>>> do have some solid performance test results to this.
>>>>
>>>> This patch fixes this by masking out PV_SEND_IPI feature when
>>>> AVIC is enabled in setting up of guest vCPUs' CPUID.
>>>>
>>>> Signed-off-by: Kele Huang <huangkele@bytedance.com>
>>>
>>> AVIC can change across migration.  I think we should instead use a new
>>> KVM_HINTS_* bit (KVM_HINTS_ACCELERATED_LAPIC or something like that).
>>> The KVM_HINTS_* bits are intended to be changeable across migration,
>>> even though we don't have for now anything equivalent to the Hyper-V
>>> reenlightenment interrupt.
>>
>> Note that the same issue exists with HyperV. It also has PV APIC,
>> which is harmful when AVIC is enabled (that is guest uses it instead
>> of using AVIC, negating AVIC benefits).
>>
>> Also note that Intel recently posted IPI virtualizaion, which
>> will make this issue relevant to APICv too soon.
> 
> The recently posted Intel IPI virtualization will accelerate unicast
> ipi but not broadcast ipis, AMD AVIC accelerates unicast ipi well but
> accelerates broadcast ipis worse than pv ipis. Could we just handle
> unicast ipi here?
> 
>      Wanpeng
> 
Depend on the number of target vCPUs, broadcast IPIs gets unstable 
performance on AVIC, and usually worse than PV Send IPI.
So agree with Wanpeng's point, is it possible to separate single IPI and 
broadcast IPI on a hardware acceleration platform?

-- 
zhenwei pi

  reply	other threads:[~2021-11-16  5:38 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-11-08  9:59 Kele Huang
2021-11-08 10:30 ` Paolo Bonzini
2021-11-08 11:08   ` Maxim Levitsky
2021-11-08 11:14     ` zhenwei pi
2021-11-08 11:18       ` Paolo Bonzini
2021-11-16  2:48     ` Wanpeng Li
2021-11-16  2:56       ` zhenwei pi [this message]
2021-11-16  9:06         ` Chao Gao
2021-11-16  9:30           ` [External] " 黄科乐
2021-11-16  9:30           ` Wanpeng Li
     [not found]           ` <CAKUug92xp7mU_KB66jGtdYRhgQpgfCm67r+3kMOMdbrGOrTQcA@mail.gmail.com>
2021-11-16 15:57             ` [External] " Sean Christopherson
2021-11-08 10:45 ` Vitaly Kuznetsov
2021-11-16  2:04 ` Wanpeng Li

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=d65fbd73-7612-8348-2fd8-8da0f5e2a3c0@bytedance.com \
    --to=pizhenwei@bytedance.com \
    --cc=bp@alien8.de \
    --cc=chaiwen.cc@bytedance.com \
    --cc=dave.hansen@linux.intel.com \
    --cc=dengliang.1214@bytedance.com \
    --cc=hpa@zytor.com \
    --cc=huangkele@bytedance.com \
    --cc=jmattson@google.com \
    --cc=joro@8bytes.org \
    --cc=kernellwp@gmail.com \
    --cc=kvm@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@redhat.com \
    --cc=mlevitsk@redhat.com \
    --cc=pbonzini@redhat.com \
    --cc=seanjc@google.com \
    --cc=tglx@linutronix.de \
    --cc=vkuznets@redhat.com \
    --cc=wanpengli@tencent.com \
    --cc=x86@kernel.org \
    --cc=xieyongji@bytedance.com \
    --subject='Re: Re: [RFC] KVM: x86: SVM: don'\''t expose PV_SEND_IPI feature with AVIC' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).