LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: "Wang, Zhi A" <zhi.a.wang@intel.com>
To: Jason Gunthorpe <jgg@nvidia.com>,
	Gerd Hoffmann <kraxel@redhat.com>,
	"Greg KH" <gregkh@linuxfoundation.org>
Cc: Christoph Hellwig <hch@lst.de>,
	Jani Nikula <jani.nikula@linux.intel.com>,
	Joonas Lahtinen <joonas.lahtinen@linux.intel.com>,
	"Vivi, Rodrigo" <rodrigo.vivi@intel.com>,
	Zhenyu Wang <zhenyuw@linux.intel.com>,
	"intel-gfx@lists.freedesktop.org"
	<intel-gfx@lists.freedesktop.org>,
	"intel-gvt-dev@lists.freedesktop.org" 
	<intel-gvt-dev@lists.freedesktop.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"dri-devel@lists.freedesktop.org"
	<dri-devel@lists.freedesktop.org>
Subject: RE: refactor the i915 GVT support
Date: Wed, 28 Jul 2021 13:38:58 +0000	[thread overview]
Message-ID: <DM4PR11MB5549EC882AA6076F3468274DCAEA9@DM4PR11MB5549.namprd11.prod.outlook.com> (raw)
In-Reply-To: <20210727121224.GA2145868@nvidia.com>

Hi Jason:

I guess those APIs you were talking about are KVM-only. For other hypervisors, e.g. Xen, ARCN cannot use the APIs you mentioned. Not sure if you have already noticed that VFIO is KVM-only right now.

GVT-g is designed for many hypervisors not only KVM. In the design, we implemented an abstraction layer for different hypervisors. You can check the link in the previous email which has an example of how the MPT module "xengt" supports GVT-g running under Xen. 
For example, injecting a msi in VFIO/KVM is via playing with eventfd. But in Xen, we need to issue a hypercall from Dom0. So does others, like querying mappings between GFN and HFN. Some GPU related emulation logic might be implemented differently under different hypervisors because different hypervisors might provide not exact the APIs we want. That's the reason why they get a callback in the MPT (yet not perfect.)  

As you can see, to survive from this situation, we have to rely on an abstraction layer so that we can prevent introducing coding blocks like in the core logic:

If (in_hypervisor_xen)
	Issue hypercalls
else if (in_hypervisor_kvm)
	Play with eventfds.
Else if (in_hypervisor_other)
	xxxx

Thus some of the APIs have to be wrapped in the MPT module in GVT-g design.

Sadly, not all customers are motivated or allowed to get their hypervisor-specific modules into the kernel. We have a customer who runs GVT-g with their private hypervisor. In this case, they don't want to get their "xxxgt" MPT module into upstream since their hypervisor has been in the kernel yet. Also, we have customers who ported the GVT-g to QNX which is another widely used commercial hypervisor in the industry. They can't get the "qnxgt" MPT module into upstream since it's not allowed.

We do understand the situation and try to figure out a solution that can fulfill expectations from different people in the community and also customers. 

According to Greg KH's comments, we are collecting the requirements of MPT modules from other open-source hypervisors in the kernel, e.g. ACRN, to see if they can get another MPT module into the kernel, which will show an example that how the MPT abstraction can benefit. Also, we are evaluating the impact on our customers if we have to remove MPT abstraction in the kernel because there is only one MPT module. 

Thanks so much for the comments.

Thanks,
Zhi.

-----Original Message-----
From: Jason Gunthorpe <jgg@nvidia.com> 
Sent: Tuesday, July 27, 2021 3:12 PM
To: Gerd Hoffmann <kraxel@redhat.com>
Cc: Wang, Zhi A <zhi.a.wang@intel.com>; Christoph Hellwig <hch@lst.de>; Jani Nikula <jani.nikula@linux.intel.com>; Joonas Lahtinen <joonas.lahtinen@linux.intel.com>; Vivi, Rodrigo <rodrigo.vivi@intel.com>; Zhenyu Wang <zhenyuw@linux.intel.com>; intel-gfx@lists.freedesktop.org; intel-gvt-dev@lists.freedesktop.org; linux-kernel@vger.kernel.org; dri-devel@lists.freedesktop.org
Subject: Re: refactor the i915 GVT support

On Thu, Jul 22, 2021 at 01:26:36PM +0200, Gerd Hoffmann wrote:
>   Hi,
> 
> > https://github.com/intel/gvt-linux/blob/topic/gvt-xengt/drivers/gpu/
> > drm/i915/gvt/xengt.c
> 
> > But it's hard for some customers to contribute their own "hypervisor"
> > module to the upstream Linux kernel. I am thinking what would be a 
> > better solution here? The MPT layer in the kernel helps a lot for 
> > customers, but only one open-source "hypervisor" module is there in 
> > the kernel. That can confuse people which don't know the story.  One 
> > thing I was thinking is to put a document about the background and 
> > more description in the MPT headers. So it won't confuse more people.
> 
> Getting the xengt module linked above merged into mainline would also 
> nicely explain why there are hypervisor modules.

It would also be nice to explain why a GPU driver needs a hypervisor specific shim like this in the first place.

        enum hypervisor_type type;
        int (*host_init)(struct device *dev, void *gvt, const void *ops);
        void (*host_exit)(struct device *dev, void *gvt);
        int (*attach_vgpu)(void *vgpu, unsigned long *handle);
        void (*detach_vgpu)(void *vgpu);

Doesn't vfio provide all this generically with notifiers?

        int (*inject_msi)(unsigned long handle, u32 addr, u16 data);

Isn't this one just an eventfd?

        unsigned long (*from_virt_to_mfn)(void *p);
        int (*read_gpa)(unsigned long handle, unsigned long gpa, void *buf,
                        unsigned long len);
        int (*write_gpa)(unsigned long handle, unsigned long gpa, void *buf,
                         unsigned long len);
        unsigned long (*gfn_to_mfn)(unsigned long handle, unsigned long gfn);

        int (*dma_map_guest_page)(unsigned long handle, unsigned long gfn,
                                  unsigned long size, dma_addr_t *dma_addr);
        void (*dma_unmap_guest_page)(unsigned long handle, dma_addr_t dma_addr);

        int (*dma_pin_guest_page)(unsigned long handle, dma_addr_t dma_addr);

        int (*map_gfn_to_mfn)(unsigned long handle, unsigned long gfn,
                              unsigned long mfn, unsigned int nr, bool map);
        bool (*is_valid_gfn)(unsigned long handle, unsigned long gfn);

Shouldn't the vfio page SW IOMMU do all of this generically?

        int (*enable_page_track)(unsigned long handle, u64 gfn);
        int (*disable_page_track)(unsigned long handle, u64 gfn);
        int (*set_trap_area)(unsigned long handle, u64 start, u64 end,
                             bool map);
        int (*set_opregion)(void *vgpu);
        int (*set_edid)(void *vgpu, int port_num);

edid depends on hypervisor??

        int (*get_vfio_device)(void *vgpu);
        void (*put_vfio_device)(void *vgpu);

Jason

  reply	other threads:[~2021-07-28 13:39 UTC|newest]

Thread overview: 60+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-07-21 15:53 Christoph Hellwig
2021-07-21 15:53 ` [PATCH 01/21] drm/i915/gvt: integrate into the main Makefile Christoph Hellwig
2021-08-03  9:27   ` Zhenyu Wang
2021-07-21 15:53 ` [PATCH 02/21] drm/i915/gvt: remove module refcounting in intel_gvt_{,un}register_hypervisor Christoph Hellwig
2021-07-21 15:53 ` [PATCH 03/21] drm/i915/gvt: remove enum hypervisor_type Christoph Hellwig
2021-07-21 15:53 ` [PATCH 04/21] drm/i915/gvt: move the gvt code into kvmgt.ko Christoph Hellwig
2021-08-09 11:29   ` Joonas Lahtinen
2021-08-09 14:29     ` Christoph Hellwig
2021-07-21 15:53 ` [PATCH 05/21] drm/i915/gvt: remove intel_gvt_ops Christoph Hellwig
2021-07-21 15:53 ` [PATCH 06/21] drm/i915/gvt: remove the map_gfn_to_mfn and set_trap_area ops Christoph Hellwig
2021-07-21 15:53 ` [PATCH 07/21] drm/i915/gvt: remove the unused from_virt_to_mfn op Christoph Hellwig
2021-07-21 15:53 ` [PATCH 08/21] drm/i915/gvt: merge struct kvmgt_vdev into struct intel_vgpu Christoph Hellwig
2021-07-21 15:53 ` [PATCH 09/21] drm/i915/gvt: merge struct kvmgt_guest_info into strut intel_vgpu Christoph Hellwig
2021-07-21 15:53 ` [PATCH 10/21] drm/i915/gvt: remove vgpu->handle Christoph Hellwig
2021-07-21 15:53 ` [PATCH 11/21] drm/i915/gvt: devirtualize ->{read,write}_gpa Christoph Hellwig
2021-07-21 15:53 ` [PATCH 12/21] drm/i915/gvt: devirtualize ->{get,put}_vfio_device Christoph Hellwig
2021-07-21 15:53 ` [PATCH 13/21] drm/i915/gvt: devirtualize ->set_edid and ->set_opregion Christoph Hellwig
2021-07-21 15:53 ` [PATCH 14/21] drm/i915/gvt: devirtualize ->detach_vgpu Christoph Hellwig
2021-07-21 15:53 ` [PATCH 15/21] drm/i915/gvt: devirtualize ->inject_msi Christoph Hellwig
2021-07-21 15:53 ` [PATCH 16/21] drm/i915/gvt: devirtualize ->is_valid_gfn Christoph Hellwig
2021-07-21 15:53 ` [PATCH 17/21] drm/i915/gvt: devirtualize ->gfn_to_mfn Christoph Hellwig
2021-07-21 15:53 ` [PATCH 18/21] drm/i915/gvt: devirtualize ->{enable,disable}_page_track Christoph Hellwig
2021-07-21 15:53 ` [PATCH 19/21] drm/i915/gvt: devirtualize ->dma_{,un}map_guest_page Christoph Hellwig
2021-07-21 15:53 ` [PATCH 20/21] drm/i915/gvt: devirtualize dma_pin_guest_page Christoph Hellwig
2021-07-21 15:53 ` [PATCH 21/21] drm/i915/gvt: remove struct intel_gvt_mpt Christoph Hellwig
2021-07-22  9:45 ` refactor the i915 GVT support Zhenyu Wang
2021-07-22 12:55   ` Christoph Hellwig
2021-07-22 10:49 ` Wang, Zhi A
2021-07-22 11:26   ` Gerd Hoffmann
2021-07-27 12:12     ` Jason Gunthorpe
2021-07-28 13:38       ` Wang, Zhi A [this message]
2021-07-28 13:43         ` Greg KH
2021-07-28 17:59         ` Jason Gunthorpe
2021-07-29  7:20           ` Christoph Hellwig
2021-07-29  7:30             ` [Intel-gfx] " Daniel Vetter
2021-08-03  9:43             ` Zhenyu Wang
2021-08-03 14:30               ` Jason Gunthorpe
2021-08-04  5:26                 ` Zhenyu Wang
2021-08-16 17:34                   ` Christoph Hellwig
2021-08-17  1:08                     ` Zhenyu Wang
2021-08-17  5:22                       ` Zhenyu Wang
2021-08-19  8:29                         ` Zhenyu Wang
2021-08-19 14:43                           ` Joonas Lahtinen
2021-08-26  6:04                             ` Zhenyu Wang
2021-08-20 14:17                           ` Christoph Hellwig
2021-08-20 19:56                             ` Luis Chamberlain
2021-08-26  6:12                               ` Zhenyu Wang
2021-09-28  7:41                                 ` Wang, Zhi A
2021-09-28 14:00                                   ` Luis Chamberlain
2021-09-28 14:35                                     ` Wang, Zhi A
2021-09-28 15:05                                       ` Jason Gunthorpe
2021-09-29 18:27                                         ` Wang, Zhi A
2021-09-29 18:55                                           ` Jason Gunthorpe
2021-10-01 13:01                                             ` Wang, Zhi A
2021-10-05  7:33                                               ` Wang, Zhi A
2021-09-30  5:24                                           ` Christoph Hellwig
2021-08-26  6:08                             ` Zhenyu Wang
2021-08-04  5:40                 ` Christoph Hellwig
2021-07-29  8:19           ` Wang, Zhi A
2021-07-22 13:16   ` Greg KH

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=DM4PR11MB5549EC882AA6076F3468274DCAEA9@DM4PR11MB5549.namprd11.prod.outlook.com \
    --to=zhi.a.wang@intel.com \
    --cc=dri-devel@lists.freedesktop.org \
    --cc=gregkh@linuxfoundation.org \
    --cc=hch@lst.de \
    --cc=intel-gfx@lists.freedesktop.org \
    --cc=intel-gvt-dev@lists.freedesktop.org \
    --cc=jani.nikula@linux.intel.com \
    --cc=jgg@nvidia.com \
    --cc=joonas.lahtinen@linux.intel.com \
    --cc=kraxel@redhat.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=rodrigo.vivi@intel.com \
    --cc=zhenyuw@linux.intel.com \
    --subject='RE: refactor the i915 GVT support' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).