LKML Archive on
help / color / mirror / Atom feed
From: "Leonardo Brás" <>
To: David Christensen <>,
	Michael Ellerman <>,
	Benjamin Herrenschmidt <>,
	Paul Mackerras <>,
	Leonardo Bras <>,
	Alexey Kardashevskiy <>,
	David Gibson <>,
	kernel test robot <>,
	Nicolin Chen <>,
	Frederic Barrat <>
Subject: Re: [PATCH v6 00/11] DDW + Indirect Mapping
Date: Tue, 31 Aug 2021 17:18:11 -0300	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <>

Hello David,

Sorry for the delay, I did not get your mail because I was not CC'd
in your reply (you sent the mail just to the mailing list).

Replies bellow:

On Mon, 2021-08-30 at 10:48 -0700, David Christensen wrote:
> On 8/16/21 11:39 PM, Leonardo Bras wrote:
> > So far it's assumed possible to map the guest RAM 1:1 to the bus,
> > which
> > works with a small number of devices. SRIOV changes it as the user
> > can
> > configure hundreds VFs and since phyp preallocates TCEs and does not
> > allow IOMMU pages bigger than 64K, it has to limit the number of TCEs
> > per a PE to limit waste of physical pages.
> > 
> > As of today, if the assumed direct mapping is not possible, DDW
> > creation
> > is skipped and the default DMA window "ibm,dma-window" is used
> > instead.
> > 
> > Using the DDW instead of the default DMA window may allow to expand
> > the
> > amount of memory that can be DMA-mapped, given the number of pages
> > (TCEs)
> > may stay the same (or increase) and the default DMA window offers
> > only
> > 4k-pages while DDW may offer larger pages (4k, 64k, 16M ...).
> So if I'm reading this correctly, VFIO applications requiring hugepage 
> DMA mappings (e.g. 16M or 2GB) can be supported on an LPAR or DLPAR 
> after this change, is that correct?  

Different DDW IOMMU page sizes were already supported in Linux (4k,
64k, 16M) for a while now, and the remaining page sizes in LoPAR were
enabled in the following patch:
(commit 472724111f0f72042deb6a9dcee9578e5398a1a1)

The thing is there are two ways of using DMA:
- Direct DMA, mapping the whole memory space of the host, which
requires a lot of DMA space if the guest memory is huge. This already
supports DDW and allows using the bigger pagesizes.
This happens on device/bus probe.

- Indirect DMA with IOMMU, mapping memory regions on demand, and un-
mapping after use. This requires much less DMA space, but causes an
overhead because an hcall is necessary for mapping and un-mapping.
Before this series, Indirect DMA was only possible with the 'default
DMA window' which allows using only 4k pages. 

This series allow Indirect DMA using DDW when available, which usually
means bigger pagesizes and more TCEs, and so more DMA space.

tl;dr this patchset means you can have more DMA space in Indirect DMA,
because you are using DDW instead of the Default DMA window.

> Any limitations based on processor 
> or pHyp revision levels?

The IOMMU page size will be limited by the sizes offered by processor
and hypervisor. They are announced at  "IO Page Sizes" output of
ibm,query-pe-dma-window, but now the biggest pagesize automatically
selected with commit 472724111f0f72042deb6a9dcee9578e5398a1a1 above

Hope this helps, please let me know if there is any remaining question.

Best regards,

  parent reply	other threads:[~2021-08-31 20:17 UTC|newest]

Thread overview: 21+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-08-17  6:39 Leonardo Bras
2021-08-17  6:39 ` [PATCH v6 01/11] powerpc/pseries/iommu: Replace hard-coded page shift Leonardo Bras
2021-08-17  6:39 ` [PATCH v6 02/11] powerpc/kernel/iommu: Add new iommu_table_in_use() helper Leonardo Bras
2021-08-27 16:52   ` Frederic Barrat
2021-08-17  6:39 ` [PATCH v6 03/11] powerpc/pseries/iommu: Add iommu_pseries_alloc_table() helper Leonardo Bras
2021-08-17  6:39 ` [PATCH v6 04/11] powerpc/pseries/iommu: Add ddw_list_new_entry() helper Leonardo Bras
2021-08-17  6:39 ` [PATCH v6 05/11] powerpc/pseries/iommu: Allow DDW windows starting at 0x00 Leonardo Bras
2021-08-17  6:39 ` [PATCH v6 06/11] powerpc/pseries/iommu: Add ddw_property_create() and refactor enable_ddw() Leonardo Bras
2021-08-27 16:55   ` Frederic Barrat
2021-08-17  6:39 ` [PATCH v6 07/11] powerpc/pseries/iommu: Reorganize iommu_table_setparms*() with new helper Leonardo Bras
2021-08-17  6:39 ` [PATCH v6 08/11] powerpc/pseries/iommu: Update remove_dma_window() to accept property name Leonardo Bras
2021-08-27 16:58   ` Frederic Barrat
2021-08-17  6:39 ` [PATCH v6 09/11] powerpc/pseries/iommu: Find existing DDW with given " Leonardo Bras
2021-08-17  6:39 ` [PATCH v6 10/11] powerpc/pseries/iommu: Make use of DDW for indirect mapping Leonardo Bras
2021-08-27 17:06   ` Frederic Barrat
2021-08-27 17:53     ` Leonardo Brás
2021-08-17  6:39 ` [PATCH v6 11/11] powerpc/pseries/iommu: Rename "direct window" to "dma window" Leonardo Bras
2021-08-31 13:56 ` [PATCH v6 00/11] DDW + Indirect Mapping Michael Ellerman
     [not found] ` <>
2021-08-31 20:18   ` Leonardo Brás [this message]
2021-08-31 20:39     ` David Christensen
2021-08-31 20:49       ` Leonardo Brás

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \ \ \ \ \ \ \
    --subject='Re: [PATCH v6 00/11] DDW + Indirect Mapping' \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).