LKML Archive on
help / color / mirror / Atom feed
From: Russell King - ARM Linux <>
To: Hante Meuleman <>
Cc: linux-wireless <>,
	brcm80211-dev-list <>,
	Will Deacon <>,
	"" <>,
	Arend Van Spriel <>,
	David Miller <>,
	Marek Szyprowski <>
Subject: Re: using DMA-API on ARM
Date: Fri, 5 Dec 2014 15:06:48 +0000	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <>

I've been doing more digging into the current DMA code, and I'm dismayed
to see that there's new bugs in it...

commit 513510ddba9650fc7da456eefeb0ead7632324f6
Author: Laura Abbott <>
Date:   Thu Oct 9 15:26:40 2014 -0700

    common: dma-mapping: introduce common remapping functions

This uses map_vm_area() to achieve the remapping of pages allocated inside
dma_alloc_coherent().  dma_alloc_coherent() is documented in a rather
round-about way in Documentation/DMA-API.txt:

| Part Ia - Using large DMA-coherent buffers
| ------------------------------------------
| void *
| dma_alloc_coherent(struct device *dev, size_t size,
|                              dma_addr_t *dma_handle, gfp_t flag)
| void
| dma_free_coherent(struct device *dev, size_t size, void *cpu_addr,
|                            dma_addr_t dma_handle)
| Free a region of consistent memory you previously allocated.  dev,
| size and dma_handle must all be the same as those passed into
| dma_alloc_coherent().  cpu_addr must be the virtual address returned by
| the dma_alloc_coherent().
| Note that unlike their sibling allocation calls, these routines
| may only be called with IRQs enabled.

Note that very last paragraph.  What this says is that it is explicitly
permitted to call dma_alloc_coherent() with IRQs disabled.

Now, the question is: is it safe to call map_vm_area() with IRQs disabled?
Well, map_vm_area() calls pud_alloc(), pmd_alloc(), and pte_alloc_kernel().
These functions all call into the kernel memory allocator *without*
GFP_ATOMIC - in other words, these allocations are permitted to sleep.
Except, IRQs are off, so it's a bug to call these functions from

Now, if we look at the previous code, it used ioremap_page_range().  This
has the same problem: it needs to allocate page tables, and it can only
do it via functions which may sleep.

If we go back even further, we find that the use of ioremap_page_range()
in dma_alloc_coherent() was introduced by:

commit e9da6e9905e639b0f842a244bc770b48ad0523e9
Author: Marek Szyprowski <>
Date:   Mon Jul 30 09:11:33 2012 +0200

    ARM: dma-mapping: remove custom consistent dma region

which is the commit which removed my pre-allocated page tables for the
DMA re-mapping region - code which I explicitly had to specifically
avoid this issue.

Obviously, this isn't a big problem, because people haven't reported
that they've hit any of the might_sleep() checks in the memory
allocators, which I think is our only saving grace - but it's still
wrong to the specified calling conditions of the DMA API.

If the problem which you (Broadcom) are suffering from is down to the
issue I suspect (that being having mappings with different cache
attributes) then I'm not sure that there's anything we can realistically
do about that.  There's a number of issues which make it hard to see a
way forward.

One example is that if we allocate memory, we need to be able to change
(or remove) the cacheable mappings associated with that memory.  We'd
need to touch the L1 page table, either to change the attributes of the
section mapping, or to convert the section mapping to a L2 page table
pointer.  We need to change the attributes in a break-flush-make sequence
to avoid TLB conflicts.

However, those mappings may be shared between other CPUs in a SMP system.
So, we would need to flush the TLBs on other CPUs before we could proceed
to create replacement mappings.  That means something like stop_machine()
or sending (and waiting for completion) of an IPI to the other CPUs.  That
is totally impractical due to dma_alloc_coherent() being allowed to be
called with IRQs off.

I'll continue to think about it, but I don't see many possibilities to
satisfy dma_alloc_coherent()'s documented requirements other than by
pre-allocating a chunk of memory at boot time to be served out as
DMA-able memory for these horrid cases.

I don't see much point in keeping the map_vm_area() approach on ARM
even if we did fallback - if we're re-establishing mappings for
the surrounding pages in lowmem, we might as well insert appropriately
attributed mappings for the DMA memory as well.

On the face of it, it would be better to allocate one section at a
time, but my unfortunate experience is that 3.x kernels are a /lot/
more trigger happy with the OOM killer, and the chances of being able
to allocate 1MB of memory at a go after the system has been running
for a while is near-on impossible.  So I don't think that's a reality.

Even if we did break up section mappings in this way, it would also
mean that over time, we'd end up with much of lowmem mapped using 4K
page table entries, which would place significant pressure on the MMU

So, we might just be far better off pre-allocating enough "DMA
coherent" memory at boot time and be done with it.  Those who want
it dynamic can use CMA instead.

FTTC broadband for 0.8mile line: currently at 9.5Mbps down 400kbps up
according to

  parent reply	other threads:[~2014-12-05 15:07 UTC|newest]

Thread overview: 41+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2014-12-05  9:22 Arend van Spriel
2014-12-05  9:45 ` Russell King - ARM Linux
2014-12-05 12:24   ` Will Deacon
2014-12-05 12:56     ` Hante Meuleman
2014-12-05 13:23       ` Russell King - ARM Linux
2014-12-05 14:20         ` Hante Meuleman
2014-12-05 14:47           ` Arend van Spriel
2014-12-08 13:47           ` Hante Meuleman
2014-12-08 15:01             ` Arnd Bergmann
2014-12-08 15:17               ` Russell King - ARM Linux
2014-12-08 15:22                 ` Arnd Bergmann
2014-12-08 16:03               ` Catalin Marinas
2014-12-08 17:01                 ` Arend van Spriel
2014-12-09 10:19                   ` Arend van Spriel
2014-12-09 10:29                     ` Russell King - ARM Linux
2014-12-09 11:07                       ` Arend van Spriel
2014-12-09 11:54                       ` Catalin Marinas
2015-01-20 15:22                       ` Fabio Estevam
2014-12-08 16:22               ` Arend van Spriel
2014-12-08 16:38                 ` Arnd Bergmann
2014-12-08 16:47                   ` Russell King - ARM Linux
2014-12-08 16:50                   ` Catalin Marinas
2014-12-08 16:54                     ` Russell King - ARM Linux
2014-12-05 15:06         ` Russell King - ARM Linux [this message]
2014-12-05 18:28           ` Catalin Marinas
2014-12-05 19:22             ` Arend van Spriel
2014-12-05 19:25               ` Russell King - ARM Linux
2014-12-05 12:43   ` Arend van Spriel
2014-12-05 12:59     ` Russell King - ARM Linux
2014-12-05  9:52 ` Arnd Bergmann
2014-12-05 11:11   ` Russell King - ARM Linux
2014-12-05 11:49     ` Hante Meuleman
2014-12-05 17:38     ` Catalin Marinas
2014-12-05 18:24       ` Russell King - ARM Linux
2014-12-05 18:31         ` Catalin Marinas
2014-12-05 18:39 ` Catalin Marinas
2014-12-05 18:53   ` Catalin Marinas
2014-12-05 19:50     ` Arend van Spriel
2014-12-08 12:55     ` Johannes Stezenbach
2014-12-08 15:55       ` Catalin Marinas
2014-12-08 16:50         ` Johannes Stezenbach

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \ \ \ \ \
    --subject='Re: using DMA-API on ARM' \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).