LKML Archive on
help / color / mirror / Atom feed
From: David Hildenbrand <>
	"Andrew Morton" <>,
	"Balbir Singh" <>,
	"Baoquan He" <>,
	"Benjamin Herrenschmidt" <>,
	"Boris Ostrovsky" <>,
	"Dan Williams" <>,
	"Dave Young" <>,
	"Greg Kroah-Hartman" <>,
	"Hari Bathini" <>,
	"Huang Ying" <>,
	"Hugh Dickins" <>,
	"Ingo Molnar" <>, "Jan Kara" <>,
	"Jérôme Glisse" <>,
	"Joonsoo Kim" <>,
	"Juergen Gross" <>,
	"Kirill A. Shutemov" <>,
	"Matthew Wilcox" <>,
	"Mel Gorman" <>,
	"Michael Ellerman" <>,
	"Michal Hocko" <>,
	"Miles Chen" <>,
	"Paul Mackerras" <>,
	"Pavel Tatashin" <>,
	"Philippe Ombredanne" <>,
	"Rashmica Gupta" <>,
	"Reza Arbab" <>,
	"Souptick Joarder" <>,
	"Tetsuo Handa" <>,
	"Thomas Gleixner" <>,
	"Vlastimil Babka" <>
Subject: Re: [PATCH RCFv2 0/7] mm: online/offline 4MB chunks controlled by device driver
Date: Wed, 9 May 2018 16:14:40 +0200	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <>

On 30.04.2018 11:42, David Hildenbrand wrote:
> I am right now working on a paravirtualized memory device ("virtio-mem").
> These devices control a memory region and the amount of memory available
> via it. Memory will not be indicated/added/onlined via ACPI and friends,
> the device driver is responsible for it.
> When the device driver starts up, it will add and online the requested
> amount of memory from its assigned physical memory region. On request, it can
> add (online) either more memory or try to remove (offline) memory. As it
> will be a virtio module, we also want to be able to have it as a loadable
> kernel module.
> Such a device can be thought of like a "resizable DIMM" or a "huge
> number of 4MB DIMMS" that can be automatically managed.
> As we want to be able to add/remove small chunks of memory to a VM without
> fragmenting guest memory ("it's not what the guest pays for" and "what if
> the hypervisor wants to sue huge pages"), it looks like we can do that
> under Linux in a 4MB granularity by using online_pages()/offline_pages()
> We add a segment and online only 4MB blocks of it on demand. So the other
> memory might not be accessible. For kdump and offlining code, we have to
> mark pages as offline before a new segment is visible to the system (e.g.
> as these pages might not be backed by real memory in the hypervisor).
> This is not a balloon driver. Main differences:
> - We can add more memory to a VM without having to use mixture of
>   technologies - e.g. ACPI for plugging, balloon for unplugging (in contrast
>   to virtio-balloon).
> - The device is responsible for its own memory only - will not inflate on
>   any system memory. (in contrast to all balloons)
> - Works on a coarser granularity (e.g. 4MB because that's what we can
>   online/offline in Linux). We are not using the buddy allocator when unplugging
>   but really search for chunks of memory we can offline. We actually
>   can support arbitrary block sizes. (in contrast to all balloons)
> - That's why we don't fragment guest memory.
> - A device can belong to exactly one NUMA node. This way we can online/offline
>   memory in a fine granularity NUMA aware. Even if the guest does not even
>   know how to spell NUMA. (in contrast to all balloons)
> - Architectures that don't have proper memory hotplug interfaces (e.g. s390x)
>   get memory hotplug support. I have a prototype for s390x.
> - Once all 4MB chunks of a memory block are offline, we can remove the
>   memory block and therefore the struct pages. (in contrast to all balloons)
> This essentially allows us to add/remove 4MB chunks to/from a VM. Especially
> without caring about the future when adding memory ("If I add a 128GB DIMM
> I can only unplug 128GB again") or running into limits ("If I want my VM to
> grow to 4TB, I have to plug at least 16GB per DIMM").
> Future work:
>  - Performance improvements
>  - Be smarter about which blocks to offline first (e.g. free ones)
>  - Automatically manage assignemnt to NORMAL/MOVABLE zone to make
>    unplug more likely to succeed.
> I will post the next prototype of virtio-mem shortly.

If there are no further comments, I'll send a v1 (!RFC) version, along
with the virtio-mem prototype after rebasing (assuming that nothing
breaks :) ).



David / dhildenb

      parent reply	other threads:[~2018-05-09 14:14 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-04-30  9:42 David Hildenbrand
2018-04-30  9:42 ` [PATCH RCFv2 1/7] mm: introduce and use PageOffline() David Hildenbrand
2018-04-30 14:35   ` Pavel Tatashin
2018-04-30 15:17     ` David Hildenbrand
2018-04-30 15:49       ` David Hildenbrand
2018-04-30  9:42 ` [PATCH RCFv2 2/7] kdump: include PAGE_OFFLINE_MAPCOUNT_VALUE in ELF info David Hildenbrand
2018-04-30  9:42 ` [PATCH RCFv2 3/7] mm/memory_hotplug: limit offline_pages() to sizes we can actually handle David Hildenbrand
2018-04-30  9:42 ` [PATCH RCFv2 4/7] mm/memory_hotplug: allow to control onlining/offlining of memory by a driver David Hildenbrand
2018-04-30  9:42 ` [PATCH RCFv2 5/7] mm/memory_hotplug: print only with DEBUG_VM in offline_pages() David Hildenbrand
2018-04-30  9:42 ` [PATCH RCFv2 6/7] mm/memory_hotplug: teach offline_pages() to not try forever David Hildenbrand
2018-04-30  9:42 ` [PATCH RCFv2 7/7] mm/memory_hotplug: allow online/offline memory by a kernel module David Hildenbrand
2018-05-09 14:14 ` David Hildenbrand [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \
    --subject='Re: [PATCH RCFv2 0/7] mm: online/offline 4MB chunks controlled by device driver' \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).