LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
* [PATCH RCFv2 0/7] mm: online/offline 4MB chunks controlled by device driver
@ 2018-04-30  9:42 David Hildenbrand
  2018-04-30  9:42 ` [PATCH RCFv2 1/7] mm: introduce and use PageOffline() David Hildenbrand
                   ` (7 more replies)
  0 siblings, 8 replies; 12+ messages in thread
From: David Hildenbrand @ 2018-04-30  9:42 UTC (permalink / raw)
  To: linux-mm
  Cc: linux-kernel, David Hildenbrand, Andrew Morton, Balbir Singh,
	Baoquan He, Benjamin Herrenschmidt, Boris Ostrovsky,
	Dan Williams, Dave Young, Greg Kroah-Hartman, Hari Bathini,
	Huang Ying, Hugh Dickins, Ingo Molnar, Jan Kara,
	Jérôme Glisse, Joonsoo Kim, Juergen Gross,
	Kirill A. Shutemov, Matthew Wilcox, Mel Gorman, Michael Ellerman,
	Michal Hocko, Miles Chen, Paul Mackerras, Pavel Tatashin,
	Philippe Ombredanne, Rashmica Gupta, Reza Arbab,
	Souptick Joarder, Tetsuo Handa, Thomas Gleixner, Vlastimil Babka

I am right now working on a paravirtualized memory device ("virtio-mem").
These devices control a memory region and the amount of memory available
via it. Memory will not be indicated/added/onlined via ACPI and friends,
the device driver is responsible for it.

When the device driver starts up, it will add and online the requested
amount of memory from its assigned physical memory region. On request, it can
add (online) either more memory or try to remove (offline) memory. As it
will be a virtio module, we also want to be able to have it as a loadable
kernel module.

Such a device can be thought of like a "resizable DIMM" or a "huge
number of 4MB DIMMS" that can be automatically managed.

As we want to be able to add/remove small chunks of memory to a VM without
fragmenting guest memory ("it's not what the guest pays for" and "what if
the hypervisor wants to sue huge pages"), it looks like we can do that
under Linux in a 4MB granularity by using online_pages()/offline_pages()

We add a segment and online only 4MB blocks of it on demand. So the other
memory might not be accessible. For kdump and offlining code, we have to
mark pages as offline before a new segment is visible to the system (e.g.
as these pages might not be backed by real memory in the hypervisor).

This is not a balloon driver. Main differences:
- We can add more memory to a VM without having to use mixture of
  technologies - e.g. ACPI for plugging, balloon for unplugging (in contrast
  to virtio-balloon).
- The device is responsible for its own memory only - will not inflate on
  any system memory. (in contrast to all balloons)
- Works on a coarser granularity (e.g. 4MB because that's what we can
  online/offline in Linux). We are not using the buddy allocator when unplugging
  but really search for chunks of memory we can offline. We actually
  can support arbitrary block sizes. (in contrast to all balloons)
- That's why we don't fragment guest memory.
- A device can belong to exactly one NUMA node. This way we can online/offline
  memory in a fine granularity NUMA aware. Even if the guest does not even
  know how to spell NUMA. (in contrast to all balloons)
- Architectures that don't have proper memory hotplug interfaces (e.g. s390x)
  get memory hotplug support. I have a prototype for s390x.
- Once all 4MB chunks of a memory block are offline, we can remove the
  memory block and therefore the struct pages. (in contrast to all balloons)

This essentially allows us to add/remove 4MB chunks to/from a VM. Especially
without caring about the future when adding memory ("If I add a 128GB DIMM
I can only unplug 128GB again") or running into limits ("If I want my VM to
grow to 4TB, I have to plug at least 16GB per DIMM").

Future work:
 - Performance improvements
 - Be smarter about which blocks to offline first (e.g. free ones)
 - Automatically manage assignemnt to NORMAL/MOVABLE zone to make
   unplug more likely to succeed.

I will post the next prototype of virtio-mem shortly.

RFC -> RFCv2:
 - "mm: introduce and use PageOffline()"
  -> Use a mapcount value instead of a page flag
  -> Rework to not require to revert a patch completely
 - "kdump: include PAGE_OFFLINE_MAPCOUNT_VALUE in ELF info$"
  -> Export the mapcount value instead
 - "mm/memory_hotplug: limit offline_pages() to sizes we can actually .."
  -> Make this look a bit nicer and drivers to also use the size
 - "mm/memory_hotplug: print only with DEBUG_VM in offline_pages()"
  -> offlining is right now fairly noisy when delaing with small chunks
 - "mm/memory_hotplug: teach offline_pages() to not try forever"
  -> We need offline_pages() to fail fast and not loop forever on persistent
     errors (e.g. -ENOMEM)
 - "mm/memory_hotplug: allow online/offline memory by a kernel module"
  -> Actually compiled it as a module and noticed that a lot was still missing

David Hildenbrand (7):
  mm: introduce and use PageOffline()
  kdump: include PAGE_OFFLINE_MAPCOUNT_VALUE in ELF info
  mm/memory_hotplug: limit offline_pages() to sizes we can actually
    handle
  mm/memory_hotplug: allow to control onlining/offlining of memory by a
    driver
  mm/memory_hotplug: print only with DEBUG_VM in offline_pages()
  mm/memory_hotplug: teach offline_pages() to not try forever
  mm/memory_hotplug: allow online/offline memory by a kernel module

 arch/powerpc/platforms/powernv/memtrace.c |   2 +-
 drivers/base/memory.c                     |  25 ++++--
 drivers/base/node.c                       |   1 -
 drivers/xen/balloon.c                     |   2 +-
 include/linux/memory.h                    |   2 +-
 include/linux/memory_hotplug.h            |  20 +++--
 include/linux/mm.h                        |   2 +
 include/linux/page-flags.h                |   9 ++
 kernel/crash_core.c                       |   1 +
 mm/memory_hotplug.c                       | 131 +++++++++++++++++++++++++-----
 mm/page_alloc.c                           |  22 +++--
 mm/sparse.c                               |  25 +++++-
 12 files changed, 195 insertions(+), 47 deletions(-)

-- 
2.14.3

^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2018-05-09 14:14 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-04-30  9:42 [PATCH RCFv2 0/7] mm: online/offline 4MB chunks controlled by device driver David Hildenbrand
2018-04-30  9:42 ` [PATCH RCFv2 1/7] mm: introduce and use PageOffline() David Hildenbrand
2018-04-30 14:35   ` Pavel Tatashin
2018-04-30 15:17     ` David Hildenbrand
2018-04-30 15:49       ` David Hildenbrand
2018-04-30  9:42 ` [PATCH RCFv2 2/7] kdump: include PAGE_OFFLINE_MAPCOUNT_VALUE in ELF info David Hildenbrand
2018-04-30  9:42 ` [PATCH RCFv2 3/7] mm/memory_hotplug: limit offline_pages() to sizes we can actually handle David Hildenbrand
2018-04-30  9:42 ` [PATCH RCFv2 4/7] mm/memory_hotplug: allow to control onlining/offlining of memory by a driver David Hildenbrand
2018-04-30  9:42 ` [PATCH RCFv2 5/7] mm/memory_hotplug: print only with DEBUG_VM in offline_pages() David Hildenbrand
2018-04-30  9:42 ` [PATCH RCFv2 6/7] mm/memory_hotplug: teach offline_pages() to not try forever David Hildenbrand
2018-04-30  9:42 ` [PATCH RCFv2 7/7] mm/memory_hotplug: allow online/offline memory by a kernel module David Hildenbrand
2018-05-09 14:14 ` [PATCH RCFv2 0/7] mm: online/offline 4MB chunks controlled by device driver David Hildenbrand

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).