LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: David Hildenbrand <david@redhat.com>
To: linux-mm@kvack.org
Cc: linux-kernel@vger.kernel.org,
	"Andrew Morton" <akpm@linux-foundation.org>,
	"Balbir Singh" <bsingharora@gmail.com>,
	"Baoquan He" <bhe@redhat.com>,
	"Benjamin Herrenschmidt" <benh@kernel.crashing.org>,
	"Boris Ostrovsky" <boris.ostrovsky@oracle.com>,
	"Dan Williams" <dan.j.williams@intel.com>,
	"Dave Young" <dyoung@redhat.com>,
	"Greg Kroah-Hartman" <gregkh@linuxfoundation.org>,
	"Hari Bathini" <hbathini@linux.vnet.ibm.com>,
	"Huang Ying" <ying.huang@intel.com>,
	"Hugh Dickins" <hughd@google.com>,
	"Ingo Molnar" <mingo@kernel.org>, "Jan Kara" <jack@suse.cz>,
	"Jérôme Glisse" <jglisse@redhat.com>,
	"Joonsoo Kim" <iamjoonsoo.kim@lge.com>,
	"Juergen Gross" <jgross@suse.com>,
	"Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>,
	"Matthew Wilcox" <mawilcox@microsoft.com>,
	"Mel Gorman" <mgorman@suse.de>,
	"Michael Ellerman" <mpe@ellerman.id.au>,
	"Michal Hocko" <mhocko@suse.com>,
	"Miles Chen" <miles.chen@mediatek.com>,
	"Paul Mackerras" <paulus@samba.org>,
	"Pavel Tatashin" <pasha.tatashin@oracle.com>,
	"Philippe Ombredanne" <pombredanne@nexb.com>,
	"Rashmica Gupta" <rashmica.g@gmail.com>,
	"Reza Arbab" <arbab@linux.vnet.ibm.com>,
	"Souptick Joarder" <jrdr.linux@gmail.com>,
	"Tetsuo Handa" <penguin-kernel@I-love.SAKURA.ne.jp>,
	"Thomas Gleixner" <tglx@linutronix.de>,
	"Vlastimil Babka" <vbabka@suse.cz>
Subject: Re: [PATCH RCFv2 0/7] mm: online/offline 4MB chunks controlled by device driver
Date: Wed, 9 May 2018 16:14:40 +0200	[thread overview]
Message-ID: <ce2abc0b-8b6c-0a9a-30a5-31c185b5f8f8@redhat.com> (raw)
In-Reply-To: <20180430094236.29056-1-david@redhat.com>

On 30.04.2018 11:42, David Hildenbrand wrote:
> I am right now working on a paravirtualized memory device ("virtio-mem").
> These devices control a memory region and the amount of memory available
> via it. Memory will not be indicated/added/onlined via ACPI and friends,
> the device driver is responsible for it.
> 
> When the device driver starts up, it will add and online the requested
> amount of memory from its assigned physical memory region. On request, it can
> add (online) either more memory or try to remove (offline) memory. As it
> will be a virtio module, we also want to be able to have it as a loadable
> kernel module.
> 
> Such a device can be thought of like a "resizable DIMM" or a "huge
> number of 4MB DIMMS" that can be automatically managed.
> 
> As we want to be able to add/remove small chunks of memory to a VM without
> fragmenting guest memory ("it's not what the guest pays for" and "what if
> the hypervisor wants to sue huge pages"), it looks like we can do that
> under Linux in a 4MB granularity by using online_pages()/offline_pages()
> 
> We add a segment and online only 4MB blocks of it on demand. So the other
> memory might not be accessible. For kdump and offlining code, we have to
> mark pages as offline before a new segment is visible to the system (e.g.
> as these pages might not be backed by real memory in the hypervisor).
> 
> This is not a balloon driver. Main differences:
> - We can add more memory to a VM without having to use mixture of
>   technologies - e.g. ACPI for plugging, balloon for unplugging (in contrast
>   to virtio-balloon).
> - The device is responsible for its own memory only - will not inflate on
>   any system memory. (in contrast to all balloons)
> - Works on a coarser granularity (e.g. 4MB because that's what we can
>   online/offline in Linux). We are not using the buddy allocator when unplugging
>   but really search for chunks of memory we can offline. We actually
>   can support arbitrary block sizes. (in contrast to all balloons)
> - That's why we don't fragment guest memory.
> - A device can belong to exactly one NUMA node. This way we can online/offline
>   memory in a fine granularity NUMA aware. Even if the guest does not even
>   know how to spell NUMA. (in contrast to all balloons)
> - Architectures that don't have proper memory hotplug interfaces (e.g. s390x)
>   get memory hotplug support. I have a prototype for s390x.
> - Once all 4MB chunks of a memory block are offline, we can remove the
>   memory block and therefore the struct pages. (in contrast to all balloons)
> 
> This essentially allows us to add/remove 4MB chunks to/from a VM. Especially
> without caring about the future when adding memory ("If I add a 128GB DIMM
> I can only unplug 128GB again") or running into limits ("If I want my VM to
> grow to 4TB, I have to plug at least 16GB per DIMM").
> 
> Future work:
>  - Performance improvements
>  - Be smarter about which blocks to offline first (e.g. free ones)
>  - Automatically manage assignemnt to NORMAL/MOVABLE zone to make
>    unplug more likely to succeed.
> 
> I will post the next prototype of virtio-mem shortly.
> 

If there are no further comments, I'll send a v1 (!RFC) version, along
with the virtio-mem prototype after rebasing (assuming that nothing
breaks :) ).

-- 

Thanks,

David / dhildenb

      parent reply	other threads:[~2018-05-09 14:14 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-04-30  9:42 David Hildenbrand
2018-04-30  9:42 ` [PATCH RCFv2 1/7] mm: introduce and use PageOffline() David Hildenbrand
2018-04-30 14:35   ` Pavel Tatashin
2018-04-30 15:17     ` David Hildenbrand
2018-04-30 15:49       ` David Hildenbrand
2018-04-30  9:42 ` [PATCH RCFv2 2/7] kdump: include PAGE_OFFLINE_MAPCOUNT_VALUE in ELF info David Hildenbrand
2018-04-30  9:42 ` [PATCH RCFv2 3/7] mm/memory_hotplug: limit offline_pages() to sizes we can actually handle David Hildenbrand
2018-04-30  9:42 ` [PATCH RCFv2 4/7] mm/memory_hotplug: allow to control onlining/offlining of memory by a driver David Hildenbrand
2018-04-30  9:42 ` [PATCH RCFv2 5/7] mm/memory_hotplug: print only with DEBUG_VM in offline_pages() David Hildenbrand
2018-04-30  9:42 ` [PATCH RCFv2 6/7] mm/memory_hotplug: teach offline_pages() to not try forever David Hildenbrand
2018-04-30  9:42 ` [PATCH RCFv2 7/7] mm/memory_hotplug: allow online/offline memory by a kernel module David Hildenbrand
2018-05-09 14:14 ` David Hildenbrand [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=ce2abc0b-8b6c-0a9a-30a5-31c185b5f8f8@redhat.com \
    --to=david@redhat.com \
    --cc=akpm@linux-foundation.org \
    --cc=arbab@linux.vnet.ibm.com \
    --cc=benh@kernel.crashing.org \
    --cc=bhe@redhat.com \
    --cc=boris.ostrovsky@oracle.com \
    --cc=bsingharora@gmail.com \
    --cc=dan.j.williams@intel.com \
    --cc=dyoung@redhat.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=hbathini@linux.vnet.ibm.com \
    --cc=hughd@google.com \
    --cc=iamjoonsoo.kim@lge.com \
    --cc=jack@suse.cz \
    --cc=jglisse@redhat.com \
    --cc=jgross@suse.com \
    --cc=jrdr.linux@gmail.com \
    --cc=kirill.shutemov@linux.intel.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=mawilcox@microsoft.com \
    --cc=mgorman@suse.de \
    --cc=mhocko@suse.com \
    --cc=miles.chen@mediatek.com \
    --cc=mingo@kernel.org \
    --cc=mpe@ellerman.id.au \
    --cc=pasha.tatashin@oracle.com \
    --cc=paulus@samba.org \
    --cc=penguin-kernel@I-love.SAKURA.ne.jp \
    --cc=pombredanne@nexb.com \
    --cc=rashmica.g@gmail.com \
    --cc=tglx@linutronix.de \
    --cc=vbabka@suse.cz \
    --cc=ying.huang@intel.com \
    --subject='Re: [PATCH RCFv2 0/7] mm: online/offline 4MB chunks controlled by device driver' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).