Linux-Fsdevel Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Muchun Song <songmuchun@bytedance.com>
To: Mike Kravetz <mike.kravetz@oracle.com>
Cc: Jonathan Corbet <corbet@lwn.net>,
	Thomas Gleixner <tglx@linutronix.de>,
	mingo@redhat.com, bp@alien8.de, x86@kernel.org, hpa@zytor.com,
	dave.hansen@linux.intel.com, luto@kernel.org,
	Peter Zijlstra <peterz@infradead.org>,
	viro@zeniv.linux.org.uk,
	Andrew Morton <akpm@linux-foundation.org>,
	paulmck@kernel.org, mchehab+huawei@kernel.org,
	pawan.kumar.gupta@linux.intel.com,
	Matthew Wilcox <willy@infradead.org>,
	Randy Dunlap <rdunlap@infradead.org>,
	oneukum@suse.com, anshuman.khandual@arm.com, jroedel@suse.de,
	Mina Almasry <almasrymina@google.com>,
	David Rientjes <rientjes@google.com>,
	linux-doc@vger.kernel.org, LKML <linux-kernel@vger.kernel.org>,
	Linux Memory Management List <linux-mm@kvack.org>,
	linux-fsdevel@vger.kernel.org,
	Xiongchun duan <duanxiongchun@bytedance.com>
Subject: Re: [External] Re: [RFC PATCH 00/24] mm/hugetlb: Free some vmemmap pages of hugetlb page
Date: Fri, 9 Oct 2020 12:13:44 +0800	[thread overview]
Message-ID: <CAMZfGtXAB7CqNxp2Et=SSY4iPDbxS92cwcDEo2C_88m92JNWoQ@mail.gmail.com> (raw)
In-Reply-To: <9d220de0-f06d-cb5b-363f-6ae97d5b4146@oracle.com>

On Thu, Oct 8, 2020 at 5:15 AM Mike Kravetz <mike.kravetz@oracle.com> wrote:
>
> On 9/29/20 2:58 PM, Mike Kravetz wrote:
> > On 9/15/20 5:59 AM, Muchun Song wrote:
> >> Hi all,
> >>
> >> This patch series will free some vmemmap pages(struct page structures)
> >> associated with each hugetlbpage when preallocated to save memory.
> > ...
> >> The mapping of the first page(index 0) and the second page(index 1) is
> >> unchanged. The remaining 6 pages are all mapped to the same page(index
> >> 1). So we only need 2 pages for vmemmap area and free 6 pages to the
> >> buddy system to save memory. Why we can do this? Because the content
> >> of the remaining 7 pages are usually same except the first page.
> >>
> >> When a hugetlbpage is freed to the buddy system, we should allocate 6
> >> pages for vmemmap pages and restore the previous mapping relationship.
> >>
> >> If we uses the 1G hugetlbpage, we can save 4095 pages. This is a very
> >> substantial gain. On our server, run some SPDK applications which will
> >> use 300GB hugetlbpage. With this feature enabled, we can save 4797MB
> >> memory.
>
> I had a hard time going through the patch series as it is currently
> structured, and instead examined all the code together.  Muchun put in
> much effort and the code does reduce memory usage.
> - For 2MB hugetlb pages, we save 5 pages of struct pages
> - For 1GB hugetlb pages, we save 4086 pages of struct pages
>
> Code is even in pace to handle poisoned pages, although I have not looked
> at this closely.  The code survives the libhugetlbfs and ltp huge page tests.
>
> To date, nobody has asked the important question "Is the added complexity
> worth the memory savings?".  I suppose it all depends on one's use case.
> Obviously, the savings are more significant when one uses 1G huge pages but
> that may not be the common case today.
>
> > At a high level this seems like a reasonable optimization for hugetlb
> > pages.  It is possible because hugetlb pages are 'special' and mostly
> > handled differently than pages in normal mm paths.
>
> Such an optimization only makes sense for something like hugetlb pages.  One
> reason is the 'special' nature of hugetlbfs as stated above.  The other is
> that this optimization mostly makes sense for huge pages that are created
> once and stick around for a long time.  hugetlb pool pages are a perfect
> example.  This is because manipulation of struct page mappings is done when
> a huge page is created or destroyed.

Yeah, in our cloud server, we have some application scenarios(e.g. SPDK,
DPDK, QEMU and jemalloc). These applications may use a lot of hugetlb
pages.

>
> > The majority of the new code is hugetlb specific, so it should not be
> > of too much concern for the general mm code paths.
>
> It is true that much of the code in this series was put in hugetlb.c.  However,
> I would argue that there is a bunch of code that only deals with remapping
> the memmap which should more generic and added to sparse-vmemmap.c.  This
> would at least allow for easier reuse.

I agree with you.

>
> Before Muchun and myself put more effort into this series, I would really
> like to get feedback on the whether or not this should move forward.
> Specifically, is the memory savings worth added complexity?  Is the removing
> of struct pages going to come back and cause issues for future features?

Some users do need this optimization to save memory. But if some users
do not need this optimization, they also can disable it by using a kernel boot
parameter 'hugetlb_free_vmemmap=off' or not configuring
CONFIG_HUGETLB_PAGE_FREE_VMEMMAP.

I have no idea about "cause issues for future features". Is there any feature
ongoing or planned?

-- 
Yours,
Muchun

      reply	other threads:[~2020-10-09  4:14 UTC|newest]

Thread overview: 48+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-09-15 12:59 Muchun Song
2020-09-15 12:59 ` [RFC PATCH 01/24] mm/memory_hotplug: Move bootmem info registration API to bootmem_info.c Muchun Song
2020-09-29 22:46   ` Mike Kravetz
2020-09-15 12:59 ` [RFC PATCH 02/24] mm/memory_hotplug: Move {get,put}_page_bootmem() " Muchun Song
2020-09-29 23:30   ` Mike Kravetz
2020-09-15 12:59 ` [RFC PATCH 03/24] mm/hugetlb: Introduce a new config HUGETLB_PAGE_FREE_VMEMMAP Muchun Song
2020-09-16  2:13   ` Randy Dunlap
2020-09-29 23:41   ` Mike Kravetz
2020-09-30  2:56     ` [External] " Muchun Song
2020-09-15 12:59 ` [RFC PATCH 04/24] mm/hugetlb: Register bootmem info when CONFIG_HUGETLB_PAGE_FREE_VMEMMAP Muchun Song
2020-09-15 12:59 ` [RFC PATCH 05/24] mm/hugetlb: Introduce nr_free_vmemmap_pages in the struct hstate Muchun Song
2020-09-30 22:41   ` Mike Kravetz
2020-10-01  2:57     ` [External] " Muchun Song
2020-09-15 12:59 ` [RFC PATCH 06/24] mm/hugetlb: Introduce pgtable allocation/freeing helpers Muchun Song
2020-09-15 12:59 ` [RFC PATCH 07/24] mm/hugetlb: Add freeing unused vmemmap pages support for x86 Muchun Song
2020-09-15 12:59 ` [RFC PATCH 08/24] mm/bootmem_info: Introduce {free,prepare}_vmemmap_page() Muchun Song
2020-09-15 12:59 ` [RFC PATCH 09/24] x86/mm: Introduce VMEMMAP_SIZE/VMEMMAP_END macro Muchun Song
2020-09-15 12:59 ` [RFC PATCH 10/24] mm/hugetlb: Free the vmemmap pages associated with each hugetlb page Muchun Song
2020-09-15 12:59 ` [RFC PATCH 11/24] mm/hugetlb: Add vmemmap_pmd_huge macro for x86 Muchun Song
2020-09-15 12:59 ` [RFC PATCH 12/24] mm/hugetlb: Defer freeing of hugetlb pages Muchun Song
2020-09-15 12:59 ` [RFC PATCH 13/24] mm/hugetlb: Allocate the vmemmap pages associated with each hugetlb page Muchun Song
2020-09-15 12:59 ` [RFC PATCH 14/24] mm/hugetlb: Introduce remap_huge_page_pmd_vmemmap helper Muchun Song
2020-09-15 12:59 ` [RFC PATCH 15/24] mm/hugetlb: Use PG_slab to indicate split pmd Muchun Song
2020-09-15 12:59 ` [RFC PATCH 16/24] mm/hugetlb: Support freeing vmemmap pages of gigantic page Muchun Song
2020-09-15 12:59 ` [RFC PATCH 17/24] mm/hugetlb: Add a BUILD_BUG_ON to check if struct page size is a power of two Muchun Song
2020-09-15 12:59 ` [RFC PATCH 18/24] mm/hugetlb: Clear PageHWPoison on the non-error memory page Muchun Song
2020-09-15 12:59 ` [RFC PATCH 19/24] mm/hugetlb: Flush work when dissolving hugetlb page Muchun Song
2020-09-15 12:59 ` [RFC PATCH 20/24] mm/hugetlb: Add a kernel parameter hugetlb_free_vmemmap Muchun Song
2020-09-16  2:10   ` Randy Dunlap
2020-09-16  2:50     ` [External] " Muchun Song
2020-09-15 12:59 ` [RFC PATCH 21/24] mm/hugetlb: Merge pte to huge pmd only for gigantic page Muchun Song
2020-09-20  9:59   ` Muchun Song
2020-09-15 12:59 ` [RFC PATCH 22/24] mm/hugetlb: Implement vmemmap_pmd_mkhuge macro Muchun Song
2020-09-15 12:59 ` [RFC PATCH 23/24] mm/hugetlb: Gather discrete indexes of tail page Muchun Song
2020-09-15 12:59 ` [RFC PATCH 24/24] mm/hugetlb: Add BUILD_BUG_ON to catch invalid usage of tail struct page Muchun Song
2020-09-15 14:32 ` [RFC PATCH 00/24] mm/hugetlb: Free some vmemmap pages of hugetlb page Matthew Wilcox
2020-09-15 14:53   ` Dave Hansen
2020-09-15 15:28   ` [External] " Muchun Song
2020-09-15 15:42     ` Matthew Wilcox
2020-09-15 17:32       ` Muchun Song
2020-09-15 17:39         ` Matthew Wilcox
2020-09-15 18:03           ` Muchun Song
2020-09-15 18:15             ` Matthew Wilcox
2020-09-16  2:45               ` Muchun Song
2020-09-29 21:58 ` Mike Kravetz
2020-09-30  3:13   ` Matthew Wilcox
2020-10-07 21:12   ` Mike Kravetz
2020-10-09  4:13     ` Muchun Song [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAMZfGtXAB7CqNxp2Et=SSY4iPDbxS92cwcDEo2C_88m92JNWoQ@mail.gmail.com' \
    --to=songmuchun@bytedance.com \
    --cc=akpm@linux-foundation.org \
    --cc=almasrymina@google.com \
    --cc=anshuman.khandual@arm.com \
    --cc=bp@alien8.de \
    --cc=corbet@lwn.net \
    --cc=dave.hansen@linux.intel.com \
    --cc=duanxiongchun@bytedance.com \
    --cc=hpa@zytor.com \
    --cc=jroedel@suse.de \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=luto@kernel.org \
    --cc=mchehab+huawei@kernel.org \
    --cc=mike.kravetz@oracle.com \
    --cc=mingo@redhat.com \
    --cc=oneukum@suse.com \
    --cc=paulmck@kernel.org \
    --cc=pawan.kumar.gupta@linux.intel.com \
    --cc=peterz@infradead.org \
    --cc=rdunlap@infradead.org \
    --cc=rientjes@google.com \
    --cc=tglx@linutronix.de \
    --cc=viro@zeniv.linux.org.uk \
    --cc=willy@infradead.org \
    --cc=x86@kernel.org \
    --subject='Re: [External] Re: [RFC PATCH 00/24] mm/hugetlb: Free some vmemmap pages of hugetlb page' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).