LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Pavel Tatashin <pasha.tatashin@oracle.com>
To: avagin@virtuozzo.com
Cc: Steven Sistare <steven.sistare@oracle.com>,
	Daniel Jordan <daniel.m.jordan@oracle.com>,
	Andrew Morton <akpm@linux-foundation.org>,
	LKML <linux-kernel@vger.kernel.org>,
	tglx@linutronix.de, Michal Hocko <mhocko@suse.com>,
	Linux Memory Management List <linux-mm@kvack.org>,
	mgorman@techsingularity.net, mingo@kernel.org,
	peterz@infradead.org, Steven Rostedt <rostedt@goodmis.org>,
	Fengguang Wu <fengguang.wu@intel.com>,
	Dennis Zhou <dennisszhou@gmail.com>
Subject: Re: [v2] mm: access to uninitialized struct page
Date: Fri, 04 May 2018 12:47:53 +0000	[thread overview]
Message-ID: <CAGM2rebLfmWLybzNDPt-HTjZY2brkJ_8Bq37xVG_QDs=G+VuxQ@mail.gmail.com> (raw)
In-Reply-To: <20180504082731.GA2782@outlook.office365.com>

Hi Andrei,

Could you please provide me with scripts to reproduce this issue?

Thank you,
Pavel
On Fri, May 4, 2018 at 4:27 AM Andrei Vagin <avagin@virtuozzo.com> wrote:

> Hello,

> We have a robot which runs criu tests on linux-next kernels.

> All tests passed on 4.17.0-rc3-next-20180502.

> But the 4.17.0-rc3-next-20180504 kernel didn't boot.

> git bisect points on this patch.

> On Thu, Apr 26, 2018 at 04:26:19PM -0400, Pavel Tatashin wrote:
> > The following two bugs were reported by Fengguang Wu:
> >
> > kernel reboot-without-warning in early-boot stage, last printk:
> > early console in setup code
> >
> >
http://lkml.kernel.org/r/20180418135300.inazvpxjxowogyge@wfg-t540p.sh.intel.com

> The problem looks similar with this one.

> [    5.596975] devtmpfs: mounted
> [    5.855754] Freeing unused kernel memory: 1704K
> [    5.858162] Write protecting the kernel read-only data: 18432k
> [    5.860772] Freeing unused kernel memory: 2012K
> [    5.861838] Freeing unused kernel memory: 160K
> [    5.862572] rodata_test: all tests were successful
> [    5.866857] random: fast init done
> early console in setup code
> [    0.000000] Linux version 4.17.0-rc3-00023-g7c4cc2d022a1
> (avagin@laptop) (gcc version 8.0.1 20180324 (Red Hat 8.0.1-0.20) (GCC))
> #13 SMP Fri May 4 01:10:51 PDT 2018
> [    0.000000] Command line: root=/dev/vda2 ro debug
> console=ttyS0,115200 LANG=en_US.UTF-8 slub_debug=FZP raid=noautodetect
> selinux=0 earlyprintk=serial,ttyS0,115200
> [    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating
> point registers'
> [    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
> [    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
> [    0.000000] x86/fpu: Supporting XSAVE feature 0x008: 'MPX bounds
> registers'

> $ git describe HEAD
> v4.17-rc3-23-g7c4cc2d022a1

> [avagin@laptop linux-next]$ git log --pretty=oneline  | head -n 1
> 7c4cc2d022a1fd56eb2ee555533b8666bc780f1e mm: access to uninitialized
struct page


> >
> > And, also:
> > [per_cpu_ptr_to_phys] PANIC: early exception 0x0d
> > IP 10:ffffffffa892f15f error 0 cr2 0xffff88001fbff000
> >
> >
http://lkml.kernel.org/r/20180419013128.iurzouiqxvcnpbvz@wfg-t540p.sh.intel.com
> >
> > Both of the problems are due to accessing uninitialized struct page from
> > trap_init(). We must first do mm_init() in order to initialize allocated
> > struct pages, and than we can access fields of any struct page that
belongs
> > to memory that's been allocated.
> >
> > Below is explanation of the root cause.
> >
> > The issue arises in this stack:
> >
> > start_kernel()
> >  trap_init()
> >   setup_cpu_entry_areas()
> >    setup_cpu_entry_area(cpu)
> >     get_cpu_gdt_paddr(cpu)
> >      per_cpu_ptr_to_phys(addr)
> >       pcpu_addr_to_page(addr)
> >        virt_to_page(addr)
> >         pfn_to_page(__pa(addr) >> PAGE_SHIFT)
> > The returned "struct page" is sometimes uninitialized, and thus
> > failing later when used. It turns out sometimes is because it depends
> > on KASLR.
> >
> > When boot is failing we have this when  pfn_to_page() is called:
> > kasrl: 0x000000000d600000
> >  addr: ffffffff83e0d000
> >     pa: 1040d000
> >    pfn: 1040d
> > page: ffff88001f113340
> > page->flags ffffffffffffffff <- Uninitialized!
> >
> > When boot is successful:
> > kaslr: 0x000000000a800000
> >  addr: ffffffff83e0d000
> >      pa: d60d000
> >     pfn: d60d
> >  page: ffff88001f05b340
> > page->flags 280000000000 <- Initialized!
> >
> > Here are physical addresses that BIOS provided to us:
> > e820: BIOS-provided physical RAM map:
> > BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
> > BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
> > BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
> > BIOS-e820: [mem 0x0000000000100000-0x000000001ffdffff] usable
> > BIOS-e820: [mem 0x000000001ffe0000-0x000000001fffffff] reserved
> > BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
> > BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
> >
> > In both cases, working and non-working the real physical address is
> > the same:
> >
> > pa - kasrl = 0x2E0D000
> >
> > The only thing that is different is PFN.
> >
> > We initialize struct pages in four places:
> >
> > 1. Early in boot a small set of struct pages is initialized to fill
> > the first section, and lower zones.
> > 2. During mm_init() we initialize "struct pages" for all the memory
> > that is allocated, i.e reserved in memblock.
> > 3. Using on-demand logic when pages are allocated after mm_init call
> > 4. After smp_init() when the rest free deferred pages are initialized.
> >
> > The above path happens before deferred memory is initialized, and thus
> > it must be covered either by 1, 2 or 3.
> >
> > So, lets check what PFNs are initialized after (1).
> >
> > memmap_init_zone() is called for pfn ranges:
> > 1 - 1000, and 1000 - 1ffe0, but it quits after reaching pfn 0x10000,
> > as it leaves the rest to be initialized as deferred pages.
> >
> > In the working scenario pfn ended up being below 1000, but in the
> > failing scenario it is above. Hence, we must initialize this page in
> > (2). But trap_init() is called before mm_init().
> >
> > The bug was introduced by "mm: initialize pages on demand during boot"
> > because we lowered amount of pages that is initialized in the step
> > (1). But, it still could happen, because the number of initialized
> > pages was a guessing.
> >
> > The current fix moves trap_init() to be called after mm_init, but as
> > alternative, we could increase pgdat->static_init_pgcnt:
> > In free_area_init_node we can increase:
> >        pgdat->static_init_pgcnt = min_t(unsigned long,
PAGES_PER_SECTION,
> >                                         pgdat->node_spanned_pages);
> > Instead of one PAGES_PER_SECTION, set several, so the text is
> > covered for all KASLR offsets. But, this would still be guessing.
> > Therefore, I prefer the current fix.
> >
> > Fixes: c9e97a1997fb ("mm: initialize pages on demand during boot")
> >
> > Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
> > Reviewed-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
> > ---
> >  init/main.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/init/main.c b/init/main.c
> > index b795aa341a3a..870f75581cea 100644
> > --- a/init/main.c
> > +++ b/init/main.c
> > @@ -585,8 +585,8 @@ asmlinkage __visible void __init start_kernel(void)
> >       setup_log_buf(0);
> >       vfs_caches_init_early();
> >       sort_main_extable();
> > -     trap_init();
> >       mm_init();
> > +     trap_init();
> >
> >       ftrace_init();
> >

  reply	other threads:[~2018-05-04 12:48 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-04-26 20:26 [PATCH v2] " Pavel Tatashin
2018-04-30 23:26 ` Andrew Morton
2018-04-30 23:58   ` Steven Rostedt
2018-05-01  0:01     ` Andrew Morton
2018-05-08 14:44   ` Pavel Tatashin
2018-05-04  8:27 ` [v2] " Andrei Vagin
2018-05-04 12:47   ` Pavel Tatashin [this message]
2018-05-04 14:50     ` Steven Rostedt
2018-05-04 16:01     ` Andrei Vagin
2018-05-04 16:03       ` Pavel Tatashin
2018-05-04 17:49         ` Andy Shevchenko
2018-05-05  1:04           ` Fengguang Wu

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAGM2rebLfmWLybzNDPt-HTjZY2brkJ_8Bq37xVG_QDs=G+VuxQ@mail.gmail.com' \
    --to=pasha.tatashin@oracle.com \
    --cc=akpm@linux-foundation.org \
    --cc=avagin@virtuozzo.com \
    --cc=daniel.m.jordan@oracle.com \
    --cc=dennisszhou@gmail.com \
    --cc=fengguang.wu@intel.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=mgorman@techsingularity.net \
    --cc=mhocko@suse.com \
    --cc=mingo@kernel.org \
    --cc=peterz@infradead.org \
    --cc=rostedt@goodmis.org \
    --cc=steven.sistare@oracle.com \
    --cc=tglx@linutronix.de \
    --subject='Re: [v2] mm: access to uninitialized struct page' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).