[PATCH] xen: convert p2m to a 3 level tree - partial revert

[PATCH] xen: convert p2m to a 3 level tree - partial revert

[Daniel Kiper]

Hi,

Durning work on Xen memory hotplug I discoverd that
2.6.38-rc2 does not boot on domU. After some investigation
it appeared that 58e05027b530ff081ecea68e38de8d59db8f87e0
commit changed CONFIG_XEN_MAX_DOMAIN_MEMORY constant value
to 128. This change does not allow to boot kernel on domU
with small memory size (I could confirm that it is even
not possible to boot kernel on domU with 2 GiB). Guest
crash silently without any warning. Durning further
research I found out that there is another commit published
recently (8e1b4cf2108488ccfb9a3e7ed7cd85a435e01d4b) which attempts
to fix this issue, however, it does not work on my platform.
I decided to reenable CONFIG_XEN_MAX_DOMAIN_MEMORY option in
kernel config and enable users to choose resonable values for
their machines until better fix will be published. I think this
solution is good because allow users to boot domU with newest
kernel and allow developers to continue their work without
time presure which could lead to new bugs.

This patch applies cleanly to current Linus' kernel tree.

Signed-off-by: Daniel Kiper <dkiper@net-space.pl>
---
 arch/x86/xen/Kconfig |   11 +++++++----
 1 files changed, 7 insertions(+), 4 deletions(-)

diff --git a/arch/x86/xen/Kconfig b/arch/x86/xen/Kconfig
index 5b54892..a234b9a 100644
--- a/arch/x86/xen/Kconfig
+++ b/arch/x86/xen/Kconfig
@@ -29,12 +29,15 @@ config XEN_PVHVM
 	depends on X86_LOCAL_APIC
 
 config XEN_MAX_DOMAIN_MEMORY
-       int
-       default 128
+       int "Maximum allowed size of a domain in gigabytes"
+       default 8 if X86_32
+       default 32 if X86_64
        depends on XEN
        help
-         This only affects the sizing of some bss arrays, the unused
-         portions of which are freed.
+         The pseudo-physical to machine address array is sized
+         according to the maximum possible memory size of a Xen
+         domain.  This array uses 1 page per gigabyte, so there's no
+         need to be too stingy here.
 
 config XEN_SAVE_RESTORE
        bool
-- 
1.5.6.5

Re: [PATCH] xen: convert p2m to a 3 level tree - partial revert

[Stefan Bader]

On 01/27/2011 03:48 PM, Daniel Kiper wrote:
> Hi,
> 
> Durning work on Xen memory hotplug I discoverd that
> 2.6.38-rc2 does not boot on domU. After some investigation
> it appeared that 58e05027b530ff081ecea68e38de8d59db8f87e0
> commit changed CONFIG_XEN_MAX_DOMAIN_MEMORY constant value
> to 128. This change does not allow to boot kernel on domU
> with small memory size (I could confirm that it is even
> not possible to boot kernel on domU with 2 GiB). Guest
> crash silently without any warning. Durning further
> research I found out that there is another commit published
> recently (8e1b4cf2108488ccfb9a3e7ed7cd85a435e01d4b) which attempts
> to fix this issue, however, it does not work on my platform.
> I decided to reenable CONFIG_XEN_MAX_DOMAIN_MEMORY option in
> kernel config and enable users to choose resonable values for
> their machines until better fix will be published. I think this
> solution is good because allow users to boot domU with newest
> kernel and allow developers to continue their work without
> time presure which could lead to new bugs.
> 

Have you tried with a version that contains

commit 8e1b4cf2108488ccfb9a3e7ed7cd85a435e01d4b
Author: Stefan Bader <stefan.bader@canonical.com>
Date:   Thu Jan 20 15:38:23 2011 +0100

    xen: p2m: correctly initialize partial p2m leaf

I still may need to send a follow up to add a matching RESERVE_BRK, but at least
I was able to boot small DomU's again.

-Stefan
> This patch applies cleanly to current Linus' kernel tree.
> 
> Signed-off-by: Daniel Kiper <dkiper@net-space.pl>
> ---
>  arch/x86/xen/Kconfig |   11 +++++++----
>  1 files changed, 7 insertions(+), 4 deletions(-)
> 
> diff --git a/arch/x86/xen/Kconfig b/arch/x86/xen/Kconfig
> index 5b54892..a234b9a 100644
> --- a/arch/x86/xen/Kconfig
> +++ b/arch/x86/xen/Kconfig
> @@ -29,12 +29,15 @@ config XEN_PVHVM
>  	depends on X86_LOCAL_APIC
>  
>  config XEN_MAX_DOMAIN_MEMORY
> -       int
> -       default 128
> +       int "Maximum allowed size of a domain in gigabytes"
> +       default 8 if X86_32
> +       default 32 if X86_64
>         depends on XEN
>         help
> -         This only affects the sizing of some bss arrays, the unused
> -         portions of which are freed.
> +         The pseudo-physical to machine address array is sized
> +         according to the maximum possible memory size of a Xen
> +         domain.  This array uses 1 page per gigabyte, so there's no
> +         need to be too stingy here.
>  
>  config XEN_SAVE_RESTORE
>         bool

Re: [PATCH] xen: convert p2m to a 3 level tree - partial revert

[Stefan Bader]

On 01/27/2011 03:59 PM, Stefan Bader wrote:
> On 01/27/2011 03:48 PM, Daniel Kiper wrote:
>> Hi,
>>
>> Durning work on Xen memory hotplug I discoverd that
>> 2.6.38-rc2 does not boot on domU. After some investigation
>> it appeared that 58e05027b530ff081ecea68e38de8d59db8f87e0
>> commit changed CONFIG_XEN_MAX_DOMAIN_MEMORY constant value
>> to 128. This change does not allow to boot kernel on domU
>> with small memory size (I could confirm that it is even
>> not possible to boot kernel on domU with 2 GiB). Guest
>> crash silently without any warning. Durning further
>> research I found out that there is another commit published
>> recently (8e1b4cf2108488ccfb9a3e7ed7cd85a435e01d4b) which attempts
>> to fix this issue, however, it does not work on my platform.
>> I decided to reenable CONFIG_XEN_MAX_DOMAIN_MEMORY option in
>> kernel config and enable users to choose resonable values for
>> their machines until better fix will be published. I think this
>> solution is good because allow users to boot domU with newest
>> kernel and allow developers to continue their work without
>> time presure which could lead to new bugs.
>>
> 
> Have you tried with a version that contains
> 
> commit 8e1b4cf2108488ccfb9a3e7ed7cd85a435e01d4b
> Author: Stefan Bader <stefan.bader@canonical.com>
> Date:   Thu Jan 20 15:38:23 2011 +0100
> 
>     xen: p2m: correctly initialize partial p2m leaf
> 
> I still may need to send a follow up to add a matching RESERVE_BRK, but at least
> I was able to boot small DomU's again.
> 
> -Stefan
>> This patch applies cleanly to current Linus' kernel tree.
>>

Sorry, did not read careful enough. So you did. Maybe you could actually try to
make the additional change of

--- a/arch/x86/xen/p2m.c
+++ b/arch/x86/xen/p2m.c
@@ -61,6 +61,7 @@ static RESERVE_BRK_ARRAY(unsigned long *, p2m_top_mfn_p, P2M_T

 RESERVE_BRK(p2m_mid, PAGE_SIZE * (MAX_DOMAIN_PAGES / (P2M_PER_PAGE * P2M_MID_PE
 RESERVE_BRK(p2m_mid_mfn, PAGE_SIZE * (MAX_DOMAIN_PAGES / (P2M_PER_PAGE * P2M_MI
+RESERVE_BRK(p2m_node, PAGE_SIZE);

 static inline unsigned p2m_top_index(unsigned long pfn)
 {

This one I missed and probably that has more effects on big memory values.

-Stefan

>> Signed-off-by: Daniel Kiper <dkiper@net-space.pl>
>> ---
>>  arch/x86/xen/Kconfig |   11 +++++++----
>>  1 files changed, 7 insertions(+), 4 deletions(-)
>>
>> diff --git a/arch/x86/xen/Kconfig b/arch/x86/xen/Kconfig
>> index 5b54892..a234b9a 100644
>> --- a/arch/x86/xen/Kconfig
>> +++ b/arch/x86/xen/Kconfig
>> @@ -29,12 +29,15 @@ config XEN_PVHVM
>>  	depends on X86_LOCAL_APIC
>>  
>>  config XEN_MAX_DOMAIN_MEMORY
>> -       int
>> -       default 128
>> +       int "Maximum allowed size of a domain in gigabytes"
>> +       default 8 if X86_32
>> +       default 32 if X86_64
>>         depends on XEN
>>         help
>> -         This only affects the sizing of some bss arrays, the unused
>> -         portions of which are freed.
>> +         The pseudo-physical to machine address array is sized
>> +         according to the maximum possible memory size of a Xen
>> +         domain.  This array uses 1 page per gigabyte, so there's no
>> +         need to be too stingy here.
>>  
>>  config XEN_SAVE_RESTORE
>>         bool
> 

Re: [PATCH] xen: convert p2m to a 3 level tree - partial revert

[Daniel Kiper]

Hi,

On Thu, Jan 27, 2011 at 04:02:31PM +0100, Stefan Bader wrote:
> On 01/27/2011 03:59 PM, Stefan Bader wrote:
> > On 01/27/2011 03:48 PM, Daniel Kiper wrote:
> >> Hi,
> >>
> >> Durning work on Xen memory hotplug I discoverd that
> >> 2.6.38-rc2 does not boot on domU. After some investigation
> >> it appeared that 58e05027b530ff081ecea68e38de8d59db8f87e0
> >> commit changed CONFIG_XEN_MAX_DOMAIN_MEMORY constant value
> >> to 128. This change does not allow to boot kernel on domU
> >> with small memory size (I could confirm that it is even
> >> not possible to boot kernel on domU with 2 GiB). Guest
> >> crash silently without any warning. Durning further
> >> research I found out that there is another commit published
> >> recently (8e1b4cf2108488ccfb9a3e7ed7cd85a435e01d4b) which attempts
> >> to fix this issue, however, it does not work on my platform.
> >> I decided to reenable CONFIG_XEN_MAX_DOMAIN_MEMORY option in
> >> kernel config and enable users to choose resonable values for
> >> their machines until better fix will be published. I think this
> >> solution is good because allow users to boot domU with newest
> >> kernel and allow developers to continue their work without
> >> time presure which could lead to new bugs.
> >
> > Have you tried with a version that contains
> >
> > commit 8e1b4cf2108488ccfb9a3e7ed7cd85a435e01d4b
> > Author: Stefan Bader <stefan.bader@canonical.com>
> > Date:   Thu Jan 20 15:38:23 2011 +0100
> >
> >     xen: p2m: correctly initialize partial p2m leaf
> >
> > I still may need to send a follow up to add a matching RESERVE_BRK, but at least
> > I was able to boot small DomU's again.
> >
> > -Stefan
> >> This patch applies cleanly to current Linus' kernel tree.
>
> Sorry, did not read careful enough. So you did. Maybe you could actually try to
> make the additional change of
>
> --- a/arch/x86/xen/p2m.c
> +++ b/arch/x86/xen/p2m.c
> @@ -61,6 +61,7 @@ static RESERVE_BRK_ARRAY(unsigned long *, p2m_top_mfn_p, P2M_T
>
>  RESERVE_BRK(p2m_mid, PAGE_SIZE * (MAX_DOMAIN_PAGES / (P2M_PER_PAGE * P2M_MID_PE
>  RESERVE_BRK(p2m_mid_mfn, PAGE_SIZE * (MAX_DOMAIN_PAGES / (P2M_PER_PAGE * P2M_MI
> +RESERVE_BRK(p2m_node, PAGE_SIZE);
>
>  static inline unsigned p2m_top_index(unsigned long pfn)
>  {
>
> This one I missed and probably that has more effects on big memory values.

No, it does not work (CONFIG_XEN_MAX_DOMAIN_MEMORY=128).

Daniel

Re: [Xen-devel] [PATCH] xen: convert p2m to a 3 level tree - partial revert

[Konrad Rzeszutek Wilk]

On Thu, Jan 27, 2011 at 03:48:43PM +0100, Daniel Kiper wrote:
> Hi,
> 
> Durning work on Xen memory hotplug I discoverd that
> 2.6.38-rc2 does not boot on domU. After some investigation
> it appeared that 58e05027b530ff081ecea68e38de8d59db8f87e0
> commit changed CONFIG_XEN_MAX_DOMAIN_MEMORY constant value
> to 128. This change does not allow to boot kernel on domU
> with small memory size (I could confirm that it is even
> not possible to boot kernel on domU with 2 GiB). Guest

Strange.. I've been booting with 1G, 2G, 3G, 4G guests.

Could you give more details? Is this a 32-bit guest? 64-bit?
What does your xm file look like?

> crash silently without any warning. Durning further
> research I found out that there is another commit published
> recently (8e1b4cf2108488ccfb9a3e7ed7cd85a435e01d4b) which attempts
> to fix this issue, however, it does not work on my platform.
> I decided to reenable CONFIG_XEN_MAX_DOMAIN_MEMORY option in
> kernel config and enable users to choose resonable values for
> their machines until better fix will be published. I think this
> solution is good because allow users to boot domU with newest
> kernel and allow developers to continue their work without
> time presure which could lead to new bugs.

I think you are hitting a completly different bug. This
patch of yours changes the size of the structure - so it could
be that you are hitting a page-table overwrite. This is something
that Stefano has been looking at - you might want to take a look at
"x86: When destroying mappings between _brk_end and _end, do not go over _end"
(it is present in his git tree).

Or you might be hitting the issue I've which is pin_pagetable_init
failing. Can you run 'xenctx' on your guest and see
where it crashed? If you crank up the serial log or Xen logging
options do you see anything?

Does it look like this:
http://darnok.org/xen/p2m-v47-stefano-nx.txt
(I added some extra logging so it might not be exact).

Re: [PATCH] xen: convert p2m to a 3 level tree - partial revert

[Stefan Bader]

On 01/27/2011 04:19 PM, Daniel Kiper wrote:
> Hi,
> 
> On Thu, Jan 27, 2011 at 04:02:31PM +0100, Stefan Bader wrote:
>> On 01/27/2011 03:59 PM, Stefan Bader wrote:
>>> On 01/27/2011 03:48 PM, Daniel Kiper wrote:
>>>> Hi,
>>>>
>>>> Durning work on Xen memory hotplug I discoverd that
>>>> 2.6.38-rc2 does not boot on domU. After some investigation
>>>> it appeared that 58e05027b530ff081ecea68e38de8d59db8f87e0
>>>> commit changed CONFIG_XEN_MAX_DOMAIN_MEMORY constant value
>>>> to 128. This change does not allow to boot kernel on domU
>>>> with small memory size (I could confirm that it is even
>>>> not possible to boot kernel on domU with 2 GiB). Guest
>>>> crash silently without any warning. Durning further
>>>> research I found out that there is another commit published
>>>> recently (8e1b4cf2108488ccfb9a3e7ed7cd85a435e01d4b) which attempts
>>>> to fix this issue, however, it does not work on my platform.
>>>> I decided to reenable CONFIG_XEN_MAX_DOMAIN_MEMORY option in
>>>> kernel config and enable users to choose resonable values for
>>>> their machines until better fix will be published. I think this
>>>> solution is good because allow users to boot domU with newest
>>>> kernel and allow developers to continue their work without
>>>> time presure which could lead to new bugs.
>>>
>>> Have you tried with a version that contains
>>>
>>> commit 8e1b4cf2108488ccfb9a3e7ed7cd85a435e01d4b
>>> Author: Stefan Bader <stefan.bader@canonical.com>
>>> Date:   Thu Jan 20 15:38:23 2011 +0100
>>>
>>>     xen: p2m: correctly initialize partial p2m leaf
>>>
>>> I still may need to send a follow up to add a matching RESERVE_BRK, but at least
>>> I was able to boot small DomU's again.
>>>
>>> -Stefan
>>>> This patch applies cleanly to current Linus' kernel tree.
>>
>> Sorry, did not read careful enough. So you did. Maybe you could actually try to
>> make the additional change of
>>
>> --- a/arch/x86/xen/p2m.c
>> +++ b/arch/x86/xen/p2m.c
>> @@ -61,6 +61,7 @@ static RESERVE_BRK_ARRAY(unsigned long *, p2m_top_mfn_p, P2M_T
>>
>>  RESERVE_BRK(p2m_mid, PAGE_SIZE * (MAX_DOMAIN_PAGES / (P2M_PER_PAGE * P2M_MID_PE
>>  RESERVE_BRK(p2m_mid_mfn, PAGE_SIZE * (MAX_DOMAIN_PAGES / (P2M_PER_PAGE * P2M_MI
>> +RESERVE_BRK(p2m_node, PAGE_SIZE);
>>
>>  static inline unsigned p2m_top_index(unsigned long pfn)
>>  {
>>
>> This one I missed and probably that has more effects on big memory values.
> 
> No, it does not work (CONFIG_XEN_MAX_DOMAIN_MEMORY=128).
> 

Ok, thanks. So out of interest, what size of guest memory is failing for you?

-Stefan

> Daniel
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [Xen-devel] [PATCH] xen: convert p2m to a 3 level tree - partial revert

[Daniel Kiper]

Hi,

On Thu, Jan 27, 2011 at 10:25:08AM -0500, Konrad Rzeszutek Wilk wrote:
> On Thu, Jan 27, 2011 at 03:48:43PM +0100, Daniel Kiper wrote:
> > Hi,
> >
> > Durning work on Xen memory hotplug I discoverd that
> > 2.6.38-rc2 does not boot on domU. After some investigation
> > it appeared that 58e05027b530ff081ecea68e38de8d59db8f87e0
> > commit changed CONFIG_XEN_MAX_DOMAIN_MEMORY constant value
> > to 128. This change does not allow to boot kernel on domU
> > with small memory size (I could confirm that it is even
> > not possible to boot kernel on domU with 2 GiB). Guest
>
> Strange.. I've been booting with 1G, 2G, 3G, 4G guests.
>
> Could you give more details? Is this a 32-bit guest? 64-bit?
> What does your xm file look like?
>
> > crash silently without any warning. Durning further
> > research I found out that there is another commit published
> > recently (8e1b4cf2108488ccfb9a3e7ed7cd85a435e01d4b) which attempts
> > to fix this issue, however, it does not work on my platform.
> > I decided to reenable CONFIG_XEN_MAX_DOMAIN_MEMORY option in
> > kernel config and enable users to choose resonable values for
> > their machines until better fix will be published. I think this
> > solution is good because allow users to boot domU with newest
> > kernel and allow developers to continue their work without
> > time presure which could lead to new bugs.
>
> I think you are hitting a completly different bug. This
> patch of yours changes the size of the structure - so it could
> be that you are hitting a page-table overwrite. This is something
> that Stefano has been looking at - you might want to take a look at
> "x86: When destroying mappings between _brk_end and _end, do not go over _end"
> (it is present in his git tree).

It appears that this patch fixed issue described above. In this
situation I think it should be applied ASAP to RC and stable kernels.

Thank you for your help.

Daniel

Re: [Xen-devel] [PATCH] xen: convert p2m to a 3 level tree - partial revert

[Stefano Stabellini]

On Mon, 31 Jan 2011, Daniel Kiper wrote:
> Hi,
> 
> On Thu, Jan 27, 2011 at 10:25:08AM -0500, Konrad Rzeszutek Wilk wrote:
> > On Thu, Jan 27, 2011 at 03:48:43PM +0100, Daniel Kiper wrote:
> > > Hi,
> > >
> > > Durning work on Xen memory hotplug I discoverd that
> > > 2.6.38-rc2 does not boot on domU. After some investigation
> > > it appeared that 58e05027b530ff081ecea68e38de8d59db8f87e0
> > > commit changed CONFIG_XEN_MAX_DOMAIN_MEMORY constant value
> > > to 128. This change does not allow to boot kernel on domU
> > > with small memory size (I could confirm that it is even
> > > not possible to boot kernel on domU with 2 GiB). Guest
> >
> > Strange.. I've been booting with 1G, 2G, 3G, 4G guests.
> >
> > Could you give more details? Is this a 32-bit guest? 64-bit?
> > What does your xm file look like?
> >
> > > crash silently without any warning. Durning further
> > > research I found out that there is another commit published
> > > recently (8e1b4cf2108488ccfb9a3e7ed7cd85a435e01d4b) which attempts
> > > to fix this issue, however, it does not work on my platform.
> > > I decided to reenable CONFIG_XEN_MAX_DOMAIN_MEMORY option in
> > > kernel config and enable users to choose resonable values for
> > > their machines until better fix will be published. I think this
> > > solution is good because allow users to boot domU with newest
> > > kernel and allow developers to continue their work without
> > > time presure which could lead to new bugs.
> >
> > I think you are hitting a completly different bug. This
> > patch of yours changes the size of the structure - so it could
> > be that you are hitting a page-table overwrite. This is something
> > that Stefano has been looking at - you might want to take a look at
> > "x86: When destroying mappings between _brk_end and _end, do not go over _end"
> > (it is present in his git tree).
> 
> It appears that this patch fixed issue described above. In this
> situation I think it should be applied ASAP to RC and stable kernels.
> 

I am reworking the patch as we speak, I should have an upstreamable
version ready very soon.