RE: [patch 14/21] Xen-paravirt: Add XEN config options and disableunsupported config options.

RE: [patch 14/21] Xen-paravirt: Add XEN config options and disableunsupported config options.

[Christian Limpach]

Jeremy Fitzhardinge wrote:
> Andrew Morton wrote:
> > On Thu, 15 Feb 2007 22:14:45 -0800 Dan Hecht 
> <dhecht@vmware.com> wrote:
> >
> >   
> >>>  config PREEMPT
> >>>  	bool "Preemptible Kernel (Low-Latency Desktop)"
> >>> +	depends on !XEN
> >>>  	help
> >>>  	  This option reduces the latency of the kernel by making
> >>>  	  all kernel code (that is not executing in a critical section)
> >>>
> >>>       
> >
> > Oh, so that's why it doesn't break when CONFIG_PREEMPT=y.  
> In which case
> > that preempt_disable() I spotted is wrong-and-unneeded.
> >
> > Why doesn't Xen work with preemption??
> >   
> 
> I've forgotten the details.  Ian?  Keir?  Steven?  Maybe it 
> can be done.

With CONFIG_PREEMPT, we can have preempted threads reference machine
addresses across save/restore.  After restore, these machine addresses
will be incorrect.

    Christian

Re: [patch 14/21] Xen-paravirt: Add XEN config options and disableunsupported config options.

[Andrew Morton]

On Fri, 16 Feb 2007 02:00:39 -0800 "Christian Limpach" <Christian.Limpach@xensource.com> wrote:

> Jeremy Fitzhardinge wrote:
> > Andrew Morton wrote:
> > > On Thu, 15 Feb 2007 22:14:45 -0800 Dan Hecht 
> > <dhecht@vmware.com> wrote:
> > >
> > >   
> > >>>  config PREEMPT
> > >>>  	bool "Preemptible Kernel (Low-Latency Desktop)"
> > >>> +	depends on !XEN
> > >>>  	help
> > >>>  	  This option reduces the latency of the kernel by making
> > >>>  	  all kernel code (that is not executing in a critical section)
> > >>>
> > >>>       
> > >
> > > Oh, so that's why it doesn't break when CONFIG_PREEMPT=y.  
> > In which case
> > > that preempt_disable() I spotted is wrong-and-unneeded.
> > >
> > > Why doesn't Xen work with preemption??
> > >   
> > 
> > I've forgotten the details.  Ian?  Keir?  Steven?  Maybe it 
> > can be done.
> 
> With CONFIG_PREEMPT, we can have preempted threads reference machine
> addresses across save/restore.  After restore, these machine addresses
> will be incorrect.
> 

It would help if you could define the terms "save/restore" and "machine
addresses".  One can guess, but there are probably subtleties here..

Are the places where the domU code references machine addresses splattered
all over the code?  If not, they can just be wrapped with
preempt_disable/preempt_enable?

Re: [patch 14/21] Xen-paravirt: Add XEN config options and disableunsupported config options.

[Keir Fraser]

On 16/2/07 10:09, "Andrew Morton" <akpm@linux-foundation.org> wrote:

> Are the places where the domU code references machine addresses splattered
> all over the code?  If not, they can just be wrapped with
> preempt_disable/preempt_enable?

The main places where machine addresses are 'visible' are any code that
holds a pte_t,pmd_t,pud_t,pgd_t. We hide the machine-to-pseudophysical and
pseudophysical-to-machine translations inside e.g., pte_val() and __pte()
(i.e., constructors and extractors for page table entries). Obviously the
users of these macros are open coded all over the place, quite apart from
the performance cost of sprinkling preempt_{enable,disable} so liberally.

 -- Keir

Re: [patch 14/21] Xen-paravirt: Add XEN config options and disableunsupported config options.

[Andrew Morton]

On Fri, 16 Feb 2007 10:47:11 +0000 Keir Fraser <keir@xensource.com> wrote:

> On 16/2/07 10:09, "Andrew Morton" <akpm@linux-foundation.org> wrote:
> 
> > Are the places where the domU code references machine addresses splattered
> > all over the code?  If not, they can just be wrapped with
> > preempt_disable/preempt_enable?
> 
> The main places where machine addresses are 'visible' are any code that
> holds a pte_t,pmd_t,pud_t,pgd_t. We hide the machine-to-pseudophysical and
> pseudophysical-to-machine translations inside e.g., pte_val() and __pte()
> (i.e., constructors and extractors for page table entries). Obviously the
> users of these macros are open coded all over the place, quite apart from
> the performance cost of sprinkling preempt_{enable,disable} so liberally.

OK, you're screwed.  I agree that the process freezer is the way out of that one.

Ingo said that he's clocked the freezer at a few milliseconds.  But if it's
any higher than that it'll need to get sped up once we convert cpu hotplug
to use it.

Re: [patch 14/21] Xen-paravirt: Add XEN config options and disableunsupported config options.

[Keir Fraser]

On 16/2/07 10:54, "Andrew Morton" <akpm@linux-foundation.org> wrote:

> OK, you're screwed.  I agree that the process freezer is the way out of that
> one.
> 
> Ingo said that he's clocked the freezer at a few milliseconds.  But if it's
> any higher than that it'll need to get sped up once we convert cpu hotplug
> to use it.

A few milliseconds is acceptable. Presumably that is under some stress
workload? What we'd really like is to be able to gang-hotplug CPUs to
amortise the freezer overhead.

This initial patchset does not include save/restore support anyway, so in
fact it would be consistent to have CONFIG_PREEMPT configurable. I'm sure
that we are going to have some nasty bugs to fix up as a result, but we
can't fix them until we find them! Then we can convert our save/restore code
to use the freezer before submitting it for inclusion.

 -- Keir

Re: [Xen-devel] Re: [patch 14/21] Xen-paravirt: Add XEN config options and disableunsupported config options.

[Jeremy Fitzhardinge]

Keir Fraser wrote:
> This initial patchset does not include save/restore support anyway, so in
> fact it would be consistent to have CONFIG_PREEMPT configurable. I'm sure
> that we are going to have some nasty bugs to fix up as a result, but we
> can't fix them until we find them! Then we can convert our save/restore code
> to use the freezer before submitting it for inclusion.
OK.  So that makes the only config restriction the 100Hz ticks.

    J

Re: [Xen-devel] Re: [patch 14/21] Xen-paravirt: Add XEN config options and disableunsupported config options.

[Keir Fraser]

On 16/2/07 17:46, "Jeremy Fitzhardinge" <jeremy@goop.org> wrote:

> Keir Fraser wrote:
>> This initial patchset does not include save/restore support anyway, so in
>> fact it would be consistent to have CONFIG_PREEMPT configurable. I'm sure
>> that we are going to have some nasty bugs to fix up as a result, but we
>> can't fix them until we find them! Then we can convert our save/restore code
>> to use the freezer before submitting it for inclusion.
> OK.  So that makes the only config restriction the 100Hz ticks.

We can extend the Xen timer interface quite easily and get rid of this one
too. In fact it doesn't *much* matter if the CONFIG_HZ differs from the Xen
ticker rate -- we modified the Linux timer ISR to handle timer interrupts at
arbitrary times already. The only drawback is that jiffies updates in burts
if CONFIG_HZ is higher than the actual tick rate, and this can affect some
calibration constants and cause Linux to print out some weird values at
start-of-day.

 -- Keir

Re: [Xen-devel] Re: [patch 14/21] Xen-paravirt: Add XEN config options and disableunsupported config options.

[Jeremy Fitzhardinge]

Keir Fraser wrote:
> We can extend the Xen timer interface quite easily and get rid of this one
> too. In fact it doesn't *much* matter if the CONFIG_HZ differs from the Xen
> ticker rate -- we modified the Linux timer ISR to handle timer interrupts at
> arbitrary times already. The only drawback is that jiffies updates in burts
> if CONFIG_HZ is higher than the actual tick rate, and this can affect some
> calibration constants and cause Linux to print out some weird values at
> start-of-day.
>   

I was wondering what would happen if we tried that.  I guess its
equivalent to having stolen time between each Xen tick.

    J

Re: [Xen-devel] Re: [patch 14/21] Xen-paravirt: Add XEN config options and disableunsupported config options.

[Zachary Amsden]

Keir Fraser wrote:
> On 16/2/07 17:46, "Jeremy Fitzhardinge" <jeremy@goop.org> wrote:
>
>   
>> Keir Fraser wrote:
>>     
>>> This initial patchset does not include save/restore support anyway, so in
>>> fact it would be consistent to have CONFIG_PREEMPT configurable. I'm sure
>>> that we are going to have some nasty bugs to fix up as a result, but we
>>> can't fix them until we find them! Then we can convert our save/restore code
>>> to use the freezer before submitting it for inclusion.
>>>       
>> OK.  So that makes the only config restriction the 100Hz ticks.
>>     
>
> We can extend the Xen timer interface quite easily and get rid of this one
> too. In fact it doesn't *much* matter if the CONFIG_HZ differs from the Xen
> ticker rate -- we modified the Linux timer ISR to handle timer interrupts at
> arbitrary times already. The only drawback is that jiffies updates in burts
> if CONFIG_HZ is higher than the actual tick rate, and this can affect some
> calibration constants and cause Linux to print out some weird values at
> start-of-day.
>   

That's why we'd very much like to get a get_cpu_speed paravirt-op 
implemented.  I think this would be useful to work around these problems 
for Xen as well.

Zach

Re: [Xen-devel] Re: [patch 14/21] Xen-paravirt: Add XEN config options and disableunsupported config options.

[Dan Hecht]

On 02/16/2007 01:51 PM, Zachary Amsden wrote:
> Keir Fraser wrote:
>> On 16/2/07 17:46, "Jeremy Fitzhardinge" <jeremy@goop.org> wrote:
>>
>>   
>>> Keir Fraser wrote:
>>>     
>>>> This initial patchset does not include save/restore support anyway, so in
>>>> fact it would be consistent to have CONFIG_PREEMPT configurable. I'm sure
>>>> that we are going to have some nasty bugs to fix up as a result, but we
>>>> can't fix them until we find them! Then we can convert our save/restore code
>>>> to use the freezer before submitting it for inclusion.
>>>>       
>>> OK.  So that makes the only config restriction the 100Hz ticks.
>>>     
>> We can extend the Xen timer interface quite easily and get rid of this one
>> too. In fact it doesn't *much* matter if the CONFIG_HZ differs from the Xen
>> ticker rate -- we modified the Linux timer ISR to handle timer interrupts at
>> arbitrary times already. The only drawback is that jiffies updates in burts
>> if CONFIG_HZ is higher than the actual tick rate, and this can affect some
>> calibration constants and cause Linux to print out some weird values at
>> start-of-day.
>>   
> 
> That's why we'd very much like to get a get_cpu_speed paravirt-op 
> implemented.  I think this would be useful to work around these problems 
> for Xen as well.
> 
>

Yes, and regardless of whether you run your periodic timer slower than 
HZ, calibrating time in a VM is always difficult due to the fact the 
kernel is time sharing the physical cpu.  Why not just ask the 
underlying hypervisor?

Really, Time is one of those areas where paravirtualization makes a lot 
of sense, no matter how much virtualization hardware you have crammed 
into your cpu....

Re: [Xen-devel] Re: [patch 14/21] Xen-paravirt: Add XEN config options and disableunsupported config options.

[Rik van Riel]

Dan Hecht wrote:

> Yes, and regardless of whether you run your periodic timer slower than 
> HZ, calibrating time in a VM is always difficult due to the fact the 
> kernel is time sharing the physical cpu.  Why not just ask the 
> underlying hypervisor?

Upstream Xen does just that.

I'm guessing we'll want something similar for KVM paravirt.

-- 
Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.