LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
* [PATCH] Drivers: hv: vmbus: prevent cpu offlining on newer hypervisors
@ 2014-11-26 13:52 Vitaly Kuznetsov
  2014-11-27  3:02 ` Greg Kroah-Hartman
  2014-12-01 10:52 ` [PATCH v2] " Vitaly Kuznetsov
  0 siblings, 2 replies; 14+ messages in thread
From: Vitaly Kuznetsov @ 2014-11-26 13:52 UTC (permalink / raw)
  To: K. Y. Srinivasan, Haiyang Zhang, devel
  Cc: Greg Kroah-Hartman, linux-kernel, Dexuan Cui

When an SMP Hyper-V guest is running on top of 2012R2 Server and secondary
cpus are sent offline (with echo 0 > /sys/devices/system/cpu/cpu$cpu/online)
the system freeze is observed. This happens due to the fact that on newer
hypervisors (Win8, WS2012R2, ...) vmbus channel handlers are distributed
across all cpus (see init_vp_index() function in drivers/hv/channel_mgmt.c)
and on cpu offlining nobody reassigns them to CPU0. Prevent cpu offlining
when vmbus is loaded until the issue is fixed host-side.

This patch also disables hibernation but it is OK as it is also broken (MCE
error is hit on resume). Suspend still works.

Tested with WS2008R2 and WS2012R2.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 drivers/hv/vmbus_drv.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c
index 4d6b269..9a82249 100644
--- a/drivers/hv/vmbus_drv.c
+++ b/drivers/hv/vmbus_drv.c
@@ -32,6 +32,7 @@
 #include <linux/completion.h>
 #include <linux/hyperv.h>
 #include <linux/kernel_stat.h>
+#include <linux/cpu.h>
 #include <asm/hyperv.h>
 #include <asm/hypervisor.h>
 #include <asm/mshyperv.h>
@@ -671,6 +672,13 @@ static void vmbus_isr(void)
 		tasklet_schedule(&msg_dpc);
 }
 
+#ifdef CONFIG_HOTPLUG_CPU
+static int hyperv_cpu_disable(void)
+{
+	return -1;
+}
+#endif
+
 /*
  * vmbus_bus_init -Main vmbus driver initialization routine.
  *
@@ -711,6 +719,12 @@ static int vmbus_bus_init(int irq)
 	if (ret)
 		goto err_alloc;
 
+#ifdef CONFIG_HOTPLUG_CPU
+	if ((vmbus_proto_version != VERSION_WS2008) &&
+	    (vmbus_proto_version != VERSION_WIN7))
+		smp_ops.cpu_disable = hyperv_cpu_disable;
+#endif
+
 	vmbus_request_offers();
 
 	return 0;
@@ -964,6 +978,11 @@ static void __exit vmbus_exit(void)
 	bus_unregister(&hv_bus);
 	hv_cleanup();
 	acpi_bus_unregister_driver(&vmbus_acpi_driver);
+#ifdef CONFIG_HOTPLUG_CPU
+	if ((vmbus_proto_version != VERSION_WS2008) &&
+	    (vmbus_proto_version != VERSION_WIN7))
+		smp_ops.cpu_disable = native_cpu_disable;
+#endif
 }
 
 
-- 
1.9.3


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH] Drivers: hv: vmbus: prevent cpu offlining on newer hypervisors
  2014-11-26 13:52 [PATCH] Drivers: hv: vmbus: prevent cpu offlining on newer hypervisors Vitaly Kuznetsov
@ 2014-11-27  3:02 ` Greg Kroah-Hartman
  2014-11-27  3:24   ` Dexuan Cui
  2014-12-01 10:52 ` [PATCH v2] " Vitaly Kuznetsov
  1 sibling, 1 reply; 14+ messages in thread
From: Greg Kroah-Hartman @ 2014-11-27  3:02 UTC (permalink / raw)
  To: Vitaly Kuznetsov; +Cc: K. Y. Srinivasan, Haiyang Zhang, devel, linux-kernel

On Wed, Nov 26, 2014 at 02:52:22PM +0100, Vitaly Kuznetsov wrote:
> When an SMP Hyper-V guest is running on top of 2012R2 Server and secondary
> cpus are sent offline (with echo 0 > /sys/devices/system/cpu/cpu$cpu/online)
> the system freeze is observed. This happens due to the fact that on newer
> hypervisors (Win8, WS2012R2, ...) vmbus channel handlers are distributed
> across all cpus (see init_vp_index() function in drivers/hv/channel_mgmt.c)
> and on cpu offlining nobody reassigns them to CPU0. Prevent cpu offlining
> when vmbus is loaded until the issue is fixed host-side.
> 
> This patch also disables hibernation but it is OK as it is also broken (MCE
> error is hit on resume). Suspend still works.
> 
> Tested with WS2008R2 and WS2012R2.
> 
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> ---
>  drivers/hv/vmbus_drv.c | 19 +++++++++++++++++++
>  1 file changed, 19 insertions(+)
> 
> diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c
> index 4d6b269..9a82249 100644
> --- a/drivers/hv/vmbus_drv.c
> +++ b/drivers/hv/vmbus_drv.c
> @@ -32,6 +32,7 @@
>  #include <linux/completion.h>
>  #include <linux/hyperv.h>
>  #include <linux/kernel_stat.h>
> +#include <linux/cpu.h>
>  #include <asm/hyperv.h>
>  #include <asm/hypervisor.h>
>  #include <asm/mshyperv.h>
> @@ -671,6 +672,13 @@ static void vmbus_isr(void)
>  		tasklet_schedule(&msg_dpc);
>  }
>  
> +#ifdef CONFIG_HOTPLUG_CPU
> +static int hyperv_cpu_disable(void)
> +{
> +	return -1;
> +}
> +#endif
> +
>  /*
>   * vmbus_bus_init -Main vmbus driver initialization routine.
>   *
> @@ -711,6 +719,12 @@ static int vmbus_bus_init(int irq)
>  	if (ret)
>  		goto err_alloc;
>  
> +#ifdef CONFIG_HOTPLUG_CPU
> +	if ((vmbus_proto_version != VERSION_WS2008) &&
> +	    (vmbus_proto_version != VERSION_WIN7))
> +		smp_ops.cpu_disable = hyperv_cpu_disable;
> +#endif
> +
>  	vmbus_request_offers();
>  
>  	return 0;
> @@ -964,6 +978,11 @@ static void __exit vmbus_exit(void)
>  	bus_unregister(&hv_bus);
>  	hv_cleanup();
>  	acpi_bus_unregister_driver(&vmbus_acpi_driver);
> +#ifdef CONFIG_HOTPLUG_CPU
> +	if ((vmbus_proto_version != VERSION_WS2008) &&
> +	    (vmbus_proto_version != VERSION_WIN7))
> +		smp_ops.cpu_disable = native_cpu_disable;
> +#endif
>  }

#ifdef in a .c file is not a good idea to do if at all possible, please
only put this in one place, using a function call to "hide" the mess.

greg k-h

^ permalink raw reply	[flat|nested] 14+ messages in thread

* RE: [PATCH] Drivers: hv: vmbus: prevent cpu offlining on newer hypervisors
  2014-11-27  3:02 ` Greg Kroah-Hartman
@ 2014-11-27  3:24   ` Dexuan Cui
  2014-11-27  9:52     ` Vitaly Kuznetsov
  0 siblings, 1 reply; 14+ messages in thread
From: Dexuan Cui @ 2014-11-27  3:24 UTC (permalink / raw)
  To: Greg Kroah-Hartman, Vitaly Kuznetsov; +Cc: devel, Haiyang Zhang, linux-kernel

> -----Original Message-----
> From: devel [mailto:driverdev-devel-bounces@linuxdriverproject.org] On
> Behalf Of Greg Kroah-Hartman
> Sent: Thursday, November 27, 2014 11:03 AM
> To: Vitaly Kuznetsov
> Cc: devel@linuxdriverproject.org; Haiyang Zhang; linux-
> kernel@vger.kernel.org
> Subject: Re: [PATCH] Drivers: hv: vmbus: prevent cpu offlining on newer
> hypervisors
> 
> On Wed, Nov 26, 2014 at 02:52:22PM +0100, Vitaly Kuznetsov wrote:
> > When an SMP Hyper-V guest is running on top of 2012R2 Server and
> secondary
> > cpus are sent offline (with echo 0 >
> /sys/devices/system/cpu/cpu$cpu/online)
> > the system freeze is observed. This happens due to the fact that on newer
> > hypervisors (Win8, WS2012R2, ...) vmbus channel handlers are
> distributed
> > across all cpus (see init_vp_index() function in
> drivers/hv/channel_mgmt.c)
> > and on cpu offlining nobody reassigns them to CPU0. Prevent cpu
> offlining
> > when vmbus is loaded until the issue is fixed host-side.
> >
> > This patch also disables hibernation but it is OK as it is also broken (MCE
> > error is hit on resume). Suspend still works.
> >
> > Tested with WS2008R2 and WS2012R2.
> >
> > Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> > ---
> >  drivers/hv/vmbus_drv.c | 19 +++++++++++++++++++
> >  1 file changed, 19 insertions(+)
> >
> > diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c
> > index 4d6b269..9a82249 100644
> > --- a/drivers/hv/vmbus_drv.c
> > +++ b/drivers/hv/vmbus_drv.c
> > @@ -32,6 +32,7 @@
> >  #include <linux/completion.h>
> >  #include <linux/hyperv.h>
> >  #include <linux/kernel_stat.h>
> > +#include <linux/cpu.h>
> >  #include <asm/hyperv.h>
> >  #include <asm/hypervisor.h>
> >  #include <asm/mshyperv.h>
> > @@ -671,6 +672,13 @@ static void vmbus_isr(void)
> >  		tasklet_schedule(&msg_dpc);
> >  }
> >
> > +#ifdef CONFIG_HOTPLUG_CPU
> > +static int hyperv_cpu_disable(void)
> > +{
> > +	return -1;
> > +}
> > +#endif
> > +
> >  /*
> >   * vmbus_bus_init -Main vmbus driver initialization routine.
> >   *
> > @@ -711,6 +719,12 @@ static int vmbus_bus_init(int irq)
> >  	if (ret)
> >  		goto err_alloc;
> >
> > +#ifdef CONFIG_HOTPLUG_CPU
> > +	if ((vmbus_proto_version != VERSION_WS2008) &&
> > +	    (vmbus_proto_version != VERSION_WIN7))
> > +		smp_ops.cpu_disable = hyperv_cpu_disable;
> > +#endif
> > +
> >  	vmbus_request_offers();
> >
> >  	return 0;
> > @@ -964,6 +978,11 @@ static void __exit vmbus_exit(void)
> >  	bus_unregister(&hv_bus);
> >  	hv_cleanup();
> >  	acpi_bus_unregister_driver(&vmbus_acpi_driver);
> > +#ifdef CONFIG_HOTPLUG_CPU
> > +	if ((vmbus_proto_version != VERSION_WS2008) &&
> > +	    (vmbus_proto_version != VERSION_WIN7))
> > +		smp_ops.cpu_disable = native_cpu_disable;
> > +#endif
> >  }
> 
> #ifdef in a .c file is not a good idea to do if at all possible, please
> only put this in one place, using a function call to "hide" the mess.
> 
> greg k-h

Hi Vitaly,
The idea of the patch is good to me.

I agree with Greg.
BTW, maybe hv_cpu_hotplug_quirk() is a better name?

Thanks,
-- Dexuan

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH] Drivers: hv: vmbus: prevent cpu offlining on newer hypervisors
  2014-11-27  3:24   ` Dexuan Cui
@ 2014-11-27  9:52     ` Vitaly Kuznetsov
  0 siblings, 0 replies; 14+ messages in thread
From: Vitaly Kuznetsov @ 2014-11-27  9:52 UTC (permalink / raw)
  To: Dexuan Cui; +Cc: Greg Kroah-Hartman, devel, Haiyang Zhang, linux-kernel

Dexuan Cui <decui@microsoft.com> writes:

>> -----Original Message-----
>> From: devel [mailto:driverdev-devel-bounces@linuxdriverproject.org] On
>> Behalf Of Greg Kroah-Hartman
>> Sent: Thursday, November 27, 2014 11:03 AM
>> To: Vitaly Kuznetsov
>> Cc: devel@linuxdriverproject.org; Haiyang Zhang; linux-
>> kernel@vger.kernel.org
>> Subject: Re: [PATCH] Drivers: hv: vmbus: prevent cpu offlining on newer
>> hypervisors
>> 
>> On Wed, Nov 26, 2014 at 02:52:22PM +0100, Vitaly Kuznetsov wrote:
>> > When an SMP Hyper-V guest is running on top of 2012R2 Server and
>> secondary
>> > cpus are sent offline (with echo 0 >
>> /sys/devices/system/cpu/cpu$cpu/online)
>> > the system freeze is observed. This happens due to the fact that on newer
>> > hypervisors (Win8, WS2012R2, ...) vmbus channel handlers are
>> distributed
>> > across all cpus (see init_vp_index() function in
>> drivers/hv/channel_mgmt.c)
>> > and on cpu offlining nobody reassigns them to CPU0. Prevent cpu
>> offlining
>> > when vmbus is loaded until the issue is fixed host-side.
>> >
>> > This patch also disables hibernation but it is OK as it is also broken (MCE
>> > error is hit on resume). Suspend still works.
>> >
>> > Tested with WS2008R2 and WS2012R2.
>> >
>> > Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
>> > ---
>> >  drivers/hv/vmbus_drv.c | 19 +++++++++++++++++++
>> >  1 file changed, 19 insertions(+)
>> >
>> > diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c
>> > index 4d6b269..9a82249 100644
>> > --- a/drivers/hv/vmbus_drv.c
>> > +++ b/drivers/hv/vmbus_drv.c
>> > @@ -32,6 +32,7 @@
>> >  #include <linux/completion.h>
>> >  #include <linux/hyperv.h>
>> >  #include <linux/kernel_stat.h>
>> > +#include <linux/cpu.h>
>> >  #include <asm/hyperv.h>
>> >  #include <asm/hypervisor.h>
>> >  #include <asm/mshyperv.h>
>> > @@ -671,6 +672,13 @@ static void vmbus_isr(void)
>> >  		tasklet_schedule(&msg_dpc);
>> >  }
>> >
>> > +#ifdef CONFIG_HOTPLUG_CPU
>> > +static int hyperv_cpu_disable(void)
>> > +{
>> > +	return -1;
>> > +}
>> > +#endif
>> > +
>> >  /*
>> >   * vmbus_bus_init -Main vmbus driver initialization routine.
>> >   *
>> > @@ -711,6 +719,12 @@ static int vmbus_bus_init(int irq)
>> >  	if (ret)
>> >  		goto err_alloc;
>> >
>> > +#ifdef CONFIG_HOTPLUG_CPU
>> > +	if ((vmbus_proto_version != VERSION_WS2008) &&
>> > +	    (vmbus_proto_version != VERSION_WIN7))
>> > +		smp_ops.cpu_disable = hyperv_cpu_disable;
>> > +#endif
>> > +
>> >  	vmbus_request_offers();
>> >
>> >  	return 0;
>> > @@ -964,6 +978,11 @@ static void __exit vmbus_exit(void)
>> >  	bus_unregister(&hv_bus);
>> >  	hv_cleanup();
>> >  	acpi_bus_unregister_driver(&vmbus_acpi_driver);
>> > +#ifdef CONFIG_HOTPLUG_CPU
>> > +	if ((vmbus_proto_version != VERSION_WS2008) &&
>> > +	    (vmbus_proto_version != VERSION_WIN7))
>> > +		smp_ops.cpu_disable = native_cpu_disable;
>> > +#endif
>> >  }
>> 
>> #ifdef in a .c file is not a good idea to do if at all possible, please
>> only put this in one place, using a function call to "hide" the mess.
>> 
>> greg k-h
>
> Hi Vitaly,
> The idea of the patch is good to me.
>
> I agree with Greg.
> BTW, maybe hv_cpu_hotplug_quirk() is a better name?

My idea was that eventually this function will start doing something
real (e.g. switching channels to cpu0 if it doesn't happen fully
host-side) so I called it with a general name 'hyperv_cpu_disable'.

I'll try addressing our and Greg's comments in v2, thanks!

>
> Thanks,
> -- Dexuan

-- 
  Vitaly

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH v2] Drivers: hv: vmbus: prevent cpu offlining on newer hypervisors
  2014-11-26 13:52 [PATCH] Drivers: hv: vmbus: prevent cpu offlining on newer hypervisors Vitaly Kuznetsov
  2014-11-27  3:02 ` Greg Kroah-Hartman
@ 2014-12-01 10:52 ` Vitaly Kuznetsov
  2014-12-01 11:12   ` Dexuan Cui
                     ` (2 more replies)
  1 sibling, 3 replies; 14+ messages in thread
From: Vitaly Kuznetsov @ 2014-12-01 10:52 UTC (permalink / raw)
  To: K. Y. Srinivasan, Haiyang Zhang
  Cc: devel, Greg Kroah-Hartman, linux-kernel, Dexuan Cui

When an SMP Hyper-V guest is running on top of 2012R2 Server and secondary
cpus are sent offline (with echo 0 > /sys/devices/system/cpu/cpu$cpu/online)
the system freeze is observed. This happens due to the fact that on newer
hypervisors (Win8, WS2012R2, ...) vmbus channel handlers are distributed
across all cpus (see init_vp_index() function in drivers/hv/channel_mgmt.c)
and on cpu offlining nobody reassigns them to CPU0. Prevent cpu offlining
when vmbus is loaded until the issue is fixed host-side.

This patch also disables hibernation but it is OK as it is also broken (MCE
error is hit on resume). Suspend still works.

Tested with WS2008R2 and WS2012R2.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>

---
Changes since v1:
- introduce hv_cpu_hotplug_quirk() function to not spread #ifdefs [Greg KH]
- add pr_notice() message "hv_vmbus: CPU offlining is not supported by hypervisor"
---
 drivers/hv/vmbus_drv.c | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c
index 4d6b269..2e6b38e 100644
--- a/drivers/hv/vmbus_drv.c
+++ b/drivers/hv/vmbus_drv.c
@@ -32,6 +32,7 @@
 #include <linux/completion.h>
 #include <linux/hyperv.h>
 #include <linux/kernel_stat.h>
+#include <linux/cpu.h>
 #include <asm/hyperv.h>
 #include <asm/hypervisor.h>
 #include <asm/mshyperv.h>
@@ -671,6 +672,36 @@ static void vmbus_isr(void)
 		tasklet_schedule(&msg_dpc);
 }
 
+#ifdef CONFIG_HOTPLUG_CPU
+static int hyperv_cpu_disable(void)
+{
+	return -1;
+}
+
+static void hv_cpu_hotplug_quirk(bool vmbus_loaded)
+{
+	/*
+	 * Offlining a CPU when running on newer hypervisors (WS2012R2, Win8,
+	 * ...) is not supported at this moment as channel interrupts are
+	 * distributed across all of them.
+	 */
+
+	if ((vmbus_proto_version == VERSION_WS2008) ||
+	    (vmbus_proto_version == VERSION_WIN7))
+		return;
+
+	if (vmbus_loaded) {
+		smp_ops.cpu_disable = hyperv_cpu_disable;
+		pr_notice("CPU offlining is not supported by hypervisor");
+	} else
+		smp_ops.cpu_disable = native_cpu_disable;
+}
+#else
+static void hv_cpu_hotplug_quirk(bool vmbus_loaded)
+{
+}
+#endif
+
 /*
  * vmbus_bus_init -Main vmbus driver initialization routine.
  *
@@ -711,6 +742,7 @@ static int vmbus_bus_init(int irq)
 	if (ret)
 		goto err_alloc;
 
+	hv_cpu_hotplug_quirk(true);
 	vmbus_request_offers();
 
 	return 0;
@@ -964,6 +996,7 @@ static void __exit vmbus_exit(void)
 	bus_unregister(&hv_bus);
 	hv_cleanup();
 	acpi_bus_unregister_driver(&vmbus_acpi_driver);
+	hv_cpu_hotplug_quirk(false);
 }
 
 
-- 
1.9.3


^ permalink raw reply	[flat|nested] 14+ messages in thread

* RE: [PATCH v2] Drivers: hv: vmbus: prevent cpu offlining on newer hypervisors
  2014-12-01 10:52 ` [PATCH v2] " Vitaly Kuznetsov
@ 2014-12-01 11:12   ` Dexuan Cui
  2015-01-09 20:57   ` Greg Kroah-Hartman
  2015-01-12 16:50   ` [PATCH v3] " Vitaly Kuznetsov
  2 siblings, 0 replies; 14+ messages in thread
From: Dexuan Cui @ 2014-12-01 11:12 UTC (permalink / raw)
  To: Vitaly Kuznetsov, KY Srinivasan, Haiyang Zhang
  Cc: devel, Greg Kroah-Hartman, linux-kernel

> -----Original Message-----
> From: Vitaly Kuznetsov [mailto:vkuznets@redhat.com]
> Sent: Monday, December 1, 2014 18:53 PM
> To: KY Srinivasan; Haiyang Zhang
> Cc: devel@linuxdriverproject.org; Greg Kroah-Hartman; linux-
> kernel@vger.kernel.org; Dexuan Cui
> Subject: [PATCH v2] Drivers: hv: vmbus: prevent cpu offlining on newer
> hypervisors
> 
> When an SMP Hyper-V guest is running on top of 2012R2 Server and
> secondary
> cpus are sent offline (with echo 0 > /sys/devices/system/cpu/cpu$cpu/online)
> the system freeze is observed. This happens due to the fact that on newer
> hypervisors (Win8, WS2012R2, ...) vmbus channel handlers are distributed
> across all cpus (see init_vp_index() function in drivers/hv/channel_mgmt.c)
> and on cpu offlining nobody reassigns them to CPU0. Prevent cpu offlining
> when vmbus is loaded until the issue is fixed host-side.
> 
> This patch also disables hibernation but it is OK as it is also broken (MCE
> error is hit on resume). Suspend still works.
> 
> Tested with WS2008R2 and WS2012R2.
> 
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> 
> ---
> Changes since v1:
> - introduce hv_cpu_hotplug_quirk() function to not spread #ifdefs [Greg KH]
> - add pr_notice() message "hv_vmbus: CPU offlining is not supported by
> hypervisor"
> ---
>  drivers/hv/vmbus_drv.c | 33 +++++++++++++++++++++++++++++++++
>  1 file changed, 33 insertions(+)
> 
> diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c
> index 4d6b269..2e6b38e 100644
> --- a/drivers/hv/vmbus_drv.c
> +++ b/drivers/hv/vmbus_drv.c
> @@ -32,6 +32,7 @@
>  #include <linux/completion.h>
>  #include <linux/hyperv.h>
>  #include <linux/kernel_stat.h>
> +#include <linux/cpu.h>
>  #include <asm/hyperv.h>
>  #include <asm/hypervisor.h>
>  #include <asm/mshyperv.h>
> @@ -671,6 +672,36 @@ static void vmbus_isr(void)
>  		tasklet_schedule(&msg_dpc);
>  }
> 
> +#ifdef CONFIG_HOTPLUG_CPU
> +static int hyperv_cpu_disable(void)
> +{
> +	return -1;
> +}
> +
> +static void hv_cpu_hotplug_quirk(bool vmbus_loaded)
> +{
> +	/*
> +	 * Offlining a CPU when running on newer hypervisors (WS2012R2,
> Win8,
> +	 * ...) is not supported at this moment as channel interrupts are
> +	 * distributed across all of them.
> +	 */
> +
> +	if ((vmbus_proto_version == VERSION_WS2008) ||
> +	    (vmbus_proto_version == VERSION_WIN7))
> +		return;
> +
> +	if (vmbus_loaded) {
> +		smp_ops.cpu_disable = hyperv_cpu_disable;
> +		pr_notice("CPU offlining is not supported by hypervisor");
> +	} else
> +		smp_ops.cpu_disable = native_cpu_disable;
> +}
> +#else
> +static void hv_cpu_hotplug_quirk(bool vmbus_loaded)
> +{
> +}
> +#endif
> +
>  /*
>   * vmbus_bus_init -Main vmbus driver initialization routine.
>   *
> @@ -711,6 +742,7 @@ static int vmbus_bus_init(int irq)
>  	if (ret)
>  		goto err_alloc;
> 
> +	hv_cpu_hotplug_quirk(true);
>  	vmbus_request_offers();
> 
>  	return 0;
> @@ -964,6 +996,7 @@ static void __exit vmbus_exit(void)
>  	bus_unregister(&hv_bus);
>  	hv_cleanup();
>  	acpi_bus_unregister_driver(&vmbus_acpi_driver);
> +	hv_cpu_hotplug_quirk(false);
>  }

Acked-by: Dexuan Cui <decui@microsoft.com>


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH v2] Drivers: hv: vmbus: prevent cpu offlining on newer hypervisors
  2014-12-01 10:52 ` [PATCH v2] " Vitaly Kuznetsov
  2014-12-01 11:12   ` Dexuan Cui
@ 2015-01-09 20:57   ` Greg Kroah-Hartman
  2015-01-12 16:50   ` [PATCH v3] " Vitaly Kuznetsov
  2 siblings, 0 replies; 14+ messages in thread
From: Greg Kroah-Hartman @ 2015-01-09 20:57 UTC (permalink / raw)
  To: Vitaly Kuznetsov
  Cc: K. Y. Srinivasan, Haiyang Zhang, devel, linux-kernel, Dexuan Cui

On Mon, Dec 01, 2014 at 11:52:41AM +0100, Vitaly Kuznetsov wrote:
> When an SMP Hyper-V guest is running on top of 2012R2 Server and secondary
> cpus are sent offline (with echo 0 > /sys/devices/system/cpu/cpu$cpu/online)
> the system freeze is observed. This happens due to the fact that on newer
> hypervisors (Win8, WS2012R2, ...) vmbus channel handlers are distributed
> across all cpus (see init_vp_index() function in drivers/hv/channel_mgmt.c)
> and on cpu offlining nobody reassigns them to CPU0. Prevent cpu offlining
> when vmbus is loaded until the issue is fixed host-side.
> 
> This patch also disables hibernation but it is OK as it is also broken (MCE
> error is hit on resume). Suspend still works.
> 
> Tested with WS2008R2 and WS2012R2.
> 
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> Acked-by: Dexuan Cui <decui@microsoft.com>
> 
> ---
> Changes since v1:
> - introduce hv_cpu_hotplug_quirk() function to not spread #ifdefs [Greg KH]
> - add pr_notice() message "hv_vmbus: CPU offlining is not supported by hypervisor"
> ---
>  drivers/hv/vmbus_drv.c | 33 +++++++++++++++++++++++++++++++++
>  1 file changed, 33 insertions(+)
> 
> diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c
> index 4d6b269..2e6b38e 100644
> --- a/drivers/hv/vmbus_drv.c
> +++ b/drivers/hv/vmbus_drv.c
> @@ -32,6 +32,7 @@
>  #include <linux/completion.h>
>  #include <linux/hyperv.h>
>  #include <linux/kernel_stat.h>
> +#include <linux/cpu.h>
>  #include <asm/hyperv.h>
>  #include <asm/hypervisor.h>
>  #include <asm/mshyperv.h>
> @@ -671,6 +672,36 @@ static void vmbus_isr(void)
>  		tasklet_schedule(&msg_dpc);
>  }
>  
> +#ifdef CONFIG_HOTPLUG_CPU
> +static int hyperv_cpu_disable(void)
> +{
> +	return -1;
> +}
> +
> +static void hv_cpu_hotplug_quirk(bool vmbus_loaded)
> +{
> +	/*
> +	 * Offlining a CPU when running on newer hypervisors (WS2012R2, Win8,
> +	 * ...) is not supported at this moment as channel interrupts are
> +	 * distributed across all of them.
> +	 */
> +
> +	if ((vmbus_proto_version == VERSION_WS2008) ||
> +	    (vmbus_proto_version == VERSION_WIN7))
> +		return;
> +
> +	if (vmbus_loaded) {
> +		smp_ops.cpu_disable = hyperv_cpu_disable;
> +		pr_notice("CPU offlining is not supported by hypervisor");
> +	} else
> +		smp_ops.cpu_disable = native_cpu_disable;
> +}
> +#else
> +static void hv_cpu_hotplug_quirk(bool vmbus_loaded)
> +{
> +}
> +#endif
> +
>  /*
>   * vmbus_bus_init -Main vmbus driver initialization routine.
>   *
> @@ -711,6 +742,7 @@ static int vmbus_bus_init(int irq)
>  	if (ret)
>  		goto err_alloc;
>  
> +	hv_cpu_hotplug_quirk(true);
>  	vmbus_request_offers();
>  
>  	return 0;
> @@ -964,6 +996,7 @@ static void __exit vmbus_exit(void)
>  	bus_unregister(&hv_bus);
>  	hv_cleanup();
>  	acpi_bus_unregister_driver(&vmbus_acpi_driver);
> +	hv_cpu_hotplug_quirk(false);
>  }
>  
>  
> -- 
> 1.9.3

This breaks the build:
ERROR: "native_cpu_disable" [drivers/hv/hv_vmbus.ko] undefined!

Please test better.

greg k-h

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH v3] Drivers: hv: vmbus: prevent cpu offlining on newer hypervisors
  2014-12-01 10:52 ` [PATCH v2] " Vitaly Kuznetsov
  2014-12-01 11:12   ` Dexuan Cui
  2015-01-09 20:57   ` Greg Kroah-Hartman
@ 2015-01-12 16:50   ` Vitaly Kuznetsov
  2015-01-12 18:54     ` KY Srinivasan
  2015-01-25 13:36     ` Greg Kroah-Hartman
  2 siblings, 2 replies; 14+ messages in thread
From: Vitaly Kuznetsov @ 2015-01-12 16:50 UTC (permalink / raw)
  To: K. Y. Srinivasan, devel
  Cc: Haiyang Zhang, Greg Kroah-Hartman, linux-kernel, Dexuan Cui

When an SMP Hyper-V guest is running on top of 2012R2 Server and secondary
cpus are sent offline (with echo 0 > /sys/devices/system/cpu/cpu$cpu/online)
the system freeze is observed. This happens due to the fact that on newer
hypervisors (Win8, WS2012R2, ...) vmbus channel handlers are distributed
across all cpus (see init_vp_index() function in drivers/hv/channel_mgmt.c)
and on cpu offlining nobody reassigns them to CPU0. Prevent cpu offlining
when vmbus is loaded until the issue is fixed host-side.

This patch also disables hibernation but it is OK as it is also broken (MCE
error is hit on resume). Suspend still works.

Tested with WS2008R2 and WS2012R2.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
Changes since v2:
- repair the build when vmbus is builded as a module [Greg KH] by saving
  current cpu_disable pointer to previous_cpu_disable and restoring it on
  unload;
- return -ENOSYS (same as native_cpu_disable when !CONFIG_HOTPLUG_CPU) instead
  of -1 in hyperv_cpu_disable().

Changes since v1:
- introduce hv_cpu_hotplug_quirk() function to not spread #ifdefs [Greg KH];
- add pr_notice() message "hv_vmbus: CPU offlining is not supported by
  hypervisor".
---
 drivers/hv/vmbus_drv.c | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c
index 4d6b269..233da0b 100644
--- a/drivers/hv/vmbus_drv.c
+++ b/drivers/hv/vmbus_drv.c
@@ -32,6 +32,7 @@
 #include <linux/completion.h>
 #include <linux/hyperv.h>
 #include <linux/kernel_stat.h>
+#include <linux/cpu.h>
 #include <asm/hyperv.h>
 #include <asm/hypervisor.h>
 #include <asm/mshyperv.h>
@@ -671,6 +672,39 @@ static void vmbus_isr(void)
 		tasklet_schedule(&msg_dpc);
 }
 
+#ifdef CONFIG_HOTPLUG_CPU
+static int hyperv_cpu_disable(void)
+{
+	return -ENOSYS;
+}
+
+static void hv_cpu_hotplug_quirk(bool vmbus_loaded)
+{
+	static void *previous_cpu_disable;
+
+	/*
+	 * Offlining a CPU when running on newer hypervisors (WS2012R2, Win8,
+	 * ...) is not supported at this moment as channel interrupts are
+	 * distributed across all of them.
+	 */
+
+	if ((vmbus_proto_version == VERSION_WS2008) ||
+	    (vmbus_proto_version == VERSION_WIN7))
+		return;
+
+	if (vmbus_loaded) {
+		previous_cpu_disable = smp_ops.cpu_disable;
+		smp_ops.cpu_disable = hyperv_cpu_disable;
+		pr_notice("CPU offlining is not supported by hypervisor\n");
+	} else if (previous_cpu_disable)
+		smp_ops.cpu_disable = previous_cpu_disable;
+}
+#else
+static void hv_cpu_hotplug_quirk(bool vmbus_loaded)
+{
+}
+#endif
+
 /*
  * vmbus_bus_init -Main vmbus driver initialization routine.
  *
@@ -711,6 +745,7 @@ static int vmbus_bus_init(int irq)
 	if (ret)
 		goto err_alloc;
 
+	hv_cpu_hotplug_quirk(true);
 	vmbus_request_offers();
 
 	return 0;
@@ -964,6 +999,7 @@ static void __exit vmbus_exit(void)
 	bus_unregister(&hv_bus);
 	hv_cleanup();
 	acpi_bus_unregister_driver(&vmbus_acpi_driver);
+	hv_cpu_hotplug_quirk(false);
 }
 
 
-- 
1.9.3


^ permalink raw reply	[flat|nested] 14+ messages in thread

* RE: [PATCH v3] Drivers: hv: vmbus: prevent cpu offlining on newer hypervisors
  2015-01-12 16:50   ` [PATCH v3] " Vitaly Kuznetsov
@ 2015-01-12 18:54     ` KY Srinivasan
  2015-01-25 13:36     ` Greg Kroah-Hartman
  1 sibling, 0 replies; 14+ messages in thread
From: KY Srinivasan @ 2015-01-12 18:54 UTC (permalink / raw)
  To: Vitaly Kuznetsov, devel
  Cc: Haiyang Zhang, Greg Kroah-Hartman, linux-kernel, Dexuan Cui



> -----Original Message-----
> From: Vitaly Kuznetsov [mailto:vkuznets@redhat.com]
> Sent: Monday, January 12, 2015 8:50 AM
> To: KY Srinivasan; devel@linuxdriverproject.org
> Cc: Haiyang Zhang; Greg Kroah-Hartman; linux-kernel@vger.kernel.org;
> Dexuan Cui
> Subject: [PATCH v3] Drivers: hv: vmbus: prevent cpu offlining on newer
> hypervisors
> 
> When an SMP Hyper-V guest is running on top of 2012R2 Server and
> secondary
> cpus are sent offline (with echo 0 >
> /sys/devices/system/cpu/cpu$cpu/online)
> the system freeze is observed. This happens due to the fact that on newer
> hypervisors (Win8, WS2012R2, ...) vmbus channel handlers are distributed
> across all cpus (see init_vp_index() function in drivers/hv/channel_mgmt.c)
> and on cpu offlining nobody reassigns them to CPU0. Prevent cpu offlining
> when vmbus is loaded until the issue is fixed host-side.
> 
> This patch also disables hibernation but it is OK as it is also broken (MCE
> error is hit on resume). Suspend still works.
> 
> Tested with WS2008R2 and WS2012R2.
> 
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Thank you.
Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
> ---
> Changes since v2:
> - repair the build when vmbus is builded as a module [Greg KH] by saving
>   current cpu_disable pointer to previous_cpu_disable and restoring it on
>   unload;
> - return -ENOSYS (same as native_cpu_disable when
> !CONFIG_HOTPLUG_CPU) instead
>   of -1 in hyperv_cpu_disable().
> 
> Changes since v1:
> - introduce hv_cpu_hotplug_quirk() function to not spread #ifdefs [Greg
> KH];
> - add pr_notice() message "hv_vmbus: CPU offlining is not supported by
>   hypervisor".
> ---
>  drivers/hv/vmbus_drv.c | 36 ++++++++++++++++++++++++++++++++++++
>  1 file changed, 36 insertions(+)
> 
> diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c
> index 4d6b269..233da0b 100644
> --- a/drivers/hv/vmbus_drv.c
> +++ b/drivers/hv/vmbus_drv.c
> @@ -32,6 +32,7 @@
>  #include <linux/completion.h>
>  #include <linux/hyperv.h>
>  #include <linux/kernel_stat.h>
> +#include <linux/cpu.h>
>  #include <asm/hyperv.h>
>  #include <asm/hypervisor.h>
>  #include <asm/mshyperv.h>
> @@ -671,6 +672,39 @@ static void vmbus_isr(void)
>  		tasklet_schedule(&msg_dpc);
>  }
> 
> +#ifdef CONFIG_HOTPLUG_CPU
> +static int hyperv_cpu_disable(void)
> +{
> +	return -ENOSYS;
> +}
> +
> +static void hv_cpu_hotplug_quirk(bool vmbus_loaded)
> +{
> +	static void *previous_cpu_disable;
> +
> +	/*
> +	 * Offlining a CPU when running on newer hypervisors (WS2012R2,
> Win8,
> +	 * ...) is not supported at this moment as channel interrupts are
> +	 * distributed across all of them.
> +	 */
> +
> +	if ((vmbus_proto_version == VERSION_WS2008) ||
> +	    (vmbus_proto_version == VERSION_WIN7))
> +		return;
> +
> +	if (vmbus_loaded) {
> +		previous_cpu_disable = smp_ops.cpu_disable;
> +		smp_ops.cpu_disable = hyperv_cpu_disable;
> +		pr_notice("CPU offlining is not supported by hypervisor\n");
> +	} else if (previous_cpu_disable)
> +		smp_ops.cpu_disable = previous_cpu_disable;
> +}
> +#else
> +static void hv_cpu_hotplug_quirk(bool vmbus_loaded)
> +{
> +}
> +#endif
> +
>  /*
>   * vmbus_bus_init -Main vmbus driver initialization routine.
>   *
> @@ -711,6 +745,7 @@ static int vmbus_bus_init(int irq)
>  	if (ret)
>  		goto err_alloc;
> 
> +	hv_cpu_hotplug_quirk(true);
>  	vmbus_request_offers();
> 
>  	return 0;
> @@ -964,6 +999,7 @@ static void __exit vmbus_exit(void)
>  	bus_unregister(&hv_bus);
>  	hv_cleanup();
>  	acpi_bus_unregister_driver(&vmbus_acpi_driver);
> +	hv_cpu_hotplug_quirk(false);
>  }
> 
> 
> --
> 1.9.3


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH v3] Drivers: hv: vmbus: prevent cpu offlining on newer hypervisors
  2015-01-12 16:50   ` [PATCH v3] " Vitaly Kuznetsov
  2015-01-12 18:54     ` KY Srinivasan
@ 2015-01-25 13:36     ` Greg Kroah-Hartman
  2015-01-26 10:38       ` Vitaly Kuznetsov
  1 sibling, 1 reply; 14+ messages in thread
From: Greg Kroah-Hartman @ 2015-01-25 13:36 UTC (permalink / raw)
  To: Vitaly Kuznetsov
  Cc: K. Y. Srinivasan, devel, Haiyang Zhang, linux-kernel, Dexuan Cui

On Mon, Jan 12, 2015 at 05:50:11PM +0100, Vitaly Kuznetsov wrote:
> When an SMP Hyper-V guest is running on top of 2012R2 Server and secondary
> cpus are sent offline (with echo 0 > /sys/devices/system/cpu/cpu$cpu/online)
> the system freeze is observed. This happens due to the fact that on newer
> hypervisors (Win8, WS2012R2, ...) vmbus channel handlers are distributed
> across all cpus (see init_vp_index() function in drivers/hv/channel_mgmt.c)
> and on cpu offlining nobody reassigns them to CPU0. Prevent cpu offlining
> when vmbus is loaded until the issue is fixed host-side.
> 
> This patch also disables hibernation but it is OK as it is also broken (MCE
> error is hit on resume). Suspend still works.
> 
> Tested with WS2008R2 and WS2012R2.
> 
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
> ---
> Changes since v2:
> - repair the build when vmbus is builded as a module [Greg KH] by saving
>   current cpu_disable pointer to previous_cpu_disable and restoring it on
>   unload;
> - return -ENOSYS (same as native_cpu_disable when !CONFIG_HOTPLUG_CPU) instead
>   of -1 in hyperv_cpu_disable().
> 
> Changes since v1:
> - introduce hv_cpu_hotplug_quirk() function to not spread #ifdefs [Greg KH];
> - add pr_notice() message "hv_vmbus: CPU offlining is not supported by
>   hypervisor".
> ---
>  drivers/hv/vmbus_drv.c | 36 ++++++++++++++++++++++++++++++++++++
>  1 file changed, 36 insertions(+)

Doesn't apply to my char-misc-test branch at all :(

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH v3] Drivers: hv: vmbus: prevent cpu offlining on newer hypervisors
  2015-01-25 13:36     ` Greg Kroah-Hartman
@ 2015-01-26 10:38       ` Vitaly Kuznetsov
  2015-01-26 16:11         ` KY Srinivasan
  2015-01-26 22:54         ` Greg Kroah-Hartman
  0 siblings, 2 replies; 14+ messages in thread
From: Vitaly Kuznetsov @ 2015-01-26 10:38 UTC (permalink / raw)
  To: Greg Kroah-Hartman
  Cc: K. Y. Srinivasan, devel, Haiyang Zhang, linux-kernel, Dexuan Cui

[-- Attachment #1: Type: text/plain, Size: 2056 bytes --]

Greg Kroah-Hartman <gregkh@linuxfoundation.org> writes:

> On Mon, Jan 12, 2015 at 05:50:11PM +0100, Vitaly Kuznetsov wrote:
>> When an SMP Hyper-V guest is running on top of 2012R2 Server and secondary
>> cpus are sent offline (with echo 0 > /sys/devices/system/cpu/cpu$cpu/online)
>> the system freeze is observed. This happens due to the fact that on newer
>> hypervisors (Win8, WS2012R2, ...) vmbus channel handlers are distributed
>> across all cpus (see init_vp_index() function in drivers/hv/channel_mgmt.c)
>> and on cpu offlining nobody reassigns them to CPU0. Prevent cpu offlining
>> when vmbus is loaded until the issue is fixed host-side.
>> 
>> This patch also disables hibernation but it is OK as it is also broken (MCE
>> error is hit on resume). Suspend still works.
>> 
>> Tested with WS2008R2 and WS2012R2.
>> 
>> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
>> Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
>> ---
>> Changes since v2:
>> - repair the build when vmbus is builded as a module [Greg KH] by saving
>>   current cpu_disable pointer to previous_cpu_disable and restoring it on
>>   unload;
>> - return -ENOSYS (same as native_cpu_disable when !CONFIG_HOTPLUG_CPU) instead
>>   of -1 in hyperv_cpu_disable().
>> 
>> Changes since v1:
>> - introduce hv_cpu_hotplug_quirk() function to not spread #ifdefs [Greg KH];
>> - add pr_notice() message "hv_vmbus: CPU offlining is not supported by
>>   hypervisor".
>> ---
>>  drivers/hv/vmbus_drv.c | 36 ++++++++++++++++++++++++++++++++++++
>>  1 file changed, 36 insertions(+)
>
> Doesn't apply to my char-misc-test branch at all :(

Another mid-air collision with K.Y's "Drivers: hv: vmbus: Implement a
clockevent device", please use the attached version. No functional
changes are required, I just fixed the merge conflict (includes).

Othere than that (and sorry for meddling), would it it be better if
you switch to 'pull requests' workflow with K.Y? There is a lot of
ongoing work in hyperv nowdays and such collisions seem otherwise
inevitable ...

-- 
  Vitaly


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Drivers-hv-vmbus-prevent-cpu-offlining-on-newer-hype.patch --]
[-- Type: text/x-patch, Size: 2781 bytes --]

>From 95f9b0ff3f73c3bbe8aa9c525414097c2c26a3ef Mon Sep 17 00:00:00 2001
From: Vitaly Kuznetsov <vkuznets@redhat.com>
Date: Mon, 12 Jan 2015 17:50:11 +0100
Subject: [PATCH] Drivers: hv: vmbus: prevent cpu offlining on newer
 hypervisors

When an SMP Hyper-V guest is running on top of 2012R2 Server and secondary
cpus are sent offline (with echo 0 > /sys/devices/system/cpu/cpu$cpu/online)
the system freeze is observed. This happens due to the fact that on newer
hypervisors (Win8, WS2012R2, ...) vmbus channel handlers are distributed
across all cpus (see init_vp_index() function in drivers/hv/channel_mgmt.c)
and on cpu offlining nobody reassigns them to CPU0. Prevent cpu offlining
when vmbus is loaded until the issue is fixed host-side.

This patch also disables hibernation but it is OK as it is also broken (MCE
error is hit on resume). Suspend still works.

Tested with WS2008R2 and WS2012R2.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
---
 drivers/hv/vmbus_drv.c | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c
index 35e3f42..90c3400 100644
--- a/drivers/hv/vmbus_drv.c
+++ b/drivers/hv/vmbus_drv.c
@@ -33,6 +33,7 @@
 #include <linux/hyperv.h>
 #include <linux/kernel_stat.h>
 #include <linux/clockchips.h>
+#include <linux/cpu.h>
 #include <asm/hyperv.h>
 #include <asm/hypervisor.h>
 #include <asm/mshyperv.h>
@@ -704,6 +705,39 @@ static void vmbus_isr(void)
 	}
 }
 
+#ifdef CONFIG_HOTPLUG_CPU
+static int hyperv_cpu_disable(void)
+{
+	return -ENOSYS;
+}
+
+static void hv_cpu_hotplug_quirk(bool vmbus_loaded)
+{
+	static void *previous_cpu_disable;
+
+	/*
+	 * Offlining a CPU when running on newer hypervisors (WS2012R2, Win8,
+	 * ...) is not supported at this moment as channel interrupts are
+	 * distributed across all of them.
+	 */
+
+	if ((vmbus_proto_version == VERSION_WS2008) ||
+	    (vmbus_proto_version == VERSION_WIN7))
+		return;
+
+	if (vmbus_loaded) {
+		previous_cpu_disable = smp_ops.cpu_disable;
+		smp_ops.cpu_disable = hyperv_cpu_disable;
+		pr_notice("CPU offlining is not supported by hypervisor\n");
+	} else if (previous_cpu_disable)
+		smp_ops.cpu_disable = previous_cpu_disable;
+}
+#else
+static void hv_cpu_hotplug_quirk(bool vmbus_loaded)
+{
+}
+#endif
+
 /*
  * vmbus_bus_init -Main vmbus driver initialization routine.
  *
@@ -744,6 +778,7 @@ static int vmbus_bus_init(int irq)
 	if (ret)
 		goto err_alloc;
 
+	hv_cpu_hotplug_quirk(true);
 	vmbus_request_offers();
 
 	return 0;
@@ -997,6 +1032,7 @@ static void __exit vmbus_exit(void)
 	bus_unregister(&hv_bus);
 	hv_cleanup();
 	acpi_bus_unregister_driver(&vmbus_acpi_driver);
+	hv_cpu_hotplug_quirk(false);
 }
 
 
-- 
1.9.3


^ permalink raw reply	[flat|nested] 14+ messages in thread

* RE: [PATCH v3] Drivers: hv: vmbus: prevent cpu offlining on newer hypervisors
  2015-01-26 10:38       ` Vitaly Kuznetsov
@ 2015-01-26 16:11         ` KY Srinivasan
  2015-01-26 22:54         ` Greg Kroah-Hartman
  1 sibling, 0 replies; 14+ messages in thread
From: KY Srinivasan @ 2015-01-26 16:11 UTC (permalink / raw)
  To: Vitaly Kuznetsov, Greg Kroah-Hartman
  Cc: devel, Haiyang Zhang, linux-kernel, Dexuan Cui



> -----Original Message-----
> From: Vitaly Kuznetsov [mailto:vkuznets@redhat.com]
> Sent: Monday, January 26, 2015 2:39 AM
> To: Greg Kroah-Hartman
> Cc: KY Srinivasan; devel@linuxdriverproject.org; Haiyang Zhang; linux-
> kernel@vger.kernel.org; Dexuan Cui
> Subject: Re: [PATCH v3] Drivers: hv: vmbus: prevent cpu offlining on newer
> hypervisors
> 
> Greg Kroah-Hartman <gregkh@linuxfoundation.org> writes:
> 
> > On Mon, Jan 12, 2015 at 05:50:11PM +0100, Vitaly Kuznetsov wrote:
> >> When an SMP Hyper-V guest is running on top of 2012R2 Server and
> >> secondary cpus are sent offline (with echo 0 >
> >> /sys/devices/system/cpu/cpu$cpu/online)
> >> the system freeze is observed. This happens due to the fact that on
> >> newer hypervisors (Win8, WS2012R2, ...) vmbus channel handlers are
> >> distributed across all cpus (see init_vp_index() function in
> >> drivers/hv/channel_mgmt.c) and on cpu offlining nobody reassigns them
> >> to CPU0. Prevent cpu offlining when vmbus is loaded until the issue is
> fixed host-side.
> >>
> >> This patch also disables hibernation but it is OK as it is also
> >> broken (MCE error is hit on resume). Suspend still works.
> >>
> >> Tested with WS2008R2 and WS2012R2.
> >>
> >> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> >> Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
> >> ---
> >> Changes since v2:
> >> - repair the build when vmbus is builded as a module [Greg KH] by saving
> >>   current cpu_disable pointer to previous_cpu_disable and restoring it on
> >>   unload;
> >> - return -ENOSYS (same as native_cpu_disable when
> !CONFIG_HOTPLUG_CPU) instead
> >>   of -1 in hyperv_cpu_disable().
> >>
> >> Changes since v1:
> >> - introduce hv_cpu_hotplug_quirk() function to not spread #ifdefs
> >> [Greg KH];
> >> - add pr_notice() message "hv_vmbus: CPU offlining is not supported by
> >>   hypervisor".
> >> ---
> >>  drivers/hv/vmbus_drv.c | 36
> ++++++++++++++++++++++++++++++++++++
> >>  1 file changed, 36 insertions(+)
> >
> > Doesn't apply to my char-misc-test branch at all :(
> 
> Another mid-air collision with K.Y's "Drivers: hv: vmbus: Implement a
> clockevent device", please use the attached version. No functional changes
> are required, I just fixed the merge conflict (includes).
> 
> Othere than that (and sorry for meddling), would it it be better if you switch
> to 'pull requests' workflow with K.Y? There is a lot of ongoing work in hyperv
> nowdays and such collisions seem otherwise inevitable ...

I will co-ordinate and forward Hyper-V patches to Greg.

K. Y
> 
> --
>   Vitaly


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH v3] Drivers: hv: vmbus: prevent cpu offlining on newer hypervisors
  2015-01-26 10:38       ` Vitaly Kuznetsov
  2015-01-26 16:11         ` KY Srinivasan
@ 2015-01-26 22:54         ` Greg Kroah-Hartman
  2015-01-26 23:53           ` KY Srinivasan
  1 sibling, 1 reply; 14+ messages in thread
From: Greg Kroah-Hartman @ 2015-01-26 22:54 UTC (permalink / raw)
  To: Vitaly Kuznetsov
  Cc: K. Y. Srinivasan, devel, Haiyang Zhang, linux-kernel, Dexuan Cui

On Mon, Jan 26, 2015 at 11:38:54AM +0100, Vitaly Kuznetsov wrote:
> Greg Kroah-Hartman <gregkh@linuxfoundation.org> writes:
> 
> > On Mon, Jan 12, 2015 at 05:50:11PM +0100, Vitaly Kuznetsov wrote:
> >> When an SMP Hyper-V guest is running on top of 2012R2 Server and secondary
> >> cpus are sent offline (with echo 0 > /sys/devices/system/cpu/cpu$cpu/online)
> >> the system freeze is observed. This happens due to the fact that on newer
> >> hypervisors (Win8, WS2012R2, ...) vmbus channel handlers are distributed
> >> across all cpus (see init_vp_index() function in drivers/hv/channel_mgmt.c)
> >> and on cpu offlining nobody reassigns them to CPU0. Prevent cpu offlining
> >> when vmbus is loaded until the issue is fixed host-side.
> >> 
> >> This patch also disables hibernation but it is OK as it is also broken (MCE
> >> error is hit on resume). Suspend still works.
> >> 
> >> Tested with WS2008R2 and WS2012R2.
> >> 
> >> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> >> Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
> >> ---
> >> Changes since v2:
> >> - repair the build when vmbus is builded as a module [Greg KH] by saving
> >>   current cpu_disable pointer to previous_cpu_disable and restoring it on
> >>   unload;
> >> - return -ENOSYS (same as native_cpu_disable when !CONFIG_HOTPLUG_CPU) instead
> >>   of -1 in hyperv_cpu_disable().
> >> 
> >> Changes since v1:
> >> - introduce hv_cpu_hotplug_quirk() function to not spread #ifdefs [Greg KH];
> >> - add pr_notice() message "hv_vmbus: CPU offlining is not supported by
> >>   hypervisor".
> >> ---
> >>  drivers/hv/vmbus_drv.c | 36 ++++++++++++++++++++++++++++++++++++
> >>  1 file changed, 36 insertions(+)
> >
> > Doesn't apply to my char-misc-test branch at all :(
> 
> Another mid-air collision with K.Y's "Drivers: hv: vmbus: Implement a
> clockevent device", please use the attached version. No functional
> changes are required, I just fixed the merge conflict (includes).
> 
> Othere than that (and sorry for meddling), would it it be better if
> you switch to 'pull requests' workflow with K.Y? There is a lot of
> ongoing work in hyperv nowdays and such collisions seem otherwise
> inevitable ...

This is a trivial number of patches for this subsystem, so no, a git
workflow isn't needed.  But if KY would collect the patches up and send
them to me for all hyperv patches from now on, that would make these
types of issues go away, KY, can you do that from now on?

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 14+ messages in thread

* RE: [PATCH v3] Drivers: hv: vmbus: prevent cpu offlining on newer hypervisors
  2015-01-26 22:54         ` Greg Kroah-Hartman
@ 2015-01-26 23:53           ` KY Srinivasan
  0 siblings, 0 replies; 14+ messages in thread
From: KY Srinivasan @ 2015-01-26 23:53 UTC (permalink / raw)
  To: Greg Kroah-Hartman, Vitaly Kuznetsov
  Cc: devel, Haiyang Zhang, linux-kernel, Dexuan Cui



> -----Original Message-----
> From: Greg Kroah-Hartman [mailto:gregkh@linuxfoundation.org]
> Sent: Monday, January 26, 2015 2:54 PM
> To: Vitaly Kuznetsov
> Cc: KY Srinivasan; devel@linuxdriverproject.org; Haiyang Zhang; linux-
> kernel@vger.kernel.org; Dexuan Cui
> Subject: Re: [PATCH v3] Drivers: hv: vmbus: prevent cpu offlining on newer
> hypervisors
> 
> On Mon, Jan 26, 2015 at 11:38:54AM +0100, Vitaly Kuznetsov wrote:
> > Greg Kroah-Hartman <gregkh@linuxfoundation.org> writes:
> >
> > > On Mon, Jan 12, 2015 at 05:50:11PM +0100, Vitaly Kuznetsov wrote:
> > >> When an SMP Hyper-V guest is running on top of 2012R2 Server and
> > >> secondary cpus are sent offline (with echo 0 >
> > >> /sys/devices/system/cpu/cpu$cpu/online)
> > >> the system freeze is observed. This happens due to the fact that on
> > >> newer hypervisors (Win8, WS2012R2, ...) vmbus channel handlers are
> > >> distributed across all cpus (see init_vp_index() function in
> > >> drivers/hv/channel_mgmt.c) and on cpu offlining nobody reassigns
> > >> them to CPU0. Prevent cpu offlining when vmbus is loaded until the
> issue is fixed host-side.
> > >>
> > >> This patch also disables hibernation but it is OK as it is also
> > >> broken (MCE error is hit on resume). Suspend still works.
> > >>
> > >> Tested with WS2008R2 and WS2012R2.
> > >>
> > >> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> > >> Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
> > >> ---
> > >> Changes since v2:
> > >> - repair the build when vmbus is builded as a module [Greg KH] by
> saving
> > >>   current cpu_disable pointer to previous_cpu_disable and restoring it
> on
> > >>   unload;
> > >> - return -ENOSYS (same as native_cpu_disable when
> !CONFIG_HOTPLUG_CPU) instead
> > >>   of -1 in hyperv_cpu_disable().
> > >>
> > >> Changes since v1:
> > >> - introduce hv_cpu_hotplug_quirk() function to not spread #ifdefs
> > >> [Greg KH];
> > >> - add pr_notice() message "hv_vmbus: CPU offlining is not supported by
> > >>   hypervisor".
> > >> ---
> > >>  drivers/hv/vmbus_drv.c | 36
> ++++++++++++++++++++++++++++++++++++
> > >>  1 file changed, 36 insertions(+)
> > >
> > > Doesn't apply to my char-misc-test branch at all :(
> >
> > Another mid-air collision with K.Y's "Drivers: hv: vmbus: Implement a
> > clockevent device", please use the attached version. No functional
> > changes are required, I just fixed the merge conflict (includes).
> >
> > Othere than that (and sorry for meddling), would it it be better if
> > you switch to 'pull requests' workflow with K.Y? There is a lot of
> > ongoing work in hyperv nowdays and such collisions seem otherwise
> > inevitable ...
> 
> This is a trivial number of patches for this subsystem, so no, a git workflow
> isn't needed.  But if KY would collect the patches up and send them to me for
> all hyperv patches from now on, that would make these types of issues go
> away, KY, can you do that from now on?

Will do.

Thanks,

K. Y
> 
> thanks,
> 
> greg k-h

^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, other threads:[~2015-01-26 23:54 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-11-26 13:52 [PATCH] Drivers: hv: vmbus: prevent cpu offlining on newer hypervisors Vitaly Kuznetsov
2014-11-27  3:02 ` Greg Kroah-Hartman
2014-11-27  3:24   ` Dexuan Cui
2014-11-27  9:52     ` Vitaly Kuznetsov
2014-12-01 10:52 ` [PATCH v2] " Vitaly Kuznetsov
2014-12-01 11:12   ` Dexuan Cui
2015-01-09 20:57   ` Greg Kroah-Hartman
2015-01-12 16:50   ` [PATCH v3] " Vitaly Kuznetsov
2015-01-12 18:54     ` KY Srinivasan
2015-01-25 13:36     ` Greg Kroah-Hartman
2015-01-26 10:38       ` Vitaly Kuznetsov
2015-01-26 16:11         ` KY Srinivasan
2015-01-26 22:54         ` Greg Kroah-Hartman
2015-01-26 23:53           ` KY Srinivasan

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).