LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
* Linux in KVM guest segfaults when hosts runs Linux 5.1
@ 2019-05-12 11:53 Marc Haber
  2019-05-13 14:07 ` Sean Christopherson
  2019-05-13 14:10 ` Radim Krčmář
  0 siblings, 2 replies; 5+ messages in thread
From: Marc Haber @ 2019-05-12 11:53 UTC (permalink / raw)
  To: kvm, linux-kernel

Hi,

since updating my home desktop machine to kernel 5.1.1, KVM guests
started on that machine segfault after booting:
general protection fault: 0000 [#1] PREEMPT SMP NOPTI
CPU: 0 PID: 13 Comm: kworker/0:1 Not tainted 5.0.13-zgsrv20080 #5.0.13.20190505.0
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014
Workqueue: events once_deferred
RIP: 0010:native_read_pmc+0x2/0x10
Code: e2 20 89 3e 48 09 d0 c3 89 f9 89 f0 0f 30 c3 66 0f 1f 84 00 00 00 00 00 89 f0 89 f9 0f 30 31 c0 c3 0f 1f 80 00 00 00 00 89 f9 <0f> 33 48 c1 e2 20 48 09 d0 c3 0f 1f 40 00 0f 20 c0 c3 66 66 2e 0f
RSP: 0018:ffff8881b9a03e50 EFLAGS: 00010083
RAX: 0000000000000001 RBX: ffff800000000001 RCX: 0000000000000000
RDX: 000000000000002f RSI: 0000000000000000 RDI: 0000000000000000
RBP: ffff8881b590e400 R08: ffff8881b590e400 R09: 0000000000000003
R10: ffffe8ffffc05440 R11: 0000000000000000 R12: ffff8881b590e5d8
R13: 0000000000000010 R14: ffff8881b590e420 R15: ffffe8ffffc05400
FS:  0000000000000000(0000) GS:ffff8881b9a00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9bcc5c61f8 CR3: 00000001b6a24000 CR4: 00000000000006f0
Call Trace:
 <IRQ>
 x86_perf_event_update+0x3b/0x80
 x86_pmu_stop+0x84/0xa0
 x86_pmu_del+0x52/0x160
 event_sched_out.isra.59+0x95/0x190
 group_sched_out.part.61+0x51/0xc0
 ctx_sched_out+0xf2/0x220
 ctx_resched+0xb8/0xc0
 __perf_install_in_context+0x175/0x1f0
 remote_function+0x3e/0x50
 flush_smp_call_function_queue+0x30/0xe0
 smp_call_function_interrupt+0x2f/0x40
 call_function_single_interrupt+0xf/0x20
 </IRQ>
RIP: 0010:smp_call_function_many+0x1ca/0x230
Code: ee 89 c7 e8 e8 f5 51 00 3b 05 a6 23 db 00 0f 83 b2 fe ff ff 48 63 d0 48 8b 0b 48 03 0c d5 20 28 db 81 8b 51 18 83 e2 01 74 0a <f3> 90 8b 51 18 83 e2 01 75 f6 eb c8 48 c7 c2 60 17 e9 81 48 89 ee
RSP: 0018:ffffc90000cc3d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff04
RAX: 0000000000000001 RBX: ffff8881b9a21e00 RCX: ffff8881b9a647c0
RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8881b9a21e08
RBP: ffff8881b9a21e08 R08: 000000000000003e R09: 0000000000000000
R10: ffffffff81c04584 R11: 0000000000000000 R12: ffffffff81027af0
R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000008
 ? arch_unregister_cpu+0x20/0x20
 ? smp_call_function_many+0x1a8/0x230
 ? inet_ehashfn+0x29/0x100
 ? arch_unregister_cpu+0x20/0x20
 ? inet_ehashfn+0x2a/0x100
 smp_call_function+0x20/0x40
 on_each_cpu+0x18/0x70
 ? inet_ehashfn+0x29/0x100
 ? inet_ehashfn+0x2a/0x100
 text_poke_bp+0x8d/0xda
 __jump_label_transform+0x10d/0x120
 arch_jump_label_transform+0x21/0x30
 __jump_label_update+0x70/0xe0
 static_key_disable_cpuslocked+0x54/0x80
 static_key_disable+0x11/0x20
 once_deferred+0x1a/0x30
 process_one_work+0x171/0x300
 worker_thread+0x2b/0x370
 ? process_one_work+0x300/0x300
 kthread+0x108/0x120
 ? kthread_park+0x80/0x80
 ret_from_fork+0x22/0x40
Modules linked in: input_leds sg led_class virtio_balloon virtio_console qemu_fw_cfg dm_mod virtio_rng ip_tables x_tables autofs4 ext4 mbcache jbd2 fscrypto sr_mod cdrom ata_generic virtio_net net_failover failover virtio_blk virtio_pci i2c_piix4 virtio_ring ata_piix virtio libata i2c_core floppy
---[ end trace 60c8d1a075894c8d ]---
RIP: 0010:native_read_pmc+0x2/0x10
Code: e2 20 89 3e 48 09 d0 c3 89 f9 89 f0 0f 30 c3 66 0f 1f 84 00 00 00 00 00 89 f0 89 f9 0f 30 31 c0 c3 0f 1f 80 00 00 00 00 89 f9 <0f> 33 48 c1 e2 20 48 09 d0 c3 0f 1f 40 00 0f 20 c0 c3 66 66 2e 0f
RSP: 0018:ffff8881b9a03e50 EFLAGS: 00010083
RAX: 0000000000000001 RBX: ffff800000000001 RCX: 0000000000000000
RDX: 000000000000002f RSI: 0000000000000000 RDI: 0000000000000000
RBP: ffff8881b590e400 R08: ffff8881b590e400 R09: 0000000000000003
R10: ffffe8ffffc05440 R11: 0000000000000000 R12: ffff8881b590e5d8
R13: 0000000000000010 R14: ffff8881b590e420 R15: ffffe8ffffc05400
FS:  0000000000000000(0000) GS:ffff8881b9a00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9bcc5c61f8 CR3: 00000001b6a24000 CR4: 00000000000006f0
Kernel panic - not syncing: Fatal exception in interrupt
Kernel Offset: disabled
---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---

The host seems to be running fine, the KVM guest crash is reproducible.
Both host and guest are running Debian unstable with a locally built
kernel; the host runs 5.1.1, the guest 5.0.13. The crash also happens
when the host is running 5.1.0; going back to 5.0.13 with the host
allows the guest to finish bootup and run fine.

Please note that my kernel 5.1.1 image is not fully broken in KVM, I
have update my APU machine which runs firewall and other infrastructure
services and the guests run fine there.

The machine in question is an older box with an AMD Phenom(tm) II X6
1090T Processor. I guess that the issue is related to the Phenom CPU.

Any idea short of bisecting?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Linux in KVM guest segfaults when hosts runs Linux 5.1
  2019-05-12 11:53 Linux in KVM guest segfaults when hosts runs Linux 5.1 Marc Haber
@ 2019-05-13 14:07 ` Sean Christopherson
  2019-05-13 14:10 ` Radim Krčmář
  1 sibling, 0 replies; 5+ messages in thread
From: Sean Christopherson @ 2019-05-13 14:07 UTC (permalink / raw)
  To: Marc Haber; +Cc: kvm, linux-kernel

On Sun, May 12, 2019 at 01:53:02PM +0200, Marc Haber wrote:
> Hi,
> 
> since updating my home desktop machine to kernel 5.1.1, KVM guests
> started on that machine segfault after booting:
> general protection fault: 0000 [#1] PREEMPT SMP NOPTI
> CPU: 0 PID: 13 Comm: kworker/0:1 Not tainted 5.0.13-zgsrv20080 #5.0.13.20190505.0
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014
> Workqueue: events once_deferred
> RIP: 0010:native_read_pmc+0x2/0x10
> Code: e2 20 89 3e 48 09 d0 c3 89 f9 89 f0 0f 30 c3 66 0f 1f 84 00 00 00 00 00 89 f0 89 f9 0f 30 31 c0 c3 0f 1f 80 00 00 00 00 89 f9 <0f> 33 48 c1 e2 20 48 09 d0 c3 0f 1f 40 00 0f 20 c0 c3 66 66 2e 0f
> RSP: 0018:ffff8881b9a03e50 EFLAGS: 00010083
> RAX: 0000000000000001 RBX: ffff800000000001 RCX: 0000000000000000
> RDX: 000000000000002f RSI: 0000000000000000 RDI: 0000000000000000
> RBP: ffff8881b590e400 R08: ffff8881b590e400 R09: 0000000000000003
> R10: ffffe8ffffc05440 R11: 0000000000000000 R12: ffff8881b590e5d8
> R13: 0000000000000010 R14: ffff8881b590e420 R15: ffffe8ffffc05400
> FS:  0000000000000000(0000) GS:ffff8881b9a00000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00007f9bcc5c61f8 CR3: 00000001b6a24000 CR4: 00000000000006f0
> Call Trace:
>  <IRQ>
>  x86_perf_event_update+0x3b/0x80
>  x86_pmu_stop+0x84/0xa0
>  x86_pmu_del+0x52/0x160
>  event_sched_out.isra.59+0x95/0x190
>  group_sched_out.part.61+0x51/0xc0
>  ctx_sched_out+0xf2/0x220
>  ctx_resched+0xb8/0xc0
>  __perf_install_in_context+0x175/0x1f0
>  remote_function+0x3e/0x50
>  flush_smp_call_function_queue+0x30/0xe0
>  smp_call_function_interrupt+0x2f/0x40
>  call_function_single_interrupt+0xf/0x20
>  </IRQ>

...

> The host seems to be running fine, the KVM guest crash is reproducible.
> Both host and guest are running Debian unstable with a locally built
> kernel; the host runs 5.1.1, the guest 5.0.13. The crash also happens
> when the host is running 5.1.0; going back to 5.0.13 with the host
> allows the guest to finish bootup and run fine.
> 
> Please note that my kernel 5.1.1 image is not fully broken in KVM, I
> have update my APU machine which runs firewall and other infrastructure
> services and the guests run fine there.
> 
> The machine in question is an older box with an AMD Phenom(tm) II X6
> 1090T Processor. I guess that the issue is related to the Phenom CPU.
> 
> Any idea short of bisecting?

It's a regression introduced by commit 672ff6cff80c ("KVM: x86: Raise
#GP when guest vCPU do not support PMU").  A fix has been submitted.

https://lkml.kernel.org/r/20190508170248.15271-1-bp@alien8.de

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Linux in KVM guest segfaults when hosts runs Linux 5.1
  2019-05-12 11:53 Linux in KVM guest segfaults when hosts runs Linux 5.1 Marc Haber
  2019-05-13 14:07 ` Sean Christopherson
@ 2019-05-13 14:10 ` Radim Krčmář
  2019-05-14  6:51   ` Marc Haber
  1 sibling, 1 reply; 5+ messages in thread
From: Radim Krčmář @ 2019-05-13 14:10 UTC (permalink / raw)
  To: Marc Haber; +Cc: kvm, linux-kernel

2019-05-12 13:53+0200, Marc Haber:
> since updating my home desktop machine to kernel 5.1.1, KVM guests
> started on that machine segfault after booting:
[...]
> Any idea short of bisecting?

It has also been spotted by Borislav and the fix [1] should land in the
next kernel update, thanks for the report.

---
1: https://patchwork.kernel.org/patch/10936271/

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Linux in KVM guest segfaults when hosts runs Linux 5.1
  2019-05-13 14:10 ` Radim Krčmář
@ 2019-05-14  6:51   ` Marc Haber
  2019-05-27  8:44     ` Marc Haber
  0 siblings, 1 reply; 5+ messages in thread
From: Marc Haber @ 2019-05-14  6:51 UTC (permalink / raw)
  To: Radim Krčmář; +Cc: kvm, linux-kernel

On Mon, May 13, 2019 at 04:10:35PM +0200, Radim Krčmář wrote:
> 2019-05-12 13:53+0200, Marc Haber:
> > since updating my home desktop machine to kernel 5.1.1, KVM guests
> > started on that machine segfault after booting:
> [...]
> > Any idea short of bisecting?
> 
> It has also been spotted by Borislav and the fix [1] should land in the
> next kernel update, thanks for the report.
> 1: https://patchwork.kernel.org/patch/10936271/

I can confirm that this patch fixes the segfaults for me.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Linux in KVM guest segfaults when hosts runs Linux 5.1
  2019-05-14  6:51   ` Marc Haber
@ 2019-05-27  8:44     ` Marc Haber
  0 siblings, 0 replies; 5+ messages in thread
From: Marc Haber @ 2019-05-27  8:44 UTC (permalink / raw)
  To: Radim Krčmář; +Cc: kvm, linux-kernel

On Tue, May 14, 2019 at 08:51:28AM +0200, Marc Haber wrote:
> On Mon, May 13, 2019 at 04:10:35PM +0200, Radim Krčmář wrote:
> > 2019-05-12 13:53+0200, Marc Haber:
> > > since updating my home desktop machine to kernel 5.1.1, KVM guests
> > > started on that machine segfault after booting:
> > [...]
> > > Any idea short of bisecting?
> > 
> > It has also been spotted by Borislav and the fix [1] should land in the
> > next kernel update, thanks for the report.
> > 1: https://patchwork.kernel.org/patch/10936271/
> 
> I can confirm that this patch fixes the segfaults for me.

And it is not yet in Linux 5.1.5.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2019-05-27  8:44 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-05-12 11:53 Linux in KVM guest segfaults when hosts runs Linux 5.1 Marc Haber
2019-05-13 14:07 ` Sean Christopherson
2019-05-13 14:10 ` Radim Krčmář
2019-05-14  6:51   ` Marc Haber
2019-05-27  8:44     ` Marc Haber

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).