2.6.24 says "serial8250: too much work for irq4" a lot.

2.6.24 says "serial8250: too much work for irq4" a lot.

[Rob Landley]

When running a 2.6.24 kernel built for x86-64 under qemu via serial console, 
doing CPU-intensive things that also produce a lot of output (such as 
compiling software) tends to produce the error message in the title.

Anybody have a clue why?  It doesn't seem to cause an actual problem, but it's 
kind of annoying.

(If it's a qemu issue, I can go bother them.  It's possible that qemu isn't 
delivering interrupts as often as it expects, since that's limited by the 
granularity of the host timer; I know the clock in qemu can run a bit slow 
because it only gets clock interrupts when the host system isn't too busy to 
schedule the emulator.  But this doesn't usually cause a problem.  I _think_ 
the message is just a "this should never happen" type warning, which is 
happening to me.  But I break stuff. :)

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: 2.6.24 says "serial8250: too much work for irq4" a lot.

[H. Peter Anvin]

Rob Landley wrote:
> When running a 2.6.24 kernel built for x86-64 under qemu via serial console, 
> doing CPU-intensive things that also produce a lot of output (such as 
> compiling software) tends to produce the error message in the title.
> 
> Anybody have a clue why?  It doesn't seem to cause an actual problem, but it's 
> kind of annoying.
> 
> (If it's a qemu issue, I can go bother them.  It's possible that qemu isn't 
> delivering interrupts as often as it expects, since that's limited by the 
> granularity of the host timer; I know the clock in qemu can run a bit slow 
> because it only gets clock interrupts when the host system isn't too busy to 
> schedule the emulator.  But this doesn't usually cause a problem.  I _think_ 
> the message is just a "this should never happen" type warning, which is 
> happening to me.  But I break stuff. :)

This is because Qemu spews data to the serial port without any rate 
limiting; this causes the in-kernel serial port driver to think the port 
is stuck.  The serial port emulation needs to make it possible to drain 
the virtual FIFO every now and then, as opposed to filling it again 
immediately.

	-hpa

Re: 2.6.24 says "serial8250: too much work for irq4" a lot.

[Ingo Molnar]

* H. Peter Anvin <hpa@zytor.com> wrote:

>> (If it's a qemu issue, I can go bother them.  It's possible that qemu 
>> isn't delivering interrupts as often as it expects, since that's 
>> limited by the granularity of the host timer; I know the clock in 
>> qemu can run a bit slow because it only gets clock interrupts when 
>> the host system isn't too busy to schedule the emulator.  But this 
>> doesn't usually cause a problem.  I _think_ the message is just a 
>> "this should never happen" type warning, which is happening to me.  
>> But I break stuff. :)
>
> This is because Qemu spews data to the serial port without any rate 
> limiting; this causes the in-kernel serial port driver to think the 
> port is stuck.  The serial port emulation needs to make it possible to 
> drain the virtual FIFO every now and then, as opposed to filling it 
> again immediately.

actually, the way i solved it for qemu+KVM+paravirt was to just turn off 
this rather silly check in the serial driver if inside a paravirt guest. 
When we are emulated then the serial 'hardware' is totally reliable and 
we should just trust it. That way i never dropped a single bit of kernel 
log output again.

	Ingo

Re: 2.6.24 says "serial8250: too much work for irq4" a lot.

[H. Peter Anvin]

Ingo Molnar wrote:
> * H. Peter Anvin <hpa@zytor.com> wrote:
> 
>>> (If it's a qemu issue, I can go bother them.  It's possible that qemu 
>>> isn't delivering interrupts as often as it expects, since that's 
>>> limited by the granularity of the host timer; I know the clock in 
>>> qemu can run a bit slow because it only gets clock interrupts when 
>>> the host system isn't too busy to schedule the emulator.  But this 
>>> doesn't usually cause a problem.  I _think_ the message is just a 
>>> "this should never happen" type warning, which is happening to me.  
>>> But I break stuff. :)
>> This is because Qemu spews data to the serial port without any rate 
>> limiting; this causes the in-kernel serial port driver to think the 
>> port is stuck.  The serial port emulation needs to make it possible to 
>> drain the virtual FIFO every now and then, as opposed to filling it 
>> again immediately.
> 
> actually, the way i solved it for qemu+KVM+paravirt was to just turn off 
> this rather silly check in the serial driver if inside a paravirt guest. 
> When we are emulated then the serial 'hardware' is totally reliable and 
> we should just trust it. That way i never dropped a single bit of kernel 
> log output again.
> 

Yes, but keying that on paravirt is silly in the extreme.  After all, 
there is no need for this to be paravirtualized.

	-hpa

Re: 2.6.24 says "serial8250: too much work for irq4" a lot.

[Rob Landley]

On Thursday 07 February 2008 11:37:12 H. Peter Anvin wrote:
> > actually, the way i solved it for qemu+KVM+paravirt was to just turn off
> > this rather silly check in the serial driver if inside a paravirt guest.
> > When we are emulated then the serial 'hardware' is totally reliable and
> > we should just trust it. That way i never dropped a single bit of kernel
> > log output again.
>
> Yes, but keying that on paravirt is silly in the extreme.  After all,
> there is no need for this to be paravirtualized.
>
> 	-hpa

Specifically, qemu isn't paravirtualized, it's fully virtualized.  The same 
kernel can run on real hardware just fine.  (Sort of the point of the 
project...)

I can yank the warning for the kernels I build (or set PASS_LIMIT to 9999999), 
but I'd rather not carry any more patches than I can avoid...

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: 2.6.24 says "serial8250: too much work for irq4" a lot.

[H. Peter Anvin]

Rob Landley wrote:
> 
> Specifically, qemu isn't paravirtualized, it's fully virtualized.  The same 
> kernel can run on real hardware just fine.  (Sort of the point of the 
> project...)
> 
> I can yank the warning for the kernels I build (or set PASS_LIMIT to 9999999), 
> but I'd rather not carry any more patches than I can avoid...
> 

The right thing to do is to add virtual FIFO exhaustion into the Qemu 
device model.  It really isn't a valid emulation of a UART that it has 
an infinite FIFO behind it that can flood the interrupt handler for an 
arbitrary number of fetches.

I was going to give a technical description of how to do it here, but 
then I realized I might as well just write it up as a patch.  Stay tuned.

	-hpa

Re: 2.6.24 says "serial8250: too much work for irq4" a lot.

[H. Peter Anvin]

[-- Attachment #1: Type: text/plain, Size: 649 bytes --]

Rob Landley wrote:
> 
> Specifically, qemu isn't paravirtualized, it's fully virtualized.  The same 
> kernel can run on real hardware just fine.  (Sort of the point of the 
> project...)
> 
> I can yank the warning for the kernels I build (or set PASS_LIMIT to 9999999), 
> but I'd rather not carry any more patches than I can avoid...
> 

Patch attached, completely untested beyond compilation.

In particular:

	- we should probably clear the burst counter when the interrupt
	  line goes from inactive to active.
	- there probably should be a timer which clears the burst
  	  counter.

However, I think I've covered most of the bases...

	-hpa

[-- Attachment #2: diff --]
[-- Type: text/plain, Size: 2376 bytes --]

diff --git a/hw/serial.c b/hw/serial.c
index b1bd0ff..c902792 100644
--- a/hw/serial.c
+++ b/hw/serial.c
@@ -73,6 +73,15 @@
 #define UART_LSR_OE	0x02	/* Overrun error indicator */
 #define UART_LSR_DR	0x01	/* Receiver data ready */
 
+/*
+ * It's common for an IRQ handler to keep reading the RBR until
+ * the LSR indicates that the FIFO is empty, expecting that the
+ * CPU is vastly faster than the serial line.  This can cause
+ * overruns or error indications if the FIFO never empties, so
+ * give the target OS a breather every so often.
+ */
+#define MAX_BURST	512
+
 struct SerialState {
     uint16_t divider;
     uint8_t rbr; /* receive register */
@@ -91,8 +100,14 @@ struct SerialState {
     int last_break_enable;
     target_phys_addr_t base;
     int it_shift;
+    int burst_len;
 };
 
+static void serial_clear_burst(SerialState *s)
+{
+    s->burst_len = 0;
+}
+
 static void serial_update_irq(SerialState *s)
 {
     if ((s->lsr & UART_LSR_DR) && (s->ier & UART_IER_RDI)) {
@@ -114,6 +129,8 @@ static void serial_update_parameters(SerialState *s)
     int speed, parity, data_bits, stop_bits;
     QEMUSerialSetParams ssp;
 
+    serial_clear_burst(s);
+
     if (s->lcr & 0x08) {
         if (s->lcr & 0x10)
             parity = 'E';
@@ -221,9 +238,12 @@ static uint32_t serial_ioport_read(void *opaque, uint32_t addr)
             ret = s->divider & 0xff;
         } else {
             ret = s->rbr;
-            s->lsr &= ~(UART_LSR_DR | UART_LSR_BI);
-            serial_update_irq(s);
-            qemu_chr_accept_input(s->chr);
+	    if (s->burst_len < MAX_BURST) {
+		s->burst_len++;
+		s->lsr &= ~(UART_LSR_DR | UART_LSR_BI);
+		serial_update_irq(s);
+		qemu_chr_accept_input(s->chr);
+	    }
         }
         break;
     case 1:
@@ -235,6 +255,7 @@ static uint32_t serial_ioport_read(void *opaque, uint32_t addr)
         break;
     case 2:
         ret = s->iir;
+	serial_clear_burst(s);
         /* reset THR pending bit */
         if ((ret & 0x7) == UART_IIR_THRI)
             s->thr_ipending = 0;
@@ -248,6 +269,10 @@ static uint32_t serial_ioport_read(void *opaque, uint32_t addr)
         break;
     case 5:
         ret = s->lsr;
+	if (s->burst_len >= MAX_BURST)
+	    ret &= ~(UART_LSR_DR|UART_LSR_BI);
+	if (!(ret & UART_LSR_DR))
+	    serial_clear_burst(s);
         break;
     case 6:
         if (s->mcr & UART_MCR_LOOP) {

Re: 2.6.24 says "serial8250: too much work for irq4" a lot.

[Rob Landley]

On Thursday 07 February 2008 15:19:39 H. Peter Anvin wrote:
> Rob Landley wrote:
> > Specifically, qemu isn't paravirtualized, it's fully virtualized.  The
> > same kernel can run on real hardware just fine.  (Sort of the point of
> > the project...)
> >
> > I can yank the warning for the kernels I build (or set PASS_LIMIT to
> > 9999999), but I'd rather not carry any more patches than I can avoid...
>
> Patch attached, completely untested beyond compilation.
>
> In particular:
>
> 	- we should probably clear the burst counter when the interrupt
> 	  line goes from inactive to active.
> 	- there probably should be a timer which clears the burst
>   	  counter.
>
> However, I think I've covered most of the bases...

Well, qemu still seems to work with the patch applied, and I haven't 
reproduced the error message yet...

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.