LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
* [PATCH] drivers: net: xgene: fix: Out of order descriptor bytes read
@ 2015-01-22 20:03 Iyappan Subramanian
  2015-01-22 22:50 ` Eric Dumazet
  0 siblings, 1 reply; 6+ messages in thread
From: Iyappan Subramanian @ 2015-01-22 20:03 UTC (permalink / raw)
  To: davem, netdev
  Cc: linux-kernel, linux-arm-kernel, mlangsdo, patches,
	Iyappan Subramanian, Keyur Chudgar

This patch fixes the following kernel crash,

	WARNING: CPU: 2 PID: 0 at net/ipv4/tcp_input.c:3079 tcp_clean_rtx_queue+0x658/0x80c()
	Call trace:
	[<fffffe0000096b7c>] dump_backtrace+0x0/0x184
	[<fffffe0000096d10>] show_stack+0x10/0x1c
	[<fffffe0000685ea0>] dump_stack+0x74/0x98
	[<fffffe00000b44e0>] warn_slowpath_common+0x88/0xb0
	[<fffffe00000b461c>] warn_slowpath_null+0x14/0x20
	[<fffffe00005b5c1c>] tcp_clean_rtx_queue+0x654/0x80c
	[<fffffe00005b6228>] tcp_ack+0x454/0x688
	[<fffffe00005b6ca8>] tcp_rcv_established+0x4a4/0x62c
	[<fffffe00005bf4b4>] tcp_v4_do_rcv+0x16c/0x350
	[<fffffe00005c225c>] tcp_v4_rcv+0x8e8/0x904
	[<fffffe000059d470>] ip_local_deliver_finish+0x100/0x26c
	[<fffffe000059dad8>] ip_local_deliver+0xac/0xc4
	[<fffffe000059d6c4>] ip_rcv_finish+0xe8/0x328
	[<fffffe000059dd3c>] ip_rcv+0x24c/0x38c
	[<fffffe0000563950>] __netif_receive_skb_core+0x29c/0x7c8
	[<fffffe0000563ea4>] __netif_receive_skb+0x28/0x7c
	[<fffffe0000563f54>] netif_receive_skb_internal+0x5c/0xe0
	[<fffffe0000564810>] napi_gro_receive+0xb4/0x110
	[<fffffe0000482a2c>] xgene_enet_process_ring+0x144/0x338
	[<fffffe0000482d18>] xgene_enet_napi+0x1c/0x50
	[<fffffe0000565454>] net_rx_action+0x154/0x228
	[<fffffe00000b804c>] __do_softirq+0x110/0x28c
	[<fffffe00000b8424>] irq_exit+0x8c/0xc0
	[<fffffe0000093898>] handle_IRQ+0x44/0xa8
	[<fffffe000009032c>] gic_handle_irq+0x38/0x7c
	[...]

Software writes poison data into the descriptor bytes[15:8] and upon
receiving the interrupt, if those bytes are overwritten by the hardware with
the valid data, software also reads bytes[7:0] and executes receive/tx
completion logic.

If the CPU executes the above two reads in out of order fashion, then the
bytes[7:0] will have older data and causing the kernel panic.  We have to
force the order of the reads and thus this patch introduces read memory
barrier between these reads.

Signed-off-by: Iyappan Subramanian <isubramanian@apm.com>
Signed-off-by: Keyur Chudgar <kchudgar@apm.com>
Tested-by: Mark Langsdorf <mlangsdo@redhat.com>
---
 drivers/net/ethernet/apm/xgene/xgene_enet_main.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/net/ethernet/apm/xgene/xgene_enet_main.c b/drivers/net/ethernet/apm/xgene/xgene_enet_main.c
index 83a5028..3622cdb 100644
--- a/drivers/net/ethernet/apm/xgene/xgene_enet_main.c
+++ b/drivers/net/ethernet/apm/xgene/xgene_enet_main.c
@@ -369,6 +369,8 @@ static int xgene_enet_process_ring(struct xgene_enet_desc_ring *ring,
 		if (unlikely(xgene_enet_is_desc_slot_empty(raw_desc)))
 			break;
 
+		/* read fpqnum field after dataaddr field */
+		smp_rmb();
 		if (is_rx_desc(raw_desc))
 			ret = xgene_enet_rx_frame(ring, raw_desc);
 		else
-- 
1.9.1


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH] drivers: net: xgene: fix: Out of order descriptor bytes read
  2015-01-22 20:03 [PATCH] drivers: net: xgene: fix: Out of order descriptor bytes read Iyappan Subramanian
@ 2015-01-22 22:50 ` Eric Dumazet
  2015-01-26 21:12   ` Iyappan Subramanian
  0 siblings, 1 reply; 6+ messages in thread
From: Eric Dumazet @ 2015-01-22 22:50 UTC (permalink / raw)
  To: Iyappan Subramanian
  Cc: davem, netdev, linux-kernel, linux-arm-kernel, mlangsdo, patches,
	Keyur Chudgar

On Thu, 2015-01-22 at 12:03 -0800, Iyappan Subramanian wrote:
> This patch fixes the following kernel crash,
> 
> 	WARNING: CPU: 2 PID: 0 at net/ipv4/tcp_input.c:3079 tcp_clean_rtx_queue+0x658/0x80c()
> 	Call trace:

> 
> Software writes poison data into the descriptor bytes[15:8] and upon
> receiving the interrupt, if those bytes are overwritten by the hardware with
> the valid data, software also reads bytes[7:0] and executes receive/tx
> completion logic.
> 
> If the CPU executes the above two reads in out of order fashion, then the
> bytes[7:0] will have older data and causing the kernel panic.  We have to
> force the order of the reads and thus this patch introduces read memory
> barrier between these reads.
> 
> Signed-off-by: Iyappan Subramanian <isubramanian@apm.com>
> Signed-off-by: Keyur Chudgar <kchudgar@apm.com>
> Tested-by: Mark Langsdorf <mlangsdo@redhat.com>
> ---
>  drivers/net/ethernet/apm/xgene/xgene_enet_main.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/drivers/net/ethernet/apm/xgene/xgene_enet_main.c b/drivers/net/ethernet/apm/xgene/xgene_enet_main.c
> index 83a5028..3622cdb 100644
> --- a/drivers/net/ethernet/apm/xgene/xgene_enet_main.c
> +++ b/drivers/net/ethernet/apm/xgene/xgene_enet_main.c
> @@ -369,6 +369,8 @@ static int xgene_enet_process_ring(struct xgene_enet_desc_ring *ring,
>  		if (unlikely(xgene_enet_is_desc_slot_empty(raw_desc)))
>  			break;
>  
> +		/* read fpqnum field after dataaddr field */
> +		smp_rmb();
>  		if (is_rx_desc(raw_desc))
>  			ret = xgene_enet_rx_frame(ring, raw_desc);
>  		else

Reading your changelog, it looks like you need a plain rmb() here.




^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH] drivers: net: xgene: fix: Out of order descriptor bytes read
  2015-01-22 22:50 ` Eric Dumazet
@ 2015-01-26 21:12   ` Iyappan Subramanian
  2015-01-26 21:27     ` Eric Dumazet
  2015-01-26 22:34     ` David Miller
  0 siblings, 2 replies; 6+ messages in thread
From: Iyappan Subramanian @ 2015-01-26 21:12 UTC (permalink / raw)
  To: Eric Dumazet
  Cc: David Miller, netdev, linux-kernel, linux-arm-kernel, mlangsdo,
	patches, Keyur Chudgar

On Thu, Jan 22, 2015 at 2:50 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> On Thu, 2015-01-22 at 12:03 -0800, Iyappan Subramanian wrote:
>> This patch fixes the following kernel crash,
>>
>>       WARNING: CPU: 2 PID: 0 at net/ipv4/tcp_input.c:3079 tcp_clean_rtx_queue+0x658/0x80c()
>>       Call trace:
>
>>
>> Software writes poison data into the descriptor bytes[15:8] and upon
>> receiving the interrupt, if those bytes are overwritten by the hardware with
>> the valid data, software also reads bytes[7:0] and executes receive/tx
>> completion logic.
>>
>> If the CPU executes the above two reads in out of order fashion, then the
>> bytes[7:0] will have older data and causing the kernel panic.  We have to
>> force the order of the reads and thus this patch introduces read memory
>> barrier between these reads.
>>
>> Signed-off-by: Iyappan Subramanian <isubramanian@apm.com>
>> Signed-off-by: Keyur Chudgar <kchudgar@apm.com>
>> Tested-by: Mark Langsdorf <mlangsdo@redhat.com>
>> ---
>>  drivers/net/ethernet/apm/xgene/xgene_enet_main.c | 2 ++
>>  1 file changed, 2 insertions(+)
>>
>> diff --git a/drivers/net/ethernet/apm/xgene/xgene_enet_main.c b/drivers/net/ethernet/apm/xgene/xgene_enet_main.c
>> index 83a5028..3622cdb 100644
>> --- a/drivers/net/ethernet/apm/xgene/xgene_enet_main.c
>> +++ b/drivers/net/ethernet/apm/xgene/xgene_enet_main.c
>> @@ -369,6 +369,8 @@ static int xgene_enet_process_ring(struct xgene_enet_desc_ring *ring,
>>               if (unlikely(xgene_enet_is_desc_slot_empty(raw_desc)))
>>                       break;
>>
>> +             /* read fpqnum field after dataaddr field */
>> +             smp_rmb();
>>               if (is_rx_desc(raw_desc))
>>                       ret = xgene_enet_rx_frame(ring, raw_desc);
>>               else
>
> Reading your changelog, it looks like you need a plain rmb() here.

rmb() translates into dsb, which in arm64 serializes everything
including instructions and thus expensive compared to dmb.

Do you see any issue with smp_rmb() (which translates into dmb) ?

>
>
>

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH] drivers: net: xgene: fix: Out of order descriptor bytes read
  2015-01-26 21:12   ` Iyappan Subramanian
@ 2015-01-26 21:27     ` Eric Dumazet
  2015-01-26 21:32       ` Eric Dumazet
  2015-01-26 22:34     ` David Miller
  1 sibling, 1 reply; 6+ messages in thread
From: Eric Dumazet @ 2015-01-26 21:27 UTC (permalink / raw)
  To: Iyappan Subramanian
  Cc: David Miller, netdev, linux-kernel, linux-arm-kernel, mlangsdo,
	patches, Keyur Chudgar

On Mon, 2015-01-26 at 13:12 -0800, Iyappan Subramanian wrote:
> On Thu, Jan 22, 2015 at 2:50 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> > On Thu, 2015-01-22 at 12:03 -0800, Iyappan Subramanian wrote:
> >> This patch fixes the following kernel crash,
> >>
> >>       WARNING: CPU: 2 PID: 0 at net/ipv4/tcp_input.c:3079 tcp_clean_rtx_queue+0x658/0x80c()
> >>       Call trace:
> >
> >>
> >> Software writes poison data into the descriptor bytes[15:8] and upon
> >> receiving the interrupt, if those bytes are overwritten by the hardware with
> >> the valid data, software also reads bytes[7:0] and executes receive/tx
> >> completion logic.
> >>
> >> If the CPU executes the above two reads in out of order fashion, then the
> >> bytes[7:0] will have older data and causing the kernel panic.  We have to
> >> force the order of the reads and thus this patch introduces read memory
> >> barrier between these reads.
> >>
> >> Signed-off-by: Iyappan Subramanian <isubramanian@apm.com>
> >> Signed-off-by: Keyur Chudgar <kchudgar@apm.com>
> >> Tested-by: Mark Langsdorf <mlangsdo@redhat.com>
> >> ---
> >>  drivers/net/ethernet/apm/xgene/xgene_enet_main.c | 2 ++
> >>  1 file changed, 2 insertions(+)
> >>
> >> diff --git a/drivers/net/ethernet/apm/xgene/xgene_enet_main.c b/drivers/net/ethernet/apm/xgene/xgene_enet_main.c
> >> index 83a5028..3622cdb 100644
> >> --- a/drivers/net/ethernet/apm/xgene/xgene_enet_main.c
> >> +++ b/drivers/net/ethernet/apm/xgene/xgene_enet_main.c
> >> @@ -369,6 +369,8 @@ static int xgene_enet_process_ring(struct xgene_enet_desc_ring *ring,
> >>               if (unlikely(xgene_enet_is_desc_slot_empty(raw_desc)))
> >>                       break;
> >>
> >> +             /* read fpqnum field after dataaddr field */
> >> +             smp_rmb();
> >>               if (is_rx_desc(raw_desc))
> >>                       ret = xgene_enet_rx_frame(ring, raw_desc);
> >>               else
> >
> > Reading your changelog, it looks like you need a plain rmb() here.
> 
> rmb() translates into dsb, which in arm64 serializes everything
> including instructions and thus expensive compared to dmb.
> 
> Do you see any issue with smp_rmb() (which translates into dmb) ?

What happens if you compile a kernel with CONFIG_SMP=n ?


Most drivers in drivers/net use rmb() in this case, not smp_rmb() or
barrier()




^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH] drivers: net: xgene: fix: Out of order descriptor bytes read
  2015-01-26 21:27     ` Eric Dumazet
@ 2015-01-26 21:32       ` Eric Dumazet
  0 siblings, 0 replies; 6+ messages in thread
From: Eric Dumazet @ 2015-01-26 21:32 UTC (permalink / raw)
  To: Iyappan Subramanian
  Cc: David Miller, netdev, linux-kernel, linux-arm-kernel, mlangsdo,
	patches, Keyur Chudgar

On Mon, 2015-01-26 at 13:27 -0800, Eric Dumazet wrote:

> What happens if you compile a kernel with CONFIG_SMP=n ?
> 
> 
> Most drivers in drivers/net use rmb() in this case, not smp_rmb() or
> barrier()

Note that dma_rmb() was recently added as well.




^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH] drivers: net: xgene: fix: Out of order descriptor bytes read
  2015-01-26 21:12   ` Iyappan Subramanian
  2015-01-26 21:27     ` Eric Dumazet
@ 2015-01-26 22:34     ` David Miller
  1 sibling, 0 replies; 6+ messages in thread
From: David Miller @ 2015-01-26 22:34 UTC (permalink / raw)
  To: isubramanian
  Cc: eric.dumazet, netdev, linux-kernel, linux-arm-kernel, mlangsdo,
	patches, kchudgar

From: Iyappan Subramanian <isubramanian@apm.com>
Date: Mon, 26 Jan 2015 13:12:23 -0800

>>> @@ -369,6 +369,8 @@ static int xgene_enet_process_ring(struct xgene_enet_desc_ring *ring,
>>>               if (unlikely(xgene_enet_is_desc_slot_empty(raw_desc)))
>>>                       break;
>>>
>>> +             /* read fpqnum field after dataaddr field */
>>> +             smp_rmb();
>>>               if (is_rx_desc(raw_desc))
>>>                       ret = xgene_enet_rx_frame(ring, raw_desc);
>>>               else
>>
>> Reading your changelog, it looks like you need a plain rmb() here.
> 
> rmb() translates into dsb, which in arm64 serializes everything
> including instructions and thus expensive compared to dmb.
> 
> Do you see any issue with smp_rmb() (which translates into dmb) ?

smp_rmb() is not appropriate.  You're not serializing accesses between
two cpus, you're serializing the cpu with the device.

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2015-01-26 22:34 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-01-22 20:03 [PATCH] drivers: net: xgene: fix: Out of order descriptor bytes read Iyappan Subramanian
2015-01-22 22:50 ` Eric Dumazet
2015-01-26 21:12   ` Iyappan Subramanian
2015-01-26 21:27     ` Eric Dumazet
2015-01-26 21:32       ` Eric Dumazet
2015-01-26 22:34     ` David Miller

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).