LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: jackm <jackm@dev.mellanox.co.il>
To: Leon Romanovsky <leon@kernel.org>
Cc: Hans Westgaard Ry <hans.westgaard.ry@oracle.com>,
	Doug Ledford <dledford@redhat.com>,
	Jason Gunthorpe <jgg@ziepe.ca>,
	Hakon Bugge <haakon.bugge@oracle.com>,
	Daniel Jurgens <danielj@mellanox.com>,
	Parav Pandit <parav@mellanox.com>,
	Pravin Shedge <pravin.shedge4linux@gmail.com>,
	linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH] IB/mad: Use ID allocator routines to allocate agent number
Date: Wed, 30 May 2018 11:18:56 +0300	[thread overview]
Message-ID: <20180530111856.0000430f@dev.mellanox.co.il> (raw)
In-Reply-To: <20180529095445.GG3697@mtr-leonro.mtl.com>

On Tue, 29 May 2018 12:54:45 +0300
Leon Romanovsky <leon@kernel.org> wrote:

> On Tue, May 29, 2018 at 11:54:59AM +0300, Leon Romanovsky wrote:
> > On Tue, May 29, 2018 at 09:38:08AM +0200, Hans Westgaard Ry wrote:  
> > > The agent TID is a 64 bit value split in two dwords.  The least
> > > significant dword is the TID running counter. The most significant
> > > dword is the agent number. In the CX-3 shared port model, the mlx4
> > > driver uses the most significant byte of the agent number to
> > > store the slave number, making agent numbers greater and equal to
> > > 2^24 (3 bytes) unusable.  The current codebase uses a variable
> > > which is incremented atomically for each new agent number giving
> > > too large agent numbers over time.  The IDA set of functions are
> > > used instead of the simple counter approach. This allows re-use
> > > of agent numbers. A sysctl variable is also introduced, to
> > > control the max agent number.  
> >
> > Why don't you simply limit this number per-driver? By default, any
> > variable is allowed and mlx4_ib will set something else.

It is much simpler to do things here -- the current allocation scheme
does have a wrap-around problem, so there is a good reason for using a
global allocator residing in the ib core.

> >
> > What is the advantage of having sysctl?  
> 
> Anyway, it doesn't pass smoke test.

REASON:  The start argument in ida_simple_get should be 1, not 0:
+       ib_mad_client_id = ida_simple_get(&ib_mad_client_ids,
+                                         1,
+                                         ib_mad_sysctl_client_id_max,
+                                         GFP_KERNEL);


The agent ID 0 is interpreted as -no agent-, is used by snoop agents,
and is never allocated.

The first allocated agent id is 1:
static atomic_t ib_mad_client_id_min = ATOMIC_INIT(0);
...
mad_agent_priv->agent.hi_tid =atomic_inc_return(&ib_mad_client_id);

If this is fixed, we do not get the panic.

However, please note Jason's feedback and my suggestion -- this
ida-based interface needs to avoid immediate re-use of freed ids.

 -Jack
> 
> [  126.428407] RPC: Unregistered rdma transport module.
> [  126.428513] RPC: Unregistered rdma backchannel transport module.
> [  194.664081] IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready
> [  209.068702] BUG: unable to handle kernel NULL pointer dereference
> at 0000000000000070 [  209.068858] PGD 80000000341cf067 P4D
> 80000000341cf067 PUD 34188067 PMD 0 [  209.068941] Oops: 0002 [#1]
> SMP PTI [  209.069006] Modules linked in: netconsole nfsv3 nfs fscache
> mlx4_ib(-) mlx4_en mlx4_core devlink ib_ipoib rdma_ucm ib_ucm
> ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_core dm_mirror
> dm_region_hash dm_log dm_mod nfsd pcspkr i2c_piix4 auth_rpcgss
> nfs_acl lockd grace sunrpc ip_tables ata_generic cirrus
> drm_kms_helper pata_acpi syscopyarea sysfillrect sysimgblt
> fb_sys_fops ttm drm e1000 virtio_console i2c_core serio_raw ata_piix
> floppy [last unloaded: mlxfw] [  209.069312] CPU: 4 PID: 11048 Comm:
> modprobe Not tainted
> 4.17.0-rc7-2018-05-29_11-04-56_Hans_Westgaard_Ry__hans_westga #1
> [  209.069413] Hardware name: Red Hat KVM, BIOS Bochs 01/01/2011
> [  209.069486] RIP: 0010:_raw_spin_lock_irqsave+0x1e/0x40
> [  209.069536] RSP: 0018:ffffc90000b4fd70 EFLAGS: 00010046
> [  209.069591] RAX: 0000000000000000 RBX: 0000000000000246 RCX:
> ffffea0004d7ed00 [  209.069653] RDX: 0000000000000001 RSI:
> 0000000000000000 RDI: 0000000000000070 [  209.069717] RBP:
> 0000000000000000 R08: ffff88013446fc00 R09: 000000018010000f
> [  209.069778] R10: 0000000000000001 R11: ffff88013446fc00 R12:
> 0000000000000070 [  209.069849] R13: 0000000000000202 R14:
> 0000000000000000 R15: 0000000000000000 [  209.069915] FS:
> 00007fc34caf7740(0000) GS:ffff88013fd00000(0000)
> knlGS:0000000000000000 [  209.069987] CS:  0010 DS: 0000 ES: 0000
> CR0: 0000000080050033 [  209.070043] CR2: 0000000000000070 CR3:
> 000000008853e000 CR4: 00000000000006e0 [  209.070128] Call Trace:
> [  209.070189]  ib_unregister_mad_agent+0x2d/0x540 [ib_core]
> [  209.070260]  ? __slab_free+0x9a/0x2d0 [  209.070332]
> ib_agent_port_close+0xad/0xf0 [ib_core] [  209.070396]
> ib_mad_remove_device+0x59/0xb0 [ib_core] [  209.070466]
> ib_unregister_device+0xd4/0x180 [ib_core] [  209.070537]
> mlx4_ib_remove+0x67/0x1f0 [mlx4_ib] [  209.070594]
> mlx4_remove_device+0x93/0xa0 [mlx4_core] [  209.070648]
> mlx4_unregister_interface+0x37/0x90 [mlx4_core] [  209.070705]
> mlx4_ib_cleanup+0xc/0x4db [mlx4_ib] [  209.072113]
> __x64_sys_delete_module+0x15b/0x260 [  209.073567]  ?
> exit_to_usermode_loop+0x7f/0x95 [  209.074945]
> do_syscall_64+0x48/0x100 [  209.076448]
> entry_SYSCALL_64_after_hwframe+0x44/0xa9 [  209.077799] RIP:
> 0033:0x7fc34bfe36b7 [  209.079122] RSP: 002b:00007ffc8ffa98b8 EFLAGS:
> 00000206 ORIG_RAX: 00000000000000b0 [  209.080500] RAX:
> ffffffffffffffda RBX: 00000000013455c0 RCX: 00007fc34bfe36b7
> [  209.081875] RDX: 0000000000000000 RSI: 0000000000000800 RDI:
> 0000000001345628 [  209.083265] RBP: 0000000000000000 R08:
> 00007fc34c2a8060 R09: 00007fc34c053a40 [  209.084655] R10:
> 00007ffc8ffa9640 R11: 0000000000000206 R12: 0000000000000000
> [  209.086028] R13: 0000000000000001 R14: 0000000001345628 R15:
> 0000000000000000 [  209.087416] Code: 66 66 66 66 2e 0f 1f 84 00 00
> 00 00 00 0f 1f 44 00 00 53 9c 58 0f 1f 44 00 00 48 89 c3 fa 66 0f 1f
> 44 00 00 31 c0 ba 01 00 00 00 <f0> 0f b1 17 85 c0 75 05 48 89 d8 5b
> c3 89 c6 e8 1e c9 81 ff eb [  209.090262] RIP:
> _raw_spin_lock_irqsave+0x1e/0x40 RSP: ffffc90000b4fd70 [  209.091720]
> CR2: 0000000000000070 [  209.093137] ---[ end trace 7b8a6faa27868861
> ]--- [  209.094546] Kernel panic - not syncing: Fatal exception
> [  209.096910] Kernel Offset: disabled [  209.098291] ---[ end Kernel
> panic - not syncing: Fatal exception ]---
> 
> Thanks
> 
> >
> > Thanks  
> 
> 

  reply	other threads:[~2018-05-30  8:19 UTC|newest]

Thread overview: 24+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-05-29  7:38 Hans Westgaard Ry
2018-05-29  8:54 ` Leon Romanovsky
2018-05-29  9:54   ` Leon Romanovsky
2018-05-30  8:18     ` jackm [this message]
2018-05-29 15:49 ` Jason Gunthorpe
2018-05-29 16:16   ` Håkon Bugge
2018-05-29 16:40     ` Jason Gunthorpe
2018-05-30  7:32       ` Håkon Bugge
2018-05-30 15:15         ` Jason Gunthorpe
2018-05-30  8:02       ` jackm
2018-05-30 12:22         ` Hans Westgaard Ry
2018-05-30 15:10           ` Jason Gunthorpe
2018-05-30 20:07             ` Håkon Bugge
2018-05-30 22:09               ` Jason Gunthorpe
2018-05-31 19:54                 ` Håkon Bugge
2018-06-07 10:52 ` [PATCH v2 0/2] IB:mad " Hans Westgaard Ry
2018-06-07 10:52   ` [PATCH v2 1/2] idr: Add ida_simple_get_cyclic Hans Westgaard Ry
2018-06-07 10:52   ` [PATCH v2 2/2] IB/mad: Use ID allocator routines to allocate agent number Hans Westgaard Ry
2018-06-07 11:14 ` [PATCH v3 0/2] IB:mad " Hans Westgaard Ry
2018-06-07 11:14   ` [PATCH v3 1/2] idr: Add ida_simple_get_cyclic Hans Westgaard Ry
2018-06-07 18:50     ` Jason Gunthorpe
2018-06-07 11:14   ` [PATCH v3 2/2] IB/mad: Use ID allocator routines to allocate agent number Hans Westgaard Ry
2018-06-07 15:37     ` Matthew Wilcox
2018-06-07 17:59       ` Håkon Bugge

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20180530111856.0000430f@dev.mellanox.co.il \
    --to=jackm@dev.mellanox.co.il \
    --cc=danielj@mellanox.com \
    --cc=dledford@redhat.com \
    --cc=haakon.bugge@oracle.com \
    --cc=hans.westgaard.ry@oracle.com \
    --cc=jgg@ziepe.ca \
    --cc=leon@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=parav@mellanox.com \
    --cc=pravin.shedge4linux@gmail.com \
    --subject='Re: [PATCH] IB/mad: Use ID allocator routines to allocate agent number' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).