Netdev Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Jason Gunthorpe <jgg@nvidia.com>
To: Leon Romanovsky <leon@kernel.org>
Cc: Doug Ledford <dledford@redhat.com>,
	Aharon Landau <aharonl@nvidia.com>,
	Jakub Kicinski <kuba@kernel.org>,
	Jason Wang <jasowang@redhat.com>,
	linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org,
	"Michael S. Tsirkin" <mst@redhat.com>,
	netdev@vger.kernel.org, Saeed Mahameed <saeedm@nvidia.com>,
	Shay Drory <shayd@nvidia.com>,
	virtualization@lists.linux-foundation.org
Subject: Re: [PATCH mlx5-next 3/5] RDMA/mlx5: Change the cache to hold mkeys instead of MRs
Date: Thu, 29 Jul 2021 16:08:16 -0300	[thread overview]
Message-ID: <20210729190816.GA2482631@nvidia.com> (raw)
In-Reply-To: <d14d051add7957f72d93fe8a9aa148d3fad20024.1624362290.git.leonro@nvidia.com>

On Tue, Jun 22, 2021 at 03:08:21PM +0300, Leon Romanovsky wrote:
> From: Aharon Landau <aharonl@nvidia.com>
> 
> Today the cache is an MR-cache, however, all members of MR, except for
> mkey, are not being used in the cache.
> Therefore, changing it to an mkey-cache so that the cache has its own
> memory and holds only the values needed for the cache.

This patch is quite big and seems to be doing a lot more than just
this

Frankly, I'm not sure what it is trying to do

> diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
> index af11a0d8ebc0..ffb6f1d41f3d 100644
> +++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
> @@ -634,6 +634,15 @@ struct mlx5_user_mmap_entry {
>  #define mlx5_update_odp_stats(mr, counter_name, value)		\
>  	atomic64_add(value, &((mr)->odp_stats.counter_name))
>  
> +struct mlx5r_cache_mkey {
> +	u32 key;
> +	struct mlx5_cache_ent *cache_ent;
> +	u32 out[MLX5_ST_SZ_DW(create_mkey_out)];
> +	struct mlx5_async_work cb_work;
> +	/* Cache list element */
> +	struct list_head list;
> +};

This is the point, right? Lift these members out of the mlx5_ib_mr?

But out abd cb_work shouldn't be stored in perpetuity in the cache, it
is only needed short-term as part of the callback for async mkey
creation.

This should also be organized to not have so many alignment holes

Actually the only thing it does is store a u32 attached to each rbtree
so this looks like rather a lot of memory overhead, plus the
kfree/allocs.

I'd probably do this with an xarray on the mlx5_cache_ent
instead. Store the 'tail index' and adding is
'xa_insert(tail_index++)' and removing is 'xa_erase(tail_index--)'

Use xa_mk_value() and I think we have less than 31 bits of mkey,
right?

>  static inline bool is_odp_mr(struct mlx5_ib_mr *mr)
>  {
>  	return IS_ENABLED(CONFIG_INFINIBAND_ON_DEMAND_PAGING) && mr->umem &&
> @@ -763,16 +758,16 @@ struct mlx5_cache_ent {
>  	u8 fill_to_high_water:1;
>  
>  	/*
> -	 * - available_mrs is the length of list head, ie the number of MRs
> +	 * - available_mkeys is the length of list head, ie the number of Mkeys
>  	 *   available for immediate allocation.
> -	 * - total_mrs is available_mrs plus all in use MRs that could be
> +	 * - total_mkeys is available_mkeys plus all in use Mkeys that could be
>  	 *   returned to the cache.
> -	 * - limit is the low water mark for available_mrs, 2* limit is the
> +	 * - limit is the low water mark for available_mkeys, 2* limit is the
>  	 *   upper water mark.
> -	 * - pending is the number of MRs currently being created
> +	 * - pending is the number of Mkeys currently being created
>  	 */
> -	u32 total_mrs;
> -	u32 available_mrs;
> +	u32 total_mkeys;
> +	u32 available_mkeys;
>  	u32 limit;
>  	u32 pending;

Put all the renaming in another patch, maybe as the last patch in the
series and do everything. Much too hard to read when renaming is
muddled with logic changes

Jason

  reply	other threads:[~2021-07-29 19:08 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-06-22 12:08 [PATCH rdma-next 0/5] mlx5 MR cache enhancements Leon Romanovsky
2021-06-22 12:08 ` [PATCH mlx5-next 1/5] RDMA/mlx5: Replace struct mlx5_core_mkey by u32 key Leon Romanovsky
2021-07-29 15:28   ` Jason Gunthorpe
2021-07-29 17:27     ` Leon Romanovsky
2021-07-29 18:08   ` Jason Gunthorpe
2021-06-22 12:08 ` [PATCH mlx5-next 2/5] RDMA/mlx5: Move struct mlx5_core_mkey to mlx5_ib Leon Romanovsky
2021-07-29 18:39   ` Jason Gunthorpe
2021-06-22 12:08 ` [PATCH mlx5-next 3/5] RDMA/mlx5: Change the cache to hold mkeys instead of MRs Leon Romanovsky
2021-07-29 19:08   ` Jason Gunthorpe [this message]
2021-06-22 12:08 ` [PATCH mlx5-next 4/5] RDMA/mlx5: Change the cache structure to an rbtree Leon Romanovsky
2021-07-29 19:45   ` Jason Gunthorpe
2021-06-22 12:08 ` [PATCH rdma-next 5/5] RDMA/mlx5: Delay the deregistration of a non-cache mkey Leon Romanovsky

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210729190816.GA2482631@nvidia.com \
    --to=jgg@nvidia.com \
    --cc=aharonl@nvidia.com \
    --cc=dledford@redhat.com \
    --cc=jasowang@redhat.com \
    --cc=kuba@kernel.org \
    --cc=leon@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=mst@redhat.com \
    --cc=netdev@vger.kernel.org \
    --cc=saeedm@nvidia.com \
    --cc=shayd@nvidia.com \
    --cc=virtualization@lists.linux-foundation.org \
    --subject='Re: [PATCH mlx5-next 3/5] RDMA/mlx5: Change the cache to hold mkeys instead of MRs' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
on how to clone and mirror all data and code used for this inbox