Netdev Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Jay Vosburgh <jay.vosburgh@canonical.com>
To: Jussi Maki <joamaki@gmail.com>
Cc: bpf@vger.kernel.org, netdev@vger.kernel.org,
	daniel@iogearbox.net, andy@greyhouse.net, vfalico@gmail.com,
	andrii@kernel.org
Subject: Re: [PATCH bpf-next 2/3] net: bonding: Use per-cpu rr_tx_counter
Date: Wed, 09 Jun 2021 17:04:29 -0700	[thread overview]
Message-ID: <22630.1623283469@famine> (raw)
In-Reply-To: <20210609135537.1460244-3-joamaki@gmail.com>

Jussi Maki <joamaki@gmail.com> wrote:

>The round-robin rr_tx_counter was shared across CPUs leading
>to significant cache trashing at high packet rates. This patch

	"trashing" -> "thrashing" ?

>switches the round-robin mechanism to use a per-cpu counter to
>decide the destination device.
>
>On a 100Gbit 64 byte packet test this reduces the CPU load from
>50% to 10% on the test system.
>
>Signed-off-by: Jussi Maki <joamaki@gmail.com>
>---
> drivers/net/bonding/bond_main.c | 18 +++++++++++++++---
> include/net/bonding.h           |  2 +-
> 2 files changed, 16 insertions(+), 4 deletions(-)
>
>diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c
>index 38eea7e096f3..917dd2cdcbf4 100644
>--- a/drivers/net/bonding/bond_main.c
>+++ b/drivers/net/bonding/bond_main.c
>@@ -4314,16 +4314,16 @@ static u32 bond_rr_gen_slave_id(struct bonding *bond)
> 		slave_id = prandom_u32();
> 		break;
> 	case 1:
>-		slave_id = bond->rr_tx_counter;
>+		slave_id = this_cpu_inc_return(*bond->rr_tx_counter);
> 		break;
> 	default:
> 		reciprocal_packets_per_slave =
> 			bond->params.reciprocal_packets_per_slave;
>-		slave_id = reciprocal_divide(bond->rr_tx_counter,
>+		slave_id = this_cpu_inc_return(*bond->rr_tx_counter);
>+		slave_id = reciprocal_divide(slave_id,
> 					     reciprocal_packets_per_slave);

	With the rr_tx_counter is per-cpu, each CPU is essentially doing
its own round-robin logic, independently of other CPUs, so the resulting
spread of transmitted packets may not be as evenly distributed (as
multiple CPUs could select the same interface to transmit on
approximately in lock-step).  I'm not sure if this could cause actual
problems in practice, though, as particular flows shouldn't skip between
CPUs (and thus rr_tx_counters) very often, and round-robin already
shouldn't be the first choice if no packet reordering is a hard
requirement.

	I think this patch could be submitted against net-next
independently of the rest of the series.

Acked-by: Jay Vosburgh <jay.vosburgh@canonical.com>

	-J

> 		break;
> 	}
>-	bond->rr_tx_counter++;
> 
> 	return slave_id;
> }
>@@ -5278,6 +5278,9 @@ static void bond_uninit(struct net_device *bond_dev)
> 
> 	list_del(&bond->bond_list);
> 
>+	if (BOND_MODE(bond) == BOND_MODE_ROUNDROBIN)
>+		free_percpu(bond->rr_tx_counter);
>+
> 	bond_debug_unregister(bond);
> }
> 
>@@ -5681,6 +5684,15 @@ static int bond_init(struct net_device *bond_dev)
> 	if (!bond->wq)
> 		return -ENOMEM;
> 
>+	if (BOND_MODE(bond) == BOND_MODE_ROUNDROBIN) {
>+		bond->rr_tx_counter = alloc_percpu(u32);
>+		if (!bond->rr_tx_counter) {
>+			destroy_workqueue(bond->wq);
>+			bond->wq = NULL;
>+			return -ENOMEM;
>+		}
>+	}
>+
> 	spin_lock_init(&bond->stats_lock);
> 	netdev_lockdep_set_classes(bond_dev);
> 
>diff --git a/include/net/bonding.h b/include/net/bonding.h
>index 34acb81b4234..8de8180f1be8 100644
>--- a/include/net/bonding.h
>+++ b/include/net/bonding.h
>@@ -232,7 +232,7 @@ struct bonding {
> 	char     proc_file_name[IFNAMSIZ];
> #endif /* CONFIG_PROC_FS */
> 	struct   list_head bond_list;
>-	u32      rr_tx_counter;
>+	u32 __percpu *rr_tx_counter;
> 	struct   ad_bond_info ad_info;
> 	struct   alb_bond_info alb_info;
> 	struct   bond_params params;
>-- 
>2.30.2
>

  reply	other threads:[~2021-06-10  0:04 UTC|newest]

Thread overview: 71+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-06-09 13:55 [PATCH bpf-next 0/3] XDP bonding support Jussi Maki
2021-06-09 13:55 ` [PATCH bpf-next 1/3] net: bonding: Add XDP support to the bonding driver Jussi Maki
2021-06-09 22:29   ` Maciej Fijalkowski
2021-06-09 23:29   ` Jay Vosburgh
2021-06-14  8:02     ` Jussi Maki
2021-06-17  3:40   ` kernel test robot
2021-06-17  6:35   ` kernel test robot
2021-06-22  7:24   ` kernel test robot
2021-06-09 13:55 ` [PATCH bpf-next 2/3] net: bonding: Use per-cpu rr_tx_counter Jussi Maki
2021-06-10  0:04   ` Jay Vosburgh [this message]
2021-06-14  7:54     ` Jussi Maki
2021-06-09 13:55 ` [PATCH bpf-next 3/3] selftests/bpf: Add tests for XDP bonding Jussi Maki
2021-06-09 22:07   ` Maciej Fijalkowski
2021-06-14  8:08     ` Jussi Maki
2021-06-14  8:48       ` Magnus Karlsson
2021-06-14 12:20         ` Jussi Maki
2021-06-10 17:24 ` [PATCH bpf-next 0/3] XDP bonding support Andrii Nakryiko
2021-06-14 12:25   ` Jussi Maki
2021-06-14 15:37     ` Jay Vosburgh
2021-06-15  5:34     ` Andrii Nakryiko
2021-06-24  9:18 ` [PATCH bpf-next v2 0/4] " joamaki
2021-06-24  9:18   ` [PATCH bpf-next v2 1/4] net: bonding: Refactor bond_xmit_hash for use with xdp_buff joamaki
2021-06-24  9:18   ` [PATCH bpf-next v2 2/4] net: core: Add support for XDP redirection to slave device joamaki
2021-06-24  9:18   ` [PATCH bpf-next v2 3/4] net: bonding: Add XDP support to the bonding driver joamaki
2021-06-24  9:18   ` [PATCH bpf-next v2 4/4] devmap: Exclude XDP broadcast to master device joamaki
2021-07-01 18:12     ` Jay Vosburgh
2021-07-05 11:44       ` Jussi Maki
2021-07-01 18:20   ` [PATCH bpf-next v2 0/4] XDP bonding support Jay Vosburgh
2021-07-05 10:32     ` Jussi Maki
2021-07-07 11:25 ` [PATCH bpf-next v3 0/5] " Jussi Maki
2021-07-07 11:25   ` [PATCH bpf-next v3 1/5] net: bonding: Refactor bond_xmit_hash for use with xdp_buff Jussi Maki
2021-07-07 11:25   ` [PATCH bpf-next v3 2/5] net: core: Add support for XDP redirection to slave device Jussi Maki
2021-07-07 11:25   ` [PATCH bpf-next v3 3/5] net: bonding: Add XDP support to the bonding driver Jussi Maki
2021-07-13  7:14     ` kernel test robot
2021-07-07 11:25   ` [PATCH bpf-next v3 4/5] devmap: Exclude XDP broadcast to master device Jussi Maki
2021-07-07 11:25   ` [PATCH bpf-next v3 5/5] net: core: Allow netdev_lower_get_next_private_rcu in bh context Jussi Maki
2021-07-28 23:43 ` [PATCH bpf-next v4 0/6] XDP bonding support joamaki
2021-07-28 23:43   ` [PATCH bpf-next v4 1/6] net: bonding: Refactor bond_xmit_hash for use with xdp_buff joamaki
2021-07-28 23:43   ` [PATCH bpf-next v4 2/6] net: core: Add support for XDP redirection to slave device joamaki
2021-07-28 23:43   ` [PATCH bpf-next v4 3/6] net: bonding: Add XDP support to the bonding driver joamaki
2021-07-28 23:43   ` [PATCH bpf-next v4 4/6] devmap: Exclude XDP broadcast to master device joamaki
2021-07-28 23:43   ` [PATCH bpf-next v4 5/6] net: core: Allow netdev_lower_get_next_private_rcu in bh context joamaki
2021-07-28 23:43   ` [PATCH bpf-next v4 6/6] selftests/bpf: Add tests for XDP bonding joamaki
2021-08-03  0:19     ` Andrii Nakryiko
2021-08-03  9:40       ` Jussi Maki
2021-07-30  6:18 ` [PATCH bpf-next v5 0/7] XDP bonding support Jussi Maki
2021-07-30  6:18   ` [PATCH bpf-next v5 1/7] net: bonding: Refactor bond_xmit_hash for use with xdp_buff Jussi Maki
2021-07-30  6:18   ` [PATCH bpf-next v5 2/7] net: core: Add support for XDP redirection to slave device Jussi Maki
2021-07-30  6:18   ` [PATCH bpf-next v5 3/7] net: bonding: Add XDP support to the bonding driver Jussi Maki
2021-07-30  6:18   ` [PATCH bpf-next v5 4/7] devmap: Exclude XDP broadcast to master device Jussi Maki
2021-07-30  6:18   ` [PATCH bpf-next v5 5/7] net: core: Allow netdev_lower_get_next_private_rcu in bh context Jussi Maki
2021-07-30  6:18   ` [PATCH bpf-next v5 6/7] selftests/bpf: Fix xdp_tx.c prog section name Jussi Maki
2021-08-04 23:35     ` Andrii Nakryiko
2021-07-30  6:18   ` [PATCH bpf-next v5 7/7] selftests/bpf: Add tests for XDP bonding Jussi Maki
2021-08-04 23:33     ` Andrii Nakryiko
2021-07-31  5:57 ` [PATCH bpf-next v6 0/7]: XDP bonding support Jussi Maki
2021-07-31  5:57   ` [PATCH bpf-next v6 1/7] net: bonding: Refactor bond_xmit_hash for use with xdp_buff Jussi Maki
2021-08-11  1:52     ` Jonathan Toppins
2021-08-11  8:22       ` Jussi Maki
2021-08-11 14:05         ` Jonathan Toppins
2021-08-16  9:05           ` Jussi Maki
2021-07-31  5:57   ` [PATCH bpf-next v6 2/7] net: core: Add support for XDP redirection to slave device Jussi Maki
2021-07-31  5:57   ` [PATCH bpf-next v6 3/7] net: bonding: Add XDP support to the bonding driver Jussi Maki
2021-07-31  5:57   ` [PATCH bpf-next v6 4/7] devmap: Exclude XDP broadcast to master device Jussi Maki
2021-07-31  5:57   ` [PATCH bpf-next v6 5/7] net: core: Allow netdev_lower_get_next_private_rcu in bh context Jussi Maki
2021-07-31  5:57   ` [PATCH bpf-next v6 6/7] selftests/bpf: Fix xdp_tx.c prog section name Jussi Maki
2021-08-06 22:53     ` Andrii Nakryiko
2021-07-31  5:57   ` [PATCH bpf-next v6 7/7] selftests/bpf: Add tests for XDP bonding Jussi Maki
2021-08-06 22:50     ` Andrii Nakryiko
2021-08-09 14:24       ` Jussi Maki
2021-08-09 21:41         ` Daniel Borkmann

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=22630.1623283469@famine \
    --to=jay.vosburgh@canonical.com \
    --cc=andrii@kernel.org \
    --cc=andy@greyhouse.net \
    --cc=bpf@vger.kernel.org \
    --cc=daniel@iogearbox.net \
    --cc=joamaki@gmail.com \
    --cc=netdev@vger.kernel.org \
    --cc=vfalico@gmail.com \
    --subject='Re: [PATCH bpf-next 2/3] net: bonding: Use per-cpu rr_tx_counter' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).