LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Alexander Lobakin <alobakin@pm.me>
To: "David S. Miller" <davem@davemloft.net>,
	Jakub Kicinski <kuba@kernel.org>
Cc: Alexei Starovoitov <ast@kernel.org>,
	Daniel Borkmann <daniel@iogearbox.net>,
	Andrii Nakryiko <andrii@kernel.org>,
	Martin KaFai Lau <kafai@fb.com>, Song Liu <songliubraving@fb.com>,
	Yonghong Song <yhs@fb.com>,
	John Fastabend <john.fastabend@gmail.com>,
	KP Singh <kpsingh@kernel.org>,
	Jonathan Lemon <jonathan.lemon@gmail.com>,
	Alexander Lobakin <alobakin@pm.me>,
	Eric Dumazet <edumazet@google.com>,
	Willem de Bruijn <willemb@google.com>,
	Kevin Hao <haokexin@gmail.com>,
	Pablo Neira Ayuso <pablo@netfilter.org>,
	Jakub Sitnicki <jakub@cloudflare.com>,
	Marco Elver <elver@google.com>, Dexuan Cui <decui@microsoft.com>,
	Vladimir Oltean <vladimir.oltean@nxp.com>,
	Ariel Levkovich <lariel@mellanox.com>,
	Wang Qing <wangqing@vivo.com>,
	Davide Caratti <dcaratti@redhat.com>,
	Guillaume Nault <gnault@redhat.com>,
	Eran Ben Elisha <eranbe@nvidia.com>,
	Mauro Carvalho Chehab <mchehab+huawei@kernel.org>,
	Kirill Tkhai <ktkhai@virtuozzo.com>,
	Bartosz Golaszewski <bgolaszewski@baylibre.com>,
	netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
	bpf@vger.kernel.org
Subject: [PATCH net-next 6/6] skbuff: micro-optimize {,__}skb_header_pointer()
Date: Fri, 12 Mar 2021 19:47:10 +0000	[thread overview]
Message-ID: <20210312194538.337504-7-alobakin@pm.me> (raw)
In-Reply-To: <20210312194538.337504-1-alobakin@pm.me>

{,__}skb_header_pointer() helpers exist mainly for preventing
accesses-beyond-end of the linear data.
In the vast majorify of cases, they bail out on the first condition.
All code going after is mostly a fallback.
Mark the most common branch as 'likely' one to move it in-line.
Also, skb_copy_bits() can return negative values only when the input
arguments are invalid, e.g. offset is greater than skb->len. It can
be safely marked as 'unlikely' branch, assuming that hotpath code
provides sane input to not fail here.

These two bump the throughput with a single Flow Dissector pass on
every packet (e.g. with RPS or driver that uses eth_get_headlen())
on 20 Mbps per flow/core.

Signed-off-by: Alexander Lobakin <alobakin@pm.me>
---
 include/linux/skbuff.h | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index 7873f24c0ae5..71f4d609819e 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -3680,11 +3680,10 @@ static inline void * __must_check
 __skb_header_pointer(const struct sk_buff *skb, int offset, int len,
 		     const void *data, int hlen, void *buffer)
 {
-	if (hlen - offset >= len)
+	if (likely(hlen - offset >= len))
 		return (void *)data + offset;

-	if (!skb ||
-	    skb_copy_bits(skb, offset, buffer, len) < 0)
+	if (!skb || unlikely(skb_copy_bits(skb, offset, buffer, len) < 0))
 		return NULL;

 	return buffer;
--
2.30.2



      parent reply	other threads:[~2021-03-12 19:48 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-03-12 19:46 [PATCH net-next 0/6] skbuff: micro-optimize flow dissection Alexander Lobakin
2021-03-12 19:46 ` [PATCH net-next 1/6] flow_dissector: constify bpf_flow_dissector's data pointers Alexander Lobakin
2021-03-12 19:46 ` [PATCH net-next 2/6] skbuff: make __skb_header_pointer()'s data argument const Alexander Lobakin
2021-03-12 19:46 ` [PATCH net-next 3/6] flow_dissector: constify raw input @data argument Alexander Lobakin
2021-03-12 19:46 ` [PATCH net-next 4/6] linux/etherdevice.h: misc trailing whitespace cleanup Alexander Lobakin
2021-03-12 19:46 ` [PATCH net-next 5/6] ethernet: constify eth_get_headlen()'s @data argument Alexander Lobakin
2021-03-12 19:47 ` Alexander Lobakin [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210312194538.337504-7-alobakin@pm.me \
    --to=alobakin@pm.me \
    --cc=andrii@kernel.org \
    --cc=ast@kernel.org \
    --cc=bgolaszewski@baylibre.com \
    --cc=bpf@vger.kernel.org \
    --cc=daniel@iogearbox.net \
    --cc=davem@davemloft.net \
    --cc=dcaratti@redhat.com \
    --cc=decui@microsoft.com \
    --cc=edumazet@google.com \
    --cc=elver@google.com \
    --cc=eranbe@nvidia.com \
    --cc=gnault@redhat.com \
    --cc=haokexin@gmail.com \
    --cc=jakub@cloudflare.com \
    --cc=john.fastabend@gmail.com \
    --cc=jonathan.lemon@gmail.com \
    --cc=kafai@fb.com \
    --cc=kpsingh@kernel.org \
    --cc=ktkhai@virtuozzo.com \
    --cc=kuba@kernel.org \
    --cc=lariel@mellanox.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mchehab+huawei@kernel.org \
    --cc=netdev@vger.kernel.org \
    --cc=pablo@netfilter.org \
    --cc=songliubraving@fb.com \
    --cc=vladimir.oltean@nxp.com \
    --cc=wangqing@vivo.com \
    --cc=willemb@google.com \
    --cc=yhs@fb.com \
    --subject='Re: [PATCH net-next 6/6] skbuff: micro-optimize {,__}skb_header_pointer()' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).