Netdev Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Johan Almbladh <johan.almbladh@anyfinetworks.com>
To: Andrii Nakryiko <andrii.nakryiko@gmail.com>
Cc: Yonghong Song <yhs@fb.com>, Alexei Starovoitov <ast@kernel.org>,
	Daniel Borkmann <daniel@iogearbox.net>,
	Andrii Nakryiko <andrii@kernel.org>,
	Martin KaFai Lau <kafai@fb.com>, Song Liu <songliubraving@fb.com>,
	John Fastabend <john.fastabend@gmail.com>,
	KP Singh <kpsingh@kernel.org>,
	Tony Ambardar <Tony.Ambardar@gmail.com>,
	Networking <netdev@vger.kernel.org>, bpf <bpf@vger.kernel.org>
Subject: Re: [PATCH] bpf: Fix off-by-one in tail call count limiting
Date: Sun, 1 Aug 2021 10:37:55 +0200	[thread overview]
Message-ID: <CAM1=_QSKa7W9SL7oXWGEHLtWqCeFWp-jtGoqPp9=MxQwUGOjaQ@mail.gmail.com> (raw)
In-Reply-To: <CAEf4BzZ1nNv12s-NJEayct5Yih_G6vNkEvFPst6dLcbhxWV_0g@mail.gmail.com>

On Fri, Jul 30, 2021 at 12:48 AM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Thu, Jul 29, 2021 at 3:29 PM Andrii Nakryiko
> <andrii.nakryiko@gmail.com> wrote:
> >
> > On Thu, Jul 29, 2021 at 2:38 PM Johan Almbladh
> > <johan.almbladh@anyfinetworks.com> wrote:
> > >
> > > On Wed, Jul 28, 2021 at 9:13 PM Yonghong Song <yhs@fb.com> wrote:
> > > > I also checked arm/arm64 jit. I saw the following comments:
> > > >
> > > >          /* if (tail_call_cnt > MAX_TAIL_CALL_CNT)
> > > >           *      goto out;
> > > >           * tail_call_cnt++;
> > > >           */
> > > >
> > > > Maybe we have this MAX_TAIL_CALL_CNT + 1 issue
> > > > for arm/arm64 jit?
> > >
> > > That wouldn't be unreasonable. I don't have an arm or arm64 setup
> > > available right now, but I can try to test it in qemu.
> >
> > On a brief check, there seems to be quite a mess in terms of the code
> > and comments.
> >
> > E.g., in arch/x86/net/bpf_jit_comp32.c:
> >
> >         /*
> >          * if (tail_call_cnt > MAX_TAIL_CALL_CNT)
> >          *     goto out;
> >          */
> >
> >                             ^^^^ here comment is wrong
> >
> >         [...]
> >
> >         /* cmp edx,hi */
> >         EMIT3(0x83, add_1reg(0xF8, IA32_EBX), hi);
> >         EMIT2(IA32_JNE, 3);
> >         /* cmp ecx,lo */
> >         EMIT3(0x83, add_1reg(0xF8, IA32_ECX), lo);
> >
> >         /* ja out */
> >         EMIT2(IA32_JAE, jmp_label(jmp_label1, 2));
> >
> >         ^^^ JAE is >=, right? But the comment says JA.
> >
> >
> > As for arch/x86/net/bpf_jit_comp.c, both comment and the code seem to
> > do > MAX_TAIL_CALL_CNT, but you are saying JIT is correct. What am I
> > missing?
> >
> > Can you please check all the places where MAX_TAIL_CALL_CNT is used
> > throughout the code? Let's clean this up in one go.
> >
> > Also, given it's so easy to do this off-by-one error, can you please
> > add a negative test validating that 33 tail calls are not allowed? I
> > assume we have a positive test that allows exactly MAX_TAIL_CALL_CNT,
> > but please double-check that as well.
>
> Ok, I see that you've added this in your bpf tests patch set. Please
> consider, additionally, implementing a similar test as part of
> selftests/bpf (specifically in test_progs). We run test_progs
> continuously in CI for every incoming patch/patchset, so it has much
> higher chances of capturing any regressions.
>
> I'm also thinking that this MAX_TAIL_CALL_CNT change should probably
> go into the bpf-next tree. First, this off-by-one behavior was around
> for a while and it doesn't cause serious issues, even if abused. But
> on the other hand, it will make your tail call tests fail, when
> applied into bpf-next without your change. So I think we should apply
> both into bpf-next.

I can confirm that the off-by-one behaviour is present on arm. Below
is the test output running on qemu. Test #4 calls itself recursively
and increments a counter each time, so the correct result should be 1
+ MAX_TAIL_CALL_CNT.

test_bpf: #0 Tail call leaf jited:1 71 PASS
test_bpf: #1 Tail call 2 jited:1 134 PASS
test_bpf: #2 Tail call 3 jited:1 164 PASS
test_bpf: #3 Tail call 4 jited:1 257 PASS
test_bpf: #4 Tail call error path, max count reached jited:1 ret 34 != 33 FAIL
test_bpf: #5 Tail call error path, NULL target jited:1 114 PASS
test_bpf: #6 Tail call error path, index out of range jited:1 112 PASS
test_bpf: test_tail_calls: Summary: 6 PASSED, 1 FAILED, [7/7 JIT'ed]

The MAX_TAIL_CALL_CNT constant is referenced in the following JITs.

arch/arm64/net/bpf_jit_comp.c
arch/arm/net/bpf_jit_32.c
arch/mips/net/ebpf_jit.c
arch/powerpc/net/bpf_jit_comp32.c
arch/powerpc/net/bpf_jit_comp64.c
arch/riscv/net/bpf_jit_comp32.c
arch/riscv/net/bpf_jit_comp64.c
arch/s390/net/bpf_jit_comp.c
arch/sparc/net/bpf_jit_comp_64.c
arch/x86/net/bpf_jit_comp32.c
arch/x86/net/bpf_jit_comp.c

The x86 JITs all pass the test, even though the comments are wrong.
The comments can easily be fixed of course. For JITs that have the
off-by-one behaviour, an easy fix would be to change all occurrences
of MAX_TAIL_CALL_CNT to MAX_TAIL_CALL_CNT - 1. We must first know
which JITs affected though.

The fix is easy but setting up the test is hard. It took me quite some
time to get the qemu/arm setup up and running. If the same has to be
done for arm64, mips64, powerpc, powerpc64, riscv32, risc64, sparc and
s390, I will need some help with this. If someone already has a
working setup for any of the systems, the test can be performed on
that.

Or perhaps there is a better way to do this? If I implement a similar
test in selftest/bpf, that would trigger the CI when the patch is
submitted and we will see which JITs we need to fix.

> On a related topic, please don't forget to include the target kernel
> tree for your patches: [PATCH bpf] or [PATCH bpf-next].

I'll add that! All patches I sent related to this are for the bpf-next tree.

Johan

  reply	other threads:[~2021-08-01  8:38 UTC|newest]

Thread overview: 27+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-07-26  8:17 [RFC PATCH 00/14] bpf/tests: Extend the eBPF test suite Johan Almbladh
2021-07-26  8:17 ` [RFC PATCH 01/14] bpf/tests: add BPF_JMP32 test cases Johan Almbladh
2021-07-26  8:17 ` [RFC PATCH 02/14] bpf/tests: add BPF_MOV tests for zero and sign extension Johan Almbladh
2021-07-26  8:17 ` [RFC PATCH 03/14] bpf/tests: fix typos in test case descriptions Johan Almbladh
2021-07-26  8:17 ` [RFC PATCH 04/14] bpf/tests: add more tests of ALU32 and ALU64 bitwise operations Johan Almbladh
2021-07-26  8:17 ` [RFC PATCH 05/14] bpf/tests: add more ALU32 tests for BPF_LSH/RSH/ARSH Johan Almbladh
2021-07-26  8:17 ` [RFC PATCH 06/14] bpf/tests: add more BPF_LSH/RSH/ARSH tests for ALU64 Johan Almbladh
2021-07-26  8:17 ` [RFC PATCH 07/14] bpf/tests: add more ALU64 BPF_MUL tests Johan Almbladh
2021-07-26  8:17 ` [RFC PATCH 08/14] bpf/tests: add tests for ALU operations implemented with function calls Johan Almbladh
2021-07-26  8:17 ` [RFC PATCH 09/14] bpf/tests: add word-order tests for load/store of double words Johan Almbladh
2021-07-26  8:17 ` [RFC PATCH 10/14] bpf/tests: add branch conversion JIT test Johan Almbladh
2021-07-26  8:17 ` [RFC PATCH 11/14] bpf/tests: add test for 32-bit context pointer argument passing Johan Almbladh
2021-07-26  8:17 ` [RFC PATCH 12/14] bpf/tests: add tests for atomic operations Johan Almbladh
2021-07-26  8:17 ` [RFC PATCH 13/14] bpf/tests: add tests for BPF_CMPXCHG Johan Almbladh
2021-07-26  8:17 ` [RFC PATCH 14/14] bpf/tests: add tail call test suite Johan Almbladh
2021-07-26 22:53 ` [RFC PATCH 00/14] bpf/tests: Extend the eBPF " Andrii Nakryiko
2021-07-28  8:27   ` Daniel Borkmann
2021-07-28 12:15     ` Johan Almbladh
2021-07-28 16:47     ` [PATCH] bpf: Fix off-by-one in tail call count limiting Johan Almbladh
2021-07-28 19:13       ` Yonghong Song
2021-07-29 21:37         ` Johan Almbladh
2021-07-29 22:29           ` Andrii Nakryiko
2021-07-29 22:48             ` Andrii Nakryiko
2021-08-01  8:37               ` Johan Almbladh [this message]
2021-08-02 20:28                 ` Andrii Nakryiko
2021-08-05 14:37                   ` Johan Almbladh
2021-08-05 22:54                     ` Andrii Nakryiko

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAM1=_QSKa7W9SL7oXWGEHLtWqCeFWp-jtGoqPp9=MxQwUGOjaQ@mail.gmail.com' \
    --to=johan.almbladh@anyfinetworks.com \
    --cc=Tony.Ambardar@gmail.com \
    --cc=andrii.nakryiko@gmail.com \
    --cc=andrii@kernel.org \
    --cc=ast@kernel.org \
    --cc=bpf@vger.kernel.org \
    --cc=daniel@iogearbox.net \
    --cc=john.fastabend@gmail.com \
    --cc=kafai@fb.com \
    --cc=kpsingh@kernel.org \
    --cc=netdev@vger.kernel.org \
    --cc=songliubraving@fb.com \
    --cc=yhs@fb.com \
    --subject='Re: [PATCH] bpf: Fix off-by-one in tail call count limiting' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
on how to clone and mirror all data and code used for this inbox