LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: 王贇 <yun.wang@linux.alibaba.com>
To: Dave Hansen <dave.hansen@linux.intel.com>,
Andy Lutomirski <luto@kernel.org>,
Peter Zijlstra <peterz@infradead.org>,
Thomas Gleixner <tglx@linutronix.de>,
Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
"maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT)"
<x86@kernel.org>, "H. Peter Anvin" <hpa@zytor.com>,
Alexei Starovoitov <ast@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
Andrii Nakryiko <andrii@kernel.org>,
Martin KaFai Lau <kafai@fb.com>, Song Liu <songliubraving@fb.com>,
Yonghong Song <yhs@fb.com>,
John Fastabend <john.fastabend@gmail.com>,
KP Singh <kpsingh@kernel.org>,
"open list:X86 MM" <linux-kernel@vger.kernel.org>,
"open list:BPF (Safe dynamic programs and tools)"
<netdev@vger.kernel.org>,
"open list:BPF (Safe dynamic programs and tools)"
<bpf@vger.kernel.org>
Subject: [PATCH] perf: fix panic by disable ftrace on fault.c
Date: Mon, 13 Sep 2021 11:30:52 +0800 [thread overview]
Message-ID: <d16e7188-1afa-7513-990c-804811747bcb@linux.alibaba.com> (raw)
In-Reply-To: <ff979a43-045a-dc56-64d1-2c31dd4db381@linux.alibaba.com>
When running with ftrace function enabled, we observed panic
as below:
traps: PANIC: double fault, error_code: 0x0
[snip]
RIP: 0010:perf_swevent_get_recursion_context+0x0/0x70
[snip]
Call Trace:
<NMI>
perf_trace_buf_alloc+0x26/0xd0
perf_ftrace_function_call+0x18f/0x2e0
kernelmode_fixup_or_oops+0x5/0x120
__bad_area_nosemaphore+0x1b8/0x280
do_user_addr_fault+0x410/0x920
exc_page_fault+0x92/0x300
asm_exc_page_fault+0x1e/0x30
RIP: 0010:__get_user_nocheck_8+0x6/0x13
perf_callchain_user+0x266/0x2f0
get_perf_callchain+0x194/0x210
perf_callchain+0xa3/0xc0
perf_prepare_sample+0xa5/0xa60
perf_event_output_forward+0x7b/0x1b0
__perf_event_overflow+0x67/0x120
perf_swevent_overflow+0xcb/0x110
perf_swevent_event+0xb0/0xf0
perf_tp_event+0x292/0x410
perf_trace_run_bpf_submit+0x87/0xc0
perf_trace_lock_acquire+0x12b/0x170
lock_acquire+0x1bf/0x2e0
perf_output_begin+0x70/0x4b0
perf_log_throttle+0xe2/0x1a0
perf_event_nmi_handler+0x30/0x50
nmi_handle+0xba/0x2a0
default_do_nmi+0x45/0xf0
exc_nmi+0x155/0x170
end_repeat_nmi+0x16/0x55
According to the trace we know the story is like this, the NMI
triggered perf IRQ throttling and call perf_log_throttle(),
which triggered the swevent overflow, and the overflow process
do perf_callchain_user() which triggered a user PF, and the PF
process triggered perf ftrace which finally lead into a suspected
stack overflow.
This patch disable ftrace on fault.c, which help to avoid the panic.
Reported-by: Abaci <abaci@linux.alibaba.com>
Signed-off-by: Michael Wang <yun.wang@linux.alibaba.com>
---
arch/x86/mm/Makefile | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile
index 5864219..1dbdca5 100644
--- a/arch/x86/mm/Makefile
+++ b/arch/x86/mm/Makefile
@@ -1,5 +1,9 @@
# SPDX-License-Identifier: GPL-2.0
# Kernel does not boot with instrumentation of tlb.c and mem_encrypt*.c
+
+# Disable ftrace to avoid stack overflow.
+CFLAGS_REMOVE_fault.o = $(CC_FLAGS_FTRACE)
+
KCOV_INSTRUMENT_tlb.o := n
KCOV_INSTRUMENT_mem_encrypt.o := n
KCOV_INSTRUMENT_mem_encrypt_identity.o := n
--
1.8.3.1
next prev parent reply other threads:[~2021-09-13 3:31 UTC|newest]
Thread overview: 39+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-09-09 3:13 [RFC PATCH] perf: fix panic by mark recursion inside perf_log_throttle 王贇
2021-09-09 6:10 ` 王贇
2021-09-10 15:38 ` Peter Zijlstra
2021-09-13 3:00 ` 王贇
2021-09-13 3:21 ` 王贇
2021-09-13 10:24 ` Peter Zijlstra
2021-09-13 10:36 ` Peter Zijlstra
2021-09-14 2:02 ` 王贇
2021-09-14 1:58 ` 王贇
2021-09-14 10:28 ` Peter Zijlstra
2021-09-15 1:51 ` 王贇
2021-09-15 15:17 ` [PATCH] x86/dumpstack/64: Add guard pages to stack_info Peter Zijlstra
2021-09-16 3:34 ` 王贇
2021-09-16 3:47 ` 王贇
2021-09-16 8:00 ` Peter Zijlstra
2021-09-16 8:03 ` Peter Zijlstra
2021-09-16 10:02 ` Peter Zijlstra
2021-09-17 2:15 ` 王贇
2021-09-17 3:02 ` 王贇
2021-09-17 10:21 ` Peter Zijlstra
2021-09-17 16:40 ` Peter Zijlstra
2021-09-18 2:30 ` 王贇
2021-09-18 6:56 ` Peter Zijlstra
2021-09-18 2:38 ` 王贇
2021-09-13 3:30 ` 王贇 [this message]
2021-09-13 14:49 ` [PATCH] perf: fix panic by disable ftrace on fault.c Dave Hansen
2021-09-14 1:52 ` 王贇
2021-09-14 3:02 ` 王贇
2021-09-14 7:23 ` 王贇
2021-09-14 16:16 ` Dave Hansen
2021-09-15 1:56 ` 王贇
2021-09-15 3:27 ` Dave Hansen
2021-09-15 7:22 ` 王贇
2021-09-15 7:34 ` 王贇
2021-09-15 15:19 ` [PATCH] x86: Increase exception stack sizes Peter Zijlstra
2021-09-16 3:42 ` 王贇
2021-09-21 7:28 ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2021-09-21 12:41 ` tip-bot2 for Peter Zijlstra
2021-09-14 2:08 ` [PATCH] perf: fix panic by disable ftrace on fault.c 王贇
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=d16e7188-1afa-7513-990c-804811747bcb@linux.alibaba.com \
--to=yun.wang@linux.alibaba.com \
--cc=andrii@kernel.org \
--cc=ast@kernel.org \
--cc=bp@alien8.de \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=dave.hansen@linux.intel.com \
--cc=hpa@zytor.com \
--cc=john.fastabend@gmail.com \
--cc=kafai@fb.com \
--cc=kpsingh@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=luto@kernel.org \
--cc=mingo@redhat.com \
--cc=netdev@vger.kernel.org \
--cc=peterz@infradead.org \
--cc=songliubraving@fb.com \
--cc=tglx@linutronix.de \
--cc=x86@kernel.org \
--cc=yhs@fb.com \
--subject='Re: [PATCH] perf: fix panic by disable ftrace on fault.c' \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).