Date: Sat, 25 Sep 2021 19:34:07 +0300 From: Andriy Gapon <avg@FreeBSD.org> To: Kristof Provost <kp@freebsd.org> Cc: net@freebsd.org, Konstantin Belousov <kib@freebsd.org> Subject: Re: page fault in pfioctl Message-ID: <991e038b-69e2-0d37-908b-1625454a6264@FreeBSD.org> In-Reply-To: <a081e770-581a-0e40-4076-ac11b59e1616@FreeBSD.org> References: <cc06b2d5-5349-33f3-4bdd-aa2ba5fa75b4@FreeBSD.org> <980E0B5C-41CF-466E-AD45-7B93532199F4@freebsd.org> <a081e770-581a-0e40-4076-ac11b59e1616@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 25/09/2021 19:25, Andriy Gapon wrote:
> On 13/06/2021 11:19, Kristof Provost wrote:
>> On 13 Jun 2021, at 09:41, Andriy Gapon <avg@freebsd.org> wrote:
>>> Based on
>>> the panic message (page fault with non-sleepable locks held), it seems that
>>> the problem is with holding the lock across the copyout. Usually that
>>> won't panic, but if the destination happens to be paged out... And only
>>> with INVARIANTS, I guess...
>>
>> Oh right. Thanks. I’ve gotten bitten by that one before, but had clearly
>> garbage collected the memory.
>>
>> I’ll fix this one and check for others on Monday.
>>
>> I’ll also see of we can persuade copyout to always panic on this bug, not
>> just when the destination memory is actually paged out. That way we’ll catch
>> this in the regression tests in the future.
>
> I upgraded to the latest stable/13 and hit a fresh panic of the same type.
> This time it's in pf_getstatus() and it's a copyout while 'pf rulesets' lock is
> held.
>
>
> <118>Enabling pf
> Kernel page fault with the following non-sleepable locks held:
> shared rm pf rulesets (pf rulesets) r = 0 (0xffffffff85764020) locked @
> /usr/devel/git/trant/sys/netpfil/pf/pf_ioctl.c:4945
> stack backtrace:
> #0 0xffffffff808cb43d at witness_debugger+0x6d
> #1 0xffffffff808cc2ab at witness_warn+0x21b
> #2 0xffffffff80b567f1 at trap_pfault+0x71
> #3 0xffffffff80b55df8 at trap+0x288
> #4 0xffffffff80b56b59 at trap_check+0x29
> #5 0xffffffff80b32298 at calltrap+0x8
> #6 0xffffffff8574cae8 at pf_getstatus+0x548
> #7 0xffffffff85747430 at pfioctl+0x2590
> #8 0xffffffff8073854f at devfs_ioctl+0xcf
> #9 0xffffffff80bd8c26 at VOP_IOCTL_APV+0x96
> #10 0xffffffff8094c424 at VOP_IOCTL+0x34
> #11 0xffffffff80947600 at vn_ioctl+0xc0
> #12 0xffffffff80738a3e at devfs_ioctl_f+0x1e
> #13 0xffffffff808cf8fb at fo_ioctl+0xb
> #14 0xffffffff808cf897 at kern_ioctl+0x1d7
> #15 0xffffffff808cf60d at sys_ioctl+0x12d
> #16 0xffffffff80b57353 at syscallenter+0x163
> #17 0xffffffff80b57025 at amd64_syscall+0x15
Hmm, there is more:
Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 02
fault virtual address = 0x800a0e000
fault code = supervisor write data, page not present
instruction pointer = 0x20:0xffffffff80b52e8a
stack pointer = 0x28:0xfffffe01b45d0150
frame pointer = 0x28:0xfffffe01b45d0150
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 1436 (pfctl)
trap number = 12
panic: page fault
cpuid = 2
time = 1632573676
KDB: stack backtrace:
db_trace_self_wrapper() at 0xffffffff805c9eeb = db_trace_self_wrapper+0x2b/frame
0xfffffe01b45cfd10
kdb_backtrace() at 0xffffffff808aafe7 = kdb_backtrace+0x37/frame 0xfffffe01b45cfdc0
vpanic() at 0xffffffff8086788c = vpanic+0x18c/frame 0xfffffe01b45cfe20
panic() at 0xffffffff808674a3 = panic+0x43/frame 0xfffffe01b45cfe80
trap_fatal() at 0xffffffff80b56725 = trap_fatal+0x375/frame 0xfffffe01b45cfee0
trap_pfault() at 0xffffffff80b56800 = trap_pfault+0x80/frame 0xfffffe01b45cff50
trap() at 0xffffffff80b55df8 = trap+0x288/frame 0xfffffe01b45d0060
trap_check() at 0xffffffff80b56b59 = trap_check+0x29/frame 0xfffffe01b45d0080
calltrap() at 0xffffffff80b32298 = calltrap+0x8/frame 0xfffffe01b45d0080
--- trap 0xc, rip = 0xffffffff80b52e8a, rsp = 0xfffffe01b45d0150, rbp =
0xfffffe01b45d0150 ---
copyout_nosmap_std() at 0xffffffff80b52e8a = copyout_nosmap_std+0x15a/frame
0xfffffe01b45d0150
pf_getstatus() at 0xffffffff8574cae8 = pf_getstatus+0x548/frame 0xfffffe01b45d0480
pfioctl() at 0xffffffff85747430 = pfioctl+0x2590/frame 0xfffffe01b45d0930
devfs_ioctl() at 0xffffffff8073854f = devfs_ioctl+0xcf/frame 0xfffffe01b45d0990
VOP_IOCTL_APV() at 0xffffffff80bd8c26 = VOP_IOCTL_APV+0x96/frame 0xfffffe01b45d09b0
VOP_IOCTL() at 0xffffffff8094c424 = VOP_IOCTL+0x34/frame 0xfffffe01b45d0a00
vn_ioctl() at 0xffffffff80947600 = vn_ioctl+0xc0/frame 0xfffffe01b45d0af0
devfs_ioctl_f() at 0xffffffff80738a3e = devfs_ioctl_f+0x1e/frame 0xfffffe01b45d0b10
fo_ioctl() at 0xffffffff808cf8fb = fo_ioctl+0xb/frame 0xfffffe01b45d0b20
kern_ioctl() at 0xffffffff808cf897 = kern_ioctl+0x1d7/frame 0xfffffe01b45d0b80
sys_ioctl() at 0xffffffff808cf60d = sys_ioctl+0x12d/frame 0xfffffe01b45d0c50
syscallenter() at 0xffffffff80b57353 = syscallenter+0x163/frame 0xfffffe01b45d0ca0
amd64_syscall() at 0xffffffff80b57025 = amd64_syscall+0x15/frame 0xfffffe01b45d0d30
fast_syscall_common() at 0xffffffff80b32bab = fast_syscall_common+0xf8/frame
0xfffffe01b45d0d30
--- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x80042adaa, rsp =
0x7fffffffe5f8, rbp = 0x7fffffffe650 ---
Uptime: 1m10s
Dumping 1462 out of 32644 MB:..2%..11%..21%..31%..41%..51%..61%..71%..81%..91%
Unfortunately kgdb itself crashes when trying to examine the dump.
I think it's strange that copyout_nosmap_std() crashes with a page fault
apparently when writing to a userland address.
--
Andriy Gapon
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?991e038b-69e2-0d37-908b-1625454a6264>