From nobody Sat Sep 25 16:34:07 2021 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id AA9C517CF98A for ; Sat, 25 Sep 2021 16:34:09 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HGvcK4Sbtz4mrM; Sat, 25 Sep 2021 16:34:09 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from [192.168.0.88] (unknown [195.64.148.76]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: avg/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 0A802C6A0; Sat, 25 Sep 2021 16:34:08 +0000 (UTC) (envelope-from avg@FreeBSD.org) From: Andriy Gapon To: Kristof Provost Cc: net@freebsd.org, Konstantin Belousov References: <980E0B5C-41CF-466E-AD45-7B93532199F4@freebsd.org> Subject: Re: page fault in pfioctl Message-ID: <991e038b-69e2-0d37-908b-1625454a6264@FreeBSD.org> Date: Sat, 25 Sep 2021 19:34:07 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Firefox/78.0 Thunderbird/78.14.0 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-ThisMailContainsUnwantedMimeParts: N On 25/09/2021 19:25, Andriy Gapon wrote: > On 13/06/2021 11:19, Kristof Provost wrote: >> On 13 Jun 2021, at 09:41, Andriy Gapon wrote: >>> Based on >>> the panic message (page fault with non-sleepable locks held), it seems that >>> the problem is with holding the lock across the copyout.  Usually that >>> won't panic, but if the destination happens to be paged out... And only >>> with INVARIANTS, I guess... >> >> Oh right. Thanks. I’ve gotten bitten by that one before, but had clearly >> garbage collected the memory. >> >> I’ll fix this one and check for others on Monday. >> >> I’ll also see of we can persuade copyout to always panic on this bug, not >> just when the destination memory is actually paged out. That way we’ll catch >> this in the regression tests in the future. > > I upgraded to the latest stable/13 and hit a fresh panic of the same type. > This time it's in pf_getstatus() and it's a copyout while 'pf rulesets' lock is > held. > > > <118>Enabling pf > Kernel page fault with the following non-sleepable locks held: > shared rm pf rulesets (pf rulesets) r = 0 (0xffffffff85764020) locked @ > /usr/devel/git/trant/sys/netpfil/pf/pf_ioctl.c:4945 > stack backtrace: > #0 0xffffffff808cb43d at witness_debugger+0x6d > #1 0xffffffff808cc2ab at witness_warn+0x21b > #2 0xffffffff80b567f1 at trap_pfault+0x71 > #3 0xffffffff80b55df8 at trap+0x288 > #4 0xffffffff80b56b59 at trap_check+0x29 > #5 0xffffffff80b32298 at calltrap+0x8 > #6 0xffffffff8574cae8 at pf_getstatus+0x548 > #7 0xffffffff85747430 at pfioctl+0x2590 > #8 0xffffffff8073854f at devfs_ioctl+0xcf > #9 0xffffffff80bd8c26 at VOP_IOCTL_APV+0x96 > #10 0xffffffff8094c424 at VOP_IOCTL+0x34 > #11 0xffffffff80947600 at vn_ioctl+0xc0 > #12 0xffffffff80738a3e at devfs_ioctl_f+0x1e > #13 0xffffffff808cf8fb at fo_ioctl+0xb > #14 0xffffffff808cf897 at kern_ioctl+0x1d7 > #15 0xffffffff808cf60d at sys_ioctl+0x12d > #16 0xffffffff80b57353 at syscallenter+0x163 > #17 0xffffffff80b57025 at amd64_syscall+0x15 Hmm, there is more: Fatal trap 12: page fault while in kernel mode cpuid = 2; apic id = 02 fault virtual address = 0x800a0e000 fault code = supervisor write data, page not present instruction pointer = 0x20:0xffffffff80b52e8a stack pointer = 0x28:0xfffffe01b45d0150 frame pointer = 0x28:0xfffffe01b45d0150 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 1436 (pfctl) trap number = 12 panic: page fault cpuid = 2 time = 1632573676 KDB: stack backtrace: db_trace_self_wrapper() at 0xffffffff805c9eeb = db_trace_self_wrapper+0x2b/frame 0xfffffe01b45cfd10 kdb_backtrace() at 0xffffffff808aafe7 = kdb_backtrace+0x37/frame 0xfffffe01b45cfdc0 vpanic() at 0xffffffff8086788c = vpanic+0x18c/frame 0xfffffe01b45cfe20 panic() at 0xffffffff808674a3 = panic+0x43/frame 0xfffffe01b45cfe80 trap_fatal() at 0xffffffff80b56725 = trap_fatal+0x375/frame 0xfffffe01b45cfee0 trap_pfault() at 0xffffffff80b56800 = trap_pfault+0x80/frame 0xfffffe01b45cff50 trap() at 0xffffffff80b55df8 = trap+0x288/frame 0xfffffe01b45d0060 trap_check() at 0xffffffff80b56b59 = trap_check+0x29/frame 0xfffffe01b45d0080 calltrap() at 0xffffffff80b32298 = calltrap+0x8/frame 0xfffffe01b45d0080 --- trap 0xc, rip = 0xffffffff80b52e8a, rsp = 0xfffffe01b45d0150, rbp = 0xfffffe01b45d0150 --- copyout_nosmap_std() at 0xffffffff80b52e8a = copyout_nosmap_std+0x15a/frame 0xfffffe01b45d0150 pf_getstatus() at 0xffffffff8574cae8 = pf_getstatus+0x548/frame 0xfffffe01b45d0480 pfioctl() at 0xffffffff85747430 = pfioctl+0x2590/frame 0xfffffe01b45d0930 devfs_ioctl() at 0xffffffff8073854f = devfs_ioctl+0xcf/frame 0xfffffe01b45d0990 VOP_IOCTL_APV() at 0xffffffff80bd8c26 = VOP_IOCTL_APV+0x96/frame 0xfffffe01b45d09b0 VOP_IOCTL() at 0xffffffff8094c424 = VOP_IOCTL+0x34/frame 0xfffffe01b45d0a00 vn_ioctl() at 0xffffffff80947600 = vn_ioctl+0xc0/frame 0xfffffe01b45d0af0 devfs_ioctl_f() at 0xffffffff80738a3e = devfs_ioctl_f+0x1e/frame 0xfffffe01b45d0b10 fo_ioctl() at 0xffffffff808cf8fb = fo_ioctl+0xb/frame 0xfffffe01b45d0b20 kern_ioctl() at 0xffffffff808cf897 = kern_ioctl+0x1d7/frame 0xfffffe01b45d0b80 sys_ioctl() at 0xffffffff808cf60d = sys_ioctl+0x12d/frame 0xfffffe01b45d0c50 syscallenter() at 0xffffffff80b57353 = syscallenter+0x163/frame 0xfffffe01b45d0ca0 amd64_syscall() at 0xffffffff80b57025 = amd64_syscall+0x15/frame 0xfffffe01b45d0d30 fast_syscall_common() at 0xffffffff80b32bab = fast_syscall_common+0xf8/frame 0xfffffe01b45d0d30 --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x80042adaa, rsp = 0x7fffffffe5f8, rbp = 0x7fffffffe650 --- Uptime: 1m10s Dumping 1462 out of 32644 MB:..2%..11%..21%..31%..41%..51%..61%..71%..81%..91% Unfortunately kgdb itself crashes when trying to examine the dump. I think it's strange that copyout_nosmap_std() crashes with a page fault apparently when writing to a userland address. -- Andriy Gapon