From nobody Wed Sep 1 03:12:58 2021 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 60A5417A15DA for ; Wed, 1 Sep 2021 03:13:08 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound1b.ore.mailhop.org (outbound1b.ore.mailhop.org [54.200.247.200]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Gzpz80HnGz3D5d for ; Wed, 1 Sep 2021 03:13:07 +0000 (UTC) (envelope-from ian@freebsd.org) ARC-Seal: i=1; a=rsa-sha256; t=1630465981; cv=none; d=outbound.mailhop.org; s=arc-outbound20181012; b=WATd8V8ZjnCnBppM9QEEQaf9jL6esIVM8cIfmONFdBIeveldEB+tI6w5WbKIXs4e0vjPjoInkLGWA mJCvj8sa8e3ZKkQWOWdVT4y1KsP/dkZnBT1XYDXy52crnX1kGksAZAJ9upxjbP6T9iRB1wDMjwp9yg /uacvKKb3gN7oTtyuEUhgN/7HpO1Fno/8cVgjtB5RAZ0L3jIh8xvGJPEgUinqc3fajddZUZXOrsxak ouwSO+MRWokjh+8QYZRGs9XaOq2aEvWy2cgTvsZyz8ixOB8BJymcYYR1MS2MzRgbzIA+ZIukWEwg8M nHStDln5GTq+migGf0KjAk8xsdxcXUw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=arc-outbound20181012; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:cc:to:from:subject:message-id:dkim-signature:from; bh=ixMyYNi+DoI6I9QGQsk9MB25GVgkhypBjjdU6y6Ic38=; b=lnsMky4+lBR/mNYYrD9r29RJHhaKGttOhmybG8qLcuvHUGmCg52eTjYTiez0NnmsiLcaWUiSxhd+W E+1BpUvx+ReK09ltf8WqbJj0Fm39S5o8JP2wfHVD8HkHcql47rs7CvM9iVYQlbO5nykSjueDRqnAbr MQeeXts9sBI03K46v4YqZr0rgWvyDSjx1i+R9gQiRrfMwNhBUSiKtjDD9/HLutOba6PSoQuNa89Eok z4rwTot1HHAV2O5vac/MWEFkm/fx3o2vWG2ZUIH0dYW5nigLfJMAhZ2OZ1iNs+Mmp9uhwYGbIz807I 8KuDFBcu8GgIdd8hHi25Q2qo1PVnzCg== ARC-Authentication-Results: i=1; outbound3.ore.mailhop.org; spf=softfail smtp.mailfrom=freebsd.org smtp.remote-ip=67.177.211.60; dmarc=none header.from=freebsd.org; arc=none header.oldest-pass=0; DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=dkim-high; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:cc:to:from:subject:message-id:from; bh=ixMyYNi+DoI6I9QGQsk9MB25GVgkhypBjjdU6y6Ic38=; b=k1vHZnU3BUbmB8X/fnWbUv1pCiaBbSRM4VYELSZ/621t8P6skJqifDXb7LC5eQy/DIltd1rq8TAkB jx/yadq9mVxIpLn1cjV3e43d6nL8icMN8Nkde2lYPkFddND2oYDGg5ixhnAIYYHO4rCwI0TOUSxz9Q G4T8vZeqA4crsBYZyMxyNlvAX0PzK2NEqV4Jmyk0JeY3LiybOQu6OWrnCh94Mf7XhFXEdklKV2G4yL gjunLnOGHwgwzu6EzyYYz9mlwhpS5+W08R1yW/Lkgk29UIGoR9JBnA8lMG33mGsj1JK0xEUg2zKjoM lsG4pzpce1kP135g4HLFlzyHNkIlSLQ== X-Originating-IP: 67.177.211.60 X-MHO-RoutePath: aGlwcGll X-MHO-User: 825c21d7-0ad2-11ec-97b4-bf9d68d023b6 X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (c-67-177-211-60.hsd1.co.comcast.net [67.177.211.60]) by outbound3.ore.mailhop.org (Halon) with ESMTPSA id 825c21d7-0ad2-11ec-97b4-bf9d68d023b6; Wed, 01 Sep 2021 03:12:59 +0000 (UTC) Received: from [172.22.42.84] (rev2.hippie.lan [172.22.42.84]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id 1813Cwpa085925; Tue, 31 Aug 2021 21:12:58 -0600 (MDT) (envelope-from ian@freebsd.org) X-Authentication-Warning: paranoia.hippie.lan: Host rev2.hippie.lan [172.22.42.84] claimed to be [172.22.42.84] Message-ID: Subject: Re: Patched gpsd and /dev/pps0 results in "sleeping thread" kernel panic From: Ian Lepore To: Warner Losh , Craig Leres Cc: FreeBSD Hackers Date: Tue, 31 Aug 2021 21:12:58 -0600 In-Reply-To: References: <5476ea21-9e8a-32f5-08ff-add46c02d910@freebsd.org> Content-Type: text/plain; charset="ISO-8859-1" User-Agent: Evolution 3.40.3 FreeBSD GNOME Team List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4Gzpz80HnGz3D5d X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N On Tue, 2021-08-31 at 21:07 -0600, Warner Losh wrote: > On Tue, Aug 31, 2021 at 8:11 PM Craig Leres > wrote: > > > > > On 8/31/21 6:54 PM, Warner Losh wrote: > > > Can you post a traceback? We recently tightened up the sleeping > > conditions > > > and many of the panics are straight forward to fix > > > > Is this sufficient? > > > > Maybe. I'm guessing that the pps driver isn't setting driver_mtx for > the > pps_state that it keeps, so when we're sleeping in pps_ioctl we're > not > dropping the the ppbus lock, which leads to this diagnostic. > > Warner > > Exactly so, I've just been digging through the code. It's not an easy fix because the ppbus pps driver doesn't have access to the ppbus mutex at the point where it's registering to be a pps driver. Craig, is there any chance you can configure your hardware so that the PPS signal is connected to either the CTS or DCD pin of a uart (even the same uart that's communicating with the gps receiver should work if you use DCD)? You'll get jitter and accuracy performance comparable to the parallel port pin, and avoid this mutex/sleeping problem. -- Ian > >                 Craig > > > > toc2 4 # kgdb /boot/kernel/kernel /var/crash/vmcore.1 > > GNU gdb (GDB) 10.2 [GDB v10.2 for FreeBSD] > > Copyright (C) 2021 Free Software Foundation, Inc. > > License GPLv3+: GNU GPL version 3 or later > > > > This is free software: you are free to change and redistribute it. > > There is NO WARRANTY, to the extent permitted by law. > > Type "show copying" and "show warranty" for details. > > This GDB was configured as "x86_64-portbld-freebsd12.2". > > Type "show configuration" for configuration details. > > For bug reporting instructions, please see: > > . > > Find the GDB manual and other documentation resources online at: > >      . > > > > For help, type "help". > > Type "apropos word" to search for commands related to "word"... > > Reading symbols from /boot/kernel/kernel... > > Reading symbols from /boot/kernel.LBLNET/kernel.debug... > > > > Unread portion of the kernel message buffer: > > Sleeping thread (tid 101408, pid 2046) owns a non-sleepable lock > > KDB: stack backtrace of thread 101408: > > sched_switch() at sched_switch+0x630/frame 0xfffffe0070e4f760 > > mi_switch() at mi_switch+0xd4/frame 0xfffffe0070e4f790 > > sleepq_catch_signals() at sleepq_catch_signals+0x403/frame > > 0xfffffe0070e4f7e0 > > sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame > > 0xfffffe0070e4f820 > > _sleep() at _sleep+0x1b3/frame 0xfffffe0070e4f8a0 > > pps_ioctl() at pps_ioctl+0x298/frame 0xfffffe0070e4f8f0 > > ppsioctl() at ppsioctl+0x48/frame 0xfffffe0070e4f920 > > devfs_ioctl() at devfs_ioctl+0xb0/frame 0xfffffe0070e4f970 > > VOP_IOCTL_APV() at VOP_IOCTL_APV+0x7b/frame 0xfffffe0070e4f9a0 > > vn_ioctl() at vn_ioctl+0x16a/frame 0xfffffe0070e4fab0 > > devfs_ioctl_f() at devfs_ioctl_f+0x1e/frame 0xfffffe0070e4fad0 > > kern_ioctl() at kern_ioctl+0x2b7/frame 0xfffffe0070e4fb30 > > sys_ioctl() at sys_ioctl+0xfa/frame 0xfffffe0070e4fc00 > > amd64_syscall() at amd64_syscall+0x387/frame 0xfffffe0070e4fd30 > > fast_syscall_common() at fast_syscall_common+0xf8/frame > > 0xfffffe0070e4fd30 > > --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x8004c899a, rsp > > = > > 0x7fffdfdfc6a8, rbp = 0x7fffdfdfc730 --- > > panic: sleeping thread > > cpuid = 8 > > time = 1630456142 > > KDB: stack backtrace: > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > > 0xfffffe005ab73ab0 > > vpanic() at vpanic+0x17b/frame 0xfffffe005ab73b00 > > panic() at panic+0x43/frame 0xfffffe005ab73b60 > > propagate_priority() at propagate_priority+0x282/frame > > 0xfffffe005ab73b90 > > turnstile_wait() at turnstile_wait+0x30c/frame 0xfffffe005ab73be0 > > __mtx_lock_sleep() at __mtx_lock_sleep+0x199/frame > > 0xfffffe005ab73c70 > > ppcintr() at ppcintr+0x2a0/frame 0xfffffe005ab73c90 > > ithread_loop() at ithread_loop+0x23c/frame 0xfffffe005ab73cf0 > > fork_exit() at fork_exit+0x7e/frame 0xfffffe005ab73d30 > > fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe005ab73d30 > > --- trap 0, rip = 0, rsp = 0, rbp = 0 --- > > Uptime: 17m20s > > Dumping 607 out of 12240 > > MB:..3%..11%..22%..32%..43%..51%..61%..72%..82%..93% > > > > __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 > > 55      /usr/src/sys/amd64/include/pcpu_aux.h: No such file or > > directory. > > (kgdb) bt > > #0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 > > #1  doadump (textdump=1) at ../../../kern/kern_shutdown.c:371 > > #2  0xffffffff80b83ada in kern_reboot (howto=260) > >      at ../../../kern/kern_shutdown.c:451 > > #3  0xffffffff80b83f33 in vpanic (fmt=, > > ap=) > >      at ../../../kern/kern_shutdown.c:880 > > #4  0xffffffff80b83d53 in panic (fmt=) > >      at ../../../kern/kern_shutdown.c:807 > > #5  0xffffffff80be7152 in propagate_priority > > (td=0xfffff80196959740) > >      at ../../../kern/subr_turnstile.c:228 > > #6  0xffffffff80be7d1c in turnstile_wait (ts=0xfffff8000357a780, > >      owner=, queue=0) at > > ../../../kern/subr_turnstile.c:785 > > #7  0xffffffff80b62ca9 in __mtx_lock_sleep (c=0xfffff80003969bd0, > >      v=) at ../../../kern/kern_mutex.c:654 > > #8  0xffffffff8086fcc0 in ppcintr (arg=0xfffff80003969b00) > >      at ../../../dev/ppc/ppc.c:1546 > > #9  0xffffffff80b4637c in intr_event_execute_handlers (p= > out>, > >      ie=0xfffff800030d9d00) at ../../../kern/kern_intr.c:1143 > > #10 ithread_execute_handlers (p=, > > ie=0xfffff800030d9d00) > >      at ../../../kern/kern_intr.c:1156 > > #11 ithread_loop (arg=0xfffff800039aa7e0) at > > ../../../kern/kern_intr.c:1236 > > #12 0xffffffff80b42e1e in fork_exit ( > >      callout=0xffffffff80b46140 , > > arg=0xfffff800039aa7e0, > >      frame=0xfffffe005ab73d40) at ../../../kern/kern_fork.c:1080 > > #13 > > (kgdb) > >