From owner-freebsd-hackers@freebsd.org Thu Apr 12 07:28:00 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A55DAF9C631 for ; Thu, 12 Apr 2018 07:28:00 +0000 (UTC) (envelope-from lists@eitanadler.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 3CBD36C279 for ; Thu, 12 Apr 2018 07:28:00 +0000 (UTC) (envelope-from lists@eitanadler.com) Received: by mailman.ysv.freebsd.org (Postfix) id 01847F9C62C; Thu, 12 Apr 2018 07:28:00 +0000 (UTC) Delivered-To: hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CEA2DF9C62B for ; Thu, 12 Apr 2018 07:27:59 +0000 (UTC) (envelope-from lists@eitanadler.com) Received: from mail-yb0-x235.google.com (mail-yb0-x235.google.com [IPv6:2607:f8b0:4002:c09::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 671456C25C for ; Thu, 12 Apr 2018 07:27:59 +0000 (UTC) (envelope-from lists@eitanadler.com) Received: by mail-yb0-x235.google.com with SMTP id k199-v6so1481338ybk.12 for ; Thu, 12 Apr 2018 00:27:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=eitanadler.com; s=0xdeadbeef; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=icj/FmmeBOnW9FSMcT0pzpu6HBc7M3gY1958t/gkCC0=; b=QleyyEzJwEkwYY2RbR56vLniidWx1TBO0H+pm4jmAtbI4qQc6QcAzpteGYxD0FuM4H zLdowNV7tm+3bVkRqmbiLYWGp2SfO+tlIaZY9s7FB+ERbMji9aubPoVRFnz9a9bEpSZd QJ1ji+so0rrJyf40RJEyXHIoRUivbitge3sqI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=icj/FmmeBOnW9FSMcT0pzpu6HBc7M3gY1958t/gkCC0=; b=AucvdxxZaG7d4CdlxDRMf5YC6gqf5iKgi3KqTmHaACgmDLSqWwix6wqa52RdUIeNDo nBczpP/N4jURzxgSbuefjwJFc/DpYuQXWBUDarZtFmBsqHGT2OLl4GK0D41VzsZcCdew 0trMDv9UqTFYSwuGFiC2MNTPck8hpRE0Qpa8PT3Q8PbQU7fb2c2jex7xlChUBoNu3XYx YRy2VWfF6Q+EQ1eNjVpq4qB8Lheg9Wtr0RmT4MPbg2LRFB6ZL66GgbffHxM0hfzBymh/ K8Gs6ls6lH/I9PfHQpNrSALF3kPECPe+Aj1zkrBAlswWFmbTYNaCbO4L0T+AmQKcCwN9 EDrQ== X-Gm-Message-State: ALQs6tDq2G5/9gkndHjHd5Fbzbohp0OiBQa2naqbFHh+OmrRw09GbECu Ibotnksc79cduBMBS1Tc8opV3/dOHzUPYQwFS3Z7RA== X-Google-Smtp-Source: AIpwx4/kxKPogVCY05BmFiCJ5u/eeeLMgdS+dG611bryG0OIDYH2a84f1TIJT1o0rFJn5yl6OxBjl09hTX22vxjJcqA= X-Received: by 2002:a25:268f:: with SMTP id m137-v6mr3484245ybm.460.1523518078608; Thu, 12 Apr 2018 00:27:58 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a25:98c3:0:0:0:0:0 with HTTP; Thu, 12 Apr 2018 00:27:28 -0700 (PDT) In-Reply-To: <33e482b2-ee50-bd72-51d6-bb320cc95b82@FreeBSD.org> References: <33e482b2-ee50-bd72-51d6-bb320cc95b82@FreeBSD.org> From: Eitan Adler Date: Thu, 12 Apr 2018 00:27:28 -0700 Message-ID: Subject: Re: panic: deadlkres: possible deadlock detected for 0xfffff80141b04560, blocked for 1801695 ticks To: Andriy Gapon Cc: "hackers@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Apr 2018 07:28:01 -0000 On 11 April 2018 at 23:11, Andriy Gapon wrote: > On 12/04/2018 07:00, Eitan Adler wrote: >> How to reproduce? >> >> # kldload sem >> # kldunload sem >> < wait debug.deadlkres.slptime_threshold seconds > >> >> https://reviews.freebsd.org/P168 >> Reading symbols from ./kernel/kernel...Reading symbols from >> /usr/home/eax/crashes/sem_load_dklres/kernel/kernel.debug...done. >> done. >> >> Unread portion of the kernel message buffer: >> [29474] panic: deadlkres: possible deadlock detected for >> 0xfffff80141b04560, blocked for 1801695 ticks > > The debugging information you provided does not help. > You need to do > (gdb) p *(struct thread*)0xfffff80141b04560 <=== taken from the panic message > find td_tid value and then do > (gdb) tid > And then examine that thread. > > I guess that the deadlkres could be enhanced to print the tid and the stack > trace of the deadlocked thread, so that it is faster to get started. (kgdb) p (*(struct thread*)0xfffff80141b04560).td_tid $2 = 0x18bd8 (kgdb) tid 18bd8 (kgdb) bt #0 sched_switch (td=0xfffff80141b04560, newtd=0xfffff80003699000, flags=) at /usr/src/sys/kern/sched_ule.c:2115 #1 0xffffffff80b7156c in mi_switch (flags=0x104, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:437 #2 0xffffffff80bba7ed in sleepq_switch (wchan=0xffffffff81deb2d8 , pri=0x0) at /usr/src/sys/kern/subr_sleepqueue.c:613 #3 0xffffffff80bba693 in sleepq_wait (wchan=0xffffffff81deb2d8 , pri=0x0) at /usr/src/sys/kern/subr_sleepqueue.c:692 #4 0xffffffff80b6f329 in _sx_xlock_hard (sx=0xffffffff81deb2d8 , x=, opts=, file=, line=) at /usr/src/sys/kern/kern_sx.c:777 #5 0xffffffff80b6ef31 in _sx_xlock (sx=0xffffffff81deb2d8 , opts=0x0, file=0xffffffff811a5d65 "/usr/src/sys/kern/kern_linker.c", line=0x42e) at /usr/src/sys/kern/kern_sx.c:319 #6 0xffffffff80b3834e in kern_kldload (td=, file=, fileid=0xfffffe00af1f8964) at /usr/src/sys/kern/kern_linker.c:1070 #7 0xffffffff80b3848b in sys_kldload (td=0xfffff80141b04560, uap=) at /usr/src/sys/kern/kern_linker.c:1097 #8 0xffffffff8102606b in syscallenter (td=0xfffff80141b04560) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:134 #9 amd64_syscall (td=0xfffff80141b04560, traced=0x0) at /usr/src/sys/amd64/amd64/trap.c:936 #10 #11 0x00000008002cfd8a in ?? () Backtrace stopped: Cannot access memory at address 0x7fffffffd478 (kgdb) frame Stack level 3, frame at 0xfffffe00af1f8820: rip = 0xffffffff80bba693 in sleepq_wait (/usr/src/sys/kern/subr_sleepqueue.c:692); saved rip = 0xffffffff80b6f329 called by frame at 0xfffffe00af1f88d0, caller of frame at 0xfffffe00af1f87f0 source language c. Arglist at 0xfffffe00af1f8810, args: wchan=0xffffffff81deb2d8 , pri=0x0 Locals at 0xfffffe00af1f8810, Previous frame's sp is 0xfffffe00af1f8820 Saved registers: rbx at 0xfffffe00af1f87f8, rbp at 0xfffffe00af1f8810, r14 at 0xfffffe00af1f8800, r15 at 0xfffffe00af1f8808, rip at 0xfffffe00af1f8818 wchan = 0xffffffff81deb2d8 pri = 0x0 td = 0xfffff80141b04560 Updated paste with more info as well -- Eitan Adler