From owner-freebsd-current@freebsd.org Sat Oct 17 19:17:58 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5924444023B for ; Sat, 17 Oct 2020 19:17:58 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (troutmask.apl.washington.edu [128.95.76.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "troutmask", Issuer "troutmask" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CDCTd4pmFz4Wwp for ; Sat, 17 Oct 2020 19:17:57 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (localhost [127.0.0.1]) by troutmask.apl.washington.edu (8.16.1/8.16.1) with ESMTPS id 09HJHtGg083221 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 17 Oct 2020 12:17:55 -0700 (PDT) (envelope-from sgk@troutmask.apl.washington.edu) Received: (from sgk@localhost) by troutmask.apl.washington.edu (8.16.1/8.16.1/Submit) id 09HJHsxZ083220; Sat, 17 Oct 2020 12:17:54 -0700 (PDT) (envelope-from sgk) Date: Sat, 17 Oct 2020 12:17:54 -0700 From: Steve Kargl To: Warner Losh Cc: "Andrey V. Elsukov" , FreeBSD Current Subject: Re: kernel panic and fun debugging Message-ID: <20201017191754.GB83033@troutmask.apl.washington.edu> References: <20201015065632.GA72261@troutmask.apl.washington.edu> <3fdb18c1-5dc3-2e06-8825-db1daa425afb@yandex.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 4CDCTd4pmFz4Wwp X-Spamd-Bar: - X-Spamd-Result: default: False [-1.85 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; DMARC_POLICY_SOFTFAIL(0.10)[washington.edu : No valid SPF, No valid DKIM,none]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_MEDIUM(-0.88)[-0.875]; NEURAL_HAM_LONG(-0.91)[-0.910]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-0.06)[-0.063]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:73, ipnet:128.95.0.0/16, country:US]; MAILMAN_DEST(0.00)[freebsd-current]; FREEMAIL_CC(0.00)[yandex.ru,freebsd.org] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Oct 2020 19:17:58 -0000 On Sat, Oct 17, 2020 at 12:58:48PM -0600, Warner Losh wrote: > On Sat, Oct 17, 2020, 12:00 PM Andrey V. Elsukov wrote: > > > On 15.10.2020 09:56, Steve Kargl wrote: > > > Just had a kernel panic. Best info I give you is > > > > > > % uname -a > > > FreeBSD mobile 13.0-CURRENT FreeBSD 13.0-CURRENT #1 r366176M: Sat Sep 26 > > 10:35:23 PDT 2020 kargl@mobile:/usr/obj/usr/src/i386.i386/sys/MOBILE > > i386 > > > > > > % kgdb gdb /usr/lib/debug/boot/kernel/kernel.debug vmcore.1 > > > ... > > > Reading symbols from /usr/lib/debug/boot/kernel/kernel.debug... > > > /usr/ports/devel/gdb/work-py37/gdb-9.2/gdb/inferior.c:283: > > internal-error: struct inferior *find_inferior_pid(int): Assertion `pid != > > 0' failed. > > > A problem internal to GDB has been detected, > > > further debugging may prove unreliable. > > > > Hi, > > > > do you have /var/crash/core.txt.1 file? > > It may have some useful info. Also did you try an old version > > /usr/libexec/kgdb ? > > > > I got this same error, btw, when I pointed kgdb at the wrong kernel for the > core file. > > But you can 'more' the uncompressed vmcore to get at least a traceback if > the proper kernel is gone... > > Warner > There are only 2 kernel.debug on the system. % locate kernel.debug /usr/lib/debug/boot/kernel/kernel.debug /usr/lib/debug/boot/kernel.old/kernel.debug Using % kgdb /usr/lib/debug/boot/kernel/kernel.debug vmcore.1 % kgdb /usr/lib/debug/boot/kernel.old/kernel.debug vmcore.1 Both have the same result. I didn't realize the panic text was recorded int vmcore.1. If I extracted everything correctly, here's what happen WARNING !drm_modeset_is_locked(&crtc->mutex) failed at /usr/ports/graphics/drm-current-kmod/work/drm-kmod-drm_v5.4.62_1/drivers/gpu/drm/drm_atomic_helper.c:621 #0 0x2318d4b9 at linux_dump_stack+0x19 #1 0x23219ec2 at drm_atomic_helper_check_modeset+0x92 #2 0x23091d90 at intel_atomic_check+0x70 #3 0x2321913e at drm_atomic_check_only+0x38e #4 0x23219461 at drm_atomic_commit+0x11 #5 0x23224793 at drm_client_modeset_commit_atomic+0xb3 #6 0x2322458e at drm_client_modeset_commit_force+0x5e #7 0x23260851 at drm_fb_helper_restore_fbdev_mode_unlocked+0x71 #8 0x2325b092 at vt_kms_postswitch+0x132 #9 0xa8bec5 at vt_fb_postswitch+0x15 #10 0xa9153d at vt_window_switch+0xfd #11 0xa8f59c at vtterm_cngrab+0x1c #12 0xbec5bf at termcn_cngrab+0xf #13 0xb4b6f6 at cngrab+0x16 #14 0xb9fce2 at vpanic+0xd2 #15 0xb9fc04 at panic+0x14 #16 0xe2c06a at vm_fault_lookup+0x13a #17 0xe2b8e7 at vm_fault+0x77 WARNING !drm_modeset_is_locked(&crtc->mutex) failed at /usr/ports/graphics/drm-current-kmod/work/drm-kmod-drm_v5.4.62_1/drivers/gpu/drm/drm_atomic_helper.c:621 #0 0x2318d4b9 at linux_dump_stack+0x19 #1 0x23219ec2 at drm_atomic_helper_check_modeset+0x92 #2 0x23091d90 at intel_atomic_check+0x70 #3 0x2321913e at drm_atomic_check_only+0x38e #4 0x23219461 at drm_atomic_commit+0x11 #5 0x23224793 at drm_client_modeset_commit_atomic+0xb3 #6 0x2322458e at drm_client_modeset_commit_force+0x5e #7 0x23260851 at drm_fb_helper_restore_fbdev_mode_unlocked+0x71 #8 0x2325b092 at vt_kms_postswitch+0x132 #9 0xa8bec5 at vt_fb_postswitch+0x15 #10 0xa9153d at vt_window_switch+0xfd #11 0xa8f59c at vtterm_cngrab+0x1c #12 0xbec5bf at termcn_cngrab+0xf #13 0xb4b6f6 at cngrab+0x16 #14 0xb9fce2 at vpanic+0xd2 #15 0xb9fc04 at panic+0x14 #16 0xe2c06a at vm_fault_lookup+0x13a #17 0xe2b8e7 at vm_fault+0x77 WARNING !drm_modeset_is_locked(&dev->mode_config.connection_mutex) failed at /usr/ports/graphics/drm-current-kmod/work/drm-kmod-drm_v5.4.62_1/drivers/gpu/drm/drm_atomic_helper.c:666 #0 0x2318d4b9 at linux_dump_stack+0x19 #1 0x2321a005 at drm_atomic_helper_check_modeset+0x1d5 #2 0x23091d90 at intel_atomic_check+0x70 #3 0x2321913e at drm_atomic_check_only+0x38e #4 0x23219461 at drm_atomic_commit+0x11 #5 0x23224793 at drm_client_modeset_commit_atomic+0xb3 #6 0x2322458e at drm_client_modeset_commit_force+0x5e #7 0x23260851 at drm_fb_helper_restore_fbdev_mode_unlocked+0x71 #8 0x2325b092 at vt_kms_postswitch+0x132 #9 0xa8bec5 at vt_fb_postswitch+0x15 #10 0xa9153d at vt_window_switch+0xfd #11 0xa8f59c at vtterm_cngrab+0x1c #12 0xbec5bf at termcn_cngrab+0xf #13 0xb4b6f6 at cngrab+0x16 #14 0xb9fce2 at vpanic+0xd2 #15 0xb9fc04 at panic+0x14 #16 0xe2c06a at vm_fault_lookup+0x13a #17 0xe2b8e7 at vm_fault+0x77 WARNING !drm_modeset_is_locked(&plane->mutex) failed at /usr/ports/graphics/drm-current-kmod/work/drm-kmod-drm_v5.4.62_1/drivers/gpu/drm/drm_atomic_helper.c:871 #0 0x2318d4b9 at linux_dump_stack+0x19 #1 0x2321adbd at drm_atomic_helper_check_planes+0x8d #2 0x23092da7 at intel_atomic_check+0x1087 #3 0x2321913e at drm_atomic_check_only+0x38e #4 0x23219461 at drm_atomic_commit+0x11 #5 0x23224793 at drm_client_modeset_commit_atomic+0xb3 #6 0x2322458e at drm_client_modeset_commit_force+0x5e #7 0x23260851 at drm_fb_helper_restore_fbdev_mode_unlocked+0x71 #8 0x2325b092 at vt_kms_postswitch+0x132 #9 0xa8bec5 at vt_fb_postswitch+0x15 #10 0xa9153d at vt_window_switch+0xfd #11 0xa8f59c at vtterm_cngrab+0x1c #12 0xbec5bf at termcn_cngrab+0xf #13 0xb4b6f6 at cngrab+0x16 #14 0xb9fce2 at vpanic+0xd2 #15 0xb9fc04 at panic+0x14 #16 0xe2c06a at vm_fault_lookup+0x13a #17 0xe2b8e7 at vm_fault+0x77 WARNING !drm_modeset_is_locked(&plane->mutex) failed at /usr/ports/graphics/drm-current-kmod/work/drm-kmod-drm_v5.4.62_1/drivers/gpu/drm/drm_atomic_helper.c:871 #0 0x2318d4b9 at linux_dump_stack+0x19 #1 0x2321adbd at drm_atomic_helper_check_planes+0x8d #2 0x23092da7 at intel_atomic_check+0x1087 #3 0x2321913e at drm_atomic_check_only+0x38e #4 0x23219461 at drm_atomic_commit+0x11 #5 0x23224793 at drm_client_modeset_commit_atomic+0xb3 #6 0x2322458e at drm_client_modeset_commit_force+0x5e #7 0x23260851 at drm_fb_helper_restore_fbdev_mode_unlocked+0x71 #8 0x2325b092 at vt_kms_postswitch+0x132 #9 0xa8bec5 at vt_fb_postswitch+0x15 #10 0xa9153d at vt_window_switch+0xfd #11 0xa8f59c at vtterm_cngrab+0x1c #12 0xbec5bf at termcn_cngrab+0xf #13 0xb4b6f6 at cngrab+0x16 #14 0xb9fce2 at vpanic+0xd2 #15 0xb9fc04 at panic+0x14 #16 0xe2c06a at vm_fault_lookup+0x13a #17 0xe2b8e7 at vm_fault+0x77 WARNING !drm_modeset_is_locked(&plane->mutex) failed at /usr/ports/graphics/drm-current-kmod/work/drm-kmod-drm_v5.4.62_1/drivers/gpu/drm/drm_atomic_helper.c:871 #0 0x2318d4b9 at linux_dump_stack+0x19 #1 0x2321adbd at drm_atomic_helper_check_planes+0x8d #2 0x23092da7 at intel_atomic_check+0x1087 #3 0x2321913e at drm_atomic_check_only+0x38e #4 0x23219461 at drm_atomic_commit+0x11 #5 0x23224793 at drm_client_modeset_commit_atomic+0xb3 #6 0x2322458e at drm_client_modeset_commit_force+0x5e #7 0x23260851 at drm_fb_helper_restore_fbdev_mode_unlocked+0x71 #8 0x2325b092 at vt_kms_postswitch+0x132 #9 0xa8bec5 at vt_fb_postswitch+0x15 #10 0xa9153d at vt_window_switch+0xfd #11 0xa8f59c at vtterm_cngrab+0x1c #12 0xbec5bf at termcn_cngrab+0xf #13 0xb4b6f6 at cngrab+0x16 #14 0xb9fce2 at vpanic+0xd2 #15 0xb9fc04 at panic+0x14 #16 0xe2c06a at vm_fault_lookup+0x13a #17 0xe2b8e7 at vm_fault+0x77 WARNING !drm_modeset_is_locked(&plane->mutex) failed at /usr/ports/graphics/drm-current-kmod/work/drm-kmod-drm_v5.4.62_1/drivers/gpu/drm/drm_atomic_helper.c:871 #0 0x2318d4b9 at linux_dump_stack+0x19 #1 0x2321adbd at drm_atomic_helper_check_planes+0x8d #2 0x23092da7 at intel_atomic_check+0x1087 #3 0x2321913e at drm_atomic_check_only+0x38e #4 0x23219461 at drm_atomic_commit+0x11 #5 0x23224793 at drm_client_modeset_commit_atomic+0xb3 #6 0x2322458e at drm_client_modeset_commit_force+0x5e #7 0x23260851 at drm_fb_helper_restore_fbdev_mode_unlocked+0x71 #8 0x2325b092 at vt_kms_postswitch+0x132 #9 0xa8bec5 at vt_fb_postswitch+0x15 #10 0xa9153d at vt_window_switch+0xfd #11 0xa8f59c at vtterm_cngrab+0x1c #12 0xbec5bf at termcn_cngrab+0xf #13 0xb4b6f6 at cngrab+0x16 #14 0xb9fce2 at vpanic+0xd2 #15 0xb9fc04 at panic+0x14 #16 0xe2c06a at vm_fault_lookup+0x13a #17 0xe2b8e7 at vm_fault+0x77 <4>WARN_ON(!mutex_is_locked(&dev->struct_mutex)) <4>WARN_ON(!mutex_is_locked(&fbc->lock))WARN_ON(!mutex_is_locked(&fbc->lock)) panic: vm_fault_lookup: fault on nofault entry, addr: 0 cpuid = 1 time = 1602744240 KDB: stack backtrace: db_trace_self_wrapper(1,22b9e8b4,9c8f69,fdabf8,12742000,...) at db_trace_self_wrapper+0x28/frame 0x22b9e898 kdb_backtrace(22c28700,1,121ad98,22b9e960,22b9e9a0,...) at kdb_backtrace+0x2b/frame 0x22b9e8f0 vpanic(f2d22a,22b9e92c,22b9e92c,22b9e950,e2c06a,...) at vpanic+0x130/frame 0x22b9e90c panic(f2d22a,f1fec9,0,22b9e974,87a2328,...) at panic+0x14/frame 0x22b9e920 vm_fault_lookup(ffffffff,0,0,0,701,...) at vm_fault_lookup+0x13a/frame 0x22b9e950 vm_fault(121ad98,0,1,0,0) at vm_fault+0x77/frame 0x22b9e9d0 vm_fault_trap(121ad98,0,1,0,0,0) at vm_fault_trap+0x44/frame 0x22b9e9f8 trap_pfault(0,0,0) at trap_pfault+0x119/frame 0x22b9ea40 trap(22b9eb08,8,28,28,40,...) at trap+0x2fb/frame 0x22b9eafc calltrap() at 0xffc0319f/frame 0x22b9eafc --- trap 0xc, eip = 0x231a72c4, esp = 0x22b9eb48, ebp = 0x22b9eb48 --- __linux_clflushopt(0,90000,0,3bea6000,2,...) at __linux_clflushopt+0x24/frame 0x22b9eb48 drm_clflush_sg(34cca3c0,34992dc0,22b9ebb0,22b9eb8c,ba82b8,...) at drm_clflush_sg+0x129/frame 0x22b9eb70 shmem_put_pages(34dab8c0,34cca3c0,0,0,22fe6038,...) at shmem_put_pages+0x3f/frame 0x22b9ebd0 __i915_gem_object_put_pages(34dab8c0,0,0,22c28700,22fe6038,...) at __i915_gem_object_put_pages+0x9d/frame 0x22b9ebf4 i915_gem_flush_free_objects(22fe6000,22b9ec5c,231981bd,22fe87bc,885aec0,...) at i915_gem_flush_free_objects+0x11e/frame 0x22b9ec20 __i915_gem_free_work(22fe87bc,885aec0,0,885aed8,0,...) at __i915_gem_free_work+0x11/frame 0x22b9ec2c linux_work_fn(22fe87bc,1) at linux_work_fn+0xed/frame 0x22b9ec5c taskqueue_run_locked(0,22c28700,22b9ecd4,b67a57,885aec0,...) at taskqueue_run_locked+0x1c4/frame 0x22b9eca8 taskqueue_thread_loop(885aec0,22b9ece8) at taskqueue_thread_loop+0x9e/frame 0x22b9ecb8 fork_exit(beb5d0,885aec0,22b9ece8,0,0,...) at fork_exit+0x67/frame 0x22b9ecd4 fork_trampoline() at 0xffc0340e/frame 0x22b9ecd4 --- kthread start KDB: enter: panic -- Steve