From owner-freebsd-current@freebsd.org Sat May 9 02:40:13 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D6C382D6BA4 for ; Sat, 9 May 2020 02:40:13 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 49Jryj24bMz44yC for ; Sat, 9 May 2020 02:40:13 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 473C72D6BA1; Sat, 9 May 2020 02:40:13 +0000 (UTC) Delivered-To: current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 46F9F2D6BA0 for ; Sat, 9 May 2020 02:40:13 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-oi1-f193.google.com (mail-oi1-f193.google.com [209.85.167.193]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49Jryh1thGz44y9; Sat, 9 May 2020 02:40:12 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-oi1-f193.google.com with SMTP id j16so10236324oih.10; Fri, 08 May 2020 19:40:12 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=etPzV5kngpNyANxLWW19MnkrIanFROgy85om3UIAwi8=; b=aFmlDikN7GMknOEnjn3OO8K3Zws0k+vDbeLkx3Ai+zV9zaqFZ/AM2UJcedQDUjNewW GcmwdA0J+xUOo0oOUXyf1WxHiaHeWF1ZMlE6XuOreSSKeazB3v89eg0I9xy8lOIKc5J3 Vgb//artbzs1lnW3NU7+DPj1BWL7uTVcaQ24QH28ceftDPyTOr+jmiZKwINvXd5utsnR SYG2Qprz8pAbfBRfXva8RO02HVTHVqPqJqV0RHZI98viIp6oZhdDV+DrWIfgRWQ46RYs q/rAQPn7QimftKWARZ5EnhwEFc4P9FjRFpiqYyX9Q/aQfPsnKutKUEJq6JO2iHaR37ee phlw== X-Gm-Message-State: AGi0Pubgf5prJYUB2M0R7MznrFh1kgEp12cJNk3TL82/8DhPB5rpSvCB KwPF7663GGij8rk5/owiFdA+/x9a31kOBe6NzSI= X-Google-Smtp-Source: APiQypJMExqB/I9p7WaSX8+lE3fqiaiwzqjoXGIVLN/YBsBAsrDmr0+5ZyKhn/v/biJvIOgNIrPcjtWum//yfbJsyP0= X-Received: by 2002:aca:4f4b:: with SMTP id d72mr12597542oib.73.1588992010775; Fri, 08 May 2020 19:40:10 -0700 (PDT) MIME-Version: 1.0 References: <20200427105124.GP2522@kib.kiev.ua> In-Reply-To: <20200427105124.GP2522@kib.kiev.ua> From: Alan Somers Date: Fri, 8 May 2020 20:39:59 -0600 Message-ID: Subject: Re: Panic in fusefs(5) on 13.0-CURRENT r359773 To: Konstantin Belousov Cc: Mateusz Piotrowski <0mp@freebsd.org>, current X-Rspamd-Queue-Id: 49Jryh1thGz44y9 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.167.193 as permitted sender) smtp.mailfrom=asomers@gmail.com X-Spamd-Result: default: False [-0.97 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.90)[-0.901,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-0.90)[-0.899,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MIME_TRACE(0.00)[0:+,1:+,2:~]; DMARC_NA(0.00)[freebsd.org]; URI_COUNT_ODD(1.00)[9]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[193.167.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-0.17)[ip: (0.00), ipnet: 209.85.128.0/17(-0.39), asn: 15169(-0.43), country: US(-0.05)]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; FREEMAIL_TO(0.00)[gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[193.167.85.209.rep.mailspike.net : 127.0.0.17]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.32 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.32 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 May 2020 02:40:13 -0000 On Mon, Apr 27, 2020 at 4:51 AM Konstantin Belousov wrote: > On Mon, Apr 27, 2020 at 11:39:41AM +0200, Mateusz Piotrowski wrote: > > Hi everyone! > > > > I'm experiencing panics every day. Apparently, they are caused by a bug > in > > our FUSE implementation. > > > > The panic occurs when I'm editing files in a folder mounted via SSHFS > from a > > bhyve VM running Ubuntu 18.04. > > > > I'm sending some parts of the core.txt file. Let me know if I can provide > > any additional information. > > > > Cheers, > > > > Mateusz > > > > ------- > > > > t480 dumped core - see /var/crash/vmcore.3 > > > > Mon Apr 27 11:30:19 CEST 2020 > > > > FreeBSD t480 13.0-CURRENT FreeBSD 13.0-CURRENT #4 r359773: Sat Apr 11 > > 05:32:31 CEST 2020 root@t480:/usr/obj/usr/src/amd64.amd64/sys/GENERIC > amd64 > > > > panic: Assertion bp->b_flags & B_VMIO failed at > > /usr/src/sys/fs/fuse/fuse_node.c:433 > > > > GNU gdb (GDB) 9.1 [GDB v9.1 for FreeBSD] > > Copyright (C) 2020 Free Software Foundation, Inc. > > License GPLv3+: GNU GPL version 3 or later > > > > This is free software: you are free to change and redistribute it. > > There is NO WARRANTY, to the extent permitted by law. > > Type "show copying" and "show warranty" for details. > > This GDB was configured as "x86_64-portbld-freebsd13.0". > > Type "show configuration" for configuration details. > > For bug reporting instructions, please see: > > . > > Find the GDB manual and other documentation resources online at: > > . > > > > For help, type "help". > > Type "apropos word" to search for commands related to "word"... > > Reading symbols from /boot/kernel/kernel... > > Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug... > > > > Unread portion of the kernel message buffer: > > WARNING !drm_modeset_is_locked(&crtc->mutex) failed at > /usr/local/sys/modules/drm-current-kmod/drivers/gpu/drm/drm_atomic_helper.c:577 > > WARNING !drm_modeset_is_locked(&crtc->mutex) failed at > /usr/local/sys/modules/drm-current-kmod/drivers/gpu/drm/drm_atomic_helper.c:577 > > WARNING !drm_modeset_is_locked(&crtc->mutex) failed at > /usr/local/sys/modules/drm-current-kmod/drivers/gpu/drm/drm_atomic_helper.c:577 > > WARNING !drm_modeset_is_locked(&dev->mode_config.connection_mutex) > failed at > /usr/local/sys/modules/drm-current-kmod/drivers/gpu/drm/drm_atomic_helper.c:622 > > WARNING !drm_modeset_is_locked(&plane->mutex) failed at > /usr/local/sys/modules/drm-current-kmod/drivers/gpu/drm/drm_atomic_helper.c:821 > > WARNING !drm_modeset_is_locked(&plane->mutex) failed at > /usr/local/sys/modules/drm-current-kmod/drivers/gpu/drm/drm_atomic_helper.c:821 > > WARNING !drm_modeset_is_locked(&plane->mutex) failed at > /usr/local/sys/modules/drm-current-kmod/drivers/gpu/drm/drm_atomic_helper.c:821 > > WARNING !drm_modeset_is_locked(&plane->mutex) failed at > /usr/local/sys/modules/drm-current-kmod/drivers/gpu/drm/drm_atomic_helper.c:821 > > WARNING !drm_modeset_is_locked(&plane->mutex) failed at > /usr/local/sys/modules/drm-current-kmod/drivers/gpu/drm/drm_atomic_helper.c:821 > > WARNING !drm_modeset_is_locked(&plane->mutex) failed at > /usr/local/sys/modules/drm-current-kmod/drivers/gpu/drm/drm_atomic_helper.c:821 > > WARNING !drm_modeset_is_locked(&plane->mutex) failed at > /usr/local/sys/modules/drm-current-kmod/drivers/gpu/drm/drm_atomic_helper.c:821 > > WARNING !drm_modeset_is_locked(&plane->mutex) failed at > /usr/local/sys/modules/drm-current-kmod/drivers/gpu/drm/drm_atomic_helper.c:821 > > WARNING !drm_modeset_is_locked(&plane->mutex) failed at > /usr/local/sys/modules/drm-current-kmod/drivers/gpu/drm/drm_atomic_helper.c:821 > > <4>WARN_ON(!mutex_is_locked(&dev->struct_mutex)) > > > > <4>WARN_ON(!mutex_is_locked(&fbc->lock)) > > > > > <4>WARN_ON(!mutex_is_locked(&fbc->lock))WARN_ON(!mutex_is_locked(&fbc->lock))WARN_ON(!mutex_is_locked(&fbc->lock)) > > panic: Assertion bp->b_flags & B_VMIO failed at > > /usr/src/sys/fs/fuse/fuse_node.c:433 > > cpuid = 5 > > time = 1587979693 > > KDB: stack backtrace: > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > > 0xfffffe00cc0ef330 > > vpanic() at vpanic+0x182/frame 0xfffffe00cc0ef380 > > panic() at panic+0x43/frame 0xfffffe00cc0ef3e0 > > fuse_vnode_setsize() at fuse_vnode_setsize+0x135/frame 0xfffffe00cc0ef430 > > fuse_internal_cache_attrs() at fuse_internal_cache_attrs+0xc4/frame > > 0xfffffe00cc0ef490 > > fuse_vnop_lookup() at fuse_vnop_lookup+0x6bf/frame 0xfffffe00cc0ef620 > > lookup() at lookup+0x5e1/frame 0xfffffe00cc0ef6c0 > > namei() at namei+0x524/frame 0xfffffe00cc0ef7b0 > > kern_statat() at kern_statat+0x7f/frame 0xfffffe00cc0ef8d0 > > sys_fstatat() at sys_fstatat+0x2f/frame 0xfffffe00cc0ef9d0 > > amd64_syscall() at amd64_syscall+0x140/frame 0xfffffe00cc0efaf0 > > fast_syscall_common() at fast_syscall_common+0x101/frame > 0xfffffe00cc0efaf0 > > --- syscall (552, FreeBSD ELF64, sys_fstatat), rip = 0x8006c2aaa, rsp = > > 0x7fffffffda78, rbp = 0x7fffffffdb20 --- > > KDB: enter: panic > > > > __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 > > 55 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct > pcpu, > > (kgdb) #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 > > #1 doadump (textdump=0) at /usr/src/sys/kern/kern_shutdown.c:394 > > #2 0xffffffff8049a43a in db_dump (dummy=, > > dummy2=, dummy3=, dummy4=) > > at /usr/src/sys/ddb/db_command.c:575 > > #3 0xffffffff8049a1fc in db_command (last_cmdp=, > > cmd_table=, dopager=1) at > > /usr/src/sys/ddb/db_command.c:482 > > #4 0xffffffff80499f6d in db_command_loop () > > at /usr/src/sys/ddb/db_command.c:535 > > #5 0xffffffff8049d168 in db_trap (type=, code= > out>) > > at /usr/src/sys/ddb/db_main.c:253 > > #6 0xffffffff80c06f24 in kdb_trap (type=3, code=0, tf=) > > at /usr/src/sys/kern/subr_kdb.c:699 > > #7 0xffffffff8105bc68 in trap (frame=0xfffffe00cc0ef260) > > at /usr/src/sys/amd64/amd64/trap.c:578 > > #8 > > #9 kdb_enter (why=0xffffffff811ea4f2 "panic", msg=) > > at /usr/src/sys/kern/subr_kdb.c:486 > > #10 0xffffffff80bbc9fe in vpanic (fmt=, ap= out>) > > at /usr/src/sys/kern/kern_shutdown.c:902 > > #11 0xffffffff80bbc793 in panic ( > > fmt=0xffffffff81c8db28 "*\350\032\201\377\377\377\377") > > at /usr/src/sys/kern/kern_shutdown.c:839 > > #12 0xffffffff83f0ba05 in fuse_vnode_setsize (vp=0xfffff801e94515b8, > > newsize=13793) at /usr/src/sys/fs/fuse/fuse_node.c:433 > > #13 0xffffffff83f15a74 in fuse_internal_cache_attrs > (vp=0xfffff801e94515b8, > > attr=0xfffffe012ba5b028, attr_valid=1, attr_valid_nsec=0, vap=0x0) > > at /usr/src/sys/fs/fuse/fuse_internal.c:267 > > #14 0xffffffff83f1346e in fuse_vnop_lookup (ap=) > > at /usr/src/sys/fs/fuse/fuse_vnops.c:1187 > > #15 0xffffffff80c873b1 in VOP_LOOKUP (dvp=0xfffff801e9451988, > > vpp=0xfffffe00cc0ef820, cnp=0xfffffe00cc0ef850) at ./vnode_if.h:54 > > #16 lookup (ndp=0xfffffe00cc0ef7c0) at /usr/src/sys/kern/vfs_lookup.c:951 > > #17 0xffffffff80c868e4 in namei (ndp=0xfffffe00cc0ef7c0) > > at /usr/src/sys/kern/vfs_lookup.c:512 > > #18 0xffffffff80ca210f in kern_statat (td=0xfffffe00cc61f800, > > flag=, fd=, > > path=0x800c9ff50 0x800c9ff50>, > > pathseg=UIO_USERSPACE, sbp=0xfffffe00cc0ef8e8, hook=0x0) > > at /usr/src/sys/kern/vfs_syscalls.c:2340 > > #19 0xffffffff80ca28cf in sys_fstatat (td=0xffffffff81c8db28 > , > > uap=0xfffffe00cc61fbd8) at /usr/src/sys/kern/vfs_syscalls.c:2317 > > #20 0xffffffff8105caa0 in syscallenter (td=) > > at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:162 > > #21 amd64_syscall (td=0xfffffe00cc61f800, traced=0) > > at /usr/src/sys/amd64/amd64/trap.c:1161 > > #22 > > #23 0x00000008006c2aaa in ?? () > > Backtrace stopped: Cannot access memory at address 0x7fffffffda78 > > (kgdb) > > I believe that in your case, the file is stat(2)-ed without being opened > before, and might be because the stat(2) is called twice, while file size > is changed, fusefs wants to resize. > > For such vnodes, v_object is not created yet, so getblk() sets B_CACHE > because both B_VMIO and B_INVAL are clear. So it might be worth just do > nothing if v_object is NULL even if newsize < oldsize. > > What I do not understand is why vfs_bio_clrbuf() is correct there at all. > It has block granularity, so either we clear too much or too little. > And the data after EOF in the last buffer is not going to be used anyway. > At worst, it might cause userspace which mapped the backing pages to still > see the data after EOF, and if this is a problem, then manual bzero() is > needed anyway. > Two questions: 1) It sounds like this panic happens pretty regularly. Did it ever happen before you updated to r358867 ? 2) Have you found a reliable way to reproduce the panic yet? -Alan