From owner-freebsd-hackers@freebsd.org Tue Jun 11 20:12:37 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DBEBA15C5A72 for ; Tue, 11 Jun 2019 20:12:36 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-lj1-f178.google.com (mail-lj1-f178.google.com [209.85.208.178]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0CB177637A for ; Tue, 11 Jun 2019 20:12:35 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-lj1-f178.google.com with SMTP id h10so7063273ljg.0 for ; Tue, 11 Jun 2019 13:12:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=aj4Sb4OSVtCIUgi5ykPzbFS/iEFCxUoroEPEdGQRqnQ=; b=fDOi2s8mhbyCUoGFwvGb5O+oci9dkgxXcVtjkqmOn2lVIVYAEmTY9eK37xAmRiGOvN cXFEUMgXgeO/yNU0s92aJNnyaanlYUijO8FNBlKdAVahtyqp0ZtGLE6qpPvYOoNtYALI iU629YfCbdxTUTfY2WEo9aXDujOPw2L+eFtCd0iXCxSR0KbNNkLd+p59W68XC9UDpxsn 9mg+34EyKHJrpKqvKmZLapdysc7Ga3+isTNpgk2EwQwUpvzby0KEnVK0e2sjxu4SCi+W V2Eyp0/cPhNiyzki1MMQ73D+Be2z7cZYtAb+ScHk1a3Spw0Svr3zSbeVwadXEe9HKvwS kNlw== X-Gm-Message-State: APjAAAVCThx1CtX75p02FpeHASqJX3XUPubt/bdOOc+GDXL3eFj5I09+ OAdBQsyRajgnEswhTl3aQoIToHhD44Z067YbCuhetOMP X-Google-Smtp-Source: APXvYqyh/9m/tHqmRpLXD6fMkUvN7eDbXRyFHEFNKPFk/4b/0GQNjoWeK9n+3i3GXDVqYUvn1D/2U7twxTmpfbVPOlY= X-Received: by 2002:a2e:12dc:: with SMTP id 89mr17883261ljs.40.1560283953759; Tue, 11 Jun 2019 13:12:33 -0700 (PDT) MIME-Version: 1.0 From: Alan Somers Date: Tue, 11 Jun 2019 14:12:22 -0600 Message-ID: Subject: panic: vm_fault_hold: fault on nofault entry in fusefs To: FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 0CB177637A X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.208.178 as permitted sender) smtp.mailfrom=asomers@gmail.com X-Spamd-Result: default: False [-4.02 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_TRACE(0.00)[0:+]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.76)[-0.757,0]; RCVD_IN_DNSWL_NONE(0.00)[178.208.85.209.list.dnswl.org : 127.0.5.0]; RCVD_TLS_LAST(0.00)[]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; IP_SCORE(-1.25)[ip: (-0.50), ipnet: 209.85.128.0/17(-3.41), asn: 15169(-2.30), country: US(-0.06)]; TO_DOM_EQ_FROM_DOM(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Jun 2019 20:12:37 -0000 Can somebody please help me to debug a fusefs problem? I have a 100% reproducible panic with the above message. Evidentially there's something I don't know about buf(9) and uiomove(9). The good news is that the panic is sufficiently reproducible and sufficiently instrumented that I know exactly what's happening; I just don't know why. Here's a summary of what happens. 1) fusefs's VOP_WRITE method gets called with a buffer that spans a logical block boundary, but does not extend the size of the file. 2) It splits the write into two parts. Each one calls getblk to allocate a struct buf, fills in the old data with a read, and fills the new data with uiomove. 3) After the file gets close()ed, VOP_INACTIVE calls vn_fsync_buf to flush dirty buffers. 4) VOP_STRATEGY successfully writes the first buffer and frees it with bufdone(). 5) VOP_STRATEGY tries to write the second buffer, but panics during uiomove. The address that caused the panic is always exactly 4KB into the buffer. So what am I doing wrong? The address that causes the panic in step 5 was successfully accessed in step 2, so this isn't some kind of buffer overrun. Does it have something to do with the fact that the read operation in step 2 called bufdone()? Seems unlikely because it did that for both buffers, yet only the second one panics. Or does the address actually fault during both VOP_WRITE and VOP_STRATEGY, but something low down handles the fault in the first case? I'd be grateful for any help that anyone can offer. -Alan P.S. Here's the panic's stack panic: vm_fault_hold: fault on nofault entry, addr: 0xfffffe0004591000 cpuid = 1 time = 1560283621 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0031c21f80 vpanic() at vpanic+0x19d/frame 0xfffffe0031c21fd0 panic() at panic+0x43/frame 0xfffffe0031c22030 vm_fault_hold() at vm_fault_hold+0x2064/frame 0xfffffe0031c22170 vm_fault() at vm_fault+0x60/frame 0xfffffe0031c221b0 trap_pfault() at trap_pfault+0x188/frame 0xfffffe0031c22200 trap() at trap+0x2b4/frame 0xfffffe0031c22310 calltrap() at calltrap+0x8/frame 0xfffffe0031c22310 --- trap 0xc, rip = 0xffffffff8108c9e6, rsp = 0xfffffe0031c223e0, rbp = 0xfffffe0031c223e0 --- memmove_erms() at memmove_erms+0x116/frame 0xfffffe0031c223e0 uiomove_faultflag() at uiomove_faultflag+0x146/frame 0xfffffe0031c22420 fuse_write_directbackend() at fuse_write_directbackend+0x1cd/frame 0xfffffe0031c224f0 fuse_io_strategy() at fuse_io_strategy+0x24d/frame 0xfffffe0031c22590 fuse_vnop_strategy() at fuse_vnop_strategy+0x2a/frame 0xfffffe0031c225a0 VOP_STRATEGY_APV() at VOP_STRATEGY_APV+0x63/frame 0xfffffe0031c225c0 bufstrategy() at bufstrategy+0x44/frame 0xfffffe0031c225f0 bufwrite() at bufwrite+0x259/frame 0xfffffe0031c22640 vn_fsync_buf() at vn_fsync_buf+0x23e/frame 0xfffffe0031c226a0 fuse_vnop_inactive() at fuse_vnop_inactive+0x7e/frame 0xfffffe0031c226e0 VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0x63/frame 0xfffffe0031c22700 vinactive() at vinactive+0xcd/frame 0xfffffe0031c22750 vputx() at vputx+0x2d0/frame 0xfffffe0031c227b0 vn_close1() at vn_close1+0x116/frame 0xfffffe0031c22820 vn_closefile() at vn_closefile+0x4c/frame 0xfffffe0031c228a0 _fdrop() at _fdrop+0x1a/frame 0xfffffe0031c228c0 closef() at closef+0x1ec/frame 0xfffffe0031c22950 closefp() at closefp+0x9c/frame 0xfffffe0031c22990 amd64_syscall() at amd64_syscall+0x276/frame 0xfffffe0031c22ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0031c22ab0 --- syscall (6, FreeBSD ELF64, sys_close), rip = 0x8006842ba, rsp = 0x7fffffffe748, rbp = 0x7fffffffe760 --- KDB: enter: panic