From owner-freebsd-fs@freebsd.org Sun Aug 19 09:42:28 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 657F51088128 for ; Sun, 19 Aug 2018 09:42:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id E1CD78F043 for ; Sun, 19 Aug 2018 09:42:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id A6A1E1088124; Sun, 19 Aug 2018 09:42:27 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 84B201088123 for ; Sun, 19 Aug 2018 09:42:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 10C7D8F03A for ; Sun, 19 Aug 2018 09:42:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 69A59158BF for ; Sun, 19 Aug 2018 09:42:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7J9gQnP021347 for ; Sun, 19 Aug 2018 09:42:26 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7J9gQGv021346 for fs@FreeBSD.org; Sun, 19 Aug 2018 09:42:26 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 227784] zfs: Fatal trap 9: general protection fault while in kernel mode on shutdown Date: Sun, 19 Aug 2018 09:42:25 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: panic X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: wulf@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Aug 2018 09:42:28 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D227784 Vladimir Kondratyev changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |wulf@freebsd.org --- Comment #10 from Vladimir Kondratyev --- (In reply to Andriy Gapon from comment #6) > Do you still have the crash dump? > If so, could you please provide full output of 'p *dd' ? I still observe the panic everyday, so I have a crash dump: (kgdb) frame 10 #10 0xffffffff8035f6dc in dsl_dir_evict_async (dbu=3D0xfffff80006b67400) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dir.c:158 158 spa_async_close(dd->dd_pool->dp_spa, dd); (kgdb) p *dd $7 =3D {dd_dbu =3D {dbu_tqent =3D {tqent_task =3D {ta_link =3D { stqe_next =3D 0xfffff8000689b400}, ta_pending =3D 0, ta_priority = =3D 0,=20 ta_func =3D 0xffffffff802f5410 ,=20 ta_context =3D 0xfffff80006b67400},=20 tqent_func =3D 0xffffffff8035f4e0 ,=20 tqent_arg =3D 0xfffff80006b67400}, dbu_evict_func_sync =3D 0x0,=20 dbu_evict_func_async =3D 0xffffffff8035f4e0 ,=20 dbu_clear_on_evict_dbufp =3D 0xfffff80006b67458}, dd_object =3D 12,=20 dd_pool =3D 0xfffff800066f5800, dd_dbuf =3D 0x0, dd_dirty_link =3D {tn_ne= xt =3D { 0x0, 0x0, 0x0, 0x0}, tn_member =3D "\000\000\000"},=20 dd_parent =3D 0xfffff80006b66c00, dd_lock =3D {lock_object =3D { lo_name =3D 0xffffffff80999c14 "dd->dd_lock", lo_flags =3D 577830912,= =20 lo_data =3D 0, lo_witness =3D 0x0}, sx_lock =3D 1}, dd_props =3D { list_size =3D 56, list_offset =3D 0, list_head =3D { list_next =3D 0xfffff80006b674c0, list_prev =3D 0xfffff80006b674c0}},= =20 dd_snap_cmtime =3D {tv_sec =3D 1534644915, tv_nsec =3D 715064905},=20 dd_origin_txg =3D 0, dd_tempreserved =3D {0, 0, 0, 0}, dd_space_towrite = =3D {0, 0,=20 0, 0}, dd_myname =3D "$ORIGIN", '\000' } (kgdb) printf "%X\n", *(int *)dd->dd_pool DEADC0DE It looks like memory referenced by dd->dd_pool is already freed when spa_async_close() is called. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sun Aug 19 17:22:15 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C120710714E8 for ; Sun, 19 Aug 2018 17:22:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 4986781433 for ; Sun, 19 Aug 2018 17:22:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 0E9AC10714E6; Sun, 19 Aug 2018 17:22:15 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C8C1010714E5 for ; Sun, 19 Aug 2018 17:22:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6A34F8142A for ; Sun, 19 Aug 2018 17:22:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id B0BE519921 for ; Sun, 19 Aug 2018 17:22:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7JHMD5b068852 for ; Sun, 19 Aug 2018 17:22:13 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7JHMDTF068851 for fs@FreeBSD.org; Sun, 19 Aug 2018 17:22:13 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230752] panic: excl->share in newnfs_request Date: Sun, 19 Aug 2018 17:22:13 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: dim@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter attachments.created Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Aug 2018 17:22:16 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230752 Bug ID: 230752 Summary: panic: excl->share in newnfs_request Product: Base System Version: CURRENT Hardware: amd64 OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: fs@FreeBSD.org Reporter: dim@FreeBSD.org Created attachment 196354 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D196354&action= =3Dedit Text dump of newnfs_request -> witness panic I recently got a few similar panics, seemingly originating from newnfs_requ= est. The panic goes like this: shared lock of (lockmgr) ufs @ /usr/src/sys/kern/vfs_lookup.c:671 while exclusively locked from /usr/src/sys/kern/vfs_subr.c:2590 panic: excl->share cpuid =3D 2 time =3D 1534686985 ... #1 doadump (textdump=3D) at /usr/src/sys/kern/kern_shutdown.c:366 #2 0xffffffff8044906b in db_dump (dummy=3D,=20 dummy2=3D, dummy3=3D, dummy4=3D) at /usr/src/sys/ddb/db_command.c:574 #3 0xffffffff80448e39 in db_command (last_cmdp=3D,=20 cmd_table=3D, dopager=3D1) at /usr/src/sys/ddb/db_comman= d.c:481 #4 0xffffffff80448bb4 in db_command_loop () at /usr/src/sys/ddb/db_command.c:534 #5 0xffffffff8044bd7f in db_trap (type=3D, code=3D) at /usr/src/sys/ddb/db_main.c:252 #6 0xffffffff80be79d5 in kdb_trap (type=3D3, code=3D0, tf=3D) at /usr/src/sys/kern/subr_kdb.c:693 #7 0xffffffff81084c3a in trap (frame=3D0xfffffe0000582fc0) at /usr/src/sys/amd64/amd64/trap.c:605 #8 #9 kdb_enter (why=3D0xffffffff8131179d "panic", msg=3D) at /usr/src/sys/kern/subr_kdb.c:479 #10 0xffffffff80b9d7b1 in vpanic (fmt=3D, ap=3D0xfffffe00005= 83130) at /usr/src/sys/kern/kern_shutdown.c:852 #11 0xffffffff80b9d550 in kassert_panic (fmt=3D0xffffffff812a3ed5 "excl->sh= are") at /usr/src/sys/kern/kern_shutdown.c:749 #12 0xffffffff80c083a8 in witness_checkorder (lock=3D0xfffff80004190ba8,=20 flags=3D1, file=3D0xffffffff8130d3f0 "/usr/src/sys/kern/vfs_lookup.c",= =20 line=3D,=20 interlock=3D0xffffffff81f64e2c ) at /usr/src/sys/kern/subr_witness.c:1176 #13 0xffffffff80b70c3e in lockmgr_slock_hard (lk=3D,=20 flags=3D2106368, ilk=3D0xfffff80004190bd8, file=3D,=20 line=3D, lwa=3D) at /usr/src/sys/kern/kern_lock.c:567 #14 0xffffffff80b71d5b in __lockmgr_args (lk=3D,=20 flags=3D, ilk=3D, wmesg=3D= ,=20 pri=3D, timo=3D,=20 file=3D0xffffffff8130d3f0 "/usr/src/sys/kern/vfs_lookup.c", line=3D671) at /usr/src/sys/kern/kern_lock.c:1195 #15 0xffffffff80ec6a05 in _lockmgr_args (lk=3D0xfffff80004190ba8,=20 flags=3D2106368, ilk=3D, wmesg=3D, prio= =3D0,=20 timo=3D0, file=3D, line=3D18) at /usr/src/sys/sys/lockmg= r.h:104 #16 ffs_lock (ap=3D0xfffffe00005833c8) at /usr/src/sys/ufs/ffs/ffs_vnops.c:= 428 #17 0xffffffff81208059 in VOP_LOCK1_APV ( vop=3D0xffffffff81b62d50 , a=3D0xfffffe00005833c8) at vnode_if.c:2087 #18 0xffffffff80c86927 in VOP_LOCK1 (vp=3D, flags=3D2106368,= =20 file=3D, line=3D671) at ./vnode_if.h:859 #19 _vn_lock (vp=3D0xfffff80004190b40, flags=3D2106368,=20 file=3D0xffffffff8130d3f0 "/usr/src/sys/kern/vfs_lookup.c", line=3D671) at /usr/src/sys/kern/vfs_vnops.c:1531 #20 0xffffffff80c68fe6 in lookup (ndp=3D0xfffffe00005835a0) at /usr/src/sys/kern/vfs_lookup.c:669 #21 0xffffffff80c68aad in namei (ndp=3D0xfffffe00005835a0) at /usr/src/sys/kern/vfs_lookup.c:450 #22 0xffffffff80c49b85 in unp_connectat (fd=3D,=20 so=3D, nam=3D, td=3D0xfffff800036f2000) at /usr/src/sys/kern/uipc_usrreq.c:1554 #23 0xffffffff80c3bc98 in soconnectat (fd=3D,=20 so=3D, nam=3D0xfffff8000367d800,=20 td=3D0xffffffff80b7d390 <_mtx_init+144>) at /usr/src/sys/kern/uipc_socket.c:1230 #24 0xffffffff80e611b9 in clnt_vc_create (so=3D0xfffff80004c316d0,=20 raddr=3D0xfffff800035fc020, prog=3D553713921, vers=3D1, sendsz=3D4096,= =20 recvsz=3D4096, intrflag=3D0) at /usr/src/sys/rpc/clnt_vc.c:159 #25 0xffffffff80e60439 in clnt_reconnect_connect (cl=3D0xfffff80003372840) at /usr/src/sys/rpc/clnt_rc.c:193 #26 clnt_reconnect_call (cl=3D0xfffff80003372840, ext=3D0xfffffe0000583ab0,= =20 proc=3D1, args=3D0xfffff800048dda00, resultsp=3D0xfffffe0000583c28,=20 utimeout=3D...) at /usr/src/sys/rpc/clnt_rc.c:265 #27 0xffffffff80a637ec in newnfs_request (nd=3D0xfffffe0000583c28, nmp=3D0x= 0,=20 clp=3D0x0, nrp=3D0xffffffff82021a18 , vp=3D0x0, td= =3D0x0,=20 cred=3D0xfffff80003e65000, prog=3D553713921, vers=3D1, retsum=3D0x0, to= plevel=3D0,=20 xidp=3D0x0, dssep=3D0x0) at /usr/src/sys/fs/nfs/nfs_commonkrpc.c:818 #28 0xffffffff80a6d5f9 in nfsrv_getuser (procnum=3D1, uid=3D= ,=20 gid=3D, name=3D0x0, p=3D0xfffffe0000582ce0) at /usr/src/sys/fs/nfs/nfs_commonsubs.c:3616 #29 0xffffffff80a6d71a in nfsrv_getgrpscred (oldcred=3D0xfffff80003deec00) at /usr/src/sys/fs/nfs/nfs_commonsubs.c:3150 #30 0xffffffff80acf53e in nfsd_excred (nd=3D0xfffffe0000583ff8,=20 exp=3D, credanon=3D0xfffffe0000582f80) at /usr/src/sys/fs/nfsserver/nfs_nfsdport.c:2932 #31 0xffffffff80aa8104 in nfsrvd_compound (nd=3D, isdgram=3D= 0,=20 tag=3D0x10 ,=20 taglen=3D, minorvers=3D, p=3D) at /usr/src/sys/fs/nfsserver/nfs_nfsdsocket.c:1008 #32 nfsrvd_dorpc (nd=3D0xfffffe0000583ff8, isdgram=3D0,=20 tag=3D0x10 , taglen=3D7,=20 minorvers=3D, p=3D0xfffff800036f2000) at /usr/src/sys/fs/nfsserver/nfs_nfsdsocket.c:560 #33 0xffffffff80abc3a7 in nfs_proc (xid=3D,=20 xprt=3D, nd=3D, rpp=3D) at /usr/src/sys/fs/nfsserver/nfs_nfsdkrpc.c:387 #34 nfssvc_program (rqst=3D0xfffff8004d1d5800, xprt=3D0xfffff800035fb600) at /usr/src/sys/fs/nfsserver/nfs_nfsdkrpc.c:272 #35 0xffffffff80e68499 in svc_executereq (rqstp=3D) at /usr/src/sys/rpc/svc.c:1031 #36 svc_run_internal (grp=3D, ismaster=3D1) at /usr/src/sys/rpc/svc.c:1306 #37 0xffffffff80e6785e in svc_run (pool=3D) at /usr/src/sys/rpc/svc.c:1385 #38 0xffffffff80abca06 in nfsrvd_nfsd (td=3D,=20 args=3D0xfffffe0000584510) at /usr/src/sys/fs/nfsserver/nfs_nfsdkrpc.c:= 542 #39 0xffffffff80ad298b in nfssvc_nfsd (td=3D0xfffff800036f2000,=20 uap=3D) at /usr/src/sys/fs/nfsserver/nfs_nfsdport.c:3451 #40 0xffffffff80e45eeb in sys_nfssvc (td=3D0xfffff800036f2000,=20 uap=3D0xfffff800036f23c0) at /usr/src/sys/nfs/nfs_nfssvc.c:111 #41 0xffffffff810859ef in syscallenter (td=3D0xfffff800036f2000) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:135 #42 amd64_syscall (td=3D0xfffff800036f2000, traced=3D0) at /usr/src/sys/amd64/amd64/trap.c:1029 #43 #44 0x00000008002dee8a in ?? () I'm adding core.txt.2 for reference. Full core dump available on request. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sun Aug 19 21:01:38 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 179171077EE2 for ; Sun, 19 Aug 2018 21:01:38 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id B41C68AEA9 for ; Sun, 19 Aug 2018 21:01:37 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id 75E811077EDA; Sun, 19 Aug 2018 21:01:37 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 543BF1077ED8 for ; Sun, 19 Aug 2018 21:01:37 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D00628AE92 for ; Sun, 19 Aug 2018 21:01:36 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 11DE51B8F5 for ; Sun, 19 Aug 2018 21:01:36 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7JL1ZjX072651 for ; Sun, 19 Aug 2018 21:01:35 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7JL1Ztv072642 for fs@FreeBSD.org; Sun, 19 Aug 2018 21:01:35 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201808192101.w7JL1Ztv072642@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: fs@FreeBSD.org Subject: Problem reports for fs@FreeBSD.org that need special attention Date: Sun, 19 Aug 2018 21:01:35 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Aug 2018 21:01:38 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 203492 | mount_unionfs -o below causes panic New | 217062 | for file systems mounted with -o noexec, exec=off New | 221909 | [ZFS] Add a sysctl to toggle send_corrupt_data Open | 140068 | [smbfs] [patch] smbfs does not allow semicolon in Open | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non f Open | 211491 | System hangs after "Uptime" on reboot with ZFS 6 problems total for which you should take action. From owner-freebsd-fs@freebsd.org Sun Aug 19 22:57:25 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D9EC5107B012 for ; Sun, 19 Aug 2018 22:57:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 776E88ECB3 for ; Sun, 19 Aug 2018 22:57:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 38BFB107B011; Sun, 19 Aug 2018 22:57:25 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1F97C107B010 for ; Sun, 19 Aug 2018 22:57:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AE2988ECB1 for ; Sun, 19 Aug 2018 22:57:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id DA8381C8DF for ; Sun, 19 Aug 2018 22:57:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7JMvNgO043529 for ; Sun, 19 Aug 2018 22:57:23 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7JMvNr1043528 for fs@FreeBSD.org; Sun, 19 Aug 2018 22:57:23 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230752] panic: excl->share in newnfs_request Date: Sun, 19 Aug 2018 22:57:23 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: panic X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: keywords Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Aug 2018 22:57:26 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230752 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |panic --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sun Aug 19 22:59:49 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5AFF5107B129 for ; Sun, 19 Aug 2018 22:59:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id E96708EDA8 for ; Sun, 19 Aug 2018 22:59:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id AE14D107B119; Sun, 19 Aug 2018 22:59:48 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9CDFE107B117 for ; Sun, 19 Aug 2018 22:59:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3E4498EDA7 for ; Sun, 19 Aug 2018 22:59:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 96AA81C8E9 for ; Sun, 19 Aug 2018 22:59:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7JMxlQ6045988 for ; Sun, 19 Aug 2018 22:59:47 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7JMxlGT045987 for fs@FreeBSD.org; Sun, 19 Aug 2018 22:59:47 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 200663] zfs allow/unallow doesn't show numeric UID when the ID no longer exists in the password file Date: Sun, 19 Aug 2018 22:59:47 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: ler@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Aug 2018 22:59:49 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D200663 --- Comment #5 from Larry Rosenman --- Any news? --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Sun Aug 19 23:25:26 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 31FFA107B9EF for ; Sun, 19 Aug 2018 23:25:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id C3A7C8FA2A for ; Sun, 19 Aug 2018 23:25:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 84EA4107B9EE; Sun, 19 Aug 2018 23:25:25 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 73913107B9ED for ; Sun, 19 Aug 2018 23:25:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0219E8FA27 for ; Sun, 19 Aug 2018 23:25:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 4D68E1CD10 for ; Sun, 19 Aug 2018 23:25:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7JNPOe1022383 for ; Sun, 19 Aug 2018 23:25:24 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7JNPOkF022382 for fs@FreeBSD.org; Sun, 19 Aug 2018 23:25:24 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230752] panic: excl->share in newnfs_request Date: Sun, 19 Aug 2018 23:25:24 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: panic X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: rmacklem@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Aug 2018 23:25:26 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230752 Rick Macklem changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |rmacklem@FreeBSD.org --- Comment #1 from Rick Macklem --- Is there a specific situation happening when these panic()s occur? (such as startup, shutdown or ??) Basically, the panic() happens when the kernel RPC code is doing a connect on an AF_LOCAL socket created by the nfsuserd daemon to do an upcall. I'll admit I haven't tested this myself recently, but I'll try running nfsuserd on current. (prior to FreeBSD12, nfsuserd uses a UDP socket, so this panic() wouldn't happen) (Since unp_connectat() does a straightforward namei(), I don't know why there would be an "excl->shared" panic? The code is using LK_SHARED for the lookup, but that happens all the time.) I am wondering if somehow the AF_LOCAL socket got deleted and that confuses the namei()/lookup(). (The socket is /var/run/nfsuserd.sock.) If the panic()s are causing you grief, you can add the -use-udpsock command line option to nfsuserd to make it use the UDP socket, like FreeBSD= -11. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Mon Aug 20 19:35:39 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C640F107A777; Mon, 20 Aug 2018 19:35:39 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (chez.mckusick.com [70.36.157.235]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 397357DF6A; Mon, 20 Aug 2018 19:35:39 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (localhost [IPv6:::1]) by chez.mckusick.com (8.15.2/8.15.2) with ESMTP id w7KJeu29072094; Mon, 20 Aug 2018 12:40:56 -0700 (PDT) (envelope-from mckusick@mckusick.com) Message-Id: <201808201940.w7KJeu29072094@chez.mckusick.com> From: Kirk McKusick To: FreeBSD Current , FreeBSD Filesystems Subject: CFT: TRIM Consolodation on UFS/FFS filesystems X-URL: http://WWW.McKusick.COM/ Reply-To: Kirk McKusick MIME-Version: 1.0 Content-ID: <71992.1534793715.0@chez.mckusick.com> Date: Mon, 20 Aug 2018 12:40:56 -0700 X-Spam-Status: No, score=-1.4 required=5.0 tests=BAYES_00,MISSING_MID, UNPARSEABLE_RELAY autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on chez.mckusick.com Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Aug 2018 19:35:40 -0000 I have recently added TRIM consolodation support for the UFS/FFS filesystem. This feature consolodates large numbers of TRIM commands into a much smaller number of commands covering larger blocks of disk space. Best described by the commit message: Author: mckusick Date: Sun Aug 19 16:56:42 2018 New Revision: 338056 URL: https://svnweb.freebsd.org/changeset/base/338056 Log: Add consolodation of TRIM / BIO_DELETE commands to the UFS/FFS filesys= tem. = When deleting files on filesystems that are stored on flash-memory (solid-state) disk drives, the filesystem notifies the underlying disk of the blocks that it is no longer using. The notification allows the drive to avoid saving these blocks when it needs to flash (zero out) one of its flash pages. These notifications of no-longer-being-used blocks are referred to as TRIM notifications. In FreeBSD these TRIM notifications are sent from the filesystem to the drive using the BIO_DELETE command. = Until now, the filesystem would send a separate message to the drive for each block of the file that was deleted. Each Gigabyte of file size resulted in over 3000 TRIM messages being sent to the drive. This burst of messages can overwhelm the drive's task queue causing multiple second delays for read and write requests. = This implementation collects runs of contiguous blocks in the file and then consolodates them into a single BIO_DELETE command to the drive. The BIO_DELETE command describes the run of blocks as a single large block being deleted. Each Gigabyte of file size can result in as few as two BIO_DELETE commands and is typically less than ten. Though these larger BIO_DELETE commands take longer to run, they do not clog the drive task queue, so read and write commands can intersperse effectively with them. = Though this new feature has been throughly reviewed and tested, it is being added disabled by default so as to minimize the possibility of disrupting the upcoming 12.0 release. It can be enabled by running ``sysctl vfs.ffs.dotrimcons=3D1''. Users are encouraged to test it. If no problems arise, we will consider requesting that it be enabled by default for 12.0. = Reviewed by: kib Tested by: Peter Holm Sponsored by: Netflix This support is off by default, but I am hoping that I can get enough testing to ensure that it (a) works, and (b) is helpful that it will be reasonable to have it turned on by default in 12.0. The cutoff for turning it on by default in 12.0 is September 19th. So I am requesting your testing feedback in the near-term. Please let me know if you have managed to use it successfully (or not) and also if it provided any performance difference (good or bad). To enable TRIM consolodation either use `sysctl vfs.ffs.dotrimcons=3D1' or just set the `dotrimcons' variable in sys/ufs/ffs/ffs_alloc.c to 1. Everything you need to test TRIM consolodation is obtained by setting the above sysctl. However, if you want to collect statistics on how effective the TRIM consolodation is working, the attached diff will allow you to easily get statitics on how the TRIM is going. Compile your kernel and the mount command. Note that if you do not do a buildworld, you will need to copy /sys/sys/mount.h to /usr/include/sys/mount.h to get the patched mount command to compile. Then run `mount -v' (or `mount -v | grep /mnt' to get just the statistics for /mnt). Removing a 30Mb file without TRIM consolodation: /dev/md0 on /mnt (ufs, local, writes: sync 10 async 482, reads: sync 7 asy= nc 0, fsid d43f795b6a7d34fb, TRIM: total 952 total blocks 7616) While removing the same file with TRIM consolodation: /dev/md0 on /mnt (ufs, local, writes: sync 10 async 482, reads: sync 7 asy= nc 0, fsid d43f795b6a7d34fb, TRIM: total 3 total blocks 7616) It also tracks pending blocks and pending files. These numbers are only printed out when they are non-zero. Here is an example running with soft updates right after a file has been rm'ed, but its blocks not yet released= : /dev/md0 on /mnt (ufs, local, soft-updates, writes: sync 2 async 251, read= s: sync 5 async 0, fsid 303f795b1be0c459, pending blocks 7616, pending fil= es 1) Finally it tracks inflight BIO_DELETEs and total blocks represented by those inflight BIO_DELETEs. These numbers are also only printed out when they are non-zero. These statistics let you see how much of a backlog of BIO_DELETEs you have backed up at/in the disk drive and you can track how quickly they drain. Kirk McKusick From owner-freebsd-fs@freebsd.org Mon Aug 20 19:53:47 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C8246107AF42; Mon, 20 Aug 2018 19:53:46 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (chez.mckusick.com [70.36.157.235]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 56F9F7F2A4; Mon, 20 Aug 2018 19:53:46 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (localhost [IPv6:::1]) by chez.mckusick.com (8.15.2/8.15.2) with ESMTP id w7KJx3iK072721; Mon, 20 Aug 2018 12:59:03 -0700 (PDT) (envelope-from mckusick@mckusick.com) Message-Id: <201808201959.w7KJx3iK072721@chez.mckusick.com> From: Kirk McKusick To: FreeBSD Current , FreeBSD Filesystems Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems X-URL: http://WWW.McKusick.COM/ Reply-To: Kirk McKusick In-reply-to: <201808201940.w7KJeu29072094@chez.mckusick.com> Comments: In-reply-to Kirk McKusick message dated "Mon, 20 Aug 2018 12:40:56 -0700." MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <72719.1534795143.1@chez.mckusick.com> Date: Mon, 20 Aug 2018 12:59:03 -0700 X-Spam-Status: No, score=-1.4 required=5.0 tests=BAYES_00,MISSING_MID, UNPARSEABLE_RELAY autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on chez.mckusick.com X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Aug 2018 19:53:47 -0000 From: Kirk McKusick To: FreeBSD Current , FreeBSD Filesystems Subject: CFT: TRIM Consolodation on UFS/FFS filesystems Date: Mon, 20 Aug 2018 12:40:56 -0700 Oops, forgot that attachments get stripped. Below are the diffs for gathering statistics. Sorry to those of you on Gmail for whom they will be mangled. Kirk McKusick =-=-= Index: sbin/mount/mount.c =================================================================== --- sbin/mount/mount.c (revision 338054) +++ sbin/mount/mount.c (working copy) @@ -686,6 +686,18 @@ prmount(struct statfs *sfp) for (i = 0; i < sizeof(sfp->f_fsid); i++) printf("%02x", ((u_char *)&sfp->f_fsid)[i]); } + if (sfp->f_trim_total != 0 || sfp->f_trim_total_blks != 0) + (void)printf(", TRIM: total %ju total blocks %ju", + (uintmax_t)sfp->f_trim_total, + (uintmax_t)sfp->f_trim_total_blks); + if (sfp->f_trim_inflight != 0 || sfp->f_trim_inflight_blks != 0) + (void)printf(", TRIM: inflight %ju inflight blocks %ju", + (uintmax_t)sfp->f_trim_inflight, + (uintmax_t)sfp->f_trim_inflight_blks); + if (sfp->f_pendingblks != 0 || sfp->f_pendingfiles != 0) + (void)printf(", pending blocks %ju, pending files %ju", + (uintmax_t)sfp->f_pendingblks, + (uintmax_t)sfp->f_pendingfiles); } (void)printf(")\n"); } Index: sys/sys/mount.h =================================================================== --- sys/sys/mount.h (revision 338054) +++ sys/sys/mount.h (working copy) @@ -85,7 +85,13 @@ struct statfs { uint64_t f_asyncwrites; /* count of async writes since mount */ uint64_t f_syncreads; /* count of sync reads since mount */ uint64_t f_asyncreads; /* count of async reads since mount */ - uint64_t f_spare[10]; /* unused spare */ + uint64_t f_trim_total; /* count of TRIM ops since mount */ + uint64_t f_trim_total_blks; /* count of TRIM blocks since mount */ + uint64_t f_trim_inflight; /* count of TRIM ops in progress */ + uint64_t f_trim_inflight_blks; /* count of TRIM blocks in progress */ + int64_t f_pendingblks; /* pending free blocks */ + int64_t f_pendingfiles; /* pending free nodes */ + uint64_t f_spare[4]; /* unused spare */ uint32_t f_namemax; /* maximum filename length */ uid_t f_owner; /* user that mounted the filesystem */ fsid_t f_fsid; /* filesystem id */ Index: sys/ufs/ffs/ffs_vfsops.c =================================================================== --- sys/ufs/ffs/ffs_vfsops.c (revision 338081) +++ sys/ufs/ffs/ffs_vfsops.c (working copy) @@ -1398,7 +1398,13 @@ ffs_statfs(mp, sbp) sbp->f_bsize = fs->fs_fsize; sbp->f_iosize = fs->fs_bsize; sbp->f_blocks = fs->fs_dsize; + sbp->f_pendingblks = dbtofsb(fs, fs->fs_pendingblocks); + sbp->f_pendingfiles = fs->fs_pendinginodes; UFS_LOCK(ump); + sbp->f_trim_total = ump->um_trim_total; + sbp->f_trim_total_blks = ump->um_trim_total_blks; + sbp->f_trim_inflight = ump->um_trim_inflight; + sbp->f_trim_inflight_blks = ump->um_trim_inflight_blks; sbp->f_bfree = fs->fs_cstotal.cs_nbfree * fs->fs_frag + fs->fs_cstotal.cs_nffree + dbtofsb(fs, fs->fs_pendingblocks); sbp->f_bavail = freespace(fs, fs->fs_minfree) + From owner-freebsd-fs@freebsd.org Tue Aug 21 11:24:09 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 47D60106DF11 for ; Tue, 21 Aug 2018 11:24:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id D903A7F20D for ; Tue, 21 Aug 2018 11:24:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 9DE9C106DF10; Tue, 21 Aug 2018 11:24:08 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8C92A106DF0F for ; Tue, 21 Aug 2018 11:24:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2E6617F205 for ; Tue, 21 Aug 2018 11:24:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 629DB2FAB7 for ; Tue, 21 Aug 2018 11:24:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7LBO7ba047793 for ; Tue, 21 Aug 2018 11:24:07 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7LBO7SP047792 for fs@FreeBSD.org; Tue, 21 Aug 2018 11:24:07 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 219760] ZFS iSCSI w/ Win10 Initiator Causes pool corruption Date: Tue, 21 Aug 2018 11:24:07 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: emz@norma.perm.ru X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Aug 2018 11:24:09 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219760 --- Comment #8 from emz@norma.perm.ru --- Workaround (provided by mav@): set the vfs.zfs.vol.immediate_write_sz to 131072. No zvol corruptiojn observed after this. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Aug 21 14:07:38 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3F36C1073798 for ; Tue, 21 Aug 2018 14:07:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id CDDD184EAF for ; Tue, 21 Aug 2018 14:07:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 930C31073796; Tue, 21 Aug 2018 14:07:37 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 81F371073795 for ; Tue, 21 Aug 2018 14:07:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 23B7B84EA4 for ; Tue, 21 Aug 2018 14:07:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 71A5B9123 for ; Tue, 21 Aug 2018 14:07:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7LE7aw6034821 for ; Tue, 21 Aug 2018 14:07:36 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7LE7a9Z034820 for fs@FreeBSD.org; Tue, 21 Aug 2018 14:07:36 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 219760] ZFS iSCSI w/ Win10 Initiator Causes pool corruption Date: Tue, 21 Aug 2018 14:07:36 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: trasz@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Aug 2018 14:07:38 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219760 --- Comment #9 from Edward Tomasz Napierala --- Good catch! Do you know _why_ does it fix the problem? --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Aug 21 23:16:47 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3717610827E1 for ; Tue, 21 Aug 2018 23:16:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id C816573347 for ; Tue, 21 Aug 2018 23:16:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 898B910827E0; Tue, 21 Aug 2018 23:16:46 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 784EC10827DF for ; Tue, 21 Aug 2018 23:16:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1995B73346 for ; Tue, 21 Aug 2018 23:16:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 61243E010 for ; Tue, 21 Aug 2018 23:16:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7LNGjP8050356 for ; Tue, 21 Aug 2018 23:16:45 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7LNGj52050355 for fs@FreeBSD.org; Tue, 21 Aug 2018 23:16:45 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230752] panic: excl->share in newnfs_request Date: Tue, 21 Aug 2018 23:16:45 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: panic X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: rmacklem@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Aug 2018 23:16:47 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230752 --- Comment #2 from Rick Macklem --- Do you have /var/run NFS mounted by any chance? --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Aug 21 23:27:57 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6875D1082AF2 for ; Tue, 21 Aug 2018 23:27:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id F0EA173785 for ; Tue, 21 Aug 2018 23:27:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id B5CF61082AF1; Tue, 21 Aug 2018 23:27:56 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A452C1082AF0 for ; Tue, 21 Aug 2018 23:27:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 45B7373783 for ; Tue, 21 Aug 2018 23:27:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 8D395E169 for ; Tue, 21 Aug 2018 23:27:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7LNRtwg071922 for ; Tue, 21 Aug 2018 23:27:55 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7LNRtoc071921 for fs@FreeBSD.org; Tue, 21 Aug 2018 23:27:55 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230752] panic: excl->share in newnfs_request Date: Tue, 21 Aug 2018 23:27:55 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: panic X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: rmacklem@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Aug 2018 23:27:57 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230752 --- Comment #3 from Rick Macklem --- Sorry, to clarify...I was wondering if the machine that panic()'d had its /var/run exported to NFS client(s) that would have it mounted via N= FS. (If that was the case, it could be a case where the nfsd thread is doing a Getattr on /var/run/nfsuserd.sock and then it tries to connect to /var/run/nfsuserd.sock to do the upcall to the nfsuserd.) --> I think I'll need to make it still use UDP sockets by default (since this could happen when the socket is in the namespace). --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Aug 22 00:48:33 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 78D7210851CC; Wed, 22 Aug 2018 00:48:33 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (www.zefox.net [50.1.20.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "www.zefox.org", Issuer "www.zefox.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id ED6EC77159; Wed, 22 Aug 2018 00:48:32 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (localhost [127.0.0.1]) by www.zefox.net (8.15.2/8.15.2) with ESMTPS id w7M0mh1P084744 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 21 Aug 2018 17:48:44 -0700 (PDT) (envelope-from fbsd@www.zefox.net) Received: (from fbsd@localhost) by www.zefox.net (8.15.2/8.15.2/Submit) id w7M0mhmj084743; Tue, 21 Aug 2018 17:48:43 -0700 (PDT) (envelope-from fbsd) Date: Tue, 21 Aug 2018 17:48:43 -0700 From: bob prohaska To: Kirk McKusick Cc: FreeBSD Current , FreeBSD Filesystems , bob prohaska Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems Message-ID: <20180822004843.GA84687@www.zefox.net> References: <201808201940.w7KJeu29072094@chez.mckusick.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201808201940.w7KJeu29072094@chez.mckusick.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 00:48:33 -0000 On Mon, Aug 20, 2018 at 12:40:56PM -0700, Kirk McKusick wrote: > I have recently added TRIM consolodation support for the UFS/FFS > filesystem. This feature consolodates large numbers of TRIM commands > into a much smaller number of commands covering larger blocks of > disk space. Best described by the commit message: > > Author: mckusick > Date: Sun Aug 19 16:56:42 2018 > New Revision: 338056 > URL: https://svnweb.freebsd.org/changeset/base/338056 > > Log: > Add consolodation of TRIM / BIO_DELETE commands to the UFS/FFS filesystem. > > When deleting files on filesystems that are stored on flash-memory > (solid-state) disk drives, the filesystem notifies the underlying > disk of the blocks that it is no longer using. The notification > allows the drive to avoid saving these blocks when it needs to > flash (zero out) one of its flash pages. These notifications of > no-longer-being-used blocks are referred to as TRIM notifications. > In FreeBSD these TRIM notifications are sent from the filesystem > to the drive using the BIO_DELETE command. > > Until now, the filesystem would send a separate message to the drive > for each block of the file that was deleted. Each Gigabyte of file > size resulted in over 3000 TRIM messages being sent to the drive. > This burst of messages can overwhelm the drive's task queue causing > multiple second delays for read and write requests. > > This implementation collects runs of contiguous blocks in the file > and then consolodates them into a single BIO_DELETE command to the > drive. The BIO_DELETE command describes the run of blocks as a > single large block being deleted. Each Gigabyte of file size can > result in as few as two BIO_DELETE commands and is typically less > than ten. Though these larger BIO_DELETE commands take longer to > run, they do not clog the drive task queue, so read and write > commands can intersperse effectively with them. > > Though this new feature has been throughly reviewed and tested, it > is being added disabled by default so as to minimize the possibility > of disrupting the upcoming 12.0 release. It can be enabled by running > ``sysctl vfs.ffs.dotrimcons=1''. Users are encouraged to test it. > If no problems arise, we will consider requesting that it be enabled > by default for 12.0. > > Reviewed by: kib > Tested by: Peter Holm > Sponsored by: Netflix > > This support is off by default, but I am hoping that I can get enough > testing to ensure that it (a) works, and (b) is helpful that it will > be reasonable to have it turned on by default in 12.0. The cutoff for > turning it on by default in 12.0 is September 19th. So I am requesting > your testing feedback in the near-term. Please let me know if you have > managed to use it successfully (or not) and also if it provided any > performance difference (good or bad). > > To enable TRIM consolodation either use `sysctl vfs.ffs.dotrimcons=1' > or just set the `dotrimcons' variable in sys/ufs/ffs/ffs_alloc.c to 1. > Will the new feature be active on a Raspberry Pi 3 using flash on microSD and USB for file systems and swap? Can the feature be turned on using one of the conf files in /etc? According to Sandisk, "All microSD or USB drives are flash memory and does support the TRIM command, however, you will not notice any difference after running TRIM command on memory cards or USB drives. TRIM command is basically used for SSD and Hard drives." The "you will not notice any difference...." qualification makes me slightly uncertain the reply was well-informed, but if there's any hope of success I'd like to try it. >From time to time there seem to be traffic jams among flash devices on the RPI3, it would a pleasant surprise if this feature helps. Thanks for reading! bob prohaska From owner-freebsd-fs@freebsd.org Wed Aug 22 01:55:47 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 72D9A1086F4C; Wed, 22 Aug 2018 01:55:47 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 29C2879EAD; Wed, 22 Aug 2018 01:55:47 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from mail-io0-f171.google.com (mail-io0-f171.google.com [209.85.223.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) (Authenticated sender: mmacy) by smtp.freebsd.org (Postfix) with ESMTPSA id E67BD101AC; Wed, 22 Aug 2018 01:55:46 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: by mail-io0-f171.google.com with SMTP id w11-v6so298885iob.2; Tue, 21 Aug 2018 18:55:46 -0700 (PDT) X-Gm-Message-State: APzg51AAEOgFIoQ/HHT24fdlrQfg0YjARlOr8p1tzwpTyM+p6ykRFNGh mB1VuRvpFISgQdC5ZRlEuNZMaK4aItVI80Q+lQ4= X-Google-Smtp-Source: ANB0VdZ8RylZhYqhLJvBkA2tH6YFMhXuPRGSDE/629i7eP/hEbKYdA85KMa+XoDKlHl/WAEYLzkzt9JN5ZKBT8RHeY0= X-Received: by 2002:a6b:500e:: with SMTP id e14-v6mr10725789iob.5.1534902946477; Tue, 21 Aug 2018 18:55:46 -0700 (PDT) MIME-Version: 1.0 From: Matthew Macy Date: Tue, 21 Aug 2018 18:55:35 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Native Encryption for ZFS on FreeBSD CFT To: freebsd-current , freebsd-fs@freebsd.org Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 01:55:47 -0000 To anyone with an interest in native encryption in ZFS please test the projects/zfs-crypto-merge-0820 branch in my freebsd repo: https://github.com/mattmacy/networking.git ( git clone https://github.com/mattmacy/networking.git -b projects/zfs-crypto-merge-0820 ) The UI is quite close to the Oracle Solaris ZFS crypto with minor differences for specifying key location. Please note that once a feature is enabled on a pool it can't be disabled. This means that if you enable encryption support on a pool you will never be able to import it in to a ZFS without encryption support. For this reason I would strongly advise against using this on any pool that can't be easily replaced until this change has made its way in to HEAD after the freeze has been lifted. By way of background the original ZoL commit can be found at: https://github.com/zfsonlinux/zfs/pull/5769/commits/5aef9bedc801830264428c64cd2242d1b786fd49 Thanks in advance. -M From owner-freebsd-fs@freebsd.org Wed Aug 22 02:27:14 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 66B7B1088540; Wed, 22 Aug 2018 02:27:14 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 111AC7B9BA; Wed, 22 Aug 2018 02:27:14 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from mail-it0-f48.google.com (mail-it0-f48.google.com [209.85.214.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) (Authenticated sender: mmacy) by smtp.freebsd.org (Postfix) with ESMTPSA id CC0D1104B8; Wed, 22 Aug 2018 02:27:13 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: by mail-it0-f48.google.com with SMTP id h23-v6so1082125ita.5; Tue, 21 Aug 2018 19:27:13 -0700 (PDT) X-Gm-Message-State: APzg51CDBOX29EwVxUKLHQxDik6kYli84VNqeTZ2sLaJu1sMSvxW9J2x ax20wE0agBZ98shspm09/AJDPaBY2t6LPVdHDFg= X-Google-Smtp-Source: ANB0VdYmAzdU1G8Optqx6uQ9rRQ5sQWo4T5PIRfzl7Sh5iBFJ2vNK2laVKRBC3lmyLuasiXh/jyixKfJy4MWF04bVqg= X-Received: by 2002:a24:704f:: with SMTP id f76-v6mr1626689itc.30.1534904833320; Tue, 21 Aug 2018 19:27:13 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Matthew Macy Date: Tue, 21 Aug 2018 19:27:02 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Native Encryption for ZFS on FreeBSD CFT To: freebsd-current , freebsd-fs@freebsd.org Cc: Sean Fagan Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 02:27:14 -0000 On Tue, Aug 21, 2018 at 6:55 PM Matthew Macy wrote: > To anyone with an interest in native encryption in ZFS please test the > projects/zfs-crypto-merge-0820 branch in my freebsd repo: > https://github.com/mattmacy/networking.git > > Oh and I neglected to state that this work is being supported by iX Systems and the tree is all built on work done by Sean Fagan at iX Systems. Please keep him in the loop on any problems encountered. Thanks. > ( git clone https://github.com/mattmacy/networking.git -b > projects/zfs-crypto-merge-0820 ) > > The UI is quite close to the Oracle Solaris ZFS crypto with minor > differences for specifying key location. > > Please note that once a feature is enabled on a pool it can't be > disabled. This means that if you enable encryption support on a pool > you will never be able to import it in to a ZFS without encryption > support. For this reason I would strongly advise against using this on > any pool that can't be easily replaced until this change has made its > way in to HEAD after the freeze has been lifted. > > > By way of background the original ZoL commit can be found at: > > https://github.com/zfsonlinux/zfs/pull/5769/commits/5aef9bedc801830264428c64cd2242d1b786fd49 > > Thanks in advance. > -M > From owner-freebsd-fs@freebsd.org Wed Aug 22 03:11:56 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D3F831089901; Wed, 22 Aug 2018 03:11:55 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-lj1-f180.google.com (mail-lj1-f180.google.com [209.85.208.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5043B7D525; Wed, 22 Aug 2018 03:11:55 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-lj1-f180.google.com with SMTP id 203-v6so324178ljj.13; Tue, 21 Aug 2018 20:11:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=2H77byjIsdHOagXX3E6B0cd9+5m33eRiPqKcdQc6duw=; b=ndDZAcx+HhcvU9zVPa85IMDB3edx9YJZLID8PZtK4pK63MO0G1a2ipFR1jAlDEB5CU aNErPir4NI3GuabzUXa22pP4E7BT+Jdrir2s963CeZJxNOscpZvaTPzcEUnmbHJGxm+3 E3VZAbY0ELMctQPUpEvo5wXa3LexogUAv46AKqqHh+Bsd7/Eqibrzaqn8u9Uoyz3Mxyn GChFn/C6+3yMrlpfCpvrkl1v1g8WfSirtX4ztPWrF42C1QiTkcIsRYoXE6bPZ9di29K/ D5cTDgBrpY+pfl/Y01BLtoK2LJuWuRB3Cng0vx/kSVSnvaZLy++Bavb3sPhylZHTp+I6 wogA== X-Gm-Message-State: AOUpUlHsx0lfic7T9mPQH94C7K74MW5A1omgTznXje0eJApRFQszsVR3 wtoePnSZ0q+ithSgAYwaUaomiIrVyetB0VdNT0VITQ== X-Google-Smtp-Source: AA+uWPwQXwYR0S7XM4YwMSO84aJc973FVyCEVHcJdMreNfRTJihx1IJYHV+i+cGYjX7k5+OsXjY6wDMrl0Oyw4emo8A= X-Received: by 2002:a2e:2067:: with SMTP id g100-v6mr35044269ljg.138.1534907507989; Tue, 21 Aug 2018 20:11:47 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Alan Somers Date: Tue, 21 Aug 2018 21:11:36 -0600 Message-ID: Subject: Re: Native Encryption for ZFS on FreeBSD CFT To: Matthew Macy Cc: FreeBSD CURRENT , freebsd-fs , Sean Fagan Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 03:11:56 -0000 The last time I looked (which was a long time ago), Oracle's ZFS encryption looked extremely vulnerable to watermarking attacks. Did anybody ever fix that? -Alan On Tue, Aug 21, 2018 at 8:28 PM Matthew Macy wrote: > On Tue, Aug 21, 2018 at 6:55 PM Matthew Macy wrote: > > > To anyone with an interest in native encryption in ZFS please test the > > projects/zfs-crypto-merge-0820 branch in my freebsd repo: > > https://github.com/mattmacy/networking.git > > > > > Oh and I neglected to state that this work is being supported by iX Systems > and the tree is all built on work done by Sean Fagan at iX Systems. Please > keep him in the loop on any problems encountered. > Thanks. > > > > > ( git clone https://github.com/mattmacy/networking.git -b > > projects/zfs-crypto-merge-0820 ) > > > > The UI is quite close to the Oracle Solaris ZFS crypto with minor > > differences for specifying key location. > > > > Please note that once a feature is enabled on a pool it can't be > > disabled. This means that if you enable encryption support on a pool > > you will never be able to import it in to a ZFS without encryption > > support. For this reason I would strongly advise against using this on > > any pool that can't be easily replaced until this change has made its > > way in to HEAD after the freeze has been lifted. > > > > > > By way of background the original ZoL commit can be found at: > > > > > https://github.com/zfsonlinux/zfs/pull/5769/commits/5aef9bedc801830264428c64cd2242d1b786fd49 > > > > Thanks in advance. > > -M > > > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Wed Aug 22 03:13:53 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 64E0E1089A4D for ; Wed, 22 Aug 2018 03:13:53 +0000 (UTC) (envelope-from sef@ixsystems.com) Received: from mail-pf1-x431.google.com (mail-pf1-x431.google.com [IPv6:2607:f8b0:4864:20::431]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DB7DE7D7F1 for ; Wed, 22 Aug 2018 03:13:52 +0000 (UTC) (envelope-from sef@ixsystems.com) Received: by mail-pf1-x431.google.com with SMTP id b11-v6so294001pfo.3 for ; Tue, 21 Aug 2018 20:13:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ixsystems-com.20150623.gappssmtp.com; s=20150623; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=9QwmT573P5z+E5cNktIb+if7LdtQ6c7G8l3Qzm5j1X4=; b=PDNCtFcEIv6ZZoL/PMt4Y18CVpjQrxE1Qn6SlXXMsjzYldeomyCgcQkgMBlRs37wp0 9KNG4Hn8tmIH7znmD2fdrhpz0kPWQ5QJ02bd3HXAE7MG+8BfPA5V/vdEr+dc7byKrolX xuj7IeHX7gKFYz2KsY9RxHdr1Fcv+grawEnrMx0tdqhAc78K9HbSUL78XDm0RPXiIkC2 oUYgRac4FzYwKnzGnJM57VO5aVT1SmBlKU6eNB7UoWQxDBZ/DzYcTWBjLd0xegowjnFH bd7g3/c7cpRF5Bs5clEunVD51agJulZdI0/6xnJZzhyV/T0zEaUl89ZZwgwzf2dm8LIB R0CA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=9QwmT573P5z+E5cNktIb+if7LdtQ6c7G8l3Qzm5j1X4=; b=mxEkKmKU87c56djKFdqV9+sfe3nrQSC2LxAkNqZCnflAJ81WEu+kdRsmZhuSj8Ii9O C5z9YdGxx69v0bIOqGH4kC3ytVYhxQxXde8wwUXtgIgmuQmWNdf3pFbUkdLopUp1eBrZ d1RY+SF1KLmvAFrsd3XKKqcAQTCmViVQpwzg7sU3eY8RRzhcFfFnhPCg5dSHpgXQkZg+ tsaxQ8DvZBd7zRsvjPPzEYcwUsp8zkvM5TYBcHCTCe5ElFaGHdPrsnWu2Nh15gRXD18r glDu7aEO7dhcD4mdfWJqUZY1mvhjeFKj4oxlkNXW1moIwvv5n7R0BIyRfD2lC8QAlPk7 XaTg== X-Gm-Message-State: AOUpUlHd0icvoWzx70EGIGxqLJPU8nWXZBd3Ew/wyh4jU8xvLSgx3G4v N09AZak9j9+FFfq80K9TK8C10A== X-Google-Smtp-Source: AA+uWPyUjjuM2lzYckIV9dJ+I2UxdVDTkq0n2tFidOeFPiBmnxopXaPTuO+dG3wUOKgoPVH6wPWctQ== X-Received: by 2002:a62:f909:: with SMTP id o9-v6mr56047791pfh.141.1534907631947; Tue, 21 Aug 2018 20:13:51 -0700 (PDT) Received: from [192.168.0.28] (173-164-180-199-SFBA.hfc.comcastbusiness.net. [173.164.180.199]) by smtp.gmail.com with ESMTPSA id x10-v6sm368355pge.23.2018.08.21.20.13.50 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 21 Aug 2018 20:13:51 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: Native Encryption for ZFS on FreeBSD CFT From: Sean Fagan In-Reply-To: Date: Tue, 21 Aug 2018 20:13:49 -0700 Cc: Matthew Macy , FreeBSD CURRENT , freebsd-fs Content-Transfer-Encoding: quoted-printable Message-Id: References: To: Alan Somers X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 03:13:53 -0000 On Aug 21, 2018, at 8:11 PM, Alan Somers wrote: > The last time I looked (which was a long time ago), Oracle's ZFS = encryption looked extremely vulnerable to watermarking attacks. Did = anybody ever fix that? This isn=E2=80=99t Oracle=E2=80=99s implementation, but I don=E2=80=99t = know how compatible or not it is with it. Sean. From owner-freebsd-fs@freebsd.org Wed Aug 22 03:22:45 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DA66F1089E9E; Wed, 22 Aug 2018 03:22:44 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-lf1-f53.google.com (mail-lf1-f53.google.com [209.85.167.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 540D37DCE5; Wed, 22 Aug 2018 03:22:44 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-lf1-f53.google.com with SMTP id e23-v6so335180lfc.13; Tue, 21 Aug 2018 20:22:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=K4sxYQFtk0zvnwlbCDWB5N32seFVM3EPQZHj3QZsVvc=; b=egcGw2pCFMOrZAgUzOAvGgiy80Qey8XVaQYmXzDhz9k6tqgNnGxG871O1bugqPJsXh E20H81fLbMpqRSPct2XD6I4mbS+HvwtESqgferFA7oYwM09ef8CUMTxKexqJE94/QmeJ vG2MAi8f1qnkuS24KPgZTUuKjOjrqvFLQkYzOAtmSNNLK2kYHgTDF+KB1msMbjgs3qBh PPrlBy+JXtboZqsP3IO4Vy8/oRfgaP+hPxcBibBT6PWXBLPRYzHQzENzUZoTdA+gsaaY iXiR2iu0lMTjgplU0fpQTDRuFhMCS+MfVLA45ItTVZam5ydIn9f0/sBg4RIUQQOl+Wia ltMw== X-Gm-Message-State: AOUpUlHIAxc4bEGqeMEmpk34xDcYUZP0E+hsQbux7xflIZPUn0CvYclP 8R68U9KB1Q4SFEuEmll10LJ4pRx9GvnmB/Rhz6Y= X-Google-Smtp-Source: AA+uWPz94t+P7yEp+c+IKPUUNpMzreD77WvNuc2a2szQz2dqWEhzHqXJ4PhDV5libp7VLnRzJUVCJ5ARSfO0Ky8308U= X-Received: by 2002:a19:c94a:: with SMTP id z71-v6mr12897641lff.34.1534907785971; Tue, 21 Aug 2018 20:16:25 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Alan Somers Date: Tue, 21 Aug 2018 21:16:14 -0600 Message-ID: Subject: Re: Native Encryption for ZFS on FreeBSD CFT To: Sean Fagan Cc: Matthew Macy , FreeBSD CURRENT , freebsd-fs Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 03:22:45 -0000 On Tue, Aug 21, 2018 at 9:13 PM Sean Fagan wrote: > On Aug 21, 2018, at 8:11 PM, Alan Somers wrote: > > The last time I looked (which was a long time ago), Oracle's ZFS > encryption looked extremely vulnerable to watermarking attacks. Did > anybody ever fix that? > > This isn=E2=80=99t Oracle=E2=80=99s implementation, but I don=E2=80=99t k= now how compatible or not > it is with it. > > Sean. > It wasn't just an implementation problem, it was in the design. IIRC, Oracle's encryption allowed encrypted blocks to be deduplicated. There's pretty much no way to defend against watermarking attacks with such a design. Does the new encryption design have the same flaw? -Alan From owner-freebsd-fs@freebsd.org Wed Aug 22 03:27:01 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C8792108A0BE; Wed, 22 Aug 2018 03:27:01 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 799A37E00E; Wed, 22 Aug 2018 03:27:01 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from mail-io0-f171.google.com (mail-io0-f171.google.com [209.85.223.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) (Authenticated sender: mmacy) by smtp.freebsd.org (Postfix) with ESMTPSA id 3FC7610AD8; Wed, 22 Aug 2018 03:27:01 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: by mail-io0-f171.google.com with SMTP id l7-v6so410844iok.6; Tue, 21 Aug 2018 20:27:01 -0700 (PDT) X-Gm-Message-State: AOUpUlERxQMew6WVd9LpaWXzQ0QbmEmKiWCQyae09ti62VzMdwQFXKDw 4THdajwndBu6zqUPSC6fNlal98APKgLuhpm0K4U= X-Google-Smtp-Source: AA+uWPwSVUNXx50T+vUJSt4grwvC4Uutr7fM4srzBnVOgh4I5teL/eXLgTjEQ98TEq8Ly/QUcz+CJ0CvJKMErAXCZCI= X-Received: by 2002:a6b:ed11:: with SMTP id n17-v6mr19517651iog.132.1534908420727; Tue, 21 Aug 2018 20:27:00 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Matthew Macy Date: Tue, 21 Aug 2018 20:26:50 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Native Encryption for ZFS on FreeBSD CFT To: Alan Somers Cc: FreeBSD CURRENT , Sean Fagan , freebsd-fs Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 03:27:02 -0000 On Tue, Aug 21, 2018 at 20:22 Alan Somers wrote: > On Tue, Aug 21, 2018 at 9:13 PM Sean Fagan wrote: > >> On Aug 21, 2018, at 8:11 PM, Alan Somers wrote: >> > The last time I looked (which was a long time ago), Oracle's ZFS >> encryption looked extremely vulnerable to watermarking attacks. Did >> anybody ever fix that? >> >> This isn=E2=80=99t Oracle=E2=80=99s implementation, but I don=E2=80=99t = know how compatible or >> not it is with it. >> >> Sean. >> > > It wasn't just an implementation problem, it was in the design. IIRC, > Oracle's encryption allowed encrypted blocks to be deduplicated. There's > pretty much no way to defend against watermarking attacks with such a > design. Does the new encryption design have the same flaw? > I would ask the original developer that question (see the commit I linked to). The current dedup Implementation is terrible, so there are very few users of it. -M > > -Alan > From owner-freebsd-fs@freebsd.org Wed Aug 22 07:06:11 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9403D108F3A2; Wed, 22 Aug 2018 07:06:11 +0000 (UTC) (envelope-from outbackdingo@gmail.com) Received: from mail-wm0-x243.google.com (mail-wm0-x243.google.com [IPv6:2a00:1450:400c:c09::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 11AAE8609B; Wed, 22 Aug 2018 07:06:11 +0000 (UTC) (envelope-from outbackdingo@gmail.com) Received: by mail-wm0-x243.google.com with SMTP id o18-v6so1102872wmc.0; Wed, 22 Aug 2018 00:06:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Kv9NBQ1OU4NjWKQca39SP/buz3RX95dJPCu8EWhHrK8=; b=F2hC6QsqP7daI3Xh0dw7BLkVlGlcxp4Z7OtoJAViqTOPiUYFPlRRQwX16IcXu9vbZk rwsxKKWWzzSYFRahVBpHxDHWd8KwLMYodF8lZm82ItEqnikMZIhlOzp71YmMfZkMLQj8 ITs24Jx2IPcgRFX13fn39KvdxcAZcGT+k+69tAMvymPjoOeciS/sZ1pQkhTtwPyDYtXq Wq/QnPNhwMIrMkrfUW/yFcdU1WPvYYskxmI3ZlDo1uMk6Tx3uRO1u390G6ExovoWU7JM I86fWM2PoIp/vQMBiTGX3PyE3aUNQCDWmLQmLcK2mXyxYN5s+F6mOfuUfFXRLvMuVpa8 W+tg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Kv9NBQ1OU4NjWKQca39SP/buz3RX95dJPCu8EWhHrK8=; b=iNUzeE5872Uz8+TpS52SyaPa6HoevaPTnygYF5K80pO95pX6ultBu00B55RxHvj8X/ YYglSSB9VaRsrRyCUndEdJcFr+Sy5BQUAcqdnBuGpao1NzFMaJ0lZiz+fdimnj/h1DS7 XCfpzqKBHqZGzbyEq+xwSkHGCX4r2WVfAjHt7hBwGBlQ5ookRoGM+EuO37K12AoZsq4m 74Mn+KBSSSIAPmuk1lLkY0j46QTdsId1A3YzqTQpkGJi4Tl9raIf4k4eFVMKX0cFg3pq O+7ebyTXfc9WpI1LCJnX9Xj8eOMSQ82BnOgq+6OQ4WFhyLBWbxKjuWF/bpXBPQ1pT/JB JMNg== X-Gm-Message-State: APzg51C0SN2TZzCXE3sqyfZHM4dErWdNOcrM/qhnDklEgYsjl7pCp97W 5ej7d5cyy6luI1tJKnPYqmLZlsu+es4MduNh8aEqPiRp X-Google-Smtp-Source: ANB0VdawG0q1D8Yq8mOTpmdMRawETmIEsROCcVOaGcO1B1yNdvGHcRUzbxA3WzzThZYYuyOtl2OuT2odSEOzbqlr9rI= X-Received: by 2002:a1c:98a:: with SMTP id 132-v6mr1537862wmj.86.1534921569731; Wed, 22 Aug 2018 00:06:09 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Outback Dingo Date: Wed, 22 Aug 2018 09:05:33 +0200 Message-ID: Subject: Re: Native Encryption for ZFS on FreeBSD CFT To: mmacy@freebsd.org Cc: freebsd-current@freebsd.org, freebsd-fs@freebsd.org, sef@ixsystems.com Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 07:06:11 -0000 of course interesting work, but unfortunately, and as you know me, what would i say next cc -target x86_64-unknown-freebsd12.0 --sysroot=/usr/obj/usr/src/amd64.amd64/tmp -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin -O2 -pipe -I/usr/src/sys/cddl/compat/opensolaris -I/usr/src/cddl/compat/opensolaris/include -I/usr/src/cddl/compat/opensolaris/lib/libumem -I/usr/src/cddl/contrib/opensolaris/lib/libzpool/common -I/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs -I/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/lua -I/usr/src/sys/cddl/contrib/opensolaris/common/zfs -I/usr/src/sys/cddl/contrib/opensolaris/uts/common -I/usr/src/cddl/contrib/opensolaris/head -I/usr/src/cddl/contrib/opensolaris/lib/libnvpair -I/usr/src/cddl/contrib/opensolaris/lib/libcmdutils -DWANTS_MUTEX_OWNED -I/usr/src/lib/libpthread/thread -I/usr/src/lib/libpthread/sys -I/usr/src/lib/libthr/arch/amd64/include -g -DDEBUG=1 -DZFS_DEBUG=1 -DNEED_SOLARIS_BOOLEAN -g -MD -MF.depend.freebsd_crypto.o -MTfreebsd_crypto.o -std=iso9899:1999 -fstack-protector-strong -Wno-pointer-sign -Wno-unknown-pragmas -Wno-empty-body -Wno-string-plus-int -Wno-unused-const-variable -Wno-tautological-compare -Wno-unused-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-conversion -Wno-unused-local-typedef -Wno-address-of-packed-member -Wno-switch -Wno-switch-enum -Wno-knr-promoted-parameter -Wno-parentheses -Qunused-arguments -c /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/freebsd_crypto.c -o freebsd_crypto.o /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/freebsd_crypto.c:62:19: warning: tentative definition of variable with internal linkage has incomplete non-array type 'struct mtx' [-Wtentative-definition-incomplete-type] static struct mtx freebsd_crypto_mutex; ^ /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/freebsd_crypto.c:62:15: note: forward declaration of 'struct mtx' static struct mtx freebsd_crypto_mutex; ^ /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/freebsd_crypto.c:63:35: error: expected identifier MTX_SYSINIT(freebsd_crypto_mutex, &freebsd_crypto_mutex, "FreeBSD ZFS Crypto mutex", MTX_DEF); ^ /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/freebsd_crypto.c:63:1: warning: type specifier missing, defaults to 'int' [-Wimplicit-int] MTX_SYSINIT(freebsd_crypto_mutex, &freebsd_crypto_mutex, "FreeBSD ZFS Crypto mutex", MTX_DEF); ^ /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/freebsd_crypto.c:62:19: error: tentative definition has type 'struct mtx' that is never completed static struct mtx freebsd_crypto_mutex; ^ /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/freebsd_crypto.c:62:15: note: forward declaration of 'struct mtx' static struct mtx freebsd_crypto_mutex; ^ 2 warnings and 2 errors generated. *** Error code 1 On Wed, Aug 22, 2018 at 4:28 AM Matthew Macy wrote: > > On Tue, Aug 21, 2018 at 6:55 PM Matthew Macy wrote: > > > To anyone with an interest in native encryption in ZFS please test the > > projects/zfs-crypto-merge-0820 branch in my freebsd repo: > > https://github.com/mattmacy/networking.git > > > > > Oh and I neglected to state that this work is being supported by iX Systems > and the tree is all built on work done by Sean Fagan at iX Systems. Please > keep him in the loop on any problems encountered. > Thanks. > > > > > ( git clone https://github.com/mattmacy/networking.git -b > > projects/zfs-crypto-merge-0820 ) > > > > The UI is quite close to the Oracle Solaris ZFS crypto with minor > > differences for specifying key location. > > > > Please note that once a feature is enabled on a pool it can't be > > disabled. This means that if you enable encryption support on a pool > > you will never be able to import it in to a ZFS without encryption > > support. For this reason I would strongly advise against using this on > > any pool that can't be easily replaced until this change has made its > > way in to HEAD after the freeze has been lifted. > > > > > > By way of background the original ZoL commit can be found at: > > > > https://github.com/zfsonlinux/zfs/pull/5769/commits/5aef9bedc801830264428c64cd2242d1b786fd49 > > > > Thanks in advance. > > -M > > > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Wed Aug 22 07:10:30 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C8AEE108F65C; Wed, 22 Aug 2018 07:10:29 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7ABB28649B; Wed, 22 Aug 2018 07:10:29 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from mail-it0-f44.google.com (mail-it0-f44.google.com [209.85.214.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) (Authenticated sender: mmacy) by smtp.freebsd.org (Postfix) with ESMTPSA id 397E612120; Wed, 22 Aug 2018 07:10:29 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: by mail-it0-f44.google.com with SMTP id d16-v6so13095092itj.0; Wed, 22 Aug 2018 00:10:29 -0700 (PDT) X-Gm-Message-State: APzg51BNvH46zbazcb5xhnEsf+7LPRroCdIfKl6LQ2HuxOJKr+RwlZOD fT+ByiGiHHT9WGjHlSybe1xUhZfZgKGXNTN/JRM= X-Google-Smtp-Source: ANB0VdZHn6+WYWMY7qpNYAUqI43Rn/hVG2+2hyAEkKKaRrLKCzj8iFLsPNC/pJ+smqIS/9LE1SHZIv15HhW9MiqGHec= X-Received: by 2002:a24:704f:: with SMTP id f76-v6mr2161624itc.30.1534921828769; Wed, 22 Aug 2018 00:10:28 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Matthew Macy Date: Wed, 22 Aug 2018 00:10:16 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Native Encryption for ZFS on FreeBSD CFT To: Outback Dingo Cc: freebsd-current , freebsd-fs , Sean Fagan Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 07:10:30 -0000 Yes. I _just_ rebased and broke world in the process. Fix coming up momentarily. -M On Wed, Aug 22, 2018 at 12:06 AM Outback Dingo wrote: > of course interesting work, but unfortunately, and as you know me, > what would i say next > > cc -target x86_64-unknown-freebsd12.0 > --sysroot=/usr/obj/usr/src/amd64.amd64/tmp > -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin -O2 -pipe > -I/usr/src/sys/cddl/compat/opensolaris > -I/usr/src/cddl/compat/opensolaris/include > -I/usr/src/cddl/compat/opensolaris/lib/libumem > -I/usr/src/cddl/contrib/opensolaris/lib/libzpool/common > -I/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs > -I/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/lua > -I/usr/src/sys/cddl/contrib/opensolaris/common/zfs > -I/usr/src/sys/cddl/contrib/opensolaris/uts/common > -I/usr/src/cddl/contrib/opensolaris/head > -I/usr/src/cddl/contrib/opensolaris/lib/libnvpair > -I/usr/src/cddl/contrib/opensolaris/lib/libcmdutils > -DWANTS_MUTEX_OWNED -I/usr/src/lib/libpthread/thread > -I/usr/src/lib/libpthread/sys -I/usr/src/lib/libthr/arch/amd64/include > -g -DDEBUG=1 -DZFS_DEBUG=1 -DNEED_SOLARIS_BOOLEAN -g -MD > -MF.depend.freebsd_crypto.o -MTfreebsd_crypto.o -std=iso9899:1999 > -fstack-protector-strong -Wno-pointer-sign -Wno-unknown-pragmas > -Wno-empty-body -Wno-string-plus-int -Wno-unused-const-variable > -Wno-tautological-compare -Wno-unused-value -Wno-parentheses-equality > -Wno-unused-function -Wno-enum-conversion -Wno-unused-local-typedef > -Wno-address-of-packed-member -Wno-switch -Wno-switch-enum > -Wno-knr-promoted-parameter -Wno-parentheses -Qunused-arguments -c > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/freebsd_crypto.c > -o freebsd_crypto.o > > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/freebsd_crypto.c:62:19: > warning: tentative definition of variable with internal linkage has > incomplete non-array type 'struct mtx' > [-Wtentative-definition-incomplete-type] > static struct mtx freebsd_crypto_mutex; > ^ > > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/freebsd_crypto.c:62:15: > note: forward declaration of 'struct mtx' > static struct mtx freebsd_crypto_mutex; > ^ > > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/freebsd_crypto.c:63:35: > error: expected identifier > MTX_SYSINIT(freebsd_crypto_mutex, &freebsd_crypto_mutex, "FreeBSD ZFS > Crypto mutex", MTX_DEF); > ^ > > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/freebsd_crypto.c:63:1: > warning: type specifier missing, defaults to 'int' [-Wimplicit-int] > MTX_SYSINIT(freebsd_crypto_mutex, &freebsd_crypto_mutex, "FreeBSD ZFS > Crypto mutex", MTX_DEF); > ^ > > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/freebsd_crypto.c:62:19: > error: tentative definition has type 'struct mtx' that is never > completed > static struct mtx freebsd_crypto_mutex; > ^ > > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/freebsd_crypto.c:62:15: > note: forward declaration of 'struct mtx' > static struct mtx freebsd_crypto_mutex; > ^ > 2 warnings and 2 errors generated. > *** Error code 1 > On Wed, Aug 22, 2018 at 4:28 AM Matthew Macy wrote: > > > > On Tue, Aug 21, 2018 at 6:55 PM Matthew Macy wrote: > > > > > To anyone with an interest in native encryption in ZFS please test the > > > projects/zfs-crypto-merge-0820 branch in my freebsd repo: > > > https://github.com/mattmacy/networking.git > > > > > > > > Oh and I neglected to state that this work is being supported by iX > Systems > > and the tree is all built on work done by Sean Fagan at iX Systems. > Please > > keep him in the loop on any problems encountered. > > Thanks. > > > > > > > > > ( git clone https://github.com/mattmacy/networking.git -b > > > projects/zfs-crypto-merge-0820 ) > > > > > > The UI is quite close to the Oracle Solaris ZFS crypto with minor > > > differences for specifying key location. > > > > > > Please note that once a feature is enabled on a pool it can't be > > > disabled. This means that if you enable encryption support on a pool > > > you will never be able to import it in to a ZFS without encryption > > > support. For this reason I would strongly advise against using this on > > > any pool that can't be easily replaced until this change has made its > > > way in to HEAD after the freeze has been lifted. > > > > > > > > > By way of background the original ZoL commit can be found at: > > > > > > > https://github.com/zfsonlinux/zfs/pull/5769/commits/5aef9bedc801830264428c64cd2242d1b786fd49 > > > > > > Thanks in advance. > > > -M > > > > > _______________________________________________ > > freebsd-current@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-current > > To unsubscribe, send any mail to " > freebsd-current-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Wed Aug 22 07:21:12 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9FC4F108FBBA; Wed, 22 Aug 2018 07:21:12 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5124886CC0; Wed, 22 Aug 2018 07:21:12 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from mail-it0-f46.google.com (mail-it0-f46.google.com [209.85.214.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) (Authenticated sender: mmacy) by smtp.freebsd.org (Postfix) with ESMTPSA id 0E94412236; Wed, 22 Aug 2018 07:21:12 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: by mail-it0-f46.google.com with SMTP id d9-v6so1729152itf.2; Wed, 22 Aug 2018 00:21:12 -0700 (PDT) X-Gm-Message-State: AOUpUlFFBCePUTEAatVbVAimUB2zs+n2Aywx0GddzmdzWRv8N52a7BSZ PoJ/KZYBdbnFDTpCQz7109reZpwCh75TwFaiX0Y= X-Google-Smtp-Source: AA+uWPxCMEZCb3hTVFW5oeAVsJlMWP8wsgGjaEYPnFNk7ofYqKwAKyThOYXekE0NW7DHfv9V7YRqDCyGPHrbzaEFxZU= X-Received: by 2002:a02:88ad:: with SMTP id n42-v6mr7045573jaj.38.1534922471509; Wed, 22 Aug 2018 00:21:11 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Matthew Macy Date: Wed, 22 Aug 2018 00:20:59 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Native Encryption for ZFS on FreeBSD CFT To: Outback Dingo Cc: freebsd-current , freebsd-fs , Sean Fagan Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 07:21:12 -0000 Fixed. Pull. bc2b257d1082112cc27e56db793f5c569f603bec On Wed, Aug 22, 2018 at 12:10 AM Matthew Macy wrote: > Yes. I _just_ rebased and broke world in the process. Fix coming up > momentarily. > -M > > On Wed, Aug 22, 2018 at 12:06 AM Outback Dingo > wrote: > >> of course interesting work, but unfortunately, and as you know me, >> what would i say next >> >> cc -target x86_64-unknown-freebsd12.0 >> --sysroot=/usr/obj/usr/src/amd64.amd64/tmp >> -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin -O2 -pipe >> -I/usr/src/sys/cddl/compat/opensolaris >> -I/usr/src/cddl/compat/opensolaris/include >> -I/usr/src/cddl/compat/opensolaris/lib/libumem >> -I/usr/src/cddl/contrib/opensolaris/lib/libzpool/common >> -I/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs >> -I/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/lua >> -I/usr/src/sys/cddl/contrib/opensolaris/common/zfs >> -I/usr/src/sys/cddl/contrib/opensolaris/uts/common >> -I/usr/src/cddl/contrib/opensolaris/head >> -I/usr/src/cddl/contrib/opensolaris/lib/libnvpair >> -I/usr/src/cddl/contrib/opensolaris/lib/libcmdutils >> -DWANTS_MUTEX_OWNED -I/usr/src/lib/libpthread/thread >> -I/usr/src/lib/libpthread/sys -I/usr/src/lib/libthr/arch/amd64/include >> -g -DDEBUG=1 -DZFS_DEBUG=1 -DNEED_SOLARIS_BOOLEAN -g -MD >> -MF.depend.freebsd_crypto.o -MTfreebsd_crypto.o -std=iso9899:1999 >> -fstack-protector-strong -Wno-pointer-sign -Wno-unknown-pragmas >> -Wno-empty-body -Wno-string-plus-int -Wno-unused-const-variable >> -Wno-tautological-compare -Wno-unused-value -Wno-parentheses-equality >> -Wno-unused-function -Wno-enum-conversion -Wno-unused-local-typedef >> -Wno-address-of-packed-member -Wno-switch -Wno-switch-enum >> -Wno-knr-promoted-parameter -Wno-parentheses -Qunused-arguments -c >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/freebsd_crypto.c >> -o freebsd_crypto.o >> >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/freebsd_crypto.c:62:19: >> warning: tentative definition of variable with internal linkage has >> incomplete non-array type 'struct mtx' >> [-Wtentative-definition-incomplete-type] >> static struct mtx freebsd_crypto_mutex; >> ^ >> >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/freebsd_crypto.c:62:15: >> note: forward declaration of 'struct mtx' >> static struct mtx freebsd_crypto_mutex; >> ^ >> >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/freebsd_crypto.c:63:35: >> error: expected identifier >> MTX_SYSINIT(freebsd_crypto_mutex, &freebsd_crypto_mutex, "FreeBSD ZFS >> Crypto mutex", MTX_DEF); >> ^ >> >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/freebsd_crypto.c:63:1: >> warning: type specifier missing, defaults to 'int' [-Wimplicit-int] >> MTX_SYSINIT(freebsd_crypto_mutex, &freebsd_crypto_mutex, "FreeBSD ZFS >> Crypto mutex", MTX_DEF); >> ^ >> >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/freebsd_crypto.c:62:19: >> error: tentative definition has type 'struct mtx' that is never >> completed >> static struct mtx freebsd_crypto_mutex; >> ^ >> >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/freebsd_crypto.c:62:15: >> note: forward declaration of 'struct mtx' >> static struct mtx freebsd_crypto_mutex; >> ^ >> 2 warnings and 2 errors generated. >> *** Error code 1 >> On Wed, Aug 22, 2018 at 4:28 AM Matthew Macy wrote: >> > >> > On Tue, Aug 21, 2018 at 6:55 PM Matthew Macy wrote: >> > >> > > To anyone with an interest in native encryption in ZFS please test the >> > > projects/zfs-crypto-merge-0820 branch in my freebsd repo: >> > > https://github.com/mattmacy/networking.git >> > > >> > > >> > Oh and I neglected to state that this work is being supported by iX >> Systems >> > and the tree is all built on work done by Sean Fagan at iX Systems. >> Please >> > keep him in the loop on any problems encountered. >> > Thanks. >> > >> > >> > >> > > ( git clone https://github.com/mattmacy/networking.git -b >> > > projects/zfs-crypto-merge-0820 ) >> > > >> > > The UI is quite close to the Oracle Solaris ZFS crypto with minor >> > > differences for specifying key location. >> > > >> > > Please note that once a feature is enabled on a pool it can't be >> > > disabled. This means that if you enable encryption support on a pool >> > > you will never be able to import it in to a ZFS without encryption >> > > support. For this reason I would strongly advise against using this on >> > > any pool that can't be easily replaced until this change has made its >> > > way in to HEAD after the freeze has been lifted. >> > > >> > > >> > > By way of background the original ZoL commit can be found at: >> > > >> > > >> https://github.com/zfsonlinux/zfs/pull/5769/commits/5aef9bedc801830264428c64cd2242d1b786fd49 >> > > >> > > Thanks in advance. >> > > -M >> > > >> > _______________________________________________ >> > freebsd-current@freebsd.org mailing list >> > https://lists.freebsd.org/mailman/listinfo/freebsd-current >> > To unsubscribe, send any mail to " >> freebsd-current-unsubscribe@freebsd.org" >> > From owner-freebsd-fs@freebsd.org Wed Aug 22 10:56:19 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 38D391084FD2 for ; Wed, 22 Aug 2018 10:56:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id CA59F8EBA2 for ; Wed, 22 Aug 2018 10:56:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 8BABA1084FD1; Wed, 22 Aug 2018 10:56:18 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7A70C1084FD0 for ; Wed, 22 Aug 2018 10:56:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1C4388EB9D for ; Wed, 22 Aug 2018 10:56:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 4B85314256 for ; Wed, 22 Aug 2018 10:56:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7MAuHYh067263 for ; Wed, 22 Aug 2018 10:56:17 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7MAuHR4067261 for fs@FreeBSD.org; Wed, 22 Aug 2018 10:56:17 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230752] panic: excl->share in newnfs_request Date: Wed, 22 Aug 2018 10:56:16 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: panic X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: dim@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 10:56:19 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230752 --- Comment #4 from Dimitry Andric --- (In reply to Rick Macklem from comment #3) > Sorry, to clarify...I was wondering if the machine that panic()'d > had its /var/run exported to NFS client(s) that would have it mounted via > NFS. > (If that was the case, it could be a case where the nfsd thread is doing > a Getattr on /var/run/nfsuserd.sock and then it tries to connect to > /var/run/nfsuserd.sock to do the upcall to the nfsuserd.) > --> I think I'll need to make it still use UDP sockets by default (since > this could happen when the socket is in the namespace). No, it's only exporting other directories. /etc/exports looks like this: /archive -maproot=3Dnfsnobody:nfsnobody -network 192.168.0.1/24 /share -maproot=3Dnfsnobody:nfsnobody -network 192.168.0.1/24 /usr/obj /usr/src -maproot=3Dnfsnobody:nfsnobody -network 192.168.0.1/24 V4: / -network 192.168.0.1/24 The following nfs related settings are in /etc/rc.conf: nfs_client_enable=3D"NO" # This host is an NFS client (or NO). nfs_server_enable=3D"YES" # This host is an NFS server (or NO). nfsv4_server_enable=3D"YES" # Enable support for NFSv4 nfscbd_enable=3D"NO" # NFSv4 client side callback daemon nfsuserd_enable=3D"YES" # NFSv4 user/group name mapping daemon --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Aug 22 11:51:06 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 625B21086274 for ; Wed, 22 Aug 2018 11:51:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 00194702BC for ; Wed, 22 Aug 2018 11:51:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id B92E41086273; Wed, 22 Aug 2018 11:51:05 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A80EE1086272 for ; Wed, 22 Aug 2018 11:51:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ACFC702B7 for ; Wed, 22 Aug 2018 11:51:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id A97F814A5A for ; Wed, 22 Aug 2018 11:51:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7MBp4Rt097240 for ; Wed, 22 Aug 2018 11:51:04 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7MBp4hK097238 for fs@FreeBSD.org; Wed, 22 Aug 2018 11:51:04 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230752] panic: excl->share in newnfs_request Date: Wed, 22 Aug 2018 11:51:04 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: panic X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: rmacklem@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 11:51:06 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230752 --- Comment #5 from Rick Macklem --- Hmm. I notice you do have the "V4: / =E2=80=A6" line in your /etc/exports. Do you know if there are clients doing an NFSv4 mount from "/"? Something like: mount -t nfs -o nfsv4 nfs-client:/ /mnt I think I will end up reverting the "use AF_LOCAL" socket patch, since that is what is causing the soconnect() on the socket and should definitely make the panic()s go away. (It did fix a problem with using UDP when jails were enabled, but that can be fixed less elegantly with a patch that adds a command line argument for an alternate IP# instead of 127.0.0.1.) --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Aug 22 12:20:22 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7E3511087564 for ; Wed, 22 Aug 2018 12:20:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id A2DBD7164F for ; Wed, 22 Aug 2018 12:20:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 67E6E1087563; Wed, 22 Aug 2018 12:20:21 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 56DF51087562 for ; Wed, 22 Aug 2018 12:20:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D24D671641 for ; Wed, 22 Aug 2018 12:20:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 3653C14D77 for ; Wed, 22 Aug 2018 12:20:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7MCKK73024777 for ; Wed, 22 Aug 2018 12:20:20 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7MCKKMs024773 for fs@FreeBSD.org; Wed, 22 Aug 2018 12:20:20 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230752] panic: excl->share in newnfs_request Date: Wed, 22 Aug 2018 12:20:20 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: panic X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 12:20:22 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230752 --- Comment #6 from commit-hook@freebsd.org --- A commit references this bug: Author: rmacklem Date: Wed Aug 22 12:20:10 UTC 2018 New revision: 338192 URL: https://svnweb.freebsd.org/changeset/base/338192 Log: Revert r320757 since it can cause "excl->shared" panics. PR#230752 shows a panic where an nfsd thread tries to do soconnect() on the AF_LOCAL socket used by the nfsuserd while already holding an exclusive lock on it. I am not 100% sure how this happens, but since an AF_LOCAL socket is in the file system namespace it is conceivable that it could lock it and then attempt an upcall to the nfsuserd. However, reverting r320757 stops the nfsuserd from using an AF_LOCAL socket, so it should avoid any such panic(). r320757 did fix a problem with running the nfsuserd when jails were enabled, but that can be dealt with less elegantly by allowing the use of an alternate address instead of 127.0.0.1. The gssd daemon also uses an AF_LOCAL socket, but it will do upcalls before the nfsd thread processes the RPC, so I think it should not be suseptible to this problem. PR: 230752 Changes: head/usr.sbin/nfsuserd/nfsuserd.c --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Aug 22 12:26:29 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 98D1B1087A42 for ; Wed, 22 Aug 2018 12:26:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id C933171BDB for ; Wed, 22 Aug 2018 12:26:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 8E37C1087A3D; Wed, 22 Aug 2018 12:26:28 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7D29C1087A3C for ; Wed, 22 Aug 2018 12:26:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1988A71BD8 for ; Wed, 22 Aug 2018 12:26:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 607EA14EE8 for ; Wed, 22 Aug 2018 12:26:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7MCQRjA039743 for ; Wed, 22 Aug 2018 12:26:27 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7MCQRhL039742 for fs@FreeBSD.org; Wed, 22 Aug 2018 12:26:27 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230752] panic: excl->share in newnfs_request Date: Wed, 22 Aug 2018 12:26:27 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: panic X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 12:26:29 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230752 --- Comment #7 from commit-hook@freebsd.org --- A commit references this bug: Author: rmacklem Date: Wed Aug 22 12:26:18 UTC 2018 New revision: 338193 URL: https://svnweb.freebsd.org/changeset/base/338193 Log: Revert r320758, which was the man page update for r320757 just reverted. This is a content change. PR: 230752 Changes: head/usr.sbin/nfsuserd/nfsuserd.8 --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Aug 22 16:33:49 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E9495108E035 for ; Wed, 22 Aug 2018 16:33:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 85FE07D38E for ; Wed, 22 Aug 2018 16:33:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 477CC108E02E; Wed, 22 Aug 2018 16:33:48 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 36487108E02C for ; Wed, 22 Aug 2018 16:33:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CC5697D388 for ; Wed, 22 Aug 2018 16:33:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 25FD51717D for ; Wed, 22 Aug 2018 16:33:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7MGXlbX019425 for ; Wed, 22 Aug 2018 16:33:47 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7MGXl4q019424 for fs@FreeBSD.org; Wed, 22 Aug 2018 16:33:47 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 229887] zfs quota related panic: solaris assert: 0 == dmu_tx_assign(tx, TXG_WAIT), Date: Wed, 22 Aug 2018 16:33:47 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 16:33:49 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D229887 --- Comment #2 from commit-hook@freebsd.org --- A commit references this bug: Author: mav Date: Wed Aug 22 16:32:53 UTC 2018 New revision: 338206 URL: https://svnweb.freebsd.org/changeset/base/338206 Log: Add dmu_tx_assign() error handling in zfs_unlinked_drain(). The error handling got lost during r334810, while according to the report error there may happen in case of dataset being over quota. In such case just leave the node in the unlinked list to be freed sometimes later. PR: 229887 Sponsored by: iXsystems, Inc. Changes: head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_dir.c --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Aug 22 16:35:01 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0D00E108E151 for ; Wed, 22 Aug 2018 16:35:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 9D97A7D4AC for ; Wed, 22 Aug 2018 16:35:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 62D2C108E150; Wed, 22 Aug 2018 16:35:00 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 516D6108E14F for ; Wed, 22 Aug 2018 16:35:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E6B9A7D4A8 for ; Wed, 22 Aug 2018 16:34:59 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 274EC17184 for ; Wed, 22 Aug 2018 16:34:59 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7MGYx7g020574 for ; Wed, 22 Aug 2018 16:34:59 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7MGYx0d020573 for fs@FreeBSD.org; Wed, 22 Aug 2018 16:34:59 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 229887] zfs quota related panic: solaris assert: 0 == dmu_tx_assign(tx, TXG_WAIT), Date: Wed, 22 Aug 2018 16:34:59 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mav@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: mav@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status assigned_to resolution Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 16:35:01 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D229887 Alexander Motin changed: What |Removed |Added ---------------------------------------------------------------------------- Status|New |Closed Assignee|fs@FreeBSD.org |mav@FreeBSD.org Resolution|--- |FIXED --- Comment #3 from Alexander Motin --- I was unable to reproduce the problem, but I believe committed patch should= fix it. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Aug 22 17:11:52 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 643CB108F998 for ; Wed, 22 Aug 2018 17:11:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 000467FA34 for ; Wed, 22 Aug 2018 17:11:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id B9720108F997; Wed, 22 Aug 2018 17:11:51 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A819B108F996 for ; Wed, 22 Aug 2018 17:11:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 465297FA2A for ; Wed, 22 Aug 2018 17:11:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 954AD176F6 for ; Wed, 22 Aug 2018 17:11:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7MHBo7e016634 for ; Wed, 22 Aug 2018 17:11:50 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7MHBoP2016608 for fs@FreeBSD.org; Wed, 22 Aug 2018 17:11:50 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230752] panic: excl->share in newnfs_request Date: Wed, 22 Aug 2018 17:11:50 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: panic X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: dim@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 17:11:52 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230752 --- Comment #8 from Dimitry Andric --- (In reply to Rick Macklem from comment #5) > Hmm. I notice you do have the "V4: / =E2=80=A6" line in your /etc/exports. > Do you know if there are clients doing an NFSv4 mount from "/"? > Something like: > mount -t nfs -o nfsv4 nfs-client:/ /mnt Yes, most clients use NFSv4, via /etc/fstab lines: nfssrv:/archive /archive nfs rw,intr,soft,nfsv4 0 0 nfssrv:/share /share nfs rw,intr,soft,nfsv4 0 0 I migrated most of them from NFSv3 a month or two ago. This panic started appearing relatively recently, for example when doing buildworlds from /sha= re (where freebsd sources are). > I think I will end up reverting the "use AF_LOCAL" socket patch, > since that is what is causing the soconnect() on the socket and > should definitely make the panic()s go away. > (It did fix a problem with using UDP when jails were enabled, but > that can be fixed less elegantly with a patch that adds a command > line argument for an alternate IP# instead of 127.0.0.1.) I'll check it out, thanks. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Aug 22 18:30:11 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6EF5C10918D2 for ; Wed, 22 Aug 2018 18:30:11 +0000 (UTC) (envelope-from sef@ixsystems.com) Received: from mail-pg1-x535.google.com (mail-pg1-x535.google.com [IPv6:2607:f8b0:4864:20::535]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E28BE83554 for ; Wed, 22 Aug 2018 18:30:10 +0000 (UTC) (envelope-from sef@ixsystems.com) Received: by mail-pg1-x535.google.com with SMTP id m3-v6so387557pgp.6 for ; Wed, 22 Aug 2018 11:30:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ixsystems-com.20150623.gappssmtp.com; s=20150623; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=5bQQP2lkwg92kML+IoMdcz/5c9zPBB0qC3JdQIS4MlQ=; b=ck553h83UCxcqDHzUVf/6nQhTcUCGMAmEQzf6rZCrCUl5nCMHn/QqXH5UkwieRV/hF nGVNmimFeT7cBTLObq2YC18a221j6SpXknm017MApI+ll9SPYFoOHv1UIIWNYdq7UGUA mnop5VkIaopR3EYa8a0pxQlWyFwH9GiRCrqCn5Epv/cspBSz5rxm4e+LZbMxgROVxEHV Ac97FwEj62JweQuDHI2g7dHhbbcIJDyPKY1bEz93itq1OaYN+RDhD7xWet1euRJN60jS 5F/Gmrgdyb4GcT8D3QN94HXCnScSt4ica1/FPIMEgvC9e0WV7XDYsav/gdPcZeXctX6/ X5rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=5bQQP2lkwg92kML+IoMdcz/5c9zPBB0qC3JdQIS4MlQ=; b=jgkJnweE/6XmKmZ+92q+xalioSsxwo92MIfNoPNISHaQE9EBJzC4stOZLsYCmZEZw+ kCsXbIkD/ceVn9BKjY+7Jn3gi0/7QWrXsDcJM3LgKj8PFog9Xa02XjfXozcalfKK8oZc VKYseOyapLBoCHlnL7tqh6aiSbbsmcpmW3HG51UVruG4ABnaw7fpojpD682jSBLJxEK0 Iq47ri548WWtp0sh4T3mbLcFMIhWnP8BIXYV8bfcI5eqT36XCSrPn/zre9IjwY0BMjDV UbVFcWMJGy1Kgl6MuUUkaDQ/UDy+AxDx+lbCf8xwt7r0EA8ckD6Az787tctZaH0RBEjS yCZA== X-Gm-Message-State: APzg51Csw5s1gl+XzCneyy7+QR5tT87dCGdo0PSKhdYC8qaNoRFG6Vok zmSra4ZSKAcmFiPC7Nhsn4tzzg== X-Google-Smtp-Source: ANB0VdaeQMKvfYHml2udXcqguR3LDNGU+XFfxHD7J2XqC3B3/jydTZvo94Mvr4IpE1SZkBF2wIG1Og== X-Received: by 2002:a62:384a:: with SMTP id f71-v6mr7929340pfa.48.1534962610065; Wed, 22 Aug 2018 11:30:10 -0700 (PDT) Received: from [10.250.1.167] ([12.229.62.29]) by smtp.gmail.com with ESMTPSA id k1-v6sm2950804pfi.62.2018.08.22.11.30.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 22 Aug 2018 11:30:08 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: Native Encryption for ZFS on FreeBSD CFT From: Sean Fagan In-Reply-To: Date: Wed, 22 Aug 2018 11:30:07 -0700 Cc: Matthew Macy , FreeBSD CURRENT , freebsd-fs Content-Transfer-Encoding: quoted-printable Message-Id: <9FDF249A-E320-4652-834E-7EEC5C4FB7CA@ixsystems.com> References: To: Alan Somers X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 18:30:11 -0000 On Aug 21, 2018, at 8:16 PM, Alan Somers wrote: >=20 > > The last time I looked (which was a long time ago), Oracle's ZFS = encryption looked extremely vulnerable to watermarking attacks. Did = anybody ever fix that? This is the comment about dedup in zio_crypt.c: * CONSIDERATIONS FOR DEDUP: * In order for dedup to work, blocks that we want to dedup with one = another * need to use the same IV and encryption key, so that they will have = the same * ciphertext. Normally, one should never reuse an IV with the same = encryption * key or else AES-GCM and AES-CCM can both actually leak the plaintext = of both * blocks. In this case, however, since we are using the same plaintext = as * well all that we end up with is a duplicate of the original = ciphertext we * already had. As a result, an attacker with read access to the raw = disk will * be able to tell which blocks are the same but this information is = given away * by dedup anyway. In order to get the same IVs and encryption keys for * equivalent blocks of data we use an HMAC of the plaintext. We use an = HMAC * here so that a reproducible checksum of the plaintext is never = available to * the attacker. The HMAC key is kept alongside the master key, = encrypted on * disk. The first 64 bits of the HMAC are used in place of the random = salt, and * the next 96 bits are used as the IV. As a result of this mechanism, = dedup * will only work within a clone family since encrypted dedup requires = use of * the same master and HMAC keys. (So, same issue. I don=E2=80=99t think encryption and deduplication = should live together, so I would not have made that choice.) Sean. From owner-freebsd-fs@freebsd.org Wed Aug 22 19:23:17 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 93EBD1092D1C for ; Wed, 22 Aug 2018 19:23:17 +0000 (UTC) (envelope-from sef@ixsystems.com) Received: from mail-pg1-x52a.google.com (mail-pg1-x52a.google.com [IPv6:2607:f8b0:4864:20::52a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0FCD485F85 for ; Wed, 22 Aug 2018 19:23:17 +0000 (UTC) (envelope-from sef@ixsystems.com) Received: by mail-pg1-x52a.google.com with SMTP id h17-v6so1366558pgv.3 for ; Wed, 22 Aug 2018 12:23:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ixsystems-com.20150623.gappssmtp.com; s=20150623; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=D/sOF9M653EyysrHa6kRvzooofB4wXSSbyiXGMKzKho=; b=eYVaWtL1ri+DsgOhZ//EhWj4+Tu/cLRI9J/yuVEb8eLeMiXK5HjZslNIuQLzFhJRIo 7835/ebDJxUaQ4VGS+28XYYX/JzaQ6QvsiLRNvqlUqAZ8xaFBFDUBFqrTf0yXkG+g5eb rYgyXZXc/2NpspVbPA8SXfg5hte4r2E9CYMHAwKrwTWMcf8KpCl6npCLRrGDic4r+tiH liLbRQxem8ZX6srx0HGrDWGjD0pKc5i/osalTEO+TKBw8LlgeeLCRAf48QaB6pdhPwI2 M/DJ2tS5nW5NcVkGdBbPqLVQhFFERfGpgzAiNSU+X/EM61eL6Mzygcvs1iUQGUIxJgYa b5mQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=D/sOF9M653EyysrHa6kRvzooofB4wXSSbyiXGMKzKho=; b=oMFYtVAcCt+Ee+17pdpVAxOh2ibc+9kxLaxXvgbpnA5wZeUXot8T3GsKcsa7pXZrDa nlPhRknNunzhabHI6I0BSXPLjeRFMSH6OS2jkYmpB/cJLbYRUWFNrvskszGhuc32QsJB dsSWITl/SKvC8lacphUIeceURM+9GJdUsXoZ7BDD+X48Ro6Zb2jQ50wIZ8Caj2k/Ux/5 nE4TUYSn5+tfmAEcnEcKX64Npnm0GdaKiHCLQGvz729AKpOhxWZnui2NVW5Fqx74+fv5 QIDkRH5/s0YkzIayfitI6erPJn4fBCuHAqSS03lY+Cy4i606yEI5t2Pi1J7MMzuYsL9M GPPA== X-Gm-Message-State: AOUpUlEeZqldO7Y3jRQuzG9Ziqe0g3c2l4zMGWLmBmMaCp/ag6xTYcI8 9X1ecCzpQdZ4jTaluqy8oSV33w== X-Google-Smtp-Source: AA+uWPy7qBnVsW/ddNGHTfD8QVfJR8B8yh+lvBsj2w0I+v7RXU2FzSkYkz9cgPgts9KVw15fI7rsug== X-Received: by 2002:a63:2b82:: with SMTP id r124-v6mr9594282pgr.159.1534965796135; Wed, 22 Aug 2018 12:23:16 -0700 (PDT) Received: from [10.250.1.167] ([12.229.62.29]) by smtp.gmail.com with ESMTPSA id h130-v6sm6345969pgc.88.2018.08.22.12.23.14 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 22 Aug 2018 12:23:15 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: Native Encryption for ZFS on FreeBSD CFT From: Sean Fagan In-Reply-To: Date: Wed, 22 Aug 2018 12:23:13 -0700 Cc: Matthew Macy , FreeBSD CURRENT , freebsd-fs Content-Transfer-Encoding: quoted-printable Message-Id: References: <9FDF249A-E320-4652-834E-7EEC5C4FB7CA@ixsystems.com> To: Alan Somers X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 19:23:17 -0000 On Aug 22, 2018, at 12:20 PM, Alan Somers wrote: > ]That doesn't answer the question about what happens when dedup is = turned off. In that case, is the HMAC still used as the IV? If so, = then watermarking attacks are still possible. If ZFS switches to a = random IV when dedup is off, then it would probably be ok. =46rom the same file: * Initialization Vector (IV): = =20 * An initialization vector for the encryption algorithms. This is used = to =20 * "tweak" the encryption algorithms so that two blocks of the same data = are =20 * encrypted into different ciphertext outputs, thus obfuscating block = patterns. =20 * The supported encryption modes (AES-GCM and AES-CCM) require that an = IV is =20 * never reused with the same encryption key. This value is stored = unencrypted =20 * and must simply be provided to the decryption function. We use a 96 = bit IV =20 * (as recommended by NIST) for all block encryption. For non-dedup = blocks we =20 * derive the IV randomly. The first 64 bits of the IV are stored in the = second =20 * word of DVA[2] and the remaining 32 bits are stored in the upper 32 = bits of =20 * blk_fill. This is safe because encrypted blocks can't use the upper = 32 bits =20 * of blk_fill. We only encrypt level 0 blocks, which normally have a = fill count =20 * of 1. The only exception is for DMU_OT_DNODE objects, where the fill = count of =20 * level 0 blocks is the number of allocated dnodes in that block. The = on-disk =20 * format supports at most 2^15 slots per L0 dnode block, because the = maximum =20 * block size is 16MB (2^24). In either case, for level 0 blocks this = number =20 * will still be smaller than UINT32_MAX so it is safe to store the IV = in the =20 * top 32 bits of blk_fill, while leaving the bottom 32 bits of the fill = count =20 * for the dnode code. = =20 Sean From owner-freebsd-fs@freebsd.org Wed Aug 22 19:41:57 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BE89F10937EE; Wed, 22 Aug 2018 19:41:57 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-lf1-f53.google.com (mail-lf1-f53.google.com [209.85.167.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2D4FA87177; Wed, 22 Aug 2018 19:41:56 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-lf1-f53.google.com with SMTP id l26-v6so2262886lfc.8; Wed, 22 Aug 2018 12:41:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=RD5GRRHc6X1znKWsJ+9utq8OFRZ9jwTiwJ24c16bRXE=; b=lCP/UYHHscX5PD+JfiP4J8DSqMnWki/u0CoG1Z99PWf7kTKYp87H5UCMUhHxxaUnO9 7ysc/0MpyQnOP/h04JPAEpGiq6ZywoUxUJkwIS8nMhV+F2y1lmG4Yyu/Jv1v2Xv4vTWm TaSQDlqxJsBTdDr7aJa8ZzPfX4II42OViNnsSS8jrRA4SmQeWIyPomgKPf4tQilOPnn2 KEBIontL/1rR+1tx72jXO2ozFLPIa37ipdOKf7zg+Vad1/9qxwjU1JvjSZ8KaaGjmkA7 C0t01CIQYbEa1tW7yQPoNRtW+tZzwrWiMb6yFYC9eOOsJpPFufPfj1oFQM21z1FCx4En H4AQ== X-Gm-Message-State: APzg51Bs1lvCmYnH+eZpEhGqVUgAxwEOq0hQUtwigY90EiSytmhH/hxN fPbsf0DnGRD/8yp+ERNYUvTsIdCKhvCPwX9ck5U= X-Google-Smtp-Source: ANB0VdazmXcqfQa30ukVrSF12oG5LmKPKkbyoSfSsZh7OXe46SQTW47gB6ZwLwFLTzhGMoaavcB2fyLJzZD2zPad8xA= X-Received: by 2002:a19:6619:: with SMTP id a25-v6mr3156876lfc.62.1534966561322; Wed, 22 Aug 2018 12:36:01 -0700 (PDT) MIME-Version: 1.0 References: <9FDF249A-E320-4652-834E-7EEC5C4FB7CA@ixsystems.com> In-Reply-To: From: Alan Somers Date: Wed, 22 Aug 2018 13:35:48 -0600 Message-ID: Subject: Re: Native Encryption for ZFS on FreeBSD CFT To: Sean Fagan Cc: Matthew Macy , FreeBSD CURRENT , freebsd-fs Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 19:41:58 -0000 Only encrypting L0 blocks also leaks a lot of information. That means that, if encryption is set to anything but "off", watermarking attacks will still be possible based on the size and sparsity of a file. Because I believe that with any encryption mode, ZFS turns continuous runs of zeros into holes. And I don't see anything in zio_crypt.c that addresses that. -Alan On Wed, Aug 22, 2018 at 1:23 PM Sean Fagan wrote: > On Aug 22, 2018, at 12:20 PM, Alan Somers wrote: > > ]That doesn't answer the question about what happens when dedup is > turned off. In that case, is the HMAC still used as the IV? If so, then > watermarking attacks are still possible. If ZFS switches to a random IV > when dedup is off, then it would probably be ok. > > From the same file: > > * Initialization Vector (IV): > > * An initialization vector for the encryption algorithms. This is used > to > * "tweak" the encryption algorithms so that two blocks of the same data > are > * encrypted into different ciphertext outputs, thus obfuscating block > patterns. > * The supported encryption modes (AES-GCM and AES-CCM) require that an IV > is > * never reused with the same encryption key. This value is stored > unencrypted > * and must simply be provided to the decryption function. We use a 96 bit > IV > * (as recommended by NIST) for all block encryption. For non-dedup blocks > we > * derive the IV randomly. The first 64 bits of the IV are stored in the > second > * word of DVA[2] and the remaining 32 bits are stored in the upper 32 > bits of > * blk_fill. This is safe because encrypted blocks can't use the upper 32 > bits > * of blk_fill. We only encrypt level 0 blocks, which normally have a fill > count > * of 1. The only exception is for DMU_OT_DNODE objects, where the fill > count of > * level 0 blocks is the number of allocated dnodes in that block. The > on-disk > * format supports at most 2^15 slots per L0 dnode block, because the > maximum > * block size is 16MB (2^24). In either case, for level 0 blocks this > number > * will still be smaller than UINT32_MAX so it is safe to store the IV in > the > * top 32 bits of blk_fill, while leaving the bottom 32 bits of the fill > count > * for the dnode code. > > > Sean > > > From owner-freebsd-fs@freebsd.org Wed Aug 22 19:46:31 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B17471093B34; Wed, 22 Aug 2018 19:46:31 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-lf1-f51.google.com (mail-lf1-f51.google.com [209.85.167.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2A7AD88045; Wed, 22 Aug 2018 19:46:31 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-lf1-f51.google.com with SMTP id j8-v6so2282605lfb.4; Wed, 22 Aug 2018 12:46:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=++NMJdGm2+cP9X5SmSlpRrJLyHTEQsGPgyG7tJMe5h8=; b=U9nBVhuiCQoX3z/ZbrhlrlFeQOIyVUnzi+cPwU4AuNivhaJddTLbh67BDLApFoxOj0 17t4/W4ei3AwzcSt+RXrisy4n+lKrwdjvwy7vqYaHyhZKYt7KPcxsF9d+V7X1BF/MBmh ZpUOj5qhGLXqZMIIROiAB4HwXgKd7OtkSBf+vy8nKty7T9u0ClOYXLc0k8fDlsbl9pAD s7rCTx/gL3SChd4WSjn3oDfsjrXN3QUaWUpLsg3/FTgnIemLiqwswNIQiFPQtw/lRjz0 ITQOLQZeXn/xIPuEovEEG9z9ax+LZcoolnf/AEe0+bfouxw8gbKiXvmdJ30a0MC3EaOi /VAg== X-Gm-Message-State: AOUpUlHMUQP8qz1hER5izdAw6eWS5IFe8Wre63sTY0tymqjEPlXc/DyR TGNkbY5CE1W+/LF9NOdpVDg3hqPr8yotNJNh05bf0A== X-Google-Smtp-Source: AA+uWPywVq7Knf1+i50rBYlrksxiyvFp4J8AjC1HFsXGRpNDhmn9WWKOOCAuLSikNgG1LrIPaO4x8J1uafv2jehzb6E= X-Received: by 2002:a19:1c4c:: with SMTP id c73-v6mr10783055lfc.90.1534965641770; Wed, 22 Aug 2018 12:20:41 -0700 (PDT) MIME-Version: 1.0 References: <9FDF249A-E320-4652-834E-7EEC5C4FB7CA@ixsystems.com> In-Reply-To: <9FDF249A-E320-4652-834E-7EEC5C4FB7CA@ixsystems.com> From: Alan Somers Date: Wed, 22 Aug 2018 13:20:29 -0600 Message-ID: Subject: Re: Native Encryption for ZFS on FreeBSD CFT To: Sean Fagan Cc: Matthew Macy , FreeBSD CURRENT , freebsd-fs Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 19:46:32 -0000 On Wed, Aug 22, 2018 at 12:30 PM Sean Fagan wrote: > On Aug 21, 2018, at 8:16 PM, Alan Somers wrote: > > > > > The last time I looked (which was a long time ago), Oracle's ZFS > encryption looked extremely vulnerable to watermarking attacks. Did > anybody ever fix that? > > This is the comment about dedup in zio_crypt.c: > > * CONSIDERATIONS FOR DEDUP: > * In order for dedup to work, blocks that we want to dedup with one > another > * need to use the same IV and encryption key, so that they will have the > same > * ciphertext. Normally, one should never reuse an IV with the same > encryption > * key or else AES-GCM and AES-CCM can both actually leak the plaintext o= f > both > * blocks. In this case, however, since we are using the same plaintext a= s > * well all that we end up with is a duplicate of the original ciphertext > we > * already had. As a result, an attacker with read access to the raw disk > will > * be able to tell which blocks are the same but this information is give= n > away > * by dedup anyway. In order to get the same IVs and encryption keys for > * equivalent blocks of data we use an HMAC of the plaintext. We use an > HMAC > * here so that a reproducible checksum of the plaintext is never > available to > * the attacker. The HMAC key is kept alongside the master key, encrypted > on > * disk. The first 64 bits of the HMAC are used in place of the random > salt, and > * the next 96 bits are used as the IV. As a result of this mechanism, > dedup > * will only work within a clone family since encrypted dedup requires us= e > of > * the same master and HMAC keys. > > (So, same issue. I don=E2=80=99t think encryption and deduplication shou= ld live > together, > so I would not have made that choice.) > > Sean. > That doesn't answer the question about what happens when dedup is turned off. In that case, is the HMAC still used as the IV? If so, then watermarking attacks are still possible. If ZFS switches to a random IV when dedup is off, then it would probably be ok. -Alan From owner-freebsd-fs@freebsd.org Wed Aug 22 19:46:55 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 21B0D1093B91 for ; Wed, 22 Aug 2018 19:46:55 +0000 (UTC) (envelope-from sef@ixsystems.com) Received: from mail-pg1-x533.google.com (mail-pg1-x533.google.com [IPv6:2607:f8b0:4864:20::533]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 96B77880A5 for ; Wed, 22 Aug 2018 19:46:54 +0000 (UTC) (envelope-from sef@ixsystems.com) Received: by mail-pg1-x533.google.com with SMTP id v66-v6so1379265pgb.10 for ; Wed, 22 Aug 2018 12:46:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ixsystems-com.20150623.gappssmtp.com; s=20150623; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=NOv7n72cDyWUU7iJNV38AFpKId3+/WvjhxWV1Nrhajo=; b=Xbc+ow6mfHnVEE2ATrYMlU/GaFjnSxEAWHW9pHeKaFcGU6bpq3BaXtwIingm83bKpc bsLiQuI+npLzBU3DqFFDZ8Ae+Zb4joGMXP+zYUDhu9hsfacqZsmdh8lssc5siSx2ApMV yzNjfqElF2YbDKxTFQ/ihNbR7aJPaRcGk7QEBqNyrjVq9w3nv/+sZbJs28wK5aixu/FW Eb9kFUktBN7PxaClWKhgDIbyQxQa4EfXQMoPNsKKESZwDQ0KBFyTa8KlLq4BEVPTR7BF d1i531DhbBbkDsQKJVy+10SkwUs0O0ieEwf8rPOLEc+yr+6xaeY3j0rMYc75EKqBZa11 VKqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=NOv7n72cDyWUU7iJNV38AFpKId3+/WvjhxWV1Nrhajo=; b=LGdLJxV7a3CYdd8wMZuxH6CzUUUS9kXsmyexxIkjmwlqFIqhB1vv7rnqKW1yau1vBg hDL8GktqjbOfMM8p0f/9npGqXxLf11MB0MvI+HVGVhz3PvOjQmrek8kudX0iSeTZpOuK 8OHbGkw5lXGfBIn14dS1XxhY0k8cbM8xWvsHeWy/mOIJP50rXneJ2u8AWTJJqIB8gA+u i56TDVIW2+72ed1tdThziu5yzWoczxIs6X7Y0lJRHhPlU0WBCfpEcW+ducXXNMbbBnTI DDU5T8RD4OYLNrOQTEx9xbEQmDSEq68Hfy2mrK7Lsa918sIh3aKl0XK+8ScCjQ4XhKCW I/+g== X-Gm-Message-State: AOUpUlHELuW0D0+6tLr6C8H8uaiwV644N48JWasl0XrKrwvfzKiy/LpQ hms9hQp4ayOQoCVzgYdKWuo99g== X-Google-Smtp-Source: AA+uWPy3k3gfkTlpp2P9WYn/Eo7SXxsZojUiXVcRVZM3PEQLFTzvZyVw5y0OFJGhkVe5p0FZp5pszg== X-Received: by 2002:a63:1760:: with SMTP id 32-v6mr22399958pgx.31.1534967213704; Wed, 22 Aug 2018 12:46:53 -0700 (PDT) Received: from [10.250.1.167] ([12.229.62.29]) by smtp.gmail.com with ESMTPSA id f11-v6sm3347916pfa.131.2018.08.22.12.46.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 22 Aug 2018 12:46:52 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: Native Encryption for ZFS on FreeBSD CFT From: Sean Fagan In-Reply-To: Date: Wed, 22 Aug 2018 12:46:51 -0700 Cc: Matthew Macy , FreeBSD CURRENT , freebsd-fs Content-Transfer-Encoding: quoted-printable Message-Id: <8C65E4ED-9AF3-4469-9C5A-BF058D50C3F5@ixsystems.com> References: <9FDF249A-E320-4652-834E-7EEC5C4FB7CA@ixsystems.com> To: Alan Somers X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 19:46:55 -0000 On Aug 22, 2018, at 12:35 PM, Alan Somers wrote: > Only encrypting L0 blocks also leaks a lot of information. That means = that, if encryption is set to anything but "off", watermarking attacks = will still be possible based on the size and sparsity of a file. = Because I believe that with any encryption mode, ZFS turns continuous = runs of zeros into holes. And I don't see anything in zio_crypt.c that = addresses that. I=E2=80=99m not sure about that. However, with compression=3Doff, dd if=3D/dev/zero of=3Dbigfile bs=3D1m count=3D1024 results in a file that is 1565148 blocks (of 128k bytes), which supports = your statement. With compression=3Don, it creates a 1 block file. Sean. From owner-freebsd-fs@freebsd.org Wed Aug 22 22:39:31 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 33B6B1098335; Wed, 22 Aug 2018 22:39:31 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D665B710C2; Wed, 22 Aug 2018 22:39:30 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from mail-io0-f172.google.com (mail-io0-f172.google.com [209.85.223.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) (Authenticated sender: mmacy) by smtp.freebsd.org (Postfix) with ESMTPSA id 9D62217F85; Wed, 22 Aug 2018 22:39:30 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: by mail-io0-f172.google.com with SMTP id l7-v6so2769117iok.6; Wed, 22 Aug 2018 15:39:30 -0700 (PDT) X-Gm-Message-State: AOUpUlFDBUy/tgXYwiysgJL5Q3SyB75azOiAknhx2XTrW4CZjHzOLsbj 1QgpyJWNVvO3G8SI8Twm8WCFxwS5FnvqYTCUqqs= X-Google-Smtp-Source: AA+uWPzqbDc7LZoKBasb7hYMlKHLigBcLUGefosuFtCCFKnKyC18ItL9dcz1+QAk9ClC6Mj2xdkRpGPkqKbb6TK9jmE= X-Received: by 2002:a6b:ed11:: with SMTP id n17-v6mr22683138iog.132.1534977570060; Wed, 22 Aug 2018 15:39:30 -0700 (PDT) MIME-Version: 1.0 References: <9FDF249A-E320-4652-834E-7EEC5C4FB7CA@ixsystems.com> In-Reply-To: From: Matthew Macy Date: Wed, 22 Aug 2018 15:39:18 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Native Encryption for ZFS on FreeBSD CFT To: tcaputi@datto.com Cc: Sean Fagan , Alan Somers , freebsd-current , freebsd-fs Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 22:39:31 -0000 Hi Thomas, Alan believes that, even with dedup disabled, the ZFS native encryption support is vulnerable to watermarking attacks. I don't have enough exposure to crypto to pass any judgement and was hoping that you'd share your point of view. Thanks in advance. -M On Wed, Aug 22, 2018 at 12:42 PM Alan Somers wrote: > Only encrypting L0 blocks also leaks a lot of information. That means > that, if encryption is set to anything but "off", watermarking attacks will > still be possible based on the size and sparsity of a file. Because I > believe that with any encryption mode, ZFS turns continuous runs of zeros > into holes. And I don't see anything in zio_crypt.c that addresses that. > -Alan > > On Wed, Aug 22, 2018 at 1:23 PM Sean Fagan wrote: > >> On Aug 22, 2018, at 12:20 PM, Alan Somers wrote: >> > ]That doesn't answer the question about what happens when dedup is >> turned off. In that case, is the HMAC still used as the IV? If so, then >> watermarking attacks are still possible. If ZFS switches to a random IV >> when dedup is off, then it would probably be ok. >> >> From the same file: >> >> * Initialization Vector (IV): >> >> * An initialization vector for the encryption algorithms. This is used >> to >> * "tweak" the encryption algorithms so that two blocks of the same data >> are >> * encrypted into different ciphertext outputs, thus obfuscating block >> patterns. >> * The supported encryption modes (AES-GCM and AES-CCM) require that an >> IV is >> * never reused with the same encryption key. This value is stored >> unencrypted >> * and must simply be provided to the decryption function. We use a 96 >> bit IV >> * (as recommended by NIST) for all block encryption. For non-dedup >> blocks we >> * derive the IV randomly. The first 64 bits of the IV are stored in the >> second >> * word of DVA[2] and the remaining 32 bits are stored in the upper 32 >> bits of >> * blk_fill. This is safe because encrypted blocks can't use the upper 32 >> bits >> * of blk_fill. We only encrypt level 0 blocks, which normally have a >> fill count >> * of 1. The only exception is for DMU_OT_DNODE objects, where the fill >> count of >> * level 0 blocks is the number of allocated dnodes in that block. The >> on-disk >> * format supports at most 2^15 slots per L0 dnode block, because the >> maximum >> * block size is 16MB (2^24). In either case, for level 0 blocks this >> number >> * will still be smaller than UINT32_MAX so it is safe to store the IV in >> the >> * top 32 bits of blk_fill, while leaving the bottom 32 bits of the fill >> count >> * for the dnode code. >> >> >> Sean >> >> >> From owner-freebsd-fs@freebsd.org Wed Aug 22 23:35:05 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6BFF010997DD for ; Wed, 22 Aug 2018 23:35:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 09C9473B81 for ; Wed, 22 Aug 2018 23:35:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id C255610997DB; Wed, 22 Aug 2018 23:35:04 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B115510997DA for ; Wed, 22 Aug 2018 23:35:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5331E73B80 for ; Wed, 22 Aug 2018 23:35:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 7DE491AB81 for ; Wed, 22 Aug 2018 23:35:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7MNZ3pH003154 for ; Wed, 22 Aug 2018 23:35:03 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7MNZ3Ul003151 for fs@FreeBSD.org; Wed, 22 Aug 2018 23:35:03 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230752] panic: excl->share in newnfs_request Date: Wed, 22 Aug 2018 23:35:03 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: panic X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: rmacklem@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 23:35:05 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230752 Rick Macklem changed: What |Removed |Added ---------------------------------------------------------------------------- Status|New |In Progress --- Comment #9 from Rick Macklem --- If you add the "-use-udpsock" command line option to nfsuserd it has the same effect as upgrading to r338192 or later. Btw, using "soft,intr" is not recommended for NFSv4 mounts. (I think it's in the BUGS section of mount_nfs(8).) --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Aug 23 03:22:05 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7BF57109F695 for ; Thu, 23 Aug 2018 03:22:05 +0000 (UTC) (envelope-from tcaputi@datto.com) Received: from mail-oi0-x22d.google.com (mail-oi0-x22d.google.com [IPv6:2607:f8b0:4003:c06::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0650A7BF4E for ; Thu, 23 Aug 2018 03:22:04 +0000 (UTC) (envelope-from tcaputi@datto.com) Received: by mail-oi0-x22d.google.com with SMTP id m11-v6so6900373oic.2 for ; Wed, 22 Aug 2018 20:22:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=datto-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=gChVgyp9EnY8azFfwvDURSLkGC2mG2iBlJVCk2Dtj/o=; b=MeuPsbT8OPmbtxB5kBcElX/KM7m0oa1Bez2fDqtU3pFX00+Q9PeO44scC26aZaBzd1 Te1e0rVaXJDkSTct0oJuk2BlXfH+XlGDt/SlswhWrj9SU1vQ0tHoFXOI4w9icF+8uOwf nijgZMEx7K9RpZq+3TAch2W1r1/jronPdT78Xw5ZTzjEWuneR6Oi4tUj3aGpjUccq7sb KwPj2NyPIYgw327wW9alDAlFV1S7dbjtyv25j8Da4URSp7nL8DLLlZcKdBTkg/zSrqcY AY7Z/XoPsbEoyBb5WRo+QccZwYO+L8W7+LarirboI0TJAXQtyZZKzGJLa0PArQCjaUoe LToQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=gChVgyp9EnY8azFfwvDURSLkGC2mG2iBlJVCk2Dtj/o=; b=RyI6XGFaJWbFDQ30dBBq4fXEzCtB+yIXvKP8lOOx2x9cT9tgnA25nhWPPaoGHouSuU U3sK83wOyJCLheygSjEmbap+BwhqQ2r/T6Z46OwdDUrHtsCF7y3E5y/QXUK6EAwC92z/ NGnIJEVkbH89ocLaTPq73PRRPshZAFOjtwXs6FY3vmTnmBjPnf16Hys2jTqaFA1tt9S6 c/9l6LDcvFXIsa+oC966hSmef+k5miCgbat3AiVDWcglQ25RaTNFqqK5nGSjFIEc4k7j xNR9IJ6Z4ZROit7SfH9Sb9ZfRsvMic2vd/frnLkNvnzVlfbcMartpgzljnSP7i65XmNL 4rzQ== X-Gm-Message-State: APzg51A1I5u2mcoYPvsRAv60tePXHqScRbIenz+qoGhOlN6xNkxNVGfT WYlCmNjXfalIjUAcWaZhDKRBBLLclcaznFKZOkCvfw== X-Google-Smtp-Source: ANB0VdY7IAlyN3YJmY4Rwt47wPz7ygUaoHjPs/JTS/QP7K5M/EiB/Sc+AH5+1zI04ryur54EBYslEwQI5dtc59kGMYY= X-Received: by 2002:aca:90d:: with SMTP id 13-v6mr6152825oij.300.1534994523936; Wed, 22 Aug 2018 20:22:03 -0700 (PDT) MIME-Version: 1.0 References: <9FDF249A-E320-4652-834E-7EEC5C4FB7CA@ixsystems.com> In-Reply-To: From: Thomas Caputi Date: Wed, 22 Aug 2018 23:21:53 -0400 Message-ID: Subject: Re: Native Encryption for ZFS on FreeBSD CFT To: mmacy@freebsd.org Cc: Sean Fagan , asomers@freebsd.org, freebsd-current@freebsd.org, freebsd-fs@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2018 03:22:05 -0000 > That doesn't answer the question about what happens when dedup is turned = off. In that case, is the HMAC still used as the IV? If so, then watermar= king attacks are still possible. Quoting the comment from the code above: "For non-dedup blocks we derive the IV randomly". When dedup is enabled, we do leak this information, but the dedup table already leaks that information anyway. The dedup table needs to be in plaintext so that we can repair it even when keys are not loaded. This is a known and documented trade off of using encryption + dedup. > Only encrypting L0 blocks also leaks a lot of information. That means th= at, if encryption is set to anything but "off", watermarking attacks will s= till be possible based on the size and sparsity of a file. Because I belie= ve that with any encryption mode, ZFS turns continuous runs of zeros into h= oles First of all, with encryption=3Doff, watermarking attacks are really quite easy :). The information that can be gained about a file from ZFS by looking at the raw disk are: 1) The size of the file (rounded up to the nearest sector size): Almost all applications that encrypt data will leak the approximate size of the protected payload. 2) The locations of holes within a file: ZFS does not turn runs of zeros into holes if you have compression off. However, data that is never written is maintained as a hole (ie if you never write any data to block 3 of a file). You are correct that technically this is a small leak of information, but we decided while designing the encryption scheme that the performance and space savings are worth it here. Is this enough information to be an attack vector? I would argue not, but if you are paranoid you could always turn compression off and fill in all the holes of your files with zeros. 3) If dedup is on, you can see which blocks have deduped against other blocks within a clone family. Encrypted dedup only works within applications that share the same master encryption key, which is essentially just snapshots and clones of snapshots. You cannot write data to one encrypted dataset and analyze the dedup tables to see i the data you wrote deduped against another dataset's data. 4) If compression + encryption is on a CRIME attack is possible, but in almost every scenario this attack is impractical. It requires the filesystem to have the key loaded, an application that appends a secret to the data controlled by an attacker, the attacker requires root access to the running system (to read the raw disk without rebooting and unloading the encryption key), and the attacker needs to be able to do many iterations of writing this attacker + secret data to disk and checking the resulting plaintext. During the implementation of native ZFS encryption we evaluated these and came to the conclusion that the security risks here are easily outweighed by the usability and performance benefits. If you have any further questions about the design, feel free to email me again or take a look at the (largely diagram based) docs on the implementation: https://docs.google.com/presentation/d/1km-z3MVNHYwlQLY6yEC3iq-TD05eredH9Ih= 4umGdkJw/edit?usp=3Dsharing On Wed, Aug 22, 2018 at 6:39 PM Matthew Macy wrote: > > Hi Thomas, > > Alan believes that, even with dedup disabled, the ZFS native encryption s= upport is vulnerable to watermarking attacks. I don't have enough exposure = to crypto to pass any judgement and was hoping that you'd share your point = of view. Thanks in advance. > > -M > > > > On Wed, Aug 22, 2018 at 12:42 PM Alan Somers wrote: >> >> Only encrypting L0 blocks also leaks a lot of information. That means t= hat, if encryption is set to anything but "off", watermarking attacks will = still be possible based on the size and sparsity of a file. Because I beli= eve that with any encryption mode, ZFS turns continuous runs of zeros into = holes. And I don't see anything in zio_crypt.c that addresses that. >> -Alan >> >> On Wed, Aug 22, 2018 at 1:23 PM Sean Fagan wrote: >>> >>> On Aug 22, 2018, at 12:20 PM, Alan Somers wrote: >>> > ]That doesn't answer the question about what happens when dedup is tu= rned off. In that case, is the HMAC still used as the IV? If so, then wat= ermarking attacks are still possible. If ZFS switches to a random IV when = dedup is off, then it would probably be ok. >>> >>> From the same file: >>> >>> * Initialization Vector (IV): >>> * An initialization vector for the encryption algorithms. This is used= to >>> * "tweak" the encryption algorithms so that two blocks of the same dat= a are >>> * encrypted into different ciphertext outputs, thus obfuscating block = patterns. >>> * The supported encryption modes (AES-GCM and AES-CCM) require that an= IV is >>> * never reused with the same encryption key. This value is stored unen= crypted >>> * and must simply be provided to the decryption function. We use a 96 = bit IV >>> * (as recommended by NIST) for all block encryption. For non-dedup blo= cks we >>> * derive the IV randomly. The first 64 bits of the IV are stored in th= e second >>> * word of DVA[2] and the remaining 32 bits are stored in the upper 32 = bits of >>> * blk_fill. This is safe because encrypted blocks can't use the upper = 32 bits >>> * of blk_fill. We only encrypt level 0 blocks, which normally have a f= ill count >>> * of 1. The only exception is for DMU_OT_DNODE objects, where the fill= count of >>> * level 0 blocks is the number of allocated dnodes in that block. The = on-disk >>> * format supports at most 2^15 slots per L0 dnode block, because the m= aximum >>> * block size is 16MB (2^24). In either case, for level 0 blocks this n= umber >>> * will still be smaller than UINT32_MAX so it is safe to store the IV = in the >>> * top 32 bits of blk_fill, while leaving the bottom 32 bits of the fil= l count >>> * for the dnode code. >>> >>> Sean >>> >>> From owner-freebsd-fs@freebsd.org Thu Aug 23 05:52:46 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C5575108194D for ; Thu, 23 Aug 2018 05:52:46 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (chez.mckusick.com [70.36.157.235]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5255D81D36 for ; Thu, 23 Aug 2018 05:52:46 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (localhost [IPv6:::1]) by chez.mckusick.com (8.15.2/8.15.2) with ESMTP id w7N5vvjj038580; Wed, 22 Aug 2018 22:57:57 -0700 (PDT) (envelope-from mckusick@mckusick.com) Message-Id: <201808230557.w7N5vvjj038580@chez.mckusick.com> From: Kirk McKusick To: bob prohaska Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems cc: FreeBSD Filesystems X-URL: http://WWW.McKusick.COM/ Reply-To: Kirk McKusick In-reply-to: <20180822004843.GA84687@www.zefox.net> Comments: In-reply-to bob prohaska message dated "Tue, 21 Aug 2018 17:48:43 -0700." MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <38578.1535003877.1@chez.mckusick.com> Date: Wed, 22 Aug 2018 22:57:57 -0700 X-Spam-Status: No, score=-1.4 required=5.0 tests=BAYES_00,MISSING_MID, UNPARSEABLE_RELAY autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on chez.mckusick.com X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2018 05:52:47 -0000 > Date: Tue, 21 Aug 2018 17:48:43 -0700 > From: bob prohaska > To: Kirk McKusick > Cc: FreeBSD Current , > FreeBSD Filesystems , > bob prohaska > Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems > X-ASK-Info: Message Queued (2018/08/21 17:55:39) > X-ASK-Info: Confirmed by User (2018/08/21 18:47:17) > > On Mon, Aug 20, 2018 at 12:40:56PM -0700, Kirk McKusick wrote: >> I have recently added TRIM consolodation support for the UFS/FFS >> filesystem. This feature consolodates large numbers of TRIM commands >> into a much smaller number of commands covering larger blocks of >> disk space. >> >> ... > > Will the new feature be active on a Raspberry Pi 3 using flash > on microSD and USB for file systems and swap? When you create the filesystem (using newfs) you need to specify the -t option to request that it send TRIM commands to the underlying media. If you have an existing filesystem, you can use the command `tunefs -t enable ' to set the TRIM-request flag. When you mount a fiesystem that has been told to send TRIM commands, it will send an ioctl to the device asking if it supports TRIM. If it replies that it does, then the TRIM commands will be sent. If it does not then the kernel will print an error message ``WARNING: : TRIM flag on fs but disk does not support TRIM'' or ``WARNING: : TRIM flag on fs but disk does not confirm that it supports TRIM''. If you get no message when you mount, then the drive will accept TRIM commands. Now whether it will do anything with them is not clear based on your quote below. > Can the feature be turned on using one of the conf files in /etc? Yes. Just add `sysctl vfs.ffs.dotrimcons=1' to your /etc/rc.local startup script (or where ever else in the startup that you handle local startup configuration). Or just log in (as root) or use sudo to run the sysctl once the system is up and running. Once the sysctl has been run, all filesystems running with TRIM requested will start consolodating their TRIM requests. > According to Sandisk, > "All microSD or USB drives are flash memory and does support the > TRIM command, however, you will not notice any difference after > running TRIM command on memory cards or USB drives. TRIM command > is basically used for SSD and Hard drives." > > The "you will not notice any difference...." qualification makes > me slightly uncertain the reply was well-informed, but if there's > any hope of success I'd like to try it. From time to time there > seem to be traffic jams among flash devices on the RPI3, it would > a pleasant surprise if this feature helps. > > Thanks for reading! > > bob prohaska It will not hurt, but if the above is to be believed will not help either. You would be better off not requesting the filesystem to send TRIM requests to avoid the overhead of doing so (e.g., if -t was specified when the filesystem was created, then you can use the command `tunefs -t disenable ' to tell the filesystem to stop sending TRIM commands). Do let me know you experiences if you try it out. Kirk McKusick From owner-freebsd-fs@freebsd.org Thu Aug 23 23:52:01 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C9C1B109B59C for ; Thu, 23 Aug 2018 23:52:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 62D2B8D718 for ; Thu, 23 Aug 2018 23:52:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 26BCF109B59A; Thu, 23 Aug 2018 23:52:01 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 13B83109B598 for ; Thu, 23 Aug 2018 23:52:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A75108D717 for ; Thu, 23 Aug 2018 23:52:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 007D127B49 for ; Thu, 23 Aug 2018 23:52:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7NNpx6t036772 for ; Thu, 23 Aug 2018 23:51:59 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7NNpxsH036771 for fs@FreeBSD.org; Thu, 23 Aug 2018 23:51:59 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 200663] zfs allow/unallow doesn't show numeric UID when the ID no longer exists in the password file Date: Thu, 23 Aug 2018 23:51:59 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: yuripv@yuripv.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2018 23:52:02 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D200663 Yuri Pankov changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |yuripv@yuripv.net --- Comment #6 from Yuri Pankov --- I'll take care of the upstream issue. --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Fri Aug 24 00:06:22 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 48A2A109BB85; Fri, 24 Aug 2018 00:06:22 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (www.zefox.net [50.1.20.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "www.zefox.org", Issuer "www.zefox.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 966D18DE71; Fri, 24 Aug 2018 00:06:21 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (localhost [127.0.0.1]) by www.zefox.net (8.15.2/8.15.2) with ESMTPS id w7O06bLd002974 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 23 Aug 2018 17:06:38 -0700 (PDT) (envelope-from fbsd@www.zefox.net) Received: (from fbsd@localhost) by www.zefox.net (8.15.2/8.15.2/Submit) id w7O06b7P002973; Thu, 23 Aug 2018 17:06:37 -0700 (PDT) (envelope-from fbsd) Date: Thu, 23 Aug 2018 17:06:37 -0700 From: bob prohaska To: Kirk McKusick Cc: FreeBSD Filesystems , freebsd-arm@freebsd.org, bob prohaska Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems Message-ID: <20180824000637.GA2157@www.zefox.net> References: <20180822004843.GA84687@www.zefox.net> <201808230557.w7N5vvjj038580@chez.mckusick.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201808230557.w7N5vvjj038580@chez.mckusick.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Aug 2018 00:06:22 -0000 On Wed, Aug 22, 2018 at 10:57:57PM -0700, Kirk McKusick wrote: > > Date: Tue, 21 Aug 2018 17:48:43 -0700 > > From: bob prohaska > > To: Kirk McKusick > > Cc: FreeBSD Current , > > FreeBSD Filesystems , > > bob prohaska > > Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems > > X-ASK-Info: Message Queued (2018/08/21 17:55:39) > > X-ASK-Info: Confirmed by User (2018/08/21 18:47:17) > > > > > > Will the new feature be active on a Raspberry Pi 3 using flash > > on microSD and USB for file systems and swap? > > When you create the filesystem (using newfs) you need to specify > the -t option to request that it send TRIM commands to the underlying > media. If you have an existing filesystem, you can use the command > `tunefs -t enable ' to set the TRIM-request > flag. When you mount a fiesystem that has been told to send TRIM > commands, it will send an ioctl to the device asking if it supports > TRIM. If it replies that it does, then the TRIM commands will be > sent. If it does not then the kernel will print an error message > ``WARNING: : TRIM flag on fs but disk does not > support TRIM'' or ``WARNING: : TRIM flag on fs but > disk does not confirm that it supports TRIM''. If you get no message > when you mount, then the drive will accept TRIM commands. Now whether > it will do anything with them is not clear based on your quote below. > Using FreeBSD 12.0-ALPHA2 #12 r338122: Tue Aug 21 14:26:18 PDT 2018 Alas, no luck. On mount TRIM isn't supported: WARNING: /usr: TRIM flag on fs but disk does not support TRIM Using tunefs on the microSD produced a different refusal: # tunefs -t enable /dev/mmcsd0s2a tunefs: issue TRIM to the disk set tunefs: /dev/mmcsd0s2a: failed to open disk for writing I tried with the device both ro and rw, same error. I expected "not supported", rather than "failed to open". If there's a mistake please tell me. Not sure if this is true of all possible storage devices, but the Sandisk Ultra microSD and Sandisk Extreme USB appear to be non-starters. Thanks very much for your help! bob prohaska From owner-freebsd-fs@freebsd.org Fri Aug 24 00:22:26 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E051A109C16C for ; Fri, 24 Aug 2018 00:22:26 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (chez.mckusick.com [70.36.157.235]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4BF7C8EBA8 for ; Fri, 24 Aug 2018 00:22:26 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (localhost [IPv6:::1]) by chez.mckusick.com (8.15.2/8.15.2) with ESMTP id w7O0Rh7f062555; Thu, 23 Aug 2018 17:27:43 -0700 (PDT) (envelope-from mckusick@mckusick.com) Message-Id: <201808240027.w7O0Rh7f062555@chez.mckusick.com> From: Kirk McKusick To: bob prohaska Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems cc: FreeBSD Filesystems X-URL: http://WWW.McKusick.COM/ Reply-To: Kirk McKusick In-reply-to: <20180824000637.GA2157@www.zefox.net> Comments: In-reply-to bob prohaska message dated "Thu, 23 Aug 2018 17:06:37 -0700." MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <62553.1535070463.1@chez.mckusick.com> Date: Thu, 23 Aug 2018 17:27:43 -0700 X-Spam-Status: No, score=-1.4 required=5.0 tests=BAYES_00,MISSING_MID, UNPARSEABLE_RELAY autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on chez.mckusick.com X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Aug 2018 00:22:27 -0000 > Date: Thu, 23 Aug 2018 17:06:37 -0700 > From: bob prohaska > To: Kirk McKusick > Cc: FreeBSD Filesystems , freebsd-arm@freebsd.org, > bob prohaska > Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems > > Using: > > FreeBSD 12.0-ALPHA2 #12 r338122: Tue Aug 21 14:26:18 PDT 2018 > > Alas, no luck. On mount TRIM isn't supported: > > WARNING: /usr: TRIM flag on fs but disk does not support TRIM > > Using tunefs on the microSD produced a different refusal: > # tunefs -t enable /dev/mmcsd0s2a > tunefs: issue TRIM to the disk set > tunefs: /dev/mmcsd0s2a: failed to open disk for writing > I tried with the device both ro and rw, same error. I > expected "not supported", rather than "failed to open". > If there's a mistake please tell me. Assuming that /dev/mmcsd0s2a is your /usr then when the filesystem was ro it should have allowed the open to work. But given that you already had the flag set when you attempted the mount of /usr it would have made no difference even if you could have set it. > Not sure if this is true of all possible storage devices, but > the Sandisk Ultra microSD and Sandisk Extreme USB appear to be > non-starters. > > Thanks very much for your help! > > bob prohaska Based on the comments by Mark Millard, it sounds like the Raspberry Pi 3 uses USB 2.0 which definitely has no ability to pass through TRIM to the underlying device (even if that device can support TRIM). There is some speculation that USB 3.1 has support for passing through TRIM commands, but I did not see it in my cursory inspection of the USB Spec. So, conclusion is that you will not be able to use TRIM on your Raspberry Pi 3. Kirk McKusick From owner-freebsd-fs@freebsd.org Fri Aug 24 00:57:07 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2B6C2109CD58 for ; Fri, 24 Aug 2018 00:57:07 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic305-22.consmr.mail.ne1.yahoo.com (sonic305-22.consmr.mail.ne1.yahoo.com [66.163.185.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AC4758FC75 for ; Fri, 24 Aug 2018 00:57:06 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: _OBaDFUVM1memAeiNljsTI7C2B1CmAXgkK.UQCwnpqiyt_BlMhYOJvS5fhozKR. vTA.UbSPR8TWjXEgpmVPYoZ6_vv46cRSuVJ2.Rgfn_Z4gZRztU0y5CfHKjbr39sxxPZdU8zQU9oz nnQf6SSlfABn0qNqir2ZwiE91FVUSihD3GZ5NQ1o2.ZuKU.0TLhPHtHiEwXrom32eyhndXEDxU1B zd29sNedXSJGcWkv0KZAdyNyiOX4jWi9mF3ELbADywuCBxbKcvovJPLy9v30ukmxclvFc2jQxQ2U D5HxKodsjT_SZK6TvdAP36hl7nt6Kv.NoQK0Uv5HDobG4g1lba_nh6zpKgXTb5PeOhe9y7tUqvtH najYozYUPx17HPbfbU5rCpotflExjaNnF6fqKxBZiqDQQNp7fJfFNIXWt9rIxeF43uDfeF3lvXJN 34UrXgsciIRwNQJvEaRXnAcYPIhyu3eRZrBtk6W0Ni0ZGAGxbsmS847oESieRzhG3gGjIokdX0Bv .qmLF_zPblMSx4T3nErJcVnZW78IPnXd4Ok0dS0z.aEEseJPnj6ykm26PeTTq5YKHGgEOj6PFSf_ 0xX4UOS1L3dJQD0PmHxg.JEIQzurlirQTQbtRYdkKDHSgkXCxfIh_Fx_P1RBKMqEDUT9RPZTAzmF mWFAZUJ5LhfAKQ4Vmv4rsCA6pGCqYCLICLyiFJhTsQvcXQCOqT7jG5I9z5oXkvRO.JlwWFrnRRUg Dz1x1IGfklbnp.WXBlOgDzyga1oG8X6GoU8m.MEdaMCOrNUvRrCQzbrV1Q4BTPv_5Z.gk4H5QXX. yH8TDZ7xdBOCHQmm5Rd91QoGKJLQsczNU5hhQgqW3cZER1wfqZ0OkkxtpHzuU5o6dNlntv8fkG_3 20ht2WRZLIoIORmk6phS9KLcAGc_gj1fp5qznhrTTObS__hynroT0MJ_Jm3qQwOgQeE6DRqYRcJe n6vsqgSivxfV1XcuyzaFelm2xFHJQwMTgLD0YX079rMefDVaeOt3Ac2ZPrNWki0ajO5Or8QKVcSU Nu_3fHy4sK1eF Received: from sonic.gate.mail.ne1.yahoo.com by sonic305.consmr.mail.ne1.yahoo.com with HTTP; Fri, 24 Aug 2018 00:57:00 +0000 Received: from ip70-189-131-151.lv.lv.cox.net (EHLO [192.168.0.105]) ([70.189.131.151]) by smtp414.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 5a8e29ccb534af955f7706ed61837f3d; Fri, 24 Aug 2018 00:56:58 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems From: Mark Millard In-Reply-To: <20180824000637.GA2157@www.zefox.net> Date: Thu, 23 Aug 2018 17:56:56 -0700 Cc: Kirk McKusick , FreeBSD Filesystems , freebsd-arm@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <1F534D6D-3CA3-4E28-AF24-7AABAB0EDBD3@yahoo.com> References: <20180822004843.GA84687@www.zefox.net> <201808230557.w7N5vvjj038580@chez.mckusick.com> <20180824000637.GA2157@www.zefox.net> To: bob prohaska X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Aug 2018 00:57:07 -0000 On 2018-Aug-23, at 5:06 PM, bob prohaska wrote: > On Wed, Aug 22, 2018 at 10:57:57PM -0700, Kirk McKusick wrote: >>> Date: Tue, 21 Aug 2018 17:48:43 -0700 >>> From: bob prohaska >>> To: Kirk McKusick >>> Cc: FreeBSD Current , >>> FreeBSD Filesystems , >>> bob prohaska >>> Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems >>> X-ASK-Info: Message Queued (2018/08/21 17:55:39) >>> X-ASK-Info: Confirmed by User (2018/08/21 18:47:17) >>>=20 >>>=20 >>> Will the new feature be active on a Raspberry Pi 3 using flash=20 >>> on microSD and USB for file systems and swap?=20 >>=20 >> When you create the filesystem (using newfs) you need to specify >> the -t option to request that it send TRIM commands to the underlying >> media. If you have an existing filesystem, you can use the command >> `tunefs -t enable ' to set the = TRIM-request >> flag. When you mount a fiesystem that has been told to send TRIM >> commands, it will send an ioctl to the device asking if it supports >> TRIM. If it replies that it does, then the TRIM commands will be >> sent. If it does not then the kernel will print an error message >> ``WARNING: : TRIM flag on fs but disk does not >> support TRIM'' or ``WARNING: : TRIM flag on fs but >> disk does not confirm that it supports TRIM''. If you get no message >> when you mount, then the drive will accept TRIM commands. Now whether >> it will do anything with them is not clear based on your quote below. >>=20 >=20 > Using > FreeBSD 12.0-ALPHA2 #12 r338122: Tue Aug 21 14:26:18 PDT 2018 >=20 > Alas, no luck. On mount TRIM isn't supported: >=20 > WARNING: /usr: TRIM flag on fs but disk does not support TRIM >=20 > Using tunefs on the microSD produced a different refusal: > # tunefs -t enable /dev/mmcsd0s2a > tunefs: issue TRIM to the disk set > tunefs: /dev/mmcsd0s2a: failed to open disk for writing > I tried with the device both ro and rw, same error. I > expected "not supported", rather than "failed to open". > If there's a mistake please tell me. For a UFS example: # umount /mnt pine64# tunefs -tenable /dev/mmcsd0s2a tunefs: issue TRIM to the disk set # mount -o noatime /dev/mmcsd0s2a /mnt # tunefs -p /mnt tunefs: POSIX.1e ACLs: (-a) disabled tunefs: NFSv4 ACLs: (-N) enabled tunefs: MAC multilabel: (-l) disabled tunefs: soft updates: (-n) enabled tunefs: soft update journaling: (-j) disabled tunefs: gjournal: (-J) disabled tunefs: trim: (-t) enabled tunefs: maximum blocks per file in a cylinder group: (-e) 4096 tunefs: average file size: (-f) 16384 tunefs: average number of files in a directory: (-s) 64 tunefs: minimum percentage of free space: (-m) 8% tunefs: space to hold for metadata blocks: (-k) 6408 tunefs: optimization preference: (-o) time tunefs: volume label: (-L) PINE642GBroot Then during a svnupdate -r477847 /usr/ports : dT: 1.006s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name 1265 144 1 32 3064 105 2597 2114 38 326 = 32.2 104.2| mmcsd0 Note the "d/s kBps ms/d" figures: it is actively in use. Context: This is from head -r337400 before the changes for how things work. The root file system and swap partition are on a USB SSD (on a powered hub). But: mmcsd0: 32GB MFG 07/2017 by 3 SD> at mmc0 = 50.0MHz/4bit/32768-block It is a SanDisk microsdhc card that I did the test with. > Not sure if this is true of all possible storage devices, but > the Sandisk Ultra microSD and Sandisk Extreme USB appear to be > non-starters. >=20 USB of various vintages has its own issues for the commands allowed. But microsd cards that support it should allow TRIM. (If some work worse with TRIM enabled is another matter. For the old way TRIM was handled by FreeBSD may be nearly certain that such works worse for such media.) =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-fs@freebsd.org Fri Aug 24 04:13:40 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C5A3F10A0AB0 for ; Fri, 24 Aug 2018 04:13:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 5860876008 for ; Fri, 24 Aug 2018 04:13:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 1AA8D10A0AAD; Fri, 24 Aug 2018 04:13:40 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 095E910A0AAC for ; Fri, 24 Aug 2018 04:13:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 96A4F76006 for ; Fri, 24 Aug 2018 04:13:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id E57349FF2 for ; Fri, 24 Aug 2018 04:13:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7O4DcDf017480 for ; Fri, 24 Aug 2018 04:13:38 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7O4Dc5a017479 for fs@FreeBSD.org; Fri, 24 Aug 2018 04:13:38 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230851] fsck sets last modified file size to zero on crash on UFS filesystem Date: Fri, 24 Aug 2018 04:13:38 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.2-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Aug 2018 04:13:40 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230851 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Aug 24 04:59:12 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2CA4910A16CE for ; Fri, 24 Aug 2018 04:59:12 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (chez.mckusick.com [70.36.157.235]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 84E4276F6B for ; Fri, 24 Aug 2018 04:59:11 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (localhost [IPv6:::1]) by chez.mckusick.com (8.15.2/8.15.2) with ESMTP id w7O54SPJ067702; Thu, 23 Aug 2018 22:04:28 -0700 (PDT) (envelope-from mckusick@mckusick.com) Message-Id: <201808240504.w7O54SPJ067702@chez.mckusick.com> From: Kirk McKusick To: Mark Millard Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems cc: bob prohaska , FreeBSD Filesystems X-URL: http://WWW.McKusick.COM/ Reply-To: Kirk McKusick In-reply-to: <1F534D6D-3CA3-4E28-AF24-7AABAB0EDBD3@yahoo.com> Comments: In-reply-to Mark Millard message dated "Thu, 23 Aug 2018 17:56:56 -0700." MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <67700.1535087068.1@chez.mckusick.com> Date: Thu, 23 Aug 2018 22:04:28 -0700 X-Spam-Status: No, score=-1.4 required=5.0 tests=BAYES_00,MISSING_MID, UNPARSEABLE_RELAY autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on chez.mckusick.com X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Aug 2018 04:59:12 -0000 So are you in a position to try out the TRIM consolidation implementation? In particular to see if it helps? Kirk McKusick From owner-freebsd-fs@freebsd.org Fri Aug 24 05:29:51 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0FFDD10A1DC9 for ; Fri, 24 Aug 2018 05:29:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id A13B277959 for ; Fri, 24 Aug 2018 05:29:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 634E510A1DC8; Fri, 24 Aug 2018 05:29:50 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 522C610A1DC7 for ; Fri, 24 Aug 2018 05:29:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E7F9F77958 for ; Fri, 24 Aug 2018 05:29:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 3BF4CA998 for ; Fri, 24 Aug 2018 05:29:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7O5Tn8h094737 for ; Fri, 24 Aug 2018 05:29:49 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7O5Tn74094736 for fs@FreeBSD.org; Fri, 24 Aug 2018 05:29:49 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230851] fsck sets last modified file size to zero on crash on UFS filesystem Date: Fri, 24 Aug 2018 05:29:49 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.2-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mckusick@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Aug 2018 05:29:51 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230851 Kirk McKusick changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mckusick@FreeBSD.org --- Comment #1 from Kirk McKusick --- What editor are you using? Most editors follow the safe update practice: write file to new_name fsync(new_name); rename(new_name, orig_name); The fsync will not return until the contents are on the disk and the rename= is atomic, so it will either point at the orig_name file contents or the new_n= ame file contents which because of the fsync will be on the disk. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Aug 24 06:08:51 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1D00F1080820 for ; Fri, 24 Aug 2018 06:08:51 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic312-23.consmr.mail.ne1.yahoo.com (sonic312-23.consmr.mail.ne1.yahoo.com [66.163.191.204]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 95C16788F0 for ; Fri, 24 Aug 2018 06:08:50 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: Ntz_0Z0VM1mg9oDZTSTTcJKV.pfPeTRZkWmwZpmTSnDnAzmIgYue0mKZ1Vhtcju Aa7UMAY2iSmzEMrfk8ZcRAyOERcwst13mGcHvwdqTE8fRLgl5ynDA7vNutLidt4mAvzum5HvZQPp tc.hLDt86bMbdiTvM6mPZJLdv2.xPLdX9aR7dM3WksZNhZSD6CC18ZDAdeddGZQ7YQCViZVTp6B2 OKkkuWiguIE8gALhlEQI8atOM40ytQ..kF1N9spvSS1hOA7f81KlRJPHNeclJkirrP.fItCx7l7P qx0Aa_nrbiaCYVugyf_YHfOBe4mAmuMv5dB7KTl1lqcteQ5wQEqok0cg0GnpDk9AfzZWEyn54bUy PH7an.6NhT5Ak3NXQQF7ybNpP1UOQSjj0eW5aAUyI7HRjX8iwd8UaksZjSM7SJv0mUd_XCWAbVnk yYoDv_SOQyylSdZwn2KE1I3El.MzeyZtBskwHFWy0M6wa1IuJTdVGA_tbbHYTdpAS2Js5s.PzYJ1 x1WYvEkKycd8GeURZzYXMoloMRr2cS5oBIbkjw5V8e6TU1qLD3QzEKX9O0JOKmQV_X5pPPXtFYrT cN9T3zHpljZdMoFCp8xxW2aTGMxbAyw1UR9W.HFMHAeJ1iPfCKZGdQvQcDSeXHCHJbpye70KAaUB wAKfZoMfOWE0lf5nLzGSK7QQvHMJ.iz1tRAOh37cekmWVbEy7gupklG6BvPlo97wyV17CSDPv8nx R3F9c1IBgpiVJOf9N4ZElAX5ByF59Tcs9ZqMpaE9qoD.hKG19rIzxS.F.9CbTPjxizPco6jd.27B SEENnkRSPyCi5MouGWG6GwcXrdxHsu_6zcLuzSda0EzM1ukvRmq7naW7VceY12bwbYTiw2sDdlnM MTjaRJqcpZ9aLpUApdYHpskbgsXhScot2ynIfmNX4YdGubU9rX1hdMI5Cxts.ste_ea92ZwVEi3u K7dtqjI0fQv6Y1eUScGEQReem9FHR9LHMNdl_SHer4b8IlGBSkF1hPJTbV_inpMyqzjLL6o38dIe owTN4NrrrRtdrBZXQnVV2p3r_gGaFfkIk_2yNRA-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic312.consmr.mail.ne1.yahoo.com with HTTP; Fri, 24 Aug 2018 06:08:43 +0000 Received: from ip70-189-131-151.lv.lv.cox.net (EHLO [192.168.0.105]) ([70.189.131.151]) by smtp426.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID c720db0d4b7270ed918f262da828e2d3; Fri, 24 Aug 2018 05:58:34 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems From: Mark Millard In-Reply-To: <201808240504.w7O54SPJ067702@chez.mckusick.com> Date: Thu, 23 Aug 2018 22:58:32 -0700 Cc: bob prohaska , FreeBSD Filesystems Content-Transfer-Encoding: 7bit Message-Id: <392FF90D-6D90-457B-94D2-9C4A05B46309@yahoo.com> References: <201808240504.w7O54SPJ067702@chez.mckusick.com> To: Kirk McKusick X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Aug 2018 06:08:51 -0000 On 2018-Aug-23, at 10:04 PM, Kirk McKusick wrote: > So are you in a position to try out the TRIM consolidation implementation? > In particular to see if it helps? The Pine64+ 2GB that I have access to is in the middle of a poudriere-devel bulk that probably has 24 hours or more to go. That will delay anything that I might try. I normally do not have the root file system and swap partition run on a microsd card. So I do not have a history to compare against. (I'd seen reports on the lists and just avoided the issues up front.) I'm not sure of a good way to test at this point. I might just end up gstat -pd monitoring a self-hosted buildworld buildkernel or some such. Suggestions welcome. Swap partitions do not get TRIMs and swap files are greatly unreliable. (For the later see: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=206048 for what I'm referring to.) So the testing would likely be just for the file system activity, with any swapping mixed in the activity. Once the poudriere-devel finishes it will take a bit to set up a microsd card environment that is modern enough to have the changes. So may be next week sometime? Hopefully the: # umount /mnt pine64# tunefs -tenable /dev/mmcsd0s2a tunefs: issue TRIM to the disk set information will enable Bob P. to be able to do some rpi3 and/or rpi2 testing. But I do not know if he has a context were he can have some microsd card(s) that he uses in an unmounted but accessible status. (I have such a context.) If FreeBSD could still boot such, I'd have tested an e.MCC on an adapter card plugged into the microsd slot. But when I took my large jump to head -r337400 I discovered that combination no longer booted. So, unless the status has changed after -r377400 when I update, I can not test the e.MMC via sdcard-slot context on the Pine64+ 2GB. === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-fs@freebsd.org Fri Aug 24 09:09:26 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 89DC01084661 for ; Fri, 24 Aug 2018 09:09:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 27CE07E312 for ; Fri, 24 Aug 2018 09:09:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id DCD4F1084660; Fri, 24 Aug 2018 09:09:25 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CB76C108465F for ; Fri, 24 Aug 2018 09:09:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6D6A37E310 for ; Fri, 24 Aug 2018 09:09:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id B4DD4C9BA for ; Fri, 24 Aug 2018 09:09:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7O99OH8009525 for ; Fri, 24 Aug 2018 09:09:24 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7O99OpW009519 for fs@FreeBSD.org; Fri, 24 Aug 2018 09:09:24 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230851] fsck sets last modified file size to zero on crash on UFS filesystem Date: Fri, 24 Aug 2018 09:09:24 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.2-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: aliovx@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Aug 2018 09:09:26 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230851 --- Comment #2 from Ali Abdallah --- I had this issue when I edited loader.conf with nano.=20 Actually I tested the edit/panic cycles many times on an old test machine. = The problem does not always occur, but sometimes. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Aug 24 16:14:33 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4C176108E20D; Fri, 24 Aug 2018 16:14:33 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (www.zefox.net [50.1.20.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "www.zefox.org", Issuer "www.zefox.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id B04FC8C981; Fri, 24 Aug 2018 16:14:32 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (localhost [127.0.0.1]) by www.zefox.net (8.15.2/8.15.2) with ESMTPS id w7OGEiuL006433 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 24 Aug 2018 09:14:45 -0700 (PDT) (envelope-from fbsd@www.zefox.net) Received: (from fbsd@localhost) by www.zefox.net (8.15.2/8.15.2/Submit) id w7OGEh2s006432; Fri, 24 Aug 2018 09:14:43 -0700 (PDT) (envelope-from fbsd) Date: Fri, 24 Aug 2018 09:14:43 -0700 From: bob prohaska To: Kirk McKusick Cc: FreeBSD Filesystems , freebsd-arm@freebsd.org, bob prohaska Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems Message-ID: <20180824161443.GB2157@www.zefox.net> References: <20180824000637.GA2157@www.zefox.net> <201808240027.w7O0Rh7f062555@chez.mckusick.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201808240027.w7O0Rh7f062555@chez.mckusick.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Aug 2018 16:14:33 -0000 On Thu, Aug 23, 2018 at 05:27:43PM -0700, Kirk McKusick wrote: > > Assuming that /dev/mmcsd0s2a is your /usr then when the filesystem Sorry, the mistake was mine. /dev/mmcsd0s2a was / in that attempt, which obviously can't work. Moving a bootable August 2nd snapshot microSD card to a USB adapter allowed root@www:/ # tunefs -t enable /dev/da2s2a tunefs: issue TRIM to the disk set When I remount the device with mount -o rw /dev/da2s2a /mnt the console reports WARNING: /mnt: TRIM flag on fs but disk does not support TRIM I hope that's a consequence of the intervening USB interface. I'll swap boot devices when the present buildworld completes and see if the tunefs command really worked. If it did, then it'll make sense to update and rebuild to a TRIM-enabled revision. That'll take a couple of days, even if I don't make any mistakes.. Sandisk has yet to clarify whether TRIM is supported on microSD. Thanks for reading, bob prohaska From owner-freebsd-fs@freebsd.org Fri Aug 24 16:26:05 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E0588108E5A5 for ; Fri, 24 Aug 2018 16:26:05 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound1a.eu.mailhop.org (outbound1a.eu.mailhop.org [52.58.109.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6A1D28CF92 for ; Fri, 24 Aug 2018 16:26:05 +0000 (UTC) (envelope-from ian@freebsd.org) X-MHO-RoutePath: aGlwcGll X-MHO-User: 61f7920b-a7ba-11e8-aff6-0b9b8210da61 X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 67.177.211.60 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (unknown [67.177.211.60]) by outbound1.eu.mailhop.org (Halon) with ESMTPSA id 61f7920b-a7ba-11e8-aff6-0b9b8210da61; Fri, 24 Aug 2018 16:26:00 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id w7OGPwMW000923; Fri, 24 Aug 2018 10:25:58 -0600 (MDT) (envelope-from ian@freebsd.org) Message-ID: <1535127958.1488.31.camel@freebsd.org> Subject: Re: CFT: TRIM Consolodation on UFS/FFS filesystems From: Ian Lepore To: bob prohaska , Kirk McKusick Cc: FreeBSD Filesystems , freebsd-arm@freebsd.org Date: Fri, 24 Aug 2018 10:25:58 -0600 In-Reply-To: <20180824161443.GB2157@www.zefox.net> References: <20180824000637.GA2157@www.zefox.net> <201808240027.w7O0Rh7f062555@chez.mckusick.com> <20180824161443.GB2157@www.zefox.net> Content-Type: text/plain; charset="ISO-8859-1" X-Mailer: Evolution 3.18.5.1 FreeBSD GNOME Team Port Mime-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Aug 2018 16:26:06 -0000 On Fri, 2018-08-24 at 09:14 -0700, bob prohaska wrote: > On Thu, Aug 23, 2018 at 05:27:43PM -0700, Kirk McKusick wrote: > > > > > > Assuming that /dev/mmcsd0s2a is your /usr then when the filesystem > Sorry, the mistake was mine. /dev/mmcsd0s2a was / in that attempt, > which obviously can't work. > > Moving a bootable August 2nd snapshot microSD card to a USB adapter > allowed > root@www:/ # tunefs -t enable /dev/da2s2a > tunefs: issue TRIM to the disk set > > When I remount the device with >  mount -o rw /dev/da2s2a /mnt > the console reports >  WARNING: /mnt: TRIM flag on fs but disk does not support TRIM > I hope that's a consequence of the intervening USB interface. > > I'll swap boot devices when the present buildworld completes and > see if the tunefs command really worked. If it did, then it'll > make sense to update and rebuild to a TRIM-enabled revision.  > > That'll take a couple of days, even if I don't make any mistakes.. > > Sandisk has yet to clarify whether TRIM is supported on microSD. > > Thanks for reading, A usb sdcard reader/writer will definitely mask the trim capability of an sdcard. Aside from that, ALL sdcards support trim on freebsd, regardless of what any support folks at sandisk might tell you. The trim operation is supported in the mmcsd driver by issuing sd erase commands (CMD32/33/38 sequence) which has been in the sd spec from the beginning. That said, it's my experience that different cards implement CMD38 different ways. Some cards treat it like a TRIM -- it's a hint that tells the card "the data in these sectors need not be preserved during future updates to the erase block they're embedded in", and thus they happen very fast and improve efficiency. Other cards treat it as a more synchronous operation, physically erasing the blocks involved while- you-wait, which can be *horrible* for performance on ufs (although potentially it might get better with the consolidation of BIO_DELETEs). -- Ian From owner-freebsd-fs@freebsd.org Fri Aug 24 23:27:43 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 12CAA1097435 for ; Fri, 24 Aug 2018 23:27:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id A4AD57C2E0 for ; Fri, 24 Aug 2018 23:27:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 65BF61097433; Fri, 24 Aug 2018 23:27:42 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 548921097432 for ; Fri, 24 Aug 2018 23:27:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D31EE7C2DE for ; Fri, 24 Aug 2018 23:27:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 3413814295 for ; Fri, 24 Aug 2018 23:27:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7ONRf2O082925 for ; Fri, 24 Aug 2018 23:27:41 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7ONRfoV082924 for fs@FreeBSD.org; Fri, 24 Aug 2018 23:27:41 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230851] fsck sets last modified file size to zero on crash on UFS filesystem Date: Fri, 24 Aug 2018 23:27:41 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.2-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mckusick@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Not A Bug X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status resolution Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Aug 2018 23:27:43 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230851 Kirk McKusick changed: What |Removed |Added ---------------------------------------------------------------------------- Status|New |Closed Resolution|--- |Not A Bug --- Comment #3 from Kirk McKusick --- (In reply to Ali Abdallah from comment #2) I have looked at nano and it does not follow the proper protocol for writing out the file (which is detailed in my comment #1). It simply does: fd =3D open(file, O_WRONLY|O_CREAT|O_TRUNC, 0666); write new contents close(fd); The O_TRUNC flag truncates the file to zero length. If the system crashes before the new contents are written, you get a zero length file. By default= the contents will not be written for up to 30 seconds. So if you panic within 30 seconds of the file being written, you will get a zero length file. The pro= per way to fix this is detailed in my comment #1. The gap could be closed significantly by adding an fsync(fd) before calling close as that would cau= se the file to be written to disk within a few milliseconds of its finishing b= eing written thus closing the gap considerably (but it would still be possible to get a zero length file). So, it really should be fixed properly. I am closing this bug because it is a bug in nano and not in FreeBSD. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Aug 25 02:37:56 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D2640109D482 for ; Sat, 25 Aug 2018 02:37:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 6FBC182F14 for ; Sat, 25 Aug 2018 02:37:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 34DD0109D480; Sat, 25 Aug 2018 02:37:56 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 23AC7109D47E for ; Sat, 25 Aug 2018 02:37:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B8AED82F0F for ; Sat, 25 Aug 2018 02:37:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 23D2A15D6D for ; Sat, 25 Aug 2018 02:37:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7P2btqx091004 for ; Sat, 25 Aug 2018 02:37:55 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7P2bt4K091003 for fs@FreeBSD.org; Sat, 25 Aug 2018 02:37:55 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230851] fsck sets last modified file size to zero on crash on UFS filesystem Date: Sat, 25 Aug 2018 02:37:55 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.2-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: cem@freebsd.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Not A Bug X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Aug 2018 02:37:57 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230851 Conrad Meyer changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |cem@freebsd.org --- Comment #4 from Conrad Meyer --- @Ali, Please file a bug with the nano project and chase up getting that fix= ed there. I know nano has a lot of users and it is a shame it isn't safe! --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Aug 25 07:04:20 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8BA7010801AB for ; Sat, 25 Aug 2018 07:04:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 28B178C0C3 for ; Sat, 25 Aug 2018 07:04:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id DEA8910801AA; Sat, 25 Aug 2018 07:04:19 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CB75A10801A9 for ; Sat, 25 Aug 2018 07:04:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 513DB8C0BE for ; Sat, 25 Aug 2018 07:04:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 96A1C18392 for ; Sat, 25 Aug 2018 07:04:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7P74Iva076749 for ; Sat, 25 Aug 2018 07:04:18 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7P74ICc076748 for fs@FreeBSD.org; Sat, 25 Aug 2018 07:04:18 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230851] fsck sets last modified file size to zero on crash on UFS filesystem Date: Sat, 25 Aug 2018 07:04:18 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.2-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: aliovx@gmail.com X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Not A Bug X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Aug 2018 07:04:20 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230851 --- Comment #5 from Ali Abdallah --- @Conrad I will file a bug against nano.=20 @Kirk thanks for looking into this.=20 BTW, while I was testing on my test system, I had done the following on a c= lean 11.2 installation 1) pkg install nano=20 2) nano /etc/sysctl.conf modify, save and exit 3) sysctl debug.kdb.panic=3D1 On the next boot, nano was registered as installed by pkg. But the nano bin= ary (together with its indexinfo and gettext-runtime files) were not present on disk.=20 I think pkg does not fsync the installed files in this case, right? --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Aug 25 16:08:26 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 502E9108F5FA for ; Sat, 25 Aug 2018 16:08:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id E135577D48 for ; Sat, 25 Aug 2018 16:08:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id A2AFB108F5F8; Sat, 25 Aug 2018 16:08:25 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 91629108F5F7 for ; Sat, 25 Aug 2018 16:08:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 32C7477D45 for ; Sat, 25 Aug 2018 16:08:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 7B2591CEB7 for ; Sat, 25 Aug 2018 16:08:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w7PG8OAX052224 for ; Sat, 25 Aug 2018 16:08:24 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w7PG8Onw052223 for fs@FreeBSD.org; Sat, 25 Aug 2018 16:08:24 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 230851] fsck sets last modified file size to zero on crash on UFS filesystem Date: Sat, 25 Aug 2018 16:08:24 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.2-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mckusick@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Not A Bug X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Aug 2018 16:08:26 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230851 --- Comment #6 from Kirk McKusick --- (In reply to Ali Abdallah from comment #5) You are correct that the installed binaries are not being fsync'ed before t= he database is being updated (which being database software DOES properly fsyn= c). I have not looked at pkg, but if it directly does the write itself, it shou= ld do the fsync before it updates the database. It may be that it is running a shell script that uses the install(1) utility, then it is install that shou= ld be doing the fsync. Obviously, more checking needs to be done. Thanks for your help in tracking down these issues. --=20 You are receiving this mail because: You are the assignee for the bug.=