From owner-freebsd-fs@freebsd.org Mon Dec 28 01:51:22 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 43327A523F1 for ; Mon, 28 Dec 2015 01:51:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 34C101379 for ; Mon, 28 Dec 2015 01:51:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id tBS1pLmS079455 for ; Mon, 28 Dec 2015 01:51:22 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 194293] FUSE program freezes when seeking pos > file size Date: Mon, 28 Dec 2015 01:51:22 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: rmacklem@uoguelph.ca X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Dec 2015 01:51:22 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D194293 rmacklem@uoguelph.ca changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |rmacklem@uoguelph.ca --- Comment #5 from rmacklem@uoguelph.ca --- Created attachment 164744 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D164744&action= =3Dedit patch fuse so that a WRONLY open becomes a RDWR open This patch modifies the FreeBSD Fuse fs (in sys/fs/fuse) so that it always does a RDWR open when a WRONLY open is requested. I believe this bug happens when a write of a partial block occurs when not doing DIRECT_IO. When this happens the buffer cache first reads the entire block in. Without a RDWR open, this read fails and leaves the block "stuck", making umount etc. fail. The problem I see w.r.t. this patch is that a WRONLY open will fail if the process doesn't have Read access to the file. An alternate way to patch this would be to force DIRECT_IO for WRONLY opens. Comments/opinions? Hopefully nishida@ can test this patch to determine if it fixes the problem? --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Mon Dec 28 16:06:01 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 705F5A543BF for ; Mon, 28 Dec 2015 16:06:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49BC81180 for ; Mon, 28 Dec 2015 16:06:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id tBSG61QZ025777 for ; Mon, 28 Dec 2015 16:06:01 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 205668] [ext2fs] [patch] cannot mount EXT4 filesystems which weren't cleanly unmounted Date: Mon, 28 Dec 2015 16:06:01 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: damjan.jov@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status keywords bug_severity priority component assigned_to reporter cc attachments.created Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Dec 2015 16:06:01 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D205668 Bug ID: 205668 Summary: [ext2fs] [patch] cannot mount EXT4 filesystems which weren't cleanly unmounted Product: Base System Version: 11.0-CURRENT Hardware: Any OS: Any Status: New Keywords: patch Severity: Affects Some People Priority: --- Component: kern Assignee: freebsd-bugs@FreeBSD.org Reporter: damjan.jov@gmail.com CC: freebsd-fs@FreeBSD.org Created attachment 164769 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D164769&action= =3Dedit Patch to add support for the EXT2F_INCOMPAT_RECOVER flag If a disk with the EXT4 filesystem wasn't cleanly unmounted, it will have t= he EXT2F_INCOMPAT_RECOVER flag set on its feature flags, visible as "needs_recovery" in the "Filesystem features" of "tune2fs -l /dev/". Our ext2fs module doesn't know about this flag, and ends up treating a disk= in this state as having unsupported features, logging the inappropriate and confusing message: WARNING: mount of denied due to unsupported optional features to dmesg and unable to mount it until fsck.ext4 even if mounting read-only = or forced. My attached patch adds support for this flag and treats it as supported, warning if the filesystem wasn't cleanly unmounted like it already does with the other 2 ways of marking it as such. --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Mon Dec 28 17:05:20 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0E9A8A537FA for ; Mon, 28 Dec 2015 17:05:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F41AA152D for ; Mon, 28 Dec 2015 17:05:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id tBSH5J2B076994 for ; Mon, 28 Dec 2015 17:05:19 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 205668] [ext2fs] [patch] cannot mount EXT4 filesystems which weren't cleanly unmounted Date: Mon, 28 Dec 2015 17:05:20 +0000 X-Bugzilla-Reason: CC AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Dec 2015 17:05:20 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D205668 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org --=20 You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Mon Dec 28 17:15:41 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3094EA53B2A for ; Mon, 28 Dec 2015 17:15:41 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id BE4C11AF4 for ; Mon, 28 Dec 2015 17:15:40 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:QMW5JRLPmeDSnFEiRdmcpTZWNBhigK39O0sv0rFitYgULvTxwZ3uMQTl6Ol3ixeRBMOAu6wC07KempujcFJDyK7JiGoFfp1IWk1NouQttCtkPvS4D1bmJuXhdS0wEZcKflZk+3amLRodQ56mNBXsq3G/pQQfBg/4fVIsYL+lRMiK14ye7KObxd76W01wnj2zYLd/fl2djD76kY0ou7ZkMbs70RDTo3FFKKx8zGJsIk+PzV6nvp/jtM0rzyMFtegs+sRbXeD0ZOxsQ6ZVAT49PyU7/+XlrxTORxCDoHwGXTNFvABPBl3/7Rr5FrL4uSj+u+81jDOfNMb1Sb0xcSml4LpmTAfoziwOYW1quFrLg9B92foI6CmqoAZyltbZ X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2CsBAB4bIFW/61jaINehAxtBohTtlYahXWBWhABAQEBAQEBAYEJgi2CDiNoASICDRkCWwQuiBSaOY9vkSkMIYEBhVWGO4JqEQGDO4FJBY4wiFaFQIlth26FMRSOJAI5K4IeggogNIM3OoEIAQEB X-IronPort-AV: E=Sophos;i="5.20,491,1444708800"; d="scan'208";a="258603905" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 28 Dec 2015 12:14:31 -0500 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 3246D15F55D for ; Mon, 28 Dec 2015 12:14:31 -0500 (EST) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id NkLh8aZ7AVDi for ; Mon, 28 Dec 2015 12:14:30 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 9A08A15F565 for ; Mon, 28 Dec 2015 12:14:30 -0500 (EST) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id a9nFcZsPnvaW for ; Mon, 28 Dec 2015 12:14:30 -0500 (EST) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 802CA15F55D for ; Mon, 28 Dec 2015 12:14:30 -0500 (EST) Date: Mon, 28 Dec 2015 12:14:30 -0500 (EST) From: Rick Macklem To: freebsd-fs Message-ID: <1044566436.145078070.1451322870420.JavaMail.zimbra@uoguelph.ca> Subject: fixing a Fuse bug when writing (PR#194293?) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.12] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF43 (Win)/8.0.9_GA_6191) Thread-Topic: fixing a Fuse bug when writing (PR#194293?) Thread-Index: 3l+My+eJ1EgHujqSUyq3l26tfcVQZg== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Dec 2015 17:15:41 -0000 Hi, I sent a rather long email to a few people, but forgot to cc freebsd-fs@. Here's a shorter version: - When a file on a Fuse mounted fs is opened WRONLY and a partial block is written (when using the buffer cache, not DIRECT_IO), it gets wedged because the buffer cache code attempts to read in the whole block and the read isn't allowed. (I can reproduce this using GlusterFS and I think PR#194293 might be the same problem.) I can think of two ways to fix this: 1 - Make any WRONLY open actually do a RDWR open. The patch at the end of the email does that and fixes the problem for my test case. XXX - I think the problem is that if a process has write but not read access for the file, this RDWR open will fail and that won't make much sense, since it has write access. 2 - Make Fuse always do DIRECT_IO when a file is opened WRONLY. (This seems to happen for GlusterFS, since it seems to always set FOPEN_DIRECT_IO in the Fuse reply to the open for WRONLY, but I suspect other Fuse filesystems don't do this? I reproduced the problem by hacking the code to ignore the FOPEN_DIRECT_IO in the reply for GlusterFS.) I have coded #1 but not #2. However I am now thinking #2 is the better solution, given (XXX) above. Any other ideas or opinions w.r.t. how to fix this? Thanks, rick ps: Here's the patch I came up with for #1: --- fs/fuse/fuse_file.h.xxx 2015-12-27 16:02:53.241174000 -0500 +++ fs/fuse/fuse_file.h 2015-12-27 15:25:01.865156000 -0500 @@ -101,6 +101,9 @@ fuse_filehandle_xlate_from_fflags(int ff if ((fflags & FREAD) && (fflags & FWRITE)) { return FUFH_RDWR; } else if (fflags & (FWRITE)) { + /* See comment in fuse_vnop_open() w.r.t. why FUFH_RDWR is needed. */ + if (datacache != 0) + return FUFH_RDWR; return FUFH_WRONLY; } else if (fflags & (FREAD)) { return FUFH_RDONLY; --- fs/fuse/fuse_vnops.c.sav 2015-12-16 16:24:43.577000000 -0500 +++ fs/fuse/fuse_vnops.c 2015-12-27 16:07:06.683806000 -0500 @@ -276,7 +276,8 @@ fuse_vnop_close(struct vop_close_args *a if (fflag & IO_NDELAY) { return 0; } - fufh_type = fuse_filehandle_xlate_from_fflags(fflag); + fufh_type = fuse_filehandle_xlate_from_fflags(fflag, + fsess_opt_datacache(vnode_mount(vp))); if (!fuse_filehandle_valid(vp, fufh_type)) { int i; @@ -1139,7 +1140,8 @@ fuse_vnop_open(struct vop_open_args *ap) if (isdir) { fufh_type = FUFH_RDONLY; } else { - fufh_type = fuse_filehandle_xlate_from_fflags(mode); + fufh_type = fuse_filehandle_xlate_from_fflags(mode, + fsess_opt_datacache(vnode_mount(vp))); } if (fuse_filehandle_valid(vp, fufh_type)) { From owner-freebsd-fs@freebsd.org Mon Dec 28 17:17:21 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DD120A53C05 for ; Mon, 28 Dec 2015 17:17:21 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BCC531C3E for ; Mon, 28 Dec 2015 17:17:21 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id tBSHHDCU000173 for ; Mon, 28 Dec 2015 17:17:21 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201512281717.tBSHHDCU000173@kenobi.freebsd.org> From: bugzilla-noreply@FreeBSD.org To: freebsd-fs@FreeBSD.org Subject: Problem reports for freebsd-fs@FreeBSD.org that need special attention Date: Mon, 28 Dec 2015 17:17:21 +0000 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Dec 2015 17:17:22 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 203492 | mount_unionfs -o below causes panic Open | 136470 | [nfs] Cannot mount / in read-only, over NFS Open | 139651 | [nfs] mount(8): read-only remount of NFS volume d Open | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non f 4 problems total for which you should take action. From owner-freebsd-fs@freebsd.org Mon Dec 28 19:25:42 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 63221A53C36 for ; Mon, 28 Dec 2015 19:25:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 510A915E0 for ; Mon, 28 Dec 2015 19:25:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id tBSJPgbk076254 for ; Mon, 28 Dec 2015 19:25:42 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 205668] [ext2fs] [patch] cannot mount EXT4 filesystems which weren't cleanly unmounted Date: Mon, 28 Dec 2015 19:25:42 +0000 X-Bugzilla-Reason: CC AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: pfg@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: pfg@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status assigned_to cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Dec 2015 19:25:42 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D205668 Pedro F. Giffuni changed: What |Removed |Added ---------------------------------------------------------------------------- Status|New |In Progress Assignee|freebsd-fs@FreeBSD.org |pfg@FreeBSD.org CC| |pfg@FreeBSD.org --- Comment #1 from Pedro F. Giffuni --- I'll take it ... --=20 You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Mon Dec 28 20:10:01 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B1C01A54928 for ; Mon, 28 Dec 2015 20:10:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A14911B8B for ; Mon, 28 Dec 2015 20:10:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id tBSKA1Xb093546 for ; Mon, 28 Dec 2015 20:10:01 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 205668] [ext2fs] [patch] cannot mount EXT4 filesystems which weren't cleanly unmounted Date: Mon, 28 Dec 2015 20:10:01 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: pfg@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: pfg@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Dec 2015 20:10:01 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D205668 --- Comment #2 from Pedro F. Giffuni --- Created attachment 164776 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D164776&action= =3Dedit Ignore EXT2F_INCOMPAT_RECOVER when mounting ext4 The flag appears only in ext4 and appears to be realated with journalling, neither of which we actually support fully. This simpler patch should "fix" it. --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Tue Dec 29 15:52:39 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 748E3A55F2B for ; Tue, 29 Dec 2015 15:52:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6592A133A for ; Tue, 29 Dec 2015 15:52:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id tBTFqd5n001409 for ; Tue, 29 Dec 2015 15:52:39 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 205668] [ext2fs] [patch] cannot mount EXT4 filesystems which weren't cleanly unmounted Date: Tue, 29 Dec 2015 15:52:39 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: pfg@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Dec 2015 15:52:39 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D205668 --- Comment #3 from commit-hook@freebsd.org --- A commit references this bug: Author: pfg Date: Tue Dec 29 15:51:52 UTC 2015 New revision: 292872 URL: https://svnweb.freebsd.org/changeset/base/292872 Log: ext2: recognize ext4 INCOMPAT_RECOVER flag This is a flag specific for journalling in ext4. Add it to the list of ext4 features we ignore for read-only purposes. PR: 205668 MFC after: 1 week Changes: head/sys/fs/ext2fs/ext2fs.h --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Tue Dec 29 19:29:32 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3E941A553F4 for ; Tue, 29 Dec 2015 19:29:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2EF601E63 for ; Tue, 29 Dec 2015 19:29:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id tBTJTVGH091194 for ; Tue, 29 Dec 2015 19:29:32 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 194293] FUSE program freezes when seeking pos > file size Date: Tue, 29 Dec 2015 19:29:32 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: nishida@asusa.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Dec 2015 19:29:32 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D194293 --- Comment #6 from nishida@asusa.net --- Sorry for my late reply. I have just tested with fopen("rw") and have confirmed that "rw" did not fr= eeze my program and FUSE. Actually it returned "Operation not permitted" which is a correct response because the file size is < 100 and fseek(100) is 'nonseekable'. I will try the patch later. Thanks Hiroshi Nishida --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Dec 29 23:18:51 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D44D1A54392 for ; Tue, 29 Dec 2015 23:18:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C5C7A155B for ; Tue, 29 Dec 2015 23:18:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id tBTNIpK7010694 for ; Tue, 29 Dec 2015 23:18:51 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 194293] FUSE program freezes when seeking pos > file size Date: Tue, 29 Dec 2015 23:18:51 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: rmacklem@uoguelph.ca X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Dec 2015 23:18:51 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D194293 --- Comment #7 from rmacklem@uoguelph.ca --- Created attachment 164831 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D164831&action= =3Dedit patch fuse so it forces DIRECT_IO for WRONLY opens This patch forces fuse to use DIRECT_IO for files opened WRONLY, so it won't try and read a block in before writing a partial block. This fix may be preferable to the other patch, since it shouldn't make a WRONLY open fail because read/write isn't allowed. To put this in -head, fuse also needs to be patched to invalidate buffers when DIRECT_IO is enabled, because otherwise reads may get stale cached data. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Dec 30 01:12:48 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5A4A7A561E4 for ; Wed, 30 Dec 2015 01:12:48 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 1EC20164A for ; Wed, 30 Dec 2015 01:12:47 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:IIoJMxJPtpHtPzd+wNmcpTZWNBhigK39O0sv0rFitYgUKfTxwZ3uMQTl6Ol3ixeRBMOAu6wC07KempujcFJDyK7JiGoFfp1IWk1NouQttCtkPvS4D1bmJuXhdS0wEZcKflZk+3amLRodQ56mNBXsq3G/pQQfBg/4fVIsYL+lRMiK14ye7KObxd76W01wnj2zYLd/fl2djD76kY0ou7ZkMbs70RDTo3FFKKx8zGJsIk+PzV6nvp/jtM0rzyMFtPY87NJNS+D2cro1SZRXCCk9L20vosrxukrtVwyKs0EdWWZetxNDAAzI6VmuRJL4uSj+u+9VxS6VIMDyVbByUj30vPQjcwPhlCpSb21xy2rQkMEl1K8= X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2CoBABFLoNW/61jaINehH+IU7Zihg+BWBABAQEBAQEBAYEJgi2CDiNWEgEiAg0ZAluIRq09kQgBCwEggQGBLoQniiKCUIFJBY4wiFaLFoQtjQmOOAI5K4QoIIQ3gQgBAQE X-IronPort-AV: E=Sophos;i="5.20,498,1444708800"; d="scan'208";a="260404941" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-annu.net.uoguelph.ca with ESMTP; 29 Dec 2015 20:12:40 -0500 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 0CF4215F55D; Tue, 29 Dec 2015 20:12:41 -0500 (EST) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id 17xmOGaPISee; Tue, 29 Dec 2015 20:12:40 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 9771C15F565; Tue, 29 Dec 2015 20:12:40 -0500 (EST) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id bofj6ks2Kuwr; Tue, 29 Dec 2015 20:12:40 -0500 (EST) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 7C28D15F55D; Tue, 29 Dec 2015 20:12:40 -0500 (EST) Date: Tue, 29 Dec 2015 20:12:40 -0500 (EST) From: Rick Macklem To: gluster-devel@gluster.org Cc: freebsd-fs Message-ID: <571237035.145690509.1451437960464.JavaMail.zimbra@uoguelph.ca> Subject: FreeBSD port of GlusterFS racks up a lot of CPU usage MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.12] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF43 (Win)/8.0.9_GA_6191) Thread-Topic: FreeBSD port of GlusterFS racks up a lot of CPU usage Thread-Index: aV4DNqRHLPOyieq8iWggfGqw6sDM/A== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Dec 2015 01:12:48 -0000 Hi, I'm been playing with the FreeBSD port of GlusterFS and it seems to be working ok. I do notice that the daemons use a lot of CPU, even when there is nothing to do (no volumes started, etc). When I ktrace the daemon, I see a small number of nanosleep() and select() syscalls and lots of poll() syscalls (close to 1000/sec). Looking at libglusterfs/src/event-poll.c, I find: ret = poll(ufds, size, 1); in a loop. The only thing the code seems to do when poll() times out is a call to event_dispatch_poll_resize(). So, is it necessary to call event_dispatch_poll_resize() 1000 times per second? Or is there a way to make event_dispatch_poll_resize() return quickly when there is nothing to do? I'm guessing that Linux uses the event-epoll stuff instead of event-poll, so it wouldn't exhibit this. Is that correct? Thanks for any information on this, rick ps: I am tempted to just crank the timeout of 1msec up to 10 or 20msec. From owner-freebsd-fs@freebsd.org Wed Dec 30 10:41:01 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DE291A55CFE for ; Wed, 30 Dec 2015 10:41:00 +0000 (UTC) (envelope-from ndevos@redhat.com) Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C78521C10 for ; Wed, 30 Dec 2015 10:41:00 +0000 (UTC) (envelope-from ndevos@redhat.com) Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id 222E6263D; Wed, 30 Dec 2015 10:31:55 +0000 (UTC) Received: from localhost (vpn1-6-200.ams2.redhat.com [10.36.6.200]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id tBUAVrYV001077 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 30 Dec 2015 05:31:54 -0500 Date: Wed, 30 Dec 2015 11:31:52 +0100 From: Niels de Vos To: Rick Macklem Cc: gluster-devel@gluster.org, freebsd-fs Subject: Re: [Gluster-devel] FreeBSD port of GlusterFS racks up a lot of CPU usage Message-ID: <20151230103152.GS13942@ndevos-x240.usersys.redhat.com> References: <571237035.145690509.1451437960464.JavaMail.zimbra@uoguelph.ca> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="gqEssfNGWsEa4HfM" Content-Disposition: inline In-Reply-To: <571237035.145690509.1451437960464.JavaMail.zimbra@uoguelph.ca> User-Agent: Mutt/1.5.24 (2015-08-30) X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Dec 2015 10:41:01 -0000 --gqEssfNGWsEa4HfM Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Dec 29, 2015 at 08:12:40PM -0500, Rick Macklem wrote: > Hi, >=20 > I'm been playing with the FreeBSD port of GlusterFS and it seems > to be working ok. I do notice that the daemons use a lot of CPU, > even when there is nothing to do (no volumes started, etc). > When I ktrace the daemon, I see a small number of nanosleep() and > select() syscalls and lots of poll() syscalls (close to 1000/sec). >=20 > Looking at libglusterfs/src/event-poll.c, I find: > ret =3D poll(ufds, size, 1); > in a loop. The only thing the code seems to do when poll() times > out is a call to event_dispatch_poll_resize(). >=20 > So, is it necessary to call event_dispatch_poll_resize() 1000 times > per second? > Or is there a way to make event_dispatch_poll_resize() return quickly > when there is nothing to do? I do not think this is critical. A longer timeout should be well acceptable. > I'm guessing that Linux uses the event-epoll stuff instead of event-poll, > so it wouldn't exhibit this. Is that correct? Well, both. most (if not all) Linux builds will use event-poll. But, that calls epoll_wait() with a timeout of 1 millisecond as well. > Thanks for any information on this, rick > ps: I am tempted to just crank the timeout of 1msec up to 10 or 20msec. Yes, that is probably what I would do too. And have both poll functions use the same timeout, have it defined in libglusterfs/src/event.h. We could make it a configurable option too, but I do not think it is very useful to have. Could you file a bug and/or send a patch for this? Thanks, Niels --gqEssfNGWsEa4HfM Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBAgAGBQJWg7KYAAoJECXo5AApwsWz1C8P/R4HbFzWERHQYuMm0Rhvh7O9 Fcs+CKP2emf9uwAeGtrtwwfCfskcd2J15jJkN50loCBNOYaBgz2Wjf/OzhLU2NAB xyrQl0jEbe7e8UEvbip0NWMh5l6j8mC9ylob9DSpBwhh6o3hk3Sbf7DqXexjKsmW evGozvFStuO2HtQBxGnQThVNRgMJFYYafjTNOMkGKahPwLNPgO9A1wWFDEE31/gJ v7PFnnQ2Zi2GA17TPV2SoG7o0dsB7BmWlGDxoD0dPLMdhCO1jcimtF9GfZQ6H81e D6NBvIGkDUMqjnbYIpTB9msqcGYBqtMlEEtVbwiw24WIEtgmbJI/5SeZiU7TGpOr 4dTBTMwUrvzNvUY32LaB5pDsaHYUZIQpY233qsHd/QumjhEsD9mZBhNExApOLgPZ qHRNY8Zlsr/o4I7bPGr2weiZ0BcEj9u3GPSe76Sk0C3AcErwXUi19844L5zM6a9z H5DsCDam68LpVLg5LUK2nK/U8k1hs2xejpvnkRLsmmYeVcp9w+gBygQx2Iwoww4w HLOCADYdBALjC2phOKZEfeRJFnvvFANnP6tYFp+ss3WhZQB4pfH5aaO8FPwYbX2a a+u7YXQ3EYFxRXUOCTVxQKrcV7Tshjdt5dCy9F/YNkG5tfCJAFXx+QgLH8v+VXNP 3Fq4eYm8K66KPAtg/xcw =MKgt -----END PGP SIGNATURE----- --gqEssfNGWsEa4HfM-- From owner-freebsd-fs@freebsd.org Wed Dec 30 14:00:41 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 49E4BA560E1 for ; Wed, 30 Dec 2015 14:00:41 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id EA2BE1AF4 for ; Wed, 30 Dec 2015 14:00:40 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:5dHjSxKI6Ruyolc8t9mcpTZWNBhigK39O0sv0rFitYgULvnxwZ3uMQTl6Ol3ixeRBMOAu6wC07KempujcFJDyK7JiGoFfp1IWk1NouQttCtkPvS4D1bmJuXhdS0wEZcKflZk+3amLRodQ56mNBXsq3G/pQQfBg/4fVIsYL+lRMiK14ye7KObxd76W01wnj2zYLd/fl2djD76kY0ou7ZkMbs70RDTo3FFKKx8zGJsIk+PzV6nvp/jtM0rzyMFnfMs89UIXaiyQaMjBeheADk4NHsd/sDntRDfCwCI4y1PfH8Rl09yAgPGpDTzVZT1vy6y4vB40SKZOcDzZa0zVimv679rDhTh3nRUfwUl+X3a35QjxJlQpwis8kRy X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2CFCwC64oNW/61jaINehDhHiFO2QIYPAoFQEAEBAQEBAQEBgQmCLYIHAQEBAwEjBFIFCwIBCBgCAg0ZAgJXAgSIOgitaZEHAQEBAQEBBAEBAQEBAR2BAYEuhCeEf4dzgUkFjjCIVosWhBcWjQmKR4NxAjkrhCgghDeBCAEBAQ X-IronPort-AV: E=Sophos;i="5.20,500,1444708800"; d="scan'208";a="258858112" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 30 Dec 2015 09:00:33 -0500 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 4A5CA15F55D; Wed, 30 Dec 2015 09:00:33 -0500 (EST) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id A3FLOPgo_-Rh; Wed, 30 Dec 2015 09:00:32 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id A111115F565; Wed, 30 Dec 2015 09:00:32 -0500 (EST) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id oE1iBT2p6AiW; Wed, 30 Dec 2015 09:00:32 -0500 (EST) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 8320D15F55D; Wed, 30 Dec 2015 09:00:32 -0500 (EST) Date: Wed, 30 Dec 2015 09:00:32 -0500 (EST) From: Rick Macklem To: Niels de Vos Cc: gluster-devel@gluster.org, freebsd-fs Message-ID: <923007690.145828058.1451484032304.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <20151230103152.GS13942@ndevos-x240.usersys.redhat.com> References: <571237035.145690509.1451437960464.JavaMail.zimbra@uoguelph.ca> <20151230103152.GS13942@ndevos-x240.usersys.redhat.com> Subject: Re: [Gluster-devel] FreeBSD port of GlusterFS racks up a lot of CPU usage MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.11] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF43 (Win)/8.0.9_GA_6191) Thread-Topic: FreeBSD port of GlusterFS racks up a lot of CPU usage Thread-Index: RUJ47Q3BSevTS1ahT09HvnlYRojsfg== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Dec 2015 14:00:41 -0000 Niels de Vos wrote: > On Tue, Dec 29, 2015 at 08:12:40PM -0500, Rick Macklem wrote: > > Hi, > > > > I'm been playing with the FreeBSD port of GlusterFS and it seems > > to be working ok. I do notice that the daemons use a lot of CPU, > > even when there is nothing to do (no volumes started, etc). > > When I ktrace the daemon, I see a small number of nanosleep() and > > select() syscalls and lots of poll() syscalls (close to 1000/sec). > > > > Looking at libglusterfs/src/event-poll.c, I find: > > ret = poll(ufds, size, 1); > > in a loop. The only thing the code seems to do when poll() times > > out is a call to event_dispatch_poll_resize(). > > > > So, is it necessary to call event_dispatch_poll_resize() 1000 times > > per second? > > Or is there a way to make event_dispatch_poll_resize() return quickly > > when there is nothing to do? > > I do not think this is critical. A longer timeout should be well > acceptable. > > > I'm guessing that Linux uses the event-epoll stuff instead of event-poll, > > so it wouldn't exhibit this. Is that correct? > > Well, both. most (if not all) Linux builds will use event-poll. But, > that calls epoll_wait() with a timeout of 1 millisecond as well. > Actually, when I look at the 3.7.6 sources in libglusterfs/src/event-epoll.c I only find one epoll_wait() at line#668: ret = epoll_wait (event_pool->fd, &event, 1, -1); so the timeout never happens in this code. (It does have code after the call to handle the timeout case.) All it seems to do (if it were to timeout) is adjust the # of threads in the event-epoll case, so my hunch is that a timeout isn't needed? For the event-poll.c case, it calls event_dispatch_poll_size() which looks like it might add new file descriptors, so someone more familiar with this code would need to decide if the timeout can be disabled (my hunch is no, but I'm not familiar with the code). > > Thanks for any information on this, rick > > ps: I am tempted to just crank the timeout of 1msec up to 10 or 20msec. > > Yes, that is probably what I would do too. And have both poll functions > use the same timeout, have it defined in libglusterfs/src/event.h. We > could make it a configurable option too, but I do not think it is very > useful to have. > > Could you file a bug and/or send a patch for this? > I will try bumping the timeout up in event-poll.c and if it seems to reduce CPU usage and not cause any obvious grief, I will file a bug report. Thanks for your help with this, rick > Thanks, > Niels > From owner-freebsd-fs@freebsd.org Wed Dec 30 15:07:56 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0491BA55C39 for ; Wed, 30 Dec 2015 15:07:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D0C0C1CE5 for ; Wed, 30 Dec 2015 15:07:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id tBUF7tvQ022303 for ; Wed, 30 Dec 2015 15:07:55 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 178238] [nullfs] nullfs don't release i-nodes on unlink. Date: Wed, 30 Dec 2015 15:07:55 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: noah.bergbauer@tum.de X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Dec 2015 15:07:56 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D178238 noah.bergbauer@tum.de changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |noah.bergbauer@tum.de --- Comment #2 from noah.bergbauer@tum.de --- This PR is tagged 9.1-STABLE, but I just rediscovered this issue on 10.2-RELEASE. Easiest repro I could find: # preparation mkdir /testbed cd /testbed mkdir tmpmnt nullmnt mount -t tmpfs -o rw,size=3D10240 tmptestbed /testbed/tmpmnt/ # test 1 mount_nullfs /testbed/tmpmnt/ /testbed/nullmnt/ df -hi | grep testbed dd if=3D/dev/zero of=3Dnullmnt/testfile # to fill up the tmpfs df -hi | grep testbed # the filesystem is now full rm nullmnt/testfile=20 df -hi | grep testbed # bug: the filesystem is still full umount /testbed/nullmnt df -hi | grep testbed # the inode is released only once the nullfs is unmou= nted # test 2: nocache mount_nullfs -o nocache /testbed/tmpmnt/ /testbed/nullmnt/ df -hi | grep testbed dd if=3D/dev/zero of=3Dnullmnt/testfile rm nullmnt/testfile=20 df -hi | grep testbed # everything is working properly here umount /testbed/nullmnt So a standard nullfs mount is unusable for long-term operation. I've only f= ound out about the nocache option from an old mailing list post related to this = PR, so after 2.5 years without a fix a note about this workaround on the manpage would be great. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Dec 30 15:31:57 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 50260A56461 for ; Wed, 30 Dec 2015 15:31:57 +0000 (UTC) (envelope-from karl@denninger.net) Received: from mail.denninger.net (wsip-70-169-168-7.pn.at.cox.net [70.169.168.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C525619E4 for ; Wed, 30 Dec 2015 15:31:56 +0000 (UTC) (envelope-from karl@denninger.net) Received: from [192.168.1.40] (Karl-Desktop.Denninger.net [192.168.1.40]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.denninger.net (Postfix) with ESMTPSA id C79A42297C2 for ; Wed, 30 Dec 2015 09:31:48 -0600 (CST) Subject: Re: Panic in ZFS during zfs recv (while snapshots being destroyed) To: freebsd-fs@freebsd.org References: <55BB443E.8040801@denninger.net> <55CF7926.1030901@denninger.net> <55DF7191.2080409@denninger.net> <55DF76AA.3040103@denninger.net> <561FB7D0.4080107@denninger.net> <562662ED.1070302@denninger.net> <562A23B0.4050306@denninger.net> From: Karl Denninger X-Enigmail-Draft-Status: N1110 Message-ID: <5683F8E4.5090007@denninger.net> Date: Wed, 30 Dec 2015 09:31:48 -0600 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: <562A23B0.4050306@denninger.net> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms060309030704010707060405" X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Dec 2015 15:31:57 -0000 This is a cryptographically signed message in MIME format. --------------ms060309030704010707060405 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable I believe that the latest change to my patch fixes this both in patched and unpatched systems. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D187594 The root cause appears to be that while it should not be possible for dirty >=3D dirty_data_max in either the patched or unpatched case it in fact can occur, and if it does you get a zero value for the delay computation if the unsigned subtraction underflows (or an outright panic on an integer divide-by-zero attempt.) The latter of course produces an immediate panic but the former corrupts the running system and results in a later panic. Some other reported cases of what I believe is the same problem occurs with a reported divide-by-zero trap but when examining the variables in question the two operands do not result in a difference of zero. I am hesitant to declare that this change fixes it since I cannot provoke the panic "on demand"; it only occurs under heavy filesystem load including snapshot activity in my case, although another reported instance occurred during /etc/periodic runs against a huge number of jails at the same time, and is likely related. But, since putting the change in on 10 December I've not had the sentinels that mark the corrupt state which I had previously identified fire, nor have I had a panic occur. If you've run into a panic in the zfs code during very heavy and unusual use I'd be real interested in whether they stop if you run the last of the above patches; it should apply against 10-STABLE of approximately r289078 or later. Thanks; On 10/23/2015 07:10, Karl Denninger wrote: > While continuing to run this down I think I've managed to duplicate the= > state that produces the panic, but I don't know exactly how or why. > > The script (modified to check all returns) now produced this on a test = case: > > ...... > receiving incremental stream of > zsr/R/10.2-STABLE@zfs-auto-snap_frequent-2015-10-23-04h00 into > backup/R/10.2-STABLE@zfs-auto-snap_frequent-2015-10-23-04h00 > received 559MB stream in 22 seconds (25.4MB/sec) > receiving incremental stream of > zsr/R/10.2-STABLE@zfs-auto-snap_hourly-2015-10-23-04h04 into > backup/R/10.2-STABLE@zfs-auto-snap_hourly-2015-10-23-04h04 > received 4.25MB stream in 1 seconds (4.25MB/sec) > receiving incremental stream of > zsr/R/10.2-STABLE@zfs-auto-snap_frequent-2015-10-23-05h00 into > backup/R/10.2-STABLE@zfs-auto-snap_frequent-2015-10-23-05h00 > received 12.5MB stream in 1 seconds (12.5MB/sec) > receiving incremental stream of > zsr/R/10.2-STABLE@zfs-auto-snap_frequent-2015-10-23-06h00 into > backup/R/10.2-STABLE@zfs-auto-snap_frequent-2015-10-23-06h00 > received 13.4MB stream in 1 seconds (13.4MB/sec) > receiving incremental stream of zsr/R/10.2-STABLE@zfs-base into > backup/R/10.2-STABLE@zfs-base > received 14.8MB stream in 1 seconds (14.8MB/sec) > *will destroy zsr/R/10.2-STABLE@zfs-old** > **will reclaim 6.50M** > **cannot destroy snapshot zsr/R/10.2-STABLE@zfs-old: dataset is busy** > **Snapshot destroy FAILED with code 1* > > And there it stopped, as I have it trapped. > > But, there's nothing holding that dataset open: > > root@NewFS:~ # zfs holds zsr/R/10.2-STABLE@zfs-old > NAME TAG TIMESTAMP > > There is also no send or receive command still running, and an attempt > to force (or defer) the destroy fails too: > > root@NewFS:~ # zfs destroy zsr/R/10.2-STABLE@zfs-old > cannot destroy snapshot zsr/R/10.2-STABLE@zfs-old: dataset is busy > root@NewFS:~ # zfs destroy -d zsr/R/10.2-STABLE@zfs-old > cannot destroy snapshot zsr/R/10.2-STABLE@zfs-old: dataset is busy > root@NewFS:~ # zfs destroy -f zsr/R/10.2-STABLE@zfs-old > cannot destroy snapshot zsr/R/10.2-STABLE@zfs-old: dataset is busy > > Parameters: > > root@NewFS:~ # zfs get all zsr/R/10.2-STABLE@zfs-old > NAME PROPERTY VALUE = =20 > SOURCE > zsr/R/10.2-STABLE@zfs-old type snapshot = - > zsr/R/10.2-STABLE@zfs-old creation Thu Oct 22 10:14 2015= - > zsr/R/10.2-STABLE@zfs-old used 6.50M = - > zsr/R/10.2-STABLE@zfs-old referenced 25.7G = - > zsr/R/10.2-STABLE@zfs-old compressratio 1.86x = - > zsr/R/10.2-STABLE@zfs-old devices on = =20 > default > zsr/R/10.2-STABLE@zfs-old exec on = =20 > default > zsr/R/10.2-STABLE@zfs-old setuid on = =20 > default > zsr/R/10.2-STABLE@zfs-old xattr off = =20 > temporary > zsr/R/10.2-STABLE@zfs-old version 5 = - > zsr/R/10.2-STABLE@zfs-old utf8only off = - > zsr/R/10.2-STABLE@zfs-old normalization none = - > zsr/R/10.2-STABLE@zfs-old casesensitivity sensitive = - > zsr/R/10.2-STABLE@zfs-old nbmand off = =20 > default > zsr/R/10.2-STABLE@zfs-old primarycache all = =20 > default > zsr/R/10.2-STABLE@zfs-old secondarycache all = =20 > default > zsr/R/10.2-STABLE@zfs-old defer_destroy off = - > zsr/R/10.2-STABLE@zfs-old userrefs 0 = - > zsr/R/10.2-STABLE@zfs-old mlslabel = - > zsr/R/10.2-STABLE@zfs-old refcompressratio 1.86x = - > zsr/R/10.2-STABLE@zfs-old written 169M = - > zsr/R/10.2-STABLE@zfs-old clones = - > zsr/R/10.2-STABLE@zfs-old logicalused 0 = - > zsr/R/10.2-STABLE@zfs-old logicalreferenced 42.7G = - > zsr/R/10.2-STABLE@zfs-old volmode default = =20 > default > zsr/R/10.2-STABLE@zfs-old com.sun:auto-snapshot true = =20 > inherited from zsr/R/10.2-STABLE > > Nothing here that looks like a problem; specifically, no clones. > > If I run the script again and it attempts recovery (since zfs-old is > present) the machine panics with the below. Once the machine has > panic'd I can remove the snapshot. > > This looks like some sort of bug internally in the zfs state -- but the= > question is, now that I'm in this state how do I get out without a cras= h > and why did it happen, given that I can't find any reason for the > snapshot to be non-removable (e.g. an active clone, etc) > > Ideas or anything that would help find the source of the problem using = zdb? > > On 10/20/2015 10:51, Karl Denninger wrote: >> More data on this crash from this morning; I caught it in-process this= >> time and know exactly where it was in the backup script when it detona= ted. >> >> Here's the section of the script that was running when it blew up: >> >> for i in $ZFS >> do >> echo Processing $i >> >> FILESYS=3D`echo $i|cut -d/ -f2` >> >> zfs list $i@zfs-base >/dev/null 2>&1 >> result=3D$? >> if [ $result -eq 1 ]; >> then >> echo "Make and send zfs-base snapshot" >> zfs snapshot -r $i@zfs-base >> zfs send -R $i@zfs-base | zfs receive -Fuvd $BACKUP >> else >> base_only=3D`zfs get -H com.backup:base-only $i|grep t= rue` >> result=3D$? >> if [ $result -eq 1 ]; >> then >> echo "Bring incremental backup up to date" >> old_present=3D`zfs list $i@zfs-old >/dev/null = 2>&1` >> old=3D$? >> if [ $old -eq 0 ]; >> then >> echo "Previous incremental sync was >> interrupted; resume" >> else >> zfs rename -r $i@zfs-base $i@zfs-old >> zfs snapshot -r $i@zfs-base >> fi >> zfs send -RI $i@zfs-old $i@zfs-base | zfs >> receive -Fudv $BACKUP >> result=3D$? >> if [ $result -eq 0 ]; >> then >> zfs destroy -r $i@zfs-old >> zfs destroy -r $BACKUP/$FILESYS@zfs-ol= d >> else >> echo "STOP - - ERROR RUNNING COPY on $= i!!!!" >> exit 1 >> fi >> else >> echo "Base-only backup configured for $i" >> fi >> fi >> echo >> done >> >> And the output on the console when it happened: >> >> Begin local ZFS backup by SEND >> Run backups of default [zsr/R/10.2-STABLE zsr/ssd-txlog zs/archive >> zs/colo-archive zs/disk zs/pgsql zs/pics dbms/ticker] >> Tue Oct 20 10:14:09 CDT 2015 >> >> Import backup pool >> Imported; ready to proceed >> Processing zsr/R/10.2-STABLE >> Bring incremental backup up to date >> _*Previous incremental sync was interrupted; resume*_ >> attempting destroy backup/R/10.2-STABLE@zfs-auto-snap_daily-2015-10-13= -00h07 >> success >> attempting destroy >> backup/R/10.2-STABLE@zfs-auto-snap_hourly-2015-10-18-12h04 >> success >> *local fs backup/R/10.2-STABLE does not have fromsnap (zfs-old in stre= am);** >> **must have been deleted locally; ignoring* >> receiving incremental stream of >> zsr/R/10.2-STABLE@zfs-auto-snap_daily-2015-10-18-00h07 into >> backup/R/10.2-STABLE@zfs-auto-snap_daily-2015-10-18-00h07 >> snap backup/R/10.2-STABLE@zfs-auto-snap_daily-2015-10-18-00h07 already= >> exists; ignoring >> received 0B stream in 1 seconds (0B/sec) >> receiving incremental stream of >> zsr/R/10.2-STABLE@zfs-auto-snap_weekly-2015-10-18-00h14 into >> backup/R/10.2-STABLE@zfs-auto-snap_weekly-2015-10-18-00h14 >> snap backup/R/10.2-STABLE@zfs-auto-snap_weekly-2015-10-18-00h14 alread= y >> exists; ignoring >> received 0B stream in 1 seconds (0B/sec) >> receiving incremental stream of zsr/R/10.2-STABLE@zfs-base into >> backup/R/10.2-STABLE@zfs-base >> snap backup/R/10.2-STABLE@zfs-base already exists; ignoring >> >> That bolded pair of lines should _*not*_ be there. It means that the= >> snapshot "zfs-old" is on the source volume, but it shouldn't be there >> because after the copy is complete we destroy it on both the source an= d >> destination. Further, when it is attempted to be sent while the >> snapshot name (zfs-old) was found in a zfs list /*the data allegedly >> associated with the phantom snapshot that shouldn't exist was not ther= e >> */(second bolded line) >> >> zfs send -RI $i@zfs-old $i@zfs-base | zfs >> receive -Fudv $BACKUP >> result=3D$? >> if [ $result -eq 0 ]; >> then >> *zfs destroy -r $i@zfs-old** >> ** zfs destroy -r $BACKUP/$FILESYS@zfs-= old* >> else >> echo "STOP - - ERROR RUNNING COPY on $= i!!!!" >> exit 1 >> fi >> >> I don't have the trace from the last run (I didn't save it) *but there= >> were no errors returned by it *as it was run by hand (from the console= ) >> while I was watching it. >> >> This STRONGLY implies that the zfs destroy /allegedly /succeeded (it r= an >> without an error and actually removed the snapshot) but left the >> snapshot _*name*_ on the volume as it was able to be found by the >> script, and then when that was referenced by the backup script in the >> incremental send the data set was invalid producing the resulting pani= c. >> >> The bad news is that the pool shows no faults and a scrub which took >> place a few days ago says the pool is fine. If I re-run the backup I >> suspect it will properly complete (it always has in the past as a resu= me >> from the interrupted one) but clearly, whatever is going on here looks= >> like some sort of race in the destroy commands (which _*are*_ being ru= n >> recursively) that leaves the snapshot name on the volume but releases >> the data stored in it, thus the panic when it is attempted to be >> referenced. >> >> I have NOT run a scrub on the pool in an attempt to preserve whatever >> evidence may be there; if there is something that I can look at with z= db >> or similar I'll leave this alone for a bit -- the machine came back up= >> and is running fine. >> >> This is a production system but I can take it down off-hours for a sho= rt >> time if there is a need to run something in single-user mode using zdb= >> to try to figure out what's going on. There have been no known hardwa= re >> issues with it and all the other pools (including the ones on the same= >> host adapter) are and have been running without incident. >> >> Ideas? >> >> >> Fatal trap 12: page fault while in kernel mode >> cpuid =3D 9; apic id =3D 21 >> fault virtual address =3D 0x378 >> fault code =3D supervisor read data, page not present >> instruction pointer =3D 0x20:0xffffffff80940ae0 >> stack pointer =3D 0x28:0xfffffe0668018680 >> frame pointer =3D 0x28:0xfffffe0668018700 >> code segment =3D base 0x0, limit 0xfffff, type 0x1b >> =3D DPL 0, pres 1, long 1, def32 0, gran 1 >> processor eflags =3D interrupt enabled, resume, IOPL =3D 0 >> current process =3D 70921 (zfs) >> trap number =3D 12 >> panic: page fault >> cpuid =3D 9 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >> 0xfffffe0668018160 >> kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe0668018210 >> vpanic() at vpanic+0x126/frame 0xfffffe0668018250 >> panic() at panic+0x43/frame 0xfffffe06680182b0 >> trap_fatal() at trap_fatal+0x36b/frame 0xfffffe0668018310 >> trap_pfault() at trap_pfault+0x2ed/frame 0xfffffe06680183b0 >> trap() at trap+0x47a/frame 0xfffffe06680185c0 >> calltrap() at calltrap+0x8/frame 0xfffffe06680185c0 >> --- trap 0xc, rip =3D 0xffffffff80940ae0, rsp =3D 0xfffffe0668018680, = rbp =3D >> 0xfffffe0668018700 --- >> *__mtx_lock_sleep() at __mtx_lock_sleep+0x1b0/frame 0xfffffe0668018700= ** >> **dounmount() at dounmount+0x58/frame 0xfffffe0668018780** >> **zfs_unmount_snap() at zfs_unmount_snap+0x114/frame 0xfffffe06680187a= 0** >> **dsl_dataset_user_release_impl() at >> dsl_dataset_user_release_impl+0x103/frame 0xfffffe0668018920** >> **dsl_dataset_user_release_onexit() at >> dsl_dataset_user_release_onexit+0x5c/frame 0xfffffe0668018940** >> **zfs_onexit_destroy() at zfs_onexit_destroy+0x56/frame 0xfffffe066801= 8970** >> **zfsdev_close() at zfsdev_close+0x52/frame 0xfffffe0668018990* >> devfs_fpdrop() at devfs_fpdrop+0xa9/frame 0xfffffe06680189b0 >> devfs_close_f() at devfs_close_f+0x45/frame 0xfffffe06680189e0 >> _fdrop() at _fdrop+0x29/frame 0xfffffe0668018a00 >> closef() at closef+0x21e/frame 0xfffffe0668018a90 >> closefp() at closefp+0x98/frame 0xfffffe0668018ad0 >> amd64_syscall() at amd64_syscall+0x35d/frame 0xfffffe0668018bf0 >> Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0668018bf0 >> --- syscall (6, FreeBSD ELF64, sys_close), rip =3D 0x801a01f5a, rsp =3D= >> 0x7fffffffd1e8, rbp =3D 0x7fffffffd200 --- >> Uptime: 5d0h58m0s >> >> >> >> On 10/15/2015 09:27, Karl Denninger wrote: >>> Got another one of these this morning, after a long absence... >>> >>> Same symptom -- happened during a backup (send/receive) and it's in >>> the same place -- when the snapshot is unmounted. I have a clean dum= p >>> and this is against a quite-recent checkout, so the >>> most-currently-rolled forward ZFS changes are in this kernel. >>> >>> >>> Fatal trap 12: page fault while in kernel mode >>> cpuid =3D 14; apic id =3D 34 >>> fault virtual address =3D 0x378 >>> fault code =3D supervisor read data, page not present >>> instruction pointer =3D 0x20:0xffffffff80940ae0 >>> stack pointer =3D 0x28:0xfffffe066796c680 >>> frame pointer =3D 0x28:0xfffffe066796c700 >>> code segment =3D base 0x0, limit 0xfffff, type 0x1b >>> =3D DPL 0, pres 1, long 1, def32 0, gran 1 >>> processor eflags =3D interrupt enabled, resume, IOPL =3D 0 >>> current process =3D 81187 (zfs) >>> trap number =3D 12 >>> panic: page fault >>> cpuid =3D 14 >>> KDB: stack backtrace: >>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >>> 0xfffffe066796c160 >>> kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe066796c210 >>> vpanic() at vpanic+0x126/frame 0xfffffe066796c250 >>> panic() at panic+0x43/frame 0xfffffe066796c2b0 >>> trap_fatal() at trap_fatal+0x36b/frame 0xfffffe066796c310 >>> trap_pfault() at trap_pfault+0x2ed/frame 0xfffffe066796c3b0 >>> trap() at trap+0x47a/frame 0xfffffe066796c5c0 >>> calltrap() at calltrap+0x8/frame 0xfffffe066796c5c0 >>> --- trap 0xc, rip =3D 0xffffffff80940ae0, rsp =3D 0xfffffe066796c680,= rbp >>> =3D 0xfffffe >>> 066796c700 --- >>> __mtx_lock_sleep() at __mtx_lock_sleep+0x1b0/frame 0xfffffe066796c700= >>> dounmount() at dounmount+0x58/frame 0xfffffe066796c780 >>> zfs_unmount_snap() at zfs_unmount_snap+0x114/frame 0xfffffe066796c7a0= >>> dsl_dataset_user_release_impl() at >>> dsl_dataset_user_release_impl+0x103/frame 0xf >>> ffffe066796c920 >>> dsl_dataset_user_release_onexit() at >>> dsl_dataset_user_release_onexit+0x5c/frame >>> 0xfffffe066796c940 >>> zfs_onexit_destroy() at zfs_onexit_destroy+0x56/frame 0xfffffe066796c= 970 >>> zfsdev_close() at zfsdev_close+0x52/frame 0xfffffe066796c990 >>> devfs_fpdrop() at devfs_fpdrop+0xa9/frame 0xfffffe066796c9b0 >>> devfs_close_f() at devfs_close_f+0x45/frame 0xfffffe066796c9e0 >>> _fdrop() at _fdrop+0x29/frame 0xfffffe066796ca00 >>> closef() at closef+0x21e/frame 0xfffffe066796ca90 >>> closefp() at closefp+0x98/frame 0xfffffe066796cad0 >>> amd64_syscall() at amd64_syscall+0x35d/frame 0xfffffe066796cbf0 >>> Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe066796cbf0 >>> --- syscall (6, FreeBSD ELF64, sys_close), rip =3D 0x801a01f5a, rsp =3D= >>> 0x7fffffffd728, rbp =3D 0x7fffffffd740 --- >>> Uptime: 3d15h37m26s >>> >>> >>> On 8/27/2015 15:44, Karl Denninger wrote: >>>> No, but that does sound like it might be involved..... >>>> >>>> And yeah, this did start when I moved the root pool to a mirrored pa= ir >>>> of Intel 530s off a pair of spinning-rust WD RE4s.... (The 530s are = darn >>>> nice performance-wise, reasonably inexpensive and thus very suitable= for >>>> a root filesystem drive and they also pass the "pull the power cord"= >>>> test, incidentally.) >>>> >>>> You may be onto something -- I'll try shutting it off, but due to th= e >>>> fact that I can't make this happen and it's a "every week or two" pa= nic, >>>> but ALWAYS when the zfs send | zfs recv is running AND it's always o= n >>>> the same filesystem it will be a fair while before I know if it's fi= xed >>>> (like over a month, given the usual pattern here, as that would be 4= >>>> "average" periods without a panic)..... >>>> >>>> I also wonder if I could tune this out with some of the other TRIM >>>> parameters instead of losing it entirely. >>>> >>>> vfs.zfs.trim.max_interval: 1 >>>> vfs.zfs.trim.timeout: 30 >>>> vfs.zfs.trim.txg_delay: 32 >>>> vfs.zfs.trim.enabled: 1 >>>> vfs.zfs.vdev.trim_max_pending: 10000 >>>> vfs.zfs.vdev.trim_max_active: 64 >>>> vfs.zfs.vdev.trim_min_active: 1 >>>> >>>> That it's panic'ing on a mtx_lock_sleep might point this way.... the= >>>> trace shows it coming from a zfs_onexit_destroy, which ends up calli= ng >>>> zfs_unmount_snap() and then it blows in dounmount() while executing >>>> mtx_lock_sleep(). >>>> >>>> I do wonder if I'm begging for new and innovative performance issues= if >>>> I run with TRIM off for an extended period of time, however..... :-)= >>>> >>>> >>>> On 8/27/2015 15:30, Sean Chittenden wrote: >>>>> Have you tried disabling TRIM? We recently ran in to an issue wher= e a `zfs delete` on a large dataset caused the host to panic because TRIM= was tripping over the ZFS deadman timer. Disabling TRIM worked as vali= d workaround for us. ? You mentioned a recent move to SSDs, so this can= happen, esp after the drive has experienced a little bit of actual work.= ? -sc >>>>> >>>>> >>>>> -- >>>>> Sean Chittenden >>>>> sean@chittenden.org >>>>> >>>>> >>>>>> On Aug 27, 2015, at 13:22, Karl Denninger wro= te: >>>>>> >>>>>> On 8/15/2015 12:38, Karl Denninger wrote: >>>>>>> Update: >>>>>>> >>>>>>> This /appears /to be related to attempting to send or receive a >>>>>>> /cloned /snapshot. >>>>>>> >>>>>>> I use /beadm /to manage boot environments and the crashes have al= l >>>>>>> come while send/recv-ing the root pool, which is the one where th= ese >>>>>>> clones get created. It is /not /consistent within a given snapsh= ot >>>>>>> when it crashes and a second attempt (which does a "recovery" >>>>>>> send/receive) succeeds every time -- I've yet to have it panic tw= ice >>>>>>> sequentially. >>>>>>> >>>>>>> I surmise that the problem comes about when a file in the cloned >>>>>>> snapshot is modified, but this is a guess at this point. >>>>>>> >>>>>>> I'm going to try to force replication of the problem on my test s= ystem. >>>>>>> >>>>>>> On 7/31/2015 04:47, Karl Denninger wrote: >>>>>>>> I have an automated script that runs zfs send/recv copies to bri= ng a >>>>>>>> backup data set into congruence with the running copies nightly.= The >>>>>>>> source has automated snapshots running on a fairly frequent basi= s >>>>>>>> through zfs-auto-snapshot. >>>>>>>> >>>>>>>> Recently I have started having a panic show up about once a week= during >>>>>>>> the backup run, but it's inconsistent. It is in the same place,= but I >>>>>>>> cannot force it to repeat. >>>>>>>> >>>>>>>> The trap itself is a page fault in kernel mode in the zfs code a= t >>>>>>>> zfs_unmount_snap(); here's the traceback from the kvm (sorry for= the >>>>>>>> image link but I don't have a better option right now.) >>>>>>>> >>>>>>>> I'll try to get a dump, this is a production machine with encryp= ted swap >>>>>>>> so it's not normally turned on. >>>>>>>> >>>>>>>> Note that the pool that appears to be involved (the backup pool)= has >>>>>>>> passed a scrub and thus I would assume the on-disk structure is = ok..... >>>>>>>> but that might be an unfair assumption. It is always occurring = in the >>>>>>>> same dataset although there are a half-dozen that are sync'd -- = if this >>>>>>>> one (the first one) successfully completes during the run then a= ll the >>>>>>>> rest will as well (that is, whenever I restart the process it ha= s always >>>>>>>> failed here.) The source pool is also clean and passes a scrub.= >>>>>>>> >>>>>>>> traceback is at http://www.denninger.net/kvmimage.png; apologies= for the >>>>>>>> image traceback but this is coming from a remote KVM. >>>>>>>> >>>>>>>> I first saw this on 10.1-STABLE and it is still happening on Fre= eBSD >>>>>>>> 10.2-PRERELEASE #9 r285890M, which I updated to in an attempt to= see if >>>>>>>> the problem was something that had been addressed. >>>>>>>> >>>>>>>> >>>>>>> --=20 >>>>>>> Karl Denninger >>>>>>> karl@denninger.net >>>>>>> /The Market Ticker/ >>>>>>> /[S/MIME encrypted email preferred]/ >>>>>> Second update: I have now taken another panic on 10.2-Stable, same= deal, >>>>>> but without any cloned snapshots in the source image. I had though= t that >>>>>> removing cloned snapshots might eliminate the issue; that is now o= ut the >>>>>> window. >>>>>> >>>>>> It ONLY happens on this one filesystem (the root one, incidentally= ) >>>>>> which is fairly-recently created as I moved this machine from spin= ning >>>>>> rust to SSDs for the OS and root pool -- and only when it is being= >>>>>> backed up by using zfs send | zfs recv (with the receive going to = a >>>>>> different pool in the same machine.) I have yet to be able to pro= voke >>>>>> it when using zfs send to copy to a different machine on the same = LAN, >>>>>> but given that it is not able to be reproduced on demand I can't b= e >>>>>> certain it's timing related (e.g. performance between the two pool= s in >>>>>> question) or just that I haven't hit the unlucky combination. >>>>>> >>>>>> This looks like some sort of race condition and I will continue to= see >>>>>> if I can craft a case to make it occur "on demand" >>>>>> >>>>>> --=20 >>>>>> Karl Denninger >>>>>> karl@denninger.net >>>>>> /The Market Ticker/ >>>>>> /[S/MIME encrypted email preferred]/ >>>>> %SPAMBLOCK-SYS: Matched [+Sean Chittenden ], m= essage ok >>>>> >>> --=20 >>> Karl Denninger >>> karl@denninger.net >>> /The Market Ticker/ >>> /[S/MIME encrypted email preferred]/ --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms060309030704010707060405 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC Bl8wggZbMIIEQ6ADAgECAgEpMA0GCSqGSIb3DQEBCwUAMIGQMQswCQYDVQQGEwJVUzEQMA4G A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND dWRhIFN5c3RlbXMgTExDIENBMB4XDTE1MDQyMTAyMjE1OVoXDTIwMDQxOTAyMjE1OVowWjEL MAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHjAcBgNVBAMTFUthcmwgRGVubmluZ2VyIChPQ1NQKTCCAiIwDQYJKoZIhvcNAQEBBQAD ggIPADCCAgoCggIBALmEWPhAdphrWd4K5VTvE5pxL3blRQPyGF3ApjUjgtavqU1Y8pbI3Byg XDj2/Uz9Si8XVj/kNbKEjkRh5SsNvx3Fc0oQ1uVjyCq7zC/kctF7yLzQbvWnU4grAPZ3IuAp 3/fFxIVaXpxEdKmyZAVDhk9az+IgHH43rdJRIMzxJ5vqQMb+n2EjadVqiGPbtG9aZEImlq7f IYDTnKyToi23PAnkPwwT+q1IkI2DTvf2jzWrhLR5DTX0fUYC0nxlHWbjgpiapyJWtR7K2YQO aevQb/3vN9gSojT2h+cBem7QIj6U69rEYcEDvPyCMXEV9VcXdcmW42LSRsPvZcBHFkWAJqMZ Myiz4kumaP+s+cIDaXitR/szoqDKGSHM4CPAZV9Yh8asvxQL5uDxz5wvLPgS5yS8K/o7zDR5 vNkMCyfYQuR6PAJxVOk5Arqvj9lfP3JSVapwbr01CoWDBkpuJlKfpQIEeC/pcCBKknllbMYq yHBO2TipLyO5Ocd1nhN/nOsO+C+j31lQHfOMRZaPQykXVPWG5BbhWT7ttX4vy5hOW6yJgeT/ o3apynlp1cEavkQRS8uJHoQszF6KIrQMID/JfySWvVQ4ksnfzwB2lRomrdrwnQ4eG/HBS+0l eozwOJNDIBlAP+hLe8A5oWZgooIIK/SulUAsfI6Sgd8dTZTTYmlhAgMBAAGjgfQwgfEwNwYI KwYBBQUHAQEEKzApMCcGCCsGAQUFBzABhhtodHRwOi8vY3VkYXN5c3RlbXMubmV0Ojg4ODgw CQYDVR0TBAIwADARBglghkgBhvhCAQEEBAMCBaAwCwYDVR0PBAQDAgXgMCwGCWCGSAGG+EIB DQQfFh1PcGVuU1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUxRyULenJaFwX RtT79aNmIB/u5VkwHwYDVR0jBBgwFoAUJHGbnYV9/N3dvbDKkpQDofrTbTUwHQYDVR0RBBYw FIESa2FybEBkZW5uaW5nZXIubmV0MA0GCSqGSIb3DQEBCwUAA4ICAQBPf3cYtmKowmGIYsm6 eBinJu7QVWvxi1vqnBz3KE+HapqoIZS8/PolB/hwiY0UAE1RsjBJ7yEjihVRwummSBvkoOyf G30uPn4yg4vbJkR9lTz8d21fPshWETa6DBh2jx2Qf13LZpr3Pj2fTtlu6xMYKzg7cSDgd2bO sJGH/rcvva9Spkx5Vfq0RyOrYph9boshRN3D4tbWgBAcX9POdXCVfJONDxhfBuPHsJ6vEmPb An+XL5Yl26XYFPiODQ+Qbk44Ot1kt9s7oS3dVUrh92Qv0G3J3DF+Vt6C15nED+f+bk4gScu+ JHT7RjEmfa18GT8DcT//D1zEke1Ymhb41JH+GyZchDRWtjxsS5OBFMzrju7d264zJUFtX7iJ 3xvpKN7VcZKNtB6dLShj3v/XDsQVQWXmR/1YKWZ93C3LpRs2Y5nYdn6gEOpL/WfQFThtfnat HNc7fNs5vjotaYpBl5H8+VCautKbGOs219uQbhGZLYTv6okuKcY8W+4EJEtK0xB08vqr9Jd0 FS9MGjQE++GWo+5eQxFt6nUENHbVYnsr6bYPQsZH0CRNycgTG9MwY/UIXOf4W034UpR82TBG 1LiMsYfb8ahQJhs3wdf1nzipIjRwoZKT1vGXh/cj3gwSr64GfenURBxaFZA5O1acOZUjPrRT n3ci4McYW/0WVVA3lDGCBRMwggUPAgEBMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4GA1UECBMH RmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExD MRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhNDdWRhIFN5 c3RlbXMgTExDIENBAgEpMA0GCWCGSAFlAwQCAwUAoIICTTAYBgkqhkiG9w0BCQMxCwYJKoZI hvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNTEyMzAxNTMxNDhaME8GCSqGSIb3DQEJBDFCBEAu /NcekvOiJzvDSzExhY7Nna6q42VWC3RLfsmEsa3g+C8GnwkwokmpghJsME/LOPyMqF/ef4dK L6c5p9Y5RfzVMGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAK BggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYI KoZIhvcNAwICASgwgacGCSsGAQQBgjcQBDGBmTCBljCBkDELMAkGA1UEBhMCVVMxEDAOBgNV BAgTB0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1z IExMQzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqGSIb3DQEJARYTQ3Vk YSBTeXN0ZW1zIExMQyBDQQIBKTCBqQYLKoZIhvcNAQkQAgsxgZmggZYwgZAxCzAJBgNVBAYT AlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1 ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG 9w0BCQEWE0N1ZGEgU3lzdGVtcyBMTEMgQ0ECASkwDQYJKoZIhvcNAQEBBQAEggIAnt3jAMQb GQFrwzseDKuApQz2F5kZ4FxJ7Y0fhH12F+MwBI6c4F5+b1vMJ6TIR0yTek3uXdFZKyhwe3WH XS7GQx+wXGeG+6EKAi04DczyfeNMiVXFBV3RcGYn1aTSwutAaXT5wMd9ewzA7jAPGIIDCIYx uqd1G2yXNDi5j4nAAOOS8DhukTZ1HEqtCyJvcrGtDQxMfDz5qjMJRgGD01S8e94qyRd/avKC VZ06xqBIWzwZMb/ej4exqgAby6WbO2SAcHYKjVOuNCg8E1i4OP6Oto6pz81daSm/84hN3XX2 9o8tshhfTbw0JIcXu/TY4nnSvdyGtGnR89gC8VjC+8cLLDtjoweykAo3kNPDVdJfHZABRuD0 BSuI3utmQ2pSNh0uJn/rl2NnLOAxvBzjaxkcwPPoudLRVMJG8RnqHY2YT5OK7CFS0Mm0xmac 0JoxIF1rro84hwLErgtqKlwz5bppzLgca8X/JDnWXSPQx5LrJhC30cTontUwSx02iDUDzLnD 1rG5x/cJ7XqAau1oOdskSXQ+2qKMOX2t8QlmXW5GmQErzXsOwC+cgSqYfHoer+EzpM6iOK59 1JbQsh+vDjDTdMbwA/pT9BR4CFmaFUpoHZIljOjErQs3ERiwnAXizMQEv2f7qVzD2hRkfYyB 5kZ7woU8JHRoN84uwCK8+z2rKM4AAAAAAAA= --------------ms060309030704010707060405-- From owner-freebsd-fs@freebsd.org Wed Dec 30 16:18:12 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 67D1AA55379 for ; Wed, 30 Dec 2015 16:18:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 58A941FDB for ; Wed, 30 Dec 2015 16:18:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id tBUGIBPM001671 for ; Wed, 30 Dec 2015 16:18:12 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 178238] [nullfs] nullfs don't release i-nodes on unlink. Date: Wed, 30 Dec 2015 16:18:11 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: kib@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Dec 2015 16:18:12 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D178238 Konstantin Belousov changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kib@FreeBSD.org --- Comment #3 from Konstantin Belousov --- Created attachment 164869 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D164869&action= =3Dedit Force reclaim of the upper vnode on unlink. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Dec 30 18:22:13 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5CEA6A55C81 for ; Wed, 30 Dec 2015 18:22:13 +0000 (UTC) (envelope-from jkh@ixsystems.com) Received: from barracuda.ixsystems.com (mail.ixsystems.com [69.198.165.135]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.ixsystems.com", Issuer "Go Daddy Secure Certificate Authority - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 43B1017B5 for ; Wed, 30 Dec 2015 18:22:12 +0000 (UTC) (envelope-from jkh@ixsystems.com) X-ASG-Debug-ID: 1451499731-08ca042abc3ef070001-3nHGF7 Received: from mail.iXsystems.com ([10.2.55.1]) by barracuda.ixsystems.com with ESMTP id 4CsQcsiE2gd6Xtud (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 30 Dec 2015 10:22:11 -0800 (PST) X-Barracuda-Envelope-From: jkh@ixsystems.com X-Barracuda-RBL-Trusted-Forwarder: 10.2.55.1 X-ASG-Whitelist: Client Received: from [10.8.0.42] (unknown [10.8.0.42]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.iXsystems.com (Postfix) with ESMTPSA id C6C6FA2EF8; Wed, 30 Dec 2015 10:22:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=ixsystems.com; s=newknight0; t=1451499730; bh=V0pYgzlJCaMtGdXkFLbD38w6SWdxPlemrvEXxgTqD5I=; h=Subject:From:In-Reply-To:Date:Cc:References:To; b=fqK1nPd/KZ1xq5AC73KREOEmr37raCzjyYneQ0zCkk+c1ML8Q6OXkZN0DqR8w0/Bv vEw3YKW93x2dI5OZLwt6xN33xYEHdHWZCzQJhAZbQJ1aDx0zlSOXG9v3MEHuJURKiJ Bowb99cOTFfd201zK3uld1N7q83IXl8mQc8eWXPw= Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: Re: [Gluster-devel] FreeBSD port of GlusterFS racks up a lot of CPU usage From: Hubbard Jordan X-ASG-Orig-Subj: Re: [Gluster-devel] FreeBSD port of GlusterFS racks up a lot of CPU usage In-Reply-To: <20151230103152.GS13942@ndevos-x240.usersys.redhat.com> Date: Wed, 30 Dec 2015 10:22:07 -0800 Cc: Rick Macklem , freebsd-fs , gluster-devel@gluster.org Content-Transfer-Encoding: quoted-printable Message-Id: <2D8C2729-D556-479B-B4E2-66E1BB222F41@ixsystems.com> References: <571237035.145690509.1451437960464.JavaMail.zimbra@uoguelph.ca> <20151230103152.GS13942@ndevos-x240.usersys.redhat.com> To: Niels de Vos X-Mailer: Apple Mail (2.3112) X-Barracuda-Connect: UNKNOWN[10.2.55.1] X-Barracuda-Start-Time: 1451499731 X-Barracuda-Encrypted: DHE-RSA-CAMELLIA256-SHA X-Barracuda-URL: https://10.2.0.41:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at ixsystems.com X-Barracuda-BRTS-Status: 1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Dec 2015 18:22:13 -0000 > On Dec 30, 2015, at 2:31 AM, Niels de Vos wrote: >=20 >> I'm guessing that Linux uses the event-epoll stuff instead of = event-poll, >> so it wouldn't exhibit this. Is that correct? >=20 > Well, both. most (if not all) Linux builds will use event-poll. But, > that calls epoll_wait() with a timeout of 1 millisecond as well. >=20 >> Thanks for any information on this, rick >> ps: I am tempted to just crank the timeout of 1msec up to 10 or = 20msec. >=20 > Yes, that is probably what I would do too. And have both poll = functions > use the same timeout, have it defined in libglusterfs/src/event.h. We > could make it a configurable option too, but I do not think it is very > useful to have. I guess this begs the question - what=E2=80=99s the actual purpose of = polling for an event with a 1 millisecond timeout? If it was some sort = of heartbeat check, one might imagine that would be better served by a = timer with nothing close to 1 millisecond as an interval (that would be = one seriously aggressive heartbeat) and if filesystem events are = incoming that glusterfs needs to respond to, why timeout at all? I also have a broader question to go with the specific one: We (at = iXsystems) were attempting to engage with some of the Red Hat folks back = when the FreeBSD port was first done, in the hope of getting it more = =E2=80=9Cofficially supported=E2=80=9D for FreeBSD and perhaps even = donating some more serious stress-testing and integration work for it, = but when those Red Hat folks moved on we lost continuity and the effort = stalled. Who at Red Hat would / could we work with in getting this back = on track? We=E2=80=99d like to integrate glusterfs with FreeNAS 10, and = in fact have already done so but it=E2=80=99s still early days and = we=E2=80=99re not even really sure what we have yet. Thanks, - Jordan From owner-freebsd-fs@freebsd.org Wed Dec 30 19:25:35 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2C9C7A56EFA for ; Wed, 30 Dec 2015 19:25:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 19BB91295 for ; Wed, 30 Dec 2015 19:25:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id tBUJPYgV024353 for ; Wed, 30 Dec 2015 19:25:34 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 194293] FUSE program freezes when seeking pos > file size Date: Wed, 30 Dec 2015 19:25:35 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: nishida@asusa.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Dec 2015 19:25:35 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D194293 --- Comment #8 from nishida@asusa.net --- (In reply to rmacklem from comment #7) I tested the patch on 10.2-RELEASE-p8 and confirmed it worked. However, the behavior of fseek() beyond EOF changed to "fseek() extends the original file and writes data at the seek point." Since fseek() beyond EOF is probably undefined, it will be OK and is much better than freeze. Thanks. Hiroshi Nishida --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Dec 30 19:50:04 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F28A0A55576 for ; Wed, 30 Dec 2015 19:50:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E337F1E22 for ; Wed, 30 Dec 2015 19:50:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id tBUJo4qR068253 for ; Wed, 30 Dec 2015 19:50:04 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 178238] [nullfs] nullfs don't release i-nodes on unlink. Date: Wed, 30 Dec 2015 19:50:05 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Dec 2015 19:50:05 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D178238 --- Comment #4 from commit-hook@freebsd.org --- A commit references this bug: Author: kib Date: Wed Dec 30 19:49:22 UTC 2015 New revision: 292961 URL: https://svnweb.freebsd.org/changeset/base/292961 Log: Force nullfs vnode reclaim after unlinking, to potentially unlink lower vnode. Otherwise, reference to the lower vnode from the upper one prevents final unlink. PR: 178238 Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week Changes: head/sys/fs/nullfs/null_vnops.c --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Dec 30 20:16:09 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 392AFA55DB8 for ; Wed, 30 Dec 2015 20:16:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 29BCE1A0C for ; Wed, 30 Dec 2015 20:16:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id tBUKG9iL052412 for ; Wed, 30 Dec 2015 20:16:09 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 178238] [nullfs] nullfs don't release i-nodes on unlink. Date: Wed, 30 Dec 2015 20:16:09 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Dec 2015 20:16:09 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D178238 --- Comment #5 from commit-hook@freebsd.org --- A commit references this bug: Author: pho Date: Wed Dec 30 20:15:30 UTC 2015 New revision: 292962 URL: https://svnweb.freebsd.org/changeset/base/292962 Log: Added a regression test. PR: 178238 Sponsored by: EMC / Isilon storage division Changes: user/pho/stress2/misc/nullfs16.sh --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Dec 30 23:26:21 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1C45EA56432 for ; Wed, 30 Dec 2015 23:26:21 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id C61D415C6 for ; Wed, 30 Dec 2015 23:26:20 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:KRIJyBY0sn/QE8ZZDoA2+Q3/LSx+4OfEezUN459isYplN5qZpci9bnLW6fgltlLVR4KTs6sC0LqI9fi4EUU7or+/81k6OKRWUBEEjchE1ycBO+WiTXPBEfjxciYhF95DXlI2t1uyMExSBdqsLwaK+i760zceF13FOBZvIaytQ8iJ35rxj7j60qaQSjsLrQL1Wal1IhSyoFeZnegtqqwmFJwMzADUqGBDYeVcyDAgD1uSmxHh+pX4p8Y7oGwD884mou5dW6/zZahwb7tCAD0gezQ3583DtAnYXBCT634HFG4Rl0wbLRLC6UTAX5zy+g7zvel51SzSadfzRLs3XTmnx7psRwLljD8HcTUwpjKEwvdshb5W9Ury7yd0xJTZNdmY X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2CsBADwZoRW/61jaINehHkGiFO2RIYPAoFQEAEBAQEBAQEBgQmCLYIHAQEBAwEjVgULAgEIDgoCAg0ZAgJXAgQTiCcIrjWQfgEBAQEGAQEBAR+BAYEuhCeEf4QmEQEdgx6BSQWCcItAiFaPLYduhTGKR4NxAjkrhCggNINSOoEIAQEB X-IronPort-AV: E=Sophos;i="5.20,502,1444708800"; d="scan'208";a="260529533" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-annu.net.uoguelph.ca with ESMTP; 30 Dec 2015 18:26:18 -0500 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id BAF7915F55D; Wed, 30 Dec 2015 18:26:18 -0500 (EST) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id RBIatIWMlVHx; Wed, 30 Dec 2015 18:26:18 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id EB79E15F565; Wed, 30 Dec 2015 18:26:17 -0500 (EST) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id m5pbA3j6r6q7; Wed, 30 Dec 2015 18:26:17 -0500 (EST) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id CDDDA15F55D; Wed, 30 Dec 2015 18:26:17 -0500 (EST) Date: Wed, 30 Dec 2015 18:26:17 -0500 (EST) From: Rick Macklem To: Hubbard Jordan Cc: Niels de Vos , freebsd-fs , gluster-devel@gluster.org Message-ID: <1083933309.146084334.1451517977647.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <2D8C2729-D556-479B-B4E2-66E1BB222F41@ixsystems.com> References: <571237035.145690509.1451437960464.JavaMail.zimbra@uoguelph.ca> <20151230103152.GS13942@ndevos-x240.usersys.redhat.com> <2D8C2729-D556-479B-B4E2-66E1BB222F41@ixsystems.com> Subject: Re: [Gluster-devel] FreeBSD port of GlusterFS racks up a lot of CPU usage MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.95.12] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF43 (Win)/8.0.9_GA_6191) Thread-Topic: FreeBSD port of GlusterFS racks up a lot of CPU usage Thread-Index: BLAXDsFlqcTtWjMaPr5EU6vvMTL3VQ== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Dec 2015 23:26:21 -0000 Jordan Hubbard wrote: >=20 > > On Dec 30, 2015, at 2:31 AM, Niels de Vos wrote: > >=20 > >> I'm guessing that Linux uses the event-epoll stuff instead of event-po= ll, > >> so it wouldn't exhibit this. Is that correct? > >=20 > > Well, both. most (if not all) Linux builds will use event-poll. But, > > that calls epoll_wait() with a timeout of 1 millisecond as well. > >=20 > >> Thanks for any information on this, rick > >> ps: I am tempted to just crank the timeout of 1msec up to 10 or 20msec= . > >=20 > > Yes, that is probably what I would do too. And have both poll functions > > use the same timeout, have it defined in libglusterfs/src/event.h. We > > could make it a configurable option too, but I do not think it is very > > useful to have. >=20 > I guess this begs the question - what=E2=80=99s the actual purpose of pol= ling for an > event with a 1 millisecond timeout? If it was some sort of heartbeat che= ck, > one might imagine that would be better served by a timer with nothing clo= se > to 1 millisecond as an interval (that would be one seriously aggressive > heartbeat) and if filesystem events are incoming that glusterfs needs to > respond to, why timeout at all? >=20 If I understand the code (I probably don't) the timeout allows the loop to call a function that may add new fd's to be polled. (If I'm right, the new ones might not get serviced.) I'll post once I've tried a longer timeout and if it seems ok, I will put it in the Redhat bugs database (as mentioned in the last post). In its current form, it's fine for testing. > I also have a broader question to go with the specific one: We (at > iXsystems) were attempting to engage with some of the Red Hat folks back > when the FreeBSD port was first done, in the hope of getting it more > =E2=80=9Cofficially supported=E2=80=9D for FreeBSD and perhaps even donat= ing some more > serious stress-testing and integration work for it, but when those Red Ha= t > folks moved on we lost continuity and the effort stalled. Who at Red Hat > would / could we work with in getting this back on track? We=E2=80=99d l= ike to > integrate glusterfs with FreeNAS 10, and in fact have already done so but > it=E2=80=99s still early days and we=E2=80=99re not even really sure what= we have yet. >=20 Just fyi..sofar, working with FreeBSD11/head and the port of 3.7.6 (the por= t tarball is in FreeBSD PR#194409), the only GlusterFS problem I've encountered is the above one. I'm not sure why this isn't in /usr/ports, but that would be nice as it might get more people trying it. (I'm a src comitter, but not a ports one.) However, I have several patches for the FreeBSD fuse interface and for a mount_glusterfs mount to work ok you need a couple of them. 1 - When an open decides to do DIRECT_IO after the file has done buffer cache I/O the buffer cache needs to be invalidated so you don't get stale cached data. 2 - For a WRONLY write, you need to force DIRECT_IO (or do a read/write ope= n). If you don't do this, the buffer cache code will get stuck when trying to read a block in before writing a partial block. (I think this is what FreeBSD PR#194293 is caused by.) Because I won't be able to do svn until April, these patches won't make it into head for a while, but they will both be in PR#194293 within hours. The others add features like extended attributes, advisory byte range locki= ng and the changes needed to export the fuse/glusterfs mount via the FreeBSD kernel nfsd. If anyone wants/needs these patches, email and I can send you them. A bit off your topic, but until you have the fixes for FreeBSD fuse, you probably can't do a lot of serious testing. (I don't know, but I'd guess that FreeNAS has about the same fuse module code as FreeBSD's head, since it hasn't been changed much in head recently= .) Thanks everyone for your help with this, rick > Thanks, >=20 > - Jordan >=20 >=20 From owner-freebsd-fs@freebsd.org Thu Dec 31 01:07:49 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D3C93A5639A for ; Thu, 31 Dec 2015 01:07:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BB30E1E8C for ; Thu, 31 Dec 2015 01:07:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id tBV17mxM030212 for ; Thu, 31 Dec 2015 01:07:49 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 194293] FUSE program freezes when seeking pos > file size Date: Thu, 31 Dec 2015 01:07:48 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: rmacklem@uoguelph.ca X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: attachments.isobsolete attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Dec 2015 01:07:49 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D194293 rmacklem@uoguelph.ca changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #164744|0 |1 is obsolete| | --- Comment #9 from rmacklem@uoguelph.ca --- Created attachment 164881 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D164881&action= =3Dedit invalidate buffer cache blocks when switching to DIRECT_IO This patch needs to be used along with 164831 so that stale data in the buffer cache isn't accessed after switching to doing direct I/O. I marked 164744 obsolete since I believe 164831 plus this patch is the preferred fix. Also, 164744 wasn't complete and wouldn't have compiled. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Dec 31 07:24:08 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 26816A563B7 for ; Thu, 31 Dec 2015 07:24:08 +0000 (UTC) (envelope-from chris@stankevitz.com) Received: from mango.stankevitz.com (mango.stankevitz.com [208.79.93.194]) by mx1.freebsd.org (Postfix) with ESMTP id 19D221EBC for ; Thu, 31 Dec 2015 07:24:07 +0000 (UTC) (envelope-from chris@stankevitz.com) Received: from Chriss-MacBook-Pro.local (209-203-101-124.static.twtelecom.net [209.203.101.124]) (using TLSv1.2 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mango.stankevitz.com (Postfix) with ESMTPSA id 2BC69B1 for ; Wed, 30 Dec 2015 23:24:01 -0800 (PST) From: Chris Stankevitz Subject: Monitoring FS changes To: FreeBSD Filesystems Message-ID: <5684D810.6070700@stankevitz.com> Date: Wed, 30 Dec 2015 23:24:00 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Dec 2015 07:24:08 -0000 Hi, I have a directory /foo that recursively contain ~250,000 files/directories. I would like my application to know when a file is added, removed, or modified under /foo. Is there a way to do that with FreeBSD? I believe on linux a facility called iNotify accomplishes this. On OSX a facility called FSEvents accomplishes this. kqueue apparently requires me to open every file and/or directory in my tree... which won't work because I have so many. Is there any other option? Perhaps i=0 while (true) { zfs snapshot pool/foo@${i} zfs diff pool/foo@${i-1} pool/foo@${i} ++i } Thank you, Chris From owner-freebsd-fs@freebsd.org Thu Dec 31 11:37:46 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CF82DA57B78 for ; Thu, 31 Dec 2015 11:37:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C0F5013A0 for ; Thu, 31 Dec 2015 11:37:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id tBVBbkNS049915 for ; Thu, 31 Dec 2015 11:37:46 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 194293] FUSE program freezes when seeking pos > file size Date: Thu, 31 Dec 2015 11:37:47 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: kib@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Dec 2015 11:37:46 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D194293 Konstantin Belousov changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kib@FreeBSD.org --- Comment #10 from Konstantin Belousov --- (In reply to rmacklem from comment #9) You might want to compare fuse_io_invalbuf() and ffs_rawread_sync(). From = the very quick inspection, they are mostly equivalent WRT write ops done. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Jan 1 14:44:02 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 19A4DA5EEFD for ; Fri, 1 Jan 2016 14:44:02 +0000 (UTC) (envelope-from artem@artem.ru) Received: from smtp50.i.mail.ru (smtp50.i.mail.ru [94.100.177.110]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7A448161B for ; Fri, 1 Jan 2016 14:44:01 +0000 (UTC) (envelope-from artem@artem.ru) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mail.ru; s=mail2; h=Content-Transfer-Encoding:Content-Type:MIME-Version:Date:Message-ID:Subject:From:To; bh=wKcWSys1NrgzHOqFcyaJ2xQqki5GXqXggPgEGjD9q1A=; b=nz+KTIX6q+ua0r8lyYAMsp2AKagK7EPF8P/Z9saFKSP03cUlMAv8v4P9Lu7n6sqKsvB/vzOy/B7e7g0YZBdcJIkMz4Va6TrqpcFCR7bgqAzR6rf2ykQNbrqpcUJTPUu8HQZt5c0hogPt2mBF+QCFvPexIjewaOaUm+B/UtYEf10=; Received: from 79-172-110-200.dyn.broadband.iskratelecom.ru ([79.172.110.200]:61909 helo=[192.168.0.160]) by smtp50.i.mail.ru with esmtpa (envelope-from ) id 1aF0vo-0007WM-0O for freebsd-fs@freebsd.org; Fri, 01 Jan 2016 17:43:52 +0300 To: freebsd-fs@freebsd.org From: Artem Kuchin Subject: fsync: giving up on dirty Message-ID: <568690A1.3040904@artem.ru> Date: Fri, 1 Jan 2016 17:43:45 +0300 User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Mras: Ok X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Jan 2016 14:44:02 -0000 Just upgraded from 10.1 to 10.2 via sources. After reboot received this: Jan 1 16:37:41 omni kernel: fsync: giving up on dirty Jan 1 16:37:41 omni kernel: 0xfffff8000f22cb10: tag devfs, type VCHR Jan 1 16:37:41 omni kernel: usecount 1, writecount 0, refcount 77 mountedhere 0xfffff8000f0b2e00 Jan 1 16:37:41 omni kernel: flags (VI_ACTIVE) Jan 1 16:37:41 omni kernel: v_object 0xfffff8000f244200 ref 0 pages 587 cleanbuf 74 dirtybuf 1 Jan 1 16:37:41 omni kernel: lock type devfs: EXCL by thread 0xfffff8000cfae4a0 (pid 22, syncer, tid 100089) Jan 1 16:37:41 omni kernel: dev mirror/root Tried to work. Got more same messages. / is gmirror # gmirror status Name Status Components mirror/boot COMPLETE ada0p1 (ACTIVE) ada1p1 (ACTIVE) mirror/swap COMPLETE ada0p2 (ACTIVE) ada1p2 (ACTIVE) mirror/root COMPLETE ada0p3 (ACTIVE) ada1p3 (ACTIVE) # tunefs -p / tunefs: POSIX.1e ACLs: (-a) disabled tunefs: NFSv4 ACLs: (-N) disabled tunefs: MAC multilabel: (-l) disabled tunefs: soft updates: (-n) enabled tunefs: soft update journaling: (-j) enabled tunefs: gjournal: (-J) disabled tunefs: trim: (-t) disabled tunefs: maximum blocks per file in a cylinder group: (-e) 4096 tunefs: average file size: (-f) 16384 tunefs: average number of files in a directory: (-s) 64 tunefs: minimum percentage of free space: (-m) 8% tunefs: space to hold for metadata blocks: (-k) 6408 tunefs: optimization preference: (-o) time tunefs: volume label: (-L) root now disabled soft updates journaling and error went away. Maybe need to disable and reenable SU journaling? I google and see such error dates back to 2012. Apparently it still happens. Why? Artem From owner-freebsd-fs@freebsd.org Fri Jan 1 19:54:05 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D1D1BA5EC05 for ; Fri, 1 Jan 2016 19:54:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C2BDF1A3E for ; Fri, 1 Jan 2016 19:54:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u01Js5eo033012 for ; Fri, 1 Jan 2016 19:54:05 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 205708] Kernel panic possibly happening while adding/removing/listing ZFS snapshot Date: Fri, 01 Jan 2016 19:54:05 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: jhb@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Jan 2016 19:54:05 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D205708 John Baldwin changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org --- Comment #5 from John Baldwin --- Assign to fs@ where ZFS issues are normally assigned. --=20 You are receiving this mail because: You are the assignee for the bug.=