From owner-freebsd-fs@freebsd.org Mon Dec 25 17:25:19 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6E219E9A72F for ; Mon, 25 Dec 2017 17:25:19 +0000 (UTC) (envelope-from mi+thun@aldan.algebra.com) Received: from symbion.zaytman.com (inet-64-112-176-10.bos.netblazr.com [64.112.176.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "symbion", Issuer "Narawntapu" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 2E8D46BC42 for ; Mon, 25 Dec 2017 17:25:18 +0000 (UTC) (envelope-from mi+thun@aldan.algebra.com) Received: from narawntapu.narawntapu (pool-108-53-87-28.nwrknj.fios.verizon.net [108.53.87.28]) by symbion.zaytman.com (8.15.2/8.15.2) with ESMTPS id vBPHPBLU003352 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Mon, 25 Dec 2017 12:25:11 -0500 (EST) (envelope-from mi+thun@aldan.algebra.com) X-Authentication-Warning: symbion.zaytman.com: Host pool-108-53-87-28.nwrknj.fios.verizon.net [108.53.87.28] claimed to be narawntapu.narawntapu Received: from aldan.narawntapu (aldan [192.168.1.10]) by narawntapu.narawntapu (8.15.2/8.15.2) with ESMTP id vBPHP96m059829 for ; Mon, 25 Dec 2017 12:25:09 -0500 (EST) (envelope-from mi+thun@aldan.algebra.com) X-Authentication-Warning: narawntapu.narawntapu: Host aldan [192.168.1.10] claimed to be aldan.narawntapu To: freebsd-fs From: "Mikhail T." Subject: kqueue/kevent - watching an entire filesystem? Message-ID: <9795fa64-b2dc-50c3-c7e0-8422e0388c15@aldan.algebra.com> Date: Mon, 25 Dec 2017 12:25:09 -0500 MIME-Version: 1.0 Content-Language: en-US X-DCC-dmv.com-Metrics: narawntapu 1095; Body=1 Fuz1=1 Fuz2=1 X-Spam-Status: No, score=-2.9 required=7.5 tests=ALL_TRUSTED,BAYES_00, HTML_MESSAGE autolearn=ham autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on narawntapu.narawntapu Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Dec 2017 17:25:19 -0000 It looks like an watching an arbitrary sub-tree with kqueue remains out of reach on FreeBSD -- one needs to traverse the entire (sub)tree first and create a separate kevent for each subdirectory. But, maybe, watching an entire filesystem for changes can be accomplished easier? If not just any filesystem -- a ZFS one? Thanks! -mi From owner-freebsd-fs@freebsd.org Mon Dec 25 17:41:25 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ED087E9B4CA for ; Mon, 25 Dec 2017 17:41:25 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: from mail-io0-f179.google.com (mail-io0-f179.google.com [209.85.223.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BAD886C39E for ; Mon, 25 Dec 2017 17:41:25 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: by mail-io0-f179.google.com with SMTP id v186so30855946iod.7 for ; Mon, 25 Dec 2017 09:41:25 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:reply-to:in-reply-to:references :from:date:message-id:subject:to:cc; bh=AUr5nH3+3Wly3mHZCWLcev8WCEiz9KG66/EoOq8Ruko=; b=DD6BQxfgZIWdE9VKi5spwdjM5saRoETNPNGcrlls+6Np/Dsj+MOT/CaOmhOpyOdvO8 C132Pc9j9AxWuwGTgX1Dy6yKu1xHwOQmXBW2jp7NHpH0zOOoWFpjwMds/kwMhAwwqY+X nJFY8xrzl0NOU5Y7J96UIgfBIDwQqfn8/SPHn7Gjv9rwxWbKW24IUjShRK9vlXfE75Cg 6w1DiTVorOG7A1tVTw2DHayWfDoZvKEx1ULht92kv/ufmdmmhxH+94x19pvTWsHrm0cZ F2uUs21B0Cc5BK2XXTCHooUjP2MEIZJGZLeDgDhx2SzbjAzlTT7HMb06vBzG1vCqYmb+ YosA== X-Gm-Message-State: AKGB3mLcqCxIr9Hd5sQPt0Hghvzd1NK6KBE2vVwEtWNyJ8md3jVqlYw9 BZDoF/j/qu03ymW9QG/6R12kqO3e X-Google-Smtp-Source: ACJfBouy+NyWzuZR7dAnbyTanRbtogatoJp6xdxPx8nMZ63pSj5XYMG6QQws/FfUka7dk4pZ2nwG2A== X-Received: by 10.107.137.146 with SMTP id t18mr17192199ioi.258.1514223679136; Mon, 25 Dec 2017 09:41:19 -0800 (PST) Received: from mail-io0-f179.google.com (mail-io0-f179.google.com. [209.85.223.179]) by smtp.gmail.com with ESMTPSA id i63sm8646255itb.35.2017.12.25.09.41.19 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 25 Dec 2017 09:41:19 -0800 (PST) Received: by mail-io0-f179.google.com with SMTP id t196so30838574iof.0 for ; Mon, 25 Dec 2017 09:41:19 -0800 (PST) X-Received: by 10.107.181.147 with SMTP id e141mr29521112iof.117.1514223678836; Mon, 25 Dec 2017 09:41:18 -0800 (PST) MIME-Version: 1.0 Reply-To: cem@freebsd.org Received: by 10.2.15.193 with HTTP; Mon, 25 Dec 2017 09:41:18 -0800 (PST) In-Reply-To: <9795fa64-b2dc-50c3-c7e0-8422e0388c15@aldan.algebra.com> References: <9795fa64-b2dc-50c3-c7e0-8422e0388c15@aldan.algebra.com> From: Conrad Meyer Date: Mon, 25 Dec 2017 09:41:18 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: kqueue/kevent - watching an entire filesystem? To: "Mikhail T." Cc: freebsd-fs Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Dec 2017 17:41:26 -0000 If you want to watch whole system activity (and not just a single filesystem), that sounds like audit. Or its crappy cousin, filemon(4). Of course, neither of these options has a kevent-style interface, and filemon(4) in particular may miss relevant events. I'm afraid there is nothing better than recursively opening subdirectories to monitor a tree (even a whole filesystem) with kqueue/kevent. Best, Conrad On Mon, Dec 25, 2017 at 9:25 AM, Mikhail T. wrote: > It looks like an watching an arbitrary sub-tree with kqueue remains out of > reach on FreeBSD -- one needs to traverse the entire (sub)tree first and > create a separate kevent for each subdirectory. > > But, maybe, watching an entire filesystem for changes can be accomplished > easier? If not just any filesystem -- a ZFS one? > > Thanks! > > -mi > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Mon Dec 25 19:07:29 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E04A6EA09F1 for ; Mon, 25 Dec 2017 19:07:29 +0000 (UTC) (envelope-from mi+thun@aldan.algebra.com) Received: from symbion.zaytman.com (inet-64-112-176-10.bos.netblazr.com [64.112.176.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "symbion", Issuer "Narawntapu" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 862E26F23B; Mon, 25 Dec 2017 19:07:28 +0000 (UTC) (envelope-from mi+thun@aldan.algebra.com) Received: from narawntapu.narawntapu (pool-108-53-87-28.nwrknj.fios.verizon.net [108.53.87.28]) by symbion.zaytman.com (8.15.2/8.15.2) with ESMTPS id vBPJ7RHb003577 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 25 Dec 2017 14:07:27 -0500 (EST) (envelope-from mi+thun@aldan.algebra.com) X-Authentication-Warning: symbion.zaytman.com: Host pool-108-53-87-28.nwrknj.fios.verizon.net [108.53.87.28] claimed to be narawntapu.narawntapu Received: from aldan.narawntapu (aldan [192.168.1.10]) by narawntapu.narawntapu (8.15.2/8.15.2) with ESMTP id vBPJ7LDY064491; Mon, 25 Dec 2017 14:07:21 -0500 (EST) (envelope-from mi+thun@aldan.algebra.com) X-Authentication-Warning: narawntapu.narawntapu: Host aldan [192.168.1.10] claimed to be aldan.narawntapu Subject: Re: kqueue/kevent - watching an entire filesystem? To: freebsd-fs References: <9795fa64-b2dc-50c3-c7e0-8422e0388c15@aldan.algebra.com> From: "Mikhail T." Message-ID: Date: Mon, 25 Dec 2017 14:07:21 -0500 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US X-DCC-Etherboy-Metrics: narawntapu 1002; Body=2 Fuz1=2 Fuz2=2 X-Spam-Status: No, score=-2.9 required=7.5 tests=ALL_TRUSTED,BAYES_00, HTML_MESSAGE autolearn=ham autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on narawntapu.narawntapu Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Dec 2017 19:07:30 -0000 On 25.12.2017 12:41, Conrad Meyer wrote: > I'm afraid there is nothing better than recursively opening > subdirectories to monitor a tree (even a whole filesystem) with > kqueue/kevent. I wonder, how difficult it would be to implement the kevent-filter neccesary for this instead. Can it be just a module, or would it have to modify other places in the kernel? -mi From owner-freebsd-fs@freebsd.org Fri Dec 29 05:29:00 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4159BEAAF4F; Fri, 29 Dec 2017 05:29:00 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 169BA7CBB6; Fri, 29 Dec 2017 05:28:53 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (115-166-0-128.dyn.iinet.net.au [115.166.0.128]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id vBT5Sl0c007101 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Thu, 28 Dec 2017 21:28:50 -0800 (PST) (envelope-from julian@freebsd.org) Subject: Re: add BIO_NORETRY flag, implement support in ata_da, use in ZFS vdev_geom To: Andriy Gapon , freebsd-fs@FreeBSD.org, freebsd-geom@FreeBSD.org References: <391f2cc7-0036-06ec-b6c9-e56681114eeb@FreeBSD.org> From: Julian Elischer Message-ID: <9512ddbe-f9a1-caa7-10ac-ef87870bf48b@freebsd.org> Date: Fri, 29 Dec 2017 13:28:41 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0) Gecko/20100101 Thunderbird/52.5.0 MIME-Version: 1.0 In-Reply-To: <391f2cc7-0036-06ec-b6c9-e56681114eeb@FreeBSD.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Dec 2017 05:29:00 -0000 On 24/11/17 6:30 pm, Andriy Gapon wrote: > https://reviews.freebsd.org/D13224 > > Anyone interested is welcome to join the review. > Thanks. I see you closed the review. However don't stop working with that problem.. It's still something that a worth a solution, even if not exactly That solution. Maybe a generic idea through the block IO layer of transaction requirements. We did face this issue in a BSD4.3 variant in 1990 where a single disk error would result in a cascading shockwave of slowdowns that took some time to resolve. The issue even comes up in networking too, especially with realtime video where "don't bother, we missed the window" comes into play.