From owner-freebsd-questions@FreeBSD.ORG Mon Oct 25 20:57:14 2010 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 22A22106566C for ; Mon, 25 Oct 2010 20:57:14 +0000 (UTC) (envelope-from temporal@gmail.com) Received: from mail-ew0-f54.google.com (mail-ew0-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id A10A88FC13 for ; Mon, 25 Oct 2010 20:57:13 +0000 (UTC) Received: by ewy28 with SMTP id 28so1898336ewy.13 for ; Mon, 25 Oct 2010 13:57:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type; bh=rAlm19+ZnWmYH42QgEdECCVp+RjgrByIY6OP0k1hxU0=; b=jNn66kTRA/+37ZeTziiw7MaaOm8ahVtDn+IgsWRNhsZF+Ji+NAUc+8grza0MZl0N5I WAvRjptTa0VkbqjrofoqDoyJDrH5ateJuzUY8v1BPBGCuwnLicmCoME/37YV9Lpcbqpo hzNqw7EkOLCAg10aShIU9ra/1os67chqAEpj8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=Orch/1S8ZibEgi+C4899yuOaPnyy+1mkkFcrAm9R8ylLgpkz8tHpZMOxb/zOMhfwt9 +UICg+qdqSakS2ocAtwxzCsEsvigFRebEsxFRtLZRPvIRhVkEkeJ2PlpGf/ZvFVr6V7w mUtzP6M0JcLdNhrGv9yiBc4u0zlTxJEPou6EA= MIME-Version: 1.0 Received: by 10.14.37.10 with SMTP id x10mr5423040eea.31.1288040232511; Mon, 25 Oct 2010 13:57:12 -0700 (PDT) Received: by 10.14.119.141 with HTTP; Mon, 25 Oct 2010 13:57:12 -0700 (PDT) In-Reply-To: References: <201010250446.o9P4kcid004004@mail.r-bonomi.com> <20101025074234.GA71353@owl.midgard.homeip.net> Date: Mon, 25 Oct 2010 13:57:12 -0700 Message-ID: From: Kenton Varda To: "Igor V. Ruzanov" Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-questions@freebsd.org Subject: Re: EVFILT_VNODE doesn't scale to large directory trees? X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Oct 2010 20:57:14 -0000 On Mon, Oct 25, 2010 at 1:54 AM, Igor V. Ruzanov wrote: > I thought so too but was not sure about his problem. With > kqueue()/kevent() we can monitor every open file descriptor. So how many > files are laid down in the directory tree, 100, 1000, 15000? In every > such way its possible to write or find already written daemon that > monitors every file in selected folder doing several jobs in separate > threads. There is might be little problem with receiving of *lots* events, > so we could create several pipes (for example) for setting up of > communication between deamon and system. > Note that we must remember to set proper meaning of kern.maxfiles sysctl > variable. > I worry that simply increasing the FD limits to meet my needs would have some negative effects, otherwise the limits would be much higher in the first place. How much kernel memory does each open FD consume? Probably most of that is wasted space, since I'm opening these FDs for no other purpose than to pass them to kqueue -- I never read or write them. But it sounds like you're saying that there is no alternative (other than polling, which would obviously be a lot worse), so I guess I'll live with it. Well, one other idea: Is there a way to simply monitor *all* I/O by all processes owned by the current user? I could then filter the events down to the directory I'm interested in. Not the ideal solution, but it would scale to a source tree of infinite size (since the machine can only be accessing a finite number of those files at once). It seems likely that this has been implemented somewhere due to the obvious system monitoring applications, but I'm not quite sure where to start looking.