From owner-freebsd-stable@freebsd.org Thu Oct 20 23:56:08 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9B74BC16A6C for ; Thu, 20 Oct 2016 23:56:08 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-lf0-x22e.google.com (mail-lf0-x22e.google.com [IPv6:2a00:1450:4010:c07::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0DF4E92D for ; Thu, 20 Oct 2016 23:56:07 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-lf0-x22e.google.com with SMTP id b81so107419570lfe.1 for ; Thu, 20 Oct 2016 16:56:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to; bh=cLZAEXiCh4PJ93dQ1ew2ZptO+0Ygx2Kl2///fFTCkbk=; b=zQhUq943NeeRt8mchcP4bwQGmrdnFtiRuMjbE31UJmaS1mGa3H+Pgpf5ovgylW3UCy 679SB3h6UWQcSirwD9UmomGjbdKh+hJu4kDWzo53r6SOjOGSwIoKPFKacXwGgLsS5Ut5 /Hrli3sjQRNLrAKfTGqHeLlKK4WfC+29brJcqPOvR+c/bhUUvsVMAmkBWGmMLCI2u68o Yn7Qo4ymT1l5qJ0B6pb/U6BTxmhlbAb0tjcYAwXh1cG0idP/x8v+ecTHTIJcV8H+uVLy zRqYmhRQs/mMwo22dn9oAI/K/PGICkfPp5hjiDK7n55nrL9I9jSi7u7R8V8ImL0A7RPX lpKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to; bh=cLZAEXiCh4PJ93dQ1ew2ZptO+0Ygx2Kl2///fFTCkbk=; b=Z+X1ZENNtUtrMT02CTIbRY8D+FY9Lu4vY7jrFuBhqY5N5RIgzO5wm2b7tpPgACoaqw ATRwfxo2p1rcZ1AGTM9qBkLGv6JjcR1R0B5S2UJ6VziUWzmvrDbR7MQ5/NaSNq24wyr8 nj1NloJ2F/7ZwPm73AISWzz4ouflxcnqzVs2I0VKxZNwdxEZujG1nKo95UPZxE8ZdPsN qSGlcUvQ0e/Tf68CJld/et8tfsudVQET8K9tXByr7W/dWkXD/CmVFj+ovCjatnGUO3MY uOUBsc+6dOfRlkO4AeVPuFmDAYXgrmpwA3C6ubNQiLoBzlluQrW6YmHZOoqI26E3ZkZE SBiw== X-Gm-Message-State: AA6/9RkwgttcGUINNL7TCUJUrmTFEnae8TsQOdYzNMV6pygcAu7stfEWSKR9BTBQRrLGurnK X-Received: by 10.28.60.7 with SMTP id j7mr4858189wma.7.1477007764601; Thu, 20 Oct 2016 16:56:04 -0700 (PDT) Received: from [10.10.1.58] ([185.97.61.26]) by smtp.gmail.com with ESMTPSA id au8sm81900802wjc.12.2016.10.20.16.56.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 20 Oct 2016 16:56:03 -0700 (PDT) Subject: Re: zfs, a directory that used to hold lot of files and listing pause To: Jonathan Chen References: <4d9269af-ed64-bb73-eb7f-98a3f5ffd5a2@norma.perm.ru> <92607b3b-a2d1-2391-5bd6-9781f426d7a6@multiplay.co.uk> Cc: freebsd-stable@freebsd.org From: Steven Hartland Message-ID: <1b187ab5-cf67-bf66-ab5f-8ec5ae29c140@multiplay.co.uk> Date: Fri, 21 Oct 2016 00:56:33 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Oct 2016 23:56:08 -0000 On 20/10/2016 23:48, Jonathan Chen wrote: > On 21 October 2016 at 11:27, Steven Hartland wrote: >> On 20/10/2016 22:18, Jonathan Chen wrote: >>> On 21 October 2016 at 09:09, Peter wrote: >>> [...] >>>> I see this on my pgsql_tmp dirs (where Postgres stores intermediate >>>> query data that gets too big for mem - usually lots of files) - in >>>> normal operation these dirs are completely empty, but make heavy disk >>>> activity (even writing!) when doing ls. >>>> Seems normal, I dont care as long as the thing is stable. One would need >>>> to check how ZFS stores directories and what kind of fragmentation can >>>> happen there. Or wait for some future feature that would do >>>> housekeeping. ;) >>> I'm seeing this as well with an Odoo ERP running on Postgresql. This >>> lag does matter to me as this is huge performance hit when running >>> Postgresql on ZFS, and it would be good to see this resolved. >>> pg_restores can make the system crawl as well. >> As mentioned before could you confirm you have disable atime? > Yup, also set the blocksize to 4K. > > 11:46am# zfs get all irontree/postgresql > NAME PROPERTY VALUE SOURCE > irontree/postgresql type filesystem - > irontree/postgresql creation Wed Sep 23 15:07 2015 - > irontree/postgresql used 43.8G - > irontree/postgresql available 592G - > irontree/postgresql referenced 43.8G - > irontree/postgresql compressratio 1.00x - > irontree/postgresql mounted yes - > irontree/postgresql quota none default > irontree/postgresql reservation none default > irontree/postgresql recordsize 8K local > irontree/postgresql mountpoint /postgresql > inherited from irontree > irontree/postgresql sharenfs off default > irontree/postgresql checksum on default > irontree/postgresql compression off default > irontree/postgresql atime off local > irontree/postgresql devices on default > irontree/postgresql exec on default > irontree/postgresql setuid on default > irontree/postgresql readonly off default > irontree/postgresql jailed off default > irontree/postgresql snapdir hidden default > irontree/postgresql aclmode discard default > irontree/postgresql aclinherit restricted default > irontree/postgresql canmount on default > irontree/postgresql xattr off temporary > irontree/postgresql copies 1 default > irontree/postgresql version 5 - > irontree/postgresql utf8only off - > irontree/postgresql normalization none - > irontree/postgresql casesensitivity sensitive - > irontree/postgresql vscan off default > irontree/postgresql nbmand off default > irontree/postgresql sharesmb off default > irontree/postgresql refquota none default > irontree/postgresql refreservation none default > irontree/postgresql primarycache all default > irontree/postgresql secondarycache all default > irontree/postgresql usedbysnapshots 0 - > irontree/postgresql usedbydataset 43.8G - > irontree/postgresql usedbychildren 0 - > irontree/postgresql usedbyrefreservation 0 - > irontree/postgresql logbias latency default > irontree/postgresql dedup off default > irontree/postgresql mlslabel - > irontree/postgresql sync standard default > irontree/postgresql refcompressratio 1.00x - > irontree/postgresql written 43.8G - > irontree/postgresql logicalused 43.4G - > irontree/postgresql logicalreferenced 43.4G - > irontree/postgresql volmode default default > irontree/postgresql filesystem_limit none default > irontree/postgresql snapshot_limit none default > irontree/postgresql filesystem_count none default > irontree/postgresql snapshot_count none default > irontree/postgresql redundant_metadata all default > When you see the stalling what does gstat -pd and top -SHz show? Regards Steve