From owner-freebsd-questions@freebsd.org Tue Jun 25 18:35:47 2019 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A0A9315D39BB for ; Tue, 25 Jun 2019 18:35:47 +0000 (UTC) (envelope-from mike@sentex.net) Received: from pyroxene.sentex.ca (unknown [IPv6:2607:f3e0:0:3::18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "pyroxene.sentex.ca", Issuer "Let's Encrypt Authority X3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D60196FCCB for ; Tue, 25 Jun 2019 18:35:46 +0000 (UTC) (envelope-from mike@sentex.net) Received: from [192.168.43.29] ([192.168.43.29]) by pyroxene.sentex.ca (8.15.2/8.15.2) with ESMTPS id x5PIZda0063828 (version=TLSv1.2 cipher=AES128-SHA bits=128 verify=NO) for ; Tue, 25 Jun 2019 14:35:39 -0400 (EDT) (envelope-from mike@sentex.net) Subject: Re: ZFS Optimizing for large directories and MANY files From: mike tancsa To: freebsd-questions@freebsd.org References: Message-ID: <37d87f73-0e67-f87d-90f1-1f098d91876d@sentex.net> Date: Tue, 25 Jun 2019 14:35:40 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.7.2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: D60196FCCB X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:3::18 as permitted sender) smtp.mailfrom=mike@sentex.net X-Spamd-Result: default: False [-1.49 / 15.00]; ARC_NA(0.00)[]; RDNS_NONE(1.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-0.998,0]; DMARC_NA(0.00)[sentex.net]; MX_GOOD(-0.01)[cached: smtp.sentex.ca]; NEURAL_HAM_SHORT(-0.96)[-0.959,0]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; IP_SCORE(-1.72)[ipnet: 2607:f3e0::/32(-4.95), asn: 11647(-3.59), country: CA(-0.09)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; HFILTER_HOSTNAME_UNKNOWN(2.50)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Jun 2019 18:35:47 -0000 On 6/25/2019 2:19 PM, mike tancsa wrote: > /|# time find . -type f -mtime -2d|/ > > /|takes 40 min after a cold boot.|/ > > /|Watching zfs disk IO, its super slow in terms of bandwidth, but gstat > shows the disks close to being pegged.  I guess the heads are thrashing > about inefficiently ? Just for comparison, I tried the same test on the new box that will be in production. With SSDs (instead of spinning metal) the same find on a cold boot takes 1.318u 55.396s 1:21.96 69.1%    35+176k 865535+0io 0pf+0w and the second time through, just 1.102u 48.666s 0:49.96 99.5%    35+177k 169189+0io 0pf+0w     ---Mike