Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 13 Dec 2012 20:38:02 +0100
From:      =?ISO-8859-1?Q?Olav_Gr=F8n=E5s_Gjerde?= <olavgg@gmail.com>
To:        Bruce Evans <brde@optusnet.com.au>
Cc:        freebsd-fs@freebsd.org, freebsd-performance@freebsd.org
Subject:   Re: find vs ls performance for walking folders, are there any faster options?
Message-ID:  <CALL7tK_VEhSRwjy1WHg3Yo%2B5-e3sEUp3qSHNN_HdqM5_byw3Zg@mail.gmail.com>
In-Reply-To: <20121213012632.M1201@besplex.bde.org>
References:  <CALL7tK9uRz2qLDqHKOqKLrM7oPc4UvYj0s6zxh6VNFQLYoyPTQ@mail.gmail.com> <20121213012632.M1201@besplex.bde.org>

next in thread | previous in thread | raw e-mail | index | archive | help
Thank you, that was a really good answer.

On Wed, Dec 12, 2012 at 3:50 PM, Bruce Evans <brde@optusnet.com.au> wrote:
> On Wed, 12 Dec 2012, [ISO-8859-1] Olav Gr=F8n=E5s Gjerde wrote:
>
>> I'm working on scanning filesystems to build a file search engine and
>> came over something interesting.
>>
>> I can walk through 300 000 folders in ~19.5seconds with this command:
>> ls -Ra | grep -e "./.*:" | sed "s/://"
>>
>> With find, it surprisingly takes ~50.5 seconds.:
>> find . -type d
>
>
> This is because 'find' with '-type' lstats all the files.  It doesn't
> use DT_DIR from dirent for some reason.  ls can be slowed down similarly
> using -F.
>
>
>> My results are based on five runs of each command to warm up the disk
>> cache.
>> I've tried both this with both UFS and ZFS, and both filesystems shows
>> the same speed difference.
>
>
> I get almost exactly the same ratio of speeds on an old version of FreeBS=
D.
> All the data was cached, and there were only 7 symlinks.  Thr file system
> was mounted with -noatime, so the cache actually worked.
>
>
>> On a modern Linux distribution(Ubuntu 12.10 with EXT4), ls is just
>> slight faster than find(about 15-20%).
>
>
> Apparently lstat() is relatively much slower in FreeBSD.  It only takes
> 5 usec here, but that is a lot for converting cached data (getpid()
> takes 0.2 usec).  A file system mounted with -atime might be much
> slower, for writing directory timestamps (the sync of the timestamps
> is delayed, but it is a very heavyweight operation).
>
>
>> Are there a faster way to walk folders on FreeBSD? Are there some
>> options(sysctl) I could tune to improve the performance?
>
>
> Nothing much faster than find without -type.  Whatever fts(3) gives.
>
> Bruce



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CALL7tK_VEhSRwjy1WHg3Yo%2B5-e3sEUp3qSHNN_HdqM5_byw3Zg>