Date: Tue, 24 Nov 2015 14:18:02 +0000 From: krad <kraduk@gmail.com> To: Albert Cervin <albert@acervin.com> Cc: freebsd-stable <freebsd-stable@freebsd.org> Subject: Re: ZFS - poor performance with "large" directories Message-ID: <CALfReyebVHCPbmxQeUy4PHa_Gc3eNa8euYPE5ekTs7_cNGyj8g@mail.gmail.com> In-Reply-To: <CAMMK2LCB5ocjyufZCMXQScQDgVxvSHOr0vmef6LKzPO35w3TQg@mail.gmail.com> References: <CAMMK2LCB5ocjyufZCMXQScQDgVxvSHOr0vmef6LKzPO35w3TQg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
make sure atime if off for starters on the filesystem On 24 November 2015 at 14:00, Albert Cervin <albert@acervin.com> wrote: > Hi all, > > Please feel free to direct me to a list that is more suitable. > > We are trying to set up a fileserver solution for a web application that we > are building. This fileserver is running FreeBSD 10.2 and ZFS. Files are > written over CIFS with Samba running on the fileserver host. > > However, we are seeing en exponential decrease in performance to write to > the file server when the number of files in the directory grows (when it > goes up to ~6000 files it becomes unusable and the write time has gone from > a fraction of a second to ten seconds). > > We ran the same setup on a Linux machine with an ext4 file system which did > NOT suffer from this performance degradation. > > Our first reaction was to remove Samba from the equation. I ran a test > where i tried to copy a folder with a large amount of files and then ran a > test with the same folder as a zip. > > So, > > cp -r folder_with_lots_of_files copy_of_folder_with_lots_of_files > > gives an iostat output that looks like this for the zpool (zpool iostat > frosting 1): > > pool alloc free read write read write > ---------- ----- ----- ----- ----- ----- ----- > frosting 48.5G 299G 2 0 267K 8.56K > frosting 48.5G 299G 401 0 50.2M 0 > frosting 48.6G 299G 384 94 47.9M 7.79M > frosting 48.6G 299G 471 0 58.9M 0 > frosting 48.6G 299G 492 0 61.4M 0 > frosting 48.6G 299G 393 0 49.0M 0 > frosting 48.6G 299G 426 0 53.3M 0 > frosting 48.6G 299G 421 147 52.5M 9.71M > frosting 48.6G 299G 507 0 63.4M 0 > frosting 48.6G 299G 376 0 47.0M 0 > frosting 48.6G 299G 447 0 55.8M 0 > frosting 48.6G 299G 433 13 54.2M 1.62M > frosting 48.6G 299G 431 85 53.8M 6.95M > frosting 48.6G 299G 288 0 36.1M 0 > frosting 48.6G 299G 329 0 41.2M 0 > frosting 48.6G 299G 340 0 42.4M 0 > frosting 48.6G 299G 398 9 49.8M 1.14M > frosting 48.6G 299G 324 126 40.4M 7.08M > frosting 48.6G 299G 391 0 48.9M 0 > frosting 48.6G 299G 261 0 32.5M 0 > frosting 48.6G 299G 314 0 39.3M 0 > frosting 48.6G 299G 317 0 39.6M 0 > frosting 48.6G 299G 346 79 43.3M 6.36M > > Are these "holes" in write speed normal. Since this is the exact symptom we > are getting when the network writes start to be slow. > > If I instead copy a large single file, I get this IO behavior: > > capacity operations bandwidth > pool alloc free read write read write > ---------- ----- ----- ----- ----- ----- ----- > frosting 50.1G 298G 7 0 953K 34.5K > frosting 50.1G 298G 224 215 27.9M 26.8M > frosting 50.2G 298G 224 364 27.8M 38.6M > frosting 50.2G 298G 225 57 27.9M 7.23M > frosting 50.3G 298G 173 477 21.5M 56.1M > frosting 50.3G 298G 219 0 27.3M 0 > frosting 50.3G 298G 265 353 33.0M 44.0M > frosting 50.3G 298G 294 172 36.6M 18.3M > frosting 50.3G 298G 237 436 29.4M 54.2M > frosting 50.4G 298G 257 108 31.9M 9.69M > frosting 50.4G 298G 211 382 26.1M 47.5M > frosting 50.4G 298G 305 162 38.0M 16.4M > frosting 50.4G 298G 253 369 31.5M 45.9M > frosting 50.5G 297G 176 177 21.8M 18.0M > frosting 50.5G 297G 197 167 24.6M 20.9M > frosting 50.6G 297G 248 375 30.9M 42.8M > frosting 50.6G 297G 322 605 39.9M 68.0M > frosting 50.6G 297G 164 36 20.4M 1.57M > frosting 50.6G 297G 259 96 32.2M 12.0M > > which looks more like what I would expect and is also similiar to the IO > behavior we get if I copy the folder with many files on an ext4 file > system. > > Any help or tips for getting this to work would be highly appreciated! > > Cheers, > Albert Cervin > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CALfReyebVHCPbmxQeUy4PHa_Gc3eNa8euYPE5ekTs7_cNGyj8g>