From owner-freebsd-performance@FreeBSD.ORG Mon Dec 17 23:13:51 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 18C8A16A417 for ; Mon, 17 Dec 2007 23:13:51 +0000 (UTC) (envelope-from cracauer@koef.zs64.net) Received: from koef.zs64.net (koef.zs64.net [212.12.50.230]) by mx1.freebsd.org (Postfix) with ESMTP id A049113C45A for ; Mon, 17 Dec 2007 23:13:50 +0000 (UTC) (envelope-from cracauer@koef.zs64.net) Received: from koef.zs64.net (koef.zs64.net [212.12.50.230]) by koef.zs64.net (8.14.1/8.14.1) with ESMTP id lBHMusZF098267; Mon, 17 Dec 2007 23:56:54 +0100 (CET) (envelope-from cracauer@koef.zs64.net) Received: (from cracauer@localhost) by koef.zs64.net (8.14.2/8.14.1/Submit) id lBHMusvt098266; Mon, 17 Dec 2007 17:56:54 -0500 (EST) (envelope-from cracauer) Date: Mon, 17 Dec 2007 17:56:54 -0500 From: Martin Cracauer To: Shantanu Ghosh Message-ID: <20071217225654.GA97600@cons.org> References: <963530.77689.qm@web54505.mail.re2.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <963530.77689.qm@web54505.mail.re2.yahoo.com> User-Agent: Mutt/1.4.2.3i Cc: freebsd-performance@freebsd.org Subject: Re: freebsd vs linux: performance problem X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Dec 2007 23:13:51 -0000 Shantanu Ghosh wrote on Thu, Dec 13, 2007 at 04:07:50AM -0800: > Hi, > > I am running FreeBSD 7.0 Beta1 and Linux FC6 on two identical pieces of > hardware - Dell poweredge with intel core2 duo. Each system has 4 CPUs. I assume that means 2 CPUs with two cores each, aka socket 771 woodcrests? Please be more specific. /proc/cpuinfo > Now, in simple memory access operations, I see the freebsd system being > noticably slower than the linux system. A simple C program that copies > from one memory buffer to another, when executed in a loop executes > between 10-30% slower on freebsd, as compared to linux. The assembly > code of the program used for testing is identical in both the cases. Please provide that simple C program. Below I assume that your assembly doesn't ever call memcpy() or similar. Please let us know which Linux kernel version, I gave up on FC and don't know the FC to kernel map. Anyway... This is most likely something I experienced myself: sometime between Linux 2.6.17 and 2.6.20 they were teaching the kernel about Core2 and about the shared cache in particular. Memory task performance such as piping around gzip output used to be horrible on Core2 systems that had some system cores sharing L2 cache and others don't, such as a dual Woodcrests system which has 4 cores total of which two and two share the L2 cache. A socket 775 system with just a Core2Duo (which means all cores in the system share the single L2 cache) used to be much better than the dual Woodcrest in 2.6.17 but in 2.6.20 it was fixed. I assume this is very simply a scheduler change that now knows which cores share L2 cache and sets affinity appropriately. On a loaded system with mixed random stuff doing on this is likely not a factor anymore (because the scheduler has too many other constraints to babysit one process), but benchmarking and single-tasking can expose it. > One observation is that freebsd system performance decreases as the > size of the buffer increases. If the buffer is under 1k, both the > sytems give the same performance. freebsd performance is about 10% > slower if the buffer size is around 4k, and about 30% slower if the > buffer is around 1Mb. A benchmark like sysbench memory read operation > performs miserably on the freebsd system, compared to linux. "buffer" here means you first read bytes, then write bytes elsewhere? How do you allocate the buffer to hold this data? Alignment plays a big role here. If you can, please give us the C program, otherwise I'd like you to print the address of the buffer in both cases. > As far as I can see, the BIOS settings are identical on both the > machines. Any idea what could be going on? Make double sure that the hardware readahead that some of the socket 771 chipsets is set in an identical manner. Also, the snoop filter in 5000x chipset suc^Hffers from underengineering and should be turned off for most applications. Also, please run the stream.c benchmark on both, including the Linux binary on FreeBSD using the Linuxulator as a third run. I put a copy on http://www.cons.org/stream.c Martin -- %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Martin Cracauer http://www.cons.org/cracauer/ FreeBSD - where you want to go, today. http://www.freebsd.org/ From owner-freebsd-performance@FreeBSD.ORG Tue Dec 18 03:15:10 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0B62316A418 for ; Tue, 18 Dec 2007 03:15:10 +0000 (UTC) (envelope-from shantanu_ghosh@yahoo.com) Received: from web54505.mail.re2.yahoo.com (web54505.mail.re2.yahoo.com [206.190.49.155]) by mx1.freebsd.org (Postfix) with SMTP id CE7DA13C461 for ; Tue, 18 Dec 2007 03:15:09 +0000 (UTC) (envelope-from shantanu_ghosh@yahoo.com) Received: (qmail 58541 invoked by uid 60001); 18 Dec 2007 03:15:09 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; b=msOq6zhEXjE99E71Bz8EnxTWUrckvarVm+R7Dto2hbv7XMuOPL/uX5aLoBwYkUY+3XbqqmYIlZAfGT+61bOdhsDkJ4Ewbbj7IEHn0EdEs162vAC5DIXQSqW1Spsivm0pdt8HB6FZs59kob9Cfa+8MkWPiIPExTSiBZQmS6IuNj4=; X-YMail-OSG: GjHoFDAVM1kDaFvJIc5zVKBdvpPyDQq7OQkdfGqbIHECfw.sBSAY4hBWX5QDtFsAeoO5lypn_KyJ22dYvRc5xE0t7T77Xt2EKUY5ZTPavw7.JerSF2Ap1g1.iNXLYQMYIHlAlIBNX6ZngvQ- Received: from [203.145.181.122] by web54505.mail.re2.yahoo.com via HTTP; Mon, 17 Dec 2007 19:15:08 PST Date: Mon, 17 Dec 2007 19:15:08 -0800 (PST) From: Shantanu Ghosh To: Martin Cracauer In-Reply-To: <20071217225654.GA97600@cons.org> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Message-ID: <979352.57503.qm@web54505.mail.re2.yahoo.com> Cc: freebsd-performance@freebsd.org Subject: Re: freebsd vs linux: performance problem X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Dec 2007 03:15:10 -0000 Hi, Thanks for your help. I am a bit tied up in something personal, I'll get back to this matter in a day or two, and give you the updates. Regards, Shantanu. --- Martin Cracauer wrote: > Shantanu Ghosh wrote on Thu, Dec 13, 2007 at 04:07:50AM -0800: > > Hi, > > > > I am running FreeBSD 7.0 Beta1 and Linux FC6 on two identical > pieces of > > hardware - Dell poweredge with intel core2 duo. Each system has 4 > CPUs. > > I assume that means 2 CPUs with two cores each, aka socket 771 > woodcrests? Please be more specific. /proc/cpuinfo > > > Now, in simple memory access operations, I see the freebsd system > being > > noticably slower than the linux system. A simple C program that > copies > > from one memory buffer to another, when executed in a loop executes > > between 10-30% slower on freebsd, as compared to linux. The > assembly > > code of the program used for testing is identical in both the > cases. > > Please provide that simple C program. Below I assume that your > assembly doesn't ever call memcpy() or similar. > > Please let us know which Linux kernel version, I gave up on FC and > don't know the FC to kernel map. > > Anyway... > > This is most likely something I experienced myself: sometime between > Linux 2.6.17 and 2.6.20 they were teaching the kernel about Core2 and > about the shared cache in particular. Memory task performance such > as > piping around gzip output used to be horrible on Core2 systems that > had some system cores sharing L2 cache and others don't, such as a > dual Woodcrests system which has 4 cores total of which two and two > share the L2 cache. A socket 775 system with just a Core2Duo (which > means all cores in the system share the single L2 cache) used to be > much better than the dual Woodcrest in 2.6.17 but in 2.6.20 it was > fixed. I assume this is very simply a scheduler change that now > knows > which cores share L2 cache and sets affinity appropriately. > > On a loaded system with mixed random stuff doing on this is likely > not > a factor anymore (because the scheduler has too many other > constraints > to babysit one process), but benchmarking and single-tasking can > expose it. > > > One observation is that freebsd system performance decreases as the > > size of the buffer increases. If the buffer is under 1k, both the > > sytems give the same performance. freebsd performance is about 10% > > slower if the buffer size is around 4k, and about 30% slower if the > > buffer is around 1Mb. A benchmark like sysbench memory read > operation > > performs miserably on the freebsd system, compared to linux. > > "buffer" here means you first read bytes, then write > bytes elsewhere? > > How do you allocate the buffer to hold this data? Alignment plays a > big role here. > > If you can, please give us the C program, otherwise I'd like you to > print the address of the buffer in both cases. > > > As far as I can see, the BIOS settings are identical on both the > > machines. Any idea what could be going on? > > Make double sure that the hardware readahead that some of the socket > 771 chipsets is set in an identical manner. Also, the snoop filter > in > 5000x chipset suc^Hffers from underengineering and should be turned > off for most applications. > > Also, please run the stream.c benchmark on both, including the Linux > binary on FreeBSD using the Linuxulator as a third run. I put a copy > on http://www.cons.org/stream.c > > Martin > -- > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > Martin Cracauer http://www.cons.org/cracauer/ > FreeBSD - where you want to go, today. http://www.freebsd.org/ > ____________________________________________________________________________________ Never miss a thing. Make Yahoo your home page. http://www.yahoo.com/r/hs From owner-freebsd-performance@FreeBSD.ORG Wed Dec 19 19:36:34 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3545916A420 for ; Wed, 19 Dec 2007 19:36:34 +0000 (UTC) (envelope-from biancalana@gmail.com) Received: from py-out-1112.google.com (py-out-1112.google.com [64.233.166.183]) by mx1.freebsd.org (Postfix) with ESMTP id DB2C113C44B for ; Wed, 19 Dec 2007 19:36:33 +0000 (UTC) (envelope-from biancalana@gmail.com) Received: by py-out-1112.google.com with SMTP id u77so357360pyb.3 for ; Wed, 19 Dec 2007 11:36:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; bh=7O7jp7VRA1SLeCkw3ysfgkWijLfi8NL/tMeAaG8BsSM=; b=q/CymL6Dyr2ICOtRr5jwB08z99Xs/te7B/BA66Y0ytTupdbGHRxIxnlfsTv5XAhguJkdRBDXm6eaaLpT4lHUgrnB9rDEXNF2kiVe7YJVHBrGhIoFOo0r4CXBZC6Ny0GmCRBLic+nGXK1Xqf20e+jbTfFzGcOykstUtgOl6J+Z10= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=Qbhsw7Zphy+AgKqSeAyecxfZItInv59Phxg21VAg2OAw/6kByFQOb+7sbMz3WdbuvBVx5kQmTpAFtPp3BuWROjt+pZ9lcDZcbVZGURxqsNf9vgAmifYn/TmBZv6OAqPwc9yPnmSjqyv8RtY0lfrwdau52hCQFmlX7XORYirs5S8= Received: by 10.65.156.2 with SMTP id i2mr21409840qbo.60.1198091388313; Wed, 19 Dec 2007 11:09:48 -0800 (PST) Received: by 10.64.184.9 with HTTP; Wed, 19 Dec 2007 11:09:48 -0800 (PST) Message-ID: <8e10486b0712191109n3d21b02cyf5183ee0cd01d8ce@mail.gmail.com> Date: Wed, 19 Dec 2007 17:09:48 -0200 From: "Alexandre Biancalana" To: freebsd-performance@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Subject: Bad performance when accessing a lot of small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Dec 2007 19:36:34 -0000 Hi List, I have a backup server running FreeBSD 7-BETA3. The cpu is CPU: Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz, 3GB Ram, 10x 500GB SATA, Areca 1231-ML, the filesystem used to backup my other servers locally is build on top of ARC-1231, 4TB (32k stripe) zfs filesystem with gzip compression. This machine receive backups from ~30 servers, (of all kinds and sizes, databases, fileservers, image servers, webservers, etc) all night, write the last day in LTO-3 tapes and store some days older days in disk. The behavior that I'm observing and that want your help is when the system is accessing some directory with many small files ( directories with ~ 1 million of ~30kb files), the performance is very poor. Here is the output of iostat -x 1 when the system is accessing (and writing tapes) with normal (no many small files) directories: extended device statistics device r/s w/s kr/s kw/s wait svc_t b ad0 0.0 0.0 0.0 0.0 0 0.0 0 da0 429.8 0.0 19774.1 0.0 6 12.1 81 sa0 0.0 3800.7 0.0 38007.3 1 0.2 88 sa1 0.0 2637.6 0.0 26375.8 1 0.3 92 pass0 0.0 0.0 0.0 0.0 0 0.0 0 pass1 0.0 0.0 0.0 0.0 0 0.0 0 pass2 0.0 0.0 0.0 0.0 0 0.0 0 pass3 0.0 0.0 0.0 0.0 0 0.0 0 pass4 0.0 0.0 0.0 0.0 0 0.0 0 ch0 0.0 0.0 0.0 0.0 0 0.0 0 extended device statistics device r/s w/s kr/s kw/s wait svc_t b ad0 0.0 0.0 0.0 0.0 0 0.0 0 da0 1196.0 0.0 58474.9 0.0 3 4.3 78 sa0 0.0 3021.1 0.0 30210.7 1 0.3 78 sa1 0.0 3029.8 0.0 30298.0 1 0.3 79 pass0 0.0 0.0 0.0 0.0 0 0.0 0 pass1 0.0 0.0 0.0 0.0 0 0.0 0 pass2 0.0 0.0 0.0 0.0 0 0.0 0 pass3 0.0 0.0 0.0 0.0 0 0.0 0 pass4 0.0 0.0 0.0 0.0 0 0.0 0 ch0 0.0 0.0 0.0 0.0 0 0.0 0 extended device statistics device r/s w/s kr/s kw/s wait svc_t b ad0 0.0 8.1 0.0 104.9 0 1.8 0 da0 1141.4 0.0 63241.1 0.0 0 2.5 39 sa0 0.0 1660.5 0.0 16605.5 0 0.3 42 sa1 0.0 1701.2 0.0 17012.5 1 0.2 42 pass0 0.0 0.0 0.0 0.0 0 0.0 0 pass1 0.0 0.0 0.0 0.0 0 0.0 0 pass2 0.0 0.0 0.0 0.0 0 0.0 0 pass3 0.0 0.0 0.0 0.0 0 0.0 0 pass4 0.0 0.0 0.0 0.0 0 0.0 0 ch0 0.0 0.0 0.0 0.0 0 0.0 0 extended device statistics device r/s w/s kr/s kw/s wait svc_t b ad0 0.0 0.0 0.0 0.0 0 0.0 0 da0 1400.8 140.1 77701.0 4185.3 0 2.0 44 sa0 0.0 2762.7 0.0 27626.7 1 0.3 72 sa1 0.0 2775.3 0.0 27753.1 1 0.3 72 pass0 0.0 0.0 0.0 0.0 0 0.0 0 pass1 0.0 0.0 0.0 0.0 0 0.0 0 pass2 0.0 0.0 0.0 0.0 0 0.0 0 pass3 0.0 0.0 0.0 0.0 0 0.0 0 pass4 0.0 0.0 0.0 0.0 0 0.0 0 ch0 0.0 0.0 0.0 0.0 0 0.0 0 Here is the output when accessing some directory with many small files: extended device statistics device r/s w/s kr/s kw/s wait svc_t b ad0 0.0 0.0 0.0 0.0 0 0.0 0 da0 738.6 0.0 38628.1 0.0 7 10.0 114 sa0 0.0 6001.2 0.0 60011.5 0 0.1 88 sa1 0.0 0.0 0.0 0.0 0 0.0 0 pass0 0.0 0.0 0.0 0.0 0 0.0 0 pass1 0.0 0.0 0.0 0.0 0 0.0 0 pass2 0.0 0.0 0.0 0.0 0 0.0 0 pass3 0.0 0.0 0.0 0.0 0 0.0 0 pass4 0.0 0.0 0.0 0.0 0 0.0 0 ch0 0.0 0.0 0.0 0.0 0 0.0 0 extended device statistics device r/s w/s kr/s kw/s wait svc_t b ad0 0.0 0.0 0.0 0.0 0 0.0 0 da0 539.9 75.6 26989.5 1808.7 15 9.2 78 sa0 0.0 4842.5 0.0 48425.0 0 0.1 72 sa1 0.0 0.0 0.0 0.0 0 0.0 0 pass0 0.0 0.0 0.0 0.0 0 0.0 0 pass1 0.0 0.0 0.0 0.0 0 0.0 0 pass2 0.0 0.0 0.0 0.0 0 0.0 0 pass3 0.0 0.0 0.0 0.0 0 0.0 0 pass4 0.0 0.0 0.0 0.0 0 0.0 0 ch0 0.0 0.0 0.0 0.0 0 0.0 0 extended device statistics device r/s w/s kr/s kw/s wait svc_t b ad0 0.0 0.0 0.0 0.0 0 0.0 0 da0 550.5 40.4 25794.9 321.5 6 12.0 99 sa0 0.0 5928.0 0.0 59280.1 1 0.1 89 sa1 0.0 0.0 0.0 0.0 0 0.0 0 pass0 0.0 0.0 0.0 0.0 0 0.0 0 pass1 0.0 0.0 0.0 0.0 0 0.0 0 pass2 0.0 0.0 0.0 0.0 0 0.0 0 pass3 0.0 0.0 0.0 0.0 0 0.0 0 pass4 0.0 0.0 0.0 0.0 0 0.0 0 ch0 0.0 0.0 0.0 0.0 0 0.0 0 extended device statistics device r/s w/s kr/s kw/s wait svc_t b ad0 0.0 0.0 0.0 0.0 0 0.0 0 da0 972.2 0.0 47698.7 0.0 8 7.6 93 sa0 0.0 3840.6 0.0 38406.2 1 0.2 63 sa1 0.0 2015.9 0.0 20158.9 1 0.2 36 pass0 0.0 0.0 0.0 0.0 0 0.0 0 pass1 0.0 0.0 0.0 0.0 0 0.0 0 pass2 0.0 0.0 0.0 0.0 0 0.0 0 pass3 0.0 0.0 0.0 0.0 0 0.0 0 pass4 0.0 0.0 0.0 0.0 0 0.0 0 ch0 0.0 0.0 0.0 0.0 0 0.0 0 extended device statistics device r/s w/s kr/s kw/s wait svc_t b ad0 0.0 10.6 0.0 169.1 0 1.9 0 da0 357.4 0.0 18103.7 0.0 6 18.7 97 sa0 0.0 3476.1 0.0 34761.3 1 0.3 94 sa1 0.0 3476.1 0.0 34761.3 1 0.3 94 pass0 0.0 0.0 0.0 0.0 0 0.0 0 pass1 0.0 0.0 0.0 0.0 0 0.0 0 pass2 0.0 0.0 0.0 0.0 0 0.0 0 pass3 0.0 0.0 0.0 0.0 0 0.0 0 pass4 0.0 0.0 0.0 0.0 0 0.0 0 ch0 0.0 0.0 0.0 0.0 0 0.0 0 The bigger difference can be seen in the times elapsed to write some directories to tape, here are some examples, following the number of files, directory size and time to write this: Normal directories (without a lot of small files): 3369 files, 169 GB, 1h 50 min 7049 files, 14 GB, 14 min 2764 files, 6 GB, 8 min Directories with small files: 1125796 files, 30 GB, 1h 31 min 1325589 files, 36 GB, 1h, 40 min 861632 files, 80 GB, 10h 40 min 1860893 files, 654 GB, 10h 54min I can provide any other necessary information. Any help is *very* appreciated ! Best Regards, From owner-freebsd-performance@FreeBSD.ORG Wed Dec 19 19:42:34 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ACC1D16A419 for ; Wed, 19 Dec 2007 19:42:34 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost2.sentex.ca (smarthost2.sentex.ca [205.211.164.50]) by mx1.freebsd.org (Postfix) with ESMTP id 7918713C45D for ; Wed, 19 Dec 2007 19:42:34 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by smarthost2.sentex.ca (8.14.1/8.13.8) with ESMTP id lBJJgXDD005451; Wed, 19 Dec 2007 14:42:33 -0500 (EST) (envelope-from mike@sentex.net) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.8/8.13.3) with ESMTP id lBJJgWmJ018860 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 19 Dec 2007 14:42:32 -0500 (EST) (envelope-from mike@sentex.net) Message-Id: <200712191942.lBJJgWmJ018860@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Wed, 19 Dec 2007 14:42:33 -0500 To: "Alexandre Biancalana" , freebsd-performance@freebsd.org From: Mike Tancsa In-Reply-To: <8e10486b0712191109n3d21b02cyf5183ee0cd01d8ce@mail.gmail.co m> References: <8e10486b0712191109n3d21b02cyf5183ee0cd01d8ce@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Cc: Subject: Re: Bad performance when accessing a lot of small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Dec 2007 19:42:34 -0000 At 02:09 PM 12/19/2007, Alexandre Biancalana wrote: >The behavior that I'm observing and that want your help is when the >system is accessing some directory with many small files ( directories >with ~ 1 million of ~30kb files), the performance is very poor. Hi, Have you adjusted the dirhash value ? What does sysctl -a vfs.ufs | grep dirhash show on your box ? Also, do you have a lot of UIDs in your passwd file ? ---Mike From owner-freebsd-performance@FreeBSD.ORG Wed Dec 19 20:07:09 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D505F16A474 for ; Wed, 19 Dec 2007 20:07:09 +0000 (UTC) (envelope-from biancalana@gmail.com) Received: from py-out-1112.google.com (py-out-1112.google.com [64.233.166.181]) by mx1.freebsd.org (Postfix) with ESMTP id 9B17E13C468 for ; Wed, 19 Dec 2007 20:07:09 +0000 (UTC) (envelope-from biancalana@gmail.com) Received: by py-out-1112.google.com with SMTP id u77so379512pyb.3 for ; Wed, 19 Dec 2007 12:07:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=X7K2rJ0IrasVYlpNhqoWPVWDhLluRolZHni9Et+76iA=; b=pGMoe8pAfkmMt/5t9ScTBYfpjB868HdU+vB6gjkBxTTAssdXGsV59mUaWlbRtCPglzSm2ypFGEc/gR8O/jjWMxLH0Kk0Zj026cwT8JuBSh+2XDDRMSICx8DLjQFqdTazKWDzwVobjhL6uEmDDxyAFH8aK7+sBD+SBTE4L50Hosk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=MyuHbRQxjPJLb0P2+KpZaj+5D4mmQzRhEig4iZjHG7l2ve+I0mBMYmS0dPQkRmjcvtkMTI2qoxGd1/ktbbmI+vNBDCg/qa15A+A6p/9Pp4872H7K/ScNwCRVBEllsX1ztPs1z4dpoOnfuiXm432hKQn/oIJKIXleEniT28JYJ8A= Received: by 10.65.153.10 with SMTP id f10mr21548035qbo.33.1198094825113; Wed, 19 Dec 2007 12:07:05 -0800 (PST) Received: by 10.64.184.9 with HTTP; Wed, 19 Dec 2007 12:07:05 -0800 (PST) Message-ID: <8e10486b0712191207r5e256b84xf6cb0bbafc6cfd20@mail.gmail.com> Date: Wed, 19 Dec 2007 18:07:05 -0200 From: "Alexandre Biancalana" To: "Mike Tancsa" In-Reply-To: <200712191942.lBJJgWmJ018860@lava.sentex.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <8e10486b0712191109n3d21b02cyf5183ee0cd01d8ce@mail.gmail.com> <200712191942.lBJJgWmJ018860@lava.sentex.ca> Cc: freebsd-performance@freebsd.org Subject: Re: Bad performance when accessing a lot of small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Dec 2007 20:07:09 -0000 On 12/19/07, Mike Tancsa wrote: > At 02:09 PM 12/19/2007, Alexandre Biancalana wrote: > > >The behavior that I'm observing and that want your help is when the > >system is accessing some directory with many small files ( directories > >with ~ 1 million of ~30kb files), the performance is very poor. > > Hi, I'm using zfs, I think this change the things.. no ? > Have you adjusted the dirhash value ? What does > > sysctl -a vfs.ufs | grep dirhash # sysctl -a vfs.ufs | grep dirhash vfs.ufs.dirhash_docheck: 0 vfs.ufs.dirhash_mem: 1410338 vfs.ufs.dirhash_maxmem: 2097152 vfs.ufs.dirhash_minsize: 2560 > > show on your box ? Also, do you have a lot of UIDs in your passwd file ? No. # wc -l /etc/passwd 45 /etc/passwd From owner-freebsd-performance@FreeBSD.ORG Wed Dec 19 20:14:43 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6F0E916A474 for ; Wed, 19 Dec 2007 20:14:43 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost2.sentex.ca (smarthost2.sentex.ca [205.211.164.50]) by mx1.freebsd.org (Postfix) with ESMTP id 3AB2913C4CE for ; Wed, 19 Dec 2007 20:14:43 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by smarthost2.sentex.ca (8.14.1/8.13.8) with ESMTP id lBJKEgX0011243; Wed, 19 Dec 2007 15:14:42 -0500 (EST) (envelope-from mike@sentex.net) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.8/8.13.3) with ESMTP id lBJKEfpY018975 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 19 Dec 2007 15:14:42 -0500 (EST) (envelope-from mike@sentex.net) Message-Id: <200712192014.lBJKEfpY018975@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Wed, 19 Dec 2007 15:14:42 -0500 To: "Alexandre Biancalana" From: Mike Tancsa In-Reply-To: <8e10486b0712191207r5e256b84xf6cb0bbafc6cfd20@mail.gmail.co m> References: <8e10486b0712191109n3d21b02cyf5183ee0cd01d8ce@mail.gmail.com> <200712191942.lBJJgWmJ018860@lava.sentex.ca> <8e10486b0712191207r5e256b84xf6cb0bbafc6cfd20@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Cc: freebsd-performance@freebsd.org Subject: Re: Bad performance when accessing a lot of small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Dec 2007 20:14:43 -0000 At 03:07 PM 12/19/2007, Alexandre Biancalana wrote: >On 12/19/07, Mike Tancsa wrote: > > At 02:09 PM 12/19/2007, Alexandre Biancalana wrote: > > > > >The behavior that I'm observing and that want your help is when the > > >system is accessing some directory with many small files ( directories > > >with ~ 1 million of ~30kb files), the performance is very poor. > > > > Hi, > >I'm using zfs, I think this change the things.. no ? Hmmm, I dont know. Quite possibly / probably. Based on your values below, its not hitting the max if it were effected. I dont know anything about ZFS to know if there is some specific tuning that needs to be done for many files :( ---Mike > > Have you adjusted the dirhash value ? What does > > > > sysctl -a vfs.ufs | grep dirhash > ># sysctl -a vfs.ufs | grep dirhash >vfs.ufs.dirhash_docheck: 0 >vfs.ufs.dirhash_mem: 1410338 >vfs.ufs.dirhash_maxmem: 2097152 >vfs.ufs.dirhash_minsize: 2560 > > > > > > show on your box ? Also, do you have a lot of UIDs in your passwd file ? > >No. > ># wc -l /etc/passwd > 45 /etc/passwd >_______________________________________________ >freebsd-performance@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-performance >To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" From owner-freebsd-performance@FreeBSD.ORG Thu Dec 20 16:56:50 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7F03616A417 for ; Thu, 20 Dec 2007 16:56:50 +0000 (UTC) (envelope-from _pppp@mail.ru) Received: from mx28.mail.ru (mx28.mail.ru [194.67.23.67]) by mx1.freebsd.org (Postfix) with ESMTP id 2F9EA13C458 for ; Thu, 20 Dec 2007 16:56:49 +0000 (UTC) (envelope-from _pppp@mail.ru) Received: from f156.mail.ru (f156.mail.ru [194.67.57.217]) by mx28.mail.ru (mPOP.Fallback_MX) with ESMTP id 25DD9772DB5 for ; Thu, 20 Dec 2007 19:41:16 +0300 (MSK) Received: from mail by f156.mail.ru with local id 1J5OSL-000Fl1-00; Thu, 20 Dec 2007 19:41:09 +0300 Received: from [89.208.20.114] by koi.mail.ru with HTTP; Thu, 20 Dec 2007 19:41:09 +0300 From: dima <_pppp@mail.ru> To: Alexandre Biancalana Mime-Version: 1.0 X-Mailer: mPOP Web-Mail 2.19 X-Originating-IP: [89.208.20.114] Date: Thu, 20 Dec 2007 19:41:09 +0300 References: <8e10486b0712191207r5e256b84xf6cb0bbafc6cfd20@mail.gmail.com> In-Reply-To: <8e10486b0712191207r5e256b84xf6cb0bbafc6cfd20@mail.gmail.com> Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: 8bit Message-Id: Cc: freebsd-performance@freebsd.org, Mike Tancsa Subject: Re[2]: Bad performance when accessing a lot of small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: dima <_pppp@mail.ru> List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Dec 2007 16:56:50 -0000 >>>The behavior that I'm observing and that want your help is when the >>>system is accessing some directory with many small files ( directories >>>with ~ 1 million of ~30kb files), the performance is very poor. >> >> Hi, > > I'm using zfs, I think this change the things.. no ? > > > Have you adjusted the dirhash value ? What does > > > > sysctl -a vfs.ufs | grep dirhash > > # sysctl -a vfs.ufs | grep dirhash > vfs.ufs.dirhash_docheck: 0 > vfs.ufs.dirhash_mem: 1410338 > vfs.ufs.dirhash_maxmem: 2097152 > vfs.ufs.dirhash_minsize: 2560 I agree with Mike. Try to run this command while listing your large directory. You'll see that dirhash_mem will reach dirhash_maxmem. So, the value of dirhash_maxmem should be increased. One more hint. Mounting filesystems with -noatime option greatly improves read performance. Apply this also, if you haven't done this yet. From owner-freebsd-performance@FreeBSD.ORG Thu Dec 20 17:10:49 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0D84C16A418 for ; Thu, 20 Dec 2007 17:10:49 +0000 (UTC) (envelope-from biancalana@gmail.com) Received: from py-out-1112.google.com (py-out-1112.google.com [64.233.166.182]) by mx1.freebsd.org (Postfix) with ESMTP id B51F613C46B for ; Thu, 20 Dec 2007 17:10:48 +0000 (UTC) (envelope-from biancalana@gmail.com) Received: by py-out-1112.google.com with SMTP id u77so307570pyb.3 for ; Thu, 20 Dec 2007 09:10:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=Gt5I63zUr+odf6cqKRQDYPhdWS7kwjkFOCb2TpAKmbk=; b=Txcfa8GP+N3T/HU9Golrzq3TkGllFGBskWk1Qoa0FUKNZmf5ifUTnfnRKcuafTUZMwo3Sj21gxj43lRAZArISNSdOy0F8sxQ1jrYYe4J4Hmlwkk2Lz0vXkkccf/mmpZNv1riI9LW/04Owqnv3zXpRG289iiSrfTZQfj8EeLUipE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=ndE60yQOxAhOShYOg0ISRnN9MVnYO0OcV5YUj0zMGEut3pbi5IHL8ss+Rcat4c5i3W2+2jMZEUo9yDDK3xe03tJ3HW7EfH/g02xnDec/UBJfyVNWd29MVznjGi21C7yyP51jva+Z+3Vo6NaDwXzkq8vicz5xLTvNUJSpNByrPcs= Received: by 10.65.214.2 with SMTP id r2mr420486qbq.62.1198170647508; Thu, 20 Dec 2007 09:10:47 -0800 (PST) Received: by 10.64.184.9 with HTTP; Thu, 20 Dec 2007 09:10:47 -0800 (PST) Message-ID: <8e10486b0712200910v32ecbfa5h72fb0dc2b82a1009@mail.gmail.com> Date: Thu, 20 Dec 2007 15:10:47 -0200 From: "Alexandre Biancalana" To: "=?ISO-8859-1?Q?Dennis_Kj=E6r_Jensen?=" In-Reply-To: <476A9AB9.7040801@signout.dk> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline References: <8e10486b0712191109n3d21b02cyf5183ee0cd01d8ce@mail.gmail.com> <476A9AB9.7040801@signout.dk> Cc: freebsd-performance@freebsd.org Subject: Re: Bad performance when accessing a lot of small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Dec 2007 17:10:49 -0000 On 12/20/07, Dennis Kj=E6r Jensen wrote: > Alexandre Biancalana wrote: > > > > Increase the dirhash maxmem > I have 'sysctl -w vfs.ufs.dirhash_maxmem=3D268435456' (256Mb)in my > rc.local on a (maildir) fileserver here > This is a zfs filesystem, by the way I raise this limit as suggested. I let you know Thank you! From owner-freebsd-performance@FreeBSD.ORG Thu Dec 20 17:12:28 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9464116A419 for ; Thu, 20 Dec 2007 17:12:27 +0000 (UTC) (envelope-from biancalana@gmail.com) Received: from py-out-1112.google.com (py-out-1112.google.com [64.233.166.177]) by mx1.freebsd.org (Postfix) with ESMTP id 7671513C45D for ; Thu, 20 Dec 2007 17:12:26 +0000 (UTC) (envelope-from biancalana@gmail.com) Received: by py-out-1112.google.com with SMTP id u77so308735pyb.3 for ; Thu, 20 Dec 2007 09:12:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=bgS4oPwwA358/RHiz+YBQ7kwTEOw+GNiXq7xnhjaby4=; b=J3AsM+mU0U3JjAhA+2y+Sl2caZCQjvwBW96MGyjdXgc3BZpSQNWuzGEJNjH14A3PYHW26fO0jz1IKHdXGhUlik9Ey55ghlP7KHMf2zSoMjk+oQrIZHH5MpMP6yO2+Ee3Uoqjy13R/rsdVGdjgj5GaonJv+WZ1d42TfW2Gt+b3Hg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=pWfcWm7d4uMwKN6R1hEeMNuSVJqMwia9b3F26e8zUH53eOyP7Cel7i98C/pmApbQaViM1WK7dVvcB9iVh/oeVBkHjE92jbOCj7fjrcQgTczx+nlK4nV9qo2/EbsCmDcu/2CQLNkiyOl7czwVglqHfEBtoe5txTT1QO4nxxAUwEc= Received: by 10.65.214.19 with SMTP id r19mr443343qbq.13.1198170735050; Thu, 20 Dec 2007 09:12:15 -0800 (PST) Received: by 10.64.184.9 with HTTP; Thu, 20 Dec 2007 09:12:14 -0800 (PST) Message-ID: <8e10486b0712200912g12966a07nfd8bb361adf0eecd@mail.gmail.com> Date: Thu, 20 Dec 2007 15:12:14 -0200 From: "Alexandre Biancalana" To: dima <_pppp@mail.ru> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <8e10486b0712191207r5e256b84xf6cb0bbafc6cfd20@mail.gmail.com> Cc: freebsd-performance@freebsd.org, Mike Tancsa Subject: Re: Re[2]: Bad performance when accessing a lot of small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Dec 2007 17:12:28 -0000 On 12/20/07, dima <_pppp@mail.ru> wrote: > > Try to run this command while listing your large directory. You'll see that dirhash_mem will reach dirhash_maxmem. So, the value of dirhash_maxmem should be increased. > > One more hint. Mounting filesystems with -noatime option greatly improves read performance. Apply this also, if you haven't done this yet. I already set this # zfs get atime backup NAME PROPERTY VALUE SOURCE backup atime off local From owner-freebsd-performance@FreeBSD.ORG Thu Dec 20 17:41:39 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 121B316A419 for ; Thu, 20 Dec 2007 17:41:39 +0000 (UTC) (envelope-from _pppp@mail.ru) Received: from f116.mail.ru (f116.mail.ru [194.67.57.123]) by mx1.freebsd.org (Postfix) with ESMTP id B709F13C469 for ; Thu, 20 Dec 2007 17:41:38 +0000 (UTC) (envelope-from _pppp@mail.ru) Received: from mail by f116.mail.ru with local id 1J5POm-000NHN-00; Thu, 20 Dec 2007 20:41:32 +0300 Received: from [89.208.20.114] by koi.mail.ru with HTTP; Thu, 20 Dec 2007 20:41:32 +0300 From: dima <_pppp@mail.ru> To: Alexandre Biancalana Mime-Version: 1.0 X-Mailer: mPOP Web-Mail 2.19 X-Originating-IP: [89.208.20.114] Date: Thu, 20 Dec 2007 20:41:32 +0300 In-Reply-To: <8e10486b0712200910v32ecbfa5h72fb0dc2b82a1009@mail.gmail.com> References: <8e10486b0712200910v32ecbfa5h72fb0dc2b82a1009@mail.gmail.com> Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: 8bit Message-Id: Cc: freebsd-performance@freebsd.org, =?koi8-r?Q?Dennis_Kj=E6r_Jensen?= Subject: Re[2]: Bad performance when accessing a lot of small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: dima <_pppp@mail.ru> List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Dec 2007 17:41:39 -0000 >> Increase the dirhash maxmem >> I have 'sysctl -w vfs.ufs.dirhash_maxmem=268435456' (256Mb)in my >> rc.local on a (maildir) fileserver here >> > This is a zfs filesystem, by the way I raise this limit as suggested. > > I let you know Can you provide sysctl vfs.zfs output then? ZFS can be tuned by increasing kern.maxvnodes sysctl (don't forget to increase vm.kmem_size and vm.kmem_size_max accordingly). Do it until vfs.numvnodes would stabilize below the kern.maxvnodes value during peak load patterns. From owner-freebsd-performance@FreeBSD.ORG Thu Dec 20 17:48:46 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4ED2516A417 for ; Thu, 20 Dec 2007 17:48:46 +0000 (UTC) (envelope-from biancalana@gmail.com) Received: from py-out-1112.google.com (py-out-1112.google.com [64.233.166.176]) by mx1.freebsd.org (Postfix) with ESMTP id F2DD013C4CE for ; Thu, 20 Dec 2007 17:48:45 +0000 (UTC) (envelope-from biancalana@gmail.com) Received: by py-out-1112.google.com with SMTP id u77so336477pyb.3 for ; Thu, 20 Dec 2007 09:48:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=tU9WOew7shjfoTmDjjN1u3G5n5K9vUjKoDJ2uARisng=; b=yBLOZZr2gdWg1abp4OoAiilMDzPDANmeYAWm/rB3WoCRQ4P2e+nwREzdbFFpksMWU4zQiluHhkgBir0P7i8GtrPCnNxd34EQohUhDhIQFe2+oZScRGxyX1VHMpy5Gw/fqoiLKruUYIIfd7fT/ITVpAR4RvNdX38s+Pu/AFtBssk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=d4giXjA7hy/QHfRLJJsS1+bEEi8i63Wty6dn4csNicKcSshgB2441ivQkG96LwOVbyPJJ+rxBrImBZGysVoMyPvckG5ial5gQnIbdPuLP+1lIw+uA17wq6NXsLeRgZsnb8vmHq7oyYMf8OuwbvyQy1IaoJmMn8iuHe8MBkMHtA4= Received: by 10.65.253.6 with SMTP id f6mr492189qbs.90.1198172921396; Thu, 20 Dec 2007 09:48:41 -0800 (PST) Received: by 10.64.184.9 with HTTP; Thu, 20 Dec 2007 09:48:41 -0800 (PST) Message-ID: <8e10486b0712200948l2bc41063h54d5579373665fc6@mail.gmail.com> Date: Thu, 20 Dec 2007 15:48:41 -0200 From: "Alexandre Biancalana" To: dima <_pppp@mail.ru> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <8e10486b0712200910v32ecbfa5h72fb0dc2b82a1009@mail.gmail.com> Cc: freebsd-performance@freebsd.org, =?KOI8-R?Q?Dennis_Kj=E6r_Jensen?= Subject: Re: Re[2]: Bad performance when accessing a lot of small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Dec 2007 17:48:46 -0000 On 12/20/07, dima <_pppp@mail.ru> wrote: > >> Increase the dirhash maxmem > >> I have 'sysctl -w vfs.ufs.dirhash_maxmem=268435456' (256Mb)in my > >> rc.local on a (maildir) fileserver here > >> > > This is a zfs filesystem, by the way I raise this limit as suggested. > > > > I let you know > > Can you provide > sysctl vfs.zfs output then? > > ZFS can be tuned by increasing kern.maxvnodes sysctl (don't forget to increase vm.kmem_size and vm.kmem_size_max accordingly). Do it until vfs.numvnodes would stabilize below the kern.maxvnodes value during peak load patterns. $ sysctl -A | grep zfs vfs.zfs.arc_min: 49152000 vfs.zfs.arc_max: 1073741824 vfs.zfs.mdcomp_disable: 0 vfs.zfs.prefetch_disable: 1 vfs.zfs.zio.taskq_threads: 0 vfs.zfs.recover: 0 vfs.zfs.vdev.cache.size: 10485760 vfs.zfs.vdev.cache.max: 16384 vfs.zfs.cache_flush_disable: 0 vfs.zfs.zil_disable: 0 vfs.zfs.debug: 0 kstat.zfs.misc.arcstats.hits: 59096254 kstat.zfs.misc.arcstats.misses: 21362360 kstat.zfs.misc.arcstats.demand_data_hits: 21209779 kstat.zfs.misc.arcstats.demand_data_misses: 17688110 kstat.zfs.misc.arcstats.demand_metadata_hits: 37886475 kstat.zfs.misc.arcstats.demand_metadata_misses: 3674250 kstat.zfs.misc.arcstats.prefetch_data_hits: 0 kstat.zfs.misc.arcstats.prefetch_data_misses: 0 kstat.zfs.misc.arcstats.prefetch_metadata_hits: 0 kstat.zfs.misc.arcstats.prefetch_metadata_misses: 0 kstat.zfs.misc.arcstats.mru_hits: 18311543 kstat.zfs.misc.arcstats.mru_ghost_hits: 16169 kstat.zfs.misc.arcstats.mfu_hits: 40784711 kstat.zfs.misc.arcstats.mfu_ghost_hits: 621909 kstat.zfs.misc.arcstats.deleted: 25885007 kstat.zfs.misc.arcstats.recycle_miss: 9413044 kstat.zfs.misc.arcstats.mutex_miss: 9253 kstat.zfs.misc.arcstats.evict_skip: 4418321 kstat.zfs.misc.arcstats.hash_elements: 232462 kstat.zfs.misc.arcstats.hash_elements_max: 489237 kstat.zfs.misc.arcstats.hash_collisions: 12684452 kstat.zfs.misc.arcstats.hash_chains: 56900 kstat.zfs.misc.arcstats.hash_chain_max: 20 kstat.zfs.misc.arcstats.p: 1023016585 kstat.zfs.misc.arcstats.c: 1040187392 kstat.zfs.misc.arcstats.c_min: 49152000 kstat.zfs.misc.arcstats.c_max: 1073741824 kstat.zfs.misc.arcstats.size: 1040184320 $ cat /etc/sysctl.conf kern.maxvnodes=400000 vfs.ufs.dirhash_maxmem=268435456 $ cat /boot/loader.conf kern.maxdsiz="2G" # Set the max data size to 4GB kern.maxssiz="1G" # Set the max stack size 2GB vfs.zfs.prefetch_disable="1" vfs.zfs.arc_max="1G" vm.kmem_size_max="1500M" vm.kmem_size="1500M" kern.ipc.nmbclusters="32768" From owner-freebsd-performance@FreeBSD.ORG Fri Dec 21 09:00:20 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 39E0E16A468 for ; Fri, 21 Dec 2007 09:00:20 +0000 (UTC) (envelope-from patpro@patpro.net) Received: from smtp.univ-lyon2.fr (smtp.univ-lyon2.fr [159.84.143.102]) by mx1.freebsd.org (Postfix) with ESMTP id F3CC213C457 for ; Fri, 21 Dec 2007 09:00:19 +0000 (UTC) (envelope-from patpro@patpro.net) Received: from localhost (localhost [127.0.0.1]) by smtp.univ-lyon2.fr (Postfix) with ESMTP id B62A25CBFD83 for ; Fri, 21 Dec 2007 09:39:42 +0100 (CET) X-Virus-Scanned: amavisd-new at univ-lyon2.fr Received: from smtp.univ-lyon2.fr ([127.0.0.1]) by localhost (smtp.univ-lyon2.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8zBa4O76JVH7 for ; Fri, 21 Dec 2007 09:39:41 +0100 (CET) Received: from [159.84.148.56] (dhcp-159-84-148-56.univ-lyon2.fr [159.84.148.56]) by smtp.univ-lyon2.fr (Postfix) with ESMTP id 97E625CBFD7C for ; Fri, 21 Dec 2007 09:39:41 +0100 (CET) Mime-Version: 1.0 (Apple Message framework v753) Content-Transfer-Encoding: 7bit Message-Id: <66621B3A-7879-4DE8-B4D5-5A46B91BAC5E@patpro.net> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed To: freebsd-performance@freebsd.org From: Patrick Proniewski Date: Fri, 21 Dec 2007 09:39:39 +0100 X-Mailer: Apple Mail (2.753) Subject: intel drivers vs. freebsd drivers X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Dec 2007 09:00:20 -0000 Hello, I've just discovered that Intel provides some drivers for the Intel PRO/1000 Family on FreeBSD: I was wondering: is there any point in running those drivers instead of the freebsd ones, performance wise of course. thanks, patpro From owner-freebsd-performance@FreeBSD.ORG Fri Dec 21 09:03:57 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7D84916A46C for ; Fri, 21 Dec 2007 09:03:57 +0000 (UTC) (envelope-from gergely.czuczy@harmless.hu) Received: from marvin.harmless.hu (marvin.harmless.hu [195.56.55.204]) by mx1.freebsd.org (Postfix) with ESMTP id 3976113C458 for ; Fri, 21 Dec 2007 09:03:57 +0000 (UTC) (envelope-from gergely.czuczy@harmless.hu) Received: from localhost (marvin-mail [192.168.0.2]) by marvin.harmless.hu (Postfix) with ESMTP id 506117C163D; Fri, 21 Dec 2007 10:03:54 +0100 (CET) X-Virus-Scanned: by amavisd-new-2.4.2 (20060627) (Debian) at harmless.hu Received: from marvin.harmless.hu ([192.168.0.2]) by localhost (marvin.harmless.hu [192.168.0.2]) (amavisd-new, port 10024) with ESMTP id rDmBldyYme+D; Fri, 21 Dec 2007 10:03:53 +0100 (CET) Received: from marvin.harmless.hu (localhost [127.0.0.1]) by marvin.harmless.hu (Postfix) with ESMTP id 275F37C0104; Fri, 21 Dec 2007 10:03:52 +0100 (CET) Date: Fri, 21 Dec 2007 10:03:52 +0100 From: Gergely CZUCZY To: Patrick Proniewski Message-ID: <20071221090352.GA6040@harmless.hu> References: <66621B3A-7879-4DE8-B4D5-5A46B91BAC5E@patpro.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=x-unknown; protocol="application/pgp-signature"; boundary="T4sUOijqQbZv57TR" Content-Disposition: inline In-Reply-To: <66621B3A-7879-4DE8-B4D5-5A46B91BAC5E@patpro.net> User-Agent: mutt-ng/devel-r804 (FreeBSD) Cc: freebsd-performance@freebsd.org Subject: Re: intel drivers vs. freebsd drivers X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Dec 2007 09:03:57 -0000 --T4sUOijqQbZv57TR Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Dec 21, 2007 at 09:39:39AM +0100, Patrick Proniewski wrote: > Hello, >=20 > I've just discovered that Intel provides some drivers for the Intel PRO/1= 000 Family on FreeBSD: >=20 > >=20 > I was wondering: is there any point in running those drivers instead of t= he freebsd ones, performance wise of course. AFAIK these are the drivers that can be found in the base system as well. Intel's drivers are pretty fine, there's no need to write new ones. >=20 > thanks, > patpro > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd= =2Eorg" Sincerely, Gergely Czuczy mailto: gergely.czuczy@harmless.hu --=20 Weenies test. Geniuses solve problems that arise. --T4sUOijqQbZv57TR Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) owGVVL9rFEEUjkYRBhTyHzxSGNDbvb1LjJfVyw9zSYxBE0xA8AdxdvfdZczuzDoz e5dLJzYWFsFGRRDBxkaw8H+wERs7EazE3sZK8M15F1Okcdli38z33vve983s3snh oaMjn969v3X28ZMXR94eW4rOZIW1suVlXLeF9CpBUPEmg4nAm/TO1WrVKI6iiYmp ajVqBkvbDzrzSlqU1tvo5hiCxR1bzlMu5AWIt7g2aOuFbXo1NsA1hMmVEVYoGYKQ qZC4v7ehuTRN1N6CjFUiZCuE+4WymHi5FtLyKEXGViUsalGCBsZQrZSgGgTngVsI psJx985dhbMBsS7BGrdaxNuwppUU2DHbAjqayoVsGi5jmqoSm65XA4qWx9oI9wpj IREmVm3UmIDdoqrLxCyFXKu2SNCAURlCogUhDDSVJhD2MWvXV8uVOguCABZ5JtIu KMcU8dJ6Ixw0urhlbR6Wy4nqyFTxJKaxUfvCVfBjlZUTtFykm9Qr9rnJd2aIfFLE drlRH2/UxmunG5SYUMQoroyfn6hO7w8BHW6go2SCuqedMI6eRuCyC7miJiQ46EJK 2qYtZf7NIqSxyBNQTbB1RkM1iXlkaEGiKUGOmqbNuIwROoLyCBergvz12dzi3PKK 60TLnLo5SQZlexrGXEJEFVUhE8fAASJOaNOlphk41mSHz3pCjpn9bFct12htF5p0 Tkp/xyGAVCDRWaTIUWGRok6Pqc8GalBnuW3IYci5Jf/oY/P/Hsroi+AdGH+2v+Yr 3YKMvHJapsJYgvfNdZHxD+DKDkfZvR0hm6p8SGHK31BQSFNEJtYionENkmDOPJfv hh09JM87kDIgR6ejukCNRxlbF4TRmHZLjC2hbtEXzO8W8W6XuapWhdD6u+zHveVZ urZZisb4WwVjnufkvIFIN4jsRGN9WKKgML3bkLadQ4ouZtY3m2vhDsWjmeHjQ+7/ MPi3jBy1H4Zevfn4c+/ElWs3zw3ffvj91FPvx53R9tDLsV/fPr98PfLq66fnX8LS yr3V38/u/gE= =horo -----END PGP SIGNATURE----- --T4sUOijqQbZv57TR-- From owner-freebsd-performance@FreeBSD.ORG Fri Dec 21 10:09:44 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8787216A418 for ; Fri, 21 Dec 2007 10:09:44 +0000 (UTC) (envelope-from patpro@patpro.net) Received: from smtp.univ-lyon2.fr (smtp.univ-lyon2.fr [159.84.143.102]) by mx1.freebsd.org (Postfix) with ESMTP id 4724413C458 for ; Fri, 21 Dec 2007 10:09:44 +0000 (UTC) (envelope-from patpro@patpro.net) Received: from localhost (localhost [127.0.0.1]) by smtp.univ-lyon2.fr (Postfix) with ESMTP id AC3925CC84C7; Fri, 21 Dec 2007 11:09:42 +0100 (CET) X-Virus-Scanned: amavisd-new at univ-lyon2.fr Received: from smtp.univ-lyon2.fr ([127.0.0.1]) by localhost (smtp.univ-lyon2.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8vqfowtPCh5b; Fri, 21 Dec 2007 11:09:41 +0100 (CET) Received: from [159.84.148.56] (dhcp-159-84-148-56.univ-lyon2.fr [159.84.148.56]) by smtp.univ-lyon2.fr (Postfix) with ESMTP id A2DA15CC84BC; Fri, 21 Dec 2007 11:09:41 +0100 (CET) In-Reply-To: <20071221090352.GA6040@harmless.hu> References: <66621B3A-7879-4DE8-B4D5-5A46B91BAC5E@patpro.net> <20071221090352.GA6040@harmless.hu> Mime-Version: 1.0 (Apple Message framework v753) Content-Type: text/plain; charset=ISO-8859-1; delsp=yes; format=flowed Message-Id: <1777A05B-B3E8-42A6-BC88-E5DA2DBB650C@patpro.net> Content-Transfer-Encoding: quoted-printable From: Patrick Proniewski Date: Fri, 21 Dec 2007 11:09:40 +0100 To: Gergely CZUCZY X-Mailer: Apple Mail (2.753) Cc: freebsd-performance@freebsd.org Subject: Re: intel drivers vs. freebsd drivers X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Dec 2007 10:09:44 -0000 On 21 d=E9c. 07, at 10:03, Gergely CZUCZY wrote: >> I've just discovered that Intel provides some drivers for the =20 >> Intel PRO/1000 Family on FreeBSD: >> I was wondering: is there any point in running those drivers =20 >> instead of the freebsd ones, performance wise of course. > AFAIK these are the drivers that can be found in the base system as =20= > well. > Intel's drivers are pretty fine, there's no need to write new ones. I've made a quick comparison between /usr/src/sys/dev/mii/e1000phy.c =20 (freebsd 6.2) and e1000_phy.c (as part of the em-6.6.6.tgz provided =20 by Intel), and the too files are quite different. But well, I'm not a developer, and that's probably not significant. regards, patpro= From owner-freebsd-performance@FreeBSD.ORG Fri Dec 21 10:48:35 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 286C116A420 for ; Fri, 21 Dec 2007 10:48:35 +0000 (UTC) (envelope-from tom@tomjudge.com) Received: from psmtp.com (s200aog14.obsmtp.com [207.126.144.128]) by mx1.freebsd.org (Postfix) with SMTP id 3FF9613C44B for ; Fri, 21 Dec 2007 10:48:34 +0000 (UTC) (envelope-from tom@tomjudge.com) Received: from source ([217.206.187.80]) by eu1sys200aob014.postini.com ([207.126.147.11]) with SMTP; Fri, 21 Dec 2007 10:48:33 UTC Received: from bill.mintel.co.uk (bill.mintel.co.uk [10.0.0.89]) by rodney.mintel.co.uk (Postfix) with ESMTP id B5FE118141B; Fri, 21 Dec 2007 10:20:57 +0000 (GMT) Message-ID: <476B9388.8000900@tomjudge.com> Date: Fri, 21 Dec 2007 10:20:56 +0000 From: Tom Judge User-Agent: Thunderbird 2.0.0.6 (X11/20071022) MIME-Version: 1.0 To: Patrick Proniewski References: <66621B3A-7879-4DE8-B4D5-5A46B91BAC5E@patpro.net> <20071221090352.GA6040@harmless.hu> <1777A05B-B3E8-42A6-BC88-E5DA2DBB650C@patpro.net> In-Reply-To: <1777A05B-B3E8-42A6-BC88-E5DA2DBB650C@patpro.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit Cc: Gergely CZUCZY , freebsd-performance@freebsd.org, Jack Vogel Subject: Re: intel drivers vs. freebsd drivers X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Dec 2007 10:48:35 -0000 Patrick Proniewski wrote: > On 21 déc. 07, at 10:03, Gergely CZUCZY wrote: > >>> I've just discovered that Intel provides some drivers for the Intel >>> PRO/1000 Family on FreeBSD: >>> I was wondering: is there any point in running those drivers instead >>> of the freebsd ones, performance wise of course. > >> AFAIK these are the drivers that can be found in the base system as well. >> Intel's drivers are pretty fine, there's no need to write new ones. > > I've made a quick comparison between /usr/src/sys/dev/mii/e1000phy.c > (freebsd 6.2) and e1000_phy.c (as part of the em-6.6.6.tgz provided by > Intel), and the too files are quite different. > But well, I'm not a developer, and that's probably not significant. > > regards, > patpro Jack Vogel (cc'd) is the Intel developer that maintains the FreeBSD drivers. He should be able to shed more light on this question. Tom From owner-freebsd-performance@FreeBSD.ORG Fri Dec 21 20:17:53 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5E74C16A419 for ; Fri, 21 Dec 2007 20:17:53 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 3A9A013C45B for ; Fri, 21 Dec 2007 20:17:53 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: by elvis.mu.org (Postfix, from userid 1192) id 8864D1A4D80; Fri, 21 Dec 2007 12:16:25 -0800 (PST) Date: Fri, 21 Dec 2007 12:16:25 -0800 From: Alfred Perlstein To: Alexandre Biancalana Message-ID: <20071221201625.GZ16982@elvis.mu.org> References: <8e10486b0712191109n3d21b02cyf5183ee0cd01d8ce@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <8e10486b0712191109n3d21b02cyf5183ee0cd01d8ce@mail.gmail.com> User-Agent: Mutt/1.4.2.3i Cc: freebsd-performance@freebsd.org Subject: Re: Bad performance when accessing a lot of small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Dec 2007 20:17:53 -0000 * Alexandre Biancalana [071219 11:35] wrote: > Hi List, > > I have a backup server running FreeBSD 7-BETA3. The cpu is CPU: > Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz, 3GB Ram, 10x 500GB > SATA, Areca 1231-ML, the filesystem used to backup my other servers > locally is build on top of ARC-1231, 4TB (32k stripe) zfs filesystem > with gzip compression. > > This machine receive backups from ~30 servers, (of all kinds and > sizes, databases, fileservers, image servers, webservers, etc) all > night, write the last day in LTO-3 tapes and store some days older > days in disk. > > The behavior that I'm observing and that want your help is when the > system is accessing some directory with many small files ( directories > with ~ 1 million of ~30kb files), the performance is very poor. There is a lot of very good tuning advice in this thread, however one thing to note is that having ~1 million files in a directory is not a very good thing to do on just about any filesystem. One trick that a lot of people do is hashing the directories themselves so that you use some kind of computation to break this huge dir into multiple smaller dirs. If you can figure out a hashing algorithm, that may help you. For instance, if you tell sendmail to use "/var/spool/mq*" for its mail spool and you happen to have 256 directories under "/var/spool/" named "mq000" through "mq256" it will randomly pick a directory to dump a file in. This makes the performance a lot better. For one million files you can probably do a two level hash, you just have to figure out a good hashing algorithm. If you you can describe the data, I may be able to help you come up with a hashing algorithm for it. -Alfred From owner-freebsd-performance@FreeBSD.ORG Fri Dec 21 20:49:58 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D6CE616A421 for ; Fri, 21 Dec 2007 20:49:58 +0000 (UTC) (envelope-from biancalana@gmail.com) Received: from wa-out-1112.google.com (wa-out-1112.google.com [209.85.146.181]) by mx1.freebsd.org (Postfix) with ESMTP id 973DC13C4F2 for ; Fri, 21 Dec 2007 20:49:58 +0000 (UTC) (envelope-from biancalana@gmail.com) Received: by wa-out-1112.google.com with SMTP id k17so765097waf.3 for ; Fri, 21 Dec 2007 12:49:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=cbQGs15tXf1sOU2nastfwyfS2g2C43mgadUJiVcwTl4=; b=YGxOfSnAC9ctksj+o8Fqam0h81jrMUmAUNSH6QuXzEbzm46SGkVfUJmocsWEWn04op2qi/J3ArbLcJgqFfuX+Voybuj2J/TgHmx6XATqLvzz0Yua//j8FbYO7H3DudYKSJo3isSLb9rK3wKoYa551qwODJbgiwqCpVETtvm1Reo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=LvMpoDp9Jt6k3FTVBIP+GYi3dIXxbG4/1+EngfjGfuIrsfX8dnP0ZhU339mvAiWGeAnI4iuOqqujr6JqsvHvzuZrNztNNlBCufgCFQJ5zEdhH0lyDheY/L9klcJNXtY4BMSDd5Gek0F5ngINWeDhP47iANtUHteKlAJ4e94XqRc= Received: by 10.114.210.2 with SMTP id i2mr661090wag.36.1198270198099; Fri, 21 Dec 2007 12:49:58 -0800 (PST) Received: by 10.114.27.7 with HTTP; Fri, 21 Dec 2007 12:49:58 -0800 (PST) Message-ID: <8e10486b0712211249v4c5571ddud21b277f686992b2@mail.gmail.com> Date: Fri, 21 Dec 2007 17:49:58 -0300 From: "Alexandre Biancalana" To: "Alfred Perlstein" In-Reply-To: <20071221201625.GZ16982@elvis.mu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <8e10486b0712191109n3d21b02cyf5183ee0cd01d8ce@mail.gmail.com> <20071221201625.GZ16982@elvis.mu.org> Cc: freebsd-performance@freebsd.org Subject: Re: Bad performance when accessing a lot of small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Dec 2007 20:49:58 -0000 On 12/21/07, Alfred Perlstein wrote: Hi Alfred ! > > There is a lot of very good tuning advice in this thread, however > one thing to note is that having ~1 million files in a directory > is not a very good thing to do on just about any filesystem. I think I was not clear, I will try explain better. This Backup Server has a /backup zfs filesystem of 4TB. Each host that do backups to this server has a /backup/ and /backup//YYYYMMDD zfs filesystems, the last contains the backups for some day of that server. My problem is with some hosts that have in your directory structure a lot of small files, independent of the hierarchy. > > One trick that a lot of people do is hashing the directories themselves > so that you use some kind of computation to break this huge dir into > multiple smaller dirs. I have the two cases, when you have a lot of files inside on directory without any directory organization/distribution but I also have problems with hosts that have files organized in a hierarchy like YYYY/MM/DD/ having no more that 200 files in the day directory level, but almost one million of files in total. Just for info, I made the previous suggested tuning (raise dirhash, maxvnodes) but this improve nothing. Thanks for your hint! From owner-freebsd-performance@FreeBSD.ORG Fri Dec 21 21:29:37 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 047B516A469 for ; Fri, 21 Dec 2007 21:29:37 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id D3F8213C448 for ; Fri, 21 Dec 2007 21:29:36 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: by elvis.mu.org (Postfix, from userid 1192) id E980E1A4D7C; Fri, 21 Dec 2007 13:28:08 -0800 (PST) Date: Fri, 21 Dec 2007 13:28:08 -0800 From: Alfred Perlstein To: Alexandre Biancalana Message-ID: <20071221212808.GE16982@elvis.mu.org> References: <8e10486b0712191109n3d21b02cyf5183ee0cd01d8ce@mail.gmail.com> <20071221201625.GZ16982@elvis.mu.org> <8e10486b0712211249v4c5571ddud21b277f686992b2@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <8e10486b0712211249v4c5571ddud21b277f686992b2@mail.gmail.com> User-Agent: Mutt/1.4.2.3i Cc: freebsd-performance@freebsd.org Subject: Re: Bad performance when accessing a lot of small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Dec 2007 21:29:37 -0000 * Alexandre Biancalana [071221 12:48] wrote: > On 12/21/07, Alfred Perlstein wrote: > > Hi Alfred ! > > > > > There is a lot of very good tuning advice in this thread, however > > one thing to note is that having ~1 million files in a directory > > is not a very good thing to do on just about any filesystem. > > I think I was not clear, I will try explain better. > > This Backup Server has a /backup zfs filesystem of 4TB. > > Each host that do backups to this server has a /backup/ and > /backup//YYYYMMDD zfs filesystems, the last contains the > backups for some day of that server. > > My problem is with some hosts that have in your directory structure a > lot of small files, independent of the hierarchy. Can you not tar these files together? > > One trick that a lot of people do is hashing the directories themselves > > so that you use some kind of computation to break this huge dir into > > multiple smaller dirs. > > I have the two cases, when you have a lot of files inside on directory > without any directory organization/distribution but I also have > problems with hosts that have files organized in a hierarchy like > YYYY/MM/DD/ having no more that 200 files in the day directory > level, but almost one million of files in total. > > Just for info, I made the previous suggested tuning (raise dirhash, > maxvnodes) but this improve nothing. > > Thanks for your hint! What application are you scanning these files with? I know I had issues with rsync in particular where I had to have it rsync smaller pieces of a collection for it to work nicely instead of going for the whole heirarchy. -- - Alfred Perlstein From owner-freebsd-performance@FreeBSD.ORG Fri Dec 21 22:26:51 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A88EB16A473 for ; Fri, 21 Dec 2007 22:26:51 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.158]) by mx1.freebsd.org (Postfix) with ESMTP id D6AE313C4DB for ; Fri, 21 Dec 2007 22:26:50 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: by fg-out-1718.google.com with SMTP id 16so269632fgg.35 for ; Fri, 21 Dec 2007 14:26:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=kGkEbt9w5g4eBzaWZfx7DWpFEZYlZGzBpXxtlWyQPHk=; b=NTDr1Wv/jht8HczVjPrQoaUJbj3IxJAmSlP9s0ShevVjhPxh8Rpl1ZjG0oaqEPenkzIUjjQqPg21syjDduggI9B/nC26n9kvnaLog006WmuaKaNFQSJV0CWmheNHXPVnSmSYwyTOt/tD0QPnz5AktTq2R/nOIzvD4kF9jl4wRlo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=FaeDR0FPArusqHXi37riFO/Hw3ElF96EKLMNDAAj83KAvVTQoydrPUCp+gfPb0frr/UkCGbQzw3StizBKZYE6VSO9UbBzqKyLQuWT7ZczmNBkFd5GQzgC53so+BAEilF9dcP2ekgIZ4CxP2E/LLBFkmEKbZc4czwf3KrNQJw4Uw= Received: by 10.86.26.11 with SMTP id 11mr1563254fgz.37.1198274459306; Fri, 21 Dec 2007 14:00:59 -0800 (PST) Received: by 10.86.97.10 with HTTP; Fri, 21 Dec 2007 14:00:59 -0800 (PST) Message-ID: <2a41acea0712211400n4f43fa5br673ff9a68c163717@mail.gmail.com> Date: Fri, 21 Dec 2007 14:00:59 -0800 From: "Jack Vogel" To: "Tom Judge" In-Reply-To: <476B9388.8000900@tomjudge.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline References: <66621B3A-7879-4DE8-B4D5-5A46B91BAC5E@patpro.net> <20071221090352.GA6040@harmless.hu> <1777A05B-B3E8-42A6-BC88-E5DA2DBB650C@patpro.net> <476B9388.8000900@tomjudge.com> Cc: Gergely CZUCZY , freebsd-performance@freebsd.org, Patrick Proniewski Subject: Re: intel drivers vs. freebsd drivers X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Dec 2007 22:26:51 -0000 On Dec 21, 2007 2:20 AM, Tom Judge wrote: > Patrick Proniewski wrote: > > On 21 d=E9c. 07, at 10:03, Gergely CZUCZY wrote: > > > >>> I've just discovered that Intel provides some drivers for the Intel > >>> PRO/1000 Family on FreeBSD: > >>> I was wondering: is there any point in running those drivers instead > >>> of the freebsd ones, performance wise of course. > > > >> AFAIK these are the drivers that can be found in the base system as we= ll. > >> Intel's drivers are pretty fine, there's no need to write new ones. > > > > I've made a quick comparison between /usr/src/sys/dev/mii/e1000phy.c > > (freebsd 6.2) and e1000_phy.c (as part of the em-6.6.6.tgz provided by > > Intel), and the too files are quite different. > > But well, I'm not a developer, and that's probably not significant. > > > > regards, > > patpro > > Jack Vogel (cc'd) is the Intel developer that maintains the FreeBSD > drivers. He should be able to shed more light on this question. > > Tom > I don't know what uses the mii code but my drivers dont :) As for the Intel published version, its my same code but there tends to be some varations due to the time involved in the machinery within Intel to release something. A couple years ago there was a pretty big gap between the FreeBSD CVS code and the Intel, but since I took this job I have been working hard to bring the two as close together as possible. For instance, right now the latest driver for 6.3 or 7.0 is newer than that 6.6.6 driver. In any case, I am the owner of both streams so you know who to call if you have problems :) BTW, I'm on vacation (home reading, relaxing, and playing a lot of bass) so other than email don't expect anything from me til 1/7 :) Jack From owner-freebsd-performance@FreeBSD.ORG Fri Dec 21 23:02:41 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DDF8716A418 for ; Fri, 21 Dec 2007 23:02:41 +0000 (UTC) (envelope-from patpro@patpro.net) Received: from postfix1-g20.free.fr (postfix1-g20.free.fr [212.27.60.42]) by mx1.freebsd.org (Postfix) with ESMTP id 8181313C4E1 for ; Fri, 21 Dec 2007 23:02:36 +0000 (UTC) (envelope-from patpro@patpro.net) Received: from smtp1-g19.free.fr (smtp1-g19.free.fr [212.27.42.27]) by postfix1-g20.free.fr (Postfix) with ESMTP id C6A65207B5F1 for ; Fri, 21 Dec 2007 23:44:36 +0100 (CET) Received: from smtp1-g19.free.fr (localhost.localdomain [127.0.0.1]) by smtp1-g19.free.fr (Postfix) with ESMTP id D80441AB2E0; Fri, 21 Dec 2007 23:44:34 +0100 (CET) Received: from boleskine.patpro.net (boleskine.patpro.net [82.235.12.223]) by smtp1-g19.free.fr (Postfix) with ESMTP id A6DFD1AB2A9; Fri, 21 Dec 2007 23:44:34 +0100 (CET) Received: from [192.168.0.2] (unknown [192.168.0.2]) by boleskine.patpro.net (Postfix) with ESMTP id E416B1CC10; Fri, 21 Dec 2007 23:44:33 +0100 (CET) Message-Id: <0EA610B6-B487-448E-9B39-D74A891903E8@patpro.net> From: Patrick Proniewski To: "Jack Vogel" In-Reply-To: <2a41acea0712211400n4f43fa5br673ff9a68c163717@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Apple Message framework v915) Date: Fri, 21 Dec 2007 23:44:34 +0100 References: <66621B3A-7879-4DE8-B4D5-5A46B91BAC5E@patpro.net> <20071221090352.GA6040@harmless.hu> <1777A05B-B3E8-42A6-BC88-E5DA2DBB650C@patpro.net> <476B9388.8000900@tomjudge.com> <2a41acea0712211400n4f43fa5br673ff9a68c163717@mail.gmail.com> X-Mailer: Apple Mail (2.915) Cc: Tom Judge , Gergely CZUCZY , freebsd-performance@freebsd.org Subject: Re: intel drivers vs. freebsd drivers X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Dec 2007 23:02:42 -0000 On 21 d=E9c. 2007, at 23:00, Jack Vogel wrote: >> Jack Vogel (cc'd) is the Intel developer that maintains the FreeBSD >> drivers. He should be able to shed more light on this question. thanks Tom > I don't know what uses the mii code but my drivers dont :) ok, it's probably not related. I've found it with a `locate e1000` I've just tried a `locate /em` with much more success. > As for the Intel published version, its my same code but there tends > to be some varations due to the time involved in the machinery > within Intel to release something. > > A couple years ago there was a pretty big gap between the > FreeBSD CVS code and the Intel, but since I took this job > I have been working hard to bring the two as close together > as possible. > > For instance, right now the latest driver for 6.3 or 7.0 is > newer than that 6.6.6 driver. I'm very pleased to read that! > BTW, I'm on vacation (home reading, relaxing, and playing > a lot of bass) so other than email don't expect anything from > me til 1/7 :) thanks a lot, and enjoy your vacation. patpro= From owner-freebsd-performance@FreeBSD.ORG Fri Dec 21 23:55:25 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6536B16A419 for ; Fri, 21 Dec 2007 23:55:25 +0000 (UTC) (envelope-from biancalana@gmail.com) Received: from mu-out-0910.google.com (mu-out-0910.google.com [209.85.134.191]) by mx1.freebsd.org (Postfix) with ESMTP id CD42F13C44B for ; Fri, 21 Dec 2007 23:55:24 +0000 (UTC) (envelope-from biancalana@gmail.com) Received: by mu-out-0910.google.com with SMTP id w9so595100mue.6 for ; Fri, 21 Dec 2007 15:55:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=+0CPQscKda3KHBycmg459ZR21D6hiXofgAaH+mGbqRY=; b=K1eYbAnKPoQkjD10LWoJJrB8RmlncLappbybdvLuFSqfnWboLb7SlG6OjBWAYonsPeMAB30XX5TLLgizzXhpxjupo7uIu8XqJCPzfmRrbk1QSYsJo6nQIL+EHRWPqZTv5b1Ba9IpGEclLAxONVHo0uCiQrvjXjv8f+08DYgId94= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=DJZtzOcMKwxIt4elanbGj4i8VsbEAmxXSVxoZYpkIDnIvk6DnoaYofuyEYSbNWTG/MNZzG8S7J2kQTKvuvezVDLCBzLqRqLQhHvFkSchFzO0UET+lFyQtFMEfinErGwjh4GNww4rizlfSSGTbCv6Nu0KPq6KKWP2L4FCVfU9iz4= Received: by 10.64.3.9 with SMTP id 9mr19856696qbc.0.1198281322381; Fri, 21 Dec 2007 15:55:22 -0800 (PST) Received: by 10.64.184.9 with HTTP; Fri, 21 Dec 2007 15:55:22 -0800 (PST) Message-ID: <8e10486b0712211555n3efe8729qff14387be128cf10@mail.gmail.com> Date: Fri, 21 Dec 2007 20:55:22 -0300 From: "Alexandre Biancalana" To: "Alfred Perlstein" In-Reply-To: <20071221212808.GE16982@elvis.mu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <8e10486b0712191109n3d21b02cyf5183ee0cd01d8ce@mail.gmail.com> <20071221201625.GZ16982@elvis.mu.org> <8e10486b0712211249v4c5571ddud21b277f686992b2@mail.gmail.com> <20071221212808.GE16982@elvis.mu.org> Cc: freebsd-performance@freebsd.org Subject: Re: Bad performance when accessing a lot of small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Dec 2007 23:55:25 -0000 On 12/21/07, Alfred Perlstein wrote: > * Alexandre Biancalana [071221 12:48] wrote: > > On 12/21/07, Alfred Perlstein wrote: > > > > Hi Alfred ! > > > > > > > > There is a lot of very good tuning advice in this thread, however > > > one thing to note is that having ~1 million files in a directory > > > is not a very good thing to do on just about any filesystem. > > > > I think I was not clear, I will try explain better. > > > > This Backup Server has a /backup zfs filesystem of 4TB. > > > > Each host that do backups to this server has a /backup/ and > > /backup//YYYYMMDD zfs filesystems, the last contains the > > backups for some day of that server. > > > > My problem is with some hosts that have in your directory structure a > > lot of small files, independent of the hierarchy. > > Can you not tar these files together? This is what I'm trying to do.... > > > > One trick that a lot of people do is hashing the directories themselves > > > so that you use some kind of computation to break this huge dir into > > > multiple smaller dirs. > > > > I have the two cases, when you have a lot of files inside on directory > > without any directory organization/distribution but I also have > > problems with hosts that have files organized in a hierarchy like > > YYYY/MM/DD/ having no more that 200 files in the day directory > > level, but almost one million of files in total. > > > > Just for info, I made the previous suggested tuning (raise dirhash, > > maxvnodes) but this improve nothing. > > > > Thanks for your hint! > > What application are you scanning these files with? I know I had > issues with rsync in particular where I had to have it rsync > smaller pieces of a collection for it to work nicely instead of > going for the whole heirarchy. tar I run tar in the /backup//YYYYMMDD writing to LTO3 tape drive, the problem is that when origin directory contains a lot of small files the process is *much* more slow.... this is my question since the thread start. From owner-freebsd-performance@FreeBSD.ORG Sat Dec 22 00:27:05 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5196B16A468 for ; Sat, 22 Dec 2007 00:27:05 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 0632313C47E for ; Sat, 22 Dec 2007 00:27:03 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: by elvis.mu.org (Postfix, from userid 1192) id 928C51A4D7C; Fri, 21 Dec 2007 16:25:35 -0800 (PST) Date: Fri, 21 Dec 2007 16:25:35 -0800 From: Alfred Perlstein To: Alexandre Biancalana Message-ID: <20071222002535.GL16982@elvis.mu.org> References: <8e10486b0712191109n3d21b02cyf5183ee0cd01d8ce@mail.gmail.com> <20071221201625.GZ16982@elvis.mu.org> <8e10486b0712211249v4c5571ddud21b277f686992b2@mail.gmail.com> <20071221212808.GE16982@elvis.mu.org> <8e10486b0712211555n3efe8729qff14387be128cf10@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <8e10486b0712211555n3efe8729qff14387be128cf10@mail.gmail.com> User-Agent: Mutt/1.4.2.3i Cc: freebsd-performance@freebsd.org Subject: Re: Bad performance when accessing a lot of small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Dec 2007 00:27:05 -0000 * Alexandre Biancalana [071221 15:53] wrote: > On 12/21/07, Alfred Perlstein wrote: > > > > What application are you scanning these files with? I know I had > > issues with rsync in particular where I had to have it rsync > > smaller pieces of a collection for it to work nicely instead of > > going for the whole heirarchy. > > tar > > I run tar in the /backup//YYYYMMDD writing to LTO3 tape > drive, the problem is that when origin directory contains a lot of > small files the process is *much* more slow.... this is my question > since the thread start. Have you tried the 'noatime' mount option? That should help. Can you provide a histogram of the count of files per directory? -- - Alfred Perlstein