From owner-freebsd-performance@FreeBSD.ORG Tue Jul 15 20:56:15 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3BDE61065674 for ; Tue, 15 Jul 2008 20:56:15 +0000 (UTC) (envelope-from rahulone@gmail.com) Received: from yx-out-2324.google.com (yx-out-2324.google.com [74.125.44.29]) by mx1.freebsd.org (Postfix) with ESMTP id E4CD08FC19 for ; Tue, 15 Jul 2008 20:56:14 +0000 (UTC) (envelope-from rahulone@gmail.com) Received: by yx-out-2324.google.com with SMTP id 8so1439713yxb.13 for ; Tue, 15 Jul 2008 13:56:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type:content-transfer-encoding :content-disposition; bh=UMrfuON1HxOquJK3QcPgYgqGJMzFV40tE9MNUN0rqZs=; b=ocHXLk5aV6TKT/rMu5NfOcfaVdzLJOPLlF2lGv6++7Iubuuc9PojbrwmBcMfkh7DTj JrFvTbu52MowVLDw7ozMCIF+7BGHiaEXjoYIo/CBsxSWeYg1jZ4AJplSokG5BwugCVDR YQqvtoSS/sDNNFYPULtzNpolUTMLYvH/wEarU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type :content-transfer-encoding:content-disposition; b=KcC6hXX9ybIYJLXJk+kRVjLqSEnHDShID1TowIZHD1W3NmZouaVMOWfwQ+IrAF93YK 4UJJyqMNDat99tbD3JxuoAvnBf9F/LchW/ZPHAVzgbCzlZOCSsP2k4291xKNc5KFmfzP kjR299gDtfJQY54W05VGvtjL2eLe9n2ScKeM8= Received: by 10.114.95.1 with SMTP id s1mr19730120wab.13.1216153681660; Tue, 15 Jul 2008 13:28:01 -0700 (PDT) Received: by 10.115.32.18 with HTTP; Tue, 15 Jul 2008 13:28:01 -0700 (PDT) Message-ID: <4804a6670807151328y2cf30363x9b808912286ea1f5@mail.gmail.com> Date: Tue, 15 Jul 2008 16:28:01 -0400 From: Rahul To: freebsd-performance@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Subject: Real Insight on Performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jul 2008 20:56:15 -0000 Hi all, I am looking to install Unix (and/or like) system for server to run some highly computational and multi-threaded applications. My background being pretty much all Windows, I am a novice to this field. I have grasped some basics by reading some material on web and how-to books but event after extensive digging around on web for real performance numbers on various operating systems, I still haven't found anything useful. Most of the data I found were basically comparisons of operating systems running MySQL or PostgreSQL to see how many connections or simple look up queries they each can server per second sort of things. But nothing that would point to underlying operations like threading, cache-ing, time slicing, I/O, etc. Now, I must admit most of the material showed Linux having upper hand. But I am not convinced FreeBSD would be behind in almost all performance benchmarks from always hearing the legendary performance and stability characteristics of FreeBSD. Can you please shed some light on what I really should be looking for in FreeBSD to optimize it to it's best performance? Am I expecting something that is just purely not BSD's priority or philosophy, per se? Is there some material I can look over? (note: I've gone through the 7.0 Preview and Tuning documentation on FreeBSD's site already.) Thanks. From owner-freebsd-performance@FreeBSD.ORG Tue Jul 15 21:17:56 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D82451065673 for ; Tue, 15 Jul 2008 21:17:56 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outN.internet-mail-service.net (outn.internet-mail-service.net [216.240.47.237]) by mx1.freebsd.org (Postfix) with ESMTP id B98BF8FC1A for ; Tue, 15 Jul 2008 21:17:56 +0000 (UTC) (envelope-from julian@elischer.org) Received: from idiom.com (mx0.idiom.com [216.240.32.160]) by out.internet-mail-service.net (Postfix) with ESMTP id 03EE12346; Tue, 15 Jul 2008 14:04:42 -0700 (PDT) Received: from julian-mac.elischer.org (localhost [127.0.0.1]) by idiom.com (Postfix) with ESMTP id 42AC22D6027; Tue, 15 Jul 2008 14:04:41 -0700 (PDT) Message-ID: <487D1069.3030801@elischer.org> Date: Tue, 15 Jul 2008 14:02:33 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: Rahul References: <4804a6670807151328y2cf30363x9b808912286ea1f5@mail.gmail.com> In-Reply-To: <4804a6670807151328y2cf30363x9b808912286ea1f5@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org Subject: Re: Real Insight on Performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jul 2008 21:17:56 -0000 Rahul wrote: > Hi all, > > I am looking to install Unix (and/or like) system for server to run > some highly computational and multi-threaded applications. My > background being pretty much all Windows, I am a novice to this field. > I have grasped some basics by reading some material on web and how-to > books but event after extensive digging around on web for real > performance numbers on various operating systems, I still haven't > found anything useful. Most of the data I found were basically > comparisons of operating systems running MySQL or PostgreSQL to see > how many connections or simple look up queries they each can server > per second sort of things. But nothing that would point to underlying > operations like threading, cache-ing, time slicing, I/O, etc. > > Now, I must admit most of the material showed Linux having upper hand. > But I am not convinced FreeBSD would be behind in almost all > performance benchmarks from always hearing the legendary performance > and stability characteristics of FreeBSD. most Linux documents will show Linux having hte upper hand of course.. For computational stuff in my experience (Image analysis software using Feedback networks with a mix of floating and fixed point work) there was not much to choose between the various OS's because the limiting factor tends to simply be the hardware calculating throughput. If you have networking or IO as part of the equation then of course it's different. One deal that we can offer you that Linux won't is that if you are prepared to work with us, we can help you find any bottlenecks that are hitting you.. (the advantage of working with a smaller group :-) > > Can you please shed some light on what I really should be looking for > in FreeBSD to optimize it to it's best performance? Am I expecting > something that is just purely not BSD's priority or philosophy, per > se? Is there some material I can look over? (note: I've gone through > the 7.0 Preview and Tuning documentation on FreeBSD's site already.) you need to tell us more about your workload. > > > > Thanks. > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" From owner-freebsd-performance@FreeBSD.ORG Wed Jul 16 00:17:47 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0EBE9106566C for ; Wed, 16 Jul 2008 00:17:47 +0000 (UTC) (envelope-from rahulone@gmail.com) Received: from wa-out-1112.google.com (wa-out-1112.google.com [209.85.146.181]) by mx1.freebsd.org (Postfix) with ESMTP id CAA208FC15 for ; Wed, 16 Jul 2008 00:17:46 +0000 (UTC) (envelope-from rahulone@gmail.com) Received: by wa-out-1112.google.com with SMTP id j4so2902109wah.3 for ; Tue, 15 Jul 2008 17:17:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=O9AFypxxTJ4Ajk9UcMRrhwN4Cy3J2EtQpeQ2ogBFBZs=; b=jq6x9lQ5t9YXrqf8bxVD3NMEKin3uK126x2I0elfLG0z0oQPkp7NpqP2nn4+oYGuWJ ImwSKRn3BBPknPiHOYbYxY0nHwkKEcuubC8N/PXCOnDPWorV8pOV+atEM9LvjdRy4/94 15BfRguUODz6mKaLkcRXovXY7+XaYmRJuRLYQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=Y/cB//U02m2uL+6drPWi1YHp6q36ReBkJoGz423G2VTJorOKmoMKSvLR+gJN0DgZ0X 8Z+EQGEGEk4afPdeTJIBRcv4e0wwwZKeXt8d3WYXSB00z92pgYC3+jJWKDCeSDaC3Im5 hIz4uhCoUHY/ysBpW4yaSXrE92K5mL276q+I4= Received: by 10.114.132.5 with SMTP id f5mr11371722wad.201.1216167466415; Tue, 15 Jul 2008 17:17:46 -0700 (PDT) Received: by 10.115.32.18 with HTTP; Tue, 15 Jul 2008 17:17:46 -0700 (PDT) Message-ID: <4804a6670807151717r54f2889fq6c9c70afebee413d@mail.gmail.com> Date: Tue, 15 Jul 2008 20:17:46 -0400 From: Rahul To: "Julian Elischer" In-Reply-To: <487D1069.3030801@elischer.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <4804a6670807151328y2cf30363x9b808912286ea1f5@mail.gmail.com> <487D1069.3030801@elischer.org> Cc: freebsd-performance@freebsd.org Subject: Re: Real Insight on Performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jul 2008 00:17:47 -0000 Thanks for the prompt response. Believe me I am fan of FreeBSD already. Your group size and discipline is what has attracted me to FreeBSD and the great respect for such work preceding it! It's just that when it comes to work/business I have to be objective. Hence, the extensive research I am doing to find the right OS for "my purpose". I know hardware has a lot to do with floating point calculations, integer math, pointers, and data caching. I don't have any utility at the moment I can run to simply measure performance. And I don't think there will be only one utility running on there all the time. For example: I would like to run a high performance web server that can handle up to 200 connections per second and serve them with great speed. That's where the concern for multi-threaded support. Depending on the request, I may have to load (and possibly unload) dynamic modules to perform calculations, and if need be, fetch data from either DB or flat file. It could involve connecting to a process on another box to get request specific command strings. This process could run for almost 20 hours straight and OS still has to be able to keep in shape. Shewww... so there is multiple parts to this. I don't want to throw hardware as the solution for anything if it could be resolved by choosing the OS and tuning it. Because hardware only solves problem temporarily, it does not give you a true measure of your capability and thus renders any prediction of linear scaling impossible. So on and so forth. Thanks again. On Tue, Jul 15, 2008 at 5:02 PM, Julian Elischer wrote: > Rahul wrote: >> >> Hi all, >> >> I am looking to install Unix (and/or like) system for server to run >> some highly computational and multi-threaded applications. My >> background being pretty much all Windows, I am a novice to this field. >> I have grasped some basics by reading some material on web and how-to >> books but event after extensive digging around on web for real >> performance numbers on various operating systems, I still haven't >> found anything useful. Most of the data I found were basically >> comparisons of operating systems running MySQL or PostgreSQL to see >> how many connections or simple look up queries they each can server >> per second sort of things. But nothing that would point to underlying >> operations like threading, cache-ing, time slicing, I/O, etc. >> >> Now, I must admit most of the material showed Linux having upper hand. >> But I am not convinced FreeBSD would be behind in almost all >> performance benchmarks from always hearing the legendary performance >> and stability characteristics of FreeBSD. > > most Linux documents will show Linux having hte upper hand of course.. > For computational stuff in my experience (Image analysis software > using Feedback networks with a mix of floating and fixed point work) > there was not much to choose between the various OS's because the > limiting factor tends to simply be the hardware calculating throughput. > If you have networking or IO as part of the equation then > of course it's different. > > One deal that we can offer you that Linux won't is that if you are prepared > to work with us, we can help you find any bottlenecks > that are hitting you.. > (the advantage of working with a smaller group :-) > > >> >> Can you please shed some light on what I really should be looking for >> in FreeBSD to optimize it to it's best performance? Am I expecting >> something that is just purely not BSD's priority or philosophy, per >> se? Is there some material I can look over? (note: I've gone through >> the 7.0 Preview and Tuning documentation on FreeBSD's site already.) > > you need to tell us more about your workload. > >> >> >> >> Thanks. >> _______________________________________________ >> freebsd-performance@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-performance >> To unsubscribe, send any mail to >> "freebsd-performance-unsubscribe@freebsd.org" > > From owner-freebsd-performance@FreeBSD.ORG Wed Jul 16 06:21:54 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D2F881065678 for ; Wed, 16 Jul 2008 06:21:54 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from redbull.bpaserver.net (redbullneu.bpaserver.net [213.198.78.217]) by mx1.freebsd.org (Postfix) with ESMTP id 63F308FC0C for ; Wed, 16 Jul 2008 06:21:54 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from outgoing.leidinger.net (p54A5542B.dip.t-dialin.net [84.165.84.43]) by redbull.bpaserver.net (Postfix) with ESMTP id 5493C2E15B; Wed, 16 Jul 2008 08:05:45 +0200 (CEST) Received: from webmail.leidinger.net (webmail.leidinger.net [192.168.1.102]) by outgoing.leidinger.net (Postfix) with ESMTP id 7721A92E17; Wed, 16 Jul 2008 08:05:33 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=Leidinger.net; s=outgoing-alex; t=1216188333; bh=kOtvvf4Z21mRaDVV71O29NK/xHSIjGOEa Y3U/ZKf6Mw=; h=Message-ID:Date:From:To:Cc:Subject:References: In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=HxD1ChcfFlQ9aMfEeNFXc5EGbNX3RzBcjo16BlPwXVNIXlbenna/0eudsO2JjV39p Ip/MRIhvLOeQmhJ0P6AcxhUCn93hh3RkbjvfxCgegdTIrBQuLAdAMzffdpUI2S25OaT OQhs9H4i+7LQO22eKkovPT3xDMotx7RbJa5c4ofmAnqIk451j0Qnwi5dLt+EBLzkP3d Afj8hF32X0RhJj5RvYrCk2LTGtBrC4peVr3fs68hJlDW/IyKC1pdiiOPJh/4oDCMxO1 /P1GAR7w/5QHKZliWFZDqNV6O7YXzppYDXrXQOskSCnpf+4ToViMswDRC3kG1jxsHRX 92p6p5SQg== Received: (from www@localhost) by webmail.leidinger.net (8.14.2/8.13.8/Submit) id m6G65Wba011839; Wed, 16 Jul 2008 08:05:32 +0200 (CEST) (envelope-from Alexander@Leidinger.net) Received: from pslux.cec.eu.int (pslux.cec.eu.int [158.169.9.14]) by webmail.leidinger.net (Horde Framework) with HTTP; Wed, 16 Jul 2008 08:05:32 +0200 Message-ID: <20080716080532.18862vjyr477xls0@webmail.leidinger.net> X-Priority: 3 (Normal) Date: Wed, 16 Jul 2008 08:05:32 +0200 From: Alexander Leidinger To: Rahul References: <4804a6670807151328y2cf30363x9b808912286ea1f5@mail.gmail.com> <487D1069.3030801@elischer.org> <4804a6670807151717r54f2889fq6c9c70afebee413d@mail.gmail.com> In-Reply-To: <4804a6670807151717r54f2889fq6c9c70afebee413d@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: quoted-printable User-Agent: Internet Messaging Program (IMP) H3 (4.2-RC2) / FreeBSD-8.0 X-BPAnet-MailScanner-Information: Please contact the ISP for more information X-BPAnet-MailScanner: Found to be clean X-BPAnet-MailScanner-SpamCheck: not spam, ORDB-RBL, SpamAssassin (not cached, score=-14.9, required 6, BAYES_00 -15.00, DKIM_SIGNED 0.00, DKIM_VERIFIED -0.00, RDNS_DYNAMIC 0.10) X-BPAnet-MailScanner-From: alexander@leidinger.net X-Spam-Status: No Cc: freebsd-performance@freebsd.org, Julian Elischer Subject: Re: Real Insight on Performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jul 2008 06:21:54 -0000 Quoting Rahul (from Tue, 15 Jul 2008 20:17:46 -0400): > For example: I would like to run a high performance web server that > can handle up to 200 connections per second and serve them with great > speed. That's where the concern for multi-threaded support. Depending Serving 200 connections per second for static data is not hard. And =20 you don't need multi-threading for that. Multiple processes is =20 actually better than multi-threading in this regard, as you don't have =20 to do locking of filedescriptors in the the webserver over multiple =20 threads. The difference between linux and FreeBSD should be not big =20 for single CPU/core systems, but if you increase the number of =20 CPUs/cores this may be different. See the postgresql (it uses =20 processes, not threads like mysql) graphs in =20 http://people.freebsd.org/~kris/scaling/7.0%20and%20beyond.pdf to get =20 an idea about the scaling of more or less independent processes in =20 FreeBSD. > on the request, I may have to load (and possibly unload) dynamic > modules to perform calculations, and if need be, fetch data from > either DB or flat file. It could involve connecting to a process on As soon as you have calculations and/or DB accesses involved, it =20 mostly depends upon the DB optimizations ("good" tables, indexes, data =20 volume, queries, good concurrency of the DB, ...) and the computation, =20 not on the OS. So without any specific workload, we can not really =20 give recommendations (besides giving FreeBSD a try and working with us =20 if there's problem). > another box to get request specific command strings. This process > could run for almost 20 hours straight and OS still has to be able to > keep in shape. The OS doesn't care about how long a process runs. But if you talk =20 about good responsiveness of the OS while a process uses a lot of =20 memory and CPU, FreeBSD will handle it good (and from what I heard and =20 seen better than Linux, but I don't have numbers at hand). Bye, Alexander. --=20 Leela: Well, someone's in a good mode. http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID =3D B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID =3D 72077137 From owner-freebsd-performance@FreeBSD.ORG Wed Jul 16 10:30:16 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 174D81065673 for ; Wed, 16 Jul 2008 10:30:16 +0000 (UTC) (envelope-from rahulone@gmail.com) Received: from rn-out-0910.google.com (rn-out-0910.google.com [64.233.170.190]) by mx1.freebsd.org (Postfix) with ESMTP id C556E8FC08 for ; Wed, 16 Jul 2008 10:30:15 +0000 (UTC) (envelope-from rahulone@gmail.com) Received: by rn-out-0910.google.com with SMTP id j71so1987325rne.12 for ; Wed, 16 Jul 2008 03:30:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=VjVqJur7MN+CEwSj5OE3r9T4wKzA2yCcWLOSk6saBlw=; b=s0XAQvmQI6GGkA/z7NCaznw15/0O2luKkS0HKhBAy2gxQ/dyriLFY+RfCerWuqWFld ABWnAVrFQZk1zTVVRYB75dvycZ0aQaBqKmjF2LdIBSzLLNKVfDZAi7dTP2uoeA4OTr6S fTc1WdEt66pgsNY/n6K4dyNz2DShiPmOt9Y3g= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=VWUvOCu1Ddm3XBxFx2JyVLSEWiV28HI1rN9+Bl4H6AjzjBBO1j1ivUKLtpudmM012L kau9aQm386ha2Yfrec0+xuNZwa+Dlk+g8ny2UwKFT+A26iWmQn0H39irS8yGcaUmK82E wtCZnis8cKkfcWJe9gqYGL5qZoDqLlZDWQMds= Received: by 10.114.148.1 with SMTP id v1mr6583134wad.199.1216204214453; Wed, 16 Jul 2008 03:30:14 -0700 (PDT) Received: by 10.115.32.18 with HTTP; Wed, 16 Jul 2008 03:30:14 -0700 (PDT) Message-ID: <4804a6670807160330n7d0265ads547297e863dcd347@mail.gmail.com> Date: Wed, 16 Jul 2008 06:30:14 -0400 From: Rahul To: "Alexander Leidinger" In-Reply-To: <20080716080532.18862vjyr477xls0@webmail.leidinger.net> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <4804a6670807151328y2cf30363x9b808912286ea1f5@mail.gmail.com> <487D1069.3030801@elischer.org> <4804a6670807151717r54f2889fq6c9c70afebee413d@mail.gmail.com> <20080716080532.18862vjyr477xls0@webmail.leidinger.net> Cc: freebsd-performance@freebsd.org Subject: Re: Real Insight on Performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jul 2008 10:30:16 -0000 Thanks Alexander, I need multi-threaded support because I will need to share data between threads. I am not just serving static pages, I am actually performing data manipulation using data that comes in the request as well as what's in memory. The memory could grow to at least 4 GB. Actually, I would like to use at least 10GB for that process (shouldn't be a problem with 16GB RAM on 64Bit OS with Intel Quad Core - Q6600 or Q9450). Although, I do prefer to launch a thread and keep it around for when there is work to do than launching a new process for every request because launching a process is more expensive to begin with and if it has to fetch data from parent process' memory somehow, it only makes things worse. I have used MySQL in the past and it has been able to handle large volume of data without complains (I am talking about 20K rows per second with 25 fields per row). So I was hoping to use MySQL this time around. I guess I should have been more clear in my question, by querying DB and reading flat files, I was meaning to ask about I/O storage and retrieval mechanism, are there any optimizations in FreeBSD regarding I/O? By hinting at possible run time, I meant it would be running 20 hours under heavy work load. So it could possibly get far more than 200 requests per second. At peak time it could get around 2500 per second. So it would be important for OS to not crash or show flaky behavior with memory management with so much memory to maintain throughout the day. Sorry if I seem to be neat-picking but I would like to pick the right OS and stick to as the demand grows and I add more servers. I would like them all to be running same OS as hardware is likely to change more frequently. Thanks. On Wed, Jul 16, 2008 at 2:05 AM, Alexander Leidinger wrote: > Quoting Rahul (from Tue, 15 Jul 2008 20:17:46 -0400): > >> For example: I would like to run a high performance web server that >> can handle up to 200 connections per second and serve them with great >> speed. That's where the concern for multi-threaded support. Depending > > Serving 200 connections per second for static data is not hard. And you > don't need multi-threading for that. Multiple processes is actually better > than multi-threading in this regard, as you don't have to do locking of > filedescriptors in the the webserver over multiple threads. The difference > between linux and FreeBSD should be not big for single CPU/core systems, but > if you increase the number of CPUs/cores this may be different. See the > postgresql (it uses processes, not threads like mysql) graphs in > http://people.freebsd.org/~kris/scaling/7.0%20and%20beyond.pdf to get an > idea about the scaling of more or less independent processes in FreeBSD. > >> on the request, I may have to load (and possibly unload) dynamic >> modules to perform calculations, and if need be, fetch data from >> either DB or flat file. It could involve connecting to a process on > > As soon as you have calculations and/or DB accesses involved, it mostly > depends upon the DB optimizations ("good" tables, indexes, data volume, > queries, good concurrency of the DB, ...) and the computation, not on the > OS. So without any specific workload, we can not really give recommendations > (besides giving FreeBSD a try and working with us if there's problem). > >> another box to get request specific command strings. This process >> could run for almost 20 hours straight and OS still has to be able to >> keep in shape. > > The OS doesn't care about how long a process runs. But if you talk about > good responsiveness of the OS while a process uses a lot of memory and CPU, > FreeBSD will handle it good (and from what I heard and seen better than > Linux, but I don't have numbers at hand). > > Bye, > Alexander. > > -- > Leela: Well, someone's in a good mode. > > http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7 > http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 > From owner-freebsd-performance@FreeBSD.ORG Thu Jul 17 02:09:53 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A307F1065675 for ; Thu, 17 Jul 2008 02:09:53 +0000 (UTC) (envelope-from astrange@ithinksw.com) Received: from fmailhost05.isp.att.net (fmailhost05.isp.att.net [204.127.217.105]) by mx1.freebsd.org (Postfix) with ESMTP id 8BA598FC19 for ; Thu, 17 Jul 2008 02:09:53 +0000 (UTC) (envelope-from astrange@ithinksw.com) Received: from [10.0.1.4] (adsl-232-6-149.asm.bellsouth.net[74.232.6.149]) by isp.att.net (frfwmhc05) with SMTP id <20080717015622H0500efs8ge>; Thu, 17 Jul 2008 01:56:24 +0000 X-Originating-IP: [74.232.6.149] Message-Id: From: Alexander Strange To: freebsd-performance@freebsd.org Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v926) Date: Wed, 16 Jul 2008 21:56:18 -0400 X-Mailer: Apple Mail (2.926) Subject: Large number of http connections immediately dropped X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jul 2008 02:09:53 -0000 We're running a rather high-load webserver using FreeBSD 7-RELEASE/ amd64/nginx on an Intel em gigabit connection. Performance is good for our current bandwidth use (about 20Mbit and ~2000 connections/sec at the moment), but a large number of HTTP requests are being immediately dropped before getting to nginx. I see complaints about this with earlier versions of FreeBSD - http://forum.lighttpd.net/topic/171 - but no solutions. Does anyone know what could be the problem, or anything we could do about it? There are several other servers running earlier FreeBSDs on i386 which don't seem to have this problem, but I still haven't ruled out upstream hardware problems or Sandvine yet. On the server: -nginx's error log is full of "accept() failed (53: Software caused connection abort)", sometimes printing three or four at the same time. -messages is full of: Limiting open port RST response from 441 to 200 packets/sec Limiting open port RST response from 488 to 200 packets/sec Limiting open port RST response from 399 to 200 packets/sec Limiting open port RST response from 434 to 200 packets/sec Limiting open port RST response from 308 to 200 packets/sec I'm not sure if that's related or not. -sysctl.conf: net.inet.tcp.tso=1 kern.ipc.somaxconn=10240 kern.ipc.nmbclusters=65536 net.inet.tcp.sendspace=65536 net.inet.tcp.recvspace=65536 net.inet.tcp.rfc1323=1 kern.ipc.maxsockbuf=262144 net.inet.tcp.blackhole=2 net.inet.udp.blackhole=1 net.inet.tcp.msl=7500 net.inet.icmp.icmplim=400 net.inet.tcp.drop_synfin=1 net.inet.tcp.icmp_may_rst=0 net.inet.tcp.fast_finwait2_recycle=1 -netstat -m: 4677/6603/11280 mbufs in use (current/cache/total) 1017/2643/3660/65536 mbuf clusters in use (current/cache/total/max) 1017/1961 mbuf+clusters out of packet secondary zone in use (current/ cache) 9/514/523/12800 4k (page size) jumbo clusters in use (current/cache/ total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 3239K/8992K/12232K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 9204 requests for I/O initiated by sendfile 0 calls to protocol drain routines nginx is not running any accept filters. Locally, after sending an HTTP request, I get a normal connection close, then one RST with sequence 1, then another (possibly more than one) RST with sequence 2. I can post a tcpdump sequence if necessary, after I sanitize some cookies away. From owner-freebsd-performance@FreeBSD.ORG Thu Jul 17 21:04:01 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B38981065675 for ; Thu, 17 Jul 2008 21:04:01 +0000 (UTC) (envelope-from leccine@gmail.com) Received: from mail.nezz.be (nezz.be [195.228.74.74]) by mx1.freebsd.org (Postfix) with ESMTP id 2CE618FC12 for ; Thu, 17 Jul 2008 21:04:01 +0000 (UTC) (envelope-from leccine@gmail.com) Received: from mail.nezz.be (db.nezz.be [127.1.0.2]) by mail.nezz.be (Postfix) with ESMTP id ABD988B167B; Thu, 17 Jul 2008 22:41:20 +0200 (CEST) Received: by mail.nezz.be (Postfix, from userid 65534) id 91E728B1672; Thu, 17 Jul 2008 22:41:20 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.2.1 (2007-05-02) on db.nezz.be X-Spam-Level: X-Spam-Status: No, score=-0.4 required=5.0 tests=AWL,BAYES_20,RCVD_IN_PBL, RDNS_NONE autolearn=no version=3.2.1 Received: from [10.0.0.20] (unknown [78.16.27.121]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.nezz.be (Postfix) with ESMTP id A5A808B1659; Thu, 17 Jul 2008 22:41:15 +0200 (CEST) Message-ID: <487FAF68.6040200@gmail.com> Date: Thu, 17 Jul 2008 21:45:28 +0100 From: Istvan Szukacs User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV using ClamSMTP Cc: Alexander Strange Subject: Re: Large number of http connections immediately dropped X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jul 2008 21:04:01 -0000 Hi! Something to read: http://people.freebsd.org/~hmp/utilities/satbl/sysctl-net.html I have these in the sysctl.conf kern.ipc.somaxconn=4096 net.inet.tcp.recvspace=78840 net.inet.tcp.sendspace=78840 kern.ipc.shmmax=67108864 kern.ipc.shmmni=200 kern.ipc.shmseg=128 kern.ipc.semmni=70 net.local.stream.sendspace=82320 net.local.stream.recvspace=82320 net.inet.tcp.local_slowstart_flightsize=10 net.inet.tcp.nolocaltimewait=1 net.inet.tcp.hostcache.expire=3900 and the loader.conf kern.maxusers=512 kern.ipc.nmbclusters=32768 kern.ipc.maxsockets=81920 kern.ipc.maxsockbuf=1048576 net.inet.tcp.tcbhashsize=4096 net.inet.tcp.hostcache.hashsize=1024 Regards, Istvan Alexander Strange wrote: > We're running a rather high-load webserver using FreeBSD > 7-RELEASE/amd64/nginx on an Intel em gigabit connection. > Performance is good for our current bandwidth use (about 20Mbit and > ~2000 connections/sec at the moment), but a large number of HTTP > requests are being immediately dropped before getting to nginx. I see > complaints about this with earlier versions of FreeBSD - > http://forum.lighttpd.net/topic/171 - but no solutions. Does anyone > know what could be the problem, or anything we could do about it? > > There are several other servers running earlier FreeBSDs on i386 which > don't seem to have this problem, but I still haven't ruled out > upstream hardware problems or Sandvine yet. > > On the server: > -nginx's error log is full of "accept() failed (53: Software caused > connection abort)", sometimes printing three or four at the same time. > > -messages is full of: > Limiting open port RST response from 441 to 200 packets/sec > Limiting open port RST response from 488 to 200 packets/sec > Limiting open port RST response from 399 to 200 packets/sec > Limiting open port RST response from 434 to 200 packets/sec > Limiting open port RST response from 308 to 200 packets/sec > I'm not sure if that's related or not. > > -sysctl.conf: > > net.inet.tcp.tso=1 > kern.ipc.somaxconn=10240 > kern.ipc.nmbclusters=65536 > net.inet.tcp.sendspace=65536 > net.inet.tcp.recvspace=65536 > net.inet.tcp.rfc1323=1 > kern.ipc.maxsockbuf=262144 > net.inet.tcp.blackhole=2 > net.inet.udp.blackhole=1 > net.inet.tcp.msl=7500 > net.inet.icmp.icmplim=400 > net.inet.tcp.drop_synfin=1 > net.inet.tcp.icmp_may_rst=0 > net.inet.tcp.fast_finwait2_recycle=1 > > -netstat -m: > 4677/6603/11280 mbufs in use (current/cache/total) > 1017/2643/3660/65536 mbuf clusters in use (current/cache/total/max) > 1017/1961 mbuf+clusters out of packet secondary zone in use > (current/cache) > 9/514/523/12800 4k (page size) jumbo clusters in use > (current/cache/total/max) > 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > 3239K/8992K/12232K bytes allocated to network (current/cache/total) > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > 0/0/0 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 9204 requests for I/O initiated by sendfile > 0 calls to protocol drain routines > > nginx is not running any accept filters. > > Locally, after sending an HTTP request, I get a normal connection > close, then one RST with sequence 1, then another (possibly more than > one) RST with sequence 2. I can post a tcpdump sequence if necessary, > after I sanitize some cookies away. > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to > "freebsd-performance-unsubscribe@freebsd.org"