From owner-freebsd-performance@FreeBSD.ORG Sun Mar 9 01:14:17 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8B8CD1065673; Sun, 9 Mar 2008 01:14:17 +0000 (UTC) (envelope-from prvs=195486f407=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (core6.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id E1E7A8FC12; Sun, 9 Mar 2008 01:14:16 +0000 (UTC) (envelope-from prvs=195486f407=killing@multiplay.co.uk) DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=multiplay.co.uk; s=Multiplay; t=1205024516; x=1205629316; q=dns/txt; h=Received: Message-ID:From:To:Cc:References:Subject:Date:MIME-Version: Content-Type:Content-Transfer-Encoding; bh=d3c9wMJ4g3x0WW9KbgTfw JlUK26ODfmwsGZqq3KV1N8=; b=FMqZIjUGBxI+AzipN3E5T2p8gPAkfx7uqbbgu ERxqvs2QYeYV3+I9OtYykDQSBzClBLw34QQGlSZkVOMoteRW3lx4BlwHsq5TuiVj ntaNwoNnbk57vJCJl7e9FJF+SwyRyX3xNGWbV4yzPOmr+rQvxHY5QWc2UJuXVZiQ l80duw= X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-14.7 required=6.0 tests=BAYES_00, USER_IN_WHITELIST, USER_IN_WHITELIST_TO autolearn=ham version=3.1.8 Received: from r2d2 by mail1.multiplay.co.uk (MDaemon PRO v9.6.3) with ESMTP id md50005243084.msg; Sun, 09 Mar 2008 01:01:50 +0000 Message-ID: <006401c88181$25cf0e30$b6db87d4@multiplay.co.uk> From: "Steven Hartland" To: "Robert Watson" References: <056601c8814c$516c0370$b6db87d4@multiplay.co.uk> <20080308221441.E11432@fledge.watson.org> Date: Sun, 9 Mar 2008 01:01:46 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3138 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3198 X-Authenticated-Sender: Killing@multiplay.co.uk X-MDRemoteIP: 212.135.219.182 X-Return-Path: prvs=195486f407=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-Spam-Processed: mail1.multiplay.co.uk, Sun, 09 Mar 2008 01:01:51 +0000 X-MDAV-Processed: mail1.multiplay.co.uk, Sun, 09 Mar 2008 01:01:56 +0000 Cc: freebsd-performance@freebsd.org Subject: Re: rrdtool / mtr causing stalling on 7.0 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Mar 2008 01:14:17 -0000 ----- Original Message ----- From: "Robert Watson" > It looks like the attachment got lost on the way through the mailing list. > > I think the first starting point is: what sort of stall is this? Is it, for > example, all network communication stalling, all disk I/O stalling, or the > entire kernel and all processes stalling? The usual diagnostics are: > > - Does the machine stop responding to pings while stalled, and/or possibly > "catch up" all at once when it recovers? > > - If you run the following loop on the machine without any network or console > I/O, do you see gaps in time stamps: > > while (1) { > sleep 1 > date >> date.log > } > > - If you write a short C program that looks a lot like the above loop, but > logs time stamps into an in-memory buffer, and have it look for gaps in the > sequence of >3 seconds, does it run across the stall? Thanks for the ideas Robert the output from the shell script this shows significant gaps:- Sun Mar 9 00:20:33 GMT 2008 Sun Mar 9 00:20:34 GMT 2008 <== Stall Sun Mar 9 00:21:09 GMT 2008 Sun Mar 9 00:21:10 GMT 2008 ... Sun Mar 9 00:25:23 GMT 2008 Sun Mar 9 00:25:24 GMT 2008 Sun Mar 9 00:25:25 GMT 2008 Sun Mar 9 00:25:27 GMT 2008 <== Stall Sun Mar 9 00:25:53 GMT 2008 Sun Mar 9 00:25:59 GMT 2008 Sun Mar 9 00:26:00 GMT 2008 Running a ping along side shows no missed responses. Enabling lock profiling for the period changes the behaviour somewhat, producing shorter but multiple stalls. Sun Mar 9 00:30:31 GMT 2008 Sun Mar 9 00:30:32 GMT 2008 Sun Mar 9 00:30:34 GMT 2008 Sun Mar 9 00:30:35 GMT 2008 Sun Mar 9 00:30:36 GMT 2008 Sun Mar 9 00:30:37 GMT 2008 Sun Mar 9 00:30:38 GMT 2008 Sun Mar 9 00:30:41 GMT 2008 Sun Mar 9 00:30:42 GMT 2008 <== Stall Sun Mar 9 00:30:44 GMT 2008 Sun Mar 9 00:30:45 GMT 2008 <== Stall Sun Mar 9 00:30:47 GMT 2008 <== Stall Sun Mar 9 00:30:49 GMT 2008 Sun Mar 9 00:30:50 GMT 2008 <== Stall Sun Mar 9 00:30:52 GMT 2008 <== Stall Sun Mar 9 00:30:54 GMT 2008 Sun Mar 9 00:30:55 GMT 2008 Sun Mar 9 00:30:56 GMT 2008 Sun Mar 9 00:30:57 GMT 2008 <== Stall Sun Mar 9 00:31:03 GMT 2008 <== Stall Sun Mar 9 00:31:05 GMT 2008 Sun Mar 9 00:31:06 GMT 2008 <== Stall Sun Mar 9 00:31:08 GMT 2008 Sun Mar 9 00:31:09 GMT 2008 Sun Mar 9 00:31:10 GMT 2008 Sun Mar 9 00:31:11 GMT 2008 <== Stall Sun Mar 9 00:31:14 GMT 2008 Sun Mar 9 00:31:15 GMT 2008 Sun Mar 9 00:31:16 GMT 2008 <== Stall Sun Mar 9 00:31:20 GMT 2008 Sun Mar 9 00:31:21 GMT 2008 Sun Mar 9 00:31:22 GMT 2008 Using the following c code we also see stalls: #include #include #include int main( char **argv, int argc ) { time_t last = time( NULL ); while ( 1 ) { time_t now = time( NULL ); time_t diff = now - last; if ( diff >= 2 ) { fprintf( stderr, "stalled for %d seconds\n", diff ); } fprintf( stderr, ctime( &now ) ); last = now; sleep( 1 ); } exit( 0 ); } [date.log] Sun Mar 9 00:55:40 GMT 2008 Sun Mar 9 00:55:43 GMT 2008 <== Stall Sun Mar 9 00:56:11 GMT 2008 Sun Mar 9 00:56:12 GMT 2008 Sun Mar 9 00:56:13 GMT 2008 Sun Mar 9 00:56:14 GMT 2008 Sun Mar 9 00:56:15 GMT 2008 [/date.log] [timec output] Sun Mar 9 00:55:40 2008 Sun Mar 9 00:55:41 2008 Sun Mar 9 00:55:42 2008 stalled for 2 seconds Sun Mar 9 00:55:44 2008 stalled for 5 seconds Sun Mar 9 00:55:49 2008 stalled for 2 seconds Sun Mar 9 00:55:51 2008 stalled for 2 seconds Sun Mar 9 00:55:53 2008 Sun Mar 9 00:55:54 2008 Sun Mar 9 00:55:55 2008 Sun Mar 9 00:55:56 2008 Sun Mar 9 00:55:57 2008 Sun Mar 9 00:55:58 2008 Sun Mar 9 00:55:59 2008 Sun Mar 9 00:56:00 2008 Sun Mar 9 00:56:01 2008 Sun Mar 9 00:56:02 2008 Sun Mar 9 00:56:03 2008 Sun Mar 9 00:56:04 2008 Sun Mar 9 00:56:05 2008 Sun Mar 9 00:56:06 2008 Sun Mar 9 00:56:07 2008 Sun Mar 9 00:56:08 2008 Sun Mar 9 00:56:09 2008 Sun Mar 9 00:56:10 2008 Sun Mar 9 00:56:11 2008 Sun Mar 9 00:56:12 2008 Sun Mar 9 00:56:13 2008 Sun Mar 9 00:56:14 2008 Sun Mar 9 00:56:15 2008 [/timec output] As the list ate the attachment, the output from the lock profile can be found here:- ftp://ftp1.multiplay.co.uk/pub/other/freebsd-7.0-rrdtool-stall.zip Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Mon Mar 10 10:48:36 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0F084106566B for ; Mon, 10 Mar 2008 10:48:36 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id D36358FC1C for ; Mon, 10 Mar 2008 10:48:35 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1JYfYQ-0003fM-Tv for freebsd-performance@freebsd.org; Mon, 10 Mar 2008 10:48:26 +0000 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 10 Mar 2008 10:48:26 +0000 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 10 Mar 2008 10:48:26 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: Ivan Voras Date: Mon, 10 Mar 2008 11:52:27 +0100 Lines: 28 Message-ID: Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig322460C8D0E4DB3A5C359FEA" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Thunderbird 2.0.0.6 (X11/20071022) X-Enigmail-Version: 0.95.0 Sender: news Subject: pgbench results X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Mar 2008 10:48:36 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig322460C8D0E4DB3A5C359FEA Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi, Has anyone been able to replicate results from http://www.kaltenbrunner.cc/blog/index.php?/archives/21-guid.html, or get close to the performance described there on similar hardware (e.g. thousands of transactions/s) ? --------------enig322460C8D0E4DB3A5C359FEA Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFH1RLxldnAQVacBcgRAh/vAJ9EyPaINisbnG0Cro4aH3/ukiAW5ACdFDie sjn/IlSk0jQt8/M6eJzaRNY= =OGvr -----END PGP SIGNATURE----- --------------enig322460C8D0E4DB3A5C359FEA-- From owner-freebsd-performance@FreeBSD.ORG Mon Mar 10 15:35:37 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A27E5106566C for ; Mon, 10 Mar 2008 15:35:37 +0000 (UTC) (envelope-from alan.bryan@yahoo.com) Received: from web50512.mail.re2.yahoo.com (web50512.mail.re2.yahoo.com [206.190.39.241]) by mx1.freebsd.org (Postfix) with SMTP id 6A5AC8FC15 for ; Mon, 10 Mar 2008 15:35:37 +0000 (UTC) (envelope-from alan.bryan@yahoo.com) Received: (qmail 92754 invoked by uid 60001); 10 Mar 2008 15:35:35 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; b=YU0eVXxzhTg5EE8xodR3ZH9KrIG5tYf4ZEOWm/j3Zn/ylnG8vkV1p3qpMonKKb8HSGC6+IQzObyKvnvExIB3exVn7kjfKog03466CuPBrjBtnqyeSyNQ5rzCssBZYk4QFo3iIDAIzPFhLCNrmkNK9x9FiN88OJQaB3lF9YjaevU=; X-YMail-OSG: wCKcqGUVM1ntB_jJq.Y9rQ9MMz24jMbuc4nHf0KC Received: from [76.91.101.96] by web50512.mail.re2.yahoo.com via HTTP; Mon, 10 Mar 2008 08:35:35 PDT Date: Mon, 10 Mar 2008 08:35:35 -0700 (PDT) From: alan bryan To: Ivan Voras , freebsd-performance@freebsd.org In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Message-ID: <571396.91912.qm@web50512.mail.re2.yahoo.com> Cc: Subject: Re: pgbench results X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Mar 2008 15:35:37 -0000 --- Ivan Voras wrote: > Hi, > > Has anyone been able to replicate results from > http://www.kaltenbrunner.cc/blog/index.php?/archives/21-guid.html, > or > get close to the performance described there on > similar hardware (e.g. > thousands of transactions/s) ? > Here's mine for a somewhat similar setup. FreeBSD 7.0 PostgreSQL 8.3 2x Intel Xeon 2.33GHZ quad cores (8 cores total), 8GB RAM, 250GB RAID 10 (4x WD Raptor 10K drives). Non-default settings: max_connections = 200 shared_buffers = 1900MB wal_buffers = 1024kB checkpoint_segments = 192 checkpoint_timeout = 30min createdb testdb pgbench -i -s 100 testdb # pgbench -c 100 -t 100000 testdb starting vacuum...end. transaction type: TPC-B (sort of) scaling factor: 100 number of clients: 100 number of transactions per client: 100000 number of transactions actually processed: 10000000/10000000 tps = 1650.806584 (including connections establishing) tps = 1650.905036 (excluding connections establishing) So, not as high as his numbers but then I've got less RAM, one less drive spindle in my array (2 vs. 3 in performance for the raid 10 setup), SATA vs. SCSI, he's got 512MB of controller cache vs my 128MB. --Alan ____________________________________________________________________________________ Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ From owner-freebsd-performance@FreeBSD.ORG Tue Mar 11 16:10:36 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EB217106566B for ; Tue, 11 Mar 2008 16:10:36 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 5A5C88FC16 for ; Tue, 11 Mar 2008 16:10:36 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1JZ73e-0007Q6-S7 for freebsd-performance@freebsd.org; Tue, 11 Mar 2008 16:10:30 +0000 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 11 Mar 2008 16:10:30 +0000 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 11 Mar 2008 16:10:30 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: Ivan Voras Date: Tue, 11 Mar 2008 17:14:40 +0100 Lines: 78 Message-ID: References: <571396.91912.qm@web50512.mail.re2.yahoo.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig2980F56084A3056FC398C300" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Thunderbird 2.0.0.6 (X11/20071022) In-Reply-To: <571396.91912.qm@web50512.mail.re2.yahoo.com> X-Enigmail-Version: 0.95.0 Sender: news Subject: Re: pgbench results X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Mar 2008 16:10:37 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig2980F56084A3056FC398C300 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable http://www.kaltenbrunner.cc/blog/index.php?/archives/21-guid.html alan bryan wrote: > Here's mine for a somewhat similar setup. =20 > FreeBSD 7.0 PostgreSQL 8.3 > 2x Intel Xeon 2.33GHZ quad cores (8 cores total), 8GB > RAM, 250GB RAID 10 (4x WD Raptor 10K drives). >=20 > Non-default settings:=20 >=20 > =20 > max_connections =3D 200=20 > shared_buffers =3D 1900MB > wal_buffers =3D 1024kB > checkpoint_segments =3D 192 =20 > checkpoint_timeout =3D 30min=20 >=20 > createdb testdb > pgbench -i -s 100 testdb >=20 > # pgbench -c 100 -t 100000 testdb > starting vacuum...end. > transaction type: TPC-B (sort of) > scaling factor: 100 > number of clients: 100 > number of transactions per client: 100000 > number of transactions actually processed: > 10000000/10000000 > tps =3D 1650.806584 (including connections establishing) > tps =3D 1650.905036 (excluding connections establishing) >=20 > So, not as high as his numbers but then I've got less > RAM, one less drive spindle in my array (2 vs. 3 in > performance for the raid 10 setup), SATA vs. SCSI, > he's got 512MB of controller cache vs my 128MB. The thing is - I *do* have a similar setup here: HP DL370 G5, 2x4-core 1.86 GHz, 4 GB RAM, 6 drives in RAID10, 512 MB cache (can pull > 200 MB/s off the array), with all settings like in the posted link except shared_buffer=3D1900 MB, and I "only" get this: tps =3D 2834.026175 (including connections establishing) tps =3D 2839.080739 (excluding connections establishing) This is still far bellow ~~ 4500 trans/s from the link and I wonder if my results are within what I should be getting. The benchmark in the link above was done with faster CPUs (but I'm not CPU bound - at least 30% idle), but with 3 times the memory and I'm guessing more memory would help here, but I'm not sure. What's strange is that toggling synchronous_commit doesn't have a significant effect on performance (it does increase CPU idle time). With synchronous_commit=3Doff, I get: tps =3D 2886.980477 (including connections establishing) tps =3D 2891.776081 (excluding connections establishing) --------------enig2980F56084A3056FC398C300 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFH1q/wldnAQVacBcgRAniVAJwJSqrKPUshKrhhG1wXhG4QkmCEtACg03ZN IkdfVNWM5rbFP+k5XZXEmUY= =kKJZ -----END PGP SIGNATURE----- --------------enig2980F56084A3056FC398C300-- From owner-freebsd-performance@FreeBSD.ORG Wed Mar 12 00:25:57 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 20F86106566C; Wed, 12 Mar 2008 00:25:57 +0000 (UTC) (envelope-from markir@paradise.net.nz) Received: from smtp4.clear.net.nz (smtp4.clear.net.nz [203.97.37.64]) by mx1.freebsd.org (Postfix) with ESMTP id E149E8FC20; Wed, 12 Mar 2008 00:25:56 +0000 (UTC) (envelope-from markir@paradise.net.nz) Received: from zmori.markir.net (121-73-161-240.dsl.telstraclear.net [121.73.161.240]) by smtp4.clear.net.nz (CLEAR Net Mail) with ESMTP id <0JXL00ESHCHOZM00@smtp4.clear.net.nz>; Wed, 12 Mar 2008 13:10:38 +1300 (NZDT) Date: Wed, 12 Mar 2008 13:10:23 +1300 From: Mark Kirkwood In-reply-to: To: Ivan Voras Message-id: <47D71F6F.2090600@paradise.net.nz> MIME-version: 1.0 Content-type: text/plain; charset=UTF-8; format=flowed Content-transfer-encoding: 7bit References: <571396.91912.qm@web50512.mail.re2.yahoo.com> User-Agent: Thunderbird 2.0.0.9 (X11/20071203) Cc: freebsd-performance@freebsd.org Subject: Re: pgbench results X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Mar 2008 00:25:57 -0000 Ivan Voras wrote: > > The thing is - I *do* have a similar setup here: HP DL370 G5, 2x4-core > 1.86 GHz, 4 GB RAM, 6 drives in RAID10, 512 MB cache (can pull > 200 > MB/s off the array), with all settings like in the posted link except > shared_buffer=1900 MB, and I "only" get this: > > tps = 2834.026175 (including connections establishing) > tps = 2839.080739 (excluding connections establishing) > > This is still far bellow ~~ 4500 trans/s from the link and I wonder if > my results are within what I should be getting. The benchmark in the > link above was done with faster CPUs (but I'm not CPU bound - at least > 30% idle), but with 3 times the memory and I'm guessing more memory > would help here, but I'm not sure. > > What's strange is that toggling synchronous_commit doesn't have a > significant effect on performance (it does increase CPU idle time). With > synchronous_commit=off, I get: > > tps = 2886.980477 (including connections establishing) > tps = 2891.776081 (excluding connections establishing) > > The article refers to a controller with a battery backed write cache - that could easily explain the difference if you do not have one (he's paying nothing for fsync wheres you are). regards Mark From owner-freebsd-performance@FreeBSD.ORG Wed Mar 12 03:42:57 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D555F106566B; Wed, 12 Mar 2008 03:42:57 +0000 (UTC) (envelope-from markir@paradise.net.nz) Received: from smtp5.clear.net.nz (smtp5.clear.net.nz [203.97.33.68]) by mx1.freebsd.org (Postfix) with ESMTP id 9F3128FC17; Wed, 12 Mar 2008 03:42:57 +0000 (UTC) (envelope-from markir@paradise.net.nz) Received: from zmori.markir.net (121-73-161-240.dsl.telstraclear.net [121.73.161.240]) by smtp5.clear.net.nz (CLEAR Net Mail) with ESMTP id <0JXL006KZMBHC010@smtp5.clear.net.nz>; Wed, 12 Mar 2008 16:42:56 +1300 (NZDT) Date: Wed, 12 Mar 2008 16:42:39 +1300 From: Mark Kirkwood In-reply-to: <47D71F6F.2090600@paradise.net.nz> To: Ivan Voras Message-id: <47D7512F.5020006@paradise.net.nz> MIME-version: 1.0 Content-type: text/plain; charset=UTF-8; format=flowed Content-transfer-encoding: 7bit References: <571396.91912.qm@web50512.mail.re2.yahoo.com> <47D71F6F.2090600@paradise.net.nz> User-Agent: Thunderbird 2.0.0.9 (X11/20071203) Cc: freebsd-performance@freebsd.org Subject: Re: pgbench results X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Mar 2008 03:42:58 -0000 Mark Kirkwood wrote: > Ivan Voras wrote: >> >> The thing is - I *do* have a similar setup here: HP DL370 G5, 2x4-core >> 1.86 GHz, 4 GB RAM, 6 drives in RAID10, 512 MB cache (can pull > 200 >> MB/s off the array), with all settings like in the posted link except >> shared_buffer=1900 MB, and I "only" get this: >> >> tps = 2834.026175 (including connections establishing) >> tps = 2839.080739 (excluding connections establishing) >> >> This is still far bellow ~~ 4500 trans/s from the link and I wonder if >> my results are within what I should be getting. The benchmark in the >> link above was done with faster CPUs (but I'm not CPU bound - at least >> 30% idle), but with 3 times the memory and I'm guessing more memory >> would help here, but I'm not sure. >> >> What's strange is that toggling synchronous_commit doesn't have a >> significant effect on performance (it does increase CPU idle time). With >> synchronous_commit=off, I get: >> >> tps = 2886.980477 (including connections establishing) >> tps = 2891.776081 (excluding connections establishing) >> >> > > The article refers to a controller with a battery backed write cache - > that could easily explain the difference if you do not have one (he's > paying nothing for fsync wheres you are). > Hmm - somehow read right past the bit where you say you have a 512MB cache - sorry! However, worth checking it is set to write-back rather than write-through. Cheers Mark From owner-freebsd-performance@FreeBSD.ORG Wed Mar 12 10:20:02 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0ADB2106566B for ; Wed, 12 Mar 2008 10:20:02 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: from wf-out-1314.google.com (wf-out-1314.google.com [209.85.200.170]) by mx1.freebsd.org (Postfix) with ESMTP id CCA2D8FC21 for ; Wed, 12 Mar 2008 10:20:01 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: by wf-out-1314.google.com with SMTP id 25so2821510wfa.7 for ; Wed, 12 Mar 2008 03:20:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=IaoYYjex8ygXSGu1iyagjjeP0mA7gQfo68D6bMNAKeE=; b=O9kmiZ9pJx28XKBRU3rEKhwtm8S6tH2dW8uUr4mFaPqPiM/Mt8v5u4gaDj8JW+8cpqkiLztKptMfKyvBS6ij7gJqZN4f64b1X+Hq6ZwKImcWdIcrJivmTLSOre1W5vcOdomzVDP4QsdXxt3Xax99LiORHGU88kMSPn36l/1tlyU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=RlsPBsGPGKJIuSE8/lIfhHaezpNDWupHZPVsYUZj0fchYDmVNarOMIcduF6HBSP4dGj7KUwJAWSUDvz88hO/kfyOVWyh3GToUKYBIoWjzAS0wTjWiB9ehGHMHBcuaqM/dQ1a+SS7/wd++Jmvebj9colmmy08q6M1t+hdTS/dph8= Received: by 10.140.142.6 with SMTP id p6mr4951417rvd.224.1205315742317; Wed, 12 Mar 2008 02:55:42 -0700 (PDT) Received: by 10.141.212.1 with HTTP; Wed, 12 Mar 2008 02:55:42 -0700 (PDT) Message-ID: <9bbcef730803120255w531636d5h642e407aa1881b6a@mail.gmail.com> Date: Wed, 12 Mar 2008 10:55:42 +0100 From: "Ivan Voras" Sender: ivoras@gmail.com To: "Mark Kirkwood" In-Reply-To: <47D7512F.5020006@paradise.net.nz> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <571396.91912.qm@web50512.mail.re2.yahoo.com> <47D71F6F.2090600@paradise.net.nz> <47D7512F.5020006@paradise.net.nz> X-Google-Sender-Auth: 0b246e4b4937d648 Cc: freebsd-performance@freebsd.org Subject: Re: pgbench results X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Mar 2008 10:20:02 -0000 On 12/03/2008, Mark Kirkwood wrote: > Hmm - somehow read right past the bit where you say you have a 512MB > cache - sorry! However, worth checking it is set to write-back rather > than write-through. As far as I can see it is set to write-through (though the HP's array configuration utility isn't explicit about it, everything performance-wise than can be turned on is turned on, including write cache). From owner-freebsd-performance@FreeBSD.ORG Thu Mar 13 00:52:29 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4E1EC106566C; Thu, 13 Mar 2008 00:52:29 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from webaccess-cl.virtdom.com (webaccess-cl.virtdom.com [216.240.101.25]) by mx1.freebsd.org (Postfix) with ESMTP id 1DED38FC13; Thu, 13 Mar 2008 00:52:29 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from [192.168.1.107] (cpe-24-94-75-93.hawaii.res.rr.com [24.94.75.93]) (authenticated bits=0) by webaccess-cl.virtdom.com (8.13.6/8.13.6) with ESMTP id m2D0qLNC007992; Wed, 12 Mar 2008 20:52:27 -0400 (EDT) (envelope-from jroberson@chesapeake.net) Date: Wed, 12 Mar 2008 14:53:24 -1000 (HST) From: Jeff Roberson X-X-Sender: jroberson@desktop To: Ivan Voras In-Reply-To: <9bbcef730803120255w531636d5h642e407aa1881b6a@mail.gmail.com> Message-ID: <20080312145307.P1091@desktop> References: <571396.91912.qm@web50512.mail.re2.yahoo.com> <47D71F6F.2090600@paradise.net.nz> <47D7512F.5020006@paradise.net.nz> <9bbcef730803120255w531636d5h642e407aa1881b6a@mail.gmail.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-performance@freebsd.org, Mark Kirkwood Subject: Re: pgbench results X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Mar 2008 00:52:29 -0000 On Wed, 12 Mar 2008, Ivan Voras wrote: > On 12/03/2008, Mark Kirkwood wrote: > >> Hmm - somehow read right past the bit where you say you have a 512MB >> cache - sorry! However, worth checking it is set to write-back rather >> than write-through. > > As far as I can see it is set to write-through (though the HP's array > configuration utility isn't explicit about it, everything > performance-wise than can be turned on is turned on, including write > cache). What kernel are you running? > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > From owner-freebsd-performance@FreeBSD.ORG Thu Mar 13 09:55:41 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 46B661065673 for ; Thu, 13 Mar 2008 09:55:41 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id F0B7F8FC24 for ; Thu, 13 Mar 2008 09:55:40 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1JZk9x-00053s-Jp for freebsd-performance@freebsd.org; Thu, 13 Mar 2008 09:55:37 +0000 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 13 Mar 2008 09:55:37 +0000 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 13 Mar 2008 09:55:37 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: Ivan Voras Date: Thu, 13 Mar 2008 11:00:05 +0100 Lines: 37 Message-ID: References: <571396.91912.qm@web50512.mail.re2.yahoo.com> <47D71F6F.2090600@paradise.net.nz> <47D7512F.5020006@paradise.net.nz> <9bbcef730803120255w531636d5h642e407aa1881b6a@mail.gmail.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigBECE8E4D46DB1FB4BC720069" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Thunderbird 2.0.0.6 (X11/20071022) In-Reply-To: <9bbcef730803120255w531636d5h642e407aa1881b6a@mail.gmail.com> X-Enigmail-Version: 0.95.0 Sender: news Subject: Re: pgbench results X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Mar 2008 09:55:41 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigBECE8E4D46DB1FB4BC720069 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Ivan Voras wrote: > On 12/03/2008, Mark Kirkwood wrote: >=20 >> Hmm - somehow read right past the bit where you say you have a 512MB >> cache - sorry! However, worth checking it is set to write-back rather= >> than write-through. >=20 > As far as I can see it is set to write-through (though the HP's array Sorry, this should be "it is NOT set to write-through" > configuration utility isn't explicit about it, everything > performance-wise than can be turned on is turned on, including write > cache). --------------enigBECE8E4D46DB1FB4BC720069 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFH2PslldnAQVacBcgRAoW1AJ0fwO3oOBDu8GV7Ou+mR2rQvI/C2QCfeTEd i1Z8o81lPn6UF6QD6wMSoSI= =D36p -----END PGP SIGNATURE----- --------------enigBECE8E4D46DB1FB4BC720069-- From owner-freebsd-performance@FreeBSD.ORG Thu Mar 13 09:55:59 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2CF091065670 for ; Thu, 13 Mar 2008 09:55:59 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: from rv-out-0910.google.com (rv-out-0910.google.com [209.85.198.186]) by mx1.freebsd.org (Postfix) with ESMTP id E1BC78FC29 for ; Thu, 13 Mar 2008 09:55:58 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: by rv-out-0910.google.com with SMTP id g13so2247015rvb.43 for ; Thu, 13 Mar 2008 02:55:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=iaslSQ+3/fbmW9ca9NJ4Q237z9kSpJuS5kmfn4LX2OA=; b=IFROE3248+/sAmHmOnNi0rJMvi/yTU94OLxK8R918/m+YhU9vbTE46GhS1N9+admeKGon5EJR8Nzg/Ne4BxHQpwhAclIwXyQCZhq55cDgOVbozUDP7F07+1SHT45Z0tnkO12lQCVCq6vfAmwVd57xcB097hl7soQbS4v0S0Tb+o= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=WYnbDWFnvc4UzvNHuC8K+txomXh5rcp2wE/931vbfzsKkYFj3exd1fQ9OBnKE2Q0ByEpQnJ2YJQbo9AzBqiYviOAk2gK0002VaqtjiKZN6HwMF2v68yunzGbX4/BJN/uwOljHrAejZTk7uB0/GjcWgQGmFkvQf2AtYcQxGEfFt4= Received: by 10.141.75.6 with SMTP id c6mr5048007rvl.286.1205402157473; Thu, 13 Mar 2008 02:55:57 -0700 (PDT) Received: by 10.141.212.1 with HTTP; Thu, 13 Mar 2008 02:55:57 -0700 (PDT) Message-ID: <9bbcef730803130255p68d35240pfc627fe5acd5b659@mail.gmail.com> Date: Thu, 13 Mar 2008 10:55:57 +0100 From: "Ivan Voras" Sender: ivoras@gmail.com To: "Jeff Roberson" In-Reply-To: <20080312145307.P1091@desktop> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <571396.91912.qm@web50512.mail.re2.yahoo.com> <47D71F6F.2090600@paradise.net.nz> <47D7512F.5020006@paradise.net.nz> <9bbcef730803120255w531636d5h642e407aa1881b6a@mail.gmail.com> <20080312145307.P1091@desktop> X-Google-Sender-Auth: 5ca2d906fff21930 Cc: freebsd-performance@freebsd.org Subject: Re: pgbench results X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Mar 2008 09:55:59 -0000 On 13/03/2008, Jeff Roberson wrote: > > On Wed, 12 Mar 2008, Ivan Voras wrote: > > > On 12/03/2008, Mark Kirkwood wrote: > > > >> Hmm - somehow read right past the bit where you say you have a 512MB > >> cache - sorry! However, worth checking it is set to write-back rather > >> than write-through. > > > > As far as I can see it is set to write-through (though the HP's array > > configuration utility isn't explicit about it, everything > > performance-wise than can be turned on is turned on, including write > > cache). > > What kernel are you running? 7-STABLE since Feb 29, amd64+ULE. From owner-freebsd-performance@FreeBSD.ORG Fri Mar 14 11:12:26 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6F56F1065670 for ; Fri, 14 Mar 2008 11:12:25 +0000 (UTC) (envelope-from hugoboy@inbox.lv) Received: from shark2.inbox.lv (shark2.inbox.lv [89.111.3.82]) by mx1.freebsd.org (Postfix) with ESMTP id 5609D8FC2A for ; Fri, 14 Mar 2008 11:12:24 +0000 (UTC) (envelope-from hugoboy@inbox.lv) Received: by shark2.inbox.lv (Postfix, from userid 1000) id 1AD2B3D90; Fri, 14 Mar 2008 12:51:51 +0200 (EET) Received: from localhost (w2 [10.0.1.151]) by shark2-plain-b64d2.inbox.lv (Postfix) with ESMTP id E98063EAD for ; Fri, 14 Mar 2008 12:51:50 +0200 (EET) Received: from 195.244.154.200 ( [195.244.154.200]) as user hugoboy@10.0.1.1 by www.inbox.lv with HTTP; Fri, 14 Mar 2008 12:51:50 +0200 X-LOCAL: 1 X-Compose: web=www.inbox.lv, node=w2, l=lv, prefs=HTML, sess=unset, fck=Compatible, compose=HTML X-REMOTE-ADDR: 195.244.154.200 X-HTTP-USER-AGENT: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.12) Gecko/20080201 Firefox/2.0.0.12 Message-ID: <1205491910.47da58c6ecb6a@www.inbox.lv> Date: Fri, 14 Mar 2008 12:51:50 +0200 From: hugoboy@inbox.lv To: freebsd-performance@freebsd.org User-Agent: inbox.lv Webmail X-Cyrus-Size: 19051 MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: FreeBSD 7.0 bridge tuning X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Mar 2008 11:12:26 -0000 Hello!=0A=0AI'm trying to tune FreeBSD 7.0 bridge.=0A=0AEnvironment:=0AServ= er - 2 x Xeon 3GHz, 2 x Gb LAN(em driver) + 1 LAN for management,=0A1GB RAM= .=0ATesters -2 x Sunrise Telecom 100Mbit Ethernet testers for traffic=0Agen= eration.=0A=0AWhat I have intended to achieve is to substitute proprietary = traffic=0Ashaper Allot with FreeBSD traffic shaper(Bridge + PF + ALTQ).=0AT= he minimum task is to make FreeBSD shaper to perform perfectly with=0A100Mb= it traffic in all spectrum of packet lengths (from 64 bytes to=0Aat least 1= 518 bytes)=0A=0AThe situation now:=0Awith pf turned off - there is no probl= em, bridge throughput is=0A100Mbit/s no packet loss (starting from 64 byte = packets)=0A=0AWith pf on I have statistics:=0Apacket lengt -> Mbit/s withou= t packet loss=0A64 -> 46=0A100 -> 66=0A150 -> 94=0A>200 -> 100=0A=0ALower c= onfiguration of kernel/sysctl is displayed.=0A=0AI don't know what else can= I tune?=0A=0AIt seems to me that bottleneck is somewhere around pf/kernel = buffers=0Aof packet headers. I read somewhere that in bridging packet paylo= ad=0Adoes not travel through all stack - just header is evaluated.=0AIn cas= e of 64 byte packets in the same time unit there are more=0Apackets for the= same bandwith on interfaces and as plain layer2=0Abridge performs 100Mbit/= s with no problem=0Athe problem is above layer2 :)=0A=0Abtw: kern.polling.e= nable=3D1 does not help - at packetlength 64 bytes=0Aperformance is 2x wors= e than with interrupts.=0Akernel:=0A---------------------------=0A=0Acpu = I686_CPU=0Aident ALLOT =0A=0A# To statically compile = in device wiring instead of=0A/boot/device.hints=0A#hints "GENERIC= .hints" # Default places to look for=0Adevices.=0A=0Amakeoptions = DEBUG=3D-g # Build kernel with gdb(1)=0Adebug symbols=0A= =0Aoptions SCHED_ULE # ULE scheduler=0A#options = SCHED_4BSD # 4BSD scheduler=0Aoptions PREEMPTION = # Enable kernel thread=0Apreemption=0Aoptions INET = # InterNETworking=0A#options INET6 # = IPv6 communications=0Aprotocols=0A#options SCTP #= Stream Control Transmission=0AProtocol=0Aoptions FFS = # Berkeley Fast Filesystem=0Aoptions SOFTUPDATES = # Enable FFS soft updates=0Asupport=0Aoptions UFS_ACL = # Support for access control=0Alists=0Aoptions UFS_DIRHASH = # Improve performance on big=0Adirectories=0Aoptions UFS_GJ= OURNAL # Enable gjournal-based UFS=0Ajournaling=0Aoptions = MD_ROOT # MD is a potential root=0Adevice=0Aoptions = NFSCLIENT # Network Filesystem Client=0Aoptions NF= SSERVER # Network Filesystem Server=0Aoptions NFS_ROO= T # NFS usable as /, requires=0ANFSCLIENT=0Aoptions = MSDOSFS # MSDOS Filesystem=0Aoptions CD9660 = # ISO 9660 Filesystem=0Aoptions PROCFS #= Process filesystem=0A(requires PSEUDOFS)=0Aoptions PSEUDOFS = # Pseudo-filesystem framework=0Aoptions GEOM_PART_GPT = # GUID Partition Tables.=0Aoptions GEOM_LABEL # = Provides labelization=0Aoptions COMPAT_43TTY # BSD 4.3 T= TY compat [KEEP=0ATHIS!]=0Aoptions COMPAT_FREEBSD4 # Compat= ible with FreeBSD4=0Aoptions COMPAT_FREEBSD5 # Compatible w= ith FreeBSD5=0Aoptions COMPAT_FREEBSD6 # Compatible with Fr= eeBSD6=0Aoptions SCSI_DELAY=3D5000 # Delay (in ms) before= =0Aprobing SCSI=0Aoptions KTRACE # ktrace(1) suppo= rt=0Aoptions SYSVSHM # SYSV-style shared memory=0Ao= ptions SYSVMSG # SYSV-style message queues=0Aoption= s SYSVSEM # SYSV-style semaphores=0Aoptions = _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B=0Areal-time extensions=0Aopti= ons KBD_INSTALL_CDEV # install a CDEV entry in=0A/dev=0Aopti= ons ADAPTIVE_GIANT # Giant mutex is adaptive.=0Aoptions = STOP_NMI # Stop CPUS using NMI instead=0Aof IPI=0Aopti= ons AUDIT # Security event auditing=0A=0Aoptions = ALTQ=0Aoptions ALTQ_CBQ=0Aoptions ALTQ_RED=0Aoptions ALTQ_RIO=0Aoptions ALT= Q_HFSC=0Aoptions ALTQ_CDNR=0Aoptions ALTQ_PRIQ=0Aoptions ALTQ_NOPCC=0Aoptio= ns HZ=3D1000=0Aoptions DEVICE_POLLING=0Aoptions IPSTEALTH=0Aoptions ZERO_CO= PY_SOCKETS=0Aoptions MPTABLE_FORCE_HTT # Enable HTT CPUs with the MP = Table=0Aoptions IPI_PREEMPTION=0A=0A# To make an SMP kernel, the next two l= ines are needed=0Aoptions SMP # Symmetric Multi= Processor=0AKernel=0Adevice apic # I/O APIC=0A-= -------------------------------=0A=0A/etc/sysctl.conf=0A#kern.polling.enabl= e=3D1=0Akern.ipc.nmbcluster=3D32768=0Akern.ipc.maxsockbufs=3D2097152=0Akern= .ipc.somaxconn=3D8192=0Akern.maxfiles=3D65536=0Akern.maxfilesperproc=3D3276= 8=0Anet.inet.tcp.delayed_ack=3D0=0Anet.inet.tcp.sendspace=3D65535=0Anet.ine= t.udp.recvspace=3D65535=0Anet.inet.udp.maxdgram=3D57344=0Anet.local.stream.= recvspace=3D65535=0Anet.local.stream.sendspace=3D65535=0Akern.polling.user_= frac=3D20=0Anet.isr.direct=3D0=0Anet.inet.ip.forwarding=3D1=0A-------------= ------------------=0A=0AP.S. I tried pfSense, but as we have used Allot bef= ore - we need to=0Asee queue statistics in graphs per queue, pfSense just o= ffers=0Anumbers..=0ASeems to me that pFsense is good for many things but no= t for=0Abridge+traffic shapeing - correct me if I'm wrong.=0A=0ABest regard= s,=0AUgis=0A=0A From owner-freebsd-performance@FreeBSD.ORG Fri Mar 14 12:56:05 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5F5791065672 for ; Fri, 14 Mar 2008 12:56:05 +0000 (UTC) (envelope-from hugoboy@inbox.lv) Received: from shark4.inbox.lv (shark4.inbox.lv [89.111.3.84]) by mx1.freebsd.org (Postfix) with ESMTP id A1CA28FC29 for ; Fri, 14 Mar 2008 12:56:03 +0000 (UTC) (envelope-from hugoboy@inbox.lv) Received: by shark4.inbox.lv (Postfix, from userid 1000) id C6C981A332; Fri, 14 Mar 2008 14:56:01 +0200 (EET) Received: from localhost (w2 [10.0.1.151]) by shark4-plain-b64d2.inbox.lv (Postfix) with ESMTP id A2A031A05F for ; Fri, 14 Mar 2008 14:56:01 +0200 (EET) Received: from 195.244.154.200 ( [195.244.154.200]) as user hugoboy@10.0.1.1 by www.inbox.lv with HTTP; Fri, 14 Mar 2008 14:56:01 +0200 X-LOCAL: 1 X-Compose: web=www.inbox.lv, node=w2, l=lv, prefs=HTML, sess=unset, fck=Compatible, compose=HTML X-REMOTE-ADDR: 195.244.154.200 X-HTTP-USER-AGENT: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.12) Gecko/20080201 Firefox/2.0.0.12 Message-ID: <1205499361.47da75e1ac733@www.inbox.lv> Date: Fri, 14 Mar 2008 14:56:01 +0200 From: hugoboy@inbox.lv To: freebsd-performance@freebsd.org User-Agent: inbox.lv Webmail X-Cyrus-Size: 17751 MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: FreeBSD 7.0 bridge tuning X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Mar 2008 12:56:05 -0000 I chose pf+altq for traffic shaper solution, because it seems to=0Abetter m= atch my needs. I use ipfw where FW is needed, but from the=0Apoint of easy = administration pf+altq is better for traffic shaper. =0ASo I have not teste= d shaping performance with ipfw as I have chosen=0Apf this time. If it will= turn out that it is not possible to achieve=0Agood enough results this way= - I'll try ipfw+dummynet.=0AServer is i386 based.=0A=0AStill I am quite su= re that it is possible to tune this configuration,=0Abut need to find bottl= eneck...=0A=0AUgis=0A>Hi,=0A>Just for my information, what performance if y= ou replace pf to ipfw=0A?=0A>and what freebsd v7.0 version ? i386 or amd64 = ?=0A>Regards=0A>Rmkml=0A On Fri, 14 Mar 2008, hugoboy@inbox.lv wrote:=0A=0A= > Date: Fri, 14 Mar 2008 12:51:50 +0200=0A> From: hugoboy@inbox.lv=0A> To: = freebsd-performance@freebsd.org=0A> Subject: FreeBSD 7.0 bridge tuning=0A> = =0A> Hello!=0A>=0A> I'm trying to tune FreeBSD 7.0 bridge.=0A>=0A> Environm= ent:=0A> Server - 2 x Xeon 3GHz, 2 x Gb LAN(em driver) + 1 LAN for=0Amanage= ment,=0A> 1GB RAM.=0A> Testers -2 x Sunrise Telecom 100Mbit Ethernet tester= s for traffic=0A> generation.=0A>=0A> What I have intended to achieve is to= substitute proprietary=0Atraffic=0A> shaper Allot with FreeBSD traffic sha= per(Bridge + PF + ALTQ).=0A> The minimum task is to make FreeBSD shaper to = perform perfectly=0Awith=0A> 100Mbit traffic in all spectrum of packet leng= ths (from 64 bytes to=0A> at least 1518 bytes)=0A>=0A> The situation now:= =0A> with pf turned off - there is no problem, bridge throughput is=0A> 100= Mbit/s no packet loss (starting from 64 byte packets)=0A>=0A> With pf on I = have statistics:=0A> packet lengt -> Mbit/s without packet loss=0A> 64 -> 4= 6=0A> 100 -> 66=0A> 150 -> 94=0A>> 200 -> 100=0A>=0A> Lower configuration o= f kernel/sysctl is displayed.=0A>=0A> I don't know what else can I tune?=0A= >=0A> It seems to me that bottleneck is somewhere around pf/kernel=0Abuffer= s=0A> of packet headers. I read somewhere that in bridging packet payload= =0A> does not travel through all stack - just header is evaluated.=0A> In c= ase of 64 byte packets in the same time unit there are more=0A> packets for= the same bandwith on interfaces and as plain layer2=0A> bridge performs 10= 0Mbit/s with no problem=0A> the problem is above layer2 :)=0A>=0A> btw: ker= n.polling.enable=3D1 does not help - at packetlength 64 bytes=0A> performan= ce is 2x worse than with interrupts.=0A> kernel:=0A> ----------------------= -----=0A>=0A> cpu I686_CPU=0A> ident ALLOT =0A>=0A> #= To statically compile in device wiring instead of=0A> /boot/device.hints= =0A> #hints "GENERIC.hints" # Default places to look=0Afor= =0A> devices.=0A>=0A> makeoptions DEBUG=3D-g # Build ker= nel with gdb(1)=0A> debug symbols=0A>=0A> options SCHED_ULE = # ULE scheduler=0A> #options SCHED_4BSD # 4BSD s= cheduler=0A> options PREEMPTION # Enable kernel thread= =0A> preemption=0A> options INET # InterNETworki= ng=0A> #options INET6 # IPv6 communications=0A> pr= otocols=0A> #options SCTP # Stream Control=0ATran= smission=0A> Protocol=0A> options FFS # Berkele= y Fast Filesystem=0A> options SOFTUPDATES # Enable FFS = soft updates=0A> support=0A> options UFS_ACL # Supp= ort for access=0Acontrol=0A> lists=0A> options UFS_DIRHASH = # Improve performance on=0Abig=0A> directories=0A> options UFS_= GJOURNAL # Enable gjournal-based UFS=0A> journaling=0A> options = MD_ROOT # MD is a potential root=0A> device=0A> opt= ions NFSCLIENT # Network Filesystem Client=0A> option= s NFSSERVER # Network Filesystem Server=0A> options = NFS_ROOT # NFS usable as /, requires=0A> NFSCLIENT=0A= > options MSDOSFS # MSDOS Filesystem=0A> options = CD9660 # ISO 9660 Filesystem=0A> options PRO= CFS # Process filesystem=0A> (requires PSEUDOFS)=0A> optio= ns PSEUDOFS # Pseudo-filesystem=0Aframework=0A> opti= ons GEOM_PART_GPT # GUID Partition Tables.=0A> options = GEOM_LABEL # Provides labelization=0A> options C= OMPAT_43TTY # BSD 4.3 TTY compat [KEEP=0A> THIS!]=0A> options = COMPAT_FREEBSD4 # Compatible with FreeBSD4=0A> options = COMPAT_FREEBSD5 # Compatible with FreeBSD5=0A> options CO= MPAT_FREEBSD6 # Compatible with FreeBSD6=0A> options SCSI_D= ELAY=3D5000 # Delay (in ms) before=0A> probing SCSI=0A> options = KTRACE # ktrace(1) support=0A> options SYSVSH= M # SYSV-style shared memory=0A> options SYSVMSG = # SYSV-style message queues=0A> options SYSVSEM = # SYSV-style semaphores=0A> options _KPOSIX_PRIORITY_SCH= EDULING # POSIX P1003_1B=0A> real-time extensions=0A> options KBD_I= NSTALL_CDEV # install a CDEV entry in=0A> /dev=0A> options A= DAPTIVE_GIANT # Giant mutex is adaptive.=0A> options STOP_= NMI # Stop CPUS using NMI=0Ainstead=0A> of IPI=0A> options = AUDIT # Security event auditing=0A>=0A> options AL= TQ=0A> options ALTQ_CBQ=0A> options ALTQ_RED=0A> options ALTQ_RIO=0A> optio= ns ALTQ_HFSC=0A> options ALTQ_CDNR=0A> options ALTQ_PRIQ=0A> options ALTQ_N= OPCC=0A> options HZ=3D1000=0A> options DEVICE_POLLING=0A> options IPSTEALTH= =0A> options ZERO_COPY_SOCKETS=0A> options MPTABLE_FORCE_HTT # Enable= HTT CPUs with the MP Table=0A> options IPI_PREEMPTION=0A>=0A> # To make an= SMP kernel, the next two lines are needed=0A> options SMP = # Symmetric MultiProcessor=0A> Kernel=0A> device apic = # I/O APIC=0A> --------------------------------=0A>=0A> = /etc/sysctl.conf=0A> #kern.polling.enable=3D1=0A> kern.ipc.nmbcluster=3D327= 68=0A> kern.ipc.maxsockbufs=3D2097152=0A> kern.ipc.somaxconn=3D8192=0A> ker= n.maxfiles=3D65536=0A> kern.maxfilesperproc=3D32768=0A> net.inet.tcp.delaye= d_ack=3D0=0A> net.inet.tcp.sendspace=3D65535=0A> net.inet.udp.recvspace=3D6= 5535=0A> net.inet.udp.maxdgram=3D57344=0A> net.local.stream.recvspace=3D655= 35=0A> net.local.stream.sendspace=3D65535=0A> kern.polling.user_frac=3D20= =0A> net.isr.direct=3D0=0A> net.inet.ip.forwarding=3D1=0A> ----------------= ---------------=0A>=0A> P.S. I tried pfSense, but as we have used Allot bef= ore - we need to=0A> see queue statistics in graphs per queue, pfSense just= offers=0A> numbers..=0A> Seems to me that pFsense is good for many things = but not for=0A> bridge+traffic shapeing - correct me if I'm wrong.=0A>=0A> = Best regards,=0A> Ugis