From owner-freebsd-performance@FreeBSD.ORG Sun May 29 15:43:55 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D3C3E16A41C for ; Sun, 29 May 2005 15:43:55 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [204.156.12.53]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9C66E43D49 for ; Sun, 29 May 2005 15:43:55 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by cyrus.watson.org (Postfix) with ESMTP id 3A3DE46B2A; Sun, 29 May 2005 11:43:55 -0400 (EDT) Date: Sun, 29 May 2005 16:44:14 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: "Matthew D. Fuller" In-Reply-To: <20050528221027.GB47448@over-yonder.net> Message-ID: <20050529164207.A52379@fledge.watson.org> References: <4297E2C4.1030505@roq.com> <20050528221027.GB47448@over-yonder.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Michael VInce , freebsd-performance@freebsd.org Subject: Re: High usage of mbufs X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 May 2005 15:43:55 -0000 On Sat, 28 May 2005, Matthew D. Fuller wrote: > On Sat, May 28, 2005 at 01:17:24PM +1000 I heard the voice of > Michael VInce, and lo! it spake thus: >> On one of my web servers I have a really high usage of mbuf clusters in >> use on a web server that does about 3million hits a day. >> 4294914731/262144 mbuf clusters in use (current/max) > > No it doesn't (have really high usage, that is). It just has messed up > statistics, which is standard on SMP machines; see the archives. I have two sets of patches that correct this problem: - One uses atomic operations to synchronize access to the mbuf-related counters instead of performing non-atomic updates, which results in races. - One uses per-cpu statistics in the mbuf allocator, and critical sections to synchronize updates. Both encounter a several percent performance hit in packet-intensive workloads, so I've been reluctant to commit either. However, most of the information required for mbuf statistics is already gathered by UMA, which is now used as the back-end for the mbuf allocator. As such, the better answer is to use the UMA statistics and then see if we still need the mbuf layer stats. I haven't had a chance to investigate this in detail yet, but I'd like to get something in before 6.0. Robert N M Watson From owner-freebsd-performance@FreeBSD.ORG Mon May 30 16:01:03 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8C04316A41F for ; Mon, 30 May 2005 16:01:03 +0000 (GMT) (envelope-from tuliogs@pgt.mpt.gov.br) Received: from mitra.mpt.gov.br (mail.pgt.mpt.gov.br [200.157.62.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id E9AE243D1D for ; Mon, 30 May 2005 16:01:01 +0000 (GMT) (envelope-from tuliogs@pgt.mpt.gov.br) Received: from [10.0.0.136] (516e.pgt.mpt.gov.br [10.0.0.136]) by mail.pgt.mpt.gov.br (8.13.1/8.13.1) with ESMTP id j4UG0wnc096423 for ; Mon, 30 May 2005 13:00:58 -0300 (BRST) (envelope-from tuliogs@pgt.mpt.gov.br) Message-ID: <429B38C7.5040405@pgt.mpt.gov.br> Date: Mon, 30 May 2005 13:01:11 -0300 From: =?ISO-8859-1?Q?Tulio_Guimar=E3es_da_Silva?= User-Agent: Mozilla Thunderbird 1.0 (Windows/20041206) X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <428CF20F.50607@pgt.mpt.gov.br> <428D0D79.7010506@rcn.com> <428DFF35.1000409@pgt.mpt.gov.br> In-Reply-To: <428DFF35.1000409@pgt.mpt.gov.br> Content-Type: multipart/mixed; boundary="------------090500060903070709060002" X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: rmt as a bottleneck - Was: Weird behaviour of AIT-3 and (g)tar X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 May 2005 16:01:03 -0000 --------------090500060903070709060002 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by mail.pgt.mpt.gov.br id j4UG0wnc096423 Updates to an old problem. I=B4m splitting the original thread in 2,=20 since there are 2 different questions, one of which I guess is for=20 -hardware and the other, for -performance, then I=B4m x-posting this one. Here are other things I=B4ve been experimenting with... first, I=B4ll=20 present the machines: - Omega: HP ML 110, 512MB RAM, 4x80GB 7200rpm SATA in RAID5 via=20 HighPoint RR8120A, AIT-3 tape via Adaptec Ultra160; - Theta: HP DL 380, 1GB RAM, 6x72.8GB@10krpm SCSI in RAID5 via HP=20 Smart Array 6i Here=B4s what I got: theta# /usr/bin/time -h gtar --atime-preserve --same-owner -cvpf=20 omega:/home/teste /home/bkp/home/ gtar: Removing leading `/' from member names home/bkp/home/ home/bkp/home/bkp/ home/bkp/home/bkp/rsh-vulcano.sh home/bkp/home/bkp/2005-01/ home/bkp/home/bkp/2005-01/vulcano_2005-01-31.tar.gz ^Ctime: command terminated abnormally 8m24.24s real 0.15s user 0.78s sys ... and then... omega# ls -laFG /home/teste -rw-r--r-- 1 root wheel 189020160 May 30 12:02 /home/teste omega# this gives me exact 375.040 Bytes/second. then I changed my approach: omega# /usr/bin/time -h rsh theta gtar --atime-preserve --same-owner -C=20 /home/bkp -cvpf - home/ >/home/teste1 home/ home/bkp/ home/bkp/rsh-vulcano.sh home/bkp/2005-01/ home/bkp/2005-01/vulcano_2005-01-31.tar.gz home/bkp/2005-01/liam_2005-01-24.tar.gz home/bkp/2005-01/cache_www2005-01-29.tar.gz ^C 9m33.75s real 8.57s user 3m6.08s sys omega# ls -laFG /home/teste1 -rw-r--r-- 1 root wheel 6396353720 May 30 12:35 /home/teste1 omega# Observe that I bypassed rmt; that bumped the transfer rate to=20 10.976.153,96 Bytes/s, almost 30x faster. Should this really happen?=20 (And yes, I read rmt(8), but found nothing about this. :( ). Thanks for your help; Tulio Tulio Guimar=E3es da Silva wrote: > Hi Gary, > ouch! That=B4s quite disappointing... :( We had already noticed this=20 > kind of behaviour with DDS-* tapes, but we got some progress varying=20 > the block size... and yup, I=B4m really using gzipped data. :S > For AIT-3, however, i thought this hardware compression was something=20 > about using lower tape=B4s phisical-rolling speeds or alikes, but I=20 > could never really find anything concrete about the methods... the=20 > only one thing I found was they could use "variable block sizes", but=20 > that=B4s all. Again, not many details. Anyway, I=B4m giving up the idea= of=20 > compression for now. > If something, I=B4m noticeing that (at least with -b 10) it becomes (a= =20 > lot) slower with time, but I guess this would be more of a question to=20 > the -performance list. > Add: while writing this message, I remembered to check the 700V=B4s=20 > "Product Specification Manual", and they mention something about=20 > dual-partitions, but it seems something that needs to be implemented=20 > at driver level, since it includes SCSI commands. In this case, I=20 > would need to format the tape as a 2-partition one... any clue about=20 > if and/or how that works on FreeBSD? > Thanks again, > > Tulio > > Gary Corcoran wrote: > >> Tulio Guimar=E3es da Silva wrote: >> >>> Hello again, >>> >>> I=B4m having some trouble putting a Sony SDX-700V SCSI AIT-3 unit=20 >>> to work on FreeBSD 5.3-RELEASE. >> >> >> ... >> >>> Besides the speed, hardware compression seems to not being=20 >>> funcional either. I already tried every 4 possible dip switch=20 >>> setting for compression, but I am still not able to transfer a 180GB=20 >>> archive to a (should-be) 260GB medium. >> >> >> >> I can't help you with most of your problems, but regarding the=20 >> "compression"... >> >> I would guess that if you have 180GB to backup, it's not all text. :) >> When I last used a tape drive years ago, when writing to a 2GB tape >> that would supposedly hold 4GB compressed, I could fit only about 1.9G= B >> before the tape was full. Turning off hardware compression, I could f= it >> 2GB. The problem was that I was saving already compressed multimedia=20 >> files, >> and the tape drive's "compression" just added overhead and took up=20 >> more space. >> So unless you're backing up text or similar files, don't believe the >> marketing hype about getting 2x the amount onto your tapes... >> >> Gary > > >------------------------------------------------------------------------ > >_______________________________________________ >freebsd-hardware@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-hardware >To unsubscribe, send any mail to "freebsd-hardware-unsubscribe@freebsd.o= rg" > =20 > --------------090500060903070709060002-- From owner-freebsd-performance@FreeBSD.ORG Tue May 31 03:18:11 2005 Return-Path: X-Original-To: freebsd-performance@FreeBSD.org Delivered-To: freebsd-performance@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 62AC816A41C for ; Tue, 31 May 2005 03:18:11 +0000 (GMT) (envelope-from bde@zeta.org.au) Received: from mailout1.pacific.net.au (mailout1.pacific.net.au [61.8.0.84]) by mx1.FreeBSD.org (Postfix) with ESMTP id CB27743D1D for ; Tue, 31 May 2005 03:18:10 +0000 (GMT) (envelope-from bde@zeta.org.au) Received: from mailproxy2.pacific.net.au (mailproxy2.pacific.net.au [61.8.0.87]) by mailout1.pacific.net.au (8.12.3/8.12.3/Debian-7.1) with ESMTP id j4V3I9rI019432; Tue, 31 May 2005 13:18:09 +1000 Received: from epsplex.bde.org (katana.zip.com.au [61.8.7.246]) by mailproxy2.pacific.net.au (8.12.3/8.12.3/Debian-7.1) with ESMTP id j4V3I5MC030969; Tue, 31 May 2005 13:18:08 +1000 Date: Tue, 31 May 2005 13:18:06 +1000 (EST) From: Bruce Evans X-X-Sender: bde@epsplex.bde.org To: =?ISO-8859-1?Q?Tulio_Guimar=E3es_da_Silva?= In-Reply-To: <429B38C7.5040405@pgt.mpt.gov.br> Message-ID: <20050531130852.U1452@epsplex.bde.org> References: <428CF20F.50607@pgt.mpt.gov.br> <428D0D79.7010506@rcn.com> <428DFF35.1000409@pgt.mpt.gov.br> <429B38C7.5040405@pgt.mpt.gov.br> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-performance@FreeBSD.org Subject: Re: rmt as a bottleneck - Was: Weird behaviour of AIT-3 and (g)tar X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 May 2005 03:18:11 -0000 > Observe that I bypassed rmt; that bumped the transfer rate to 10.976.153,96 > Bytes/s, almost 30x faster. Should this really happen? (And yes, I read > rmt(8), but found nothing about this. :( ). > Thanks for your help; ISTR that remote tars have a delay of 10 msec or so for each block because the protocol needs to talk after each block (it doesn't stream) and there is a TCP startup delay of this amount. Bruce From owner-freebsd-performance@FreeBSD.ORG Tue May 31 14:24:19 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2D74A16A41C for ; Tue, 31 May 2005 14:24:19 +0000 (GMT) (envelope-from dgilbert@daveg.ca) Received: from ox.eicat.ca (ox.eicat.ca [66.96.30.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id D359043D4C for ; Tue, 31 May 2005 14:24:18 +0000 (GMT) (envelope-from dgilbert@daveg.ca) Received: by ox.eicat.ca (Postfix, from userid 66) id 9241EFDEA; Tue, 31 May 2005 10:24:17 -0400 (EDT) Received: by canoe.dclg.ca (Postfix, from userid 101) id F0E0C1A08CB; Tue, 31 May 2005 10:24:14 -0400 (EDT) From: David Gilbert MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17052.29582.462629.587978@canoe.dclg.ca> Date: Tue, 31 May 2005 10:24:14 -0400 To: Bruce Evans In-Reply-To: <20050531130852.U1452@epsplex.bde.org> References: <428CF20F.50607@pgt.mpt.gov.br> <428D0D79.7010506@rcn.com> <428DFF35.1000409@pgt.mpt.gov.br> <429B38C7.5040405@pgt.mpt.gov.br> <20050531130852.U1452@epsplex.bde.org> X-Mailer: VM 7.17 under 21.4 (patch 17) "Jumbo Shrimp" XEmacs Lucid Cc: freebsd-performance@FreeBSD.org, =?ISO-8859-1?Q?Tulio_Guimar=E3es_da_Silva?= Subject: Re: rmt as a bottleneck - Was: Weird behaviour of AIT-3 and (g)tar X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 May 2005 14:24:19 -0000 >>>>> "Bruce" == Bruce Evans writes: >> Observe that I bypassed rmt; that bumped the transfer rate to >> 10.976.153,96 Bytes/s, almost 30x faster. Should this really >> happen? (And yes, I read rmt(8), but found nothing about this. :( >> ). Thanks for your help; Bruce> ISTR that remote tars have a delay of 10 msec or so for each Bruce> block because the protocol needs to talk after each block (it Bruce> doesn't stream) and there is a TCP startup delay of this Bruce> amount. I have often considered the problem of remote tape devices. By way of introduction: I've done a lot of tape handling on UNIX and other systems for complex block formats. The UN*X tape model is rather simplistic. The primary problem with rmt (and UN*X tape in general) is that it cannot stream. If I signal "EOT" (end-of-tape) to tar, tar blithly assumes I meant the block you just wrote caused an EOT. Some tape hardware approximates streaming by estimating EOT ahead of time --- it certainly performs better, but it's not a great solution. Some tape programs on UN*X attempt to buffer heavily in RAM (team, etc)... but to overcome this shortcoming in the multi-tape case, they all need multi-volume smarts. Pretty much a mess. So... yes... there's a chatter for every block because the protocol needs to return the status of every block. I keep threatening to write the difinitive tape suite for FreeBSD ... but something more important always comes along. Problem is, everything needs rejiging ... tar, dump, tape drivers, rmt, etc. Dave. -- ============================================================================ |David Gilbert, Independent Contractor. | Two things can only be | |Mail: dave@daveg.ca | equal if and only if they | |http://daveg.ca | are precisely opposite. | =========================================================GLO================