From owner-freebsd-stable@FreeBSD.ORG Fri Oct 1 21:56:42 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5AF021065672 for ; Fri, 1 Oct 2010 21:56:42 +0000 (UTC) (envelope-from artemb@gmail.com) Received: from mail-gw0-f54.google.com (mail-gw0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 124658FC1B for ; Fri, 1 Oct 2010 21:56:41 +0000 (UTC) Received: by gwb15 with SMTP id 15so1653695gwb.13 for ; Fri, 01 Oct 2010 14:56:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:sender:received :in-reply-to:references:date:x-google-sender-auth:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=w6a3yxfmlPCGR0iAhodqDc9Gamw86bkaXPcR6NFxw8g=; b=u4fv0oS9vV8U4n/kwCVPo7DD0cexCtjLTSCT5TZQpDmWO6n0b937up1OIdRFbwaaM9 ECCXdoWnsPV5OFyFNDEXnuAmj8YAyQbBC2re+ZBCHcwjxcmSYFDKRf73iUdRdJTz1ySJ u4CbjJTMDLMQKn48862H5DUDet/wgYNRrJwCA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=sXWPVpduKNJ+6XaqEbiYa0SgmiJhO94u2dkLeqdWzQmCNR62zQe3hnrQFAgJytUFg9 xGA6VE5SqR0Sn6NTUikNEyHAvv5PKF9azL9cMQFP9PZ/mYoTDzKziCTgFva1JaBvcLsB vG0bLPbdyBXCMYg5R5Q8VpsIs6oLFjWgp0Fxg= MIME-Version: 1.0 Received: by 10.236.102.147 with SMTP id d19mr1526266yhg.69.1285970201271; Fri, 01 Oct 2010 14:56:41 -0700 (PDT) Sender: artemb@gmail.com Received: by 10.220.176.77 with HTTP; Fri, 1 Oct 2010 14:56:41 -0700 (PDT) In-Reply-To: <45cfd27021fb93f9b0877a1596089776.squirrel@nyi.unixathome.org> References: <45cfd27021fb93f9b0877a1596089776.squirrel@nyi.unixathome.org> Date: Fri, 1 Oct 2010 14:56:41 -0700 X-Google-Sender-Auth: ela4voIjdbjswM2V8g7FeJPleRw Message-ID: From: Artem Belevich To: Dan Langille Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-stable@freebsd.org Subject: Re: zfs send/receive: is this slow? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Oct 2010 21:56:42 -0000 Hmm. It did help me a lot when I was replicating ~2TB worth of data over GigE. Without mbuffer things were roughly in the ballpark of your numbers. With mbuffer I've got around 100MB/s. Assuming that you have two boxes connected via ethernet, it would be good to check that nobody generates PAUSE frames. Some time back I've discovered that el-cheapo switch I've been using for some reason could not keep up with traffic bursts and generated tons of PAUSE frames that severely limited throughput. If you're using Intel adapters, check xon/xoff counters in "sysctl dev.em.0.mac_stats". If you see them increasing, that may explain slow speed. If you have a switch between your boxes, try bypassing it and connect boxes directly. --Artem On Fri, Oct 1, 2010 at 11:51 AM, Dan Langille wrote: > > On Wed, September 29, 2010 2:04 pm, Dan Langille wrote: >> $ zpool iostat 10 >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0capacity =A0 =A0 operations =A0 =A0bandwi= dth >> pool =A0 =A0 =A0 =A0 used =A0avail =A0 read =A0write =A0 read =A0write >> ---------- =A0----- =A0----- =A0----- =A0----- =A0----- =A0----- >> storage =A0 =A0 7.67T =A05.02T =A0 =A0358 =A0 =A0 38 =A043.1M =A01.96M >> storage =A0 =A0 7.67T =A05.02T =A0 =A0317 =A0 =A0475 =A039.4M =A030.9M >> storage =A0 =A0 7.67T =A05.02T =A0 =A0357 =A0 =A0533 =A044.3M =A034.4M >> storage =A0 =A0 7.67T =A05.02T =A0 =A0371 =A0 =A0556 =A046.0M =A035.8M >> storage =A0 =A0 7.67T =A05.02T =A0 =A0313 =A0 =A0521 =A038.9M =A028.7M >> storage =A0 =A0 7.67T =A05.02T =A0 =A0309 =A0 =A0457 =A038.4M =A030.4M >> storage =A0 =A0 7.67T =A05.02T =A0 =A0388 =A0 =A0589 =A048.2M =A037.8M >> storage =A0 =A0 7.67T =A05.02T =A0 =A0377 =A0 =A0581 =A046.8M =A036.5M >> storage =A0 =A0 7.67T =A05.02T =A0 =A0310 =A0 =A0559 =A038.4M =A030.4M >> storage =A0 =A0 7.67T =A05.02T =A0 =A0430 =A0 =A0611 =A053.4M =A041.3M > > Now that I'm using mbuffer: > > $ zpool iostat 10 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 capacity =A0 =A0 operations =A0 =A0bandwidth > pool =A0 =A0 =A0 =A0 used =A0avail =A0 read =A0write =A0 read =A0write > ---------- =A0----- =A0----- =A0----- =A0----- =A0----- =A0----- > storage =A0 =A0 9.96T =A02.73T =A02.01K =A0 =A0131 =A0 151M =A06.72M > storage =A0 =A0 9.96T =A02.73T =A0 =A0615 =A0 =A0515 =A076.3M =A033.5M > storage =A0 =A0 9.96T =A02.73T =A0 =A0360 =A0 =A0492 =A044.7M =A033.7M > storage =A0 =A0 9.96T =A02.73T =A0 =A0388 =A0 =A0554 =A048.3M =A038.4M > storage =A0 =A0 9.96T =A02.73T =A0 =A0403 =A0 =A0562 =A050.1M =A039.6M > storage =A0 =A0 9.96T =A02.73T =A0 =A0313 =A0 =A0468 =A038.9M =A028.0M > storage =A0 =A0 9.96T =A02.73T =A0 =A0462 =A0 =A0677 =A057.3M =A022.4M > storage =A0 =A0 9.96T =A02.73T =A0 =A0383 =A0 =A0581 =A047.5M =A021.6M > storage =A0 =A0 9.96T =A02.72T =A0 =A0142 =A0 =A0571 =A017.7M =A015.4M > storage =A0 =A0 9.96T =A02.72T =A0 =A0 80 =A0 =A0598 =A010.0M =A018.8M > storage =A0 =A0 9.96T =A02.72T =A0 =A0718 =A0 =A0503 =A089.1M =A013.6M > storage =A0 =A0 9.96T =A02.72T =A0 =A0594 =A0 =A0517 =A073.8M =A014.1M > storage =A0 =A0 9.96T =A02.72T =A0 =A0367 =A0 =A0528 =A045.6M =A015.1M > storage =A0 =A0 9.96T =A02.72T =A0 =A0338 =A0 =A0520 =A041.9M =A016.4M > storage =A0 =A0 9.96T =A02.72T =A0 =A0348 =A0 =A0499 =A043.3M =A021.5M > storage =A0 =A0 9.96T =A02.72T =A0 =A0398 =A0 =A0553 =A049.4M =A014.4M > storage =A0 =A0 9.96T =A02.72T =A0 =A0346 =A0 =A0481 =A043.0M =A06.78M > > If anything, it's slower. > > The above was without -s 128. =A0The following used that setting: > > =A0$ zpool iostat 10 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 capacity =A0 =A0 operations =A0 =A0bandwidth > pool =A0 =A0 =A0 =A0 used =A0avail =A0 read =A0write =A0 read =A0write > ---------- =A0----- =A0----- =A0----- =A0----- =A0----- =A0----- > storage =A0 =A0 9.78T =A02.91T =A01.98K =A0 =A0137 =A0 149M =A06.92M > storage =A0 =A0 9.78T =A02.91T =A0 =A0761 =A0 =A0577 =A094.4M =A042.6M > storage =A0 =A0 9.78T =A02.91T =A0 =A0462 =A0 =A0411 =A057.4M =A024.6M > storage =A0 =A0 9.78T =A02.91T =A0 =A0492 =A0 =A0497 =A061.1M =A027.6M > storage =A0 =A0 9.78T =A02.91T =A0 =A0632 =A0 =A0446 =A078.5M =A022.5M > storage =A0 =A0 9.78T =A02.91T =A0 =A0554 =A0 =A0414 =A068.7M =A021.8M > storage =A0 =A0 9.78T =A02.91T =A0 =A0459 =A0 =A0434 =A057.0M =A031.4M > storage =A0 =A0 9.78T =A02.91T =A0 =A0398 =A0 =A0570 =A049.4M =A032.7M > storage =A0 =A0 9.78T =A02.91T =A0 =A0338 =A0 =A0495 =A041.9M =A026.5M > storage =A0 =A0 9.78T =A02.91T =A0 =A0358 =A0 =A0526 =A044.5M =A033.3M > storage =A0 =A0 9.78T =A02.91T =A0 =A0385 =A0 =A0555 =A047.8M =A039.8M > storage =A0 =A0 9.78T =A02.91T =A0 =A0271 =A0 =A0453 =A033.6M =A023.3M > storage =A0 =A0 9.78T =A02.91T =A0 =A0270 =A0 =A0456 =A033.5M =A028.8M > > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >