From owner-freebsd-stable@FreeBSD.ORG Fri Oct 1 23:00:16 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1987A106566B for ; Fri, 1 Oct 2010 23:00:16 +0000 (UTC) (envelope-from artemb@gmail.com) Received: from mail-qw0-f54.google.com (mail-qw0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id E18C18FC1F for ; Fri, 1 Oct 2010 23:00:14 +0000 (UTC) Received: by qwd6 with SMTP id 6so2154038qwd.13 for ; Fri, 01 Oct 2010 16:00:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:sender:received :in-reply-to:references:date:x-google-sender-auth:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=sCXAY6tIx6/OL29Y7iATwJ63wKUXk4/VUhVJV/Ygs1g=; b=PUgfmbIlWO7Fp7K16dLlNekQRT3xffbL2oa5sRBtIySs28RP1hS0ffyRP36NK3MMwz VACHs+csKYeLvGaQcCJs0B626EP47HkDFs/LdSwsCOL7ZzamOgWNoiWuU4hiF1vaUkd7 uNcwIIHZRxnratjwSOuW6LUCi3Yg9borXv9hA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=WkPQ2EbB/ckyS5k12wcGJQwjHC3v5kWvT9K2hlCkDq80Sk7qHJzB/OUiuiAoHo9x1w eYrwmUluyAB6BagePKiOXj69uH8gdS4g8ZMT5OzZDXNsyB+0GLfgY8aJ95SJSC+aHIck sfHgAHaudlSrACXfHuEtzTfTuTId+SY/46dMw= MIME-Version: 1.0 Received: by 10.220.63.15 with SMTP id z15mr1453796vch.70.1285974013979; Fri, 01 Oct 2010 16:00:13 -0700 (PDT) Sender: artemb@gmail.com Received: by 10.220.176.77 with HTTP; Fri, 1 Oct 2010 16:00:13 -0700 (PDT) In-Reply-To: <4C511EF8-591C-4BB9-B7AA-30D5C3DDC0FF@langille.org> References: <45cfd27021fb93f9b0877a1596089776.squirrel@nyi.unixathome.org> <4C511EF8-591C-4BB9-B7AA-30D5C3DDC0FF@langille.org> Date: Fri, 1 Oct 2010 16:00:13 -0700 X-Google-Sender-Auth: aVwa1-MUmdu9FqDXfCCWBCNQlWk Message-ID: From: Artem Belevich To: Dan Langille Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-stable@freebsd.org" Subject: Re: zfs send/receive: is this slow? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Oct 2010 23:00:16 -0000 On Fri, Oct 1, 2010 at 3:49 PM, Dan Langille wrote: > FYI: this is all on the same box. In one of the previous emails you've used this command line: > # mbuffer -s 128k -m 1G -I 9090 | zfs receive You've used mbuffer in network client mode. I assumed that you did do your transfer over network. If you're running send/receive locally just pipe the data through mbuffer -- zfs send|mbuffer|zfs receive --Artem > > -- > Dan Langille > http://langille.org/ > > > On Oct 1, 2010, at 5:56 PM, Artem Belevich wrote: > >> Hmm. It did help me a lot when I was replicating ~2TB worth of data >> over GigE. Without mbuffer things were roughly in the ballpark of your >> numbers. With mbuffer I've got around 100MB/s. >> >> Assuming that you have two boxes connected via ethernet, it would be >> good to check that nobody generates PAUSE frames. Some time back I've >> discovered that el-cheapo switch I've been using for some reason could >> not keep up with traffic bursts and generated tons of PAUSE frames >> that severely limited throughput. >> >> If you're using Intel adapters, check xon/xoff counters in "sysctl >> dev.em.0.mac_stats". If you see them increasing, that may explain slow >> speed. >> If you have a switch between your boxes, try bypassing it and connect >> boxes directly. >> >> --Artem >> >> >> >> On Fri, Oct 1, 2010 at 11:51 AM, Dan Langille wrote: >>> >>> On Wed, September 29, 2010 2:04 pm, Dan Langille wrote: >>>> $ zpool iostat 10 >>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0capacity =A0 =A0 operations =A0 =A0band= width >>>> pool =A0 =A0 =A0 =A0 used =A0avail =A0 read =A0write =A0 read =A0write >>>> ---------- =A0----- =A0----- =A0----- =A0----- =A0----- =A0----- >>>> storage =A0 =A0 7.67T =A05.02T =A0 =A0358 =A0 =A0 38 =A043.1M =A01.96M >>>> storage =A0 =A0 7.67T =A05.02T =A0 =A0317 =A0 =A0475 =A039.4M =A030.9M >>>> storage =A0 =A0 7.67T =A05.02T =A0 =A0357 =A0 =A0533 =A044.3M =A034.4M >>>> storage =A0 =A0 7.67T =A05.02T =A0 =A0371 =A0 =A0556 =A046.0M =A035.8M >>>> storage =A0 =A0 7.67T =A05.02T =A0 =A0313 =A0 =A0521 =A038.9M =A028.7M >>>> storage =A0 =A0 7.67T =A05.02T =A0 =A0309 =A0 =A0457 =A038.4M =A030.4M >>>> storage =A0 =A0 7.67T =A05.02T =A0 =A0388 =A0 =A0589 =A048.2M =A037.8M >>>> storage =A0 =A0 7.67T =A05.02T =A0 =A0377 =A0 =A0581 =A046.8M =A036.5M >>>> storage =A0 =A0 7.67T =A05.02T =A0 =A0310 =A0 =A0559 =A038.4M =A030.4M >>>> storage =A0 =A0 7.67T =A05.02T =A0 =A0430 =A0 =A0611 =A053.4M =A041.3M >>> >>> Now that I'm using mbuffer: >>> >>> $ zpool iostat 10 >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 capacity =A0 =A0 operations =A0 =A0bandwidt= h >>> pool =A0 =A0 =A0 =A0 used =A0avail =A0 read =A0write =A0 read =A0write >>> ---------- =A0----- =A0----- =A0----- =A0----- =A0----- =A0----- >>> storage =A0 =A0 9.96T =A02.73T =A02.01K =A0 =A0131 =A0 151M =A06.72M >>> storage =A0 =A0 9.96T =A02.73T =A0 =A0615 =A0 =A0515 =A076.3M =A033.5M >>> storage =A0 =A0 9.96T =A02.73T =A0 =A0360 =A0 =A0492 =A044.7M =A033.7M >>> storage =A0 =A0 9.96T =A02.73T =A0 =A0388 =A0 =A0554 =A048.3M =A038.4M >>> storage =A0 =A0 9.96T =A02.73T =A0 =A0403 =A0 =A0562 =A050.1M =A039.6M >>> storage =A0 =A0 9.96T =A02.73T =A0 =A0313 =A0 =A0468 =A038.9M =A028.0M >>> storage =A0 =A0 9.96T =A02.73T =A0 =A0462 =A0 =A0677 =A057.3M =A022.4M >>> storage =A0 =A0 9.96T =A02.73T =A0 =A0383 =A0 =A0581 =A047.5M =A021.6M >>> storage =A0 =A0 9.96T =A02.72T =A0 =A0142 =A0 =A0571 =A017.7M =A015.4M >>> storage =A0 =A0 9.96T =A02.72T =A0 =A0 80 =A0 =A0598 =A010.0M =A018.8M >>> storage =A0 =A0 9.96T =A02.72T =A0 =A0718 =A0 =A0503 =A089.1M =A013.6M >>> storage =A0 =A0 9.96T =A02.72T =A0 =A0594 =A0 =A0517 =A073.8M =A014.1M >>> storage =A0 =A0 9.96T =A02.72T =A0 =A0367 =A0 =A0528 =A045.6M =A015.1M >>> storage =A0 =A0 9.96T =A02.72T =A0 =A0338 =A0 =A0520 =A041.9M =A016.4M >>> storage =A0 =A0 9.96T =A02.72T =A0 =A0348 =A0 =A0499 =A043.3M =A021.5M >>> storage =A0 =A0 9.96T =A02.72T =A0 =A0398 =A0 =A0553 =A049.4M =A014.4M >>> storage =A0 =A0 9.96T =A02.72T =A0 =A0346 =A0 =A0481 =A043.0M =A06.78M >>> >>> If anything, it's slower. >>> >>> The above was without -s 128. =A0The following used that setting: >>> >>> =A0$ zpool iostat 10 >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 capacity =A0 =A0 operations =A0 =A0bandwidt= h >>> pool =A0 =A0 =A0 =A0 used =A0avail =A0 read =A0write =A0 read =A0write >>> ---------- =A0----- =A0----- =A0----- =A0----- =A0----- =A0----- >>> storage =A0 =A0 9.78T =A02.91T =A01.98K =A0 =A0137 =A0 149M =A06.92M >>> storage =A0 =A0 9.78T =A02.91T =A0 =A0761 =A0 =A0577 =A094.4M =A042.6M >>> storage =A0 =A0 9.78T =A02.91T =A0 =A0462 =A0 =A0411 =A057.4M =A024.6M >>> storage =A0 =A0 9.78T =A02.91T =A0 =A0492 =A0 =A0497 =A061.1M =A027.6M >>> storage =A0 =A0 9.78T =A02.91T =A0 =A0632 =A0 =A0446 =A078.5M =A022.5M >>> storage =A0 =A0 9.78T =A02.91T =A0 =A0554 =A0 =A0414 =A068.7M =A021.8M >>> storage =A0 =A0 9.78T =A02.91T =A0 =A0459 =A0 =A0434 =A057.0M =A031.4M >>> storage =A0 =A0 9.78T =A02.91T =A0 =A0398 =A0 =A0570 =A049.4M =A032.7M >>> storage =A0 =A0 9.78T =A02.91T =A0 =A0338 =A0 =A0495 =A041.9M =A026.5M >>> storage =A0 =A0 9.78T =A02.91T =A0 =A0358 =A0 =A0526 =A044.5M =A033.3M >>> storage =A0 =A0 9.78T =A02.91T =A0 =A0385 =A0 =A0555 =A047.8M =A039.8M >>> storage =A0 =A0 9.78T =A02.91T =A0 =A0271 =A0 =A0453 =A033.6M =A023.3M >>> storage =A0 =A0 9.78T =A02.91T =A0 =A0270 =A0 =A0456 =A033.5M =A028.8M >>> >>> >>> _______________________________________________ >>> freebsd-stable@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >>> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.or= g" >>> >> >