From owner-freebsd-scsi@FreeBSD.ORG  Tue Sep 25 11:57:45 2012
Return-Path: <owner-freebsd-scsi@FreeBSD.ORG>
Delivered-To: freebsd-scsi@FreeBSD.ORG
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 7617D1065679;
	Tue, 25 Sep 2012 11:57:45 +0000 (UTC)
	(envelope-from gpalmer@freebsd.org)
Received: from noop.in-addr.com (mail.in-addr.com [IPv6:2001:470:8:162::1])
	by mx1.freebsd.org (Postfix) with ESMTP id 3A7C58FC0C;
	Tue, 25 Sep 2012 11:57:45 +0000 (UTC)
Received: from gjp by noop.in-addr.com with local (Exim 4.80 (FreeBSD))
	(envelope-from <gpalmer@freebsd.org>)
	id 1TGTlh-000K0W-9I; Tue, 25 Sep 2012 07:57:37 -0400
Date: Tue, 25 Sep 2012 07:57:37 -0400
From: Gary Palmer <gpalmer@freebsd.org>
To: John <jwd@FreeBSD.ORG>
Message-ID: <20120925115737.GE77784@in-addr.com>
References: <20120925030953.GA84605@FreeBSD.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20120925030953.GA84605@FreeBSD.org>
X-SA-Exim-Connect-IP: <locally generated>
X-SA-Exim-Mail-From: gpalmer@freebsd.org
X-SA-Exim-Scanned: No (on noop.in-addr.com); SAEximRunCond expanded to false
Cc: FreeBSD iSCSI <freebsd-scsi@FreeBSD.ORG>
Subject: Re: Performance question - istgt with dual 10g data links to linux
 client
X-BeenThere: freebsd-scsi@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: SCSI subsystem <freebsd-scsi.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-scsi>,
	<mailto:freebsd-scsi-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-scsi>
List-Post: <mailto:freebsd-scsi@freebsd.org>
List-Help: <mailto:freebsd-scsi-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-scsi>,
	<mailto:freebsd-scsi-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 25 Sep 2012 11:57:45 -0000

On Tue, Sep 25, 2012 at 03:09:53AM +0000, John wrote:
> Hi Folks,
> 
>    I have a bsd 9.1 zfs server running the latest istgt connected to a
> RHEL 6.1 system.  Regardless of how I configure the systems, I cannot seem
> to exceed 1GB throughput. If I create a 25G /dev/md0 and export it via
> istgt (no mpio here), format it with default xfs values, place a
> 20G file on it, I get the following:
> 
> dd if=/usr2/20g of=/dev/null bs=512K
> 40960+0 records in
> 40960+0 records out
> 21474836480 bytes (21 GB) copied, 21.4256 s, 1.0 GB/s
> 
>    Running the above /dev/md0 with mpio, dual paths on 10G cards, with
> rr_minio set anywhere from 1 to 100 on the linux side:
> 
> [PortalGroup2]
>    Comment "Two networks - one port"
>    Portal DA1 10.59.6.14:5020          # 10G mtu 9000
>    Portal DA2 10.60.6.14:5020          # 10G mtu 9000
>    Comment "END: PortalGroup2"
> 
> mpatha (33000000051ed39a4) dm-0 FreeBSD,USE136EXHF_iSCSI
> size=25G features='0' hwhandler='0' wp=rw
> `-+- policy='round-robin 0' prio=1 status=active
>   |- 11:0:0:0 sdd 8:48 active ready  running
>   `- 12:0:0:0 sde 8:64 active ready  running
> 
> 
> dd if=/usr2/20g of=/dev/null bs=1M
> 20480+0 records in
> 20480+0 records out
> 21474836480 bytes (21 GB) copied, 20.0076 s, 1.1 GB/s
> 
>    I can see the traffic evenly across both interfaces. I simply
> can't seem to get the parallelization factor up. Higher levels
> of mpio have no effect.
> 
>    I realize I haven't included the entire configuration. I'm hoping
> someone can give some high-level thoughts. I do need to maximize
> single process large file i/o..

Have you tried doing two dd's at the same time when you have mpio running?
I'd be curious to know if you get double the throughput or if each dd
gets 0.5GB/s

Gary