From owner-freebsd-scsi@FreeBSD.ORG Thu Jun 18 08:20:16 2015 Return-Path: Delivered-To: freebsd-scsi@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 73542D6D for ; Thu, 18 Jun 2015 08:20:16 +0000 (UTC) (envelope-from maxg@mellanox.com) Received: from emea01-db3-obe.outbound.protection.outlook.com (mail-db3on0091.outbound.protection.outlook.com [157.55.234.91]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9A6025EA for ; Thu, 18 Jun 2015 08:20:14 +0000 (UTC) (envelope-from maxg@mellanox.com) Received: from DB3PR05CA0020.eurprd05.prod.outlook.com (10.160.41.148) by AMXPR05MB165.eurprd05.prod.outlook.com (10.242.72.139) with Microsoft SMTP Server (TLS) id 15.1.195.15; Thu, 18 Jun 2015 08:05:46 +0000 Received: from AM1FFO11OLC007.protection.gbl (2a01:111:f400:7e00::117) by DB3PR05CA0020.outlook.office365.com (2a01:111:e400:9428::20) with Microsoft SMTP Server (TLS) id 15.1.195.15 via Frontend Transport; Thu, 18 Jun 2015 08:05:46 +0000 Authentication-Results: spf=none (sender IP is 193.47.165.134) smtp.mailfrom=mellanox.com; freebsd.org; dkim=none (message not signed) header.d=none; Received-SPF: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) Received: from mtlcas13.mtl.com (193.47.165.134) by AM1FFO11OLC007.mail.protection.outlook.com (10.174.64.133) with Microsoft SMTP Server (TLS) id 15.1.184.11 via Frontend Transport; Thu, 18 Jun 2015 08:05:44 +0000 Received: from MTLCAS13.mtl.com (10.0.8.78) by mtlcas13.mtl.com (10.0.8.78) with Microsoft SMTP Server (TLS) id 15.0.775.38; Thu, 18 Jun 2015 11:05:25 +0300 Received: from MTLCAS01.mtl.com (10.0.8.71) by MTLCAS13.mtl.com (10.0.8.78) with Microsoft SMTP Server (TLS) id 15.0.775.38 via Frontend Transport; Thu, 18 Jun 2015 11:05:25 +0300 Received: from [10.223.0.78] (10.223.0.78) by MTLCAS01.mtl.com (10.0.8.71) with Microsoft SMTP Server (TLS) id 14.3.123.3; Thu, 18 Jun 2015 11:05:24 +0300 Message-ID: <55827BBF.6040206@mellanox.com> Date: Thu, 18 Jun 2015 11:05:19 +0300 From: Max Gurtovoy User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: , Sagi Grimberg , Oren Duer , Hans Petter Selasky Subject: Re: gmultipath HA over iscsi/iser References: <557DA8C0.1020209@mellanox.com> In-Reply-To: <557DA8C0.1020209@mellanox.com> Content-Type: text/plain; charset="windows-1255"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [10.223.0.78] X-EOPAttributedMessage: 0 X-Microsoft-Exchange-Diagnostics: 1; AM1FFO11OLC007; 1:IJwhiUyei1igiwphiM3pU57+IDbjCRyKbaxzU+dzJt/iopgwW4rQ0872XV/CyMpLd0PfRvYOs0RaRLBZWaKbcUlj/yX2bqQWODW+CRzjDzu+YvmBMDWkd7MJgoEaPzkjr3HFVll+y/yBNV3tP6Q6JWIAqhPfrFtN3P2BCSOl6YLqiDW85HNo0QeUIuipw/4awlpRGTWFZDrtUAphi7i7jStWalyonUoLOmebC+ALXvy6e29HzqWqKEvUGI59NiZInmwFW0/5s3DZC47m5ATjyQ== X-Forefront-Antispam-Report: CIP:193.47.165.134; CTRY:IL; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(10009020)(6009001)(428002)(199003)(51704005)(24454002)(377454003)(164054003)(479174004)(189002)(77096005)(50466002)(2950100001)(189998001)(105586002)(6806004)(80316001)(92566002)(4001350100001)(106466001)(107886002)(5001770100001)(83506001)(101416001)(46102003)(65806001)(65956001)(47776003)(54356999)(65816999)(76176999)(450100001)(87266999)(50986999)(33656002)(87936001)(86362001)(77156002)(62966003)(36756003)(3940600001)(4001450100001); DIR:OUT; SFP:1101; SCL:1; SRVR:AMXPR05MB165; H:mtlcas13.mtl.com; FPR:; SPF:None; MLV:sfv; A:1; MX:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; AMXPR05MB165; 2:Mcn8JoD9OKr1D5WkOqYYDsqfTRSyhLz9KbUTdOdiZlKS4KrLFHdsS9F5ACrJpt7Z; 3:x1n777pW4A5pSWk3huehnjrCHcRxvuIwbE+WQ1DwfdGi8Gh1eaAzVKTploxOHKis3rM6IRceLXtK7N5+GAPxlwPtO/2l8D+ajrEIvO2UhmDU8KA7TNl+/HRDdVi9BnG3mPrMFP9AH/Hyp6lJU6m1loIT1oa9UDO/ayffnD29P8RnUdbHJrxXdlPFupdf2XVPxMMW/m6XYjJXnbp9lrRJ0yduOwGZYK2I0bUnRQ3SwbPhdFW+QKH+eDzm6u7YbV0I; 20:HU0krm7INUUR0rZgXLgF/h1qgRjLcjn4fuHxLwyKbqtW/XX871vdYlRWdfvKbIDTe+W8q0tyGh56iBihXqEg8yN7/7xZ1C0EWFfTwSe5i+Sj0mfNa0Ug1VJnW0yUoCKzGzaZ14WZbc22DJy9HxBbdW+dmpR1Y3KQmWs8XMr9TMPAnCEtRwQEKgYcNxQcITYIZGrg/mhWvlMGS5oQFK+hW+RakiO+fVjAnYIjEVZwUkdFaRpV3Ks3JHLFDQNQcd2SmdLT6gPWwcyI92mOvIz38rPrfCF15FbC9zJ9kD2cghfS3GEBU2vewLc0sOkIL0p1bDuhxupglpmRv/ky3/TLD5YmW8hmBd9N0CFKpREDf6TNXlC9AQJsY80yZ0Zkw0SVg1L4CAqEK5ugAMGjt6yH/sGnEObMeEbATaqQSkxv581alvl7sWcEKBcQRHE7NoV6qnCK/MEaSMucxCqP4kgzOiLQ9Af1LwYC/fhx8UD8vTPVgf/iVO/CwH1Md5jMhY7l X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:AMXPR05MB165; X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(5005006)(3002001); SRVR:AMXPR05MB165; BCL:0; PCL:0; RULEID:; SRVR:AMXPR05MB165; X-Microsoft-Exchange-Diagnostics: 1; AMXPR05MB165; 4:nx18qpledMG6ItTo1+4aMx73VCuPnNbt4cn0tjUabqxF3GYXELESbp66LX30OSWOypSta4vL0n7Cdwx/vlLENJ+ICAWMIoskYldPfpQr9NG4oNzNPo0R452Q1ED7laeMJecB5VLsumAjPdX2frt1a+37S4hqcUL7tnRw12RtUbO3JGQ5F5HjFBX5rUPlA1GPXVFp0CMstoZ3FSLuDMR/8gctAy8ritay3r8/GfZ5MYVTi6W38GygKBEvlRMFg5pZm7lEIETQtLccyXxQRrX7/gF5smCcYOeM9Tuf/cTMLS8= X-Forefront-PRVS: 0611A21987 X-Microsoft-Exchange-Diagnostics: =?windows-1255?Q?1; AMXPR05MB165; 23:PT96L67e1HMGwr8ZYv8OTtQYQAy43zHd/w6iyJ?= =?windows-1255?Q?pCxTKuTfiWT4yHDB4LEq06ErVguqHSbrFAYIoPN0DeL///Zx1f0ABmnd?= =?windows-1255?Q?2DWSzPev/HmJiKKOCK2XOd+rEArOWn2W+duVC+Dm2TPvxPDQFmbORuje?= =?windows-1255?Q?EJHrpgbq8YCwpTO5GCRtmoP2SwnSE9+7H/HQJGyFgzKNkjBBe9B2TaEs?= =?windows-1255?Q?oudFckWCYTTjk82JN1ZLx69c6DWaSlLyH13j5m+o2QOfwNtWHH7s7JsG?= =?windows-1255?Q?NCA4S/xCOX2uizM+xJLhfR9b5vu3ElOB6mzHDLWyQQbpr4ZsMJHJNRJ/?= =?windows-1255?Q?E3aAqwureUYLd5wQQchUZdGWvGIb/016nFCu+DftSk5ZcNdTqbEQEd36?= =?windows-1255?Q?zhUkXWV3UshUJHCy/cg910qQtyF+/AezUioPVMqEHD65PoqAZs3GG4FD?= =?windows-1255?Q?K8mGi1dHk95McCIxWq43saPJLTrxVKv6lVMoPH4cjqiFIkKG/facSXPw?= =?windows-1255?Q?xfT8PiLdTISBxOTPIN7tFZGeBVVCQmrzh03me3SgOKBOoLwTTJOxHSby?= =?windows-1255?Q?VH3vyZDdV+QiM1fnghSAOXpLmpKQQwGRUEyMiclWfUGFNv8HZR/Smzyj?= =?windows-1255?Q?I2m2OGWfxI5bIg5PdZOKYRZhsV9Ov1AvWZq3y8zCpa0Tu5uOX+L69odB?= =?windows-1255?Q?0YZKDQi+Vud9a7ahnRLP3/Ufy07urE3dslimGRp/Bvz25ArWecNUGK08?= =?windows-1255?Q?W98b2gBOyjCFMk+Sek7ROc+E/gL6qX1HNEXW2hN9SHa4NL/ZZ6p9hQJP?= =?windows-1255?Q?NHloZw0i56IDxctaP4DgY5Js2FoVrEjO86rgOrcAVngB21wvWuKRl5Vq?= =?windows-1255?Q?7Vl0o4NdDZTxjthH/M1/0mLuNJVjX5b+GUe5clnqL6BHDNBqvVhWAXjo?= =?windows-1255?Q?8NjC1SR2OyGwrU6ygKaiDlS7VBdm8u3pkwmfTyai6CFB3NoPbAZgZIxR?= =?windows-1255?Q?0Xn/eszSqS04QBA2/R67TU6YDQ0I3zpvwhstfM1SwOj2VhKezNfp4qPy?= =?windows-1255?Q?bolIzllL54aSnRisc0lkcplJ6vRXdaT9q2mI/VN1Ezvzo3dpSr/d42Uu?= =?windows-1255?Q?2U9oUKWlHj8o/1Uy6oPOI=3D?= X-Microsoft-Exchange-Diagnostics: 1; AMXPR05MB165; 5:XjwIbM7niG+Gk3cmtd7sKBVR+cIb8l+ZXKbyKDNH5viEQM9PC7AbKEbtiVktX+DRRYCRyXvVVd9SuwJbJlcWpzekLCMzBLcTCha4ZcaJ39unzZSNYLJ0RxDKiFRmvJOxXZBXZrLns849cDQbjuD3Ww==; 24:KDx55JoGHLwk6bPMmmHrpd4K5ciOkSK4qGpCeuyz5BRpeMBTIVf1Ey3i3KgeETl4t+mz5BKmss0WR2eWvTFi4vT+vrrClaoTxBIxicaAOFA=; 20:Ztn5woKdaB+w+mw5E97+2b8m2Gv3SALI7mJM9T/sTNLnb33LPqLXPFVvtATUGjOHikLDidzQ4qb0smzWdidngw== X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jun 2015 08:05:44.7525 (UTC) X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=a652971c-7d2e-4d9b-a6a4-d149256f461b; Ip=[193.47.165.134]; Helo=[mtlcas13.mtl.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AMXPR05MB165 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Jun 2015 08:20:16 -0000 Anyone checked gmultipath utility ? Thanks, Max. On 6/14/2015 7:16 PM, Max Gurtovoy wrote: > Hello, > lately I was testing HA using gmultipath utility over iSCSI/iSER devices. > I'm working on 11-current code base. > I created 1 LUN on the target side and connected via 2 different > physical ports from the initiator side. > On the initiator side I see see /dev/da0 and /dev/da1. > I created multipath device using: > gmultipath label dm0 /dev/da0 /dev/da1. > Now I have new device /dev/multipath/dm0. > I set kern.iscsi.fail_on_disconnection=1 (to fail IO fast). > > Issue 1: > ------------- > I can't run simple fio/dd traffice over /dev/da0 nor /dev/da1. > The only traffic that possible is using the multipath device dm0. > Is this by design ? > In the linux implementation we can run traffic on both block devices > and multipath devices. > > Issue 2: > -------------- > I run some fio traffic utility over multipath device dm0 on initiator > side with port toggling in a loop > > Port 1 down --> sleep 2 mins (iSCSI/ISER device reconnecting meanwhile > with no success) --> port 1 up --> sleep 5 mins (iSCSI/ISER device > reconnecting successecfully) > Port 2 down --> sleep 2 mins (iSCSI/ISER device reconnecting meanwhile > with no success) --> port 2 up --> sleep 5 mins (iSCSI/ISER device > reconnecting successecfully) > > The expected result is that when the port N is down than the traffic > moves to the available port and continue succesfully. > I run this test for many hours and traffic FAILED (even though there > was at least 1 suitable path between initiator and target). > > log: > > # gmultipath status > > Name Status Components > > multipath/dm_tcp OPTIMAL da0 (ACTIVE) > > da1 (PASSIVE) > > multipath/dm_iser OPTIMAL da2 (ACTIVE) > > da3 (PASSIVE) > > > # fio ..... (over /dev/multipath/dm_iser or /dev/multipath/dm_tcp) > > > fio: this platform does not support process shared mutexes, forcing > use of threads. Use the 'thread' option to get rid of this warning. > > task1: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=psync, iodepth=8 > > ... > > task1: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=psync, iodepth=8 > > fio-2.1.3 > > Starting 8 threads > > fio: pid=101071, err=6/file:filesetup.c:575, > func=open(/dev/multipath/dm_tcp), error=Device not configured > > task1: (groupid=0, jobs=8): err= 6 (file:filesetup.c:575, > func=open(/dev/multipath/dm_tcp), error=Device not configured): > pid=101071: Thu Jun 11 17:25:47 2015 > > read : io=296400MB, bw=32122KB/s, iops=8030, runt=9448911msec > > clat (usec): min=131, max=5541.8K, avg=504.40, stdev=5660.23 > > lat (usec): min=132, max=5541.8K, avg=504.55, stdev=5660.23 > > clat percentiles (usec): > > | 1.00th=[ 251], 5.00th=[ 298], 10.00th=[ 330], 20.00th=[ 370], > > | 30.00th=[ 406], 40.00th=[ 446], 50.00th=[ 478], 60.00th=[ 510], > > | 70.00th=[ 540], 80.00th=[ 580], 90.00th=[ 644], 95.00th=[ 700], > > | 99.00th=[ 1448], 99.50th=[ 1704], 99.90th=[ 1976], 99.95th=[ 2064], > > | 99.99th=[ 2256] > > bw (KB /s): min= 2, max= 5576, per=12.64%, avg=4060.97, > stdev=352.37 > > write: io=295596MB, bw=32034KB/s, iops=8008, runt=9448911msec > > clat (usec): min=125, max=5541.8K, avg=490.13, stdev=5143.96 > > lat (usec): min=125, max=5541.8K, avg=490.41, stdev=5143.96 > > clat percentiles (usec): > > | 1.00th=[ 239], 5.00th=[ 282], 10.00th=[ 310], 20.00th=[ 354], > > | 30.00th=[ 390], 40.00th=[ 426], 50.00th=[ 466], 60.00th=[ 502], > > | 70.00th=[ 532], 80.00th=[ 572], 90.00th=[ 628], 95.00th=[ 692], > > | 99.00th=[ 1432], 99.50th=[ 1688], 99.90th=[ 1960], 99.95th=[ 2040], > > | 99.99th=[ 2256] > > bw (KB /s): min= 3, max= 5512, per=12.64%, avg=4049.74, > stdev=355.11 > > lat (usec) : 250=1.29%, 500=56.84%, 750=38.78%, 1000=0.94% > > lat (msec) : 2=2.08%, 4=0.07%, 10=0.01%, 20=0.01%, 50=0.01% > > lat (msec) : 100=0.01%, >=2000=0.01% > > cpu : usr=0.61%, sys=4.33%, ctx=151634083, majf=0, minf=3 > > IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, > >=64=0.0% > > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, > >=64=0.0% > > complete : 0=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, > >=64=0.0% > > issued : total=r=75878522/w=75672554/d=0, short=r=0/w=0/d=0 > > Run status group 0 (all jobs): > > READ: io=296400MB, aggrb=32121KB/s, minb=32121KB/s, maxb=32121KB/s, > mint=9448911msec, maxt=9448911msec > > WRITE: io=295596MB, aggrb=32034KB/s, minb=32034KB/s, maxb=32034KB/s, > mint=9448911msec, maxt=9448911msec > > > # gmultipath status > > Name Status Components > > multipath/dm_tcp DEGRADED da1 (ACTIVE) > > multipath/dm_iser DEGRADED da3 (ACTIVE) > > > We can see that there is Active paths to multipath device but still > traffice failed. > Any suggestions ? Anyone saw this before ? > > Thanks, > Max Gurtovoy. > Mellanox Technologies. > > > > > > >