From owner-freebsd-scsi@FreeBSD.ORG Sun Jun 14 21:50:37 2015 Return-Path: Delivered-To: freebsd-scsi@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6CBD7AFB for ; Sun, 14 Jun 2015 21:50:37 +0000 (UTC) (envelope-from maxg@mellanox.com) Received: from emea01-am1-obe.outbound.protection.outlook.com (mail-am1on0098.outbound.protection.outlook.com [157.56.112.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BB65582C for ; Sun, 14 Jun 2015 21:50:35 +0000 (UTC) (envelope-from maxg@mellanox.com) Received: from DB3PR05CA0077.eurprd05.prod.outlook.com (10.163.44.45) by AMSPR05MB161.eurprd05.prod.outlook.com (10.242.86.11) with Microsoft SMTP Server (TLS) id 15.1.190.14; Sun, 14 Jun 2015 16:16:12 +0000 Received: from DB3FFO11FD050.protection.gbl (2a01:111:f400:7e04::185) by DB3PR05CA0077.outlook.office365.com (2a01:111:e400:9448::45) with Microsoft SMTP Server (TLS) id 15.1.190.14 via Frontend Transport; Sun, 14 Jun 2015 16:16:12 +0000 Authentication-Results: spf=none (sender IP is 193.47.165.134) smtp.mailfrom=mellanox.com; freebsd.org; dkim=none (message not signed) header.d=none; Received-SPF: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) Received: from mtlcas13.mtl.com (193.47.165.134) by DB3FFO11FD050.mail.protection.outlook.com (10.47.217.81) with Microsoft SMTP Server (TLS) id 15.1.190.9 via Frontend Transport; Sun, 14 Jun 2015 16:16:10 +0000 Received: from MTLCAS13.mtl.com (10.0.8.78) by mtlcas13.mtl.com (10.0.8.78) with Microsoft SMTP Server (TLS) id 15.0.775.38; Sun, 14 Jun 2015 19:16:25 +0300 Received: from MTLCAS01.mtl.com (10.0.8.71) by MTLCAS13.mtl.com (10.0.8.78) with Microsoft SMTP Server (TLS) id 15.0.775.38 via Frontend Transport; Sun, 14 Jun 2015 19:16:25 +0300 Received: from [10.223.0.78] (10.223.0.78) by MTLCAS01.mtl.com (10.0.8.71) with Microsoft SMTP Server (TLS) id 14.3.123.3; Sun, 14 Jun 2015 19:16:17 +0300 Message-ID: <557DA8C0.1020209@mellanox.com> Date: Sun, 14 Jun 2015 19:16:00 +0300 From: Max Gurtovoy User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: , Sagi Grimberg , Oren Duer , Hans Petter Selasky Subject: gmultipath HA over iscsi/iser X-Originating-IP: [10.223.0.78] X-EOPAttributedMessage: 0 X-Microsoft-Exchange-Diagnostics: 1; DB3FFO11FD050; 1:RtXU2Ek7/rFXoKftJ2X2xAjZAN/IBa1ALqX3pFRr/RWArAjgxfcAOBI2C+8D+dB+HrAS5PxddVSw1EqCMP8lu8LPXKbdwmVt/y+oKh4r83c1VbtGzKK494Ul2ie8Imj4DM3Vaf2FObn2wnOabSqU9gVSG1TMUPM57cKAOf7IRxqQ3IVnNogotoIg0FeZDEQGcdU7I4CKhhaz57UrjYUjFbSQ27aqZtGcUyVWiTjoTWSdtPgkWaqWHa1uhuwcVVx10bShOr/ijoz7FBoCWt88eRwTr74GjR/SfxfIjE03syE= X-Forefront-Antispam-Report: CIP:193.47.165.134; CTRY:IL; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(10009020)(428002)(199003)(189002)(164054003)(16236675004)(101416001)(59896002)(92566002)(189998001)(77096005)(450100001)(62966003)(77156002)(46102003)(33656002)(36756003)(107886002)(65956001)(65806001)(5001920100001)(105586002)(6806004)(87936001)(19625215002)(84326002)(50986999)(83506001)(54356999)(4001350100001)(87266999)(86362001)(65816999)(5001770100001)(106466001)(19580395003)(229853001)(3940600001)(4001450100001)(579004)(559001); DIR:OUT; SFP:1101; SCL:1; SRVR:AMSPR05MB161; H:mtlcas13.mtl.com; FPR:; SPF:None; MLV:sfv; A:1; MX:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; AMSPR05MB161; 2:+mzFA92Z4OVnGTMEA+XfOeI6nRuzxV564bV9Q88w1tWUSMg/AXHi7MxufzWYD6o2; 2:X1Fdemxy4RFo398rYJ7AAb5TH9J8t9m8xqo7p6esk/vvMdJJPUl9Q3bSr5j6eBY+HEAidbNubYNEeU7WTUaQ9Pga+29tUxfmts9nSo8rCDvmIIUx75NY8zW+arGWV3DEQJvrebD9r8WZDHGO2mMsPYAE19EGcc3SZfKZV3X/c4a7Ytn/Ibi5zd2Reli2AMVSW0ocevXXYFaIKVJIykR3Bl+K/6YNs/sLQjyyQ8+LZdN5rq5AdFfE5qKNCH4m2iCM; 6:eKoFTAmCmabwONV/sN4VMFnDXOeJIbJWQh6Nrrj0JvO6GmLCMcQawwz5mZjkXqwNbEAWVlLqzPyQXk3RWMYkGKa1J+/3k1KoxpFos0MgMhHFw2vCGZoLKjxkeI3xIbr4iqv16ChDLVKsaUx1MNRbEfNNYL0AbUTujG6zskOikcmNlsR0Q2BblCqJcCKj2tAGP1V+GRLRv0ZjhajpmS5YzYcQ/QczEz4C7Bw8fVNf/QyacIudeDB8667rLlCriqXzZPE5KD4aeGCN1zhowmpjx28UzNwAqFyxo2W2Qt/vanWW+6C4AyMqwV6UmzRDSYe/e0proasRo4Eb+gzpkIK9K3HqDkh6fgYNpc5LVAcSExVvzhmM7WCZskIjG7Fj/xrtuNTzx2DPqIsg6wFk9UFqRm0PWsfvVSQTJcUJ6PdkdTSht623NpNVZGMCp02MpqtnnGNapS2Avo0mw0xUmfMwpJrdxFEdO29PSJLM11bA/c0YHpP9N2wQ5HxIK/pHrxPX X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:AMSPR05MB161; X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(520003)(5005006)(3002001); SRVR:AMSPR05MB161; BCL:0; PCL:0; RULEID:; SRVR:AMSPR05MB161; X-Microsoft-Exchange-Diagnostics: 1; AMSPR05MB161; 3:8WQLDQw1s6eEFkbXGpyouGK9RG+XMtRDYFxjVbxCPrifrKcsJLfgeiFQ7ZPO1cTVb2SD3ruYPRPDpYbbnS+XBuW3tY8tOCn47Qd7c9qXJtJkyhiLYJbZ55TOX8WPkIJSJHgGtNWDNVLOW15mXX8/3M2zjRs3MKxL96nSi1vnwh3ixkxLEmwXtnR8vUSJaXOE7RbWW9UjmTGK1VL1JbDs5aUzvusfR/Tlwx3m6StkvFJua9OD1ALpKaLRXycLIfdhqiNXLRhSn33wrWTuEsVkb+c5h647Tmj7x9p9TjUNmiSYswYKFJOpYw5iBuwcdSj+ X-Forefront-PRVS: 06070568C5 X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; AMSPR05MB161; 9:aCdUyOw5N8g7dh63MMiI/IwIzBIrhMHtv0yGudCh1fR?= =?us-ascii?Q?wWxAFRpTsVrwerqf5H/LbyUxMa7puDSSAKIjjrPJ2CCaE1lQhPB/opKU46YL?= =?us-ascii?Q?Y9Uxlwia3gW5ZJOJREsVzn+CR9kkZDBhS2pcCjV3ucvPBZsropI5hT4TUSpd?= =?us-ascii?Q?d85kuPh+YGcyrHhoPimdaoMgDuZmnbJHwxtvRsmmEK2hWNK+yPvz02ZBpU46?= =?us-ascii?Q?MaE7nbjJK74O706Mbd7lFxw2B7eiTblhfwN78wVleBRBokiF977xDkIoqh8g?= =?us-ascii?Q?HrO3vzzuEJvF+Qg4CepNr1cqvcByzLF+Lap7/TaeKNQLDYD68ua+Kp1HDWr+?= =?us-ascii?Q?8lx+K/gvCaav72zO048w5Zaj+oFvF3i3kc530NJrtF+v7pIjR6P/ofS1KSmu?= =?us-ascii?Q?vjENO7tp2lxmVjTBryG/x3NiC9Bx0pNniohRSDn660/VbCudl/nCRN9V9+ZL?= =?us-ascii?Q?QK7b7fUWQyirzGOnbqiFZCfkVeCvg4EykX3jf2RzPS/Oe5kfIv97FNJBLyfQ?= =?us-ascii?Q?7sZnROaZCM21xf0OSlJvvJuxk0PNGmpsL+AbVcjmEgQDvlWTR8Coo7YPHitJ?= =?us-ascii?Q?NeyqKkYl7JTaUfvwgmFE4OGiWZBMdwdYV6552cDvRskuVznbu2Jbo3W1hFtU?= =?us-ascii?Q?6Wyq4Je6e/gfcIm9HErvd7VthsOehLWpTX29UAvOhYCfL4hStVa6STE327hf?= =?us-ascii?Q?vysk4LQ1KmY9h1hCIeIx/1RSgxUikxmXIaxARVTGYH9Xyi65CeKT4IC4xF1L?= =?us-ascii?Q?2uLIQ8szWUt0SYDVETo6EnuYh3HXTgCairmRjE2zBKJ+l+xtjweg1Mzp6CB6?= =?us-ascii?Q?2DLuMiupmCiWeHPRr6V16SnR+DTQKm91nRA90b1lStcTysGJYMBlumgXqaim?= =?us-ascii?Q?g2aUogT+TzfQzyas81HhjZKN7RClo/mEWymI4Dowo8Vw2Ij3FF1jTsYpFhxi?= =?us-ascii?Q?wgTzcEfw5+0WZRxDd0kA02cm3tSoj0ZBPDf2386wyCNVJD2jeJ5SYta7h+Gs?= =?us-ascii?Q?V61TUaTYeZ2OnYX98/fjfmL9h1E+JeHAsSwi9KeocNv/1sbpd8WMTk6EjcHY?= =?us-ascii?Q?f/Ls=3D?= X-Microsoft-Exchange-Diagnostics: 1; AMSPR05MB161; 3:OScjCrm9Z8voaxuQuKBahtO6TlNqCus47HUWb4XBm2a1he1WGy7y5KJwNNINqFM9++x/wjb4uw6x4VJNjxkI8x4bpRGJ26aqi6aRop37676zD3cBBjSAUITA88FD2JTjj/hDasKRfF+25bFvfBPX9Q==; 10:KPLDgzEKU5MrBYszjY0xgTzFUV/lteH8Uzzrf0BXdHpEoNDhq1tXRETqg5nSnMmsfPvkiS7LVXNUFkW9AAYNaDubJMO6qoMx7/aPjJEo484=; 6:FboOhOeZYz7YIgtW9qovkDKqQV6XUDcsvUE8flDQW2lQgsdhtezzSAge0fPmu5HV X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Jun 2015 16:16:10.5131 (UTC) X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=a652971c-7d2e-4d9b-a6a4-d149256f461b; Ip=[193.47.165.134]; Helo=[mtlcas13.mtl.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AMSPR05MB161 Content-Type: text/plain; charset="windows-1255"; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Jun 2015 21:50:37 -0000 Hello, lately I was testing HA using gmultipath utility over iSCSI/iSER devices. I'm working on 11-current code base. I created 1 LUN on the target side and connected via 2 different physical ports from the initiator side. On the initiator side I see see /dev/da0 and /dev/da1. I created multipath device using: gmultipath label dm0 /dev/da0 /dev/da1. Now I have new device /dev/multipath/dm0. I set kern.iscsi.fail_on_disconnection=1 (to fail IO fast). Issue 1: ------------- I can't run simple fio/dd traffice over /dev/da0 nor /dev/da1. The only traffic that possible is using the multipath device dm0. Is this by design ? In the linux implementation we can run traffic on both block devices and multipath devices. Issue 2: -------------- I run some fio traffic utility over multipath device dm0 on initiator side with port toggling in a loop Port 1 down --> sleep 2 mins (iSCSI/ISER device reconnecting meanwhile with no success) --> port 1 up --> sleep 5 mins (iSCSI/ISER device reconnecting successecfully) Port 2 down --> sleep 2 mins (iSCSI/ISER device reconnecting meanwhile with no success) --> port 2 up --> sleep 5 mins (iSCSI/ISER device reconnecting successecfully) The expected result is that when the port N is down than the traffic moves to the available port and continue succesfully. I run this test for many hours and traffic FAILED (even though there was at least 1 suitable path between initiator and target). log: # gmultipath status Name Status Components multipath/dm_tcp OPTIMAL da0 (ACTIVE) da1 (PASSIVE) multipath/dm_iser OPTIMAL da2 (ACTIVE) da3 (PASSIVE) # fio ..... (over /dev/multipath/dm_iser or /dev/multipath/dm_tcp) fio: this platform does not support process shared mutexes, forcing use of threads. Use the 'thread' option to get rid of this warning. task1: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=psync, iodepth=8 ... task1: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=psync, iodepth=8 fio-2.1.3 Starting 8 threads fio: pid=101071, err=6/file:filesetup.c:575, func=open(/dev/multipath/dm_tcp),error=Device not configured task1: (groupid=0, jobs=8): err= 6 (file:filesetup.c:575, func=open(/dev/multipath/dm_tcp), error=Device not configured): pid=101071: Thu Jun 11 17:25:47 2015 read : io=296400MB, bw=32122KB/s, iops=8030, runt=9448911msec clat (usec): min=131, max=5541.8K, avg=504.40, stdev=5660.23 lat (usec): min=132, max=5541.8K, avg=504.55, stdev=5660.23 clat percentiles (usec): | 1.00th=[ 251], 5.00th=[ 298], 10.00th=[ 330], 20.00th=[ 370], | 30.00th=[ 406], 40.00th=[ 446], 50.00th=[ 478], 60.00th=[ 510], | 70.00th=[ 540], 80.00th=[ 580], 90.00th=[ 644], 95.00th=[ 700], | 99.00th=[ 1448], 99.50th=[ 1704], 99.90th=[ 1976], 99.95th=[ 2064], | 99.99th=[ 2256] bw (KB /s): min= 2, max= 5576, per=12.64%, avg=4060.97, stdev=352.37 write: io=295596MB, bw=32034KB/s, iops=8008, runt=9448911msec clat (usec): min=125, max=5541.8K, avg=490.13, stdev=5143.96 lat (usec): min=125, max=5541.8K, avg=490.41, stdev=5143.96 clat percentiles (usec): | 1.00th=[ 239], 5.00th=[ 282], 10.00th=[ 310], 20.00th=[ 354], | 30.00th=[ 390], 40.00th=[ 426], 50.00th=[ 466], 60.00th=[ 502], | 70.00th=[ 532], 80.00th=[ 572], 90.00th=[ 628], 95.00th=[ 692], | 99.00th=[ 1432], 99.50th=[ 1688], 99.90th=[ 1960], 99.95th=[ 2040], | 99.99th=[ 2256] bw (KB /s): min= 3, max= 5512, per=12.64%, avg=4049.74, stdev=355.11 lat (usec) : 250=1.29%, 500=56.84%, 750=38.78%, 1000=0.94% lat (msec) : 2=2.08%, 4=0.07%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, >=2000=0.01% cpu : usr=0.61%, sys=4.33%, ctx=151634083, majf=0, minf=3 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=75878522/w=75672554/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): READ: io=296400MB, aggrb=32121KB/s, minb=32121KB/s, maxb=32121KB/s, mint=9448911msec, maxt=9448911msec WRITE: io=295596MB, aggrb=32034KB/s, minb=32034KB/s, maxb=32034KB/s, mint=9448911msec, maxt=9448911msec # gmultipath status Name Status Components multipath/dm_tcp DEGRADED da1 (ACTIVE) multipath/dm_iser DEGRADED da3 (ACTIVE) We can see that there is Active paths to multipath device but still traffice failed. Any suggestions ? Anyone saw this before ? Thanks, Max Gurtovoy. Mellanox Technologies. From owner-freebsd-scsi@FreeBSD.ORG Mon Jun 15 17:18:08 2015 Return-Path: Delivered-To: freebsd-scsi@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1C033A23 for ; Mon, 15 Jun 2015 17:18:08 +0000 (UTC) (envelope-from syoung@switchways.com) Received: from mail-pa0-f44.google.com (mail-pa0-f44.google.com [209.85.220.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E6E6B90D for ; Mon, 15 Jun 2015 17:18:07 +0000 (UTC) (envelope-from syoung@switchways.com) Received: by padev16 with SMTP id ev16so69522555pad.0 for ; Mon, 15 Jun 2015 10:18:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:subject:date:message-id:mime-version :content-type:thread-index:content-language; bh=CU67jsYGD7s+YFVGwAXHDK5i4kW622x7pqoDM+4d8vo=; b=hKexFnrquepHiRfP8dM+Gko4aJZIwJG1+t+jIN9FFhH0+vw4V1leVRZ70lO58CqyF+ a0mNy3NqyvWtfAwLS2PoqoZGTbIpnd5kCOL2li5/bhfnMORHADWhEU9WcY9tE9tEL/Wj 9wr2LbpxAKxarSGiMH3dGEswyVvmQjEH9TYn1ew8hYeZJcKTMYUM9FQGPl7RZEFiqLiy DNr2nWhkrOe6cZk6+KIb2rJNs7nfDoBn4VGOQ1EmjV6w5F3bvdTduvV+BRpsC5fOe5sJ xKqpUN75J2OYF82jX8CU6a6YL/vBs/S5XJSa+PFEefy3nU/AOiSoD2Tzu6B08/I6Nm7N GeCg== X-Gm-Message-State: ALoCoQmy5ADE/SKW5SOi89Wbv+ulRjoSzyP5q9FmUhMWcNVlACyCak1z08ja7hkD6XDuN3f/X5In X-Received: by 10.70.88.145 with SMTP id bg17mr50506074pdb.167.1434388680906; Mon, 15 Jun 2015 10:18:00 -0700 (PDT) Received: from IIPLPraveen ([103.6.157.158]) by mx.google.com with ESMTPSA id k9sm5308328pbq.71.2015.06.15.10.17.57 for (version=TLSv1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 15 Jun 2015 10:17:58 -0700 (PDT) From: "Shaun Young" To: Subject: Windows Server Users List Date: Mon, 15 Jun 2015 13:13:30 -0400 Message-ID: <097a01d0a78f$39733d80$ac59b880$@com> MIME-Version: 1.0 X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: AdCnjn0ggftz709bT7e5TnqJ6XZ8Gg== Content-Language: en-us X-Antivirus: avast! (VPS 150615-0, 06/15/2015), Outbound message X-Antivirus-Status: Clean Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Jun 2015 17:18:08 -0000 Hi, Would you be interested in acquiring an Email list of "Windows Server Users" from USA? We have data for Microsoft BizTalk Server Users, Microsoft Exchange Server Users, Microsoft FAST Search Server Users, Microsoft Live Communication Server Users, Microsoft Office PerformancePoint Server Users, Microsoft Systems Management Server Users, Microsoft Windows Cluster users, Microsoft Windows XP Users, Microsoft Windows 8 Users, Microsoft Windows Azure Users, Microsoft Windows Virtual PC Users, iPhone Users, iTune Users, Security Software Users, Virtualization Technology Users, CRM Users, Telecommunication Users, Servers Users, Programming Tools Users and many more.. Each Record in the List includes: Company, Name, First Name, Last Name, Title, HQ Phone, Direct No, Email, Address1, Address2, City, State, Zip, Country, Industry, Revenue, Employees, IT Budget, IT Employees, Website, Technology, Company HQ Address1, Company HQ Address2, Company HQ City, Company HQ State, Company HQ Zip, Company HQ Country and LinkedIn link. All the contacts are opt-in verified, 100% permission based and can be used for unlimited multi-channel marketing. Please let me know your thoughts towards procuring the "Windows Server Users List" . Please brief us on the type of emails you wish to target. Look forward to your prompt response. Thanks & Regards Shaun Young Research Analyst If this message is not relevant to you please forward to decision maker.We respect your privacy, if you do not wish to receive any further emails from our end, please reply with a subject "Leave Out". --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus From owner-freebsd-scsi@FreeBSD.ORG Thu Jun 18 08:20:16 2015 Return-Path: Delivered-To: freebsd-scsi@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 73542D6D for ; Thu, 18 Jun 2015 08:20:16 +0000 (UTC) (envelope-from maxg@mellanox.com) Received: from emea01-db3-obe.outbound.protection.outlook.com (mail-db3on0091.outbound.protection.outlook.com [157.55.234.91]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9A6025EA for ; Thu, 18 Jun 2015 08:20:14 +0000 (UTC) (envelope-from maxg@mellanox.com) Received: from DB3PR05CA0020.eurprd05.prod.outlook.com (10.160.41.148) by AMXPR05MB165.eurprd05.prod.outlook.com (10.242.72.139) with Microsoft SMTP Server (TLS) id 15.1.195.15; Thu, 18 Jun 2015 08:05:46 +0000 Received: from AM1FFO11OLC007.protection.gbl (2a01:111:f400:7e00::117) by DB3PR05CA0020.outlook.office365.com (2a01:111:e400:9428::20) with Microsoft SMTP Server (TLS) id 15.1.195.15 via Frontend Transport; Thu, 18 Jun 2015 08:05:46 +0000 Authentication-Results: spf=none (sender IP is 193.47.165.134) smtp.mailfrom=mellanox.com; freebsd.org; dkim=none (message not signed) header.d=none; Received-SPF: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) Received: from mtlcas13.mtl.com (193.47.165.134) by AM1FFO11OLC007.mail.protection.outlook.com (10.174.64.133) with Microsoft SMTP Server (TLS) id 15.1.184.11 via Frontend Transport; Thu, 18 Jun 2015 08:05:44 +0000 Received: from MTLCAS13.mtl.com (10.0.8.78) by mtlcas13.mtl.com (10.0.8.78) with Microsoft SMTP Server (TLS) id 15.0.775.38; Thu, 18 Jun 2015 11:05:25 +0300 Received: from MTLCAS01.mtl.com (10.0.8.71) by MTLCAS13.mtl.com (10.0.8.78) with Microsoft SMTP Server (TLS) id 15.0.775.38 via Frontend Transport; Thu, 18 Jun 2015 11:05:25 +0300 Received: from [10.223.0.78] (10.223.0.78) by MTLCAS01.mtl.com (10.0.8.71) with Microsoft SMTP Server (TLS) id 14.3.123.3; Thu, 18 Jun 2015 11:05:24 +0300 Message-ID: <55827BBF.6040206@mellanox.com> Date: Thu, 18 Jun 2015 11:05:19 +0300 From: Max Gurtovoy User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: , Sagi Grimberg , Oren Duer , Hans Petter Selasky Subject: Re: gmultipath HA over iscsi/iser References: <557DA8C0.1020209@mellanox.com> In-Reply-To: <557DA8C0.1020209@mellanox.com> Content-Type: text/plain; charset="windows-1255"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [10.223.0.78] X-EOPAttributedMessage: 0 X-Microsoft-Exchange-Diagnostics: 1; AM1FFO11OLC007; 1:IJwhiUyei1igiwphiM3pU57+IDbjCRyKbaxzU+dzJt/iopgwW4rQ0872XV/CyMpLd0PfRvYOs0RaRLBZWaKbcUlj/yX2bqQWODW+CRzjDzu+YvmBMDWkd7MJgoEaPzkjr3HFVll+y/yBNV3tP6Q6JWIAqhPfrFtN3P2BCSOl6YLqiDW85HNo0QeUIuipw/4awlpRGTWFZDrtUAphi7i7jStWalyonUoLOmebC+ALXvy6e29HzqWqKEvUGI59NiZInmwFW0/5s3DZC47m5ATjyQ== X-Forefront-Antispam-Report: CIP:193.47.165.134; CTRY:IL; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(10009020)(6009001)(428002)(199003)(51704005)(24454002)(377454003)(164054003)(479174004)(189002)(77096005)(50466002)(2950100001)(189998001)(105586002)(6806004)(80316001)(92566002)(4001350100001)(106466001)(107886002)(5001770100001)(83506001)(101416001)(46102003)(65806001)(65956001)(47776003)(54356999)(65816999)(76176999)(450100001)(87266999)(50986999)(33656002)(87936001)(86362001)(77156002)(62966003)(36756003)(3940600001)(4001450100001); DIR:OUT; SFP:1101; SCL:1; SRVR:AMXPR05MB165; H:mtlcas13.mtl.com; FPR:; SPF:None; MLV:sfv; A:1; MX:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; AMXPR05MB165; 2:Mcn8JoD9OKr1D5WkOqYYDsqfTRSyhLz9KbUTdOdiZlKS4KrLFHdsS9F5ACrJpt7Z; 3:x1n777pW4A5pSWk3huehnjrCHcRxvuIwbE+WQ1DwfdGi8Gh1eaAzVKTploxOHKis3rM6IRceLXtK7N5+GAPxlwPtO/2l8D+ajrEIvO2UhmDU8KA7TNl+/HRDdVi9BnG3mPrMFP9AH/Hyp6lJU6m1loIT1oa9UDO/ayffnD29P8RnUdbHJrxXdlPFupdf2XVPxMMW/m6XYjJXnbp9lrRJ0yduOwGZYK2I0bUnRQ3SwbPhdFW+QKH+eDzm6u7YbV0I; 20:HU0krm7INUUR0rZgXLgF/h1qgRjLcjn4fuHxLwyKbqtW/XX871vdYlRWdfvKbIDTe+W8q0tyGh56iBihXqEg8yN7/7xZ1C0EWFfTwSe5i+Sj0mfNa0Ug1VJnW0yUoCKzGzaZ14WZbc22DJy9HxBbdW+dmpR1Y3KQmWs8XMr9TMPAnCEtRwQEKgYcNxQcITYIZGrg/mhWvlMGS5oQFK+hW+RakiO+fVjAnYIjEVZwUkdFaRpV3Ks3JHLFDQNQcd2SmdLT6gPWwcyI92mOvIz38rPrfCF15FbC9zJ9kD2cghfS3GEBU2vewLc0sOkIL0p1bDuhxupglpmRv/ky3/TLD5YmW8hmBd9N0CFKpREDf6TNXlC9AQJsY80yZ0Zkw0SVg1L4CAqEK5ugAMGjt6yH/sGnEObMeEbATaqQSkxv581alvl7sWcEKBcQRHE7NoV6qnCK/MEaSMucxCqP4kgzOiLQ9Af1LwYC/fhx8UD8vTPVgf/iVO/CwH1Md5jMhY7l X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:AMXPR05MB165; X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(5005006)(3002001); SRVR:AMXPR05MB165; BCL:0; PCL:0; RULEID:; SRVR:AMXPR05MB165; X-Microsoft-Exchange-Diagnostics: 1; AMXPR05MB165; 4:nx18qpledMG6ItTo1+4aMx73VCuPnNbt4cn0tjUabqxF3GYXELESbp66LX30OSWOypSta4vL0n7Cdwx/vlLENJ+ICAWMIoskYldPfpQr9NG4oNzNPo0R452Q1ED7laeMJecB5VLsumAjPdX2frt1a+37S4hqcUL7tnRw12RtUbO3JGQ5F5HjFBX5rUPlA1GPXVFp0CMstoZ3FSLuDMR/8gctAy8ritay3r8/GfZ5MYVTi6W38GygKBEvlRMFg5pZm7lEIETQtLccyXxQRrX7/gF5smCcYOeM9Tuf/cTMLS8= X-Forefront-PRVS: 0611A21987 X-Microsoft-Exchange-Diagnostics: =?windows-1255?Q?1; AMXPR05MB165; 23:PT96L67e1HMGwr8ZYv8OTtQYQAy43zHd/w6iyJ?= =?windows-1255?Q?pCxTKuTfiWT4yHDB4LEq06ErVguqHSbrFAYIoPN0DeL///Zx1f0ABmnd?= =?windows-1255?Q?2DWSzPev/HmJiKKOCK2XOd+rEArOWn2W+duVC+Dm2TPvxPDQFmbORuje?= =?windows-1255?Q?EJHrpgbq8YCwpTO5GCRtmoP2SwnSE9+7H/HQJGyFgzKNkjBBe9B2TaEs?= =?windows-1255?Q?oudFckWCYTTjk82JN1ZLx69c6DWaSlLyH13j5m+o2QOfwNtWHH7s7JsG?= =?windows-1255?Q?NCA4S/xCOX2uizM+xJLhfR9b5vu3ElOB6mzHDLWyQQbpr4ZsMJHJNRJ/?= =?windows-1255?Q?E3aAqwureUYLd5wQQchUZdGWvGIb/016nFCu+DftSk5ZcNdTqbEQEd36?= =?windows-1255?Q?zhUkXWV3UshUJHCy/cg910qQtyF+/AezUioPVMqEHD65PoqAZs3GG4FD?= =?windows-1255?Q?K8mGi1dHk95McCIxWq43saPJLTrxVKv6lVMoPH4cjqiFIkKG/facSXPw?= =?windows-1255?Q?xfT8PiLdTISBxOTPIN7tFZGeBVVCQmrzh03me3SgOKBOoLwTTJOxHSby?= =?windows-1255?Q?VH3vyZDdV+QiM1fnghSAOXpLmpKQQwGRUEyMiclWfUGFNv8HZR/Smzyj?= =?windows-1255?Q?I2m2OGWfxI5bIg5PdZOKYRZhsV9Ov1AvWZq3y8zCpa0Tu5uOX+L69odB?= =?windows-1255?Q?0YZKDQi+Vud9a7ahnRLP3/Ufy07urE3dslimGRp/Bvz25ArWecNUGK08?= =?windows-1255?Q?W98b2gBOyjCFMk+Sek7ROc+E/gL6qX1HNEXW2hN9SHa4NL/ZZ6p9hQJP?= =?windows-1255?Q?NHloZw0i56IDxctaP4DgY5Js2FoVrEjO86rgOrcAVngB21wvWuKRl5Vq?= =?windows-1255?Q?7Vl0o4NdDZTxjthH/M1/0mLuNJVjX5b+GUe5clnqL6BHDNBqvVhWAXjo?= =?windows-1255?Q?8NjC1SR2OyGwrU6ygKaiDlS7VBdm8u3pkwmfTyai6CFB3NoPbAZgZIxR?= =?windows-1255?Q?0Xn/eszSqS04QBA2/R67TU6YDQ0I3zpvwhstfM1SwOj2VhKezNfp4qPy?= =?windows-1255?Q?bolIzllL54aSnRisc0lkcplJ6vRXdaT9q2mI/VN1Ezvzo3dpSr/d42Uu?= =?windows-1255?Q?2U9oUKWlHj8o/1Uy6oPOI=3D?= X-Microsoft-Exchange-Diagnostics: 1; AMXPR05MB165; 5:XjwIbM7niG+Gk3cmtd7sKBVR+cIb8l+ZXKbyKDNH5viEQM9PC7AbKEbtiVktX+DRRYCRyXvVVd9SuwJbJlcWpzekLCMzBLcTCha4ZcaJ39unzZSNYLJ0RxDKiFRmvJOxXZBXZrLns849cDQbjuD3Ww==; 24:KDx55JoGHLwk6bPMmmHrpd4K5ciOkSK4qGpCeuyz5BRpeMBTIVf1Ey3i3KgeETl4t+mz5BKmss0WR2eWvTFi4vT+vrrClaoTxBIxicaAOFA=; 20:Ztn5woKdaB+w+mw5E97+2b8m2Gv3SALI7mJM9T/sTNLnb33LPqLXPFVvtATUGjOHikLDidzQ4qb0smzWdidngw== X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jun 2015 08:05:44.7525 (UTC) X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=a652971c-7d2e-4d9b-a6a4-d149256f461b; Ip=[193.47.165.134]; Helo=[mtlcas13.mtl.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AMXPR05MB165 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Jun 2015 08:20:16 -0000 Anyone checked gmultipath utility ? Thanks, Max. On 6/14/2015 7:16 PM, Max Gurtovoy wrote: > Hello, > lately I was testing HA using gmultipath utility over iSCSI/iSER devices. > I'm working on 11-current code base. > I created 1 LUN on the target side and connected via 2 different > physical ports from the initiator side. > On the initiator side I see see /dev/da0 and /dev/da1. > I created multipath device using: > gmultipath label dm0 /dev/da0 /dev/da1. > Now I have new device /dev/multipath/dm0. > I set kern.iscsi.fail_on_disconnection=1 (to fail IO fast). > > Issue 1: > ------------- > I can't run simple fio/dd traffice over /dev/da0 nor /dev/da1. > The only traffic that possible is using the multipath device dm0. > Is this by design ? > In the linux implementation we can run traffic on both block devices > and multipath devices. > > Issue 2: > -------------- > I run some fio traffic utility over multipath device dm0 on initiator > side with port toggling in a loop > > Port 1 down --> sleep 2 mins (iSCSI/ISER device reconnecting meanwhile > with no success) --> port 1 up --> sleep 5 mins (iSCSI/ISER device > reconnecting successecfully) > Port 2 down --> sleep 2 mins (iSCSI/ISER device reconnecting meanwhile > with no success) --> port 2 up --> sleep 5 mins (iSCSI/ISER device > reconnecting successecfully) > > The expected result is that when the port N is down than the traffic > moves to the available port and continue succesfully. > I run this test for many hours and traffic FAILED (even though there > was at least 1 suitable path between initiator and target). > > log: > > # gmultipath status > > Name Status Components > > multipath/dm_tcp OPTIMAL da0 (ACTIVE) > > da1 (PASSIVE) > > multipath/dm_iser OPTIMAL da2 (ACTIVE) > > da3 (PASSIVE) > > > # fio ..... (over /dev/multipath/dm_iser or /dev/multipath/dm_tcp) > > > fio: this platform does not support process shared mutexes, forcing > use of threads. Use the 'thread' option to get rid of this warning. > > task1: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=psync, iodepth=8 > > ... > > task1: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=psync, iodepth=8 > > fio-2.1.3 > > Starting 8 threads > > fio: pid=101071, err=6/file:filesetup.c:575, > func=open(/dev/multipath/dm_tcp), error=Device not configured > > task1: (groupid=0, jobs=8): err= 6 (file:filesetup.c:575, > func=open(/dev/multipath/dm_tcp), error=Device not configured): > pid=101071: Thu Jun 11 17:25:47 2015 > > read : io=296400MB, bw=32122KB/s, iops=8030, runt=9448911msec > > clat (usec): min=131, max=5541.8K, avg=504.40, stdev=5660.23 > > lat (usec): min=132, max=5541.8K, avg=504.55, stdev=5660.23 > > clat percentiles (usec): > > | 1.00th=[ 251], 5.00th=[ 298], 10.00th=[ 330], 20.00th=[ 370], > > | 30.00th=[ 406], 40.00th=[ 446], 50.00th=[ 478], 60.00th=[ 510], > > | 70.00th=[ 540], 80.00th=[ 580], 90.00th=[ 644], 95.00th=[ 700], > > | 99.00th=[ 1448], 99.50th=[ 1704], 99.90th=[ 1976], 99.95th=[ 2064], > > | 99.99th=[ 2256] > > bw (KB /s): min= 2, max= 5576, per=12.64%, avg=4060.97, > stdev=352.37 > > write: io=295596MB, bw=32034KB/s, iops=8008, runt=9448911msec > > clat (usec): min=125, max=5541.8K, avg=490.13, stdev=5143.96 > > lat (usec): min=125, max=5541.8K, avg=490.41, stdev=5143.96 > > clat percentiles (usec): > > | 1.00th=[ 239], 5.00th=[ 282], 10.00th=[ 310], 20.00th=[ 354], > > | 30.00th=[ 390], 40.00th=[ 426], 50.00th=[ 466], 60.00th=[ 502], > > | 70.00th=[ 532], 80.00th=[ 572], 90.00th=[ 628], 95.00th=[ 692], > > | 99.00th=[ 1432], 99.50th=[ 1688], 99.90th=[ 1960], 99.95th=[ 2040], > > | 99.99th=[ 2256] > > bw (KB /s): min= 3, max= 5512, per=12.64%, avg=4049.74, > stdev=355.11 > > lat (usec) : 250=1.29%, 500=56.84%, 750=38.78%, 1000=0.94% > > lat (msec) : 2=2.08%, 4=0.07%, 10=0.01%, 20=0.01%, 50=0.01% > > lat (msec) : 100=0.01%, >=2000=0.01% > > cpu : usr=0.61%, sys=4.33%, ctx=151634083, majf=0, minf=3 > > IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, > >=64=0.0% > > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, > >=64=0.0% > > complete : 0=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, > >=64=0.0% > > issued : total=r=75878522/w=75672554/d=0, short=r=0/w=0/d=0 > > Run status group 0 (all jobs): > > READ: io=296400MB, aggrb=32121KB/s, minb=32121KB/s, maxb=32121KB/s, > mint=9448911msec, maxt=9448911msec > > WRITE: io=295596MB, aggrb=32034KB/s, minb=32034KB/s, maxb=32034KB/s, > mint=9448911msec, maxt=9448911msec > > > # gmultipath status > > Name Status Components > > multipath/dm_tcp DEGRADED da1 (ACTIVE) > > multipath/dm_iser DEGRADED da3 (ACTIVE) > > > We can see that there is Active paths to multipath device but still > traffice failed. > Any suggestions ? Anyone saw this before ? > > Thanks, > Max Gurtovoy. > Mellanox Technologies. > > > > > > > From owner-freebsd-scsi@FreeBSD.ORG Fri Jun 19 16:11:06 2015 Return-Path: Delivered-To: freebsd-scsi@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C6A43F5D for ; Fri, 19 Jun 2015 16:11:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B0EEA92C for ; Fri, 19 Jun 2015 16:11:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t5JGB6XL045187 for ; Fri, 19 Jun 2015 16:11:06 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-scsi@FreeBSD.org Subject: [Bug 191717] [isci] smartctl -H gives "ATA output registers missing" for a disk using the isci driver Date: Fri, 19 Jun 2015 16:11:05 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: martin@lispworks.com X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-scsi@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Jun 2015 16:11:06 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191717 --- Comment #6 from martin@lispworks.com --- It still fails in the same way with smartctl svn. The machine is now running unpatched 10.1-RELEASE-p13. According to http://lists.freebsd.org/pipermail/freebsd-scsi/2014-July/006383.html on the same thread as the compatibility patch, the kernel is returning the wrong value for byte 7 in the fixed format sense data buffer, which should be 10 rather than 0. He seems to have a good point, so it looks like sati_scsi_fixed_sense_data_construct and sati_scsi_common_fixed_sense_construct need to byte 7, like sati_scsi_common_descriptor_sense_construct and sati_scsi_descriptor_sense_data_construct do. This is a production machine, so I'm not confident enough to just change this without additional input from someone who knows about this stuff in FreeBSD. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-scsi@FreeBSD.ORG Fri Jun 19 16:20:17 2015 Return-Path: Delivered-To: freebsd-scsi@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E3F1B3A3 for ; Fri, 19 Jun 2015 16:20:17 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "gold.funkthat.com", Issuer "gold.funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 9EF22B0D for ; Fri, 19 Jun 2015 16:20:17 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.14.5/8.14.5) with ESMTP id t5JGKFoP059126 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 19 Jun 2015 09:20:16 -0700 (PDT) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.14.5/8.14.5/Submit) id t5JGKFwe059125; Fri, 19 Jun 2015 09:20:15 -0700 (PDT) (envelope-from jmg) Date: Fri, 19 Jun 2015 09:20:15 -0700 From: John-Mark Gurney To: Max Gurtovoy Cc: freebsd-scsi@freebsd.org, Sagi Grimberg , Oren Duer , Hans Petter Selasky Subject: Re: gmultipath HA over iscsi/iser Message-ID: <20150619162015.GD96349@funkthat.com> References: <557DA8C0.1020209@mellanox.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <557DA8C0.1020209@mellanox.com> X-Operating-System: FreeBSD 9.1-PRERELEASE amd64 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.5.21 (2010-09-15) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (gold.funkthat.com [127.0.0.1]); Fri, 19 Jun 2015 09:20:16 -0700 (PDT) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Jun 2015 16:20:18 -0000 Max Gurtovoy wrote this message on Sun, Jun 14, 2015 at 19:16 +0300: > lately I was testing HA using gmultipath utility over iSCSI/iSER devices. > I'm working on 11-current code base. > I created 1 LUN on the target side and connected via 2 different > physical ports from the initiator side. > On the initiator side I see see /dev/da0 and /dev/da1. > I created multipath device using: > gmultipath label dm0 /dev/da0 /dev/da1. > Now I have new device /dev/multipath/dm0. > I set kern.iscsi.fail_on_disconnection=1 (to fail IO fast). > > Issue 1: > ------------- > I can't run simple fio/dd traffice over /dev/da0 nor /dev/da1. > The only traffic that possible is using the multipath device dm0. > Is this by design ? This is probably because geom only lets one writer open the device at a time, and when the dm0 device is open, it blocks any other writers to it's underlying devices... There is a special geom debug flag that you can set to avoid this... > In the linux implementation we can run traffic on both block devices and > multipath devices. > > Issue 2: > -------------- > I run some fio traffic utility over multipath device dm0 on initiator > side with port toggling in a loop > > Port 1 down --> sleep 2 mins (iSCSI/ISER device reconnecting meanwhile > with no success) --> port 1 up --> sleep 5 mins (iSCSI/ISER device > reconnecting successecfully) > Port 2 down --> sleep 2 mins (iSCSI/ISER device reconnecting meanwhile > with no success) --> port 2 up --> sleep 5 mins (iSCSI/ISER device > reconnecting successecfully) > > The expected result is that when the port N is down than the traffic > moves to the available port and continue succesfully. > I run this test for many hours and traffic FAILED (even though there was > at least 1 suitable path between initiator and target). Someone else will need to answer this one... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."