From owner-freebsd-scsi@FreeBSD.ORG Tue Jun 23 15:42:17 2015 Return-Path: Delivered-To: freebsd-scsi@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C039B26E for ; Tue, 23 Jun 2015 15:42:17 +0000 (UTC) (envelope-from maxg@mellanox.com) Received: from emea01-db3-obe.outbound.protection.outlook.com (mail-db3on0093.outbound.protection.outlook.com [157.55.234.93]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 19B8AB88 for ; Tue, 23 Jun 2015 15:42:16 +0000 (UTC) (envelope-from maxg@mellanox.com) Received: from AM3PR05CA023.eurprd05.prod.outlook.com (10.141.192.33) by DB3PR05MB217.eurprd05.prod.outlook.com (10.242.133.17) with Microsoft SMTP Server (TLS) id 15.1.195.15; Tue, 23 Jun 2015 15:42:14 +0000 Received: from DB3FFO11FD037.protection.gbl (2a01:111:f400:7e04::151) by AM3PR05CA023.outlook.office365.com (2a01:111:e400:882a::33) with Microsoft SMTP Server (TLS) id 15.1.201.16 via Frontend Transport; Tue, 23 Jun 2015 15:42:13 +0000 Authentication-Results: spf=none (sender IP is 193.47.165.134) smtp.mailfrom=mellanox.com; freebsd.org; dkim=none (message not signed) header.d=none; Received-SPF: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) Received: from mtlcas13.mtl.com (193.47.165.134) by DB3FFO11FD037.mail.protection.outlook.com (10.47.217.68) with Microsoft SMTP Server (TLS) id 15.1.190.9 via Frontend Transport; Tue, 23 Jun 2015 15:42:11 +0000 Received: from MTLCAS13.mtl.com (10.0.8.78) by mtlcas13.mtl.com (10.0.8.78) with Microsoft SMTP Server (TLS) id 15.0.775.38; Tue, 23 Jun 2015 18:42:29 +0300 Received: from MTLCAS01.mtl.com (10.0.8.71) by MTLCAS13.mtl.com (10.0.8.78) with Microsoft SMTP Server (TLS) id 15.0.775.38 via Frontend Transport; Tue, 23 Jun 2015 18:42:29 +0300 Received: from [10.223.0.78] (10.223.0.78) by MTLCAS01.mtl.com (10.0.8.71) with Microsoft SMTP Server (TLS) id 14.3.123.3; Tue, 23 Jun 2015 18:42:04 +0300 Message-ID: <55897E4B.8060608@mellanox.com> Date: Tue, 23 Jun 2015 18:42:03 +0300 From: Max Gurtovoy User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: John-Mark Gurney CC: , Sagi Grimberg , Oren Duer , Hans Petter Selasky Subject: Re: gmultipath HA over iscsi/iser References: <557DA8C0.1020209@mellanox.com> <20150619162015.GD96349@funkthat.com> In-Reply-To: <20150619162015.GD96349@funkthat.com> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [10.223.0.78] X-EOPAttributedMessage: 0 X-Microsoft-Exchange-Diagnostics: 1; DB3FFO11FD037; 1:DNxNDIfcdpMNNnBww7c8ZNGXZpizIfAmyElLfkz9Dd8sTH50CkvXR5GBzEWyoESU5gU1sTfWTksi0xIdQroT012s3sNxhAsEtts1RRY22DQ76d2mkvqMCw11FPoaE6YKnAAnkyHkmjrFIU1prnz9ZgnWMO6Mb0loAp+0w//wabMrq8LtSy7ac9sesmxKLfj6Z8epgWpJ7pIZ7jX7ytD2ugxPncgnb7zYiR4X+cvVdK0OLKyqaLmpfsqooCVe4jGacS2HgfRDQvA/Bt/PSxwV2WDfMZZpgDU0XikOKcQkRv8jj75RcWcwooHV5lGkca16 X-Forefront-Antispam-Report: CIP:193.47.165.134; CTRY:IL; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(10009020)(6009001)(2980300002)(428002)(189002)(51704005)(377454003)(479174004)(199003)(164054003)(24454002)(77096005)(105586002)(6806004)(110136002)(50466002)(189998001)(47776003)(65806001)(62966003)(65956001)(86362001)(77156002)(107886002)(36756003)(83506001)(33656002)(87936001)(46102003)(4001350100001)(106466001)(64126003)(23746002)(101416001)(92566002)(80316001)(99136001)(2950100001)(50986999)(59896002)(76176999)(87266999)(65816999)(54356999)(3940600001)(4001430100001); DIR:OUT; SFP:1101; SCL:1; SRVR:DB3PR05MB217; H:mtlcas13.mtl.com; FPR:; SPF:None; MLV:sfv; MX:1; A:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; DB3PR05MB217; 2:OVlV6G/a84fdR3NBqLmIN1xGYFpGr4yhKjstNh//nIivosc2+hW+m3MOiMU9RZbd; 3:0YTbQgVI/fMoAh7oXTwLV3i5TNktisXhEpt1AuSLLvWBFGby1Cu7es7MzbYlepa3O+p1VUYFY3YQDrz7Lg2GMWhPQEgD1jatbM9HHTzFqRNaknoz0mNYvq362oiVptaKT7tdkPbNZJ37adiHiKAYCddVC2bE3g2iMxLRKaoHPZJTrVfc6oi2EgCb4S2icihbOWUxT0FXXJ3MSFglK1vU8vUP6zl7K54DSoY8Iop3xxVmhVzVyfA7PBdShgUIR2KZ; 20:N45uZ/qQZWDorazQzdWwk79rEr5/f0FqRw2+IsY5hE3+ZjdnSSGZ4c1Be/Hk21dcCMeFna8Rd/JtesZ6dl30zRBgjSasciUdeTIOnPPisA5qyNwEMRp6odG2keNkvvLeop9CEQnP36uzLFKDr8EmT3LoBYCicwKQ7LYoTsgjJSokhuNc6AvSrMjhWIaywGcLXVpe6XxIbq7ZWWkaU0J21jRKuRK9NSZLOwKyN1Mjo+R9/jTx3jM5y+p3R74MW15cjLHnRoZqldoihxpDuT67azNye9K4KVo+p6kfRJMFwbwVY2t9h1BHK89+t2vjF4tAiMtL0v+jk2m5QKJ9iZKA+LBWKkQMgZngET5Qs3pYss+CkLF7Z9gvnlylpmkXEaHJnAYgyllegwgMHRuiS5+U8o0mBSyM73Um00w0nJrc0IXh5VBUIEhDTRMNnxggYmz9Ruo1NYyrJQZO+eNTyrgktgoCGTlDTEyxkQ9KOxltulYAgjTtcWK3IBho+TptQgeo X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:DB3PR05MB217; X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(5005006)(3002001); SRVR:DB3PR05MB217; BCL:0; PCL:0; RULEID:; SRVR:DB3PR05MB217; X-Microsoft-Exchange-Diagnostics: 1; DB3PR05MB217; 4:MwUN4ESZjfeJ47+f7bhxomchwfkncxFAJgR9oP5M9BqBFrbB84nISALuew7GRj460pDbZ5xVFE7fi/1TTJ5d2ac30mUYtrYtTNGBS6ZEjkTffQ9lIbrrcZCpgaIk8prOIisbX+GiHV3SfkaFRHgEoPFgse3SsnqgP7gcr+5f0KmcIn/S3YJPfgQGhB8faK+gVtHjfb0e6aL0/wFeJqJLMAWo8GrbfIAlSQ/a85k6sea1oDB5ERaCcMNsLI3vdQFZfdk98kHDmGS5XSVAj8cgHf7PvzRiPlVvVAxGIgToiaQ= X-Forefront-PRVS: 06167FAD59 X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1; DB3PR05MB217; 23:LTLdXOduEHGN1XpaGEsQiisp1eS9VGHMTBrsx8?= =?Windows-1252?Q?oFxGmyKHZ49mTCeZWxLC2FMj5p/F8Oi8AX+8r8lKygAPw7G/Q/XuQdYN?= =?Windows-1252?Q?ScLD+iw7sgE5sCp/PHwk10XlW9TzKgcx20/IHGtXE48PUMR4QN4eBnPJ?= =?Windows-1252?Q?RL3QHACJJNYeka72C3l/mAklJ33LJUXfnD3UF25uyyFBv1W52sCDczZK?= =?Windows-1252?Q?Cf2IB6LmV1hDDE/Pswh8uvcM3JGYNmQy9Db1ESljxpr9zJ09UN2imwzJ?= =?Windows-1252?Q?rXhyX22h9RDof3K3vnyXmf9MbR7KF1ZYWChlKBmJJf2NB5GPosUxcWVZ?= =?Windows-1252?Q?+9ws3MPJtVotI7anhObnJQnP2tB4eBvKHuEmHaTI9243DuEvHHP39XYr?= =?Windows-1252?Q?El3xmGAfN9YjO/CgTcMRqzK63EbQdTBw0NKDCe+fuzxWmPSvWELp9DN/?= =?Windows-1252?Q?IeaI1PDEPzSDZKYEh8gZxGhiB1g6waWK33URcI2BVLwlkNqEJsvA2BPm?= =?Windows-1252?Q?nEXzzs9XHLOKxyJGaMrr0Kt59Yqz9RrK0Nl3Yrie9OFl86nGkz79322o?= =?Windows-1252?Q?Hq+kdH/7iOV2Vcu6ZZOowW5T8C7hklal8sHAT50ACccklN12GTYnrFdw?= =?Windows-1252?Q?wrzHp7eg9JK12qabB1K1w2V391xLFaFIGnPU4WORHvQpsbbswBYRiYPW?= =?Windows-1252?Q?1h8JoWreYgpb9lx9aMlxBq9YNDB8dNJM8HG0NHS+VUmeilpHEeeJDsSN?= =?Windows-1252?Q?vSR/jHqGHWFmS4p6WB6Xzl2B4G7O3xcXgnICgPs5pdwmuJvBGbrdhuJq?= =?Windows-1252?Q?TG+jbS+dZHddRolxr84ZFg1U0KftvvO25A0dYAY3EAk62iynrLzezkDt?= =?Windows-1252?Q?eQ+CPItPZMzfL597Jz5nYSY2wEQJ0UApie55ORHUgxZoj18+yDYr42Ps?= =?Windows-1252?Q?c0Wyzs7j97mB9h49VYU9Di2ayhnlr1r0mJmQgPRiWw69omso/oYoqfi7?= =?Windows-1252?Q?/UoDrdBk0vEPvRSGB9AAezSpBIvLvuTYLYsHhZ09hdQ8pM2Gr8O6Qepl?= =?Windows-1252?Q?r1nnRzgWaXvqX2xvYh0RrQqL+TmtP75PS4/3zRuclkdTT43e4N1oAIFk?= =?Windows-1252?Q?k1wtPQDDfHiANYMzwdvkhzMMQZwgrssIGx6yWj823rwvr9qYAEnCmdAQ?= =?Windows-1252?Q?xSfiqD4BYg3Hhmn0Lu+XNrCz4EgNQEl3PlpX8PUbdh2+SHjW8M?= X-Microsoft-Exchange-Diagnostics: 1; DB3PR05MB217; 5:Oy2IJlvgTfAYYB33bHN3ep2SDKOEENy9GKWa3CPH7pfb7jBm+MWriPjLEtppS3fErTlpsmryjvbcTQsyUUiMv9b2lzzinXCj4Wud2dVX0UKd8vethpH7nUJaMOuQqhvL2zBbnniqCekkyEIlKA0yDw==; 24:TSunMG24EXjUwfD+ZeS4Mcm4Tl1pdwDju2jKtIzV+urP4LsaGw4oKY4q8oBWGQksUTg1xqeLhtNFhMau2bRHqHR4/XCPlqZ57opeDhkbZnA=; 20:mbmPN/myon+ANeMnkvsWifCouM97uQ9awQRiG5AvmmI8/EWabJZf5LvhKN6EK/6wlqxHdStjlwZnHI2mpb9g3A== X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Jun 2015 15:42:11.6962 (UTC) X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=a652971c-7d2e-4d9b-a6a4-d149256f461b; Ip=[193.47.165.134]; Helo=[mtlcas13.mtl.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB3PR05MB217 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jun 2015 15:42:17 -0000 Hi All, On 6/19/2015 7:20 PM, John-Mark Gurney wrote: > Max Gurtovoy wrote this message on Sun, Jun 14, 2015 at 19:16 +0300: >> lately I was testing HA using gmultipath utility over iSCSI/iSER devices. >> I'm working on 11-current code base. >> I created 1 LUN on the target side and connected via 2 different >> physical ports from the initiator side. >> On the initiator side I see see /dev/da0 and /dev/da1. >> I created multipath device using: >> gmultipath label dm0 /dev/da0 /dev/da1. >> Now I have new device /dev/multipath/dm0. >> I set kern.iscsi.fail_on_disconnection=1 (to fail IO fast). >> >> Issue 1: >> ------------- >> I can't run simple fio/dd traffice over /dev/da0 nor /dev/da1. >> The only traffic that possible is using the multipath device dm0. >> Is this by design ? > > This is probably because geom only lets one writer open the device at > a time, and when the dm0 device is open, it blocks any other writers > to it's underlying devices... There is a special geom debug flag that > you can set to avoid this... > I tried this debug exclusive flag with iSCSI/iSER but when I set it to 0 the system crushed at the 'gmultipath label ...' stage. John, did you try it over iSCSI ? >> In the linux implementation we can run traffic on both block devices and >> multipath devices. >> >> Issue 2: >> -------------- >> I run some fio traffic utility over multipath device dm0 on initiator >> side with port toggling in a loop >> >> Port 1 down --> sleep 2 mins (iSCSI/ISER device reconnecting meanwhile >> with no success) --> port 1 up --> sleep 5 mins (iSCSI/ISER device >> reconnecting successecfully) >> Port 2 down --> sleep 2 mins (iSCSI/ISER device reconnecting meanwhile >> with no success) --> port 2 up --> sleep 5 mins (iSCSI/ISER device >> reconnecting successecfully) >> >> The expected result is that when the port N is down than the traffic >> moves to the available port and continue succesfully. >> I run this test for many hours and traffic FAILED (even though there was >> at least 1 suitable path between initiator and target). > > Someone else will need to answer this one... > I'm pretty new in the FreeBSD community so I don't know who is the maintainer of the gmultipath driver. Maybe we can discuss about testing the gmultipath driver over iscsi/iser devices and fix some bugs together ? We are planning to add it to our test plan and HA is in high priority for us. Thanks, Max