From owner-freebsd-scsi@freebsd.org Wed Jun 24 05:32:07 2015 Return-Path: Delivered-To: freebsd-scsi@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 07E69141E5B5 for ; Wed, 24 Jun 2015 05:32:07 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-lb0-x231.google.com (mail-lb0-x231.google.com [IPv6:2a00:1450:4010:c04::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id ADA5C9A4 for ; Wed, 24 Jun 2015 05:21:56 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: by lbbvz5 with SMTP id vz5so19383602lbb.0 for ; Tue, 23 Jun 2015 22:21:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=B2KqhI+pbpdh0m2HufWM8X/t7WM5KtmYeMU1GYavQjw=; b=j4Y42U5JAcEYLMvs1KCIO2nd3572OYT5pH+94PAUM3W7Fyztw0xzz1Aut+5WBoR+sX mZBwvzHtfcFYmPfL9CTW9ZGcHF8iKaCAvtVukjnpUA9UwJJLPGJZgCp64kb+pEQ2ohnc pvuohlWzH3GkhM/k0fWrfAuxqYYNi51FZGQWdj0JQR9hgc3XW3kmMrFj3GRSZIIbLrVD oc1NM6Px6qW5511yRkKFYqKT979zG63SRcPgz2NArPIcx1ozfN6tDVcmgkKLlP8NDBpN Uad043TqFTJKgVDUTOSQgP/KRb5ZVa0I41IoCQrh4Omr57S0GrCzMzQama9GgKRT+0CQ 6GtQ== X-Received: by 10.112.135.131 with SMTP id ps3mr38037750lbb.84.1435123314897; Tue, 23 Jun 2015 22:21:54 -0700 (PDT) Received: from mavbook.mavhome.dp.ua ([134.249.139.101]) by mx.google.com with ESMTPSA id jr6sm5255553lab.12.2015.06.23.22.21.53 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 23 Jun 2015 22:21:53 -0700 (PDT) Sender: Alexander Motin Message-ID: <558A3E6F.90903@FreeBSD.org> Date: Wed, 24 Jun 2015 08:21:51 +0300 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: John-Mark Gurney , Max Gurtovoy CC: freebsd-scsi@freebsd.org, Sagi Grimberg , Oren Duer , Hans Petter Selasky Subject: Re: gmultipath HA over iscsi/iser References: <557DA8C0.1020209@mellanox.com> <20150619162015.GD96349@funkthat.com> <55897E4B.8060608@mellanox.com> <20150624001412.GS96349@funkthat.com> In-Reply-To: <20150624001412.GS96349@funkthat.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Jun 2015 05:32:07 -0000 Hi, Max. On 24.06.2015 03:14, John-Mark Gurney wrote: > Max Gurtovoy wrote this message on Tue, Jun 23, 2015 at 18:42 +0300: >> On 6/19/2015 7:20 PM, John-Mark Gurney wrote: >>> Max Gurtovoy wrote this message on Sun, Jun 14, 2015 at 19:16 +0300: >>>> lately I was testing HA using gmultipath utility over iSCSI/iSER devices. >>>> I'm working on 11-current code base. >>>> I created 1 LUN on the target side and connected via 2 different >>>> physical ports from the initiator side. >>>> On the initiator side I see see /dev/da0 and /dev/da1. >>>> I created multipath device using: >>>> gmultipath label dm0 /dev/da0 /dev/da1. >>>> Now I have new device /dev/multipath/dm0. >>>> I set kern.iscsi.fail_on_disconnection=1 (to fail IO fast). >>>> >>>> Issue 1: >>>> ------------- >>>> I can't run simple fio/dd traffice over /dev/da0 nor /dev/da1. >>>> The only traffic that possible is using the multipath device dm0. >>>> Is this by design ? Yes, that is by design. Otherwise geom tasting process could be confused in some situations. For example, it may be not obvious whether partition tables should be handled over multipath or over raw devices. And if user mounts file system via partition labels or GPT IDs, it is not predictable which of three (or more) instances he will mount. There is sysctl kern.geom.multipath.exclusive, documented in man page, to control that behavior, but I would not recommend you to disable it. >>>> In the linux implementation we can run traffic on both block devices and >>>> multipath devices. There is an Easter egg in GEOM that allows to do the same: kern.geom.debugflags=16. It allows access to raw GEOM devices in any situation. But that is a hack and should be considered as such with all possible cautions. >>>> Issue 2: >>>> -------------- >>>> I run some fio traffic utility over multipath device dm0 on initiator >>>> side with port toggling in a loop >>>> >>>> Port 1 down --> sleep 2 mins (iSCSI/ISER device reconnecting meanwhile >>>> with no success) --> port 1 up --> sleep 5 mins (iSCSI/ISER device >>>> reconnecting successecfully) >>>> Port 2 down --> sleep 2 mins (iSCSI/ISER device reconnecting meanwhile >>>> with no success) --> port 2 up --> sleep 5 mins (iSCSI/ISER device >>>> reconnecting successecfully) >>>> >>>> The expected result is that when the port N is down than the traffic >>>> moves to the available port and continue succesfully. >>>> I run this test for many hours and traffic FAILED (even though there was >>>> at least 1 suitable path between initiator and target). I am not aware about such bug. We may need some deeper debugging to diagnose that. > Though I realize it's difficult, it's easiest to look at the source to > see who's been touching it last: > https://svnweb.freebsd.org/base/head/sys/geom/multipath/g_multipath.c?view=log > > Looks like mav has been somewhat active recently... I've cc'd him... > >> Maybe we can discuss about testing the gmultipath driver over iscsi/iser >> devices and fix some bugs together ? > > I can help out some, but engaging mav, or some of the others that > are more active in storage would be better... > >> We are planning to add it to our test plan and HA is in high priority >> for us. > > You should definately talk/coordinate w/ the people at iXsystems (mav > works there), as they work in storage w/ iSCSI, etc. I was not a original gmultipath author, but I've rewritten significant part of it to make it work, and now it works fine for us. I am open to bug reports and propositions about improving it. -- Alexander Motin