From owner-freebsd-infiniband@FreeBSD.ORG Tue Feb 10 16:19:03 2015 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 75EFC28A for ; Tue, 10 Feb 2015 16:19:03 +0000 (UTC) Received: from proofpoint4.lanl.gov (proofpoint4.lanl.gov [IPv6:2001:400:4210:400::a4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4BE74170 for ; Tue, 10 Feb 2015 16:19:03 +0000 (UTC) Received: from mailrelay2.lanl.gov (mailrelay2.lanl.gov [128.165.4.103]) by mailgate4.lanl.gov (8.14.7/8.14.7) with ESMTP id t1AGJ21f002095 for ; Tue, 10 Feb 2015 09:19:02 -0700 Received: from localhost (localhost.localdomain [127.0.0.1]) by mailrelay2.lanl.gov (Postfix) with ESMTP id 04CD6E9EAFD for ; Tue, 10 Feb 2015 09:19:02 -0700 (MST) X-NIE-2-Virus-Scanner: amavisd-new at mailrelay2.lanl.gov Received: from tykhe.lanl.gov (tykhe.lanl.gov [128.165.248.149]) by mailrelay2.lanl.gov (Postfix) with ESMTP id E7E0BE9EAFE for ; Tue, 10 Feb 2015 09:19:01 -0700 (MST) Message-ID: <54DA2F75.3000307@lanl.gov> Date: Tue, 10 Feb 2015 09:19:01 -0700 From: Ron Croonenberg User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: freebsd-infiniband@freebsd.org Subject: IBSRP and switches Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.13.68, 1.0.33, 0.0.0000 definitions=2015-02-10_06:2015-02-10,2015-02-10,1970-01-01 signatures=0 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Feb 2015 16:19:03 -0000 From what I understand IBSRP is a point to point protocol and cannot be switched over an IB switch, correct? (and the presented devices cannot be see by initiators 'through' a switch. Is there a 'scsi protocol' that I can run over IB that can be switched? thanks, Ron From owner-freebsd-infiniband@FreeBSD.ORG Tue Feb 10 17:39:35 2015 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4BA7CA53 for ; Tue, 10 Feb 2015 17:39:35 +0000 (UTC) Received: from webmail2.jnielsen.net (webmail2.jnielsen.net [50.114.224.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "webmail2.jnielsen.net", Issuer "freebsdsolutions.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 295A1C93 for ; Tue, 10 Feb 2015 17:39:34 +0000 (UTC) Received: from [10.10.1.196] (office.betterlinux.com [199.58.199.60]) (authenticated bits=0) by webmail2.jnielsen.net (8.15.1/8.14.9) with ESMTPSA id t1AHdOIR063468 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 10 Feb 2015 10:39:26 -0700 (MST) (envelope-from lists@jnielsen.net) X-Authentication-Warning: webmail2.jnielsen.net: Host office.betterlinux.com [199.58.199.60] claimed to be [10.10.1.196] Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2070.6\)) Subject: Re: IBSRP and switches From: John Nielsen In-Reply-To: <54DA2F75.3000307@lanl.gov> Date: Tue, 10 Feb 2015 10:39:25 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: <54DA2F75.3000307@lanl.gov> To: Ron Croonenberg X-Mailer: Apple Mail (2.2070.6) Cc: freebsd-infiniband@freebsd.org X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Feb 2015 17:39:35 -0000 On Feb 10, 2015, at 9:19 AM, Ron Croonenberg wrote: > =46rom what I understand IBSRP is a point to point protocol and cannot = be switched over an IB switch, correct? (and the presented devices = cannot be see by initiators 'through' a switch. At $work we use SRP over InfiniBand with lots of switches in the mix. = Never had an issue. > Is there a 'scsi protocol' that I can run over IB that can be = switched? Unfortunately, there isn't currently SRP support on FreeBSD. There is = RDMA support, so it might be feasible to port the SRP code from Linux, = but I haven't really looked in to it. There's always iSCSI. FreeBSD 10.0 saw the addition of the kernel-based = target and initiator (with some improvements since), but I don't think = it takes advantage of RDMA or other acceleration techniques when used = with InfiniBand. I'd love to be wrong. :) JN From owner-freebsd-infiniband@FreeBSD.ORG Tue Feb 10 18:01:09 2015 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 35C47218 for ; Tue, 10 Feb 2015 18:01:09 +0000 (UTC) Received: from proofpoint5.lanl.gov (proofpoint5.lanl.gov [IPv6:2001:400:4210:400::a5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E5823E8C for ; Tue, 10 Feb 2015 18:01:08 +0000 (UTC) Received: from mailrelay1.lanl.gov (mailrelay1.lanl.gov [128.165.4.101]) by mailgate5.lanl.gov (8.14.7/8.14.7) with ESMTP id t1AI171I009198; Tue, 10 Feb 2015 11:01:07 -0700 Received: from localhost (localhost.localdomain [127.0.0.1]) by mailrelay1.lanl.gov (Postfix) with ESMTP id C416713777A9; Tue, 10 Feb 2015 11:01:07 -0700 (MST) X-NIE-2-Virus-Scanner: amavisd-new at mailrelay1.lanl.gov Received: from tykhe.lanl.gov (tykhe.lanl.gov [128.165.248.149]) by mailrelay1.lanl.gov (Postfix) with ESMTP id B1BC113777A0; Tue, 10 Feb 2015 11:01:07 -0700 (MST) Message-ID: <54DA4763.6070203@lanl.gov> Date: Tue, 10 Feb 2015 11:01:07 -0700 From: Ron Croonenberg User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: John Nielsen , Ron Croonenberg Subject: Re: IBSRP and switches References: <54DA2F75.3000307@lanl.gov> In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.13.68, 1.0.33, 0.0.0000 definitions=2015-02-10_06:2015-02-10,2015-02-10,1970-01-01 signatures=0 Cc: freebsd-infiniband@freebsd.org X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Feb 2015 18:01:09 -0000 On 02/10/2015 10:39 AM, John Nielsen wrote: > On Feb 10, 2015, at 9:19 AM, Ron Croonenberg wrote: > >> From what I understand IBSRP is a point to point protocol and cannot be switched over an IB switch, correct? (and the presented devices cannot be see by initiators 'through' a switch. > > At $work we use SRP over InfiniBand with lots of switches in the mix. Never had an issue. > Ok, I'll try some of that. I assume you were able to see all drives on all ports. What I am trying to do is see all drives on all hosts (connected to a switch). IBSRP seems to be 'point to point' So I wonder if it (ibsrp) will let me mount a drive multiple times. (I am doing this in a 'not so posix' kind of way) >> Is there a 'scsi protocol' that I can run over IB that can be switched? > > Unfortunately, there isn't currently SRP support on FreeBSD. There is RDMA support, so it might be feasible to port the SRP code from Linux, but I haven't really looked in to it. That is why I am looking into it this way, because of RDMA. There is also something called ISER (iSCSI extensions over RDMA) > There's always iSCSI. FreeBSD 10.0 saw the addition of the kernel-based target and initiator (with some improvements since), but I don't think it takes advantage of RDMA or other acceleration techniques when used with InfiniBand. I'd love to be wrong. :) right, but I am trying to go for the highest performance that is reliable. > JN > Ron From owner-freebsd-infiniband@FreeBSD.ORG Wed Feb 11 05:15:10 2015 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C8946D91 for ; Wed, 11 Feb 2015 05:15:10 +0000 (UTC) Received: from emea01-am1-obe.outbound.protection.outlook.com (mail-am1on0649.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe00::649]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 528C41CC for ; Wed, 11 Feb 2015 05:15:10 +0000 (UTC) Received: from DB3PR05MB026.eurprd05.prod.outlook.com (10.242.137.13) by DB3PR05MB026.eurprd05.prod.outlook.com (10.242.137.13) with Microsoft SMTP Server (TLS) id 15.1.81.19; Wed, 11 Feb 2015 05:14:47 +0000 Received: from DB3PR05MB026.eurprd05.prod.outlook.com ([169.254.2.160]) by DB3PR05MB026.eurprd05.prod.outlook.com ([169.254.2.160]) with mapi id 15.01.0081.018; Wed, 11 Feb 2015 05:14:47 +0000 From: Oded Shanoon To: Ron Croonenberg , John Nielsen Subject: RE: IBSRP and switches Thread-Topic: IBSRP and switches Thread-Index: AQHQRU1QfcNbVMRuDkGs/SN2nhUHYpzqJmuAgAAGEICAALvI8A== Date: Wed, 11 Feb 2015 05:14:47 +0000 Message-ID: References: <54DA2F75.3000307@lanl.gov> <54DA4763.6070203@lanl.gov> In-Reply-To: <54DA4763.6070203@lanl.gov> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [193.47.165.251] authentication-results: lanl.gov; dkim=none (message not signed) header.d=none; x-microsoft-antispam: BCL:0;PCL:0;RULEID:;SRVR:DB3PR05MB026; x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:;SRVR:DB3PR05MB026; x-forefront-prvs: 0484063412 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(51704005)(479174004)(13464003)(24454002)(377454003)(76176999)(46102003)(50986999)(2900100001)(54356999)(2656002)(19580405001)(19580395003)(86362001)(76576001)(74316001)(87936001)(102836002)(15975445007)(2950100001)(62966003)(77156002)(66066001)(33656002)(106116001)(1720100001)(40100003)(122556002)(92566002); DIR:OUT; SFP:1101; SCL:1; SRVR:DB3PR05MB026; H:DB3PR05MB026.eurprd05.prod.outlook.com; FPR:; SPF:None; MLV:sfv; LANG:en; Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Feb 2015 05:14:47.2989 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB3PR05MB026 Cc: "freebsd-infiniband@freebsd.org" X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Feb 2015 05:15:11 -0000 Hi, We (Mellanox) are now working on iSER for FreeBSD. We should have something running in a couple of months. I'm not sure when e= xactly it will be commited to upstream. You are welcome to contact me if you have questions. Regards, Oded Shanoon OFED-FreeBSD Team Leader Mellanox Technologies, Raanana -----Original Message----- From: owner-freebsd-infiniband@freebsd.org [mailto:owner-freebsd-infiniband= @freebsd.org] On Behalf Of Ron Croonenberg Sent: Tuesday, February 10, 2015 8:01 PM To: John Nielsen; Ron Croonenberg Cc: freebsd-infiniband@freebsd.org Subject: Re: IBSRP and switches On 02/10/2015 10:39 AM, John Nielsen wrote: > On Feb 10, 2015, at 9:19 AM, Ron Croonenberg wrote: > >> From what I understand IBSRP is a point to point protocol and cannot be= switched over an IB switch, correct? (and the presented devices cannot be = see by initiators 'through' a switch. > > At $work we use SRP over InfiniBand with lots of switches in the mix. Nev= er had an issue. > Ok, I'll try some of that. I assume you were able to see all drives on all= ports. What I am trying to do is see all drives on all hosts (connected to= a switch). IBSRP seems to be 'point to point' So I wonder if it (ibsrp) wi= ll let me mount a drive multiple times. (I am doing this in a 'not so posix= ' kind of way) >> Is there a 'scsi protocol' that I can run over IB that can be switched? > > Unfortunately, there isn't currently SRP support on FreeBSD. There is RDM= A support, so it might be feasible to port the SRP code from Linux, but I h= aven't really looked in to it. That is why I am looking into it this way, because of RDMA. There is also something called ISER (iSCSI extensions over RDMA) > There's always iSCSI. FreeBSD 10.0 saw the addition of the=20 > kernel-based target and initiator (with some improvements since), but=20 > I don't think it takes advantage of RDMA or other acceleration=20 > techniques when used with InfiniBand. I'd love to be wrong. :) right, but I am trying to go for the highest performance that is reliable. > JN > Ron _______________________________________________ freebsd-infiniband@freebsd.org mailing list http://lists.freebsd.org/mailma= n/listinfo/freebsd-infiniband To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe@freebsd.or= g" From owner-freebsd-infiniband@FreeBSD.ORG Wed Feb 11 21:35:11 2015 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 32923878 for ; Wed, 11 Feb 2015 21:35:11 +0000 (UTC) Received: from proofpoint5.lanl.gov (proofpoint5.lanl.gov [IPv6:2001:400:4210:400::a5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 040BBBB for ; Wed, 11 Feb 2015 21:35:10 +0000 (UTC) Received: from mailrelay1.lanl.gov (mailrelay1.lanl.gov [128.165.4.101]) by mailgate5.lanl.gov (8.14.7/8.14.7) with ESMTP id t1BLZ903006753 for ; Wed, 11 Feb 2015 14:35:09 -0700 Received: from localhost (localhost.localdomain [127.0.0.1]) by mailrelay1.lanl.gov (Postfix) with ESMTP id C6A4C13778F3; Wed, 11 Feb 2015 14:35:09 -0700 (MST) X-NIE-2-Virus-Scanner: amavisd-new at mailrelay1.lanl.gov Received: from tykhe.lanl.gov (tykhe.lanl.gov [128.165.248.149]) by mailrelay1.lanl.gov (Postfix) with ESMTP id B227B137768D; Wed, 11 Feb 2015 14:35:09 -0700 (MST) Message-ID: <54DBCB0D.9030709@lanl.gov> Date: Wed, 11 Feb 2015 14:35:09 -0700 From: Ron Croonenberg User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: Ron Croonenberg Subject: Re: IBSRP and switches References: <54DA2F75.3000307@lanl.gov> <54DA4763.6070203@lanl.gov> In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.13.68, 1.0.33, 0.0.0000 definitions=2015-02-11_04:2015-02-11,2015-02-11,1970-01-01 signatures=0 Cc: "freebsd-infiniband@freebsd.org" X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Feb 2015 21:35:11 -0000 I am actually looking into iSER thanks, Ron On 02/10/2015 10:14 PM, Oded Shanoon wrote: > Hi, > > We (Mellanox) are now working on iSER for FreeBSD. > We should have something running in a couple of months. I'm not sure when exactly it will be commited to upstream. > > You are welcome to contact me if you have questions. > > > Regards, > > Oded Shanoon > OFED-FreeBSD Team Leader > Mellanox Technologies, Raanana > > -----Original Message----- > From: owner-freebsd-infiniband@freebsd.org [mailto:owner-freebsd-infiniband@freebsd.org] On Behalf Of Ron Croonenberg > Sent: Tuesday, February 10, 2015 8:01 PM > To: John Nielsen; Ron Croonenberg > Cc: freebsd-infiniband@freebsd.org > Subject: Re: IBSRP and switches > > > > On 02/10/2015 10:39 AM, John Nielsen wrote: >> On Feb 10, 2015, at 9:19 AM, Ron Croonenberg wrote: >> >>> From what I understand IBSRP is a point to point protocol and cannot be switched over an IB switch, correct? (and the presented devices cannot be see by initiators 'through' a switch. >> >> At $work we use SRP over InfiniBand with lots of switches in the mix. Never had an issue. >> > > Ok, I'll try some of that. I assume you were able to see all drives on all ports. What I am trying to do is see all drives on all hosts (connected to a switch). IBSRP seems to be 'point to point' So I wonder if it (ibsrp) will let me mount a drive multiple times. (I am doing this in a 'not so posix' kind of way) > >>> Is there a 'scsi protocol' that I can run over IB that can be switched? >> >> Unfortunately, there isn't currently SRP support on FreeBSD. There is RDMA support, so it might be feasible to port the SRP code from Linux, but I haven't really looked in to it. > > That is why I am looking into it this way, because of RDMA. > > There is also something called ISER (iSCSI extensions over RDMA) > >> There's always iSCSI. FreeBSD 10.0 saw the addition of the >> kernel-based target and initiator (with some improvements since), but >> I don't think it takes advantage of RDMA or other acceleration >> techniques when used with InfiniBand. I'd love to be wrong. :) > > right, but I am trying to go for the highest performance that is reliable. > >> JN >> > > > Ron > _______________________________________________ > freebsd-infiniband@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe@freebsd.org" >