From owner-freebsd-infiniband@freebsd.org Sun May 22 18:28:19 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3CAE0B45F70 for ; Sun, 22 May 2016 18:28:19 +0000 (UTC) (envelope-from justin@postgresql.org) Received: from meldrar.postgresql.org (meldrar.postgresql.org [IPv6:2a02:c0:301:0:ffff::31]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "*.postgresql.org", Issuer "Gandi Standard SSL CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 048451C0A for ; Sun, 22 May 2016 18:28:18 +0000 (UTC) (envelope-from justin@postgresql.org) Received: from 82-69-92-196.dsl.in-addr.zen.co.uk ([82.69.92.196] helo=[172.16.1.14]) by meldrar.postgresql.org with esmtpsa (TLS1.0:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1b4Y6m-0006ca-T3 for freebsd-infiniband@freebsd.org; Sun, 22 May 2016 18:28:15 +0000 From: Justin Clift Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Subject: OT: Mellanox Ethernet mode support added to FreeNAS :) Message-Id: Date: Sun, 22 May 2016 19:28:10 +0100 To: freebsd-infiniband@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) X-Mailer: Apple Mail (2.1878.6) X-Pg-Spam-Score: -2.9 (--) X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 May 2016 18:28:19 -0000 Hi all, Ethernet mode support for Mellanox adapters has been merged into the FreeNAS 9.10 and 10 source trees now: * 9.10 - https://github.com/freenas/freenas-build/pull/5 * 10 - https://github.com/freenas/freenas-build/pull/6 It'll be in their nightly builds from here on, and may get included with the main releases if it seems stable enough. :) If people have time to test it and report back, that would be great: http://download.freenas.org/9.10/MASTER/ For getting the native IB side of things included, I'm not sure when I'll have time to investigate that. I wasn't able to get Windows (ugh) running in IPoIB mode with my ConnectX card, so couldn't do interoperability testing to FreeBSD nor FreeNAS. Thus the switch to Ethernet mode testing. I do have some single port ConnectX-2 cards now, but haven't plugged them in yet, and probably won't over the next 2-3 weeks. :/ Maybe after that though. :) + Justin -- "My grandfather once told me that there are two kinds of people: those who work and those who take the credit. He told me to try to be in the first group; there was less competition there." - Indira Gandhi From owner-freebsd-infiniband@freebsd.org Mon May 23 15:38:38 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6B402B475D1 for ; Mon, 23 May 2016 15:38:38 +0000 (UTC) (envelope-from Krishna2@chelsio.com) Received: from stargate.chelsio.com (stargate.chelsio.com [12.32.117.8]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4E2861180 for ; Mon, 23 May 2016 15:38:37 +0000 (UTC) (envelope-from Krishna2@chelsio.com) Received: from nice.asicdesigners.com (nice.asicdesigners.com [10.192.160.7]) by stargate.chelsio.com (8.13.8/8.13.8) with ESMTP id u4NFcbRO016871; Mon, 23 May 2016 08:38:37 -0700 Received: from NICE.asicdesigners.com ([fe80::51b2:ba95:9d72:babc]) by nice.asicdesigners.com ([fe80::51b2:ba95:9d72:babc%15]) with mapi id 14.03.0123.003; Mon, 23 May 2016 08:38:33 -0700 From: KrishnamRaju ErapaRaju To: Hans Petter Selasky , "freebsd-infiniband@freebsd.org" Subject: RE: write() system call(via rdma_get_cm_event) is not honoring a signal with flag SA_RESTART. Thread-Topic: write() system call(via rdma_get_cm_event) is not honoring a signal with flag SA_RESTART. Thread-Index: AdGxDdzFSQDg/DwRTmyaP7SBO3H+vgAAVIMQAA9IBoAAHy0U0AAPFVIAAMCpauA= Date: Mon, 23 May 2016 15:38:32 +0000 Message-ID: <4EAB2619577FCA4EBCE11F17B7373447C4313CE5@nice.asicdesigners.com> References: <4EAB2619577FCA4EBCE11F17B7373447C4313548@nice.asicdesigners.com> <3500e324-7d7c-ed94-5913-911fd9877bef@selasky.org> <4EAB2619577FCA4EBCE11F17B7373447C431372B@nice.asicdesigners.com> <6085b7db-aab4-0496-1789-0992bc1f1c5d@selasky.org> In-Reply-To: <6085b7db-aab4-0496-1789-0992bc1f1c5d@selasky.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.193.191.70] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 May 2016 15:38:38 -0000 Hi Hans, Did you get a chance to have a closer look into this issue? Looping in application for errno: 512 would be workaround, but the fix(I th= ink) should be in RDMA core, as there are many rdma applications, which alr= eady written assuming system calls gets restarted when interrupted by a sig= nal. Thanks, Krishna. -----Original Message----- From: Hans Petter Selasky [mailto:hps@selasky.org]=20 Sent: Thursday, May 19, 2016 6:04 PM To: KrishnamRaju ErapaRaju ; freebsd-infiniband@freeb= sd.org Subject: Re: write() system call(via rdma_get_cm_event) is not honoring a s= ignal with flag SA_RESTART. On 05/19/16 14:22, KrishnamRaju ErapaRaju wrote: > Hans, > > I think it's not a regression. > > To verify it I have fetched the latest sources(both FreeBSD & OFED 3.2) a= nd started building... > Between, may I know when will OFED 3.2 patches be pushed to upstream? > > Also, before started building, I'd quickly replaced -ERESTARTSYS with -5= 12 in ucma_get_event, I still don't see the write() system call getting res= tarted. > > Here are the return codes: > ucma_get_event, returns -512 > rdma_get_cm_event, returns -1, errorno: 512 > > So can FreeBSD understand the interrupted system call return value(-512) = like Linux? Hi, I need to check this a bit closer. Possibly you'll need to loop in the appl= ication or library when you see errno 512. --HPS From owner-freebsd-infiniband@freebsd.org Thu May 26 09:45:44 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E519DB499D8 for ; Thu, 26 May 2016 09:45:44 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from msa1.earth.yoonka.com (yoonka.com [185.24.122.233]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "msa1.earth.yoonka.com", Issuer "msa1.earth.yoonka.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 76971197C for ; Thu, 26 May 2016 09:45:43 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from crayon2.yoonka.com (crayon2.yoonka.com [192.168.1.20]) (authenticated bits=0) by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id u4Q9jfKe054300 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Thu, 26 May 2016 09:45:41 GMT (envelope-from list1@gjunka.com) To: freebsd-infiniband@freebsd.org From: Grzegorz Junka Subject: InfiniBand hw support out of the box? Message-ID: Date: Thu, 26 May 2016 09:45:41 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.1.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 May 2016 09:45:45 -0000 Is there any InfiniBand card supported in FreeBSD by the generic kernel/modules without having to recompile kernel or compile kernel modules? From owner-freebsd-infiniband@freebsd.org Thu May 26 12:14:20 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ED119B4946F for ; Thu, 26 May 2016 12:14:20 +0000 (UTC) (envelope-from justin@postgresql.org) Received: from meldrar.postgresql.org (meldrar.postgresql.org [IPv6:2a02:c0:301:0:ffff::31]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "*.postgresql.org", Issuer "Gandi Standard SSL CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B3FF1104C for ; Thu, 26 May 2016 12:14:20 +0000 (UTC) (envelope-from justin@postgresql.org) Received: from 82-69-92-196.dsl.in-addr.zen.co.uk ([82.69.92.196] helo=[172.16.1.14]) by meldrar.postgresql.org with esmtpsa (TLS1.0:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1b5uB3-000273-7s; Thu, 26 May 2016 12:14:16 +0000 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: InfiniBand hw support out of the box? From: Justin Clift In-Reply-To: Date: Thu, 26 May 2016 13:14:10 +0100 Cc: freebsd-infiniband@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <1AB1786D-53C6-4FFB-9606-13C48A79FDF1@postgresql.org> References: To: Grzegorz Junka X-Mailer: Apple Mail (2.1878.6) X-Pg-Spam-Score: -2.9 (--) X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 May 2016 12:14:21 -0000 On 26 May 2016, at 10:45, Grzegorz Junka wrote: > Is there any InfiniBand card supported in FreeBSD by the generic = kernel/modules without having to recompile kernel or compile kernel = modules? Not yet. Recompiling the kernel + kernel modules is extremely easy = though, for anyone with *nix command line experience of any kind, if that's the concern. :) There's even a guide with cut-n-paste instructions if that helps. Useful? Regards and best wishes, Justin Clift -- "My grandfather once told me that there are two kinds of people: those who work and those who take the credit. He told me to try to be in the first group; there was less competition there." - Indira Gandhi From owner-freebsd-infiniband@freebsd.org Thu May 26 12:56:40 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EFD08B4B16E for ; Thu, 26 May 2016 12:56:40 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from msa1.earth.yoonka.com (yoonka.com [185.24.122.233]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "msa1.earth.yoonka.com", Issuer "msa1.earth.yoonka.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 9A47A189C for ; Thu, 26 May 2016 12:56:39 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from crayon2.yoonka.com (crayon2.yoonka.com [192.168.1.20]) (authenticated bits=0) by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id u4QCuVqV057415 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Thu, 26 May 2016 12:56:31 GMT (envelope-from list1@gjunka.com) Subject: Re: InfiniBand hw support out of the box? References: <1AB1786D-53C6-4FFB-9606-13C48A79FDF1@postgresql.org> To: freebsd-infiniband@freebsd.org From: Grzegorz Junka Message-ID: <05d08345-fdb3-107f-464c-bd5599bb530b@gjunka.com> Date: Thu, 26 May 2016 12:56:31 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.1.0 MIME-Version: 1.0 In-Reply-To: <1AB1786D-53C6-4FFB-9606-13C48A79FDF1@postgresql.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 May 2016 12:56:41 -0000 Somehow. I used to compile the kernel but now I try to postpone if possible. The trouble isn't the compilation, it's the deployment. I can't compile on my server, I would need to compile on my desktop and somehow move the kernel to the server, upgrade it there, and restart. Then any further upgrades to a newer FreeBSD release would become a much bigger hassle. It's just a matter of spending some time figuring out the process. Grzegorz On 26/05/2016 12:14, Justin Clift wrote: > On 26 May 2016, at 10:45, Grzegorz Junka wrote: >> Is there any InfiniBand card supported in FreeBSD by the generic kernel/modules without having to recompile kernel or compile kernel modules? > Not yet. Recompiling the kernel + kernel modules is extremely easy though, > for anyone with *nix command line experience of any kind, if that's the > concern. :) > > There's even a guide with cut-n-paste instructions if that helps. > > Useful? > > Regards and best wishes, > > Justin Clift > > -- > "My grandfather once told me that there are two kinds of people: those > who work and those who take the credit. He told me to try to be in the > first group; there was less competition there." > - Indira Gandhi > > From owner-freebsd-infiniband@freebsd.org Thu May 26 13:00:25 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 958B0B4B259 for ; Thu, 26 May 2016 13:00:25 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from msa1.earth.yoonka.com (yoonka.com [185.24.122.233]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "msa1.earth.yoonka.com", Issuer "msa1.earth.yoonka.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 27D901A57 for ; Thu, 26 May 2016 13:00:24 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from crayon2.yoonka.com (crayon2.yoonka.com [192.168.1.20]) (authenticated bits=0) by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id u4QD0M72057547 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Thu, 26 May 2016 13:00:23 GMT (envelope-from list1@gjunka.com) To: freebsd-infiniband@freebsd.org From: Grzegorz Junka Subject: InfiniBand supported hw Message-ID: <9f3323d3-fc01-c1e7-8a93-3132a61c9235@gjunka.com> Date: Thu, 26 May 2016 13:00:22 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.1.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 May 2016 13:00:25 -0000 Is 500EX-D supported on FreeBSD? I read that it's a Voltaire version and it is the same as Mellanox ConnectX, which supposed to be supported. Also, most InfiniBand cards provide two ports. When connecting two computers together (IPoIB) is it possible to use both ports to double the speed? Lastly, is NFS RDMA properly supported on FreeBSD? Grzegorz From owner-freebsd-infiniband@freebsd.org Thu May 26 13:03:23 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 68AF1B4B363 for ; Thu, 26 May 2016 13:03:23 +0000 (UTC) (envelope-from bacon4000@gmail.com) Received: from mail-it0-x22a.google.com (mail-it0-x22a.google.com [IPv6:2607:f8b0:4001:c0b::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 379CC1D02 for ; Thu, 26 May 2016 13:03:23 +0000 (UTC) (envelope-from bacon4000@gmail.com) Received: by mail-it0-x22a.google.com with SMTP id e62so93462911ita.1 for ; Thu, 26 May 2016 06:03:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding; bh=J0z/LyfBQ8ksKBpyZkGECtLyNhCowIcUZKxFuUTN+7Y=; b=KRwznqf/mEdChwfxFNprD09Rtx8F+/GShDXBI2iCx6jrLCe52q4OjyCgBVXC8phA8x K2C4fgkmqKO4Dt/XclU63nHhxAvutn7XyLGua1x4kfTmNk10WOFT4/fpSk5Yy3B630nb lbvWh1lw0hmBmvH15e+oT/kt62Qz+FwOzf+uMy7ubKiNpFalV0vbwBUspNfo7Fvx97BU BmLMrKWwvwNRUSiyq02qJkqTc0P91Pnr6OAnVs4f0O8k+rFmIt4AaUhUhE/F9TyYrTF+ W4WIWuktMfKj0DKf0HepzBFn51SvNH/i8inDis/0ejxJboumApYb09x/pARu414VtcbQ /M5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=J0z/LyfBQ8ksKBpyZkGECtLyNhCowIcUZKxFuUTN+7Y=; b=FicfRX/ZP2ZY1knBIvzZF391UyHKdASzxHCkcog8WGIcMgB5tNB0BwbtKsF6pg0xbL ZOrxMO6llJeghuMyb+ZEtkWCgIu6a+D9Ysqo2sj/iB5IFup5r/wd7kWmLh7netme4RA9 Mz5OblXRkCy8j5GJSWVaV8Zt/6IGU2wXOqlErG2NcQHTB01YPb0jmadbeQJjTCpxBb5I K2U49RjDhrZdlCD236jvJOrmW92Ka/vnHPcuJAcGhx4vPYBSHKKajxhNVTzup6SX8WgQ f1FWlTseIf8FMoaS3MKvteDjYCiMZpy1K5bpLQSc4luANnn+bfNc/GxRRgH0FxkQli0n 4Zyw== X-Gm-Message-State: ALyK8tIN45F/D8vIdM6gw6sNpR3begSkusfyDi9FAjiQVj9p6dPpFihzbz1UVVsZvCy3gg== X-Received: by 10.36.124.65 with SMTP id a62mr3100528itd.33.1464267802304; Thu, 26 May 2016 06:03:22 -0700 (PDT) Received: from imacbsd.acadix.biz (cpe-174-102-163-140.wi.res.rr.com. [174.102.163.140]) by smtp.gmail.com with ESMTPSA id mx2sm1049645igb.0.2016.05.26.06.03.21 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 26 May 2016 06:03:21 -0700 (PDT) Subject: Re: InfiniBand hw support out of the box? To: Grzegorz Junka , freebsd-infiniband@freebsd.org References: From: Jason Bacon Message-ID: <5746F418.1050105@gmail.com> Date: Thu, 26 May 2016 08:03:20 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.7.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 May 2016 13:03:23 -0000 I think IB support in the generic kernel is unlikely any time soon, if ever, but building the kernel modules only takes a few minutes. Instructions are on the wiki in case you weren't aware: https://wiki.freebsd.org/InfiniBand You still have to do a buildworld to update your userland tools, though. I've made the argument that the userland tools could and should be built with IB support now. It adds a trivial number of files and only modifies a few binaries. This would allow someone to get IB support running in minutes instead of hours, and would prevent freebsd-update from replacing ib-enabled files with generic ones. Regards, Jason On 05/26/16 04:45, Grzegorz Junka wrote: > Is there any InfiniBand card supported in FreeBSD by the generic > kernel/modules without having to recompile kernel or compile kernel > modules? > _______________________________________________ > freebsd-infiniband@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > To unsubscribe, send any mail to > "freebsd-infiniband-unsubscribe@freebsd.org" -- All wars are civil wars, because all men are brothers ... Each one owes infinitely more to the human race than to the particular country in which he was born. -- Francois Fenelon From owner-freebsd-infiniband@freebsd.org Thu May 26 13:07:35 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 82EBAB4B3FF for ; Thu, 26 May 2016 13:07:35 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from msa1.earth.yoonka.com (yoonka.com [185.24.122.233]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "msa1.earth.yoonka.com", Issuer "msa1.earth.yoonka.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 2DCB71D5F for ; Thu, 26 May 2016 13:07:34 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from crayon2.yoonka.com (crayon2.yoonka.com [192.168.1.20]) (authenticated bits=0) by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id u4QD7WoS057615 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Thu, 26 May 2016 13:07:32 GMT (envelope-from list1@gjunka.com) Subject: Re: InfiniBand hw support out of the box? To: freebsd-infiniband@freebsd.org References: <5746F418.1050105@gmail.com> From: Grzegorz Junka Message-ID: Date: Thu, 26 May 2016 13:07:32 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.1.0 MIME-Version: 1.0 In-Reply-To: <5746F418.1050105@gmail.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 May 2016 13:07:35 -0000 Yeah, I agree, I don't quite understand why those modules aren't build with the generic kernel if so many other much less useful modules are still being build and shipped wit the generic kernel. It's quite a hassle to maintain custom kernels/modules on production servers. Grzegorz On 26/05/2016 13:03, Jason Bacon wrote: > > I think IB support in the generic kernel is unlikely any time soon, if > ever, but building the kernel modules only takes a few minutes. > Instructions are on the wiki in case you weren't aware: > > https://wiki.freebsd.org/InfiniBand > > You still have to do a buildworld to update your userland tools, > though. I've made the argument that the userland tools could and > should be built with IB support now. It adds a trivial number of > files and only modifies a few binaries. > > This would allow someone to get IB support running in minutes instead > of hours, and would prevent freebsd-update from replacing ib-enabled > files with generic ones. > > Regards, > > Jason > > On 05/26/16 04:45, Grzegorz Junka wrote: >> Is there any InfiniBand card supported in FreeBSD by the generic >> kernel/modules without having to recompile kernel or compile kernel >> modules? >> _______________________________________________ >> freebsd-infiniband@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-infiniband >> To unsubscribe, send any mail to >> "freebsd-infiniband-unsubscribe@freebsd.org" > > From owner-freebsd-infiniband@freebsd.org Thu May 26 15:31:24 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3575CB4B621 for ; Thu, 26 May 2016 15:31:24 +0000 (UTC) (envelope-from bacon4000@gmail.com) Received: from mail-io0-x236.google.com (mail-io0-x236.google.com [IPv6:2607:f8b0:4001:c06::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F2E8F179B for ; Thu, 26 May 2016 15:31:23 +0000 (UTC) (envelope-from bacon4000@gmail.com) Received: by mail-io0-x236.google.com with SMTP id f8so55100646ioe.3 for ; Thu, 26 May 2016 08:31:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding; bh=oFJifZDsmFhucp9cDScZqQxJI33wJFjH66pnZVrPJ2M=; b=APJFFaXpY5MrAyGYWL9/We8kOnQbj3+Gi/Y9wEtUaC0a1/rbeOcW/0DqX2LV+EsaIh ZEsGiC42oeOb4u3Dsl7MNky08jHSrKW+gUwOsL1nEzJ79zvVjov9T0wKKO7ypJh97lDq GI59mLg6nGvHdaNAOATKrtrHLtmhvpePO4DJcqyU5x6JlWH/5i6Fv3rpKGUHZU3ces53 BjpnKMj1A0YwC30lFXxjwswEGWBKECgBz3w1Ozc6+BQMy9D17sTIxc3o7uLGlUr8PfCg YtRaVBY9uJaAzTOiwfhP4Uk7PnWkDZee2TVqzYUNHGXR/EgX+aQ/eowILuR7YDzW959F MItg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=oFJifZDsmFhucp9cDScZqQxJI33wJFjH66pnZVrPJ2M=; b=ZCTtnV5+w5Rv7Jo152F9dgnipmIl0ad/SqAajJZSkKBW6qyW/oSm++uCQSwuKzlHR/ HZcakXH2Fb51xNx9O857BvZ3ylonisKIMKdQ1VYvGJnny13maVFQEC3488prKNT8Uaa9 J8wzI08gUHbyH62dzBc5jnJrH6BAGOa08xvMYqHt8JnorUn4qkO1YmceJ5vMWglVKnT/ yl09koPCxw9BD0VlgiHSflMVIYc2TAIIdbu5W7vuMkhO7FrJp3bvXRMft6lfGpIYQVRK EDQH70kmEb420y9O19sMeBzrRkIOSEMglR9AaglSA8xEFwlgK3rRh6Rkqmd8iICWmqEy Oy0w== X-Gm-Message-State: ALyK8tL2Na7PMcMom+ppL3K9Rihazk82p1l5Axf+BJwO0NVr577RwRwIDLg07owfJI5Sgg== X-Received: by 10.107.8.67 with SMTP id 64mr8748684ioi.85.1464276683273; Thu, 26 May 2016 08:31:23 -0700 (PDT) Received: from imacbsd.acadix.biz (cpe-174-102-163-140.wi.res.rr.com. [174.102.163.140]) by smtp.gmail.com with ESMTPSA id s124sm1223499itd.20.2016.05.26.08.31.22 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 26 May 2016 08:31:22 -0700 (PDT) Subject: Re: InfiniBand hw support out of the box? To: Grzegorz Junka , freebsd-infiniband@freebsd.org References: <5746F418.1050105@gmail.com> From: Jason Bacon Message-ID: <574716C9.7080903@gmail.com> Date: Thu, 26 May 2016 10:31:21 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.7.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 May 2016 15:31:24 -0000 Well, the kernel code and kernel modules are still a work-in-progress and could cause instability, including panics. Forcing the user to build them manually is a form of notification that they're not ready for prime-time. Since the kernel modules only take a few minutes to build, there wouldn't be much advantage to including them by default. On the flip side, installing the IB userland basically amounts to installing IB headers and libraries and enabling a feature in a few networking commands. This poses minimal risk and saves the end-user hours of compilation every time the base is updated. Regards, JB On 05/26/16 08:07, Grzegorz Junka wrote: > Yeah, I agree, I don't quite understand why those modules aren't build > with the generic kernel if so many other much less useful modules are > still being build and shipped wit the generic kernel. It's quite a > hassle to maintain custom kernels/modules on production servers. > > Grzegorz > > > On 26/05/2016 13:03, Jason Bacon wrote: >> >> I think IB support in the generic kernel is unlikely any time soon, >> if ever, but building the kernel modules only takes a few minutes. >> Instructions are on the wiki in case you weren't aware: >> >> https://wiki.freebsd.org/InfiniBand >> >> You still have to do a buildworld to update your userland tools, >> though. I've made the argument that the userland tools could and >> should be built with IB support now. It adds a trivial number of >> files and only modifies a few binaries. >> >> This would allow someone to get IB support running in minutes instead >> of hours, and would prevent freebsd-update from replacing ib-enabled >> files with generic ones. >> >> Regards, >> >> Jason >> >> On 05/26/16 04:45, Grzegorz Junka wrote: >>> Is there any InfiniBand card supported in FreeBSD by the generic >>> kernel/modules without having to recompile kernel or compile kernel >>> modules? >>> _______________________________________________ >>> freebsd-infiniband@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-infiniband >>> To unsubscribe, send any mail to >>> "freebsd-infiniband-unsubscribe@freebsd.org" >> >> > > _______________________________________________ > freebsd-infiniband@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > To unsubscribe, send any mail to > "freebsd-infiniband-unsubscribe@freebsd.org" -- All wars are civil wars, because all men are brothers ... Each one owes infinitely more to the human race than to the particular country in which he was born. -- Francois Fenelon From owner-freebsd-infiniband@freebsd.org Thu May 26 17:07:09 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2220BB4A394 for ; Thu, 26 May 2016 17:07:09 +0000 (UTC) (envelope-from justin@postgresql.org) Received: from meldrar.postgresql.org (meldrar.postgresql.org [IPv6:2a02:c0:301:0:ffff::31]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "*.postgresql.org", Issuer "Gandi Standard SSL CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DCEA61A29 for ; Thu, 26 May 2016 17:07:08 +0000 (UTC) (envelope-from justin@postgresql.org) Received: from 82-69-92-196.dsl.in-addr.zen.co.uk ([82.69.92.196] helo=[172.16.1.14]) by meldrar.postgresql.org with esmtpsa (TLS1.0:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1b5ykR-0004B5-3x; Thu, 26 May 2016 17:07:06 +0000 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: InfiniBand supported hw From: Justin Clift In-Reply-To: <9f3323d3-fc01-c1e7-8a93-3132a61c9235@gjunka.com> Date: Thu, 26 May 2016 18:07:01 +0100 Cc: freebsd-infiniband@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <9f3323d3-fc01-c1e7-8a93-3132a61c9235@gjunka.com> To: Grzegorz Junka X-Mailer: Apple Mail (2.1878.6) X-Pg-Spam-Score: -2.9 (--) X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 May 2016 17:07:09 -0000 On 26 May 2016, at 14:00, Grzegorz Junka wrote: > Is 500EX-D supported on FreeBSD? I read that it's a Voltaire version = and it is the same as Mellanox ConnectX, which supposed to be supported. Yep, they should work. You may need to flash them with their Mellanox = equivalent firmware, but I'm not sure. (doing that flash isn't very hard btw) > Also, most InfiniBand cards provide two ports. When connecting two = computers together (IPoIB) is it possible to use both ports to double = the speed? Not sure, as I've not needed to personally. None of my storage (or = other uses) even fills one DDR port. Yet. :) I *think* the answer to that is yes, > Lastly, is NFS RDMA properly supported on FreeBSD? Again not sure. Other people here would likely know. :) + Justin -- "My grandfather once told me that there are two kinds of people: those who work and those who take the credit. He told me to try to be in the first group; there was less competition there." - Indira Gandhi From owner-freebsd-infiniband@freebsd.org Thu May 26 17:44:35 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0F2A8B4AF75 for ; Thu, 26 May 2016 17:44:35 +0000 (UTC) (envelope-from lists@jnielsen.net) Received: from webmail2.jnielsen.NET (webmail2.jnielsen.net [50.114.224.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "webmail2.jnielsen.net", Issuer "freebsdsolutions.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id E91441F15 for ; Thu, 26 May 2016 17:44:34 +0000 (UTC) (envelope-from lists@jnielsen.net) Received: from jnielse-ml.domo.com (50-207-241-62-static.hfc.comcastbusiness.net [50.207.241.62]) (authenticated bits=0) by webmail2.jnielsen.NET (8.15.2/8.15.2) with ESMTPSA id u4QHiTjE006605 (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 26 May 2016 11:44:33 -0600 (MDT) (envelope-from lists@jnielsen.net) X-Authentication-Warning: webmail2.jnielsen.NET: Host 50-207-241-62-static.hfc.comcastbusiness.net [50.207.241.62] claimed to be jnielse-ml.domo.com Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: InfiniBand supported hw From: John Nielsen In-Reply-To: Date: Thu, 26 May 2016 11:44:32 -0600 Cc: Grzegorz Junka , freebsd-infiniband@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <05BB7487-B000-4800-88BD-EC7DEC508160@jnielsen.net> References: <9f3323d3-fc01-c1e7-8a93-3132a61c9235@gjunka.com> To: Justin Clift X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 May 2016 17:44:35 -0000 On May 26, 2016, at 11:07 AM, Justin Clift = wrote: > On 26 May 2016, at 14:00, Grzegorz Junka wrote: >> Is 500EX-D supported on FreeBSD? I read that it's a Voltaire version = and it is the same as Mellanox ConnectX, which supposed to be supported. >=20 > Yep, they should work. You may need to flash them with their Mellanox = equivalent > firmware, but I'm not sure. (doing that flash isn't very hard btw) >=20 >> Also, most InfiniBand cards provide two ports. When connecting two = computers together (IPoIB) is it possible to use both ports to double = the speed? >=20 > Not sure, as I've not needed to personally. None of my storage (or = other uses) > even fills one DDR port. Yet. :) >=20 > I *think* the answer to that is yes, It depends on what you mean by "use both ports." You can of course use = both ports at the same time, but if you're envisioning something like an = IPoIB version of an Ethernet LAGG I don't think that is supported. If you are using the IB for storage protocols, then active/active = multipathing is one way to utilize the full bandwidth of multiple ports. >> Lastly, is NFS RDMA properly supported on FreeBSD? >=20 > Again not sure. Other people here would likely know. :) From owner-freebsd-infiniband@freebsd.org Thu May 26 18:33:03 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E3870B4BDDB for ; Thu, 26 May 2016 18:33:03 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from msa1.earth.yoonka.com (yoonka.com [185.24.122.233]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "msa1.earth.yoonka.com", Issuer "msa1.earth.yoonka.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 73B5E1436 for ; Thu, 26 May 2016 18:33:02 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from crayon2.yoonka.com (crayon2.yoonka.com [192.168.1.20]) (authenticated bits=0) by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id u4QIX0dD062781 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Thu, 26 May 2016 18:33:00 GMT (envelope-from list1@gjunka.com) Subject: Re: InfiniBand supported hw References: <9f3323d3-fc01-c1e7-8a93-3132a61c9235@gjunka.com> <05BB7487-B000-4800-88BD-EC7DEC508160@jnielsen.net> To: freebsd-infiniband@freebsd.org From: Grzegorz Junka Message-ID: <1063e37f-8a71-0804-b96c-13ef90dcdcb0@gjunka.com> Date: Thu, 26 May 2016 18:33:00 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.1.0 MIME-Version: 1.0 In-Reply-To: <05BB7487-B000-4800-88BD-EC7DEC508160@jnielsen.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 May 2016 18:33:04 -0000 On 26/05/2016 17:44, John Nielsen wrote: > On May 26, 2016, at 11:07 AM, Justin Clift wrote: > >> On 26 May 2016, at 14:00, Grzegorz Junka wrote: >>> Is 500EX-D supported on FreeBSD? I read that it's a Voltaire version and it is the same as Mellanox ConnectX, which supposed to be supported. >> Yep, they should work. You may need to flash them with their Mellanox equivalent >> firmware, but I'm not sure. (doing that flash isn't very hard btw) >> >>> Also, most InfiniBand cards provide two ports. When connecting two computers together (IPoIB) is it possible to use both ports to double the speed? >> Not sure, as I've not needed to personally. None of my storage (or other uses) >> even fills one DDR port. Yet. :) >> >> I *think* the answer to that is yes, > It depends on what you mean by "use both ports." You can of course use both ports at the same time, but if you're envisioning something like an IPoIB version of an Ethernet LAGG I don't think that is supported. > > If you are using the IB for storage protocols, then active/active multipathing is one way to utilize the full bandwidth of multiple ports. > >>> Lastly, is NFS RDMA properly supported on FreeBSD? >> Again not sure. Other people here would likely know. :) > Yes, that's exactly what I was hoping for, a sort of LAGG to double the speed of the link between computer by utilizing both ports together. LAGG is independent from the network interface, if I could set up an IPoIB NIC for each port separately what could prevent me from creating a LAGG interface on top of them? And if that's not supported, what do people usually do with the other interface? Just leave it unconnected? Grzegorz From owner-freebsd-infiniband@freebsd.org Thu May 26 19:00:59 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 71565B4B55B for ; Thu, 26 May 2016 19:00:59 +0000 (UTC) (envelope-from chrisr@spectralogic.com) Received: from mail1.bemta12.messagelabs.com (mail1.bemta12.messagelabs.com [216.82.251.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail1.bemta12.messagelabs.com", Issuer "Symantec Class 3 Secure Server CA - G4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 319651B94 for ; Thu, 26 May 2016 19:00:58 +0000 (UTC) (envelope-from chrisr@spectralogic.com) Received: from [216.82.251.33] by server-1.bemta-12.messagelabs.com id 84/E7-21085-D6647475; Thu, 26 May 2016 18:54:37 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrOIsWRWlGSWpSXmKPExsVyQG6fiG6um3u 4wZkeeYu/xyayWryf/ITJgcljxqf5LB6brtcFMEWxZuYl5VcksGZ8ObKRqWC/WMXMu9NYGxhv C3UxcnKwCehIdKw7zQxiiwhkSlzof8QKYgsLqEhsu/oKKq4qsevaJTYI201i6+JJLCA2C1B8y g+IXl4BI4mbO18Axbk4hARuM0rc2f6ECSTBKWAnMaO3E6yBUUBM4vupNWBxZgFxiVtP5oPZEg ICEkv2nGeGsEUlXj7+xwphG0hsXbqPBaJeT+LG1ClsELa2xLKFr6EWC0qcnPmEZQKj4CwkY2c haZmFpGUWkpYFjCyrGDWKU4vKUot0jcz1kooy0zNKchMzc3QNDY30clOLixPTU3MSk4r1kvNz NzECA7yegYFxB+OmRq9DjJIcTEqivLEG7uFCfEn5KZUZicUZ8UWlOanFhxhlODiUJHjTXYFyg kWp6akVaZk5wFiDSUtw8CiJ8BaApHmLCxJzizPTIVKnGBWlxHkFQBICIImM0jy4Nlh8X2KUlR LmZWRgYBDiKUgtys0sQZV/xSjOwagkzBsJMoUnM68EbvoroMVMQIv9vziDLC5JREhJNTB6r+3 iU7k9QTtO7XZpRfO0fR1WvgyBl5VEVZ7NsZzs3cwd3/H/WcHz7NTVx37V2W+f/Wq7EvPC5D+P Xq1Ivbl8k6cuy4K8j/9O7/ux3njhQY3fOxpfflS6Md88OEtR0X7/jLQ1WRKG58V2i0zSaau3z 9Hkvv5y2o/jVwqyb9Yr+z9xnLanjK1NiaU4I9FQi7moOBEAiWsMDOoCAAA= X-Env-Sender: chrisr@spectralogic.com X-Msg-Ref: server-4.tower-130.messagelabs.com!1464288877!34351730!1 X-Originating-IP: [192.30.190.20] X-StarScan-Received: X-StarScan-Version: 8.34; banners=-,-,- X-VirusChecked: Checked Received: (qmail 118394 invoked from network); 26 May 2016 18:54:37 -0000 Received: from outmx2.spectralogic.com (HELO mail.spectralogic.com) (192.30.190.20) by server-4.tower-130.messagelabs.com with AES256-GCM-SHA384 encrypted SMTP; 26 May 2016 18:54:37 -0000 From: Chris Reeves To: Grzegorz Junka , "freebsd-infiniband@freebsd.org" Subject: Re: InfiniBand supported hw Thread-Topic: InfiniBand supported hw Thread-Index: AQHRt06WrtyxxhHDmkO+I7/v2i4P3J/L2AGAgAAKfACAAA2KAP//oEig Date: Thu, 26 May 2016 18:54:36 +0000 Message-ID: <1464288876261.66558@spectralogic.com> References: <9f3323d3-fc01-c1e7-8a93-3132a61c9235@gjunka.com> <05BB7487-B000-4800-88BD-EC7DEC508160@jnielsen.net>, <1063e37f-8a71-0804-b96c-13ef90dcdcb0@gjunka.com> In-Reply-To: <1063e37f-8a71-0804-b96c-13ef90dcdcb0@gjunka.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 May 2016 19:00:59 -0000 I've tried to LAGG two IPOIB ports in the past on FreeBSD, but was not succ= essful.=0A= I'm not certain, but I believe the following from sys/net/if_lagg.c is the = issue:=0A= =0A= /* XXX Disallow non-ethernet interfaces (this should be any of 802)= */=0A= if (ifp->if_type !=3D IFT_ETHER) {=0A= LAGG_WUNLOCK(sc);=0A= return (EPROTONOSUPPORT);=0A= }=0A= =0A= I think IPOIB is not considered an Ethernet port.=0A= ________________________________________=0A= From: owner-freebsd-infiniband@freebsd.org on behalf of Grzegorz Junka =0A= Sent: Thursday, May 26, 2016 12:33 PM=0A= To: freebsd-infiniband@freebsd.org=0A= Subject: Re: InfiniBand supported hw=0A= =0A= On 26/05/2016 17:44, John Nielsen wrote:=0A= > On May 26, 2016, at 11:07 AM, Justin Clift wrote:= =0A= >=0A= >> On 26 May 2016, at 14:00, Grzegorz Junka wrote:=0A= >>> Is 500EX-D supported on FreeBSD? I read that it's a Voltaire version an= d it is the same as Mellanox ConnectX, which supposed to be supported.=0A= >> Yep, they should work. You may need to flash them with their Mellanox e= quivalent=0A= >> firmware, but I'm not sure. (doing that flash isn't very hard btw)=0A= >>=0A= >>> Also, most InfiniBand cards provide two ports. When connecting two comp= uters together (IPoIB) is it possible to use both ports to double the speed= ?=0A= >> Not sure, as I've not needed to personally. None of my storage (or othe= r uses)=0A= >> even fills one DDR port. Yet. :)=0A= >>=0A= >> I *think* the answer to that is yes,=0A= > It depends on what you mean by "use both ports." You can of course use bo= th ports at the same time, but if you're envisioning something like an IPoI= B version of an Ethernet LAGG I don't think that is supported.=0A= >=0A= > If you are using the IB for storage protocols, then active/active multipa= thing is one way to utilize the full bandwidth of multiple ports.=0A= >=0A= >>> Lastly, is NFS RDMA properly supported on FreeBSD?=0A= >> Again not sure. Other people here would likely know. :)=0A= >=0A= =0A= Yes, that's exactly what I was hoping for, a sort of LAGG to double the=0A= speed of the link between computer by utilizing both ports together.=0A= LAGG is independent from the network interface, if I could set up an=0A= IPoIB NIC for each port separately what could prevent me from creating a=0A= LAGG interface on top of them? And if that's not supported, what do=0A= people usually do with the other interface? Just leave it unconnected?=0A= =0A= Grzegorz=0A= _______________________________________________=0A= freebsd-infiniband@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-infiniband=0A= To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe@freebsd.or= g"=0A= From owner-freebsd-infiniband@freebsd.org Thu May 26 19:55:54 2016 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 033D7B4B6CF for ; Thu, 26 May 2016 19:55:54 +0000 (UTC) (envelope-from lists@jnielsen.net) Received: from webmail2.jnielsen.NET (webmail2.jnielsen.net [50.114.224.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "webmail2.jnielsen.net", Issuer "freebsdsolutions.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D0F2B11CC for ; Thu, 26 May 2016 19:55:53 +0000 (UTC) (envelope-from lists@jnielsen.net) Received: from [10.180.26.108] (mobile-166-171-249-213.mycingular.net [166.171.249.213]) (authenticated bits=0) by webmail2.jnielsen.NET (8.15.2/8.15.2) with ESMTPSA id u4QJtnhb054591 (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 26 May 2016 13:55:52 -0600 (MDT) (envelope-from lists@jnielsen.net) X-Authentication-Warning: webmail2.jnielsen.NET: Host mobile-166-171-249-213.mycingular.net [166.171.249.213] claimed to be [10.180.26.108] Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (1.0) Subject: Re: InfiniBand supported hw From: John Nielsen X-Mailer: iPhone Mail (13F69) In-Reply-To: <1063e37f-8a71-0804-b96c-13ef90dcdcb0@gjunka.com> Date: Thu, 26 May 2016 13:55:47 -0600 Cc: freebsd-infiniband@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <7C7D1835-52DD-4F27-B0CA-A59650E59639@jnielsen.net> References: <9f3323d3-fc01-c1e7-8a93-3132a61c9235@gjunka.com> <05BB7487-B000-4800-88BD-EC7DEC508160@jnielsen.net> <1063e37f-8a71-0804-b96c-13ef90dcdcb0@gjunka.com> To: Grzegorz Junka X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 May 2016 19:55:54 -0000 > On May 26, 2016, at 12:33 PM, Grzegorz Junka wrote: >=20 >=20 >> On 26/05/2016 17:44, John Nielsen wrote: >>> On May 26, 2016, at 11:07 AM, Justin Clift wrote= : >>>=20 >>>> On 26 May 2016, at 14:00, Grzegorz Junka wrote: >>>> Is 500EX-D supported on FreeBSD? I read that it's a Voltaire version an= d it is the same as Mellanox ConnectX, which supposed to be supported. >>> Yep, they should work. You may need to flash them with their Mellanox e= quivalent >>> firmware, but I'm not sure. (doing that flash isn't very hard btw) >>>=20 >>>> Also, most InfiniBand cards provide two ports. When connecting two comp= uters together (IPoIB) is it possible to use both ports to double the speed?= >>> Not sure, as I've not needed to personally. None of my storage (or othe= r uses) >>> even fills one DDR port. Yet. :) >>>=20 >>> I *think* the answer to that is yes, >> It depends on what you mean by "use both ports." You can of course use bo= th ports at the same time, but if you're envisioning something like an IPoIB= version of an Ethernet LAGG I don't think that is supported. >>=20 >> If you are using the IB for storage protocols, then active/active multipa= thing is one way to utilize the full bandwidth of multiple ports. >>=20 >>>> Lastly, is NFS RDMA properly supported on FreeBSD? >>> Again not sure. Other people here would likely know. :) >=20 > Yes, that's exactly what I was hoping for, a sort of LAGG to double the sp= eed of the link between computer by utilizing both ports together. LAGG is i= ndependent from the network interface, if I could set up an IPoIB NIC for ea= ch port separately what could prevent me from creating a LAGG interface on t= op of them? And if that's not supported, what do people usually do with the o= ther interface? Just leave it unconnected? LAGG is designed to work with Ethernet. InfiniBand is a very different layer= 2 technology with its own advantages and disadvantages. I don't want to get= too far out of my depth so I'll just suggest you do your own research on th= e subject. How you use the second port is entirely up to you. You can use it for separa= te applications, for redundancy or higher throughput. In my experience it us= ually makes sense to give it its own IP. If you want higher bandwidth than a= single port can provide then you should at layer 3 or higher for ways to ag= gregate bandwidth. (You should also do some research on PCI-e throughput. A x= 8 PCI-e 3.0 slot maxes out well before 112Gbit/s.) As I mentioned previously you can use active/active multipath to get more th= roughput for block storage applications. You could also run a routing daemon= and do ECMP for any layer 3 traffic. Or your particular application may hav= e other ways to intelligently use two distinct IPoIB paths.