From owner-freebsd-infiniband@FreeBSD.ORG Sun Sep 15 20:20:35 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 430F1141 for ; Sun, 15 Sep 2013 20:20:35 +0000 (UTC) (envelope-from jwbacon@tds.net) Received: from mail-qc0-f171.google.com (mail-qc0-f171.google.com [209.85.216.171]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 04BE123CF for ; Sun, 15 Sep 2013 20:20:29 +0000 (UTC) Received: by mail-qc0-f171.google.com with SMTP id x19so2283615qcw.30 for ; Sun, 15 Sep 2013 13:20:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :subject:content-type:content-transfer-encoding; bh=xnsyR0udjmnGav93QRvgQdptj+bK2NM00+Yy9x5ftko=; b=OP7uXZOBB60X68/v2qx9MFXXsulUM0IaYQ0z2JYcKvhdyXdkbD1FGvmH8KJ5tnmA7v SrW8ggIVOX0PqBOMJZJmfl+rETZkruLgXKS+pZTWiJSmVqL+yUI20aHSo1tJ831BEv3R oSzlYIYPtqR1+nbPkWeeP42j9CbO0W/sxbXDAeXBgl8tPWGCPgJJf457TXOqUoCN7SLN js6z8ykz6+SL2gyrL1zPNQsIdS2KmIfCFirJIGlJMMruQrTlJSgCT8mUy7xPSMEHaA8V ymhS3PkianTAr+wsVMdLws/z9LPTxl5lc3MeGtMljZbw77Pzw8K5tJIBKkarZFOvK73H g45Q== X-Gm-Message-State: ALoCoQkEDh6tBiIicFb8KG5gXL2SHAdStQKYpl6UaP/7einof9wm2wDEEVcn49hZeFgsCwUEpPOO X-Received: by 10.229.174.3 with SMTP id r3mr42008597qcz.10.1379276423763; Sun, 15 Sep 2013 13:20:23 -0700 (PDT) Received: from [192.168.0.52] (h69-131-100-239.nwblwi.broadband.dynamic.tds.net. [69.131.100.239]) by mx.google.com with ESMTPSA id fy7sm41668696qeb.1.1969.12.31.16.00.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 15 Sep 2013 13:20:22 -0700 (PDT) Message-ID: <52361685.8000901@tds.net> Date: Sun, 15 Sep 2013 15:20:21 -0500 From: Jason bacon User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.20) Gecko/20110804 Thunderbird/3.1.12 MIME-Version: 1.0 To: freebsd-infiniband@freebsd.org Subject: Supported NICs Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 15 Sep 2013 20:20:35 -0000 Is there a list of supported IB NICs out there somewhere? I followed the wiki instructions for rebuilding with IB support and now have mlx4ib, mlxen, etc. Was hoping there would be man pages for the drivers that list known working cards, but there don't seem to be. I'm hoping to test a file server using IPOIB and possible roll a FreeNAS ISO with IB support if it works out. Thanks, -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Jason W. Bacon jwbacon@tds.net Circumstances don't make a man: They reveal him. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From owner-freebsd-infiniband@FreeBSD.ORG Mon Sep 16 03:17:52 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 85FA07DF for ; Mon, 16 Sep 2013 03:17:52 +0000 (UTC) (envelope-from accornehl@gmail.com) Received: from mail-ee0-x229.google.com (mail-ee0-x229.google.com [IPv6:2a00:1450:4013:c00::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 20DB326F0 for ; Mon, 16 Sep 2013 03:17:51 +0000 (UTC) Received: by mail-ee0-f41.google.com with SMTP id d17so1755626eek.28 for ; Sun, 15 Sep 2013 20:17:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=CRW7XgLT+MvV4ACjNZyieMisXnge3Ahn5uhsJMELx5Y=; b=TkIwCYG/uHuHY9pA/Qu96Aa2LUwPmDuFg4w4vTf0t73k59yshnXnDtpNhQeyf0Uj5y ekquzwVuCNFnqYatMEAi5a1DGTqCeH2P16DxtbW2NZdcWZaKA9tya1LsX878vX6RzV35 7eHMvMevu6K9dYSURExR8sgO4aoRbodGG4r2UwT8UMwiXEjCdf3YMamatn2Dwi61Ow/x igxM/mq/SqzrrC/nbEOXAnwsFIzoBaZQMolKMcoQxUhagr1/PSE9R9zxnhREaDXe/qEj rmMQpg0m8BEQ/9ZiYtu/wrfR0i9wAAnhb5NhM/VESMmK72ngFEN7uYOKJmXVBnLmN+B0 FzdQ== MIME-Version: 1.0 X-Received: by 10.14.45.70 with SMTP id o46mr39967172eeb.19.1379301470563; Sun, 15 Sep 2013 20:17:50 -0700 (PDT) Received: by 10.223.152.69 with HTTP; Sun, 15 Sep 2013 20:17:50 -0700 (PDT) Received: by 10.223.152.69 with HTTP; Sun, 15 Sep 2013 20:17:50 -0700 (PDT) In-Reply-To: <52361685.8000901@tds.net> References: <52361685.8000901@tds.net> Date: Sun, 15 Sep 2013 20:17:50 -0700 Message-ID: Subject: Re: Supported NICs From: Anthony Cornehl To: Jason bacon Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-infiniband@freebsd.org X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Sep 2013 03:17:52 -0000 On Sep 15, 2013 1:20 PM, "Jason bacon" wrote: > > > Is there a list of supported IB NICs out there somewhere? > > I followed the wiki instructions for rebuilding with IB support and now have mlx4ib, mlxen, etc. > > Was hoping there would be man pages for the drivers that list known working cards, but there don't seem to be. I'm hoping to test a file server using IPOIB and possible roll a FreeNAS ISO with IB support if it works out. > > Thanks, > > -- > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Jason W. Bacon > jwbacon@tds.net > > Circumstances don't make a man: > They reveal him. > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > _______________________________________________ > freebsd-infiniband@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > To unsubscribe, send any mail to " freebsd-infiniband-unsubscribe@freebsd.org" Hey Jason, Mellanox ConnectX series cards are the only ones supported currently... http://www.mellanox.com/page/infiniband_cards_overview Don't forget to compile with IPoIB as well, since the IPoIB support is compiled into the driver, unlike in Linux. Just be aware that IPoIB performance is also improved by enabling connected mode when you compile the kernel module. The IB code in FreeBSD is a few years older the what is in Linux, but, the following forum thread is probably relevant... http://forums.servethehome.com/networking/1554-infiniband-ipoib-performance-problems.html Cheers! From owner-freebsd-infiniband@FreeBSD.ORG Mon Sep 16 03:38:09 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id E57F4968 for ; Mon, 16 Sep 2013 03:38:09 +0000 (UTC) (envelope-from accornehl@gmail.com) Received: from mail-ea0-x22c.google.com (mail-ea0-x22c.google.com [IPv6:2a00:1450:4013:c01::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 70DF427BC for ; Mon, 16 Sep 2013 03:38:09 +0000 (UTC) Received: by mail-ea0-f172.google.com with SMTP id r16so1729603ead.17 for ; Sun, 15 Sep 2013 20:38:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=arNv1/2iBSfR5vTqCp6CUpADT0sE0YnhPl9d0B1aAoc=; b=kCQeXsA0bXn585IbrUw1DcApDT1/ot4g+E7VZggNl7hgAG2PEqztGPOaClJuwZFSwq ic1erhcsIJ4a9WBGRN+RNGZ237zDmxgHAtktwYFw9vAKbvi2B2mIyEGbLWLBKlh5uFMq ZGcdgw8RMlpf2z4n1FNP3yjWewe0/kxx9h/P2kFTKumtm2gUejs30mrCvEdpkaxTK9CR YKfw2eYgQYNllX6Yvj3ClpIYlZYRbc92N4LOEL3tB+qb5NmPOIZ+FSF8l8VGjrJN2+sU xUdnD5KqlK1WNqhGJE/UzQqWDsAVpjHHXQllOnafSkwPxidvoeXdeaMvCvMPzaT8+9Dn +tkw== MIME-Version: 1.0 X-Received: by 10.14.88.65 with SMTP id z41mr9487441eee.38.1379302687824; Sun, 15 Sep 2013 20:38:07 -0700 (PDT) Received: by 10.223.152.69 with HTTP; Sun, 15 Sep 2013 20:38:07 -0700 (PDT) Received: by 10.223.152.69 with HTTP; Sun, 15 Sep 2013 20:38:07 -0700 (PDT) In-Reply-To: References: <52361685.8000901@tds.net> Date: Sun, 15 Sep 2013 20:38:07 -0700 Message-ID: Subject: Re: Supported NICs From: Anthony Cornehl To: Jason bacon Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-infiniband@freebsd.org X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Sep 2013 03:38:10 -0000 On Sep 15, 2013 8:17 PM, "Anthony Cornehl" wrote: > > > On Sep 15, 2013 1:20 PM, "Jason bacon" wrote: > > > > > > Is there a list of supported IB NICs out there somewhere? > > > > I followed the wiki instructions for rebuilding with IB support and now have mlx4ib, mlxen, etc. > > > > Was hoping there would be man pages for the drivers that list known working cards, but there don't seem to be. I'm hoping to test a file server using IPOIB and possible roll a FreeNAS ISO with IB support if it works out. > > > > Thanks, > > > > -- > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > Jason W. Bacon > > jwbacon@tds.net > > > > Circumstances don't make a man: > > They reveal him. > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > > > _______________________________________________ > > freebsd-infiniband@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > > To unsubscribe, send any mail to " freebsd-infiniband-unsubscribe@freebsd.org" > > Hey Jason, > > Mellanox ConnectX series cards are the only ones supported currently... > > http://www.mellanox.com/page/infiniband_cards_overview > > Don't forget to compile with IPoIB as well, since the IPoIB support is compiled into the driver, unlike in Linux. > > Just be aware that IPoIB performance is also improved by enabling connected mode when you compile the kernel module. The IB code in FreeBSD is a few years older the what is in Linux, but, the following forum thread is probably relevant... > > http://forums.servethehome.com/networking/1554-infiniband-ipoib-performance-problems.html > > Cheers! It also appears that Jeff fixed SDP a few weeks ago, which is more capable of reaching line-speed for IB-connected devices. http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-7.html Cheers! From owner-freebsd-infiniband@FreeBSD.ORG Mon Sep 16 13:31:43 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 6CD6A681 for ; Mon, 16 Sep 2013 13:31:43 +0000 (UTC) (envelope-from jwbacon@tds.net) Received: from mail-vb0-f51.google.com (mail-vb0-f51.google.com [209.85.212.51]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 290A22D92 for ; Mon, 16 Sep 2013 13:31:41 +0000 (UTC) Received: by mail-vb0-f51.google.com with SMTP id x16so2777360vbf.38 for ; Mon, 16 Sep 2013 06:31:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=uKIgA2cjPM0XYPrUCKpR0We46xGrz1r6vsjDKiJ+tLY=; b=Y50SEonD2J8bmjhuwk4DJ59sRKoFTCSW92ShvF/6aYrc3gwXNBa5dAITGIGtgPO3o3 bYEiN6gTgalsYqbYD6jfycLuBp98Y4afhBV+EQO2/2/6aFp0+3nUVPC+6OzWAgsbBph3 gA+GXCzyZhh3tWXDRTa4+kavtkCvDonFfELTqyI0rMhIJzqfisl+MdwIGorU5a/Pz+HH KZDa6MCOeORXnGVr+/Ww3dab0rz9jFcyow9pDhuNN5Th89UoWZLSPZevHAQz8T15ZmO0 TuIduPbl94skx9D8bFLeyrQOtchtrCXZg5L0ppW7IxBEPxCJLPd37+jyUT3J04AwVzlt OFdg== X-Gm-Message-State: ALoCoQlAlo57bwfoU96hFo7ttt4hHx3gNAlk0d2hbfzOF6VS6nix3mTmZL80ZTRvEk8BYMBbBW+n X-Received: by 10.52.122.68 with SMTP id lq4mr22851359vdb.21.1379338295577; Mon, 16 Sep 2013 06:31:35 -0700 (PDT) Received: from sculpin.jbacon.dyndns.org (h69-131-100-239.nwblwi.broadband.dynamic.tds.net. [69.131.100.239]) by mx.google.com with ESMTPSA id bl10sm18883144vec.8.1969.12.31.16.00.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 16 Sep 2013 06:31:35 -0700 (PDT) Message-ID: <52370835.7020901@tds.net> Date: Mon, 16 Sep 2013 08:31:33 -0500 From: Jason Bacon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:10.0.3) Gecko/20120327 Thunderbird/10.0.3 MIME-Version: 1.0 To: Anthony Cornehl Subject: Re: Supported NICs References: <52361685.8000901@tds.net> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-infiniband@freebsd.org X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Sep 2013 13:31:43 -0000 Thanks Anthony & Oded! Would you say that IB support in 9.2 is stable enough for a production HPC file server? Will do plenty of my own testing first, of course. Regards, Jason On 09/15/13 22:38, Anthony Cornehl wrote: > > > On Sep 15, 2013 8:17 PM, "Anthony Cornehl" > wrote: > > > > > > On Sep 15, 2013 1:20 PM, "Jason bacon" > wrote: > > > > > > > > > Is there a list of supported IB NICs out there somewhere? > > > > > > I followed the wiki instructions for rebuilding with IB support > and now have mlx4ib, mlxen, etc. > > > > > > Was hoping there would be man pages for the drivers that list > known working cards, but there don't seem to be. I'm hoping to test a > file server using IPOIB and possible roll a FreeNAS ISO with IB > support if it works out. > > > > > > Thanks, > > > > > > -- > > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > > Jason W. Bacon > > > jwbacon@tds.net > > > > > > Circumstances don't make a man: > > > They reveal him. > > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > > > > > _______________________________________________ > > > freebsd-infiniband@freebsd.org > mailing list > > > http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > > > To unsubscribe, send any mail to > "freebsd-infiniband-unsubscribe@freebsd.org > " > > > > Hey Jason, > > > > Mellanox ConnectX series cards are the only ones supported currently... > > > > http://www.mellanox.com/page/infiniband_cards_overview > > > > Don't forget to compile with IPoIB as well, since the IPoIB support > is compiled into the driver, unlike in Linux. > > > > Just be aware that IPoIB performance is also improved by enabling > connected mode when you compile the kernel module. The IB code in > FreeBSD is a few years older the what is in Linux, but, the following > forum thread is probably relevant... > > > > > http://forums.servethehome.com/networking/1554-infiniband-ipoib-performance-problems.html > > > > Cheers! > > It also appears that Jeff fixed SDP a few weeks ago, which is more > capable of reaching line-speed for IB-connected devices. > > http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-7.html > > Cheers! > -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Jason W. Bacon jwbacon@tds.net Circumstances don't make a man: They reveal him. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~