From owner-freebsd-infiniband@FreeBSD.ORG Mon Nov 19 02:18:10 2012 Return-Path: Delivered-To: freebsd-infiniband@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8432DC1 for ; Mon, 19 Nov 2012 02:18:10 +0000 (UTC) (envelope-from daichi@bsd-consulting.jp) Received: from natial.ongs.co.jp (natial.ongs.co.jp [202.216.246.90]) by mx1.freebsd.org (Postfix) with ESMTP id 4DDFB8FC0C for ; Mon, 19 Nov 2012 02:18:09 +0000 (UTC) Received: from parancell.ongs.co.jp (dullmdaler.ongs.co.jp [202.216.246.94]) by natial.ongs.co.jp (Postfix) with ESMTPSA id 7914112543B; Mon, 19 Nov 2012 11:09:52 +0900 (JST) Date: Mon, 19 Nov 2012 11:09:45 +0900 From: Daichi GOTO To: freebsd-infiniband@FreeBSD.org Subject: Infiniband device information Message-Id: <20121119110945.ff40669f8feefc2e996df959@bsd-consulting.jp> Organization: BSD Consulting Inc. X-Mailer: Sylpheed 3.2.0 (GTK+ 2.24.6; amd64-portbld-freebsd10.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Nov 2012 02:18:10 -0000 Hi infiniband guys, thanks for IB ml :-) InfiniBand devices are becoming more and more important in Japanese HPC business environments (In Japan, word "BigData" is used instead of HPC in these days, a business buzz word). To promoto FreeBSD as the best product to use in BigData businesses, we need InfiniBand devices lists, devices working well on FreeBSD. http://wiki.freebsd.org/InfiniBand Should we do add "IB devices working list" on FreeBSD IB wiki page? Some ISPs are trying to use IBs on their servers and they are going to choose CentOS because of the lack of FreeBSD IB support. Not good situation for FreeBSD. Needs improvement I guess. -- Daichi GOTO Director | BSD Consulting Inc. 3F SS bldg. 3-4-2, Nishishinbashi, Minato, Tokyo 81-80-4174-6161 | daichi@bsd-consulting.jp From owner-freebsd-infiniband@FreeBSD.ORG Mon Nov 19 03:00:45 2012 Return-Path: Delivered-To: freebsd-infiniband@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5C4958D2 for ; Mon, 19 Nov 2012 03:00:45 +0000 (UTC) (envelope-from daichi@bsdconsulting.co.jp) Received: from pivot.bsdconsulting.co.jp (pivot.bsdconsulting.co.jp [110.50.242.36]) by mx1.freebsd.org (Postfix) with ESMTP id 2A2768FC08 for ; Mon, 19 Nov 2012 03:00:44 +0000 (UTC) Received: from parancell.ongs.co.jp (dullmdaler.ongs.co.jp [202.216.246.94]) by pivot.bsdconsulting.co.jp (Postfix) with ESMTPSA id 2927D3C019B; Mon, 19 Nov 2012 11:53:57 +0900 (JST) Date: Mon, 19 Nov 2012 11:53:50 +0900 From: Daichi GOTO To: freebsd-infiniband@FreeBSD.org Subject: Infiniband experiment environment setting up Message-Id: <20121119115350.3456c580209197b4aea33d75@bsdconsulting.co.jp> Organization: BSD Consulting Inc. X-Mailer: Sylpheed 3.2.0 (GTK+ 2.24.6; amd64-portbld-freebsd10.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Nov 2012 03:00:45 -0000 Hi, Currently, we are getting ready to test InfiniBands on FreeBSD using follow devices: CPU: Intel Xeon E3-1240v2 M/B: Super Micro X9SCM RAM: Kingston KVR16E11/8 (8GBx4) IB Card: Mellanox Technologies ConnectX-2 VPI MHQH19B-XTR IB Card: Mellanox Technologies ConnectX-3 VPI MCX353A-QCBT IB Switch: Mellanox Technologies EXW-IS5022 QSFP cable: MC22096-130-001 Is there some information about above InfiniBand devices? -- Daichi GOTO Director | BSD Consulting Inc. 3F SS bldg. 3-4-2, Nishishinbashi, Minato, Tokyo 81-80-4174-6161 | daichi@bsdconsulting.co.jp From owner-freebsd-infiniband@FreeBSD.ORG Tue Nov 20 18:58:15 2012 Return-Path: Delivered-To: freebsd-infiniband@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C161AA00 for ; Tue, 20 Nov 2012 18:58:15 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from vps.hungerhost.com (vps.hungerhost.com [216.38.53.176]) by mx1.freebsd.org (Postfix) with ESMTP id 920358FC0C for ; Tue, 20 Nov 2012 18:58:15 +0000 (UTC) Received: from [38.105.238.108] (port=53068 helo=[10.7.1.235]) by vps.hungerhost.com with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.80) (envelope-from ) id 1Tat1S-0006rv-GO; Tue, 20 Nov 2012 13:58:14 -0500 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: Infiniband device information From: George Neville-Neil In-Reply-To: <20121119110945.ff40669f8feefc2e996df959@bsd-consulting.jp> Date: Tue, 20 Nov 2012 13:58:13 -0500 Content-Transfer-Encoding: 7bit Message-Id: <827EEDA6-277A-417E-9660-92BDFCFF9ADC@neville-neil.com> References: <20121119110945.ff40669f8feefc2e996df959@bsd-consulting.jp> To: Daichi GOTO X-Mailer: Apple Mail (2.1499) X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - vps.hungerhost.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - neville-neil.com X-Get-Message-Sender-Via: vps.hungerhost.com: authenticated_id: gnn@neville-neil.com Cc: freebsd-infiniband@FreeBSD.org X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 18:58:15 -0000 On Nov 18, 2012, at 21:09 , Daichi GOTO wrote: > Hi infiniband guys, thanks for IB ml :-) > > InfiniBand devices are becoming more and more important > in Japanese HPC business environments (In Japan, word "BigData" > is used instead of HPC in these days, a business buzz word). > > To promoto FreeBSD as the best product to use in BigData > businesses, we need InfiniBand devices lists, devices working > well on FreeBSD. > > http://wiki.freebsd.org/InfiniBand > > Should we do add "IB devices working list" on FreeBSD IB > wiki page? > > Some ISPs are trying to use IBs on their servers and they > are going to choose CentOS because of the lack of FreeBSD IB > support. Not good situation for FreeBSD. Needs improvement > I guess. > Yes, you should add the page, and you should also try to get the folks like Jeff Roberson, who know about our Infiniband stack to point out any issues. Best, George From owner-freebsd-infiniband@FreeBSD.ORG Tue Nov 20 18:59:57 2012 Return-Path: Delivered-To: freebsd-infiniband@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3F195A11 for ; Tue, 20 Nov 2012 18:59:57 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from vps.hungerhost.com (vps.hungerhost.com [216.38.53.176]) by mx1.freebsd.org (Postfix) with ESMTP id 0AAF98FC0C for ; Tue, 20 Nov 2012 18:59:56 +0000 (UTC) Received: from [38.105.238.108] (port=53084 helo=[10.7.1.235]) by vps.hungerhost.com with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.80) (envelope-from ) id 1Tat35-0008LO-EB; Tue, 20 Nov 2012 13:59:55 -0500 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: Infiniband experiment environment setting up From: George Neville-Neil In-Reply-To: <20121119115350.3456c580209197b4aea33d75@bsdconsulting.co.jp> Date: Tue, 20 Nov 2012 13:59:54 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: <2A385CE3-72E8-4D40-8413-69E00841536D@neville-neil.com> References: <20121119115350.3456c580209197b4aea33d75@bsdconsulting.co.jp> To: Daichi GOTO X-Mailer: Apple Mail (2.1499) X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - vps.hungerhost.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - neville-neil.com X-Get-Message-Sender-Via: vps.hungerhost.com: authenticated_id: gnn@neville-neil.com Cc: freebsd-infiniband@FreeBSD.org X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 18:59:57 -0000 On Nov 18, 2012, at 21:53 , Daichi GOTO = wrote: > Hi, >=20 > Currently, we are getting ready to test InfiniBands on FreeBSD > using follow devices: >=20 > CPU: Intel Xeon E3-1240v2 > M/B: Super Micro X9SCM > RAM: Kingston KVR16E11/8 (8GBx4) >=20 > IB Card: Mellanox Technologies ConnectX-2 VPI MHQH19B-XTR > IB Card: Mellanox Technologies ConnectX-3 VPI = MCX353A-QCBT > IB Switch: Mellanox Technologies EXW-IS5022 > QSFP cable: MC22096-130-001 >=20 > Is there some information about above InfiniBand devices? Well it's only the cards you have to worry about. I know the ConnectX-2 = works but have not tested the ConnectX-3. Also, note that it depends on what you are testing. That is, Infiniband = has low latency at low payloads. The numbers quoted by Mellanox about low = latency depend on you sending one byte (yes, 1 byte). At real packet sizes Infiniband = is no faster than 10GbE. Best, George= From owner-freebsd-infiniband@FreeBSD.ORG Tue Nov 20 19:09:19 2012 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C8569E20 for ; Tue, 20 Nov 2012 19:09:19 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 82ED28FC13 for ; Tue, 20 Nov 2012 19:09:19 +0000 (UTC) Received: by mail-ob0-f182.google.com with SMTP id 16so8020141obc.13 for ; Tue, 20 Nov 2012 11:09:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=KKjHLnp4WYi1CSVYqqjH8RFs5g2j6K1aOJyk6pKSZrU=; b=Xu4pt5x6O4SfrhMytI1lyOn42JhfhHsenz8/TBiE6E3w6B5cNWSZcWgQG6RAdtV7hB PEBKJRE2Mf/Sk9u/tJ7h32zbo+2+j1t4NyAHnRfJus+UqHOBM9jNapkpyPqm7LjsCqxl 7T7SP3KZPwW7e+VLtTAiPpkEONOyjoRACMqGP+S2r2lMgG1aV63Txu/2sM3m3WG9EAne CdwSBdhfr7m6b8ecM4Fbz5dlAM/LO4i1SxgvPcGlbo2S+HUZflE3oSTfsXS0G07Fu5lZ fZ36atvPHBuzxo26Gi8z1JydvGktcBjHFsqD4CgTOIYC2hh72uT29cswSfmAxtoNU41R edhA== MIME-Version: 1.0 Received: by 10.60.25.104 with SMTP id b8mr13901687oeg.18.1353438558948; Tue, 20 Nov 2012 11:09:18 -0800 (PST) Received: by 10.76.143.33 with HTTP; Tue, 20 Nov 2012 11:09:18 -0800 (PST) In-Reply-To: <2A385CE3-72E8-4D40-8413-69E00841536D@neville-neil.com> References: <20121119115350.3456c580209197b4aea33d75@bsdconsulting.co.jp> <2A385CE3-72E8-4D40-8413-69E00841536D@neville-neil.com> Date: Tue, 20 Nov 2012 11:09:18 -0800 Message-ID: Subject: Re: Infiniband experiment environment setting up From: Garrett Cooper To: George Neville-Neil Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-infiniband@freebsd.org, Daichi GOTO X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 19:09:20 -0000 On Tue, Nov 20, 2012 at 10:59 AM, George Neville-Neil wrote: > > On Nov 18, 2012, at 21:53 , Daichi GOTO wrote: > >> Hi, >> >> Currently, we are getting ready to test InfiniBands on FreeBSD >> using follow devices: >> >> CPU: Intel Xeon E3-1240v2 >> M/B: Super Micro X9SCM >> RAM: Kingston KVR16E11/8 (8GBx4) >> >> IB Card: Mellanox Technologies ConnectX-2 VPI MHQH19B-XTR >> IB Card: Mellanox Technologies ConnectX-3 VPI MCX353A-QCBT >> IB Switch: Mellanox Technologies EXW-IS5022 >> QSFP cable: MC22096-130-001 >> >> Is there some information about above InfiniBand devices? > > Well it's only the cards you have to worry about. I know the ConnectX-2 works > but have not tested the ConnectX-3. We're getting access to ConnectX-3 cards at Isilon soon, but I'm not sure what sourcebase we'll be testing against (our local sourcebase, CURRENT, or CURRENT + new OFED). My team can best speak to what we'll be doing here. > Also, note that it depends on what you are testing. That is, Infiniband has low > latency at low payloads. The numbers quoted by Mellanox about low latency depend > on you sending one byte (yes, 1 byte). At real packet sizes Infiniband is no faster than > 10GbE. I heard similar. Thanks, -Garrett From owner-freebsd-infiniband@FreeBSD.ORG Tue Nov 20 19:39:38 2012 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9EC2E851 for ; Tue, 20 Nov 2012 19:39:38 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from mail-oa0-f54.google.com (mail-oa0-f54.google.com [209.85.219.54]) by mx1.freebsd.org (Postfix) with ESMTP id 5DC788FC08 for ; Tue, 20 Nov 2012 19:39:38 +0000 (UTC) Received: by mail-oa0-f54.google.com with SMTP id n9so8062274oag.13 for ; Tue, 20 Nov 2012 11:39:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=UwtltNc3qvlAP6bRFyslWjTc4WCOE+sTdJsVqZefrbw=; b=IpwDvdDQP/ZZdRpqGxeCAbk9q7aZKJN5jEIwmabtPkwO8QJLhWGpboOMZyPJMiOg4x 9Mt0+BgtpBdXfwmj9GeAPgVtLAxdeAYofXHHvZLAM0j4a92YKMhq2nw0PsvElCIjWVvD uGBmgzDP5PGvvb3KXLcY0wIgHf4sFHMQlvj9Ox9yVQL5Ckb0fV1iEgZtBwS6IjMiEX21 8W6ndplgl39d99x6c4x5PFCdY/50qo33qIU5we4t2+qq6iUz9g2eDBxXVicBzxjm2nqu bIemNEqZRc+6FN32ilzJlWO7ue+etTdjHn8pu5PypyZencYpLmYwYRYfyWPpz2FwoDEk qaJw== MIME-Version: 1.0 Received: by 10.60.169.170 with SMTP id af10mr14349551oec.17.1353440377620; Tue, 20 Nov 2012 11:39:37 -0800 (PST) Received: by 10.76.143.33 with HTTP; Tue, 20 Nov 2012 11:39:37 -0800 (PST) In-Reply-To: <20121119110945.ff40669f8feefc2e996df959@bsd-consulting.jp> References: <20121119110945.ff40669f8feefc2e996df959@bsd-consulting.jp> Date: Tue, 20 Nov 2012 11:39:37 -0800 Message-ID: Subject: Re: Infiniband device information From: Garrett Cooper To: Daichi GOTO Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-infiniband@freebsd.org X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 19:39:38 -0000 On Sun, Nov 18, 2012 at 6:09 PM, Daichi GOTO wrote: > Hi infiniband guys, thanks for IB ml :-) > > InfiniBand devices are becoming more and more important > in Japanese HPC business environments (In Japan, word "BigData" > is used instead of HPC in these days, a business buzz word). > > To promoto FreeBSD as the best product to use in BigData > businesses, we need InfiniBand devices lists, devices working > well on FreeBSD. > > http://wiki.freebsd.org/InfiniBand That should have been updated. I'll go and polish it up a bit later on today with more build instructions; the instructions only mention how to build some of the userland pieces -- not the kernel pieces, and the wiki page could be reformatted so the instructions are a bit more straightforward. > Should we do add "IB devices working list" on FreeBSD IB > wiki page? > > Some ISPs are trying to use IBs on their servers and they > are going to choose CentOS because of the lack of FreeBSD IB > support. Not good situation for FreeBSD. Needs improvement > I guess. A little birdy told me that you should shoot for ConnectX-2/-3 as your primary models to test. YMMV if you use other vendors/models (even though our version of the OFED stack supports more). mthca for instance is ancient, it has been validated internally, but there's a delta between what we have at Isilon and what's available to the community for various product-specific reasons. Thanks, -Garrett From owner-freebsd-infiniband@FreeBSD.ORG Tue Nov 20 20:09:27 2012 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0AD6CC8F; Tue, 20 Nov 2012 20:09:27 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id AC9DA8FC13; Tue, 20 Nov 2012 20:09:26 +0000 (UTC) Received: by mail-ob0-f182.google.com with SMTP id 16so8098643obc.13 for ; Tue, 20 Nov 2012 12:09:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=QUUBM39+/PGPk1DzyDLcA4MwGvPdR4sLJsfhIrYkj1M=; b=dF5f4v0Ma5SxuZo9uxx0cjxpXrA8/1RTurNRW0PonRLzlc4915F30EKOO3o5qlWj+L CDJLaKjjuq+Gog0YQQ5PoFFny3rtPCJ6IGARzs1+pKYufViJdpUw6sf34hq9g6ihqsVf brLmc+gVUc22R72xxteQiy3eCbZZRONy0LKsnF+GWWY274fLEyQDtNsoawon41mTYAU+ k5r9aFk82/RrKkL9RhSUCFDFPOmVbuWzLvBpHYXC9EmaWz40UiTbfvCAiVz+dWyV5S11 eY/2Pbu/BGBz7eQxYgPMwcd4zi1s0K/f/ct9dTqSJctwy33rCzw4JlCBaykOJsLs2paw UGeQ== MIME-Version: 1.0 Received: by 10.182.95.234 with SMTP id dn10mr14120396obb.97.1353442165851; Tue, 20 Nov 2012 12:09:25 -0800 (PST) Received: by 10.76.143.33 with HTTP; Tue, 20 Nov 2012 12:09:25 -0800 (PST) In-Reply-To: References: <20121119110945.ff40669f8feefc2e996df959@bsd-consulting.jp> Date: Tue, 20 Nov 2012 12:09:25 -0800 Message-ID: Subject: Re: Infiniband device information From: Garrett Cooper To: daichi@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-infiniband@freebsd.org X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 20:09:27 -0000 On Tue, Nov 20, 2012 at 11:39 AM, Garrett Cooper wrote: > On Sun, Nov 18, 2012 at 6:09 PM, Daichi GOTO wrote: >> Hi infiniband guys, thanks for IB ml :-) >> >> InfiniBand devices are becoming more and more important >> in Japanese HPC business environments (In Japan, word "BigData" >> is used instead of HPC in these days, a business buzz word). >> >> To promoto FreeBSD as the best product to use in BigData >> businesses, we need InfiniBand devices lists, devices working >> well on FreeBSD. >> >> http://wiki.freebsd.org/InfiniBand > > That should have been updated. I'll go and polish it up a bit later on > today with more build instructions; the instructions only mention how > to build some of the userland pieces -- not the kernel pieces, and the > wiki page could be reformatted so the instructions are a bit more > straightforward. The wiki page has been updated a bit. Thanks, -Garrett