From owner-freebsd-questions@FreeBSD.ORG Wed Dec 25 01:43:00 2013 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1C190BEC for ; Wed, 25 Dec 2013 01:43:00 +0000 (UTC) Received: from mail-wi0-x229.google.com (mail-wi0-x229.google.com [IPv6:2a00:1450:400c:c05::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 9E5B1139D for ; Wed, 25 Dec 2013 01:42:59 +0000 (UTC) Received: by mail-wi0-f169.google.com with SMTP id j9so3986554wiv.4 for ; Tue, 24 Dec 2013 17:42:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=PFJgaDVv/t5RrXBv62r2ZnVQsz46CxJxo0+LBz7ibF8=; b=F1YCD4HYkrCfrB9zz8N8oksbPh8DmmHWL33B4oJBduhn2u6XTm8Q1ihM680C/q6Tr+ e67LOFIzra3n1o321B784MZnOHGQReXx/UoXnGx0QInf3PkG8rY+53vyOW8Xpek8GITl epJrqe6PaIvEc1CakoMn5W3gZc27xrGaVKGPqEjXAgYjl2JAWN0BZkPPo3ubJQXY7Bs0 /NwbWSfyZMfZJ/3N6Hx5544Hrv6jy/utIz9NlGZbdjuWSysXXGsIhZRawAxCVhKVHT1E nc9pVoaYM5pjT1x5qXb+mkuD2T1l7b+ko7qpKM5TIgiI4Hto35roLQA1W29UB0b2DzuC B72w== MIME-Version: 1.0 X-Received: by 10.180.149.175 with SMTP id ub15mr25004299wib.44.1387935778079; Tue, 24 Dec 2013 17:42:58 -0800 (PST) Received: by 10.216.108.195 with HTTP; Tue, 24 Dec 2013 17:42:57 -0800 (PST) Received: by 10.216.108.195 with HTTP; Tue, 24 Dec 2013 17:42:57 -0800 (PST) In-Reply-To: <1387923358.71651.122.camel@btw.pki2.com> References: <1387915155.71651.118.camel@btw.pki2.com> <1387920273.71651.119.camel@btw.pki2.com> <1387923358.71651.122.camel@btw.pki2.com> Date: Tue, 24 Dec 2013 19:42:57 -0600 Message-ID: Subject: Re: InfiniBand in 10.0? From: iamatt To: Dennis Glatting Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: FreeBSD Mailing List X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Dec 2013 01:43:00 -0000 And then you need to see what mpi are you going to use. I'm curious which one you plan on using and how well it works on FreeBSD. On Dec 24, 2013 4:16 PM, "Dennis Glatting" wrote: > On Tue, 2013-12-24 at 15:46 -0600, iamatt wrote: > > A good batch of mellanox cables. Melanic had a bad run of FDR early > > in the year. Most FDR centers have been plagued with them. Even the > > back end of our DDN lustres has had issues. They typically do not > > just fail but downgrade to QDR rate. > > > > Ah. I though they were stripping the technology InfinBand. > > > > > On Dec 24, 2013 3:24 PM, "Dennis Glatting" wrote: > > On Tue, 2013-12-24 at 14:43 -0600, iamatt wrote: > > > Not sure if I would even bother with running IB on BSD. We > > have enough > > > problems just getting things stable with Linux and mellanox > > ofed . Add all > > > of the documented build problems with the mellanox FDR > > cables and recalls > > > and you are guaranteed a nightmare. TACC, navy, and almost > > every center I > > > am aware of has replaced all or most or some of their FDR, > > including us. > > > > > > > Replaced them with what? > > > > > > > As far as cards go it's all mellanox. Qlogic lost the game, > > still dealing > > > with a 12800 qlogic switch which is a total nightmare. > > Stick with > > > mellanox. Different vendors used to make their own fabric > > managers in the > > > early time of IB but now its best to stick with open ibd. > > > > > > FreeBSD and IB gives me the chills just thinking about it. > > > On Dec 24, 2013 1:59 PM, "Dennis Glatting" > > wrote: > > > > > > > > > > > This site appears dated: > > > > > > > > https://wiki.freebsd.org/InfiniBand > > > > > > > > I have a couple of questions. > > > > > > > > 1) Under 10.0-RC2, is InfiniBand a part of GENERIC or did > > I still have > > > > to modify the conf as specified in the wiki? I don't see > > any of the > > > > options from the wiki in 10.0-RC2 GENERIC. > > > > > > > > 2) Any recommendation on cards? What works, what doesn't? > > > > > > > > 3) What things to watch out for? > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > freebsd-questions@freebsd.org mailing list > > > > > > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > > > > To unsubscribe, send any mail to " > > > > freebsd-questions-unsubscribe@freebsd.org" > > > > > > > _______________________________________________ > > > freebsd-questions@freebsd.org mailing list > > > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > > > To unsubscribe, send any mail to > > "freebsd-questions-unsubscribe@freebsd.org" > > > > > > >