From owner-freebsd-infiniband@FreeBSD.ORG Tue May 27 13:46:45 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BDB4FA27 for ; Tue, 27 May 2014 13:46:45 +0000 (UTC) Received: from mail-ie0-x235.google.com (mail-ie0-x235.google.com [IPv6:2607:f8b0:4001:c03::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8CBC72BA8 for ; Tue, 27 May 2014 13:46:45 +0000 (UTC) Received: by mail-ie0-f181.google.com with SMTP id rp18so7255242iec.40 for ; Tue, 27 May 2014 06:46:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=WRF0l8c4KCT7KjqLU24u0ItiRvUbjQSBojnmqntPmNY=; b=mLZsW3T6u3rb6JtXU0I5D8eXIklrbiPX4aFZdyV0zDIf2+M0fFujyv+h0mjJX91cnB 6f9HWu9d9z+XlFyDQ2Ai6jIc36nOP/tkOGnq9Rl34rgbkhKkcAFRSaQMhQ6bJz1Qe4v3 TcaJ9sSNDFK8OB0WL/sjhWQrnWk2L81BTyUCX4yFoaLz0bN++9ZR0GfA5t2hi9j/i4fu aBpryGv3l03FdCaRzMyxk9YfHLbShupK7tEL1Mm3H66WELeZMB9a6ww3+pvKm+97Yx6F u6tBeDWsrrruSoLZXMTVwzIlX5uaFpZFjeNzyzHxrgQMQcDtOZTg70wR+hNp3YWteHd3 8dYA== X-Received: by 10.50.147.101 with SMTP id tj5mr34074468igb.27.1401198405056; Tue, 27 May 2014 06:46:45 -0700 (PDT) Received: from [10.1.68.236] (gs-sv-1-49-ac1.gsfc.nasa.gov. [198.119.56.43]) by mx.google.com with ESMTPSA id ri2sm7858423igc.1.2014.05.27.06.46.41 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 27 May 2014 06:46:41 -0700 (PDT) Message-ID: <5384973F.8030304@gmail.com> Date: Tue, 27 May 2014 09:46:39 -0400 From: John Jasen User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: freebsd-infiniband@freebsd.org Subject: basic questions on using Mellanox IB/40GbE converged cards Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 May 2014 13:46:45 -0000 I'm not quite sure if this belongs in freebsd-infiniband or another list, such as freebsd-net or -questions. If it should be filed elsewhere, my apologies. I am attempting to prototype a 40GbE router/packet filter, using FreeBSD 10.0 as the base operating system, with the test platform being a Dell R820. I have three dual port Mellanox cards in the system, one a ConnectX-3, the others OEM-branded; and now attempting performance tests. I'm hoping to get answers to the following questions: A) As the cards come up, I see, on each interface, "64B EQEs/CQEs supported by the device but not enabled". My googling has not yielded FreeBSD-specific results on this. Can 64B EQEs and CQEs be enabled under FreeBSD? Will it help with performance across multiple cards? If so, how can 64B EQE and CQEs be enabled? B) I also see "Using 5 TX rings" and "Using 2 RX rings" as the cards come up. Is this tunable under FreeBSD? If it is tunable, should it be tuned for a router? Maybe 4 TX and 3 RX, perhaps? And, if it is tunable and should be tuned, how is this accomplished? It appears hw.mlxen$N.conf.rx_rings and hw.mlxen$N.conf.tx_rings are read-only sysctls. Does that mean /boot/loader.conf? C) network tuning guides for FreeBSD are easy to find. Ones discussing 40GbE are not. Are there specific recommendations to obtain higher performance out of these cards, either general or specific to the mlx modules, that I should be using? D) In my tests, I am seeing various TCP retransmissions, which decrease throughput. What can be done to mitigate those? Thanks in advance for any assistance! -- John Jasen (jjasen@gmail.com)