From owner-freebsd-net@freebsd.org  Mon Aug 24 07:20:47 2015
Return-Path: <owner-freebsd-net@freebsd.org>
Delivered-To: freebsd-net@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id CBB169C1502;
 Mon, 24 Aug 2015 07:20:47 +0000 (UTC) (envelope-from hps@selasky.org)
Received: from mail.turbocat.net (mail.turbocat.net
 [IPv6:2a01:4f8:d16:4514::2])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id 8B58B1730;
 Mon, 24 Aug 2015 07:20:47 +0000 (UTC) (envelope-from hps@selasky.org)
Received: from laptop015.home.selasky.org
 (cm-176.74.213.204.customer.telag.net [176.74.213.204])
 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
 (No client certificate requested)
 by mail.turbocat.net (Postfix) with ESMTPSA id 2B5121FE023;
 Mon, 24 Aug 2015 09:20:44 +0200 (CEST)
Subject: Re: ix(intel) vs mlxen(mellanox) 10Gb performance
To: Rick Macklem <rmacklem@uoguelph.ca>, Daniel Braniss <danny@cs.huji.ac.il>
References: <1D52028A-B39F-4F9B-BD38-CB1D73BF5D56@cs.huji.ac.il>
 <55D43615.1030401@selasky.org>
 <2013503980.25726607.1439989235806.JavaMail.zimbra@uoguelph.ca>
 <20150820023024.GB996@michelle.fasterthan.com>
 <1153838447.28656490.1440193567940.JavaMail.zimbra@uoguelph.ca>
 <15D19823-08F7-4E55-BBD0-CE230F67D26E@cs.huji.ac.il>
 <818666007.28930310.1440244756872.JavaMail.zimbra@uoguelph.ca>
 <49173B1F-7B5E-4D59-8651-63D97B0CB5AC@cs.huji.ac.il>
 <1815942485.29539597.1440370972998.JavaMail.zimbra@uoguelph.ca>
Cc: pyunyh@gmail.com, FreeBSD stable <freebsd-stable@freebsd.org>,
 FreeBSD Net <freebsd-net@freebsd.org>, Slawa Olhovchenkov <slw@zxy.spb.ru>,
 Gleb Smirnoff <glebius@FreeBSD.org>,
 Christopher Forgeron <csforgeron@gmail.com>
From: Hans Petter Selasky <hps@selasky.org>
Message-ID: <55DAC623.60006@selasky.org>
Date: Mon, 24 Aug 2015 09:22:11 +0200
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101
 Thunderbird/38.1.0
MIME-Version: 1.0
In-Reply-To: <1815942485.29539597.1440370972998.JavaMail.zimbra@uoguelph.ca>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.20
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 24 Aug 2015 07:20:47 -0000

On 08/24/15 01:02, Rick Macklem wrote:
> The other thing is the degradation seems to cut the rate by about half each time.
> 300-->150-->70 I have no idea if this helps to explain it.

Might be a NUMA binding issue for the processes involved.

man cpuset

--HPS