From owner-freebsd-net@freebsd.org Sun Oct 18 21:00:40 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 222B6A181AB for ; Sun, 18 Oct 2015 21:00:40 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F17048B5 for ; Sun, 18 Oct 2015 21:00:39 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9IL0dgp089503 for ; Sun, 18 Oct 2015 21:00:39 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201510182100.t9IL0dgp089503@kenobi.freebsd.org> From: bugzilla-noreply@FreeBSD.org To: freebsd-net@FreeBSD.org Subject: Problem reports for freebsd-net@FreeBSD.org that need special attention X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 Date: Sun, 18 Oct 2015 21:00:39 +0000 Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Oct 2015 21:00:40 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- Open | 156226 | [lagg]: failover does not announce the failover t Open | 194515 | Fatal Trap 12 Kernel with vimage Open | 199136 | [if_tap] Added down_on_close sysctl variable to t 3 problems total for which you should take action. From owner-freebsd-net@freebsd.org Mon Oct 19 03:42:39 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 92A21A103E8 for ; Mon, 19 Oct 2015 03:42:39 +0000 (UTC) (envelope-from njwilliams@swin.edu.au) Received: from gpo1.cc.swin.edu.au (gpo1.cc.swin.edu.au [136.186.1.30]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 17AF41762 for ; Mon, 19 Oct 2015 03:42:38 +0000 (UTC) (envelope-from njwilliams@swin.edu.au) Received: from [136.186.242.235] (vpn242-235.cc.swin.edu.au [136.186.242.235]) by gpo1.cc.swin.edu.au (8.14.3/8.14.3) with ESMTP id t9J3gYdT006265 for ; Mon, 19 Oct 2015 14:42:35 +1100 To: freebsd-net@freebsd.org From: Nigel Williams Subject: MPTCP for FreeBSD repository on BitBucket/v0.51 update Message-ID: <562466AA.7020707@swin.edu.au> Date: Mon, 19 Oct 2015 14:42:34 +1100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2015 03:42:39 -0000 Hi, The MPTCP code is now available as a mercurial repository: - Repository: https://bitbucket.org/nw-swin/caia-mptcp-freebsd - Wiki: https://bitbucket.org/nw-swin/caia-mptcp-freebsd/wiki/ For those interested in trying the implementation/looking at the code, this should hopefully make the process a little easier (and save having to patch in updates). It should also make it possible to contribute code for those wishing to do so. Some details: - Has been branched off 'freebsd-head' at 'http://hg-beta.freebsd.org/base', and will be merged on a weekly basis. - I will be working off this repository so it will be up-to-date with recent changes. - In place of patch releases, release versions will now be tagged. - I'll also start to populate the 'Issues' section so that there is a better picture of current bugs/things TBD. The version has also been updated to v0.51. See: - http://caia.swin.edu.au/newtcp/mptcp/tools.html - OR https://bitbucket.org/nw-swin/caia-mptcp-freebsd/wiki/Home Functionally-wise this hasn't changed from the previous version, but has been merged with a recent revision of head. cheers, nigel From owner-freebsd-net@freebsd.org Mon Oct 19 04:02:17 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3646DA108DD for ; Mon, 19 Oct 2015 04:02:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 236931ED2 for ; Mon, 19 Oct 2015 04:02:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9J42Hlk038619 for ; Mon, 19 Oct 2015 04:02:17 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 203630] [Hyper-V] [nat] [tcp] 10.2 NAT bug in TCP stack or hyperv netsvc driver Date: Mon, 19 Oct 2015 04:02:15 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: weh@microsoft.com X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-bugs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2015 04:02:17 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203630 --- Comment #12 from Wei Hu --- (In reply to Eddy from comment #11) >I just tried to build a new kernel with the last "disable_csum_20151016.patch" you provided but I am stuck with an error: >/usr/src/sys/dev/hyperv/netvsc/hv_rndis_filter.c:828:11 error: unused variable `dev` [-Werror,-Wunused-variable] device_t dev = device->device; ^ You can just comment out this line since this variable is not used after applying the the patch. Let me know how it goes. -- You are receiving this mail because: You are on the CC list for the bug. From owner-freebsd-net@freebsd.org Mon Oct 19 05:59:33 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4F712A180B7 for ; Mon, 19 Oct 2015 05:59:33 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: from phabric-backend.isc.freebsd.org (phabric-backend.isc.freebsd.org [IPv6:2001:4f8:3:ffe0:406a:0:50:2]) by mx1.freebsd.org (Postfix) with ESMTP id 3BCCF7B2 for ; Mon, 19 Oct 2015 05:59:33 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: by phabric-backend.isc.freebsd.org (Postfix, from userid 1346) id 31F0210745E; Mon, 19 Oct 2015 05:59:33 +0000 (UTC) Date: Mon, 19 Oct 2015 05:59:33 +0000 To: freebsd-net@freebsd.org From: "lakshmi.n_msystechnologies.com (LN)" Reply-to: D1986+325+381818416dc12ca2@reviews.freebsd.org Subject: [Differential] [Commented On] D1986: Teach lagg(4) to change MTU Message-ID: <99ecb86f27ca628c92610867edb93730@localhost.localdomain> X-Priority: 3 X-Phabricator-Sent-This-Message: Yes X-Mail-Transport-Agent: MetaMTA X-Auto-Response-Suppress: All X-Phabricator-Mail-Tags: Thread-Topic: D1986: Teach lagg(4) to change MTU X-Herald-Rules: none X-Phabricator-To: X-Phabricator-To: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: Precedence: bulk In-Reply-To: References: Thread-Index: ODZhMzNlYThiYzMxOTgzYmRhMDE5M2Q2Yzk4IFYkhsU= MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset="utf-8" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2015 05:59:33 -0000 lakshmi.n_msystechnologies.com added a comment. @sbruno, A gentle remainder on em driver's LOR issue. Can you please share your findings. REPOSITORY rS FreeBSD src repository REVISION DETAIL https://reviews.freebsd.org/D1986 EMAIL PREFERENCES https://reviews.freebsd.org/settings/panel/emailpreferences/ To: rpokala-panasas.com, rstone Cc: sbruno, lakshmi.n_msystechnologies.com, emaste, ae, freebsd-net-list From owner-freebsd-net@freebsd.org Mon Oct 19 06:21:16 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 39DE1A18837 for ; Mon, 19 Oct 2015 06:21:16 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: from phabric-backend.isc.freebsd.org (phabric-backend.isc.freebsd.org [IPv6:2001:4f8:3:ffe0:406a:0:50:2]) by mx1.freebsd.org (Postfix) with ESMTP id 2444610B7 for ; Mon, 19 Oct 2015 06:21:16 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: by phabric-backend.isc.freebsd.org (Postfix, from userid 1346) id 20895107D2F; Mon, 19 Oct 2015 06:21:16 +0000 (UTC) Date: Mon, 19 Oct 2015 06:21:16 +0000 To: freebsd-net@freebsd.org From: "hrs (Hiroki Sato)" Reply-to: D1986+325+381818416dc12ca2@reviews.freebsd.org Subject: [Differential] [Commented On] D1986: Teach lagg(4) to change MTU Message-ID: X-Priority: 3 X-Phabricator-Sent-This-Message: Yes X-Mail-Transport-Agent: MetaMTA X-Auto-Response-Suppress: All X-Phabricator-Mail-Tags: , Thread-Topic: D1986: Teach lagg(4) to change MTU X-Herald-Rules: none X-Phabricator-To: X-Phabricator-To: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: Precedence: bulk In-Reply-To: References: Thread-Index: ODZhMzNlYThiYzMxOTgzYmRhMDE5M2Q2Yzk4IFYki9w= MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset="utf-8" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2015 06:21:16 -0000 hrs added a subscriber: hrs. hrs added a comment. It is true that this LOR is driver-specific but calling SIOCSIFMTU after acquiring a lock in lagg ioctl is not always safe. Change of lladdr suffers from the same situation and it was solved by using an asynchronous task queue to update addresses on each port. What do you think about piggybacking an MTU change to the queue by extending struct lagg_llq to a more generic one which makes it possible to handle per-port properties? REPOSITORY rS FreeBSD src repository REVISION DETAIL https://reviews.freebsd.org/D1986 EMAIL PREFERENCES https://reviews.freebsd.org/settings/panel/emailpreferences/ To: rpokala-panasas.com, rstone Cc: hrs, sbruno, lakshmi.n_msystechnologies.com, emaste, ae, freebsd-net-list From owner-freebsd-net@freebsd.org Mon Oct 19 08:15:40 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5A077A18A13 for ; Mon, 19 Oct 2015 08:15:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4715B8A6 for ; Mon, 19 Oct 2015 08:15:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9J8FehL006478 for ; Mon, 19 Oct 2015 08:15:40 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 202680] Silent data corruption on em(4) interfaces Date: Mon, 19 Oct 2015 08:15:39 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.2-STABLE X-Bugzilla-Keywords: IntelNetworking X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: KOT@MATPOCKuH.Ru X-Bugzilla-Status: In Progress X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2015 08:15:40 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=202680 --- Comment #14 from Dmitry Afanasiev --- I tried to use freebsd-current from nightly snapshot: FreeBSD sunrise0 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r289044: Thu Oct 8 21:21:40 UTC 2015 root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 But nothing changed - after 2 days uptime I again got incorrect MD5 checksum and messages from ssh: Corrupted MAC on input. Disconnecting: Packet corrupt -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-net@freebsd.org Mon Oct 19 08:46:57 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BAA06A126B7 for ; Mon, 19 Oct 2015 08:46:57 +0000 (UTC) (envelope-from sepherosa@gmail.com) Received: from mail-io0-x244.google.com (mail-io0-x244.google.com [IPv6:2607:f8b0:4001:c06::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8798E61A for ; Mon, 19 Oct 2015 08:46:57 +0000 (UTC) (envelope-from sepherosa@gmail.com) Received: by iodz80 with SMTP id z80so16603766iod.3 for ; Mon, 19 Oct 2015 01:46:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=12UV9DqVx7popEa8h8OAeRa1r8pAa4DaMVwo0FButmA=; b=Z07PWgJlSn+EVIqesYrt08KJ2DMyB1HjEp8WQMGqfqN4w8afp9mED2uvfS3/AQy/BC L/XQZ52TGlk5h8jK4nD/Fz7MaVPJ5L7EJbzxlQuQUrqhPi2n6uCW7XFeH9cxMSxnC3J7 xAVstPTMmMraEFvdxTqvxsyHXXqvefrqnHJkA9pmLaIM7bNNUZcqd+6sMBFkj+MZVr2M GWyKLVdaSLaJQXPHkxK9bIrN7KOKbV8appGbhVtKDzCcLxIn7dGdQzP0FJ6hpQnVyepp qhnvYW5qhrLXwSVwGCp5m8qfaz5yc7LWsYtUwZp5/kHbaf4Am1Jz8z7oo/qsHJRfhJYT VdWQ== MIME-Version: 1.0 X-Received: by 10.107.158.10 with SMTP id h10mr7634667ioe.24.1445244416947; Mon, 19 Oct 2015 01:46:56 -0700 (PDT) Received: by 10.107.37.137 with HTTP; Mon, 19 Oct 2015 01:46:56 -0700 (PDT) In-Reply-To: References: <561F6BFB.7080103@freebsd.org> <20151016154512.T15983@sola.nimnet.asn.au> Date: Mon, 19 Oct 2015 16:46:56 +0800 Message-ID: Subject: Re: nice stuff from cloudflare (and, we need something like ethtool!) From: Sepherosa Ziehau To: Jim Thompson Cc: Ian Smith , "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2015 08:46:57 -0000 On Fri, Oct 16, 2015 at 1:25 PM, Jim Thompson wrote: > > > >> On Oct 16, 2015, at 12:06 AM, Ian Smith wrote: >> >>> On Thu, 15 Oct 2015 17:03:55 +0800, Julian Elischer wrote: >>>> On 10/10/15 10:59 PM, Luigi Rizzo wrote: >>>> the nice folks at cloudflare implemented a nice feature >>>> in netmap that puts some queues of the NIC in netmap mode >>>> leaving others attached to the host stack >>>> >>>> https://blog.cloudflare.com/single-rx-queue-kernel-bypass-with-netmap/ >>>> >>>> and use ethtool (and native NIC filters) to steer traffic around. >>>> [FWIW, the chelsio native netmap driver is similar except that >>>> the netmap queue has a different MAC address] >>>> >>>> While their code was developed on linux, it should run >>>> almost unmodified on FreeBSD (and we plan to import it soon), >>>> except for the fact that we don't have ethtool hence no >>>> device-independent mechanism to configure traffic steering. >>>> >>>> We really need to address the latter. >>> >>> I suspect the answer may be a device dependent sysctl >> >> Interesting; care to flesh out your ideas a bit on how that might work? >> >> I've done nothing more than skim ethtool(8) on linuxcommand.org, and >> wondered why its functionality wasn't incorporated into ifconfig, but >> then ifconfig (on FreeBSD anyway) is tending towards obesity already > > Luigi already did netlink sockets for FreeBSD. > > https://github.com/luigirizzo/netlink-freebsd ha, the netlink for BSD, interesting :) -- Tomorrow Will Never Die From owner-freebsd-net@freebsd.org Mon Oct 19 09:18:58 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4BE72A185AD for ; Mon, 19 Oct 2015 09:18:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 38F3F1C52 for ; Mon, 19 Oct 2015 09:18:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9J9IwjB092967 for ; Mon, 19 Oct 2015 09:18:58 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 203630] [Hyper-V] [nat] [tcp] 10.2 NAT bug in TCP stack or hyperv netsvc driver Date: Mon, 19 Oct 2015 09:18:57 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: onyx@netfusion.fr X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-bugs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2015 09:18:58 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203630 --- Comment #13 from Eddy --- (In reply to Wei Hu from comment #12) After some tests, "disable_csum_20151016.patch" doesn't solve the issue for me. The last r285236 patch worked. Do I have to first apply the r285236 patch and then the disable_csum_20151016.patch? I only applied the new patch on clean sources. -- You are receiving this mail because: You are on the CC list for the bug. From owner-freebsd-net@freebsd.org Mon Oct 19 13:52:43 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CD9CAA1849F for ; Mon, 19 Oct 2015 13:52:43 +0000 (UTC) (envelope-from lars@netapp.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id ADD6625F for ; Mon, 19 Oct 2015 13:52:43 +0000 (UTC) (envelope-from lars@netapp.com) Received: by mailman.ysv.freebsd.org (Postfix) id AAAE5A1849E; Mon, 19 Oct 2015 13:52:43 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AA2C2A1849D for ; Mon, 19 Oct 2015 13:52:43 +0000 (UTC) (envelope-from lars@netapp.com) Received: from mx143.netapp.com (mx143.netapp.com [216.240.21.24]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "mx143.netapp.com", Issuer "Symantec Class 3 Secure Server CA - G4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6AB5525D; Mon, 19 Oct 2015 13:52:43 +0000 (UTC) (envelope-from lars@netapp.com) X-IronPort-AV: E=Sophos;i="5.17,702,1437462000"; d="asc'?scan'208";a="73100698" Received: from hioexcmbx04-prd.hq.netapp.com ([10.122.105.37]) by mx143-out.netapp.com with ESMTP; 19 Oct 2015 06:52:18 -0700 Received: from HIOEXCMBX07-PRD.hq.netapp.com (10.122.105.40) by hioexcmbx04-prd.hq.netapp.com (10.122.105.37) with Microsoft SMTP Server (TLS) id 15.0.1104.5; Mon, 19 Oct 2015 06:52:18 -0700 Received: from HIOEXCMBX07-PRD.hq.netapp.com ([::1]) by hioexcmbx07-prd.hq.netapp.com ([fe80::e1d9:911e:3048:d510%21]) with mapi id 15.00.1104.000; Mon, 19 Oct 2015 06:52:18 -0700 From: "Eggert, Lars" To: "net@freebsd.org" CC: "jfv@FreeBSD.org" , "ricera10@gmail.com" Subject: ixl 40G bad performance? Thread-Topic: ixl 40G bad performance? Thread-Index: AQHRCnVeigW3mf41PU+t9k/mAjNcOg== Date: Mon, 19 Oct 2015 13:52:17 +0000 Message-ID: <79830D9D-94E6-47A9-92B9-D63DF5432272@netapp.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-mailer: Apple Mail (2.3094) x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [10.122.56.79] Content-Type: multipart/signed; boundary="Apple-Mail=_B52087A4-0A91-43E5-894C-055F2D0B0D7F"; protocol="application/pgp-signature"; micalg=pgp-sha256 MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2015 13:52:44 -0000 --Apple-Mail=_B52087A4-0A91-43E5-894C-055F2D0B0D7F Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi, I'm running a few simple tests on -CURRENT with a pair of dual-port = Intel XL710 boards, which are seen by the kernel as: ixl0: mem = 0xdc800000-0xdcffffff,0xdd808000-0xdd80ffff irq 32 at device 0.0 on pci3 ixl0: Using MSIX interrupts with 33 vectors ixl0: f4.40 a1.4 n04.53 e80001dca ixl0: Using defaults for TSO: 65518/35/2048 ixl0: Ethernet address: 68:05:ca:32:0b:98 ixl0: PCI Express Bus: Speed 8.0GT/s Width x8 ixl0: netmap queues/slots: TX 32/1024, RX 32/1024 ixl1: mem = 0xdc000000-0xdc7fffff,0xdd800000-0xdd807fff irq 32 at device 0.1 on pci3 ixl1: Using MSIX interrupts with 33 vectors ixl1: f4.40 a1.4 n04.53 e80001dca ixl1: Using defaults for TSO: 65518/35/2048 ixl1: Ethernet address: 68:05:ca:32:0b:99 ixl1: PCI Express Bus: Speed 8.0GT/s Width x8 ixl1: netmap queues/slots: TX 32/1024, RX 32/1024 ixl0: link state changed to UP ixl1: link state changed to UP I have two identical machines connected with patch cables (no switch). = iperf performance is bad: # iperf -c 10.0.1.2 ------------------------------------------------------------ Client connecting to 10.0.1.2, TCP port 5001 TCP window size: 32.5 KByte (default) ------------------------------------------------------------ [ 3] local 10.0.1.1 port 19238 connected with 10.0.1.2 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 3.91 GBytes 3.36 Gbits/sec As is flood ping latency: # sudo ping -f 10.0.1.2 PING 10.0.1.2 (10.0.1.2): 56 data bytes .^C --- 10.0.1.2 ping statistics --- 41927 packets transmitted, 41926 packets received, 0.0% packet loss round-trip min/avg/max/stddev =3D 0.084/0.116/0.145/0.002 ms Any ideas on what's going on here? Testing 10G ix interfaces between the = same two machines results in 9.39 Gbits/sec and flood ping latencies of = 17 usec. Thanks, Lars PS: Full dmesg attached. Copyright (c) 1992-2015 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights = reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 11.0-CURRENT #2 483de3c(muclab)-dirty: Mon Oct 19 11:01:16 CEST = 2015 = elars@laurel.muccbc.hq.netapp.com:/usr/home/elars/obj/usr/home/elars/src/s= ys/MUCLAB amd64 FreeBSD clang version 3.7.0 (tags/RELEASE_370/final 246257) 20150906 VT(vga): resolution 640x480 CPU: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz (2000.05-MHz K8-class CPU) Origin=3D"GenuineIntel" Id=3D0x206d7 Family=3D0x6 Model=3D0x2d = Stepping=3D7 = Features=3D0xbfebfbff = Features2=3D0x1fbee3ff AMD Features=3D0x2c100800 AMD Features2=3D0x1 XSAVE Features=3D0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics real memory =3D 137438953472 (131072 MB) avail memory =3D 133484290048 (127300 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: < > FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs FreeBSD/SMP: 2 package(s) x 8 core(s) x 2 SMT threads cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 6 cpu7 (AP): APIC ID: 7 cpu8 (AP): APIC ID: 8 cpu9 (AP): APIC ID: 9 cpu10 (AP): APIC ID: 10 cpu11 (AP): APIC ID: 11 cpu12 (AP): APIC ID: 12 cpu13 (AP): APIC ID: 13 cpu14 (AP): APIC ID: 14 cpu15 (AP): APIC ID: 15 cpu16 (AP): APIC ID: 32 cpu17 (AP): APIC ID: 33 cpu18 (AP): APIC ID: 34 cpu19 (AP): APIC ID: 35 cpu20 (AP): APIC ID: 36 cpu21 (AP): APIC ID: 37 cpu22 (AP): APIC ID: 38 cpu23 (AP): APIC ID: 39 cpu24 (AP): APIC ID: 40 cpu25 (AP): APIC ID: 41 cpu26 (AP): APIC ID: 42 cpu27 (AP): APIC ID: 43 cpu28 (AP): APIC ID: 44 cpu29 (AP): APIC ID: 45 cpu30 (AP): APIC ID: 46 cpu31 (AP): APIC ID: 47 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard ioapic2 irqs 48-71 on motherboard random: entropy device external interface module_register_init: MOD_LOAD (vesa, 0xffffffff8094fb90, 0) error 19 netmap: loaded module vtvga0: on motherboard smbios0: at iomem 0xf04d0-0xf04ee on = motherboard smbios0: Version: 2.7, BCD Revision: 2.7 cryptosoft0: on motherboard aesni0: on motherboard acpi0: on motherboard acpi0: Power Button (fixed) cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cpu4: on acpi0 cpu5: on acpi0 cpu6: on acpi0 cpu7: on acpi0 cpu8: on acpi0 cpu9: on acpi0 cpu10: on acpi0 cpu11: on acpi0 cpu12: on acpi0 cpu13: on acpi0 cpu14: on acpi0 cpu15: on acpi0 cpu16: on acpi0 cpu17: on acpi0 cpu18: on acpi0 cpu19: on acpi0 cpu20: on acpi0 cpu21: on acpi0 cpu22: on acpi0 cpu23: on acpi0 cpu24: on acpi0 cpu25: on acpi0 cpu26: on acpi0 cpu27: on acpi0 cpu28: on acpi0 cpu29: on acpi0 cpu30: on acpi0 cpu31: on acpi0 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: port 0x70-0x71 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 350 Event timer "HPET1" frequency 14318180 Hz quality 340 Event timer "HPET2" frequency 14318180 Hz quality 340 Event timer "HPET3" frequency 14318180 Hz quality 340 Event timer "HPET4" frequency 14318180 Hz quality 340 Event timer "HPET5" frequency 14318180 Hz quality 340 Event timer "HPET6" frequency 14318180 Hz quality 340 Event timer "HPET7" frequency 14318180 Hz quality 340 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: irq 26 at device 1.0 on pci0 pci1: on pcib1 pcib2: irq 26 at device 1.1 on pci0 pci2: on pcib2 igb0: port = 0x8020-0x803f mem 0xdf820000-0xdf83ffff,0xdf844000-0xdf847fff irq 27 at = device 0.0 on pci2 igb0: Using MSIX interrupts with 9 vectors igb0: Ethernet address: 00:25:90:9b:73:2e igb0: Bound queue 0 to cpu 0 igb0: Bound queue 1 to cpu 1 igb0: Bound queue 2 to cpu 2 igb0: Bound queue 3 to cpu 3 igb0: Bound queue 4 to cpu 4 igb0: Bound queue 5 to cpu 5 igb0: Bound queue 6 to cpu 6 igb0: Bound queue 7 to cpu 7 igb0: netmap queues/slots: TX 8/1024, RX 8/1024 igb1: port = 0x8000-0x801f mem 0xdf800000-0xdf81ffff,0xdf840000-0xdf843fff irq 30 at = device 0.1 on pci2 igb1: Using MSIX interrupts with 9 vectors igb1: Ethernet address: 00:25:90:9b:73:2f igb1: Bound queue 0 to cpu 8 igb1: Bound queue 1 to cpu 9 igb1: Bound queue 2 to cpu 10 igb1: Bound queue 3 to cpu 11 igb1: Bound queue 4 to cpu 12 igb1: Bound queue 5 to cpu 13 igb1: Bound queue 6 to cpu 14 igb1: Bound queue 7 to cpu 15 igb1: netmap queues/slots: TX 8/1024, RX 8/1024 pcib3: irq 33 at device 2.0 on pci0 pci3: on pcib3 pci3: at device 0.0 (no driver attached) pci3: at device 0.1 (no driver attached) pcib4: irq 33 at device 2.2 on pci0 pci4: on pcib4 pcib5: irq 41 at device 3.0 on pci0 pci5: on pcib5 pcib6: irq 41 at device 3.2 on pci0 pci6: on pcib6 ix0: = port 0x7020-0x703f mem 0xdf180000-0xdf1fffff,0xdf604000-0xdf607fff irq = 42 at device 0.0 on pci6 ix0: Using MSIX interrupts with 33 vectors ix0: Ethernet address: 90:e2:ba:77:d4:9c ix0: PCI Express Bus: Speed 5.0GT/s Width x8 ix0: netmap queues/slots: TX 32/2048, RX 32/2048 ix1: = port 0x7000-0x701f mem 0xdf100000-0xdf17ffff,0xdf600000-0xdf603fff irq = 45 at device 0.1 on pci6 ix1: Using MSIX interrupts with 33 vectors ix1: Ethernet address: 90:e2:ba:77:d4:9d ix1: PCI Express Bus: Speed 5.0GT/s Width x8 ix1: netmap queues/slots: TX 32/2048, RX 32/2048 pcib7: irq 16 at device 17.0 on pci0 pci7: on pcib7 isci0: port = 0x6000-0x60ff mem 0xde07c000-0xde07ffff,0xddc00000-0xddffffff irq 16 at = device 0.0 on pci7 pci0: at device 22.0 (no driver attached) pci0: at device 22.1 (no driver attached) ehci0: mem 0xdf923000-0xdf9233ff irq = 16 at device 26.0 on pci0 usbus0: EHCI version 1.0 usbus0 on ehci0 pcib8: irq 17 at device 28.0 on pci0 pci8: on pcib8 pcib9: irq 19 at device 28.7 on pci0 pci9: on pcib9 pcib10: at device 0.0 on pci9 pci10: on pcib10 pcib11: at device 0.0 on pci10 pci11: on pcib11 pcib12: at device 0.0 on pci11 pci12: on pcib12 vgapci0: mem = 0xdb000000-0xdbffffff,0xdf000000-0xdf003fff,0xde800000-0xdeffffff irq 19 = at device 0.0 on pci12 vgapci0: Boot video device pcib13: at device 1.0 on pci10 pci13: on pcib13 ehci1: mem 0xdf922000-0xdf9223ff irq = 23 at device 29.0 on pci0 usbus1: EHCI version 1.0 usbus1 on ehci1 pcib14: at device 30.0 on pci0 pci14: on pcib14 isab0: at device 31.0 on pci0 isa0: on isab0 ahci0: port = 0x9050-0x9057,0x9040-0x9043,0x9030-0x9037,0x9020-0x9023,0x9000-0x901f = mem 0xdf921000-0xdf9217ff irq 18 at device 31.2 on pci0 ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 ahcich4: at channel 4 on ahci0 ahcich5: at channel 5 on ahci0 ahciem0: on ahci0 ichsmb0: port 0x1180-0x119f mem = 0xdf920000-0xdf9200ff irq 18 at device 31.3 on pci0 smbus0: on ichsmb0 smb0: on smbus0 pcib15: on acpi0 pci15: on pcib15 pcib16: on acpi0 pci16: on pcib16 pcib17: irq 57 at device 2.0 on pci16 pci17: on pcib17 pcib18: irq 64 at device 3.0 on pci16 pci18: on pcib18 pcib19: irq 64 at device 3.2 on pci16 pci19: on pcib19 pcib20: on acpi0 pci20: on pcib20 acpi_button0: on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: console (115200,n,8,1) uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 ipmi0: port 0xca2,0xca3 on acpi0 ipmi0: KCS mode found at io 0xca2 on acpi ichwd0 on isa0 ichwd0: ICH WDT present but disabled in BIOS or hardware device_attach: ichwd0 attach returned 6 ichwd0 at port 0x430-0x437,0x460-0x47f on isa0 ichwd0: ICH WDT present but disabled in BIOS or hardware device_attach: ichwd0 attach returned 6 orm0: at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff on isa0 coretemp0: on cpu0 est0: on cpu0 coretemp1: on cpu1 est1: on cpu1 coretemp2: on cpu2 est2: on cpu2 coretemp3: on cpu3 est3: on cpu3 coretemp4: on cpu4 est4: on cpu4 coretemp5: on cpu5 est5: on cpu5 coretemp6: on cpu6 est6: on cpu6 coretemp7: on cpu7 est7: on cpu7 coretemp8: on cpu8 est8: on cpu8 coretemp9: on cpu9 est9: on cpu9 coretemp10: on cpu10 est10: on cpu10 coretemp11: on cpu11 est11: on cpu11 coretemp12: on cpu12 est12: on cpu12 coretemp13: on cpu13 est13: on cpu13 coretemp14: on cpu14 est14: on cpu14 coretemp15: on cpu15 est15: on cpu15 coretemp16: on cpu16 est16: on cpu16 coretemp17: on cpu17 est17: on cpu17 coretemp18: on cpu18 est18: on cpu18 coretemp19: on cpu19 est19: on cpu19 coretemp20: on cpu20 est20: on cpu20 coretemp21: on cpu21 est21: on cpu21 coretemp22: on cpu22 est22: on cpu22 coretemp23: on cpu23 est23: on cpu23 coretemp24: on cpu24 est24: on cpu24 coretemp25: on cpu25 est25: on cpu25 coretemp26: on cpu26 est26: on cpu26 coretemp27: on cpu27 est27: on cpu27 coretemp28: on cpu28 est28: on cpu28 coretemp29: on cpu29 est29: on cpu29 coretemp30: on cpu30 est30: on cpu30 coretemp31: on cpu31 est31: on cpu31 fuse-freebsd: version 0.4.4, FUSE ABI 7.8 Timecounters tick every 1.000 msec iw_cxgb: Chelsio T3 RDMA Driver loaded IPsec: Initialized Security Association Processing. ipfw2 (+ipv6) initialized, divert loadable, nat enabled, default to = accept, logging disabled DUMMYNET 0 with IPv6 initialized (100409) load_dn_sched dn_sched FIFO loaded load_dn_sched dn_sched PRIO loaded load_dn_sched dn_sched QFQ loaded load_dn_sched dn_sched RR loaded load_dn_sched dn_sched WF2Q+ loaded ipmi0: IPMI device rev. 1, firmware rev. 2.35, version 2.0 ipmi0: Number of channels 3 ipmi0: Attached watchdog usbus0: 480Mbps High Speed USB v2.0 usbus1: 480Mbps High Speed USB v2.0 ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 ses0 at ahciem0 bus 0 scbus7 target 0 lun 0 ses0: SEMB S-E-S 2.00 device ses0: SEMB SES Device ada0 at ahcich0 bus 0 scbus1 target 0 lun 0 ada0: ACS-2 ATA SATA 3.x device ada0: Serial Number CVCV3102050X180EGN ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 171705MB (351651888 512 byte sectors) ada0: quirks=3D0x1<4K> random: unblocking device. Sending DHCP Discover packet from interface igb0 (00:25:90:9b:73:2e) Sending DHCP Discover packet from interface igb1 (00:25:90:9b:73:2f) Sending DHCP Discover packet from interface ix0 (90:e2:ba:77:d4:9c) Sending DHCP Discover packet from interface ix1 (90:e2:ba:77:d4:9d) ix0: link state changed to UP ix1: link state changed to UP uhub1: 2 ports with 2 removable, self powered uhub0: 2 ports with 2 removable, self powered ugen1.2: at usbus1 uhub2: = on usbus1 ugen0.2: at usbus0 uhub3: = on usbus0 uhub3: 6 ports with 6 removable, self powered uhub2: 8 ports with 8 removable, self powered ugen0.3: at usbus0 ukbd0: on usbus0 ums0: on usbus0 ums0: 3 buttons and [Z] coordinates ID=3D0 igb0: link state changed to UP Received DHCP Offer packet on igb0 from 192.168.0.2 (accepted) (no root = path) (boot_file) Received DHCP Offer packet on igb0 from 192.168.0.2 (ignored) (no root = path) (boot_file) Received DHCP Offer packet on igb0 from 192.168.0.2 (ignored) (no root = path) (boot_file) Sending DHCP Request packet from interface igb0 (00:25:90:9b:73:2e) Received DHCP Ack packet on igb0 from 192.168.0.2 (accepted) (got root = path) DHCP timeout for interface igb1 DHCP timeout for interface ix0 DHCP timeout for interface ix1 Wired loader interface (IP 192.168.11.1) is igb0 igb0 at 192.168.11.1 server 192.168.0.2 boot file /pxe/pxelinux.0 subnet mask 255.255.0.0 router 192.168.0.2 rootfs = 192.168.0.10:/muclab/image/machines/phobos2 hostname phobos2 Adjusted interface igb0 Shutdown interface igb1 Shutdown interface ix0 ix0: link state changed to DOWN Shutdown interface ix1 ix1: link state changed to DOWN SMP: AP CPU #31 Launched! SMP: AP CPU #10 Launched! SMP: AP CPU #25 Launched! SMP: AP CPU #14 Launched! SMP: AP CPU #30 Launched! SMP: AP CPU #12 Launched! SMP: AP CPU #17 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #28 Launched! SMP: AP CPU #13 Launched! SMP: AP CPU #24 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #27 Launched! SMP: AP CPU #8 Launched! SMP: AP CPU #29 Launched! SMP: AP CPU #11 Launched! SMP: AP CPU #20 Launched! SMP: AP CPU #15 Launched! SMP: AP CPU #26 Launched! SMP: AP CPU #9 Launched! SMP: AP CPU #22 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #18 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #21 Launched! SMP: AP CPU #16 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #23 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #19 Launched! Timecounter "TSC" frequency 2000045308 Hz quality 1000 hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 = IAP/4/48/0x3ff = IAF/3/48/0x67 Trying to mount root from nfs: []... NFS ROOT: 192.168.0.10:/muclab/image/machines/phobos2 ixl0: mem = 0xdc800000-0xdcffffff,0xdd808000-0xdd80ffff irq 32 at device 0.0 on pci3 ixl0: Using MSIX interrupts with 33 vectors ixl0: f4.40 a1.4 n04.53 e80001dca ixl0: Using defaults for TSO: 65518/35/2048 ixl0: Ethernet address: 68:05:ca:32:15:d0 ixl0: PCI Express Bus: Speed 8.0GT/s Width x8 queues is 0xfffffe0008d03000 ixl0: netmap queues/slots: TX 32/1024, RX 32/1024 ixl1: mem = 0xdc000000-0xdc7fffff,0xdd800000-0xdd807fff irq 32 at device 0.1 on pci3 ixl1: Using MSIX interrupts with 33 vectors ixl1: f4.40 a1.4 n04.53 e80001dca ixl1: Using defaults for TSO: 65518/35/2048 ixl1: Ethernet address: 68:05:ca:32:15:d1 ixl1: PCI Express Bus: Speed 8.0GT/s Width x8 queues is 0xfffffe0009227000 ixl1: netmap queues/slots: TX 32/1024, RX 32/1024 ix0: link state changed to UP ix1: link state changed to UP ixl0: link state changed to UP ixl1: link state changed to UP --Apple-Mail=_B52087A4-0A91-43E5-894C-055F2D0B0D7F Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="signature.asc" Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- iQCVAwUBViT1lNZcnpRveo1xAQiTrwP/azi5wlBySkUjnLAZxrTkQCeU/nGOwPDh NRm6hJrMr45GViOTUtFYHrKf6Lc9eLgnMvZfxGEfKbtWZXrE82kOMe+7zTYmpSkr KMtxx8mDC89GYnqxmmQMqUz3esezFj/uDnGl2tlAr50Tx9hfkA+OnA56PWdpvXUb 7nca8/RVEU8= =tpRf -----END PGP SIGNATURE----- --Apple-Mail=_B52087A4-0A91-43E5-894C-055F2D0B0D7F-- From owner-freebsd-net@freebsd.org Mon Oct 19 14:20:26 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 83B40A18F7A for ; Mon, 19 Oct 2015 14:20:26 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 606931865 for ; Mon, 19 Oct 2015 14:20:26 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 5D2F6A18F78; Mon, 19 Oct 2015 14:20:26 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5CB84A18F77 for ; Mon, 19 Oct 2015 14:20:26 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-lb0-x234.google.com (mail-lb0-x234.google.com [IPv6:2a00:1450:4010:c04::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D35F51864; Mon, 19 Oct 2015 14:20:25 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by lbbwb3 with SMTP id wb3so83911206lbb.1; Mon, 19 Oct 2015 07:20:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=E9/V31xGJa61hKG2qfyAqbFTMw63SDMZNygjzFSLYlA=; b=dCGL0Xmwntm5kHVHFlolnkuqdirrdh8k60sIvZJZmBbQ7V2/dYnimnJQoHeXoW40dD 3F3fcQ/ruPyEHhBco/s0Xrp9IwZFbhzOdYQxvgRXQyaZdkzFd2IgTeLpsmFNDY1Ya+/X W2vsvqPrtTZYyNmuDi1NiZdSPCC3PNAjweDCKUzlWazRVih7cdEQFZLprL1MZkYp4jsW foff9/UdiocCwQCqDw05jYzR26QnkuW0RD+Y5MRrlFnzbqnfHFQPaPLa38KO6jxL2W7c tHsH3Q0cpcyy7EHd8otDIuKLcLNa03lBbo4wmVUfa3fuPnsJhtl/32G3GSOIUgSb6KM2 Cduw== MIME-Version: 1.0 X-Received: by 10.112.167.101 with SMTP id zn5mr14698187lbb.18.1445264423910; Mon, 19 Oct 2015 07:20:23 -0700 (PDT) Sender: rizzo.unipi@gmail.com Received: by 10.114.177.4 with HTTP; Mon, 19 Oct 2015 07:20:23 -0700 (PDT) In-Reply-To: <79830D9D-94E6-47A9-92B9-D63DF5432272@netapp.com> References: <79830D9D-94E6-47A9-92B9-D63DF5432272@netapp.com> Date: Mon, 19 Oct 2015 07:20:23 -0700 X-Google-Sender-Auth: _ac3sdREsU6bohIquX8TSNkYOFA Message-ID: Subject: Re: ixl 40G bad performance? From: Luigi Rizzo To: "Eggert, Lars" Cc: "net@freebsd.org" , "jfv@FreeBSD.org" , "ricera10@gmail.com" , Giuseppe Lettieri Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2015 14:20:26 -0000 i would look at the following: - c states and clock speed - make sure you never go below C1, and fix the clock speed to max. Sure these parameters also affect the 10G card, but there may be strange interaction that trigger the power saving modes in different ways - interrupt moderation (may affect ping latency, do not remember how it is set in ixl but probably a sysctl - number of queues (32 is a lot i wouldn't use more than 4-8), may affect cpu-socket affinity - tso and flow director - i have seen bad effects of accelerations so i would run the iperf test with of these features disabled on both sides, and then enable them one at a time - queue sizes - the driver seems to use 1024 slots which is about 1.5 MB queued, which in turn means you have 300us (and possibly half of that) to drain the queue at 40Gbit/s. 150-300us may seem an eternity, but if a couple of cores fall into c7 your budget is gone and the loss will trigger a retransmission and window halving etc. cheers luigi On Mon, Oct 19, 2015 at 6:52 AM, Eggert, Lars wrote: > Hi, > > I'm running a few simple tests on -CURRENT with a pair of dual-port Intel XL710 boards, which are seen by the kernel as: > > ixl0: mem 0xdc800000-0xdcffffff,0xdd808000-0xdd80ffff irq 32 at device 0.0 on pci3 > ixl0: Using MSIX interrupts with 33 vectors > ixl0: f4.40 a1.4 n04.53 e80001dca > ixl0: Using defaults for TSO: 65518/35/2048 > ixl0: Ethernet address: 68:05:ca:32:0b:98 > ixl0: PCI Express Bus: Speed 8.0GT/s Width x8 > ixl0: netmap queues/slots: TX 32/1024, RX 32/1024 > ixl1: mem 0xdc000000-0xdc7fffff,0xdd800000-0xdd807fff irq 32 at device 0.1 on pci3 > ixl1: Using MSIX interrupts with 33 vectors > ixl1: f4.40 a1.4 n04.53 e80001dca > ixl1: Using defaults for TSO: 65518/35/2048 > ixl1: Ethernet address: 68:05:ca:32:0b:99 > ixl1: PCI Express Bus: Speed 8.0GT/s Width x8 > ixl1: netmap queues/slots: TX 32/1024, RX 32/1024 > ixl0: link state changed to UP > ixl1: link state changed to UP > > I have two identical machines connected with patch cables (no switch). iperf performance is bad: > > # iperf -c 10.0.1.2 > ------------------------------------------------------------ > Client connecting to 10.0.1.2, TCP port 5001 > TCP window size: 32.5 KByte (default) > ------------------------------------------------------------ > [ 3] local 10.0.1.1 port 19238 connected with 10.0.1.2 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-10.0 sec 3.91 GBytes 3.36 Gbits/sec > > As is flood ping latency: > > # sudo ping -f 10.0.1.2 > PING 10.0.1.2 (10.0.1.2): 56 data bytes > .^C > --- 10.0.1.2 ping statistics --- > 41927 packets transmitted, 41926 packets received, 0.0% packet loss > round-trip min/avg/max/stddev = 0.084/0.116/0.145/0.002 ms > > Any ideas on what's going on here? Testing 10G ix interfaces between the same two machines results in 9.39 Gbits/sec and flood ping latencies of 17 usec. > > Thanks, > Lars > > PS: Full dmesg attached. > > Copyright (c) 1992-2015 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 11.0-CURRENT #2 483de3c(muclab)-dirty: Mon Oct 19 11:01:16 CEST 2015 > elars@laurel.muccbc.hq.netapp.com:/usr/home/elars/obj/usr/home/elars/src/sys/MUCLAB amd64 > FreeBSD clang version 3.7.0 (tags/RELEASE_370/final 246257) 20150906 > VT(vga): resolution 640x480 > CPU: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz (2000.05-MHz K8-class CPU) > Origin="GenuineIntel" Id=0x206d7 Family=0x6 Model=0x2d Stepping=7 > Features=0xbfebfbff > Features2=0x1fbee3ff > AMD Features=0x2c100800 > AMD Features2=0x1 > XSAVE Features=0x1 > VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID > TSC: P-state invariant, performance statistics > real memory = 137438953472 (131072 MB) > avail memory = 133484290048 (127300 MB) > Event timer "LAPIC" quality 600 > ACPI APIC Table: < > > FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs > FreeBSD/SMP: 2 package(s) x 8 core(s) x 2 SMT threads > cpu0 (BSP): APIC ID: 0 > cpu1 (AP): APIC ID: 1 > cpu2 (AP): APIC ID: 2 > cpu3 (AP): APIC ID: 3 > cpu4 (AP): APIC ID: 4 > cpu5 (AP): APIC ID: 5 > cpu6 (AP): APIC ID: 6 > cpu7 (AP): APIC ID: 7 > cpu8 (AP): APIC ID: 8 > cpu9 (AP): APIC ID: 9 > cpu10 (AP): APIC ID: 10 > cpu11 (AP): APIC ID: 11 > cpu12 (AP): APIC ID: 12 > cpu13 (AP): APIC ID: 13 > cpu14 (AP): APIC ID: 14 > cpu15 (AP): APIC ID: 15 > cpu16 (AP): APIC ID: 32 > cpu17 (AP): APIC ID: 33 > cpu18 (AP): APIC ID: 34 > cpu19 (AP): APIC ID: 35 > cpu20 (AP): APIC ID: 36 > cpu21 (AP): APIC ID: 37 > cpu22 (AP): APIC ID: 38 > cpu23 (AP): APIC ID: 39 > cpu24 (AP): APIC ID: 40 > cpu25 (AP): APIC ID: 41 > cpu26 (AP): APIC ID: 42 > cpu27 (AP): APIC ID: 43 > cpu28 (AP): APIC ID: 44 > cpu29 (AP): APIC ID: 45 > cpu30 (AP): APIC ID: 46 > cpu31 (AP): APIC ID: 47 > ioapic0 irqs 0-23 on motherboard > ioapic1 irqs 24-47 on motherboard > ioapic2 irqs 48-71 on motherboard > random: entropy device external interface > module_register_init: MOD_LOAD (vesa, 0xffffffff8094fb90, 0) error 19 > netmap: loaded module > vtvga0: on motherboard > smbios0: at iomem 0xf04d0-0xf04ee on motherboard > smbios0: Version: 2.7, BCD Revision: 2.7 > cryptosoft0: on motherboard > aesni0: on motherboard > acpi0: on motherboard > acpi0: Power Button (fixed) > cpu0: on acpi0 > cpu1: on acpi0 > cpu2: on acpi0 > cpu3: on acpi0 > cpu4: on acpi0 > cpu5: on acpi0 > cpu6: on acpi0 > cpu7: on acpi0 > cpu8: on acpi0 > cpu9: on acpi0 > cpu10: on acpi0 > cpu11: on acpi0 > cpu12: on acpi0 > cpu13: on acpi0 > cpu14: on acpi0 > cpu15: on acpi0 > cpu16: on acpi0 > cpu17: on acpi0 > cpu18: on acpi0 > cpu19: on acpi0 > cpu20: on acpi0 > cpu21: on acpi0 > cpu22: on acpi0 > cpu23: on acpi0 > cpu24: on acpi0 > cpu25: on acpi0 > cpu26: on acpi0 > cpu27: on acpi0 > cpu28: on acpi0 > cpu29: on acpi0 > cpu30: on acpi0 > cpu31: on acpi0 > attimer0: port 0x40-0x43 irq 0 on acpi0 > Timecounter "i8254" frequency 1193182 Hz quality 0 > Event timer "i8254" frequency 1193182 Hz quality 100 > atrtc0: port 0x70-0x71 irq 8 on acpi0 > Event timer "RTC" frequency 32768 Hz quality 0 > hpet0: iomem 0xfed00000-0xfed003ff on acpi0 > Timecounter "HPET" frequency 14318180 Hz quality 950 > Event timer "HPET" frequency 14318180 Hz quality 350 > Event timer "HPET1" frequency 14318180 Hz quality 340 > Event timer "HPET2" frequency 14318180 Hz quality 340 > Event timer "HPET3" frequency 14318180 Hz quality 340 > Event timer "HPET4" frequency 14318180 Hz quality 340 > Event timer "HPET5" frequency 14318180 Hz quality 340 > Event timer "HPET6" frequency 14318180 Hz quality 340 > Event timer "HPET7" frequency 14318180 Hz quality 340 > Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 > pcib0: port 0xcf8-0xcff on acpi0 > pci0: on pcib0 > pcib1: irq 26 at device 1.0 on pci0 > pci1: on pcib1 > pcib2: irq 26 at device 1.1 on pci0 > pci2: on pcib2 > igb0: port 0x8020-0x803f mem 0xdf820000-0xdf83ffff,0xdf844000-0xdf847fff irq 27 at device 0.0 on pci2 > igb0: Using MSIX interrupts with 9 vectors > igb0: Ethernet address: 00:25:90:9b:73:2e > igb0: Bound queue 0 to cpu 0 > igb0: Bound queue 1 to cpu 1 > igb0: Bound queue 2 to cpu 2 > igb0: Bound queue 3 to cpu 3 > igb0: Bound queue 4 to cpu 4 > igb0: Bound queue 5 to cpu 5 > igb0: Bound queue 6 to cpu 6 > igb0: Bound queue 7 to cpu 7 > igb0: netmap queues/slots: TX 8/1024, RX 8/1024 > igb1: port 0x8000-0x801f mem 0xdf800000-0xdf81ffff,0xdf840000-0xdf843fff irq 30 at device 0.1 on pci2 > igb1: Using MSIX interrupts with 9 vectors > igb1: Ethernet address: 00:25:90:9b:73:2f > igb1: Bound queue 0 to cpu 8 > igb1: Bound queue 1 to cpu 9 > igb1: Bound queue 2 to cpu 10 > igb1: Bound queue 3 to cpu 11 > igb1: Bound queue 4 to cpu 12 > igb1: Bound queue 5 to cpu 13 > igb1: Bound queue 6 to cpu 14 > igb1: Bound queue 7 to cpu 15 > igb1: netmap queues/slots: TX 8/1024, RX 8/1024 > pcib3: irq 33 at device 2.0 on pci0 > pci3: on pcib3 > pci3: at device 0.0 (no driver attached) > pci3: at device 0.1 (no driver attached) > pcib4: irq 33 at device 2.2 on pci0 > pci4: on pcib4 > pcib5: irq 41 at device 3.0 on pci0 > pci5: on pcib5 > pcib6: irq 41 at device 3.2 on pci0 > pci6: on pcib6 > ix0: port 0x7020-0x703f mem 0xdf180000-0xdf1fffff,0xdf604000-0xdf607fff irq 42 at device 0.0 on pci6 > ix0: Using MSIX interrupts with 33 vectors > ix0: Ethernet address: 90:e2:ba:77:d4:9c > ix0: PCI Express Bus: Speed 5.0GT/s Width x8 > ix0: netmap queues/slots: TX 32/2048, RX 32/2048 > ix1: port 0x7000-0x701f mem 0xdf100000-0xdf17ffff,0xdf600000-0xdf603fff irq 45 at device 0.1 on pci6 > ix1: Using MSIX interrupts with 33 vectors > ix1: Ethernet address: 90:e2:ba:77:d4:9d > ix1: PCI Express Bus: Speed 5.0GT/s Width x8 > ix1: netmap queues/slots: TX 32/2048, RX 32/2048 > pcib7: irq 16 at device 17.0 on pci0 > pci7: on pcib7 > isci0: port 0x6000-0x60ff mem 0xde07c000-0xde07ffff,0xddc00000-0xddffffff irq 16 at device 0.0 on pci7 > pci0: at device 22.0 (no driver attached) > pci0: at device 22.1 (no driver attached) > ehci0: mem 0xdf923000-0xdf9233ff irq 16 at device 26.0 on pci0 > usbus0: EHCI version 1.0 > usbus0 on ehci0 > pcib8: irq 17 at device 28.0 on pci0 > pci8: on pcib8 > pcib9: irq 19 at device 28.7 on pci0 > pci9: on pcib9 > pcib10: at device 0.0 on pci9 > pci10: on pcib10 > pcib11: at device 0.0 on pci10 > pci11: on pcib11 > pcib12: at device 0.0 on pci11 > pci12: on pcib12 > vgapci0: mem 0xdb000000-0xdbffffff,0xdf000000-0xdf003fff,0xde800000-0xdeffffff irq 19 at device 0.0 on pci12 > vgapci0: Boot video device > pcib13: at device 1.0 on pci10 > pci13: on pcib13 > ehci1: mem 0xdf922000-0xdf9223ff irq 23 at device 29.0 on pci0 > usbus1: EHCI version 1.0 > usbus1 on ehci1 > pcib14: at device 30.0 on pci0 > pci14: on pcib14 > isab0: at device 31.0 on pci0 > isa0: on isab0 > ahci0: port 0x9050-0x9057,0x9040-0x9043,0x9030-0x9037,0x9020-0x9023,0x9000-0x901f mem 0xdf921000-0xdf9217ff irq 18 at device 31.2 on pci0 > ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported > ahcich0: at channel 0 on ahci0 > ahcich1: at channel 1 on ahci0 > ahcich2: at channel 2 on ahci0 > ahcich3: at channel 3 on ahci0 > ahcich4: at channel 4 on ahci0 > ahcich5: at channel 5 on ahci0 > ahciem0: on ahci0 > ichsmb0: port 0x1180-0x119f mem 0xdf920000-0xdf9200ff irq 18 at device 31.3 on pci0 > smbus0: on ichsmb0 > smb0: on smbus0 > pcib15: on acpi0 > pci15: on pcib15 > pcib16: on acpi0 > pci16: on pcib16 > pcib17: irq 57 at device 2.0 on pci16 > pci17: on pcib17 > pcib18: irq 64 at device 3.0 on pci16 > pci18: on pcib18 > pcib19: irq 64 at device 3.2 on pci16 > pci19: on pcib19 > pcib20: on acpi0 > pci20: on pcib20 > acpi_button0: on acpi0 > uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 > uart0: console (115200,n,8,1) > uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 > ipmi0: port 0xca2,0xca3 on acpi0 > ipmi0: KCS mode found at io 0xca2 on acpi > ichwd0 on isa0 > ichwd0: ICH WDT present but disabled in BIOS or hardware > device_attach: ichwd0 attach returned 6 > ichwd0 at port 0x430-0x437,0x460-0x47f on isa0 > ichwd0: ICH WDT present but disabled in BIOS or hardware > device_attach: ichwd0 attach returned 6 > orm0: at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff on isa0 > coretemp0: on cpu0 > est0: on cpu0 > coretemp1: on cpu1 > est1: on cpu1 > coretemp2: on cpu2 > est2: on cpu2 > coretemp3: on cpu3 > est3: on cpu3 > coretemp4: on cpu4 > est4: on cpu4 > coretemp5: on cpu5 > est5: on cpu5 > coretemp6: on cpu6 > est6: on cpu6 > coretemp7: on cpu7 > est7: on cpu7 > coretemp8: on cpu8 > est8: on cpu8 > coretemp9: on cpu9 > est9: on cpu9 > coretemp10: on cpu10 > est10: on cpu10 > coretemp11: on cpu11 > est11: on cpu11 > coretemp12: on cpu12 > est12: on cpu12 > coretemp13: on cpu13 > est13: on cpu13 > coretemp14: on cpu14 > est14: on cpu14 > coretemp15: on cpu15 > est15: on cpu15 > coretemp16: on cpu16 > est16: on cpu16 > coretemp17: on cpu17 > est17: on cpu17 > coretemp18: on cpu18 > est18: on cpu18 > coretemp19: on cpu19 > est19: on cpu19 > coretemp20: on cpu20 > est20: on cpu20 > coretemp21: on cpu21 > est21: on cpu21 > coretemp22: on cpu22 > est22: on cpu22 > coretemp23: on cpu23 > est23: on cpu23 > coretemp24: on cpu24 > est24: on cpu24 > coretemp25: on cpu25 > est25: on cpu25 > coretemp26: on cpu26 > est26: on cpu26 > coretemp27: on cpu27 > est27: on cpu27 > coretemp28: on cpu28 > est28: on cpu28 > coretemp29: on cpu29 > est29: on cpu29 > coretemp30: on cpu30 > est30: on cpu30 > coretemp31: on cpu31 > est31: on cpu31 > fuse-freebsd: version 0.4.4, FUSE ABI 7.8 > Timecounters tick every 1.000 msec > iw_cxgb: Chelsio T3 RDMA Driver loaded > IPsec: Initialized Security Association Processing. > ipfw2 (+ipv6) initialized, divert loadable, nat enabled, default to accept, logging disabled > DUMMYNET 0 with IPv6 initialized (100409) > load_dn_sched dn_sched FIFO loaded > load_dn_sched dn_sched PRIO loaded > load_dn_sched dn_sched QFQ loaded > load_dn_sched dn_sched RR loaded > load_dn_sched dn_sched WF2Q+ loaded > ipmi0: IPMI device rev. 1, firmware rev. 2.35, version 2.0 > ipmi0: Number of channels 3 > ipmi0: Attached watchdog > usbus0: 480Mbps High Speed USB v2.0 > usbus1: 480Mbps High Speed USB v2.0 > ugen0.1: at usbus0 > uhub0: on usbus0 > ugen1.1: at usbus1 > uhub1: on usbus1 > ses0 at ahciem0 bus 0 scbus7 target 0 lun 0 > ses0: SEMB S-E-S 2.00 device > ses0: SEMB SES Device > ada0 at ahcich0 bus 0 scbus1 target 0 lun 0 > ada0: ACS-2 ATA SATA 3.x device > ada0: Serial Number CVCV3102050X180EGN > ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) > ada0: Command Queueing enabled > ada0: 171705MB (351651888 512 byte sectors) > ada0: quirks=0x1<4K> > random: unblocking device. > Sending DHCP Discover packet from interface igb0 (00:25:90:9b:73:2e) > Sending DHCP Discover packet from interface igb1 (00:25:90:9b:73:2f) > Sending DHCP Discover packet from interface ix0 (90:e2:ba:77:d4:9c) > Sending DHCP Discover packet from interface ix1 (90:e2:ba:77:d4:9d) > ix0: link state changed to UP > ix1: link state changed to UP > uhub1: 2 ports with 2 removable, self powered > uhub0: 2 ports with 2 removable, self powered > ugen1.2: at usbus1 > uhub2: on usbus1 > ugen0.2: at usbus0 > uhub3: on usbus0 > uhub3: 6 ports with 6 removable, self powered > uhub2: 8 ports with 8 removable, self powered > ugen0.3: at usbus0 > ukbd0: on usbus0 > ums0: on usbus0 > ums0: 3 buttons and [Z] coordinates ID=0 > igb0: link state changed to UP > Received DHCP Offer packet on igb0 from 192.168.0.2 (accepted) (no root path) (boot_file) > Received DHCP Offer packet on igb0 from 192.168.0.2 (ignored) (no root path) (boot_file) > Received DHCP Offer packet on igb0 from 192.168.0.2 (ignored) (no root path) (boot_file) > Sending DHCP Request packet from interface igb0 (00:25:90:9b:73:2e) > Received DHCP Ack packet on igb0 from 192.168.0.2 (accepted) (got root path) > DHCP timeout for interface igb1 > DHCP timeout for interface ix0 > DHCP timeout for interface ix1 > Wired loader interface (IP 192.168.11.1) is igb0 > igb0 at 192.168.11.1 server 192.168.0.2 boot file /pxe/pxelinux.0 > subnet mask 255.255.0.0 router 192.168.0.2 rootfs 192.168.0.10:/muclab/image/machines/phobos2 hostname phobos2 > Adjusted interface igb0 > Shutdown interface igb1 > Shutdown interface ix0 > ix0: link state changed to DOWN > Shutdown interface ix1 > ix1: link state changed to DOWN > SMP: AP CPU #31 Launched! > SMP: AP CPU #10 Launched! > SMP: AP CPU #25 Launched! > SMP: AP CPU #14 Launched! > SMP: AP CPU #30 Launched! > SMP: AP CPU #12 Launched! > SMP: AP CPU #17 Launched! > SMP: AP CPU #7 Launched! > SMP: AP CPU #28 Launched! > SMP: AP CPU #13 Launched! > SMP: AP CPU #24 Launched! > SMP: AP CPU #6 Launched! > SMP: AP CPU #27 Launched! > SMP: AP CPU #8 Launched! > SMP: AP CPU #29 Launched! > SMP: AP CPU #11 Launched! > SMP: AP CPU #20 Launched! > SMP: AP CPU #15 Launched! > SMP: AP CPU #26 Launched! > SMP: AP CPU #9 Launched! > SMP: AP CPU #22 Launched! > SMP: AP CPU #5 Launched! > SMP: AP CPU #18 Launched! > SMP: AP CPU #4 Launched! > SMP: AP CPU #21 Launched! > SMP: AP CPU #16 Launched! > SMP: AP CPU #3 Launched! > SMP: AP CPU #1 Launched! > SMP: AP CPU #23 Launched! > SMP: AP CPU #2 Launched! > SMP: AP CPU #19 Launched! > Timecounter "TSC" frequency 2000045308 Hz quality 1000 > hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 IAP/4/48/0x3ff IAF/3/48/0x67 > Trying to mount root from nfs: []... > NFS ROOT: 192.168.0.10:/muclab/image/machines/phobos2 > ixl0: mem 0xdc800000-0xdcffffff,0xdd808000-0xdd80ffff irq 32 at device 0.0 on pci3 > ixl0: Using MSIX interrupts with 33 vectors > ixl0: f4.40 a1.4 n04.53 e80001dca > ixl0: Using defaults for TSO: 65518/35/2048 > ixl0: Ethernet address: 68:05:ca:32:15:d0 > ixl0: PCI Express Bus: Speed 8.0GT/s Width x8 > queues is 0xfffffe0008d03000 > ixl0: netmap queues/slots: TX 32/1024, RX 32/1024 > ixl1: mem 0xdc000000-0xdc7fffff,0xdd800000-0xdd807fff irq 32 at device 0.1 on pci3 > ixl1: Using MSIX interrupts with 33 vectors > ixl1: f4.40 a1.4 n04.53 e80001dca > ixl1: Using defaults for TSO: 65518/35/2048 > ixl1: Ethernet address: 68:05:ca:32:15:d1 > ixl1: PCI Express Bus: Speed 8.0GT/s Width x8 > queues is 0xfffffe0009227000 > ixl1: netmap queues/slots: TX 32/1024, RX 32/1024 > ix0: link state changed to UP > ix1: link state changed to UP > ixl0: link state changed to UP > ixl1: link state changed to UP > -- -----------------------------------------+------------------------------- Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione http://www.iet.unipi.it/~luigi/ . Universita` di Pisa TEL +39-050-2217533 . via Diotisalvi 2 Mobile +39-338-6809875 . 56122 PISA (Italy) -----------------------------------------+------------------------------- From owner-freebsd-net@freebsd.org Mon Oct 19 15:05:02 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ED085A19E19 for ; Mon, 19 Oct 2015 15:05:01 +0000 (UTC) (envelope-from lars@netapp.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id CAB05182 for ; Mon, 19 Oct 2015 15:05:01 +0000 (UTC) (envelope-from lars@netapp.com) Received: by mailman.ysv.freebsd.org (Postfix) id C9665A19E18; Mon, 19 Oct 2015 15:05:01 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AF183A19E17 for ; Mon, 19 Oct 2015 15:05:01 +0000 (UTC) (envelope-from lars@netapp.com) Received: from mx144.netapp.com (mx144.netapp.com [216.240.21.25]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "mx144.netapp.com", Issuer "Symantec Class 3 Secure Server CA - G4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 649BA181; Mon, 19 Oct 2015 15:05:00 +0000 (UTC) (envelope-from lars@netapp.com) X-IronPort-AV: E=Sophos;i="5.17,702,1437462000"; d="asc'?scan'208";a="74705270" Received: from hioexcmbx02-prd.hq.netapp.com ([10.122.105.35]) by mx144-out.netapp.com with ESMTP; 19 Oct 2015 08:03:54 -0700 Received: from HIOEXCMBX07-PRD.hq.netapp.com (10.122.105.40) by hioexcmbx02-prd.hq.netapp.com (10.122.105.35) with Microsoft SMTP Server (TLS) id 15.0.1104.5; Mon, 19 Oct 2015 08:03:53 -0700 Received: from HIOEXCMBX07-PRD.hq.netapp.com ([::1]) by hioexcmbx07-prd.hq.netapp.com ([fe80::e1d9:911e:3048:d510%21]) with mapi id 15.00.1104.000; Mon, 19 Oct 2015 08:03:54 -0700 From: "Eggert, Lars" To: Luigi Rizzo CC: "jfv@FreeBSD.org" , "ricera10@gmail.com" , Giuseppe Lettieri , "net@freebsd.org" Subject: Re: ixl 40G bad performance? Thread-Topic: ixl 40G bad performance? Thread-Index: AQHRCnVeClivuZWSFkK4WCM/ynkiH55zUwOAgAAMKAA= Date: Mon, 19 Oct 2015 15:03:53 +0000 Message-ID: References: <79830D9D-94E6-47A9-92B9-D63DF5432272@netapp.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-mailer: Apple Mail (2.3094) x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [10.122.56.79] Content-Type: multipart/signed; boundary="Apple-Mail=_5D33675E-C592-4762-8246-EC3CEF5B4EE2"; protocol="application/pgp-signature"; micalg=pgp-sha256 MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2015 15:05:02 -0000 --Apple-Mail=_5D33675E-C592-4762-8246-EC3CEF5B4EE2 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi, On 2015-10-19, at 16:20, Luigi Rizzo wrote: >=20 > i would look at the following: > - c states and clock speed - make sure you never go below C1, > and fix the clock speed to max. > Sure these parameters also affect the 10G card, but there > may be strange interaction that trigger the power saving > modes in different ways I already have powerd_flags=3D"-a max -b max -n max" in rc.conf, which I = hope should be enough. > - interrupt moderation (may affect ping latency, > do not remember how it is set in ixl but probably a sysctl ixl(4) describes two sysctls that sound like they control AIM, and they = default to off: hw.ixl.dynamic_tx_itr: 0 hw.ixl.dynamic_rx_itr: 0 > - number of queues (32 is a lot i wouldn't use more than 4-8), > may affect cpu-socket affinity With hw.ixl.max_queues=3D4 in loader.conf, performance is still = unchanged. > - tso and flow director - i have seen bad effects of > accelerations so i would run the iperf test with > of these features disabled on both sides, and then enable > them one at a time No change with "ifconfig -tso4 -tso6 -rxcsum -txcsum -lro". How do I turn off flow director? > - queue sizes - the driver seems to use 1024 slots which is > about 1.5 MB queued, which in turn means you have 300us > (and possibly half of that) to drain the queue at 40Gbit/s. > 150-300us may seem an eternity, but if a couple of cores fall > into c7 your budget is gone and the loss will trigger a > retransmission and window halving etc. Also no change with "hw.ixl.ringsz=3D256" in loader.conf. This is really weird. Lars --Apple-Mail=_5D33675E-C592-4762-8246-EC3CEF5B4EE2 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="signature.asc" Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- iQCVAwUBViUGW9ZcnpRveo1xAQjoBAP9F6YSpjWoZXl+C4yVN3eDWuzw9lCHfM7k r87TQ0avA1puKOTE84chmNCuhonnCvFpGWAscgmufZ/PeYkyR5TynFjwpMwjkVXB 4I18y/i7sOZqV98JOUWoVIBY+hhYb9BeitX7s+Ip6zoBgMILg82/mkn45tGb2gtV 6gC8c7VtXz4= =pAzg -----END PGP SIGNATURE----- --Apple-Mail=_5D33675E-C592-4762-8246-EC3CEF5B4EE2-- From owner-freebsd-net@freebsd.org Mon Oct 19 15:11:22 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6CBE3A190D6 for ; Mon, 19 Oct 2015 15:11:22 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 4AAE4A01 for ; Mon, 19 Oct 2015 15:11:22 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 4923CA190D5; Mon, 19 Oct 2015 15:11:22 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 48B9BA190D4 for ; Mon, 19 Oct 2015 15:11:22 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-lf0-x22b.google.com (mail-lf0-x22b.google.com [IPv6:2a00:1450:4010:c07::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B14C59FF; Mon, 19 Oct 2015 15:11:21 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by lffy185 with SMTP id y185so113268797lff.2; Mon, 19 Oct 2015 08:11:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=cmggJVsJ0IpkPuioz/Oqe0ScHX7iFNs/4Fu0L9mcCPM=; b=ifvs0CU5ETUC3rrnRxOfvSgPDvGJJAKyxnPlNJ8kKmGWYRNezzuE/BzKyoe2iCrrAo JpGh2pSGkw2n52dtniril/auqyY8ix+wluN5HTcQBGebtuuEIn8aF0D438mfh1qb6/pL uR6Y5fpT8SURnm2cF2VAkl2cOh5B+gHnXMjLfJ9Z3MEppljD48VfQEqY9UtG0UuYl5vh sNqcs50xY0rD70tFr7VN8TsoIUeiK+IxDuaQapAW3KjBMdsywRBUjfJMK+dWj0UtIHxr fNB4FsepUcJzm/v11H4TJoj4WCE7dHOm2waqAg8wlapMBQN7ui2Ar9/QXekB1cw4QQLQ S1iA== MIME-Version: 1.0 X-Received: by 10.25.24.195 with SMTP id 64mr9864471lfy.71.1445267479620; Mon, 19 Oct 2015 08:11:19 -0700 (PDT) Sender: rizzo.unipi@gmail.com Received: by 10.114.177.4 with HTTP; Mon, 19 Oct 2015 08:11:19 -0700 (PDT) In-Reply-To: References: <79830D9D-94E6-47A9-92B9-D63DF5432272@netapp.com> Date: Mon, 19 Oct 2015 08:11:19 -0700 X-Google-Sender-Auth: hkj5Fxk2sqeMzOek1vpVKf1Eaig Message-ID: Subject: Re: ixl 40G bad performance? From: Luigi Rizzo To: "Eggert, Lars" Cc: "jfv@FreeBSD.org" , "ricera10@gmail.com" , Giuseppe Lettieri , "net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2015 15:11:22 -0000 On Monday, October 19, 2015, Eggert, Lars wrote: > Hi, > > On 2015-10-19, at 16:20, Luigi Rizzo > > wrote: > > > > i would look at the following: > > - c states and clock speed - make sure you never go below C1, > > and fix the clock speed to max. > > Sure these parameters also affect the 10G card, but there > > may be strange interaction that trigger the power saving > > modes in different ways > > I already have powerd_flags="-a max -b max -n max" in rc.conf, which I > hope should be enough. I suspect it might not touch the c states, but better check. The safest is disable them in the bios. > > > - interrupt moderation (may affect ping latency, > > do not remember how it is set in ixl but probably a sysctl > > ixl(4) describes two sysctls that sound like they control AIM, and they > default to off: > > hw.ixl.dynamic_tx_itr: 0 > hw.ixl.dynamic_rx_itr: 0 > > There must be some other control for the actual (fixed, not dynamic) moderation. > > - number of queues (32 is a lot i wouldn't use more than 4-8), > > may affect cpu-socket affinity > > With hw.ixl.max_queues=4 in loader.conf, performance is still unchanged. > > > - tso and flow director - i have seen bad effects of > > accelerations so i would run the iperf test with > > of these features disabled on both sides, and then enable > > them one at a time > > No change with "ifconfig -tso4 -tso6 -rxcsum -txcsum -lro". > > How do I turn off flow director? I am not sure if it is enabled I'm FreeBSD. It is in linux and almost halves the pkt rate with netmap (from 35 down to 19mpps). Maybe it is not too bad for bulk TCP. > > > - queue sizes - the driver seems to use 1024 slots which is > > about 1.5 MB queued, which in turn means you have 300us > > (and possibly half of that) to drain the queue at 40Gbit/s. > > 150-300us may seem an eternity, but if a couple of cores fall > > into c7 your budget is gone and the loss will trigger a > > retransmission and window halving etc. > > Also no change with "hw.ixl.ringsz=256" in loader.conf. Any better success with 2048 slots? 3.5 gbit is what I used to see on the ixgbe with tso disabled, probably hitting a CPU bound. Cheers Luigi > This is really weird. > > Lars > -- -----------------------------------------+------------------------------- Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione http://www.iet.unipi.it/~luigi/ . Universita` di Pisa TEL +39-050-2217533 . via Diotisalvi 2 Mobile +39-338-6809875 . 56122 PISA (Italy) -----------------------------------------+------------------------------- From owner-freebsd-net@freebsd.org Mon Oct 19 15:36:06 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9E057A1965F for ; Mon, 19 Oct 2015 15:36:06 +0000 (UTC) (envelope-from lars@netapp.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 79FB9179F for ; Mon, 19 Oct 2015 15:36:06 +0000 (UTC) (envelope-from lars@netapp.com) Received: by mailman.ysv.freebsd.org (Postfix) id 7695AA1965E; Mon, 19 Oct 2015 15:36:06 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5C5E9A1965D for ; Mon, 19 Oct 2015 15:36:06 +0000 (UTC) (envelope-from lars@netapp.com) Received: from mx141.netapp.com (mx141.netapp.com [216.240.21.12]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "mx141.netapp.com", Issuer "Symantec Class 3 Secure Server CA - G4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2AEE5179E; Mon, 19 Oct 2015 15:36:05 +0000 (UTC) (envelope-from lars@netapp.com) X-IronPort-AV: E=Sophos;i="5.17,702,1437462000"; d="asc'?scan'208";a="75767988" Received: from hioexcmbx03-prd.hq.netapp.com ([10.122.105.36]) by mx141-out.netapp.com with ESMTP; 19 Oct 2015 08:34:51 -0700 Received: from HIOEXCMBX07-PRD.hq.netapp.com (10.122.105.40) by hioexcmbx03-prd.hq.netapp.com (10.122.105.36) with Microsoft SMTP Server (TLS) id 15.0.1104.5; Mon, 19 Oct 2015 08:34:48 -0700 Received: from HIOEXCMBX07-PRD.hq.netapp.com ([::1]) by hioexcmbx07-prd.hq.netapp.com ([fe80::e1d9:911e:3048:d510%21]) with mapi id 15.00.1104.000; Mon, 19 Oct 2015 08:34:48 -0700 From: "Eggert, Lars" To: Luigi Rizzo CC: "jfv@FreeBSD.org" , "ricera10@gmail.com" , Giuseppe Lettieri , "net@freebsd.org" Subject: Re: ixl 40G bad performance? Thread-Topic: ixl 40G bad performance? Thread-Index: AQHRCnVeClivuZWSFkK4WCM/ynkiH55zUwOAgAAMKACAAAITgIAABpIA Date: Mon, 19 Oct 2015 15:34:47 +0000 Message-ID: References: <79830D9D-94E6-47A9-92B9-D63DF5432272@netapp.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-mailer: Apple Mail (2.3094) x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [10.122.56.79] Content-Type: multipart/signed; boundary="Apple-Mail=_3EA9F340-C819-48AB-BA2A-F8B6B26FAA02"; protocol="application/pgp-signature"; micalg=pgp-sha256 MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2015 15:36:06 -0000 --Apple-Mail=_3EA9F340-C819-48AB-BA2A-F8B6B26FAA02 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi, in order to eliminate network or hardware weirdness, I've rerun the test = with Linux 4.3rc6, where I get 13.1 Gbits/sec throughput and 52 usec = flood ping latency. Not great either, but in line with earlier = experiments with Mellanox NICs and an untuned Linux system. On 2015-10-19, at 17:11, Luigi Rizzo wrote: > I suspect it might not touch the c states, but better check. The = safest is > disable them in the bios. I'll try that. >> hw.ixl.dynamic_tx_itr: 0 >> hw.ixl.dynamic_rx_itr: 0 >>=20 >>=20 > There must be some other control for the actual (fixed, not dynamic) > moderation. The only other sysctls in ixl(4) that look relevant are: hw.ixl.rx_itr The RX interrupt rate value, set to 8K by default. hw.ixl.tx_itr The TX interrupt rate value, set to 4K by default. I'll play with those. >> Also no change with "hw.ixl.ringsz=3D256" in loader.conf. >=20 > Any better success with 2048 slots? > 3.5 gbit is what I used to see on the ixgbe with tso disabled, = probably > hitting a CPU bound. Will try. Thanks! Lars --Apple-Mail=_3EA9F340-C819-48AB-BA2A-F8B6B26FAA02 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="signature.asc" Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- iQCVAwUBViUNm9ZcnpRveo1xAQihLQP/TL9IlhVgdX5D4soii576IC/ZSqq4xYw3 h2iT2QjQBNks6US3p2pv8ZT+5SkmtbN0c8GPFd3mmJ/PM2aov/JSb2wp+Xmz+zTD MOqjhGXKs84g5dc1Cb83dlkm2EmrrGxotpIvYp+ioui13BNqxDWFYVvIXyPN/wQP Dx+sdHklhHo= =74hg -----END PGP SIGNATURE----- --Apple-Mail=_3EA9F340-C819-48AB-BA2A-F8B6B26FAA02-- From owner-freebsd-net@freebsd.org Mon Oct 19 15:55:56 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AD027A19C2B for ; Mon, 19 Oct 2015 15:55:56 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 8B702B4 for ; Mon, 19 Oct 2015 15:55:56 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 8A3AEA19C2A; Mon, 19 Oct 2015 15:55:56 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 89D2EA19C29 for ; Mon, 19 Oct 2015 15:55:56 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-lb0-x22d.google.com (mail-lb0-x22d.google.com [IPv6:2a00:1450:4010:c04::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1051FB3; Mon, 19 Oct 2015 15:55:56 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by lbbwb3 with SMTP id wb3so86199714lbb.1; Mon, 19 Oct 2015 08:55:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=F9AMt1eUK1vcQHHjhQaO7coZmUGhztKS/cTtirL5EVA=; b=CKo3RgXTywk0dVOUwwIDphzGVAZ6ZBp5+sJcxmWC/wqiYorwHWiqEzKv1yXqpmcdVg BUOX/UIeJJflyVQ/ef7YJ8PSLwhdO1v62/t8lBuBxyKcvImEBMwlOae66v6hyJwRoueA 5etGro0H+EZ1bk3XKpAcJhlbJpHoaJbSnna6VIsXAmH+aJyIvCqiLEjGuJ0t32VKOShE hdCNsdIT0vndFn5TP+n7isDDqlGbdGr5BvRm9/SQCZG/kjF/lHC3gMa9OzSrxMc0Mco9 wBpUYEba6tUOxJPHJNIgbnyR+OgaBxVSkxxbIeieP3LgH3WBjBn/1rXfY6Kjh8fbsrbC uAWw== MIME-Version: 1.0 X-Received: by 10.112.134.73 with SMTP id pi9mr14993048lbb.83.1445270154005; Mon, 19 Oct 2015 08:55:54 -0700 (PDT) Sender: rizzo.unipi@gmail.com Received: by 10.114.177.4 with HTTP; Mon, 19 Oct 2015 08:55:53 -0700 (PDT) In-Reply-To: References: <79830D9D-94E6-47A9-92B9-D63DF5432272@netapp.com> Date: Mon, 19 Oct 2015 08:55:53 -0700 X-Google-Sender-Auth: Gjof0kR7rV8irksJdTek3unGLww Message-ID: Subject: Re: ixl 40G bad performance? From: Luigi Rizzo To: "Eggert, Lars" Cc: "jfv@FreeBSD.org" , "ricera10@gmail.com" , Giuseppe Lettieri , "net@freebsd.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2015 15:55:56 -0000 On Mon, Oct 19, 2015 at 8:34 AM, Eggert, Lars wrote: > Hi, > > in order to eliminate network or hardware weirdness, I've rerun the test = with Linux 4.3rc6, where I get 13.1 Gbits/sec throughput and 52 usec flood = ping latency. Not great either, but in line with earlier experiments with M= ellanox NICs and an untuned Linux system. > ... >> There must be some other control for the actual (fixed, not dynamic) >> moderation. > > The only other sysctls in ixl(4) that look relevant are: > > hw.ixl.rx_itr > The RX interrupt rate value, set to 8K by default. > > hw.ixl.tx_itr > The TX interrupt rate value, set to 4K by default. > yes those. raise to 20-50k and see what you get in terms of ping latency. Note that 4k on tx means you get to reclaim buffers in the tx queue (unless it is done opportunistically) every 250us which is dangerously close to the 300us capacity of the queue itself. cheers luigi > I'll play with those. > >>> Also no change with "hw.ixl.ringsz=3D256" in loader.conf. >> >> Any better success with 2048 slots? >> 3.5 gbit is what I used to see on the ixgbe with tso disabled, probably >> hitting a CPU bound. > > Will try. > > Thanks! > > Lars --=20 -----------------------------------------+------------------------------- Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione http://www.iet.unipi.it/~luigi/ . Universita` di Pisa TEL +39-050-2217533 . via Diotisalvi 2 Mobile +39-338-6809875 . 56122 PISA (Italy) -----------------------------------------+------------------------------- From owner-freebsd-net@freebsd.org Mon Oct 19 16:47:40 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 24CC9A10E64 for ; Mon, 19 Oct 2015 16:47:40 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 09E0192F for ; Mon, 19 Oct 2015 16:47:40 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: by mailman.ysv.freebsd.org (Postfix) id 088E7A10E63; Mon, 19 Oct 2015 16:47:40 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 08287A10E62 for ; Mon, 19 Oct 2015 16:47:40 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from mail.strugglingcoder.info (strugglingcoder.info [65.19.130.35]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.strugglingcoder.info", Issuer "mail.strugglingcoder.info" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id E7FE092E; Mon, 19 Oct 2015 16:47:39 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from localhost (unknown [10.1.1.3]) (Authenticated sender: hiren@strugglingcoder.info) by mail.strugglingcoder.info (Postfix) with ESMTPA id 2804D10A34B; Mon, 19 Oct 2015 09:47:39 -0700 (PDT) Date: Mon, 19 Oct 2015 09:47:39 -0700 From: hiren panchasara To: Luigi Rizzo Cc: "Eggert, Lars" , "jfv@FreeBSD.org" , "ricera10@gmail.com" , Giuseppe Lettieri , "net@freebsd.org" Subject: Re: ixl 40G bad performance? Message-ID: <20151019164739.GH87252@strugglingcoder.info> References: <79830D9D-94E6-47A9-92B9-D63DF5432272@netapp.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="ggHp9WSrPOeNxb0b" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2015 16:47:40 -0000 --ggHp9WSrPOeNxb0b Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 10/19/15 at 08:11P, Luigi Rizzo wrote: > On Monday, October 19, 2015, Eggert, Lars wrote: >=20 > > > > How do I turn off flow director? >=20 >=20 > I am not sure if it is enabled I'm FreeBSD. It is in linux and almost > halves the pkt rate with netmap (from 35 down to 19mpps). > Maybe it is not too bad for bulk TCP. > Flow director support is incomplete on FreeBSD and that's why it is disabled by default. Cheers, Hiren --ggHp9WSrPOeNxb0b Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAABCgBmBQJWJR6lXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRBNEUyMEZBMUQ4Nzg4RjNGMTdFNjZGMDI4 QjkyNTBFMTU2M0VERkU1AAoJEIuSUOFWPt/lHsEH/0FSPU2SR7j/LdwvLLwoidgH oIl6o8HyInsiaVEGB81t4TTfFv+vz089KfNreEZFXQ7DLTrPN45uUHigD6z9oyzd vZzY8yjfaLdmC9EN9z1ZbeenoLoPviX2bt8T8AdnNH0+NoChtHKohyL0xJlZN6I/ J4X2fEEBUAUJyZKVH5/TsLqecfw8rFtj9W/jt6bcd7oNzuh8ATLlp0mf57is54qX MXJ8UhGBE9QsmDcxVOz5IN4hOTLAYB7ahw44kqJvBvAQM7VFQhg5EIy2aCLmh6pI sH/xbW9rs+UdtacEY4+rmjRmh5JHkijSiK9ntJbFccYTCz9kk0k3RJ2u6z5bEHc= =5oRL -----END PGP SIGNATURE----- --ggHp9WSrPOeNxb0b-- From owner-freebsd-net@freebsd.org Mon Oct 19 21:30:36 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C1F11A19AB1 for ; Mon, 19 Oct 2015 21:30:36 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9894A194B; Mon, 19 Oct 2015 21:30:36 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from ralph.baldwin.cx (c-73-231-226-104.hsd1.ca.comcast.net [73.231.226.104]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 5FE41B923; Mon, 19 Oct 2015 17:30:35 -0400 (EDT) From: John Baldwin To: Maxim Sobolev Cc: FreeBSD Net Subject: Re: Some MSI are not routed correctly Date: Mon, 19 Oct 2015 14:03:35 -0700 Message-ID: <1608354.LQmTMSsd5C@ralph.baldwin.cx> User-Agent: KMail/4.14.3 (FreeBSD/10.2-PRERELEASE; KDE/4.14.3; amd64; ; ) In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Mon, 19 Oct 2015 17:30:35 -0400 (EDT) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2015 21:30:37 -0000 On Thursday, October 08, 2015 07:33:27 AM Maxim Sobolev wrote: > Hi John & others, > > We've came across a weird MSI routing issue on one of our newest dual > E5-2690v3 (haswell) Supermicro X10DRL-i boxes running latest 10.2-p4. It is > fitted with dual port Intel I350 card, in addition to the built-in I210 > chip that is not used. The hw.igb.num_queues is set to 4, and the driver > reports binding to the CPUs 0-3 for the first port and CPUs 4-7 for the > second, however when verified with top -P under the load, interrupts are > only delivered to the CPUs 0-3, no interrupt time is recorded on the CPUs > 4-7. systat -vm shows that all 8 queues are firing interrupts, so my guess > that for whatever reason bus_bind_intr() is not doing what's expected to do > for half of those interrupts. > > What's interesting is that on a similar box (same chassis/mobo/cpu) but > equipped with the quad-port X540-AT2 10Gig card, interrupts are routed > properly. The latter is running with hw.ix.num_queues="3". > > pcib2: port 0xcf8-0xcff on acpi0 > pci0: on pcib2 > pcib3: irq 26 at device 1.0 on pci0 > pci1: on pcib3 > igb0: mem > 0xc7200000-0xc72fffff,0xc7304000-0xc7307fff irq 26 at device 0.0 on pci1 > igb0: Using MSIX interrupts with 5 vectors > igb0: Ethernet address: a0:36:9f:76:af:20 > igb0: Bound queue 0 to cpu0 > igb0: Bound queue 1 to cpu1 > igb0: Bound queue 2 to cpu2 > igb0: Bound queue 3 to cpu3 > igb0: netmap queues/slots: TX 4/4096, RX 4/4096 > igb1: mem > 0xc7100000-0xc71fffff,0xc7300000-0xc7303fff irq 28 at device 0.1 on pci1 > igb1: Using MSIX interrupts with 5 vectors > igb1: Ethernet address: a0:36:9f:76:af:21 > igb1: Bound queue 0 to cpu4 > igb1: Bound queue 1 to cpu5 > igb1: Bound queue 2 to cpu6 > igb1: Bound queue 3 to cpu7 > igb1: netmap queues/slots: TX 4/4096, RX 4/4096 > > pcib2: port 0xcf8-0xcff on acpi0 > pci0: on pcib2 > pcib3: irq 26 at device 1.0 on pci0 > pci1: on pcib3 > pcib4: irq 32 at device 2.0 on pci0 > pci2: on pcib4 > pcib5: irq 40 at device 3.0 on pci0 > pci3: on pcib5 > ix0: port > 0x6020-0x603f mem 0xc7c00000-0xc7dfffff,0xc7e04000-0xc7e07fff irq 40 at > device 0.0 on pci3 > ix0: Using MSIX interrupts with 4 vectors > ix0: Bound queue 0 to cpu 0 > ix0: Bound queue 1 to cpu 1 > ix0: Bound queue 2 to cpu 2 > ix0: Ethernet address: 0c:c4:7a:5e:be:64 > ix0: PCI Express Bus: Speed 5.0GT/s Width x8 > ix0: netmap queues/slots: TX 3/4096, RX 3/4096 > ix1: port > 0x6000-0x601f mem 0xc7a00000-0xc7bfffff,0xc7e00000-0xc7e03fff irq 44 at > device 0.1 on pci3 > ix1: Using MSIX interrupts with 4 vectors > ix1: Bound queue 0 to cpu 3 > ix1: Bound queue 1 to cpu 4 > ix1: Bound queue 2 to cpu 5 > ix1: Ethernet address: 0c:c4:7a:5e:be:65 > ix1: PCI Express Bus: Speed 5.0GT/s Width x8 > ix1: netmap queues/slots: TX 3/4096, RX 3/4096 > > Some extra debug is here: > > http://sobomax.sippysoft.com/haswell_bug/bad.dmesg > http://sobomax.sippysoft.com/haswell_bug/lstopo_bad.png > http://sobomax.sippysoft.com/haswell_bug/systat_vm_bad.png > http://sobomax.sippysoft.com/haswell_bug/top_P_bad.png > > http://sobomax.sippysoft.com/haswell_bug/good.dmesg > http://sobomax.sippysoft.com/haswell_bug/lstopo_good.png > http://sobomax.sippysoft.com/haswell_bug/systat_vm_good.png > http://sobomax.sippysoft.com/haswell_bug/top_P_good.png > > Any ideas on how to debug that further are welcome. The box in the > production, but we can remove traffic during off-peak to run some > test/debug code on. Can you get procstat -S output for the interrupt threads? (Usually interrupt threads are in pid 12, so 'procstat -S 12' would suffice.) -- John Baldwin From owner-freebsd-net@freebsd.org Mon Oct 19 22:50:42 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 41319A183F0 for ; Mon, 19 Oct 2015 22:50:42 +0000 (UTC) (envelope-from outbackdingo@gmail.com) Received: from mail-oi0-x232.google.com (mail-oi0-x232.google.com [IPv6:2607:f8b0:4003:c06::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 09551D4D for ; Mon, 19 Oct 2015 22:50:42 +0000 (UTC) (envelope-from outbackdingo@gmail.com) Received: by oiao187 with SMTP id o187so50601958oia.3 for ; Mon, 19 Oct 2015 15:50:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=1sfc1JfWMRTEjb5NyuP3+TiNmzL3Mar+7nFwjNUgxPA=; b=U5lLVb300nj8wcfEsqOj1sSaRNliC2pPUGhJ8EtdC8WZR53q1AJSgyXpPy6FWh5/Hb 64IfWrjrCzWJkl/TfPn/Gf89atGmYrT6xfeJJVyQjUuGhtoEOdnJfk+/BEwDNeTpZ5Kk qM3G53B1/mBZlA+0OlgSuOxs9rDFCbitp3U+GMkP7l+Qb2Hci7FkvvVuxQ4QAOMyrEW1 jhlbFgtGFGhsUnyUtHyl8zXj1V3YLk+jsO7QdsY1rPKZm4Ek/eLxajM5IlO9WE3uyHQq LSS2I9qhrfklPwEAnaFieEZYz390VRaV8ZdWXiPqHSehj9vZ95udzUqZex59P+KmWc5j 5Njw== X-Received: by 10.202.87.76 with SMTP id l73mr18084561oib.33.1445295041345; Mon, 19 Oct 2015 15:50:41 -0700 (PDT) MIME-Version: 1.0 Received: by 10.202.107.84 with HTTP; Mon, 19 Oct 2015 15:50:01 -0700 (PDT) In-Reply-To: <562466AA.7020707@swin.edu.au> References: <562466AA.7020707@swin.edu.au> From: Outback Dingo Date: Tue, 20 Oct 2015 08:50:01 +1000 Message-ID: Subject: Re: MPTCP for FreeBSD repository on BitBucket/v0.51 update To: Nigel Williams Cc: freebsd-net@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2015 22:50:42 -0000 Nigel... seriously... /*-^M * Copyright (c) 2012-2015^M * Swinburne University of Technology, Melbourne, Australia.^M * All rights reserved.^M *^M * This software was developed at the Centre for Advanced Internet^M * Architectures, Swinburne University of Technology, by Nigel Williams and^M * Lawrence Stewart, made possible in part by a gift from the FreeBSD^M * Foundation and The Cisco University Research Program Fund, a corporate^M * advised fund of Silicon Valley Community Foundation.^M *^M * Redistribution and use in source and binary forms, with or without^M * modification, are permitted provided that the following conditions^M * are met:^M * 1. Redistributions of source code must retain the above copyright^M * notice, this list of conditions and the following disclaimer.^M * 2. Redistributions in binary form must reproduce the above copyright^M * notice, this list of conditions and the following disclaimer in the^M * documentation and/or other materials provided with the distribution.^M *^M * THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND^M * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE^M * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE^M * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE^M * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL^M * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS^M * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)^M * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT^M * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY^M * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF^M * SUCH DAMAGE.^M */^M ^M /*^M * mptcp.h^M *^M * Created on: 15/05/2012^M * Author: nwilliams^M */^M ^M #ifndef MPTCP_H_^M #define MPTCP_H_^M ^M ^M #include ^M ^M #define MPTCP_64BIT_KEY 8^M ^M typedef u_int64_t mptcp_seq;^M ^M /* MPTCP subtypes */^M #define MPTCP_SUBTYPE_MP_CAPABLE 0^M #define MPTCP_SUBLEN_MP_CAPABLE_SYN 12^M #define MPTCP_SUBLEN_MP_CAPABLE_ACK 20^M ^M #define MPTCP_SUBTYPE_MP_JOIN 1^M #define MPTCP_SUBLEN_MP_JOIN_SYN 12^M #define MPTCP_SUBLEN_MP_JOIN_SYNACK 16 // should be 16, but run out of option space^M #define MPTCP_SUBLEN_MP_JOIN_ACK 24 // should be 24, but run out of option space^M ^M #define MPTCP_SUBTYPE_DSS 2^M #define MPTCP_SUBLEN_DSS_DATA_ACK XX^M #define MPTCP_SUBLEN_DSS_DATA_DSN XX^M ^M #define MPTCP_SUBTYPE_ADD_ADDR 3^M #define MPTCP_SUBLEN_ADD_ADDRV4 8^M #define MPTCP_SUBLEN_ADD_ADDRV6 20^M ^M #define MPTCP_SUBTYPE_REMOVE_ADDR 4^M #define MPTCP_SUBLEN_REMOVE_ADDR 4^M ^M #define MPTCP_SUBTYPE_MP_PRIO 5^M ^M #define MPTCP_SUBTYPE_MP_FAIL 6^M #define MPTCP_SUBTYPELEN_MP_FAIL 12^M ^M #define MPTCP_SUBTYPE_MP_FASTCLOSE 7^M #define MPTCP_SUBTYPELEN_MP_FASTCLOSE 12^M ^M #define MAX_MP_OPLEN 28^M ^M /* mptcp errors */^M ^M #define EMAXSUBFLOWSREACHED 01^M #define ENOMPCB 02^M #define ENOTCPCB 03^M ^M /* mptcp funcs */^M ^M ^M #define MPTCP_SA_NAME_MAX 16 /* max scheduler discipline name length */^M ^M #endif /* MPTCP_H_ */^M On Mon, Oct 19, 2015 at 1:42 PM, Nigel Williams wrote: > Hi, > > The MPTCP code is now available as a mercurial repository: > - Repository: https://bitbucket.org/nw-swin/caia-mptcp-freebsd > - Wiki: https://bitbucket.org/nw-swin/caia-mptcp-freebsd/wiki/ > > For those interested in trying the implementation/looking at the code, > this should hopefully make the process a little easier (and save having to > patch in updates). It should also make it possible to contribute code for > those wishing to do so. > > Some details: > - Has been branched off 'freebsd-head' at 'http://hg-beta.freebsd.org/base', > and will be merged on a weekly basis. > - I will be working off this repository so it will be up-to-date with > recent changes. > - In place of patch releases, release versions will now be tagged. > - I'll also start to populate the 'Issues' section so that there is a > better picture of current bugs/things TBD. > > The version has also been updated to v0.51. See: > - http://caia.swin.edu.au/newtcp/mptcp/tools.html > - OR https://bitbucket.org/nw-swin/caia-mptcp-freebsd/wiki/Home > > Functionally-wise this hasn't changed from the previous version, but has > been merged with a recent revision of head. > > cheers, > nigel > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@freebsd.org Mon Oct 19 23:41:29 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A5FBBA194C5 for ; Mon, 19 Oct 2015 23:41:29 +0000 (UTC) (envelope-from njwilliams@swin.edu.au) Received: from gpo1.cc.swin.edu.au (gpo1.cc.swin.edu.au [136.186.1.30]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3CEABA9B for ; Mon, 19 Oct 2015 23:41:27 +0000 (UTC) (envelope-from njwilliams@swin.edu.au) Received: from [136.186.242.188] (vpn242-188.cc.swin.edu.au [136.186.242.188]) by gpo1.cc.swin.edu.au (8.14.3/8.14.3) with ESMTP id t9JNfHmb014439; Tue, 20 Oct 2015 10:41:17 +1100 Subject: Re: MPTCP for FreeBSD repository on BitBucket/v0.51 update To: Outback Dingo References: <562466AA.7020707@swin.edu.au> Cc: freebsd-net@freebsd.org From: Nigel Williams Message-ID: <56257F9C.1080201@swin.edu.au> Date: Tue, 20 Oct 2015 10:41:16 +1100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2015 23:41:29 -0000 Hi, Fixed. It should just have been that file. cheers, nigel On 20/10/15 09:50, Outback Dingo wrote: > Nigel... > > seriously... > > /*-^M > * Copyright (c) 2012-2015^M > * Swinburne University of Technology, Melbourne, Australia.^M > * All rights reserved.^M > *^M > * This software was developed at the Centre for Advanced Internet^M > * Architectures, Swinburne University of Technology, by Nigel Williams > and^M > * Lawrence Stewart, made possible in part by a gift from the FreeBSD^M > * Foundation and The Cisco University Research Program Fund, a corporate^M > * advised fund of Silicon Valley Community Foundation.^M > *^M > * Redistribution and use in source and binary forms, with or without^M > * modification, are permitted provided that the following conditions^M > * are met:^M > * 1. Redistributions of source code must retain the above copyright^M > * notice, this list of conditions and the following disclaimer.^M > * 2. Redistributions in binary form must reproduce the above copyright^M > * notice, this list of conditions and the following disclaimer in the^M > * documentation and/or other materials provided with the > distribution.^M > *^M > * THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' > AND^M > * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE^M > * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR > PURPOSE^M > * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE > LIABLE^M > * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR > CONSEQUENTIAL^M > * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE > GOODS^M > * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)^M > * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, > STRICT^M > * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN > ANY WAY^M > * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF^M > * SUCH DAMAGE.^M > */^M > ^M > /*^M > * mptcp.h^M > *^M > * Created on: 15/05/2012^M > * Author: nwilliams^M > */^M > ^M > #ifndef MPTCP_H_^M > #define MPTCP_H_^M > ^M > ^M > #include ^M > ^M > #define MPTCP_64BIT_KEY 8^M > ^M > typedef u_int64_t mptcp_seq;^M > ^M > /* MPTCP subtypes */^M > #define MPTCP_SUBTYPE_MP_CAPABLE 0^M > #define MPTCP_SUBLEN_MP_CAPABLE_SYN 12^M > #define MPTCP_SUBLEN_MP_CAPABLE_ACK 20^M > ^M > #define MPTCP_SUBTYPE_MP_JOIN 1^M > #define MPTCP_SUBLEN_MP_JOIN_SYN 12^M > #define MPTCP_SUBLEN_MP_JOIN_SYNACK 16 // > should be 16, but run out of option space^M > #define MPTCP_SUBLEN_MP_JOIN_ACK 24 // > should be 24, but run out of option space^M > ^M > #define MPTCP_SUBTYPE_DSS 2^M > #define MPTCP_SUBLEN_DSS_DATA_ACK XX^M > #define MPTCP_SUBLEN_DSS_DATA_DSN XX^M > ^M > #define MPTCP_SUBTYPE_ADD_ADDR 3^M > #define MPTCP_SUBLEN_ADD_ADDRV4 8^M > #define MPTCP_SUBLEN_ADD_ADDRV6 20^M > ^M > #define MPTCP_SUBTYPE_REMOVE_ADDR 4^M > #define MPTCP_SUBLEN_REMOVE_ADDR 4^M > ^M > #define MPTCP_SUBTYPE_MP_PRIO 5^M > ^M > #define MPTCP_SUBTYPE_MP_FAIL 6^M > #define MPTCP_SUBTYPELEN_MP_FAIL 12^M > ^M > #define MPTCP_SUBTYPE_MP_FASTCLOSE 7^M > #define MPTCP_SUBTYPELEN_MP_FASTCLOSE 12^M > ^M > #define MAX_MP_OPLEN 28^M > ^M > /* mptcp errors */^M > ^M > #define EMAXSUBFLOWSREACHED 01^M > #define ENOMPCB 02^M > #define ENOTCPCB 03^M > ^M > /* mptcp funcs */^M > ^M > ^M > #define MPTCP_SA_NAME_MAX 16 /* max scheduler discipline name > length */^M > ^M > #endif /* MPTCP_H_ */^M > > > On Mon, Oct 19, 2015 at 1:42 PM, Nigel Williams > wrote: > > Hi, > > The MPTCP code is now available as a mercurial repository: > - Repository: https://bitbucket.org/nw-swin/caia-mptcp-freebsd > - Wiki: https://bitbucket.org/nw-swin/caia-mptcp-freebsd/wiki/ > > For those interested in trying the implementation/looking at the > code, this should hopefully make the process a little easier (and > save having to patch in updates). It should also make it possible to > contribute code for those wishing to do so. > > Some details: > - Has been branched off 'freebsd-head' at > 'http://hg-beta.freebsd.org/base', and will be merged on a weekly basis. > - I will be working off this repository so it will be up-to-date > with recent changes. > - In place of patch releases, release versions will now be tagged. > - I'll also start to populate the 'Issues' section so that there is > a better picture of current bugs/things TBD. > > The version has also been updated to v0.51. See: > - http://caia.swin.edu.au/newtcp/mptcp/tools.html > - OR https://bitbucket.org/nw-swin/caia-mptcp-freebsd/wiki/Home > > Functionally-wise this hasn't changed from the previous version, but > has been merged with a recent revision of head. > > cheers, > nigel > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org > " > > From owner-freebsd-net@freebsd.org Mon Oct 19 23:43:17 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2D09CA1963B for ; Mon, 19 Oct 2015 23:43:17 +0000 (UTC) (envelope-from jhellenthal@dataix.net) Received: from mail-io0-x234.google.com (mail-io0-x234.google.com [IPv6:2607:f8b0:4001:c06::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EDD1CC8C for ; Mon, 19 Oct 2015 23:43:16 +0000 (UTC) (envelope-from jhellenthal@dataix.net) Received: by ioll68 with SMTP id l68so4184305iol.3 for ; Mon, 19 Oct 2015 16:43:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dataix.net; s=rsa; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=VSijDBsLnMTvgeBCwm8atLuAhqgLOYFoN9JgIwtne4U=; b=WR1beivIiJxCgCJc8d16/vQfkBru3hRWSKjPDFpIh/UgDAcKxS+NgTd8GvzXjc09J3 1spPV3o4YeaxxpPpiefQfRkwHXIYasY8/NGr7mbvTSU8tUtkvKz23oqsb2rBlWN7ENqD e7CuThMMIrdQB7CJf9YvsKtrEWF4SIFs11eHw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:content-type:mime-version:subject:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to; bh=VSijDBsLnMTvgeBCwm8atLuAhqgLOYFoN9JgIwtne4U=; b=U93AYuGtBF/VF02qEQkQkfKKNiG5aPfNhv/YovyrXiDpRNb8F1dN+okkIJSRX2MVx6 sl0oDYoTZ60gTTzwsZleZUruziM5tHco1s5/eay0YC2vo00fk/lfNla5VQrWWoeBrGZp LcsxNQccR7/ThO3QhI4d3z4Jm/3ocf4b4ozAUHmRZwj6azZ1PC4mJ57X2qaKq8Ve94FM H7aqpRncHjWcafT1jmeJOy/IsVjVALwpy2XJACLsY9B+LZO1QDP/gC+x7UrbDf05orpP xKOe3H65x0+rZe3/VtHhW9GEO5D+rG4xch14ym9eLr+5UnF1yOO+Rp14mce/wj4GSK3p TGCA== X-Gm-Message-State: ALoCoQnQNKOKacqe3DLRqnHIbyWtbGoeP7hv7S1NA0ohlFaixft6Jpk/hL/BhDNYp1Pt2mlgDqBr X-Received: by 10.107.9.91 with SMTP id j88mr558799ioi.191.1445298196026; Mon, 19 Oct 2015 16:43:16 -0700 (PDT) Received: from [172.31.32.31] (cpe-65-26-238-24.wi.res.rr.com. [65.26.238.24]) by smtp.gmail.com with ESMTPSA id m137sm297256iom.10.2015.10.19.16.43.15 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 19 Oct 2015 16:43:15 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (1.0) Subject: Re: MPTCP for FreeBSD repository on BitBucket/v0.51 update From: Jason Hellenthal X-Mailer: iPhone Mail (13A452) In-Reply-To: Date: Mon, 19 Oct 2015 18:43:15 -0500 Cc: Nigel Williams , freebsd-net@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <41F00B1A-25BB-4D14-8752-0B620259FC85@dataix.net> References: <562466AA.7020707@swin.edu.au> To: Outback Dingo X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2015 23:43:17 -0000 Hahaha was this written in notepad or did someone forget to turn off dos sty= le line encodings. --=20 Jason Hellenthal JJH48-ARIN On Oct 19, 2015, at 17:50, Outback Dingo wrote: Nigel... seriously... /*-^M * Copyright (c) 2012-2015^M * Swinburne University of Technology, Melbourne, Australia.^M * All rights reserved.^M *^M * This software was developed at the Centre for Advanced Internet^M * Architectures, Swinburne University of Technology, by Nigel Williams and^M * Lawrence Stewart, made possible in part by a gift from the FreeBSD^M * Foundation and The Cisco University Research Program Fund, a corporate^M * advised fund of Silicon Valley Community Foundation.^M *^M * Redistribution and use in source and binary forms, with or without^M * modification, are permitted provided that the following conditions^M * are met:^M * 1. Redistributions of source code must retain the above copyright^M * notice, this list of conditions and the following disclaimer.^M * 2. Redistributions in binary form must reproduce the above copyright^M * notice, this list of conditions and the following disclaimer in the^M * documentation and/or other materials provided with the distribution.^M *^M * THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND^M * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE^M * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE^M * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE^M * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL^M * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS^M * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)^M * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT^M * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY^M * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF^M * SUCH DAMAGE.^M */^M ^M /*^M * mptcp.h^M *^M * Created on: 15/05/2012^M * Author: nwilliams^M */^M ^M #ifndef MPTCP_H_^M #define MPTCP_H_^M ^M ^M #include ^M ^M #define MPTCP_64BIT_KEY 8^M ^M typedef u_int64_t mptcp_seq;^M ^M /* MPTCP subtypes */^M #define MPTCP_SUBTYPE_MP_CAPABLE 0^M #define MPTCP_SUBLEN_MP_CAPABLE_SYN 12^M #define MPTCP_SUBLEN_MP_CAPABLE_ACK 20^M ^M #define MPTCP_SUBTYPE_MP_JOIN 1^M #define MPTCP_SUBLEN_MP_JOIN_SYN 12^M #define MPTCP_SUBLEN_MP_JOIN_SYNACK 16 // should be 16, but run out of option space^M #define MPTCP_SUBLEN_MP_JOIN_ACK 24 // should be 24, but run out of option space^M ^M #define MPTCP_SUBTYPE_DSS 2^M #define MPTCP_SUBLEN_DSS_DATA_ACK XX^M #define MPTCP_SUBLEN_DSS_DATA_DSN XX^M ^M #define MPTCP_SUBTYPE_ADD_ADDR 3^M #define MPTCP_SUBLEN_ADD_ADDRV4 8^M #define MPTCP_SUBLEN_ADD_ADDRV6 20^M ^M #define MPTCP_SUBTYPE_REMOVE_ADDR 4^M #define MPTCP_SUBLEN_REMOVE_ADDR 4^M ^M #define MPTCP_SUBTYPE_MP_PRIO 5^M ^M #define MPTCP_SUBTYPE_MP_FAIL 6^M #define MPTCP_SUBTYPELEN_MP_FAIL 12^M ^M #define MPTCP_SUBTYPE_MP_FASTCLOSE 7^M #define MPTCP_SUBTYPELEN_MP_FASTCLOSE 12^M ^M #define MAX_MP_OPLEN 28^M ^M /* mptcp errors */^M ^M #define EMAXSUBFLOWSREACHED 01^M #define ENOMPCB 02^M #define ENOTCPCB 03^M ^M /* mptcp funcs */^M ^M ^M #define MPTCP_SA_NAME_MAX 16 /* max scheduler discipline name length */^M ^M #endif /* MPTCP_H_ */^M On Mon, Oct 19, 2015 at 1:42 PM, Nigel Williams wrote: > Hi, >=20 > The MPTCP code is now available as a mercurial repository: > - Repository: https://bitbucket.org/nw-swin/caia-mptcp-freebsd > - Wiki: https://bitbucket.org/nw-swin/caia-mptcp-freebsd/wiki/ >=20 > For those interested in trying the implementation/looking at the code, > this should hopefully make the process a little easier (and save having to= > patch in updates). It should also make it possible to contribute code for > those wishing to do so. >=20 > Some details: > - Has been branched off 'freebsd-head' at 'http://hg-beta.freebsd.org/base= ', > and will be merged on a weekly basis. > - I will be working off this repository so it will be up-to-date with > recent changes. > - In place of patch releases, release versions will now be tagged. > - I'll also start to populate the 'Issues' section so that there is a > better picture of current bugs/things TBD. >=20 > The version has also been updated to v0.51. See: > - http://caia.swin.edu.au/newtcp/mptcp/tools.html > - OR https://bitbucket.org/nw-swin/caia-mptcp-freebsd/wiki/Home >=20 > Functionally-wise this hasn't changed from the previous version, but has > been merged with a recent revision of head. >=20 > cheers, > nigel > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" _______________________________________________ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@freebsd.org Tue Oct 20 01:29:39 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CA49CA1936F for ; Tue, 20 Oct 2015 01:29:39 +0000 (UTC) (envelope-from lstewart@freebsd.org) Received: from lauren.room52.net (lauren.room52.net [210.50.193.198]) by mx1.freebsd.org (Postfix) with ESMTP id 92F4318C2 for ; Tue, 20 Oct 2015 01:29:39 +0000 (UTC) (envelope-from lstewart@freebsd.org) Received: from lgwl-lstewart2.corp.netflix.com (lstewart-laptop.caia.swin.edu.au [136.186.229.148]) by lauren.room52.net (Postfix) with ESMTPSA id 766E87E839; Tue, 20 Oct 2015 12:22:50 +1100 (EST) Subject: Re: MPTCP for FreeBSD repository on BitBucket/v0.51 update To: Jason Hellenthal , Outback Dingo References: <562466AA.7020707@swin.edu.au> <41F00B1A-25BB-4D14-8752-0B620259FC85@dataix.net> Cc: freebsd-net@freebsd.org From: Lawrence Stewart X-Enigmail-Draft-Status: N1110 Message-ID: <56259724.8030107@freebsd.org> Date: Tue, 20 Oct 2015 12:21:40 +1100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <41F00B1A-25BB-4D14-8752-0B620259FC85@dataix.net> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=2.4 required=5.0 tests=DNS_FROM_AHBL_RHSBL, UNPARSEABLE_RELAY autolearn=no version=3.3.2 X-Spam-Level: ** X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on lauren.room52.net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Oct 2015 01:29:39 -0000 Outback Dingo and Jason, On 20/10/2015 10:43, Jason Hellenthal wrote: > Hahaha was this written in notepad or did someone forget to turn off dos style line encodings. > >> On 20/10/2015 09:50, Outback Dingo wrote: >> Nigel... >> >> seriously... >> Publicly ridiculing someone for such a minor issue when a friendly private email or even better a pull request on Bitbucket would have been appropriate makes you both look pathetic. Shut up and code, or failing that shut up and let other people code, and please keep disparaging comments off the list in future. Lawrence From owner-freebsd-net@freebsd.org Tue Oct 20 03:14:23 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6E8AFA1AE75 for ; Tue, 20 Oct 2015 03:14:23 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from mail.strugglingcoder.info (strugglingcoder.info [65.19.130.35]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.strugglingcoder.info", Issuer "mail.strugglingcoder.info" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 5BEF51337; Tue, 20 Oct 2015 03:14:22 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from localhost (unknown [10.1.1.3]) (Authenticated sender: hiren@strugglingcoder.info) by mail.strugglingcoder.info (Postfix) with ESMTPA id 7C39D10A28D; Mon, 19 Oct 2015 20:14:21 -0700 (PDT) Date: Mon, 19 Oct 2015 20:14:21 -0700 From: hiren panchasara To: Lawrence Stewart Cc: Jason Hellenthal , Outback Dingo , freebsd-net@freebsd.org Subject: Re: MPTCP for FreeBSD repository on BitBucket/v0.51 update Message-ID: <20151020031421.GK87252@strugglingcoder.info> References: <562466AA.7020707@swin.edu.au> <41F00B1A-25BB-4D14-8752-0B620259FC85@dataix.net> <56259724.8030107@freebsd.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="2fSbKhQ/kwrfWINy" Content-Disposition: inline In-Reply-To: <56259724.8030107@freebsd.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Oct 2015 03:14:23 -0000 --2fSbKhQ/kwrfWINy Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 10/20/15 at 12:21P, Lawrence Stewart wrote: > Outback Dingo and Jason, >=20 > On 20/10/2015 10:43, Jason Hellenthal wrote: > > Hahaha was this written in notepad or did someone forget to turn off do= s style line encodings. > > > >> On 20/10/2015 09:50, Outback Dingo wrote: > >> Nigel... > >>=20 > >> seriously... > >>=20 >=20 > Publicly ridiculing someone for such a minor issue when a friendly > private email or even better a pull request on Bitbucket would have been > appropriate makes you both look pathetic. Shut up and code, or failing > that shut up and let other people code, and please keep disparaging > comments off the list in future. So true. How could you guys pick this "nit" over the huge overhaul Nigel is doing in tcp stack to bring this useful feature into FreeBSD? Please be constructive in your responses. We want to encourage and support people who work on FreeBSD and not make fun of them. Cheers, Hiren --2fSbKhQ/kwrfWINy Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAABCgBmBQJWJbGKXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRBNEUyMEZBMUQ4Nzg4RjNGMTdFNjZGMDI4 QjkyNTBFMTU2M0VERkU1AAoJEIuSUOFWPt/lkO4H/joUMLrTeM2SwJ9zi25OxHWk nZ2CUtkljW3kddybDiT2xTjBR++u3SiZej46PNCA12KGxVLErrL2nI/vUdfbG/1k mASOXybfSbVUgTd1L37K4kZblBXuoSGaYeHxDww6d3i0UZPGi1hOegprejeaqy4R jHnKIdutSzp8+4fsefuaWxm9U0c/bzsGELaL0bEnWt9LWsbuxeBK0dI7PkSYY/vM c/hnWLD6k+KatPa93ijilptP3QukNFEKTQhRubu3PcJ+pOcwOzA/nwfXUsQIUXXB EpILBhu48Y7nkIZPXqx4qijIrFSwidcu2rIbWxDxW0Mf6mVxk3d3foeQIh31oiU= =ROAv -----END PGP SIGNATURE----- --2fSbKhQ/kwrfWINy-- From owner-freebsd-net@freebsd.org Tue Oct 20 04:47:37 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ADF1AA19971 for ; Tue, 20 Oct 2015 04:47:37 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 8AAD6DBF for ; Tue, 20 Oct 2015 04:47:37 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 88552A19970; Tue, 20 Oct 2015 04:47:37 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6D0ECA1996B for ; Tue, 20 Oct 2015 04:47:37 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mail-ob0-x233.google.com (mail-ob0-x233.google.com [IPv6:2607:f8b0:4003:c01::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2D40BDBD; Tue, 20 Oct 2015 04:47:37 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: by obbwb3 with SMTP id wb3so5073370obb.0; Mon, 19 Oct 2015 21:47:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=DHjkwGXOWXe8ywU+0Yug+WjCCirn7TM+TDU4lEYDYOM=; b=pHr+cLuphL+Y9eY3eaaRk5cxEHAjdVJJ0RUB6z28QgmPfeQOUcgUu8a68Xs00ME2RE 5kD7FiPfHZnOTDwOzhdCdNYahclx/tTSqgaImZETCHfv8DwAcGv7HpY/ccEGu7Uh3tM1 KDeLG5N0dd1I4DIqSeIDmfRk3BUcfr2jiYvNFEEFY1xbxiKVI07yze0uWd4LpfwJyzj8 hC7Rw0BOhfpL8aTCzDShyO+E+aNq385b6PB9PaIRc09fj0EAXc3IaXv9R/nt7A4omccy wEfKFuP58PyplEpqNpmge5RhP9NbHx0DmSLWggy5rDhILrkSiM1xqVrCSACpy3YMtsQY 4B/g== MIME-Version: 1.0 X-Received: by 10.182.95.68 with SMTP id di4mr666650obb.23.1445316456188; Mon, 19 Oct 2015 21:47:36 -0700 (PDT) Sender: kob6558@gmail.com Received: by 10.202.50.136 with HTTP; Mon, 19 Oct 2015 21:47:36 -0700 (PDT) In-Reply-To: References: <79830D9D-94E6-47A9-92B9-D63DF5432272@netapp.com> Date: Mon, 19 Oct 2015 21:47:36 -0700 X-Google-Sender-Auth: mu20mEJtIp-v9tnindqvu94Bx5g Message-ID: Subject: Re: ixl 40G bad performance? From: Kevin Oberman To: Luigi Rizzo Cc: "Eggert, Lars" , "jfv@FreeBSD.org" , "ricera10@gmail.com" , Giuseppe Lettieri , "net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Oct 2015 04:47:37 -0000 On Mon, Oct 19, 2015 at 8:11 AM, Luigi Rizzo wrote: > On Monday, October 19, 2015, Eggert, Lars wrote: > > > Hi, > > > > On 2015-10-19, at 16:20, Luigi Rizzo > > > wrote: > > > > > > i would look at the following: > > > - c states and clock speed - make sure you never go below C1, > > > and fix the clock speed to max. > > > Sure these parameters also affect the 10G card, but there > > > may be strange interaction that trigger the power saving > > > modes in different ways > > > > I already have powerd_flags="-a max -b max -n max" in rc.conf, which I > > hope should be enough. > > > I suspect it might not touch the c states, but better check. The safest is > disable them in the bios. > To disable C-States: sysctl dev.cpu.0.cx_lowest=C1 -- Kevin Oberman, Part time kid herder and retired Network Engineer From owner-freebsd-net@freebsd.org Tue Oct 20 08:24:54 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 31265A18987 for ; Tue, 20 Oct 2015 08:24:54 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 1765D2E0 for ; Tue, 20 Oct 2015 08:24:54 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: by mailman.ysv.freebsd.org (Postfix) id 1752AA18985; Tue, 20 Oct 2015 08:24:54 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 16F28A18984 for ; Tue, 20 Oct 2015 08:24:54 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [115.70.110.159]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 79FFB2DF for ; Tue, 20 Oct 2015 08:24:52 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id t9K8ObpU066842; Tue, 20 Oct 2015 19:24:37 +1100 (EST) (envelope-from smithi@nimnet.asn.au) Date: Tue, 20 Oct 2015 19:24:37 +1100 (EST) From: Ian Smith To: Kevin Oberman cc: Luigi Rizzo , "jfv@FreeBSD.org" , "ricera10@gmail.com" , "Eggert, Lars" , Giuseppe Lettieri , "net@freebsd.org" Subject: Re: ixl 40G bad performance? In-Reply-To: Message-ID: <20151020190541.B15983@sola.nimnet.asn.au> References: <79830D9D-94E6-47A9-92B9-D63DF5432272@netapp.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Oct 2015 08:24:54 -0000 On Mon, 19 Oct 2015 21:47:36 -0700, Kevin Oberman wrote: > > I suspect it might not touch the c states, but better check. The safest is > > disable them in the bios. > > > > To disable C-States: > sysctl dev.cpu.0.cx_lowest=C1 Actually, you want to set hw.acpi.cpu.cx_lowest=C1 instead. Otherwise you've only changed cpu.0; if you try it you should see that other CPUs will have retained their previous C-state setting - up to 9.3 at least. Setting performance_cx_lowest=C1 in rc.conf (and economy_cx_lowest=C1 on laptops) performs that by setting hw.acpi.cpu.cx_lowest on boot (and on every change to/from battery power) in power_profile via devd notifies. cheers, Ian From owner-freebsd-net@freebsd.org Tue Oct 20 11:03:36 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 52D48A19523 for ; Tue, 20 Oct 2015 11:03:36 +0000 (UTC) (envelope-from lars@netapp.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 2ACD4121B for ; Tue, 20 Oct 2015 11:03:36 +0000 (UTC) (envelope-from lars@netapp.com) Received: by mailman.ysv.freebsd.org (Postfix) id 29DE5A19522; Tue, 20 Oct 2015 11:03:36 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 29743A19521 for ; Tue, 20 Oct 2015 11:03:36 +0000 (UTC) (envelope-from lars@netapp.com) Received: from mx141.netapp.com (mx141.netapp.com [216.240.21.12]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "mx141.netapp.com", Issuer "Symantec Class 3 Secure Server CA - G4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D07F1121A; Tue, 20 Oct 2015 11:03:35 +0000 (UTC) (envelope-from lars@netapp.com) X-IronPort-AV: E=Sophos;i="5.17,707,1437462000"; d="asc'?scan'208";a="75991804" Received: from hioexcmbx04-prd.hq.netapp.com ([10.122.105.37]) by mx141-out.netapp.com with ESMTP; 20 Oct 2015 04:03:15 -0700 Received: from HIOEXCMBX07-PRD.hq.netapp.com (10.122.105.40) by hioexcmbx04-prd.hq.netapp.com (10.122.105.37) with Microsoft SMTP Server (TLS) id 15.0.1104.5; Tue, 20 Oct 2015 04:03:15 -0700 Received: from HIOEXCMBX07-PRD.hq.netapp.com ([::1]) by hioexcmbx07-prd.hq.netapp.com ([fe80::e1d9:911e:3048:d510%21]) with mapi id 15.00.1104.000; Tue, 20 Oct 2015 04:03:15 -0700 From: "Eggert, Lars" To: Ian Smith CC: Kevin Oberman , Luigi Rizzo , "jfv@FreeBSD.org" , "ricera10@gmail.com" , Giuseppe Lettieri , "net@freebsd.org" Subject: Re: ixl 40G bad performance? Thread-Topic: ixl 40G bad performance? Thread-Index: AQHRCnVeClivuZWSFkK4WCM/ynkiH55zUwOAgAAMKACAAAITgIAA5BEAgAA8ooCAACxPAA== Date: Tue, 20 Oct 2015 11:03:14 +0000 Message-ID: <6CD6754D-FC0E-4B24-AAEC-7C9D68284141@netapp.com> References: <79830D9D-94E6-47A9-92B9-D63DF5432272@netapp.com> <20151020190541.B15983@sola.nimnet.asn.au> In-Reply-To: <20151020190541.B15983@sola.nimnet.asn.au> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-mailer: Apple Mail (2.3094) x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [10.122.56.79] Content-Type: multipart/signed; boundary="Apple-Mail=_D6D2EEA7-5DE1-488B-A132-48E5E1E99B3B"; protocol="application/pgp-signature"; micalg=pgp-sha256 MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Oct 2015 11:03:36 -0000 --Apple-Mail=_D6D2EEA7-5DE1-488B-A132-48E5E1E99B3B Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi, On 2015-10-20, at 10:24, Ian Smith wrote: > Actually, you want to set hw.acpi.cpu.cx_lowest=3DC1 instead. Done. On 2015-10-19, at 17:55, Luigi Rizzo wrote: > On Mon, Oct 19, 2015 at 8:34 AM, Eggert, Lars wrote: >> The only other sysctls in ixl(4) that look relevant are: >>=20 >> hw.ixl.rx_itr >> The RX interrupt rate value, set to 8K by default. >>=20 >> hw.ixl.tx_itr >> The TX interrupt rate value, set to 4K by default. >>=20 >=20 > yes those. raise to 20-50k and see what you get in > terms of ping latency. While ixl(4) talks about 8K and 4K, the defaults actually seem to be: hw.ixl.tx_itr: 122 hw.ixl.rx_itr: 62 Doubling those values *increases* flood ping latency to ~200 usec (from = ~116 usec). Halving them to 62/31 decreases flood ping latency to ~50 usec, but = still doesn't increase iperf throughput (still 2.8 Gb/s). Going to 31/16 = further drops latency to 24 usec, with no change in throughput. (Looking at the "interrupt Moderation parameters" #defines in = sys/dev/ixl/ixl.h it seems that ixl likes to have its irq rates = specified with some weird divider scheme.) With 5/5 (which corresponds to IXL_ITR_100K), I get down to 16 usec. = Unfortunately, throughput is then also down to about 2 Gb/s. One thing I noticed in top is that one queue irq is using quite a bit of = CPU when I run iperf: 11 0 -92 - 0K 1152K CPU2 2 0:19 50.98% = intr{irq293: ixl1:q2} 11 0 -92 - 0K 1152K WAIT 3 0:02 5.18% = intr{irq294: ixl1:q3} 0 0 -92 0 0K 8944K - 25 0:01 1.07% = kernel{ixl1 que} 11 0 -92 - 0K 1152K WAIT 1 0:01 0.00% = intr{irq292: ixl1:q1} 11 0 -92 - 0K 1152K WAIT 0 0:00 0.00% = intr{irq291: ixl1:q0} 0 0 -92 0 0K 8944K - 22 0:00 0.00% = kernel{ixl1 adminq} 0 0 -92 0 0K 8944K - 31 0:00 0.00% = kernel{ixl1 que} 0 0 -92 0 0K 8944K - 31 0:00 0.00% = kernel{ixl1 que} 0 0 -92 0 0K 8944K - 31 0:00 0.00% = kernel{ixl1 que} 11 0 -92 - 0K 1152K WAIT -1 0:00 0.00% = intr{irq290: ixl1:aq} With 10G ix interfaces and a throughput of ~9Gb/s, the CPU load is much = lower: 11 0 -92 - 0K 1152K WAIT 0 0:05 7.67% = intr{irq274: ix0:que } 0 0 -92 0 0K 8944K - 27 0:00 0.29% = kernel{ix0 que} 0 0 -92 0 0K 8944K - 10 0:00 0.00% = kernel{ix0 linkq} 11 0 -92 - 0K 1152K WAIT 1 0:00 0.00% = intr{irq275: ix0:que } 11 0 -92 - 0K 1152K WAIT 3 0:00 0.00% = intr{irq277: ix0:que } 11 0 -92 - 0K 1152K WAIT 2 0:00 0.00% = intr{irq276: ix0:que } 11 0 -92 - 0K 1152K WAIT 18 0:00 0.00% = intr{irq278: ix0:link} 0 0 -92 0 0K 8944K - 0 0:00 0.00% = kernel{ix0 que} 0 0 -92 0 0K 8944K - 0 0:00 0.00% = kernel{ix0 que} 0 0 -92 0 0K 8944K - 0 0:00 0.00% = kernel{ix0 que} Lars --Apple-Mail=_D6D2EEA7-5DE1-488B-A132-48E5E1E99B3B Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="signature.asc" Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- iQCVAwUBViYfcdZcnpRveo1xAQiS1gP/bGeKFsEN+BVIzUbtCDdGkoDsDL7eZIuZ Qfdw8J63Azn4DfnPvgKlH330ESm45UaSYvojSQev16rO6BfU53Sa0K+7nzaSyqzf kkyiOseAXHTTsAOnL/IA4djPcdYLSD2qGRyx1LpeNc3uMaVshcxlzeCgFmSLyJa4 17FAveCAMk4= =1HCx -----END PGP SIGNATURE----- --Apple-Mail=_D6D2EEA7-5DE1-488B-A132-48E5E1E99B3B-- From owner-freebsd-net@freebsd.org Tue Oct 20 14:51:12 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E60ADA1A3F3 for ; Tue, 20 Oct 2015 14:51:11 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id C8855DF3 for ; Tue, 20 Oct 2015 14:51:11 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: by mailman.ysv.freebsd.org (Postfix) id C7BC0A1A3F2; Tue, 20 Oct 2015 14:51:11 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AD854A1A3F1 for ; Tue, 20 Oct 2015 14:51:11 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail105.syd.optusnet.com.au (mail105.syd.optusnet.com.au [211.29.132.249]) by mx1.freebsd.org (Postfix) with ESMTP id 59182DF1; Tue, 20 Oct 2015 14:51:11 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from c211-30-166-197.carlnfd1.nsw.optusnet.com.au (c211-30-166-197.carlnfd1.nsw.optusnet.com.au [211.30.166.197]) by mail105.syd.optusnet.com.au (Postfix) with ESMTPS id 81F6310452E2; Wed, 21 Oct 2015 01:51:01 +1100 (AEDT) Date: Wed, 21 Oct 2015 01:51:00 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: "Eggert, Lars" cc: Ian Smith , Kevin Oberman , "net@freebsd.org" , "jfv@FreeBSD.org" , "ricera10@gmail.com" , Luigi Rizzo , Giuseppe Lettieri Subject: Re: ixl 40G bad performance? In-Reply-To: <6CD6754D-FC0E-4B24-AAEC-7C9D68284141@netapp.com> Message-ID: <20151020232218.G1833@besplex.bde.org> References: <79830D9D-94E6-47A9-92B9-D63DF5432272@netapp.com> <20151020190541.B15983@sola.nimnet.asn.au> <6CD6754D-FC0E-4B24-AAEC-7C9D68284141@netapp.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=R6/+YolX c=1 sm=1 tr=0 a=KA6XNC2GZCFrdESI5ZmdjQ==:117 a=PO7r1zJSAAAA:8 a=JzwRw_2MAAAA:8 a=kj9zAlcOel0A:10 a=JDjsHSkAAAAA:8 a=p_OtkrAIStXhInMiOlUA:9 a=CjuIK1q_8ugA:10 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Oct 2015 14:51:12 -0000 On Tue, 20 Oct 2015, Eggert, Lars wrote: > Hi, > > On 2015-10-20, at 10:24, Ian Smith wrote: >> Actually, you want to set hw.acpi.cpu.cx_lowest=C1 instead. > > Done. > > On 2015-10-19, at 17:55, Luigi Rizzo wrote: >> On Mon, Oct 19, 2015 at 8:34 AM, Eggert, Lars wrote: >>> The only other sysctls in ixl(4) that look relevant are: >>> >>> hw.ixl.rx_itr >>> The RX interrupt rate value, set to 8K by default. >>> >>> hw.ixl.tx_itr >>> The TX interrupt rate value, set to 4K by default. >>> >> >> yes those. raise to 20-50k and see what you get in >> terms of ping latency. > > While ixl(4) talks about 8K and 4K, the defaults actually seem to be: > > hw.ixl.tx_itr: 122 > hw.ixl.rx_itr: 62 ixl seems to have a different set of itr sysctl bugs than em. In em, 122 for the itr means 125 initially, but it is documented (only by sysctl -d, not by the man page) as having units usecs/4. The units are actually usecs*4 except initially, and these units take effect if you write the initial value back -- writing back 122 changes the active period from 125 to 488. 122 instead of 125 is the result of confusion between powers of 2 and powers of 10. The first obvious bug in ixl is that the above sysctls are read-only global tunables (not documented as sysctls of course), but you can write them using per-device sysctls (dev.ixl.[0-N].*itr?). Writing them for 1 device clobbers the globals and probably the settings for all ixl devices. sysctl -d doesn't say anything useful about ixl's itrs. It misdocuments the units for all of them as being rates. Actually, the units for 2 of them are boolean and the units for the other 2 are periods. ixl(4) uses better wording for the booleans but even worse wording for the periods ("rate value"). em uses better wording for its itr sysctl but em(4) has no documentation for any sysctl or its itr tunable. igb is more like em than ixl here. 122 seems to be the result of mis-scaling 125, and 62 from correctly scaling 62.5, but these numbers are also off by a factor of 2. Either there is a scaling bug or the undocumented units are usecs/2 where em's documented units are usecs/4. In em, the default itr rate is 8 kHz (power of 10), but in ixl it is unclear if 4K and 8K are actually 4000 and 8000, since they are scaled more in hardware (IXL_ITR_4K is hard-coded as 122; the scale is linear but their aren't enough bits to preserve linearity; it is unclear if the hard-coded values are defined by the hardware or are the result of precomputing the values (using hard-coded 0x7A (122) where em uses 1000000 / SCALE (100000 being user-friendly microseconds and SCALE a hardware clock frequency)). I think 122 really does mean a period that approximates the period for a frequency of 4 khz. The period for this frequency is 250 usecs, and 122 is 250 with units of usec*2, with an approximate error of 3 units. Or 122 is the period for the documented frequency of 4K (binary power of 2 with undocumented units which I assume are Hz), with the weird usec*2 units and a tiny error. Similarly for 62 and 8K, except there is a rounding error of almost 1. > Doubling those values *increases* flood ping latency to ~200 usec (from ~116 usec). Since they are periods and not frequencies, doubling them should double the latency. Since their units are weird and undocumented, it is hard to predict what the latency actually is. But I predict that if the units are usecs*2, then the unscaled values give average latencies from interrupt moderation. This gives 122 + 62 = 184 plus maybe another 20 for other delays. Since the observed average latency is less than half that, the units seem to usecs*1 and it is the documented frequencies that are off by a power of 2. > Halving them to 62/31 decreases flood ping latency to ~50 usec, but still doesn't increase iperf throughput (still 2.8 Gb/s). Going to 31/16 further drops latency to 24 usec, with no change in throughput. For em and lem, I use itr = 0 or 1 when optimizing for latency. This reduces the latency to 50 for lem but only to 73 for em (where the connection goes through a slow switch to not so slow bge). 24 seems quite good, and the lowest I have seen for 1 Gbps is 26, but this requires kludges like a direct connection and polling, and I would hope for 40 times lower at 40 Gbps. > (Looking at the "interrupt Moderation parameters" #defines in sys/dev/ixl/ixl.h it seems that ixl likes to have its irq rates specified with some weird divider scheme.) > > With 5/5 (which corresponds to IXL_ITR_100K), I get down to 16 usec. Unfortunately, throughput is then also down to about 2 Gb/s. Lowering (improving) latency always lowers (unimproves) throughput by increasing load. itr = 8 kHz is resonable for 1 Gbps (it gives higher latency than I like), but scaling that to 40 Gbps gives itr = 320 kHz and it is impossible to scale up the speed of a single CPU to reasonbly keep up with that. Fix for em: X diff -u2 if_em.c~ if_em.c X --- if_em.c~ 2015-09-28 06:29:35.000000000 +0000 X +++ if_em.c 2015-10-18 18:49:36.876699000 +0000 X @@ -609,8 +609,8 @@ X em_tx_abs_int_delay_dflt); X em_add_int_delay_sysctl(adapter, "itr", X - "interrupt delay limit in usecs/4", X + "interrupt delay limit in usecs", X &adapter->tx_itr, X E1000_REGISTER(hw, E1000_ITR), X - DEFAULT_ITR); X + 1000000 / MAX_INTS_PER_SEC); X X /* Sysctl for limiting the amount of work done in the taskqueue */ "delay limit" is fairly good wording. Other parameters tend to give long delays, but itr limits the longest delay due to interrupt moderation to whatever the itr respresents. Bruce From owner-freebsd-net@freebsd.org Tue Oct 20 18:32:35 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F3887A1A2BD; Tue, 20 Oct 2015 18:32:35 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D22A41B45; Tue, 20 Oct 2015 18:32:35 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from ralph.baldwin.cx (c-73-231-226-104.hsd1.ca.comcast.net [73.231.226.104]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 2547BB91E; Tue, 20 Oct 2015 14:32:34 -0400 (EDT) From: John Baldwin To: freebsd-drivers@freebsd.org Cc: David Somayajulu , "freebsd-net@freebsd.org" Subject: Re: Question on mbufs Date: Tue, 20 Oct 2015 11:31:16 -0700 Message-ID: <3003826.l9YxHWDkZK@ralph.baldwin.cx> User-Agent: KMail/4.14.3 (FreeBSD/10.2-PRERELEASE; KDE/4.14.3; amd64; ; ) In-Reply-To: <49F5640B08EAA94DAF2F6B6145E6A08A0194383DC4@AVMB1.qlogic.org> References: <49F5640B08EAA94DAF2F6B6145E6A08A0194383DC4@AVMB1.qlogic.org> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 20 Oct 2015 14:32:34 -0400 (EDT) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Oct 2015 18:32:36 -0000 On Friday, October 16, 2015 05:56:33 PM David Somayajulu wrote: > Hi All, > When indicating a chain of mbufs to the network via ifp->if_input(), what are the repercussions of setting M_PKTHDR bit in all the mbufs in the chain, instead of just the first mbuf ? > Thanks > David S. Right now the various input routines do not expect a chain of packets and probably assume that the chain is a single packet. I assume you mean that the chain here represents a single mbuf still and they are chained via m_next and not m_nextpkt? I suspect that this is probably fine and that most things will only check for M_PKTHDR and look for the fields in the first mbuf in the chain. -- John Baldwin From owner-freebsd-net@freebsd.org Tue Oct 20 18:53:15 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CC1BDA1AB33 for ; Tue, 20 Oct 2015 18:53:15 +0000 (UTC) (envelope-from freebsd@wagsky.com) Received: from bmx.allycomm.com (bmx.allycomm.com [198.199.108.230]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AE30EF5A for ; Tue, 20 Oct 2015 18:53:14 +0000 (UTC) (envelope-from freebsd@wagsky.com) Received: from jmk-mbp.guidewire.com (inet.guidewire.com [199.91.42.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by bmx.allycomm.com (Postfix) with ESMTPSA id 55B93D9D3B; Tue, 20 Oct 2015 11:43:15 -0700 (PDT) To: freebsd-net@freebsd.org From: Jeff Kletsky Subject: netgraph snooping failing using tcpdump with ng_tee and ng_eiface Message-ID: <56268B27.5000809@wagsky.com> Date: Tue, 20 Oct 2015 11:42:47 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Oct 2015 18:53:16 -0000 I'm in the process of trying to debug a deeper question with netgraph, but am puzzled as to why I can't seem to use tcpdump with ng_tee and ng_eiface. I don't see any packets with tcpdump on either the ng_eiface connected to ng_tee left2right or to ng_tee right2left when there are packets flowing through the ng_tee. TL;DR I can't see packets using tcpdump on ng_eiface connected to ng_tee The configuration can be seen in detail with a graphic from ncgtl dot: In summary: re0 (ether) ----------\ | | re0_tee_upper re0_tee_lower | | re0_bridge -----------/ | ng0_testjail_tee | ng0_testjail (eiface, passed to a vnet-enabled jail) The jail can clearly communicate through ng0_testjail to the outside world (physically connected to re0) (ifconfig and netstat -rn for host and jail at the bottom of this message) I've added ng_eiface nodes to all the left2right and right2left tees: + mkpeer ng0_testjail_tee: eiface left2right ether + mkpeer ng0_testjail_tee: eiface right2left ether + mkpeer re0_tee_lower: eiface left2right ether + mkpeer re0_tee_lower: eiface right2left ether + mkpeer re0_tee_upper: eiface left2right ether + mkpeer re0_tee_upper: eiface right2left ether If I run 'tcpdump -i ngeth1' on the host (left2right tap on ng_tee between the jail's VNET ng_eiface and the ng_bridge), I can see it is put into promiscuous mode: ngeth1: flags=8902 metric 0 mtu 1500 options=28 ether 00:00:00:00:00:00 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active If I make a connection to the outside world from inside the jail, I would expect the packets to flow through ng0_testjail (eiface in jail) ng0_testjail_tee re0_bridge re0_tee_lower or re0_tee_upper re0 and back again. Based on this, I would expect there to be packets copied to the taps of the ng0_testjail_tee and then to the ng_eiface tap attached to the ng_tee. However, I don't see anything with tcpdump on the ng_eiface tap. What am I missing here in being able to "snoop" the traffic within my virtual netgraph network? Are the packets somehow bypassing the virtual network and being routed directly to re0? TIA, Jeff Host: ----- re0: flags=8943 metric 0 mtu 1500 options=8209b ether d0:50:99:51:38:eb inet 192.168.6.13 netmask 0xffffff00 broadcast 192.168.6.255 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active Routing tables Internet: Destination Gateway Flags Netif Expire default 192.168.6.1 UGS re0 127.0.0.1 link#2 UH lo0 192.168.6.0/24 link#1 U re0 192.168.6.13 link#1 UHS lo0 VNET jail: ---------- ng0_testjail: flags=8843 metric 0 mtu 1500 options=28 ether 02:00:28:51:38:eb inet 192.168.6.213 netmask 0xffffff00 broadcast 192.168.6.255 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active Routing tables Internet: Destination Gateway Flags Netif Expire default 192.168.6.1 UGS ng0_test 127.0.0.1 link#1 UH lo0 192.168.6.0/24 link#2 U ng0_test 192.168.6.213 link#2 UHS lo0 arp -a: wildside.pn.wagsky.com (192.168.6.1) at 68:05:ca:34:34:7f on ng0_testjail expires in 966 seconds [ethernet] ? (192.168.6.213) at 02:00:28:51:38:eb on ng0_testjail permanent [ethernet] From owner-freebsd-net@freebsd.org Wed Oct 21 01:31:50 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 18E0EA19769 for ; Wed, 21 Oct 2015 01:31:50 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-wi0-x232.google.com (mail-wi0-x232.google.com [IPv6:2a00:1450:400c:c05::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 91571338 for ; Wed, 21 Oct 2015 01:31:49 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by wicll6 with SMTP id ll6so69615507wic.0 for ; Tue, 20 Oct 2015 18:31:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sippysoft_com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=V705hSiPRkXnu6/nUOKwFKgz8X0tV3gQfzM+GdS0m38=; b=ngQm+6zw7emVASS3MI39+PhHLBDeb49u6TaGLvUME/nn/ffBwvSzah05y5ui4VPM37 K6wko60MrXcFSHfOP6DyFi4KdSQKcq97ZjDOaAzs5vdj8rZnExn7MqYdPABGf+/vAE91 D1xFz2MXYPwd2quvtHTioZqc17K5obSEwyKoBXvOSTw/XRnRnOfkPg1LDB70uDc7SET0 OASPlIzdNCPX/lH2E9+G4AHQG7RgkOZT/4B33rmZ3b8J6rad5opcCK1IO3hXMSpSUHGC rhNDiH8fUj/mr9DJigJEoCVBavqKPyHne81pkLdf/+v9eCt7d3eQB1ALEhvwhbnRVxFW SLag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=V705hSiPRkXnu6/nUOKwFKgz8X0tV3gQfzM+GdS0m38=; b=gx4eNTAUWm/ZO8Yu1FRDSkmP8yXk0G11o3FNmIY5uHbq2XLSxRuVWOSHanSfqpnSH+ 5Fs0t8Fg+TvmijpfT2bzE7BocsU2fcaTV7vl0Elvf9cFo08TXedwD9XtCE1JWnv1HtSa j822vwsxlcTufChONf2EqriryL8Dp81ZLOEJ9gYkSHN+6L2V2sdkhhtYvadXxXJQeutR od6g8+A56TOt4sRSG2BXL0itbfFvpgN1IAsLya9VLs9cgRWUQROAb+uckegYnfcyXmO7 StiyKtCvgMf9R0HHg/UmgRIlUSCZs2HdfQPxZImco6OAxzmyCgGO2/3wynUZQPusfZXP UFbA== X-Gm-Message-State: ALoCoQmHPqYrVXa2NLUwPnon+RDmiamIWaTk9+ryNBLmjfj86FYuiNJaC1pso9jFuFXEUDj/GYHh MIME-Version: 1.0 X-Received: by 10.180.72.177 with SMTP id e17mr29438440wiv.8.1445391107699; Tue, 20 Oct 2015 18:31:47 -0700 (PDT) Sender: sobomax@sippysoft.com Received: by 10.27.11.228 with HTTP; Tue, 20 Oct 2015 18:31:47 -0700 (PDT) In-Reply-To: <1608354.LQmTMSsd5C@ralph.baldwin.cx> References: <1608354.LQmTMSsd5C@ralph.baldwin.cx> Date: Tue, 20 Oct 2015 18:31:47 -0700 X-Google-Sender-Auth: COULeuZRMgTv_hswFpUVyAgCQj8 Message-ID: Subject: Re: Some MSI are not routed correctly From: Maxim Sobolev To: John Baldwin Cc: FreeBSD Net Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Oct 2015 01:31:50 -0000 Here you go: $ sudo procstat -S 11 PID TID COMM TDNAME CPU CSID CPU MASK 11 100026 intr swi3: vm 0 1 0-23 11 100027 intr swi4: clock 0 1 0-23 11 100028 intr swi4: clock 0 1 0-23 11 100029 intr swi4: clock 0 1 0-23 11 100030 intr swi4: clock 0 1 0-23 11 100031 intr swi4: clock 0 1 0-23 11 100032 intr swi4: clock 0 1 0-23 11 100033 intr swi4: clock 0 1 0-23 11 100034 intr swi4: clock 0 1 0-23 11 100035 intr swi4: clock 0 1 0-23 11 100036 intr swi4: clock 0 1 0-23 11 100037 intr swi4: clock 0 1 0-23 11 100038 intr swi4: clock 0 1 0-23 11 100039 intr swi4: clock 0 1 0-23 11 100040 intr swi4: clock 0 1 0-23 11 100041 intr swi4: clock 0 1 0-23 11 100042 intr swi4: clock 0 1 0-23 11 100043 intr swi4: clock 0 1 0-23 11 100044 intr swi4: clock 0 1 0-23 11 100045 intr swi4: clock 0 1 0-23 11 100046 intr swi4: clock 0 1 0-23 11 100047 intr swi4: clock 0 1 0-23 11 100048 intr swi4: clock 0 1 0-23 11 100049 intr swi4: clock 0 1 0-23 11 100050 intr swi4: clock 0 1 0-23 11 100051 intr swi1: netisr 0 18 1 0-23 11 100067 intr swi6: task queue 21 1 0-23 11 100068 intr swi6: Giant task 15 1 0-23 11 100070 intr swi5: fast taskq 0 1 0-23 11 100073 intr irq264: igb0:que 0 1 0 11 100075 intr irq265: igb0:que 1 1 1 11 100077 intr irq266: igb0:que 2 1 2 11 100079 intr irq267: igb0:que 3 1 3 11 100081 intr irq268: igb0:lin 6 1 0-23 11 100082 intr irq269: igb1:que 4 1 4 11 100084 intr irq270: igb1:que 5 1 5 11 100086 intr irq271: igb1:que 6 1 6 11 100088 intr irq272: igb1:que 7 1 7 11 100090 intr irq273: igb1:lin 7 1 0-23 11 100091 intr irq274: ahci0 0 1 0-23 11 100092 intr irq19: xhci0 5 1 0-23 11 100097 intr irq18: ehci0 ehc 4 1 0-23 11 100102 intr irq275: igb2:que 0 1 8 11 100104 intr irq276: igb2:que 0 1 9 11 100106 intr irq277: igb2:que 0 1 10 11 100108 intr irq278: igb2:que 0 1 11 11 100110 intr irq279: igb2:lin 0 1 0-23 11 100111 intr irq280: igb3:que 0 1 12 11 100113 intr irq281: igb3:que 0 1 13 11 100115 intr irq282: igb3:que 0 1 14 11 100117 intr irq283: igb3:que 0 1 15 11 100119 intr irq284: igb3:lin 0 1 0-23 11 100125 intr irq285: mrsas0 11 1 0-23 11 100126 intr irq286: mrsas0 14 1 0-23 11 100127 intr irq287: mrsas0 13 1 0-23 11 100128 intr irq288: mrsas0 14 1 0-23 11 100129 intr irq289: mrsas0 15 1 0-23 11 100130 intr irq290: mrsas0 16 1 0-23 11 100131 intr irq291: mrsas0 17 1 0-23 11 100132 intr irq292: mrsas0 18 1 0-23 11 100133 intr irq293: mrsas0 19 1 0-23 11 100134 intr irq294: mrsas0 20 1 0-23 11 100135 intr irq295: mrsas0 21 1 0-23 11 100136 intr irq296: mrsas0 22 1 0-23 11 100137 intr irq297: mrsas0 23 1 0-23 11 100138 intr irq298: mrsas0 0 1 0-23 11 100139 intr irq299: mrsas0 4 1 0-23 11 100140 intr irq300: mrsas0 2 1 0-23 11 100142 intr swi0: uart uart 0 1 0-23 On Mon, Oct 19, 2015 at 2:03 PM, John Baldwin wrote: > On Thursday, October 08, 2015 07:33:27 AM Maxim Sobolev wrote: > > Hi John & others, > > > > We've came across a weird MSI routing issue on one of our newest dual > > E5-2690v3 (haswell) Supermicro X10DRL-i boxes running latest 10.2-p4. It > is > > fitted with dual port Intel I350 card, in addition to the built-in I210 > > chip that is not used. The hw.igb.num_queues is set to 4, and the driver > > reports binding to the CPUs 0-3 for the first port and CPUs 4-7 for the > > second, however when verified with top -P under the load, interrupts are > > only delivered to the CPUs 0-3, no interrupt time is recorded on the CPUs > > 4-7. systat -vm shows that all 8 queues are firing interrupts, so my > guess > > that for whatever reason bus_bind_intr() is not doing what's expected to > do > > for half of those interrupts. > > > > What's interesting is that on a similar box (same chassis/mobo/cpu) but > > equipped with the quad-port X540-AT2 10Gig card, interrupts are routed > > properly. The latter is running with hw.ix.num_queues="3". > > > > pcib2: port 0xcf8-0xcff on acpi0 > > pci0: on pcib2 > > pcib3: irq 26 at device 1.0 on pci0 > > pci1: on pcib3 > > igb0: mem > > 0xc7200000-0xc72fffff,0xc7304000-0xc7307fff irq 26 at device 0.0 on pci1 > > igb0: Using MSIX interrupts with 5 vectors > > igb0: Ethernet address: a0:36:9f:76:af:20 > > igb0: Bound queue 0 to cpu0 > > igb0: Bound queue 1 to cpu1 > > igb0: Bound queue 2 to cpu2 > > igb0: Bound queue 3 to cpu3 > > igb0: netmap queues/slots: TX 4/4096, RX 4/4096 > > igb1: mem > > 0xc7100000-0xc71fffff,0xc7300000-0xc7303fff irq 28 at device 0.1 on pci1 > > igb1: Using MSIX interrupts with 5 vectors > > igb1: Ethernet address: a0:36:9f:76:af:21 > > igb1: Bound queue 0 to cpu4 > > igb1: Bound queue 1 to cpu5 > > igb1: Bound queue 2 to cpu6 > > igb1: Bound queue 3 to cpu7 > > igb1: netmap queues/slots: TX 4/4096, RX 4/4096 > > > > pcib2: port 0xcf8-0xcff on acpi0 > > pci0: on pcib2 > > pcib3: irq 26 at device 1.0 on pci0 > > pci1: on pcib3 > > pcib4: irq 32 at device 2.0 on pci0 > > pci2: on pcib4 > > pcib5: irq 40 at device 3.0 on pci0 > > pci3: on pcib5 > > ix0: > port > > 0x6020-0x603f mem 0xc7c00000-0xc7dfffff,0xc7e04000-0xc7e07fff irq 40 at > > device 0.0 on pci3 > > ix0: Using MSIX interrupts with 4 vectors > > ix0: Bound queue 0 to cpu 0 > > ix0: Bound queue 1 to cpu 1 > > ix0: Bound queue 2 to cpu 2 > > ix0: Ethernet address: 0c:c4:7a:5e:be:64 > > ix0: PCI Express Bus: Speed 5.0GT/s Width x8 > > ix0: netmap queues/slots: TX 3/4096, RX 3/4096 > > ix1: > port > > 0x6000-0x601f mem 0xc7a00000-0xc7bfffff,0xc7e00000-0xc7e03fff irq 44 at > > device 0.1 on pci3 > > ix1: Using MSIX interrupts with 4 vectors > > ix1: Bound queue 0 to cpu 3 > > ix1: Bound queue 1 to cpu 4 > > ix1: Bound queue 2 to cpu 5 > > ix1: Ethernet address: 0c:c4:7a:5e:be:65 > > ix1: PCI Express Bus: Speed 5.0GT/s Width x8 > > ix1: netmap queues/slots: TX 3/4096, RX 3/4096 > > > > Some extra debug is here: > > > > http://sobomax.sippysoft.com/haswell_bug/bad.dmesg > > http://sobomax.sippysoft.com/haswell_bug/lstopo_bad.png > > http://sobomax.sippysoft.com/haswell_bug/systat_vm_bad.png > > http://sobomax.sippysoft.com/haswell_bug/top_P_bad.png > > > > http://sobomax.sippysoft.com/haswell_bug/good.dmesg > > http://sobomax.sippysoft.com/haswell_bug/lstopo_good.png > > http://sobomax.sippysoft.com/haswell_bug/systat_vm_good.png > > http://sobomax.sippysoft.com/haswell_bug/top_P_good.png > > > > Any ideas on how to debug that further are welcome. The box in the > > production, but we can remove traffic during off-peak to run some > > test/debug code on. > > Can you get procstat -S output for the interrupt threads? (Usually > interrupt > threads are in pid 12, so 'procstat -S 12' would suffice.) > > -- > John Baldwin > > From owner-freebsd-net@freebsd.org Wed Oct 21 09:48:45 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 32975A1BFB9 for ; Wed, 21 Oct 2015 09:48:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1EE921376 for ; Wed, 21 Oct 2015 09:48:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9L9mim5005597 for ; Wed, 21 Oct 2015 09:48:45 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 203916] ethernet and wlan interfaces both have the same mac-address after upgrade to r289486 Date: Wed, 21 Oct 2015 09:48:44 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to keywords Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Oct 2015 09:48:45 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203916 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-net@FreeBSD.org Keywords| |regression -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-net@freebsd.org Wed Oct 21 11:50:55 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DCD6EA1ACF4 for ; Wed, 21 Oct 2015 11:50:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C990415BC for ; Wed, 21 Oct 2015 11:50:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9LBotg1071748 for ; Wed, 21 Oct 2015 11:50:55 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 203922] The kern.ipc.acceptqueue limit is too low Date: Wed, 21 Oct 2015 11:50:55 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ngie@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc keywords assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Oct 2015 11:50:56 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203922 Garrett Cooper,425-314-3911 changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ngie@FreeBSD.org Keywords| |patch Assignee|freebsd-bugs@FreeBSD.org |freebsd-net@FreeBSD.org -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-net@freebsd.org Wed Oct 21 12:26:17 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CC65BA1AE31 for ; Wed, 21 Oct 2015 12:26:17 +0000 (UTC) (envelope-from lars@netapp.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id A515A1061 for ; Wed, 21 Oct 2015 12:26:17 +0000 (UTC) (envelope-from lars@netapp.com) Received: by mailman.ysv.freebsd.org (Postfix) id A47BBA1AE30; Wed, 21 Oct 2015 12:26:17 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A41D4A1AE2F for ; Wed, 21 Oct 2015 12:26:17 +0000 (UTC) (envelope-from lars@netapp.com) Received: from mx143.netapp.com (mx143.netapp.com [216.240.21.24]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "mx143.netapp.com", Issuer "Symantec Class 3 Secure Server CA - G4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 55737105E; Wed, 21 Oct 2015 12:26:16 +0000 (UTC) (envelope-from lars@netapp.com) X-IronPort-AV: E=Sophos;i="5.17,711,1437462000"; d="asc'?scan'208";a="73587828" Received: from hioexcmbx06-prd.hq.netapp.com ([10.122.105.39]) by mx143-out.netapp.com with ESMTP; 21 Oct 2015 05:26:00 -0700 Received: from HIOEXCMBX07-PRD.hq.netapp.com (10.122.105.40) by hioexcmbx06-prd.hq.netapp.com (10.122.105.39) with Microsoft SMTP Server (TLS) id 15.0.1104.5; Wed, 21 Oct 2015 05:26:00 -0700 Received: from HIOEXCMBX07-PRD.hq.netapp.com ([::1]) by hioexcmbx07-prd.hq.netapp.com ([fe80::e1d9:911e:3048:d510%21]) with mapi id 15.00.1104.000; Wed, 21 Oct 2015 05:26:00 -0700 From: "Eggert, Lars" To: Bruce Evans CC: Ian Smith , Kevin Oberman , "net@freebsd.org" , "jfv@FreeBSD.org" , "ricera10@gmail.com" , Luigi Rizzo , Giuseppe Lettieri Subject: Re: ixl 40G bad performance? Thread-Topic: ixl 40G bad performance? Thread-Index: AQHRCnVeClivuZWSFkK4WCM/ynkiH55zUwOAgAAMKACAAAITgIAA5BEAgAA8ooCAACxPAIAAP6UAgAFpzQA= Date: Wed, 21 Oct 2015 12:25:59 +0000 Message-ID: <12938D48-44B9-456A-AD7C-630D7E09BB35@netapp.com> References: <79830D9D-94E6-47A9-92B9-D63DF5432272@netapp.com> <20151020190541.B15983@sola.nimnet.asn.au> <6CD6754D-FC0E-4B24-AAEC-7C9D68284141@netapp.com> <20151020232218.G1833@besplex.bde.org> In-Reply-To: <20151020232218.G1833@besplex.bde.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-mailer: Apple Mail (2.3094) x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [10.122.56.79] Content-Type: multipart/signed; boundary="Apple-Mail=_0220844F-3D93-4781-B1B5-E7999C199CB0"; protocol="application/pgp-signature"; micalg=pgp-sha256 MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Oct 2015 12:26:18 -0000 --Apple-Mail=_0220844F-3D93-4781-B1B5-E7999C199CB0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi Bruce, thanks for the very detailed analysis of the ixl sysctls! On 2015-10-20, at 16:51, Bruce Evans wrote: >=20 > Lowering (improving) latency always lowers (unimproves) throughput by > increasing load. That, I also understand. But even when I back off the itr values to = something more reasonable, throughput still remains low. With all the tweaking I have tried, I have yet to top 3 Gb/s with ixl = cards, whereas they do ~13 Gb/s on Linux straight out of the box. Lars --Apple-Mail=_0220844F-3D93-4781-B1B5-E7999C199CB0 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="signature.asc" Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- iQCVAwUBVieEVdZcnpRveo1xAQh9egQAqxX1TeZj7SMuFuecd2js5ODhooofXz7w V8yXFQgjeK6MzthBqlr4jK+LA1osZOWcLIzfXxlIXPYb1MhG5Ei8QMGCH2ZW7B9b Wzu6yavZovncQc/tC89ZfETqTTalqXueCH3vsXlcItvbefuszekBjhBsc+b4LM6I X0jIObpPXho= =EdNi -----END PGP SIGNATURE----- --Apple-Mail=_0220844F-3D93-4781-B1B5-E7999C199CB0-- From owner-freebsd-net@freebsd.org Wed Oct 21 14:14:51 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8B51FA1B6ED for ; Wed, 21 Oct 2015 14:14:51 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 65B80ECA for ; Wed, 21 Oct 2015 14:14:51 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 62AD8A1B6EB; Wed, 21 Oct 2015 14:14:51 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 48676A1B6EA for ; Wed, 21 Oct 2015 14:14:51 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-yk0-x22e.google.com (mail-yk0-x22e.google.com [IPv6:2607:f8b0:4002:c07::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 034FDEC8; Wed, 21 Oct 2015 14:14:51 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: by ykba4 with SMTP id a4so44203191ykb.3; Wed, 21 Oct 2015 07:14:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Coe9FL2Gmi6JbUaHArwbKu5XaK7fQDSvS2y6jrQEBn4=; b=PNLeZStpgRHlJtQdPOy4icRc0DVXOs8tq2vmXyT2exF2Hhafp2n5FWUZguAvlkSX15 BzBBIm7zJ8Wgv6vzemOT6BH9lqSH1QBXBJ231ObLrY4jL77YYjt3nIxhIWMy4W64kNdJ FZ7t7EdwZxqmVfRAn5/CVBJM4nweb1NyAa0tkB+kWSo/Mln63zTIKKcCCRz99SV8IOge 8xg11aiZD5xfQUhyZG1p7S7QT+Hl2SqtEdgBWZr1/LqfGpL7kPufjieRLPn7TbK1vX76 uqcZoXb6fb+HbljWZLfJS2cecBNa/KJDg98PrWLAQwhWOWtTRw2kczXMk47SH1oIzIqr plsg== MIME-Version: 1.0 X-Received: by 10.13.204.65 with SMTP id o62mr1299419ywd.303.1445436889942; Wed, 21 Oct 2015 07:14:49 -0700 (PDT) Received: by 10.37.36.204 with HTTP; Wed, 21 Oct 2015 07:14:49 -0700 (PDT) In-Reply-To: <12938D48-44B9-456A-AD7C-630D7E09BB35@netapp.com> References: <79830D9D-94E6-47A9-92B9-D63DF5432272@netapp.com> <20151020190541.B15983@sola.nimnet.asn.au> <6CD6754D-FC0E-4B24-AAEC-7C9D68284141@netapp.com> <20151020232218.G1833@besplex.bde.org> <12938D48-44B9-456A-AD7C-630D7E09BB35@netapp.com> Date: Wed, 21 Oct 2015 07:14:49 -0700 Message-ID: Subject: Re: ixl 40G bad performance? From: Jack Vogel To: "Eggert, Lars" Cc: Bruce Evans , Ian Smith , Kevin Oberman , "net@freebsd.org" , "jfv@FreeBSD.org" , "ricera10@gmail.com" , Luigi Rizzo , Giuseppe Lettieri Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Oct 2015 14:14:51 -0000 The 40G hardware is absolutely dependent on firmware, if you have a mismatch for instance, it can totally bork things. So, I would work with your Intel rep and be sure you have the correct version for your specific hardware. Good luck, Jack On Wed, Oct 21, 2015 at 5:25 AM, Eggert, Lars wrote: > Hi Bruce, > > thanks for the very detailed analysis of the ixl sysctls! > > On 2015-10-20, at 16:51, Bruce Evans wrote: > > > > Lowering (improving) latency always lowers (unimproves) throughput by > > increasing load. > > That, I also understand. But even when I back off the itr values to > something more reasonable, throughput still remains low. > > With all the tweaking I have tried, I have yet to top 3 Gb/s with ixl > cards, whereas they do ~13 Gb/s on Linux straight out of the box. > > Lars > From owner-freebsd-net@freebsd.org Wed Oct 21 14:59:54 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 70F7CA1B414 for ; Wed, 21 Oct 2015 14:59:54 +0000 (UTC) (envelope-from elof2@sentor.se) Received: from smtp-out.sentor.se (smtp-out.sentor.se [176.124.225.2]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 37DD3132C for ; Wed, 21 Oct 2015 14:59:53 +0000 (UTC) (envelope-from elof2@sentor.se) Received: from localhost (localhost [127.0.0.1]) by farmermaggot.shire.sentor.se (Postfix) with ESMTP id 9616CB61D235 for ; Wed, 21 Oct 2015 16:50:08 +0200 (CEST) Date: Wed, 21 Oct 2015 16:50:08 +0200 (CEST) From: elof2@sentor.se To: freebsd-net Subject: sysctl and signed net.bpf.maxbufsize variable Message-ID: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Oct 2015 14:59:54 -0000 Isn't this a bug? # sysctl net.bpf.maxbufsize=3000000000 net.bpf.maxbufsize: 524288 -> -1294967296 No error message and exit status is 0. Shouldn't net.bpf.maxbufsize be unsigned? I would like sysctl to have a crude sanity control and return an error if you set a positive value but the result becomes negative. ...and also have some specific sanity control to test if you try to set a value waaaay out of bounds (like setting net.bpf.maxbufsize to a value greater than the maximum RAM). /Elof From owner-freebsd-net@freebsd.org Wed Oct 21 15:01:19 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 30403A1B4BD for ; Wed, 21 Oct 2015 15:01:19 +0000 (UTC) (envelope-from lars@netapp.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 0784514D1 for ; Wed, 21 Oct 2015 15:01:19 +0000 (UTC) (envelope-from lars@netapp.com) Received: by mailman.ysv.freebsd.org (Postfix) id 05D68A1B4BB; Wed, 21 Oct 2015 15:01:19 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DFB2BA1B4B9 for ; Wed, 21 Oct 2015 15:01:18 +0000 (UTC) (envelope-from lars@netapp.com) Received: from mx144.netapp.com (mx144.netapp.com [216.240.21.25]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "mx144.netapp.com", Issuer "Symantec Class 3 Secure Server CA - G4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8FCEF14CD; Wed, 21 Oct 2015 15:01:18 +0000 (UTC) (envelope-from lars@netapp.com) X-IronPort-AV: E=Sophos;i="5.17,712,1437462000"; d="asc'?scan'208";a="75225202" Received: from hioexcmbx03-prd.hq.netapp.com ([10.122.105.36]) by mx144-out.netapp.com with ESMTP; 21 Oct 2015 07:59:38 -0700 Received: from HIOEXCMBX07-PRD.hq.netapp.com (10.122.105.40) by hioexcmbx03-prd.hq.netapp.com (10.122.105.36) with Microsoft SMTP Server (TLS) id 15.0.1104.5; Wed, 21 Oct 2015 07:59:38 -0700 Received: from HIOEXCMBX07-PRD.hq.netapp.com ([::1]) by hioexcmbx07-prd.hq.netapp.com ([fe80::e1d9:911e:3048:d510%21]) with mapi id 15.00.1104.000; Wed, 21 Oct 2015 07:59:38 -0700 From: "Eggert, Lars" To: Jack Vogel CC: Ian Smith , Kevin Oberman , "net@freebsd.org" , "jfv@FreeBSD.org" , "ricera10@gmail.com" , Luigi Rizzo , Giuseppe Lettieri Subject: Re: ixl 40G bad performance? Thread-Topic: ixl 40G bad performance? Thread-Index: AQHRCnVeClivuZWSFkK4WCM/ynkiH55zUwOAgAAMKACAAAITgIAA5BEAgAA8ooCAACxPAIAAP6UAgAFpzQCAAB5sgIAADIMA Date: Wed, 21 Oct 2015 14:59:37 +0000 Message-ID: <99F14384-271F-4543-A7F8-E19F4006998A@netapp.com> References: <79830D9D-94E6-47A9-92B9-D63DF5432272@netapp.com> <20151020190541.B15983@sola.nimnet.asn.au> <6CD6754D-FC0E-4B24-AAEC-7C9D68284141@netapp.com> <20151020232218.G1833@besplex.bde.org> <12938D48-44B9-456A-AD7C-630D7E09BB35@netapp.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-mailer: Apple Mail (2.3094) x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [10.122.56.79] Content-Type: multipart/signed; boundary="Apple-Mail=_2A524B7E-B9ED-47C1-8665-B537C5FD8D99"; protocol="application/pgp-signature"; micalg=pgp-sha256 MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Oct 2015 15:01:19 -0000 --Apple-Mail=_2A524B7E-B9ED-47C1-8665-B537C5FD8D99 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi Jack, On 2015-10-21, at 16:14, Jack Vogel wrote: > The 40G hardware is absolutely dependent on firmware, if you have a = mismatch > for instance, it can totally bork things. So, I would work with your = Intel > rep and be sure you have the correct version for your specific = hardware. I got these tester cards from Amazon, so I don't have a rep. I flashed the latest NVM (1.2.5), because previously the FreeBSD driver = was complaining about the firmware being too old. But I did that before = the experiments. If there is anything else I should be doing, I'd appreciate being put in = contact with someone at Intel who can help. Thanks, Lars Lars --Apple-Mail=_2A524B7E-B9ED-47C1-8665-B537C5FD8D99 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="signature.asc" Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- iQCVAwUBVieoWNZcnpRveo1xAQi/xQP/eWaotyEHEKIlboc1lKRPWvxQtHJZvauF 56hq3K3ep4bwdoa4KKeTWd67e/vjIbNZEjDmesyJ8GnPjyItCzsWr1TTYdzEVXWx E9EvmCeuppY+y21Cbhuwz9TZ/IXjAGtFN5pq7N0Sxr5nmDvbpZ+Edbvyk99Dy8HL UuwYXE0HlTY= =4v3N -----END PGP SIGNATURE----- --Apple-Mail=_2A524B7E-B9ED-47C1-8665-B537C5FD8D99-- From owner-freebsd-net@freebsd.org Wed Oct 21 15:33:23 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A9EF7A1BF8B for ; Wed, 21 Oct 2015 15:33:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 96D991355 for ; Wed, 21 Oct 2015 15:33:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9LFXNa5013093 for ; Wed, 21 Oct 2015 15:33:23 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 193579] [axge] axge driver issue with tcp checksum offload with pf nat Date: Wed, 21 Oct 2015 15:33:23 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Oct 2015 15:33:23 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193579 --- Comment #5 from commit-hook@freebsd.org --- A commit references this bug: Author: kp Date: Wed Oct 21 15:32:21 UTC 2015 New revision: 289703 URL: https://svnweb.freebsd.org/changeset/base/289703 Log: MFC r289316: pf: Fix TSO issues In certain configurations (mostly but not exclusively as a VM on Xen) pf produced packets with an invalid TCP checksum. The problem was that pf could only handle packets with a full checksum. The FreeBSD IP stack produces TCP packets with a pseudo-header checksum (only addresses, length and protocol). Certain network interfaces expect to see the pseudo-header checksum, so they end up producing packets with invalid checksums. To fix this stop calculating the full checksum and teach pf to only update TCP checksums if TSO is disabled or the change affects the pseudo-header checksum. PR: 154428, 193579, 198868 Relnotes: yes Sponsored by: RootBSD Changes: _U stable/10/ stable/10/sys/net/pfvar.h stable/10/sys/netpfil/pf/pf.c stable/10/sys/netpfil/pf/pf_ioctl.c stable/10/sys/netpfil/pf/pf_norm.c -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-net@freebsd.org Wed Oct 21 16:19:36 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0587DA1BB8A for ; Wed, 21 Oct 2015 16:19:36 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id DCD5C10A8 for ; Wed, 21 Oct 2015 16:19:35 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: by mailman.ysv.freebsd.org (Postfix) id D998AA1BB89; Wed, 21 Oct 2015 16:19:35 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D935DA1BB88 for ; Wed, 21 Oct 2015 16:19:35 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from mail.strugglingcoder.info (strugglingcoder.info [65.19.130.35]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.strugglingcoder.info", Issuer "mail.strugglingcoder.info" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id C273110A7; Wed, 21 Oct 2015 16:19:35 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from localhost (unknown [10.1.1.3]) (Authenticated sender: hiren@strugglingcoder.info) by mail.strugglingcoder.info (Postfix) with ESMTPA id 2D74010A844; Wed, 21 Oct 2015 09:19:35 -0700 (PDT) Date: Wed, 21 Oct 2015 09:19:35 -0700 From: hiren panchasara To: "Eggert, Lars" , erj@FreeBSD.org Cc: "net@freebsd.org" Subject: Re: ixl 40G bad performance? Message-ID: <20151021161935.GF28288@strugglingcoder.info> References: <20151020190541.B15983@sola.nimnet.asn.au> <6CD6754D-FC0E-4B24-AAEC-7C9D68284141@netapp.com> <20151020232218.G1833@besplex.bde.org> <12938D48-44B9-456A-AD7C-630D7E09BB35@netapp.com> <99F14384-271F-4543-A7F8-E19F4006998A@netapp.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="kbCYTQG2MZjuOjyn" Content-Disposition: inline In-Reply-To: <99F14384-271F-4543-A7F8-E19F4006998A@netapp.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Oct 2015 16:19:36 -0000 --kbCYTQG2MZjuOjyn Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable + Eric from Intel (Also trimming the CC list as it wouldn't let me send the message otherwise.) On 10/21/15 at 02:59P, Eggert, Lars wrote: > Hi Jack, >=20 > On 2015-10-21, at 16:14, Jack Vogel wrote: > > The 40G hardware is absolutely dependent on firmware, if you have a mis= match > > for instance, it can totally bork things. So, I would work with your In= tel > > rep and be sure you have the correct version for your specific hardware. >=20 > I got these tester cards from Amazon, so I don't have a rep. >=20 > I flashed the latest NVM (1.2.5), because previously the FreeBSD driver w= as complaining about the firmware being too old. But I did that before the = experiments. >=20 > If there is anything else I should be doing, I'd appreciate being put in = contact with someone at Intel who can help. Eric, Can you think of anything else that could explain this low performance? Cheers, Hiren --kbCYTQG2MZjuOjyn Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAABCgBmBQJWJ7sXXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRBNEUyMEZBMUQ4Nzg4RjNGMTdFNjZGMDI4 QjkyNTBFMTU2M0VERkU1AAoJEIuSUOFWPt/lBWwH/jebXgUhrOqZLIzi+xKtMGAY l3ay6N9UnmofXeukvaiW9JIyjTkr9dWEDFVTFZfIOAIxzsmaRmBLQnfE9x3djgjm HGl/2X2o/WEeJzHQzL2Yt/M0IaYx8N5t/FqPrSvTOPU3JLKUZMxz6sctO7jtsb+A dZYk43vFSZjrOKz/9UJrb8q2qULROzez3s5+2LQb4wDo+3TGYSOiTEx1hfVGDi0K 5Y6fsEMJootPKYlNRjwWkEclXzNof5k2L3tiODrM1/3+MiAQGk1DyWKi5E18sOMc 7EN2aKxOPp4RZLCHuYYAD4x7hzFjBl9B+yL2kcMDJnq/PonMq2CgFurXIaeQm00= =M8/2 -----END PGP SIGNATURE----- --kbCYTQG2MZjuOjyn-- From owner-freebsd-net@freebsd.org Wed Oct 21 17:41:46 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 52273A1BDD0 for ; Wed, 21 Oct 2015 17:41:46 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2EC631E36; Wed, 21 Oct 2015 17:41:46 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from ralph.baldwin.cx (c-73-231-226-104.hsd1.ca.comcast.net [73.231.226.104]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 88EF4B98F; Wed, 21 Oct 2015 13:41:44 -0400 (EDT) From: John Baldwin To: Maxim Sobolev Cc: FreeBSD Net Subject: Re: Some MSI are not routed correctly Date: Wed, 21 Oct 2015 10:41:01 -0700 Message-ID: <2050066.YiOUntKNB6@ralph.baldwin.cx> User-Agent: KMail/4.14.3 (FreeBSD/10.2-PRERELEASE; KDE/4.14.3; amd64; ; ) In-Reply-To: References: <1608354.LQmTMSsd5C@ralph.baldwin.cx> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Wed, 21 Oct 2015 13:41:44 -0400 (EDT) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Oct 2015 17:41:46 -0000 On Tuesday, October 20, 2015 06:31:47 PM Maxim Sobolev wrote: > Here you go: > > $ sudo procstat -S 11 > PID TID COMM TDNAME CPU CSID CPU MASK > 11 100082 intr irq269: igb1:que 4 1 4 > 11 100084 intr irq270: igb1:que 5 1 5 > 11 100086 intr irq271: igb1:que 6 1 6 > 11 100088 intr irq272: igb1:que 7 1 7 These are clearly what you want, and you can see that the last CPU they ran on is the CPU you want as well. If you run 'top -SHz' do you see the threads running on other CPUs? -- John Baldwin From owner-freebsd-net@freebsd.org Wed Oct 21 18:29:19 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5A9D6A1A7A0 for ; Wed, 21 Oct 2015 18:29:19 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-wi0-x22b.google.com (mail-wi0-x22b.google.com [IPv6:2a00:1450:400c:c05::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E18E115F9 for ; Wed, 21 Oct 2015 18:29:18 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by wicll6 with SMTP id ll6so103598942wic.0 for ; Wed, 21 Oct 2015 11:29:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sippysoft_com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=MXOZl1V1oCB7mgb7MHKxVMwvXyWWINsCwjbD7Syf7Cs=; b=fgCmn+i83gnR8Ru7fz092IYQUklyPdPPe3JCfx+/urXhpw5mMrmoYsK3pTms+Q49Co s0rL0o4kspp+fisTO76Z/rFd2bLHKQM6epeYZq8OO8Kcat5pMH5/jIjUd0E6nq0om5hb tNY80xHwiybC2C2sgIZ6iEf85HXYfh1SPLb1xWxwjwo2WCO8K9LRF8O+Zmad8SaJK1wo m4TVyrd5VLH905e5RGGufEiANYmmKh29N1RTVCu4FLevHGqIhmo7MVxl+x/RavTo5K8J 8UI0beiWpw/4Q6b3e6WVvOuEdEc2R/s7Jy5axexV+fzev+jOpwJzBBL9oXCZ/TiG3XRS vKGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=MXOZl1V1oCB7mgb7MHKxVMwvXyWWINsCwjbD7Syf7Cs=; b=FSA3tRGdHLKtbHduN4WQ52Hp71yGRnorX1MuM1I3ZOf79Cm4Ru95kkuj8c3X+VZAlp n96h4PffGnMXXAcf0jmToAkYt8Ut8G4AZLJzEQji5ayumH/m0nG8Co7sqqrDOvAl9b50 vX/gRZXnplpQ5ru4vgHfA/3LiQjqzg3k5Sm5qHQ7bCdDsML8eJ6Ve+MT/dtwvxO8IGF1 lSyp6Sf7aCocHLiH6DGCRM/YigwEbnRVPhxV1c8fMjwGyAtntPxoCAevdYYRWdav5qdz yT3TmU9L8PcJGxX7vYpchWi7is2ZgRfQVRWOb9XmuI0zA5CDLJPJqkcoYWrpxf1suwRC aJ1A== X-Gm-Message-State: ALoCoQniyY/jPMtpvs08xyPPX/7Q+J3zwplB6/x7MOm7D5ZYoNBj2Aa4bNLOqIJ7Epym7oOgCQ2t MIME-Version: 1.0 X-Received: by 10.194.222.227 with SMTP id qp3mr14337874wjc.36.1445452157208; Wed, 21 Oct 2015 11:29:17 -0700 (PDT) Received: by 10.27.11.228 with HTTP; Wed, 21 Oct 2015 11:29:17 -0700 (PDT) In-Reply-To: <2050066.YiOUntKNB6@ralph.baldwin.cx> References: <1608354.LQmTMSsd5C@ralph.baldwin.cx> <2050066.YiOUntKNB6@ralph.baldwin.cx> Date: Wed, 21 Oct 2015 11:29:17 -0700 Message-ID: Subject: Re: Some MSI are not routed correctly From: Maxim Sobolev To: John Baldwin Cc: FreeBSD Net Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Oct 2015 18:29:19 -0000 Yes, I do. However, please note that for some reason they are not using nearly as much CPU time as the other 4 for some reason. 11 root -92 - 0K 1104K WAIT 3 95.3H 28.96% intr{irq267: igb0:que} 11 root -92 - 0K 1104K WAIT 1 95.5H 24.41% intr{irq265: igb0:que} 11 root -92 - 0K 1104K CPU2 2 95.2H 23.73% intr{irq266: igb0:que} 11 root -92 - 0K 1104K WAIT 0 95.2H 23.05% intr{irq264: igb0:que} 11 root -92 - 0K 1104K WAIT 6 286:37 1.12% intr{irq271: igb1:que} 11 root -92 - 0K 1104K WAIT 7 278:05 1.12% intr{irq272: igb1:que} 11 root -92 - 0K 1104K WAIT 5 284:26 1.07% intr{irq270: igb1:que} 11 root -92 - 0K 1104K WAIT 4 290:41 0.98% intr{irq269: igb1:que} CPU 0: 0.0% user, 0.0% nice, 0.9% system, 24.9% interrupt, 74.2% idle CPU 1: 0.5% user, 0.0% nice, 0.0% system, 26.3% interrupt, 73.2% idle CPU 2: 0.0% user, 0.0% nice, 1.4% system, 25.4% interrupt, 73.2% idle CPU 3: 0.0% user, 0.0% nice, 0.5% system, 23.9% interrupt, 75.6% idle CPU 4: 0.9% user, 0.0% nice, 2.3% system, 2.3% interrupt, 94.4% idle CPU 5: 1.4% user, 0.0% nice, 4.2% system, 4.2% interrupt, 90.1% idle CPU 6: 1.4% user, 0.0% nice, 3.8% system, 1.4% interrupt, 93.4% idle CPU 7: 2.8% user, 0.0% nice, 0.0% system, 3.8% interrupt, 93.4% idle 34263 igb0:que 0 32308 igb0:que 1 35022 igb0:que 2 34593 igb0:que 3 14931 igb1:que 0 13059 igb1:que 1 12971 igb1:que 2 13032 igb1:que 3 So I guess interrupts are routed correctly after all, but for some reason driver takes some 5 times less time to process it on cpus 4-7 (per-interrupt). Weird. On Wed, Oct 21, 2015 at 10:41 AM, John Baldwin wrote: > On Tuesday, October 20, 2015 06:31:47 PM Maxim Sobolev wrote: > > Here you go: > > > > $ sudo procstat -S 11 > > PID TID COMM TDNAME CPU CSID CPU MASK > > 11 100082 intr irq269: igb1:que 4 1 4 > > 11 100084 intr irq270: igb1:que 5 1 5 > > 11 100086 intr irq271: igb1:que 6 1 6 > > 11 100088 intr irq272: igb1:que 7 1 7 > > These are clearly what you want, and you can see that the last CPU they > ran on is the CPU you want as well. If you run 'top -SHz' do you see > the threads running on other CPUs? > > -- > John Baldwin > > -- Maksym Sobolyev Sippy Software, Inc. Internet Telephony (VoIP) Experts Tel (Canada): +1-778-783-0474 Tel (Toll-Free): +1-855-747-7779 Fax: +1-866-857-6942 Web: http://www.sippysoft.com MSN: sales@sippysoft.com Skype: SippySoft From owner-freebsd-net@freebsd.org Wed Oct 21 18:44:04 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 51B1BA1AB7E for ; Wed, 21 Oct 2015 18:44:04 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2DB0A1F2C for ; Wed, 21 Oct 2015 18:44:04 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from ralph.baldwin.cx (c-73-231-226-104.hsd1.ca.comcast.net [73.231.226.104]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 1AD21B97D; Wed, 21 Oct 2015 14:44:03 -0400 (EDT) From: John Baldwin To: Maxim Sobolev Cc: FreeBSD Net Subject: Re: Some MSI are not routed correctly Date: Wed, 21 Oct 2015 11:31:49 -0700 Message-ID: <206504538.Si4KCGV8IQ@ralph.baldwin.cx> User-Agent: KMail/4.14.3 (FreeBSD/10.2-PRERELEASE; KDE/4.14.3; amd64; ; ) In-Reply-To: References: <2050066.YiOUntKNB6@ralph.baldwin.cx> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Wed, 21 Oct 2015 14:44:03 -0400 (EDT) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Oct 2015 18:44:04 -0000 On Wednesday, October 21, 2015 11:29:17 AM Maxim Sobolev wrote: > Yes, I do. However, please note that for some reason they are not using > nearly as much CPU time as the other 4 for some reason. > > 11 root -92 - 0K 1104K WAIT 3 95.3H 28.96% > intr{irq267: igb0:que} > 11 root -92 - 0K 1104K WAIT 1 95.5H 24.41% > intr{irq265: igb0:que} > 11 root -92 - 0K 1104K CPU2 2 95.2H 23.73% > intr{irq266: igb0:que} > 11 root -92 - 0K 1104K WAIT 0 95.2H 23.05% > intr{irq264: igb0:que} > 11 root -92 - 0K 1104K WAIT 6 286:37 1.12% > intr{irq271: igb1:que} > 11 root -92 - 0K 1104K WAIT 7 278:05 1.12% > intr{irq272: igb1:que} > 11 root -92 - 0K 1104K WAIT 5 284:26 1.07% > intr{irq270: igb1:que} > 11 root -92 - 0K 1104K WAIT 4 290:41 0.98% > intr{irq269: igb1:que} > > CPU 0: 0.0% user, 0.0% nice, 0.9% system, 24.9% interrupt, 74.2% idle > CPU 1: 0.5% user, 0.0% nice, 0.0% system, 26.3% interrupt, 73.2% idle > CPU 2: 0.0% user, 0.0% nice, 1.4% system, 25.4% interrupt, 73.2% idle > CPU 3: 0.0% user, 0.0% nice, 0.5% system, 23.9% interrupt, 75.6% idle > CPU 4: 0.9% user, 0.0% nice, 2.3% system, 2.3% interrupt, 94.4% idle > CPU 5: 1.4% user, 0.0% nice, 4.2% system, 4.2% interrupt, 90.1% idle > CPU 6: 1.4% user, 0.0% nice, 3.8% system, 1.4% interrupt, 93.4% idle > CPU 7: 2.8% user, 0.0% nice, 0.0% system, 3.8% interrupt, 93.4% idle > > 34263 igb0:que 0 > 32308 igb0:que 1 > 35022 igb0:que 2 > 34593 igb0:que 3 > 14931 igb1:que 0 > 13059 igb1:que 1 > 12971 igb1:que 2 > 13032 igb1:que 3 > > So I guess interrupts are routed correctly after all, but for some reason > driver takes some 5 times less time to process it on cpus 4-7 > (per-interrupt). Weird. Are the pps rates the same? It seems like the interrupt rates on igb0 are double those of igb1? -- John Baldwin From owner-freebsd-net@freebsd.org Wed Oct 21 19:07:37 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 21FAEA1B20A for ; Wed, 21 Oct 2015 19:07:37 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-wi0-x236.google.com (mail-wi0-x236.google.com [IPv6:2a00:1450:400c:c05::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B5A57FA3 for ; Wed, 21 Oct 2015 19:07:36 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by wicfx6 with SMTP id fx6so105267110wic.1 for ; Wed, 21 Oct 2015 12:07:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sippysoft_com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=CrUiBbKYG60OKz11QpwUNuJJ78PNnyX2qcB2L7G5RyA=; b=x2EmBMuDI/vIppuTtH0j8nBhCM5hMsDf3VHBv7XaPmtaq2bwU1B2Y4Sxuc1PlEOmi3 KNFXu5S13BKZFMZ52N5DZNelP4sFQCQhv4BX4CCnfSaXn9mT8PIsMc/B0WqZ7GVrxMty e7Mpid/yMAYJsKYsLw2hWlxRqTyCJxscZGlEn190hLNiqYdTKIWyK73sAhm/dExy8eIz GADGjZmHtHHJ1wW0JxfFhDpAWzRHln3xtoIwTVUjB3rFhJ/tXrXkTyCbmOwaUlRw3/61 wI5KM/dnzkcYoWfTktTUN4ymmKRCRWXu10ir23DoJ8n4nrquJHKKF1QcNzg5Ccw8Af1y BuWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=CrUiBbKYG60OKz11QpwUNuJJ78PNnyX2qcB2L7G5RyA=; b=ZVKZTeHE3+EIQ18HbF0tbWOBwTU9nTgOTqgivfOUoiALZEhuodexLPQJCpcn2sVqrO 0mJ5Ns62UikyKeiGgG8juSZuI1HNmEeOvjViknbBot1dQ3HV0yHzJH45JuuQbcDOhTVN N0wrnlN1vpVCpOolvK+LcQ2G2gU/g8rpqGRfSjqWzFupuUQTVpPEWZi8Rb6clyKIKDzq /RziS6H7swxbFbtdwoL9woO2tsV/+L9A+oZGvdguZgrIYhxnMROoOvI9udJetSgFgBEa SWdM+OcS55u13HvhnzJJzSvTz5E2UuP626m+TW+zYeTVkzzHMczJGYGZ0X5vrjo9OsdD 22dQ== X-Gm-Message-State: ALoCoQlgbCj/pAktmIPR7Mx5wXZD6BYfdqbkuyNVMiNjfCqGMk7ulJARI7GvVz5lW12pty3ADYhZ MIME-Version: 1.0 X-Received: by 10.180.99.8 with SMTP id em8mr12345288wib.8.1445454454904; Wed, 21 Oct 2015 12:07:34 -0700 (PDT) Sender: sobomax@sippysoft.com Received: by 10.27.11.228 with HTTP; Wed, 21 Oct 2015 12:07:34 -0700 (PDT) In-Reply-To: <206504538.Si4KCGV8IQ@ralph.baldwin.cx> References: <2050066.YiOUntKNB6@ralph.baldwin.cx> <206504538.Si4KCGV8IQ@ralph.baldwin.cx> Date: Wed, 21 Oct 2015 12:07:34 -0700 X-Google-Sender-Auth: dmlnxWj_HtG2PhncuEEFAptbh2o Message-ID: Subject: Re: Some MSI are not routed correctly From: Maxim Sobolev To: John Baldwin Cc: FreeBSD Net Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Oct 2015 19:07:37 -0000 Oh, bingo! Just checked packets count and found the following: $ sysctl -a | grep dev.igb.1.queue | grep packets dev.igb.1.queue3.rx_packets: 2997 dev.igb.1.queue3.tx_packets: 21045801676 dev.igb.1.queue2.rx_packets: 3084 dev.igb.1.queue2.tx_packets: 21265692009 dev.igb.1.queue1.rx_packets: 3016 dev.igb.1.queue1.tx_packets: 21496134503 dev.igb.1.queue0.rx_packets: 48868 dev.igb.1.queue0.tx_packets: 21729900371 $ sysctl -a | grep dev.igb.0.queue | grep packets dev.igb.0.queue3.rx_packets: 40760861870 dev.igb.0.queue3.tx_packets: 21068449957 dev.igb.0.queue2.rx_packets: 40724698310 dev.igb.0.queue2.tx_packets: 21288469372 dev.igb.0.queue1.rx_packets: 40739376158 dev.igb.0.queue1.tx_packets: 21519768656 dev.igb.0.queue0.rx_packets: 40602824141 dev.igb.0.queue0.tx_packets: 21754065014 Apparently all incoming packets are going through igb0, while outbound get distributed. This means the upstream switch is not doing proper load balancing between two ports. We'll take it to the DC to fix. Thanks John, for helping to drill that down! On Wed, Oct 21, 2015 at 11:31 AM, John Baldwin wrote: > On Wednesday, October 21, 2015 11:29:17 AM Maxim Sobolev wrote: > > Yes, I do. However, please note that for some reason they are not using > > nearly as much CPU time as the other 4 for some reason. > > > > 11 root -92 - 0K 1104K WAIT 3 95.3H 28.96% > > intr{irq267: igb0:que} > > 11 root -92 - 0K 1104K WAIT 1 95.5H 24.41% > > intr{irq265: igb0:que} > > 11 root -92 - 0K 1104K CPU2 2 95.2H 23.73% > > intr{irq266: igb0:que} > > 11 root -92 - 0K 1104K WAIT 0 95.2H 23.05% > > intr{irq264: igb0:que} > > 11 root -92 - 0K 1104K WAIT 6 286:37 1.12% > > intr{irq271: igb1:que} > > 11 root -92 - 0K 1104K WAIT 7 278:05 1.12% > > intr{irq272: igb1:que} > > 11 root -92 - 0K 1104K WAIT 5 284:26 1.07% > > intr{irq270: igb1:que} > > 11 root -92 - 0K 1104K WAIT 4 290:41 0.98% > > intr{irq269: igb1:que} > > > > CPU 0: 0.0% user, 0.0% nice, 0.9% system, 24.9% interrupt, 74.2% idle > > CPU 1: 0.5% user, 0.0% nice, 0.0% system, 26.3% interrupt, 73.2% idle > > CPU 2: 0.0% user, 0.0% nice, 1.4% system, 25.4% interrupt, 73.2% idle > > CPU 3: 0.0% user, 0.0% nice, 0.5% system, 23.9% interrupt, 75.6% idle > > CPU 4: 0.9% user, 0.0% nice, 2.3% system, 2.3% interrupt, 94.4% idle > > CPU 5: 1.4% user, 0.0% nice, 4.2% system, 4.2% interrupt, 90.1% idle > > CPU 6: 1.4% user, 0.0% nice, 3.8% system, 1.4% interrupt, 93.4% idle > > CPU 7: 2.8% user, 0.0% nice, 0.0% system, 3.8% interrupt, 93.4% idle > > > > 34263 igb0:que 0 > > 32308 igb0:que 1 > > 35022 igb0:que 2 > > 34593 igb0:que 3 > > 14931 igb1:que 0 > > 13059 igb1:que 1 > > 12971 igb1:que 2 > > 13032 igb1:que 3 > > > > So I guess interrupts are routed correctly after all, but for some reason > > driver takes some 5 times less time to process it on cpus 4-7 > > (per-interrupt). Weird. > > Are the pps rates the same? It seems like the interrupt rates on igb0 > are double those of igb1? > > -- > John Baldwin > From owner-freebsd-net@freebsd.org Thu Oct 22 07:39:22 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0DB3AA1B10E for ; Thu, 22 Oct 2015 07:39:22 +0000 (UTC) (envelope-from lars@netapp.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id DB1DB158F for ; Thu, 22 Oct 2015 07:39:21 +0000 (UTC) (envelope-from lars@netapp.com) Received: by mailman.ysv.freebsd.org (Postfix) id D7E20A1B10C; Thu, 22 Oct 2015 07:39:21 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D6835A1B10B for ; Thu, 22 Oct 2015 07:39:21 +0000 (UTC) (envelope-from lars@netapp.com) Received: from mx141.netapp.com (mx141.netapp.com [216.240.21.12]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "mx141.netapp.com", Issuer "Symantec Class 3 Secure Server CA - G4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 80D50158C; Thu, 22 Oct 2015 07:39:21 +0000 (UTC) (envelope-from lars@netapp.com) X-IronPort-AV: E=Sophos;i="5.20,181,1444719600"; d="asc'?scan'208";a="76437850" Received: from hioexcmbx01-prd.hq.netapp.com ([10.122.105.34]) by mx141-out.netapp.com with ESMTP; 22 Oct 2015 00:39:00 -0700 Received: from HIOEXCMBX07-PRD.hq.netapp.com (10.122.105.40) by hioexcmbx01-prd.hq.netapp.com (10.122.105.34) with Microsoft SMTP Server (TLS) id 15.0.1104.5; Thu, 22 Oct 2015 00:38:59 -0700 Received: from HIOEXCMBX07-PRD.hq.netapp.com ([::1]) by hioexcmbx07-prd.hq.netapp.com ([fe80::e1d9:911e:3048:d510%21]) with mapi id 15.00.1104.000; Thu, 22 Oct 2015 00:38:59 -0700 From: "Eggert, Lars" To: hiren panchasara CC: "erj@FreeBSD.org" , "net@freebsd.org" Subject: Re: ixl 40G bad performance? Thread-Topic: ixl 40G bad performance? Thread-Index: AQHRCnVeClivuZWSFkK4WCM/ynkiH55zUwOAgAAMKACAAAITgIAA5BEAgAA8ooCAACxPAIAAP6UAgAFpzQCAAB5sgIAADIMAgAAWWYCAAQDgAA== Date: Thu, 22 Oct 2015 07:38:59 +0000 Message-ID: References: <20151020190541.B15983@sola.nimnet.asn.au> <6CD6754D-FC0E-4B24-AAEC-7C9D68284141@netapp.com> <20151020232218.G1833@besplex.bde.org> <12938D48-44B9-456A-AD7C-630D7E09BB35@netapp.com> <99F14384-271F-4543-A7F8-E19F4006998A@netapp.com> <20151021161935.GF28288@strugglingcoder.info> In-Reply-To: <20151021161935.GF28288@strugglingcoder.info> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-mailer: Apple Mail (2.3096.5) x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [10.122.56.79] Content-Type: multipart/signed; boundary="Apple-Mail=_63C5D3A6-200E-4EC8-9021-A0EDB1CFB4AD"; protocol="application/pgp-signature"; micalg=pgp-sha256 MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Oct 2015 07:39:22 -0000 --Apple-Mail=_63C5D3A6-200E-4EC8-9021-A0EDB1CFB4AD Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi, for those of you following along, I did try jumbograms and throughput = increases roughly 5x. So it looks like I'm hitting a packet-rate limit = somewhere. Lars --Apple-Mail=_63C5D3A6-200E-4EC8-9021-A0EDB1CFB4AD Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="signature.asc" Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- iQCVAwUBViiSktZcnpRveo1xAQhQ1AP+L/xwOa1SJ71KEmMT5snufIG77uqPHp+p T96vspaI9h+Gm0PUxTDAFFei6r48SzQqi0B9juDWyL0/6aGkoUdZJkWDgZBKiLdY cCa9L+BwzOqQJ8/Hzzrkod2XlaOV9eLO2nmxM1649v6QBXYcnd7ScY2DMvMj4xH3 AuxDFiBP1IA= =OeXA -----END PGP SIGNATURE----- --Apple-Mail=_63C5D3A6-200E-4EC8-9021-A0EDB1CFB4AD-- From owner-freebsd-net@freebsd.org Thu Oct 22 08:22:32 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 89098A1BBDB for ; Thu, 22 Oct 2015 08:22:32 +0000 (UTC) (envelope-from lars@netapp.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 616721B12 for ; Thu, 22 Oct 2015 08:22:32 +0000 (UTC) (envelope-from lars@netapp.com) Received: by mailman.ysv.freebsd.org (Postfix) id 5F969A1BBDA; Thu, 22 Oct 2015 08:22:32 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5E518A1BBD8 for ; Thu, 22 Oct 2015 08:22:32 +0000 (UTC) (envelope-from lars@netapp.com) Received: from mx142.netapp.com (mx142.netapp.com [216.240.21.19]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "mx142.netapp.com", Issuer "Symantec Class 3 Secure Server CA - G4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0A9151B10; Thu, 22 Oct 2015 08:22:30 +0000 (UTC) (envelope-from lars@netapp.com) X-IronPort-AV: E=Sophos;i="5.20,181,1444719600"; d="asc'?scan'208";a="72286764" Received: from hioexcmbx04-prd.hq.netapp.com ([10.122.105.37]) by mx142-out.netapp.com with ESMTP; 22 Oct 2015 01:21:20 -0700 Received: from HIOEXCMBX07-PRD.hq.netapp.com (10.122.105.40) by hioexcmbx04-prd.hq.netapp.com (10.122.105.37) with Microsoft SMTP Server (TLS) id 15.0.1104.5; Thu, 22 Oct 2015 01:21:19 -0700 Received: from HIOEXCMBX07-PRD.hq.netapp.com ([::1]) by hioexcmbx07-prd.hq.netapp.com ([fe80::e1d9:911e:3048:d510%21]) with mapi id 15.00.1104.000; Thu, 22 Oct 2015 01:21:19 -0700 From: "Eggert, Lars" To: hiren panchasara CC: "erj@FreeBSD.org" , "net@freebsd.org" Subject: Re: ixl 40G bad performance? Thread-Topic: ixl 40G bad performance? Thread-Index: AQHRCnVeClivuZWSFkK4WCM/ynkiH55zUwOAgAAMKACAAAITgIAA5BEAgAA8ooCAACxPAIAAP6UAgAFpzQCAAB5sgIAADIMAgAAWWYCAAQDgAIAAC9QA Date: Thu, 22 Oct 2015 08:21:19 +0000 Message-ID: <25588FF3-E5A7-47A7-990C-72740F30B654@netapp.com> References: <20151020190541.B15983@sola.nimnet.asn.au> <6CD6754D-FC0E-4B24-AAEC-7C9D68284141@netapp.com> <20151020232218.G1833@besplex.bde.org> <12938D48-44B9-456A-AD7C-630D7E09BB35@netapp.com> <99F14384-271F-4543-A7F8-E19F4006998A@netapp.com> <20151021161935.GF28288@strugglingcoder.info> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-mailer: Apple Mail (2.3096.5) x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [10.122.56.79] Content-Type: multipart/signed; boundary="Apple-Mail=_B7DB74BD-B267-4025-B452-665EDBF0AEAE"; protocol="application/pgp-signature"; micalg=pgp-sha256 MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Oct 2015 08:22:32 -0000 --Apple-Mail=_B7DB74BD-B267-4025-B452-665EDBF0AEAE Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii On 2015-10-22, at 9:38, Eggert, Lars wrote: > for those of you following along, I did try jumbograms and throughput = increases roughly 5x. So it looks like I'm hitting a packet-rate limit = somewhere. Does the ixl driver have an issue with TSO/LRO? If I tcpdump on the receiver when testing the 10G ix interfaces, I see = that most "packets" are up to 64KB in the traces on both sender and = receiver, which is expected with TSO/LRO. When I look at the traffic over the ixl interfaces, I see that most = "packets" on the sender are much smaller (~2896 aka 2 segments; although = some few are >40K). On the receiver, I only see 1448 byte packets. Lars --Apple-Mail=_B7DB74BD-B267-4025-B452-665EDBF0AEAE Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="signature.asc" Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- iQCUAwUBViicftZcnpRveo1xAQhJTgP4jHc0k76zbXDySTBQCB4ABoDKD60Pxsaf zlMzTxdu7it9oZhj1TQA2Qew7Xi6NtU6TJgo11nRcaztioXTQ0MMT8UNI5V5fbkQ bOBv1HFQNPiqA8tgnVUO+ZWSTbUNUXzyvYYmfhxQJbfL5QnaWPSGuphZj94yVoTU 1gdcj7/irg== =MpJK -----END PGP SIGNATURE----- --Apple-Mail=_B7DB74BD-B267-4025-B452-665EDBF0AEAE-- From owner-freebsd-net@freebsd.org Thu Oct 22 09:58:52 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1CD56A1CDCB for ; Thu, 22 Oct 2015 09:58:52 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from smtp.hungerhost.com (smtp.hungerhost.com [216.38.53.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id ED63A1A9F for ; Thu, 22 Oct 2015 09:58:51 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from [38.127.159.5] (port=49445 helo=[10.70.193.138]) by vps.hungerhost.com with esmtpsa (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.86) (envelope-from ) id 1ZpCe0-0004P1-3d; Thu, 22 Oct 2015 05:58:48 -0400 From: "George Neville-Neil" To: "Nigel Williams" Cc: freebsd-net@freebsd.org Subject: Re: MPTCP for FreeBSD repository on BitBucket/v0.51 update Date: Thu, 22 Oct 2015 05:58:45 -0400 Message-ID: <137EE92F-7611-402D-874F-37AA627F41CA@neville-neil.com> In-Reply-To: <562466AA.7020707@swin.edu.au> References: <562466AA.7020707@swin.edu.au> MIME-Version: 1.0 Content-Type: text/plain; format=flowed X-Mailer: MailMate (1.9.2r5141) X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - vps.hungerhost.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - neville-neil.com X-Get-Message-Sender-Via: vps.hungerhost.com: authenticated_id: gnn@neville-neil.com X-Authenticated-Sender: vps.hungerhost.com: gnn@neville-neil.com X-Source: X-Source-Args: X-Source-Dir: X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Oct 2015 09:58:52 -0000 On 18 Oct 2015, at 23:42, Nigel Williams wrote: > Hi, > > The MPTCP code is now available as a mercurial repository: > - Repository: https://bitbucket.org/nw-swin/caia-mptcp-freebsd > - Wiki: https://bitbucket.org/nw-swin/caia-mptcp-freebsd/wiki/ > > For those interested in trying the implementation/looking at the code, > this should hopefully make the process a little easier (and save > having to patch in updates). It should also make it possible to > contribute code for those wishing to do so. > > Some details: > - Has been branched off 'freebsd-head' at > 'http://hg-beta.freebsd.org/base', and will be merged on a weekly > basis. > - I will be working off this repository so it will be up-to-date with > recent changes. > - In place of patch releases, release versions will now be tagged. > - I'll also start to populate the 'Issues' section so that there is a > better picture of current bugs/things TBD. > > The version has also been updated to v0.51. See: > - http://caia.swin.edu.au/newtcp/mptcp/tools.html > - OR https://bitbucket.org/nw-swin/caia-mptcp-freebsd/wiki/Home > > Functionally-wise this hasn't changed from the previous version, but > has been merged with a recent revision of head. > Very nice! Just wondering how you're testing this out. I've been working on a lot of networking tests and I'm sure MPTCP introduces some interesting complications. Thanks, George From owner-freebsd-net@freebsd.org Thu Oct 22 14:27:35 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D19B8A1C8C3 for ; Thu, 22 Oct 2015 14:27:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BEB1B1B02 for ; Thu, 22 Oct 2015 14:27:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9MERZKb035252 for ; Thu, 22 Oct 2015 14:27:35 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 203916] ethernet and wlan interfaces both have the same mac-address after upgrade to r289486 Date: Thu, 22 Oct 2015 14:27:35 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: shuriku@shurik.kiev.ua X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Oct 2015 14:27:35 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203916 --- Comment #3 from Alexandr Krivulya --- After complete power down laptop and removing battery re0 restored its original mac and network is working now. But it stop working when re0 becomes a lagg member and problem with fake mac-address repeats again until power down. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-net@freebsd.org Thu Oct 22 18:34:13 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7A54FA1CB3F for ; Thu, 22 Oct 2015 18:34:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 65DD8186F for ; Thu, 22 Oct 2015 18:34:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9MIYDAp048273 for ; Thu, 22 Oct 2015 18:34:13 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 202983] ixv driver in 11.0-CURRENT(10.1 & 10.2 RELEASE) doesn't pass traffic using XEN hypervisor(AWS EC2) Date: Thu, 22 Oct 2015 18:34:13 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: IntelNetworking X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: jeffrey.e.pieper@intel.com X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Oct 2015 18:34:13 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=202983 Jeff Pieper changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jeffrey.e.pieper@intel.com --- Comment #1 from Jeff Pieper --- We are able to reproduce this, but we will need logs from the host during the time of the failure. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-net@freebsd.org Thu Oct 22 18:43:20 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 35B89A1CD3F for ; Thu, 22 Oct 2015 18:43:20 +0000 (UTC) (envelope-from guru@unixarea.de) Received: from ms-10.1blu.de (ms-10.1blu.de [178.254.4.101]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EC7471D71 for ; Thu, 22 Oct 2015 18:43:19 +0000 (UTC) (envelope-from guru@unixarea.de) Received: from [89.204.135.136] (helo=localhost.unixarea.de) by ms-10.1blu.de with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.76) (envelope-from ) id 1ZpKpQ-0000mR-LZ for freebsd-net@freebsd.org; Thu, 22 Oct 2015 20:43:08 +0200 Received: from localhost.my.domain (c720-r276659 [127.0.0.1]) by localhost.unixarea.de (8.14.9/8.14.9) with ESMTP id t9MIh7f7002106 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 22 Oct 2015 20:43:07 +0200 (CEST) (envelope-from guru@unixarea.de) Received: (from guru@localhost) by localhost.my.domain (8.14.9/8.14.9/Submit) id t9MIh7a6002105 for freebsd-net@freebsd.org; Thu, 22 Oct 2015 20:43:07 +0200 (CEST) (envelope-from guru@unixarea.de) X-Authentication-Warning: localhost.my.domain: guru set sender to guru@unixarea.de using -f Date: Thu, 22 Oct 2015 20:43:07 +0200 From: Matthias Apitz To: freebsd-net@freebsd.org Subject: Ubuntu-phone BQ as Wifi router to the Internet Message-ID: <20151022184307.GA2044@c720-r276659> Reply-To: Matthias Apitz Mail-Followup-To: Matthias Apitz , freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit X-Operating-System: FreeBSD 11.0-CURRENT r269739 (i386) User-Agent: Mutt/1.5.23 (2014-03-12) X-Con-Id: 51246 X-Con-U: 0-guru X-Originating-IP: 89.204.135.136 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Oct 2015 18:43:20 -0000 Hello, I have a mobile phone, a Ubuntu BQ which works really nice as any kind of Linux box, SSH access etc. (not like this Android crap). At the moment I'm using my BQ as a router to the Internet as described here: https://gurucubano.gitbooks.io/bq-aquaris-e-4-5-ubuntu-phone/content/chapter4.html i.e. via USB and tethering: FreeBSD-netbook ---(USB tethering)---> BQ ---(data mobile) ---> Internet I'd like to get rid of the USB cable and I'm thinking about the following: My FreeBSD C720 netbook acts as an AP, the BQ connects, but all the routing and DNS is scripted the way that the traffic from the netbook goes over Wifi to the BQ and from this to Internet; I investigated the option and technical it seems to work once changed routing etc. in the BQ; Any comments? Thanks matthias -- Matthias Apitz, ✉ guru@unixarea.de, 🌐 http://www.unixarea.de/ ☎ +49-176-38902045 From owner-freebsd-net@freebsd.org Thu Oct 22 22:08:11 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C27E2A1C756 for ; Thu, 22 Oct 2015 22:08:11 +0000 (UTC) (envelope-from lists@jnielsen.net) Received: from webmail2.jnielsen.net (webmail2.jnielsen.net [50.114.224.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "webmail2.jnielsen.net", Issuer "freebsdsolutions.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 9E9FC6CA for ; Thu, 22 Oct 2015 22:08:11 +0000 (UTC) (envelope-from lists@jnielsen.net) Received: from [10.10.1.196] (office.betterlinux.com [199.58.199.60]) (authenticated bits=0) by webmail2.jnielsen.net (8.15.1/8.15.1) with ESMTPSA id t9MM86g0018658 (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 22 Oct 2015 16:08:10 -0600 (MDT) (envelope-from lists@jnielsen.net) X-Authentication-Warning: webmail2.jnielsen.net: Host office.betterlinux.com [199.58.199.60] claimed to be [10.10.1.196] Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.1 \(3096.5\)) Subject: Re: Ubuntu-phone BQ as Wifi router to the Internet From: John Nielsen In-Reply-To: <20151022184307.GA2044@c720-r276659> Date: Thu, 22 Oct 2015 16:08:06 -0600 Cc: freebsd-net@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <29547EF6-8485-4357-A4AF-89BB1C2695BD@jnielsen.net> References: <20151022184307.GA2044@c720-r276659> To: Matthias Apitz X-Mailer: Apple Mail (2.3096.5) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Oct 2015 22:08:11 -0000 On Oct 22, 2015, at 12:43 PM, Matthias Apitz wrote: > I have a mobile phone, a Ubuntu BQ which works really nice as any kind > of Linux box, SSH access etc. (not like this Android crap). >=20 > At the moment I'm using my BQ as a router to the Internet as described > here: = https://gurucubano.gitbooks.io/bq-aquaris-e-4-5-ubuntu-phone/content/chapt= er4.html > i.e. via USB and tethering: >=20 > FreeBSD-netbook ---(USB tethering)---> BQ ---(data mobile) ---> = Internet >=20 > I'd like to get rid of the USB cable and I'm thinking about the > following: >=20 > My FreeBSD C720 netbook acts as an AP, the BQ connects, but all the = routing and > DNS is scripted the way that the traffic from the netbook goes over > Wifi to the BQ and from this to Internet; > I investigated the option and technical it seems to work once changed > routing etc. in the BQ; >=20 > Any comments? Should work fine with static IPs, etc on the wireless subnet. Using DHCP = will probably be problematic. From owner-freebsd-net@freebsd.org Thu Oct 22 23:08:40 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B2D34A1C63D for ; Thu, 22 Oct 2015 23:08:40 +0000 (UTC) (envelope-from alarig@swordarmor.fr) Received: from bulbizarre.swordarmor.fr (bulbizarre.swordarmor.fr [IPv6:2001:470:1f13:138:715d:2fa0:b591:532f]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6B63FD3B for ; Thu, 22 Oct 2015 23:08:40 +0000 (UTC) (envelope-from alarig@swordarmor.fr) Received: from drscott.swordarmor.fr (drscott-he.swordarmor.fr [IPv6:2001:470:1f13:138::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bulbizarre.swordarmor.fr (Postfix) with ESMTPS id DF4504EDF25 for ; Fri, 23 Oct 2015 01:08:38 +0200 (CEST) Authentication-Results: bulbizarre.swordarmor.fr; dkim=pass reason="1024-bit key" header.d=swordarmor.fr header.i=@swordarmor.fr header.b=fZzYY6ta; dkim-adsp=pass; dkim-atps=neutral Received: from drscott.swordarmor.fr (localhost [127.0.0.1]); by drscott.swordarmor.fr (OpenSMTPD) with ESMTP id be334b78; for ; Thu, 22 Oct 2015 23:08:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=simple; d=swordarmor.fr; h=date:from :to:subject:message-id:mime-version:content-type; s=default; bh= GAQOT+E9vSutKsnKZZroT3wXJLQ=; b=fZzYY6taWw78e1MytU2bLMfLK7vRd7+3 gbohIyxdCeQNHBsC4ALkGrwPnoMiYE3U7HB+oe5XBEY1QGe9XLuyNDNY0/KH/HFe BxE/CFmqbD66kOPpp4reN40hBNG9IT7hY/vxYtEVt9FvBU8EG2cc6W8/FjlmXAM4 xRi54oZJPIE= DomainKey-Signature: a=rsa-sha1; c=simple; d=swordarmor.fr; h=date:from :to:subject:message-id:mime-version:content-type; q=dns; s= default; b=jFoHv/SOjw8bIi9huF+cc0tnIQdjZ7fuK2PDzBuhh+zhyNehw4mJm LHe1cjeV6xmQwXyO8crkTvo2/d/cfDkPaPf+jaZTzzTcWwou/5w6NRnRA2M7iibv kB4CeGljjhRV86dx9c91ZFxKT13Kv5IzQF0/kb+cabla8X5ZVxe0NU= Received: from drscott.swordarmor.fr (localhost [IPv6:::1]); by drscott.swordarmor.fr (OpenSMTPD) with ESMTP id 904f644e; for ; Thu, 22 Oct 2015 23:08:38 +0000 (UTC) Date: Fri, 23 Oct 2015 01:08:38 +0200 From: Alarig Le lay To: freebsd-net@freebsd.org Subject: Cannot add an ipv6 route with -interface Message-ID: <20151022230838.GA22065@drscott.swordarmor.fr> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Re7H+V5lQR2Zv/pu" Content-Disposition: inline User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Oct 2015 23:08:40 -0000 --Re7H+V5lQR2Zv/pu Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, I try to add an IPv6 route on a 10.1 FreeBSD router by using the option -interface. It works with IPv4 but not with IPv6. In IPv4: alarig@nominoe:~ % ping 192.168.1.1 PING 192.168.1.1 (192.168.1.1): 56 data bytes 76 bytes from te0-0-2-2.rcr11.uro01.atlas.cogentco.com (130.117.1.242): Destination Net Unreachable Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 68 0054 0bea 0 0000 3e 01 d145 149.6.72.98 192.168.1.1=20 alarig@nominoe:~ % sudo route add 192.168.1.0/24 -interface em1 add net 192.168.1.0: gateway em1 alarig@nominoe:~ % ping 192.168.1.1 PING 192.168.1.1 (192.168.1.1): 56 data bytes 64 bytes from 192.168.1.1: icmp_seq=3D75 ttl=3D64 time=3D0.831 ms In IPv6: alarig@nominoe:~ % ping6 2a00:5884:8200::100 PING6(56=3D40+8+8 bytes) 2001:978:2:4e::5:2 --> 2a00:5884:8200::100 ^C --- 2a00:5884:8200::100 ping6 statistics --- 3 packets transmitted, 0 packets received, 100.0% packet loss alarig@nominoe:~ % sudo route add -6 2a00:5884:8200::/40 -interface em1 add net 2a00:5884:8200::/40: gateway em1 alarig@nominoe:~ % ping6 2a00:5884:8200::100 PING6(56=3D40+8+8 bytes) 2a00:5884::1 --> 2a00:5884:8200::100 ping6: sendmsg: Operation not permitted ping6: wrote 2a00:5884:8200::100 16 chars, ret=3D-1 This operation works on linux: alarig@judicael:~$ ping6 2a00:5884:8200::100 PING 2a00:5884:8200::100(2a00:5884:8200::100) 56 data bytes =46rom 2a00:5884::1 icmp_seq=3D1 Destination unreachable: Address unreachable =46rom 2a00:5884::1 icmp_seq=3D2 Destination unreachable: Address unreachable ^C --- 2a00:5884:8200::100 ping statistics --- 2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1007ms alarig@judicael:~$ sudo ip -6 route add 2a00:5884:8200::/40 dev eth0 alarig@judicael:~$ ping6 2a00:5884:8200::100 PING 2a00:5884:8200::100(2a00:5884:8200::100) 56 data bytes 64 bytes from 2a00:5884:8200::100: icmp_seq=3D1 ttl=3D63 time=3D1.41 ms 64 bytes from 2a00:5884:8200::100: icmp_seq=3D2 ttl=3D63 time=3D0.588 ms 64 bytes from 2a00:5884:8200::100: icmp_seq=3D3 ttl=3D63 time=3D0.459 ms ^C --- 2a00:5884:8200::100 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2000ms rtt min/avg/max/mdev =3D 0.459/0.819/1.410/0.421 ms It also works if I add an intermediate router: alarig@nominoe:~ % sudo route add -6 2a00:5884:8200::/40 2a00:5884::3 add net 2a00:5884:8200::/40: gateway 2a00:5884::3 alarig@nominoe:~ % ping6 2a00:5884:8200::100 PING6(56=3D40+8+8 bytes) 2a00:5884::1 --> 2a00:5884:8200::100 16 bytes from 2a00:5884:8200::100, icmp_seq=3D0 hlim=3D64 time=3D0.590 ms 16 bytes from 2a00:5884:8200::100, icmp_seq=3D1 hlim=3D64 time=3D0.699 ms 16 bytes from 2a00:5884:8200::100, icmp_seq=3D2 hlim=3D64 time=3D0.764 ms 16 bytes from 2a00:5884:8200::100, icmp_seq=3D3 hlim=3D64 time=3D0.806 ms ^C=20 --- 2a00:5884:8200::100 ping6 statistics --- 4 packets transmitted, 4 packets received, 0.0% packet loss round-trip min/avg/max/std-dev =3D 0.590/0.715/0.806/0.081 ms This router is theoretically useless as it is on the same network than the FreeBSD one. Do you have any idea about this issue? Did I make a mistake or is it a bug? Thanks, --=20 Alarig Le Lay --Re7H+V5lQR2Zv/pu Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJWKWx2AAoJEK84SsFrICuI5qEH/3PVrhKlaDV2IJV2F/10ravJ 5VKR0RwcNn6reZW+iSuEJ1oZx7FjKAfx7gIdILn+gtD7FrOZ8+TnbrRxmyOEtL8y dlFkWMsuM365V09H3Zvnr1cAKkvONacbIumgbO847z3W7J5Fq5CpXznN2T7NTDVJ tepeOu9cUjj/5d/UvotTg5aZvfYEGZs1IiIEPEI5WWlrbhRtYq4kGGFikNTQyTTj VLDaePTiAyHpcbLM3ju5/cEGkMP/FF23fm94Greb8g5UO/Cg7YYyatGGtBIEEr3Q /uHNQW3ETFtCFWJyMlX68Cx5iCYmRDvRYBePsFSOIqEVyf+t4PIAFL5OGroczHI= =rhq7 -----END PGP SIGNATURE----- --Re7H+V5lQR2Zv/pu-- From owner-freebsd-net@freebsd.org Thu Oct 22 23:34:17 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5D140A1CAC5 for ; Thu, 22 Oct 2015 23:34:17 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mail-ob0-x230.google.com (mail-ob0-x230.google.com [IPv6:2607:f8b0:4003:c01::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1AC04164E for ; Thu, 22 Oct 2015 23:34:17 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: by obbwb3 with SMTP id wb3so81289170obb.0 for ; Thu, 22 Oct 2015 16:34:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=hBE1Kk/j5PgXPrE49zep5rpKo4ueg6P686JrbU95E5g=; b=pTy0i/EBPlFPJ6RkXHRwy/g45yJajTWd3h4xr3tYZz70wP10ZWJBChgKVmn1FQYXFj bCsk0KeKBVLwCS2nqDyWBKFITFXp7nIHeJU5vMdnJ6W1JWjAoZmDbiPa1FxRlEq6Gmdk fXX6caOMxyBJLpgCzGT/V1Suaad/44FYluTuMoVm7zSg42ncZ3aZgPq4c8eo08nKxbq/ yFuiD8geRjUFetXjk+AI2TPZ37fn5838rD4sKZ9WaU9q+fSLUVCtCI9tXZWMK9c63kvn 62MentEMP3SZxJ6WIvyuZVw79kKqmeTN6Ky5WFp7xErGuPzFJQHJc9MxwgaylJ00FoEj CuUA== MIME-Version: 1.0 X-Received: by 10.60.41.9 with SMTP id b9mr12648831oel.37.1445556856336; Thu, 22 Oct 2015 16:34:16 -0700 (PDT) Sender: kob6558@gmail.com Received: by 10.202.50.136 with HTTP; Thu, 22 Oct 2015 16:34:16 -0700 (PDT) In-Reply-To: <20151022230838.GA22065@drscott.swordarmor.fr> References: <20151022230838.GA22065@drscott.swordarmor.fr> Date: Thu, 22 Oct 2015 16:34:16 -0700 X-Google-Sender-Auth: 82wgWscKGYmzYpAtusgJ60L_uVU Message-ID: Subject: Re: Cannot add an ipv6 route with -interface From: Kevin Oberman To: Alarig Le lay Cc: "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Oct 2015 23:34:17 -0000 On Thu, Oct 22, 2015 at 4:08 PM, Alarig Le lay wrote: > Hi, > > I try to add an IPv6 route on a 10.1 FreeBSD router by using the option > -interface. It works with IPv4 but not with IPv6. > > In IPv4: > alarig@nominoe:~ % ping 192.168.1.1 > PING 192.168.1.1 (192.168.1.1): 56 data bytes > 76 bytes from te0-0-2-2.rcr11.uro01.atlas.cogentco.com (130.117.1.242): > Destination Net Unreachable > Vr HL TOS Len ID Flg off TTL Pro cks Src Dst > 4 5 68 0054 0bea 0 0000 3e 01 d145 149.6.72.98 192.168.1.1 > alarig@nominoe:~ % sudo route add 192.168.1.0/24 -interface em1 > add net 192.168.1.0: gateway em1 > alarig@nominoe:~ % ping 192.168.1.1 > PING 192.168.1.1 (192.168.1.1): 56 data bytes > 64 bytes from 192.168.1.1: icmp_seq=75 ttl=64 time=0.831 ms > > In IPv6: > alarig@nominoe:~ % ping6 2a00:5884:8200::100 > PING6(56=40+8+8 bytes) 2001:978:2:4e::5:2 --> 2a00:5884:8200::100 > ^C > --- 2a00:5884:8200::100 ping6 statistics --- > 3 packets transmitted, 0 packets received, 100.0% packet loss > > alarig@nominoe:~ % sudo route add -6 2a00:5884:8200::/40 -interface em1 > add net 2a00:5884:8200::/40: gateway em1 > alarig@nominoe:~ % ping6 2a00:5884:8200::100 > PING6(56=40+8+8 bytes) 2a00:5884::1 --> 2a00:5884:8200::100 > ping6: sendmsg: Operation not permitted > ping6: wrote 2a00:5884:8200::100 16 chars, ret=-1 > > This operation works on linux: > alarig@judicael:~$ ping6 2a00:5884:8200::100 > PING 2a00:5884:8200::100(2a00:5884:8200::100) 56 data bytes > From 2a00:5884::1 icmp_seq=1 Destination unreachable: Address > unreachable > From 2a00:5884::1 icmp_seq=2 Destination unreachable: Address > unreachable > ^C > --- 2a00:5884:8200::100 ping statistics --- > 2 packets transmitted, 0 received, +2 errors, 100% packet loss, time > 1007ms > alarig@judicael:~$ sudo ip -6 route add 2a00:5884:8200::/40 dev eth0 > alarig@judicael:~$ ping6 2a00:5884:8200::100 > PING 2a00:5884:8200::100(2a00:5884:8200::100) 56 data bytes > 64 bytes from 2a00:5884:8200::100: icmp_seq=1 ttl=63 time=1.41 ms > 64 bytes from 2a00:5884:8200::100: icmp_seq=2 ttl=63 time=0.588 ms > 64 bytes from 2a00:5884:8200::100: icmp_seq=3 ttl=63 time=0.459 ms > ^C > --- 2a00:5884:8200::100 ping statistics --- > 3 packets transmitted, 3 received, 0% packet loss, time 2000ms > rtt min/avg/max/mdev = 0.459/0.819/1.410/0.421 ms > > It also works if I add an intermediate router: > alarig@nominoe:~ % sudo route add -6 2a00:5884:8200::/40 2a00:5884::3 > add net 2a00:5884:8200::/40: gateway 2a00:5884::3 > alarig@nominoe:~ % ping6 2a00:5884:8200::100 > PING6(56=40+8+8 bytes) 2a00:5884::1 --> 2a00:5884:8200::100 > 16 bytes from 2a00:5884:8200::100, icmp_seq=0 hlim=64 time=0.590 ms > 16 bytes from 2a00:5884:8200::100, icmp_seq=1 hlim=64 time=0.699 ms > 16 bytes from 2a00:5884:8200::100, icmp_seq=2 hlim=64 time=0.764 ms > 16 bytes from 2a00:5884:8200::100, icmp_seq=3 hlim=64 time=0.806 ms > ^C > --- 2a00:5884:8200::100 ping6 statistics --- > 4 packets transmitted, 4 packets received, 0.0% packet loss > round-trip min/avg/max/std-dev = 0.590/0.715/0.806/0.081 ms > > > This router is theoretically useless as it is on the same network than the > FreeBSD one. > > Do you have any idea about this issue? Did I make a mistake or is it a > bug? > > Thanks, > -- > Alarig Le Lay > Take a look at route(8). There are significant syntax differences. E.g. On FreeBSD route(8) has no "-6" option. It uses "-inet6". That may not be the only issue, but I need to get out the door to get to a hockey match. the rest is at least close. -- Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkoberman@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 From owner-freebsd-net@freebsd.org Fri Oct 23 00:13:47 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3E29DA1C51A for ; Fri, 23 Oct 2015 00:13:47 +0000 (UTC) (envelope-from hrs@FreeBSD.org) Received: from mail.allbsd.org (gatekeeper.allbsd.org [IPv6:2001:2f0:104:e001::32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.allbsd.org", Issuer "RapidSSL CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id BE23890C for ; Fri, 23 Oct 2015 00:13:46 +0000 (UTC) (envelope-from hrs@FreeBSD.org) Received: from alph.d.allbsd.org (alph.d.allbsd.org [IPv6:2001:2f0:104:e010:862b:2bff:febc:8956] (may be forged)) (authenticated bits=56) by mail.allbsd.org (8.14.9/8.14.9) with ESMTP id t9N0DX3G023628 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 23 Oct 2015 09:13:34 +0900 (JST) (envelope-from hrs@FreeBSD.org) Received: from localhost (localhost [IPv6:::1]) (authenticated bits=0) by alph.d.allbsd.org (8.15.2/8.14.9) with ESMTPA id t9N0DSZG014137; Fri, 23 Oct 2015 09:13:33 +0900 (JST) (envelope-from hrs@FreeBSD.org) Date: Fri, 23 Oct 2015 09:05:53 +0900 (JST) Message-Id: <20151023.090553.1908755081181502110.hrs@allbsd.org> To: alarig@swordarmor.fr Cc: freebsd-net@freebsd.org Subject: Re: Cannot add an ipv6 route with -interface From: Hiroki Sato In-Reply-To: <20151022230838.GA22065@drscott.swordarmor.fr> References: <20151022230838.GA22065@drscott.swordarmor.fr> X-PGPkey-fingerprint: BDB3 443F A5DD B3D0 A530 FFD7 4F2C D3D8 2793 CF2D X-Mailer: Mew version 6.7 on Emacs 24.5 / Mule 6.0 (HANACHIRUSATO) Mime-Version: 1.0 Content-Type: Multipart/Signed; protocol="application/pgp-signature"; micalg=pgp-sha1; boundary="--Security_Multipart(Fri_Oct_23_09_05_53_2015_791)--" Content-Transfer-Encoding: 7bit X-Virus-Scanned: clamav-milter 0.98.6 at gatekeeper.allbsd.org X-Virus-Status: Clean X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (mail.allbsd.org [IPv6:2001:2f0:104:e001::32]); Fri, 23 Oct 2015 09:13:39 +0900 (JST) X-Spam-Status: No, score=-98.0 required=13.0 tests=CONTENT_TYPE_PRESENT, RCVD_IN_AHBL, RCVD_IN_AHBL_PROXY, RCVD_IN_AHBL_SPAM, RDNS_NONE, USER_IN_WHITELIST autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on gatekeeper.allbsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Oct 2015 00:13:47 -0000 ----Security_Multipart(Fri_Oct_23_09_05_53_2015_791)-- Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Alarig Le lay wrote in <20151022230838.GA22065@drscott.swordarmor.fr>: al> Do you have any idea about this issue? Did I make a mistake or is it a al> bug? Is 2a00:5884::1/64 configured on em1? An address within 2a00:5884:8200::/40 is required on em1 in this case. -- Hiroki ----Security_Multipart(Fri_Oct_23_09_05_53_2015_791)-- Content-Type: application/pgp-signature Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iEYEABECAAYFAlYpeeEACgkQTyzT2CeTzy1QWQCffSiIiLstJbF0n7FguUgkF+4p 6FYAn1kQ5TH8nJnqouz0liMxeK6RMjlT =HqpS -----END PGP SIGNATURE----- ----Security_Multipart(Fri_Oct_23_09_05_53_2015_791)---- From owner-freebsd-net@freebsd.org Fri Oct 23 05:36:23 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 21B47A164D2 for ; Fri, 23 Oct 2015 05:36:23 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 03CC77A5 for ; Fri, 23 Oct 2015 05:36:23 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: by mailman.ysv.freebsd.org (Postfix) id 0142DA164D1; Fri, 23 Oct 2015 05:36:23 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F40FBA164D0 for ; Fri, 23 Oct 2015 05:36:22 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail109.syd.optusnet.com.au (mail109.syd.optusnet.com.au [211.29.132.80]) by mx1.freebsd.org (Postfix) with ESMTP id A7E0C7A3; Fri, 23 Oct 2015 05:36:22 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from c211-30-166-197.carlnfd1.nsw.optusnet.com.au (c211-30-166-197.carlnfd1.nsw.optusnet.com.au [211.30.166.197]) by mail109.syd.optusnet.com.au (Postfix) with ESMTPS id 31840D67561; Fri, 23 Oct 2015 16:36:09 +1100 (AEDT) Date: Fri, 23 Oct 2015 16:36:07 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Bruce Evans cc: "Eggert, Lars" , Ian Smith , Kevin Oberman , "net@freebsd.org" , "jfv@FreeBSD.org" , "ricera10@gmail.com" , Luigi Rizzo , Giuseppe Lettieri Subject: Re: ixl 40G bad performance? In-Reply-To: <20151020232218.G1833@besplex.bde.org> Message-ID: <20151023162337.L1149@besplex.bde.org> References: <79830D9D-94E6-47A9-92B9-D63DF5432272@netapp.com> <20151020190541.B15983@sola.nimnet.asn.au> <6CD6754D-FC0E-4B24-AAEC-7C9D68284141@netapp.com> <20151020232218.G1833@besplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=cK4dyQqN c=1 sm=1 tr=0 a=KA6XNC2GZCFrdESI5ZmdjQ==:117 a=PO7r1zJSAAAA:8 a=JzwRw_2MAAAA:8 a=kj9zAlcOel0A:10 a=7HLWPhpPLtQj6zYKw5gA:9 a=CjuIK1q_8ugA:10 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Oct 2015 05:36:23 -0000 On Wed, 21 Oct 2015, Bruce Evans wrote: > Fix for em: > > X diff -u2 if_em.c~ if_em.c > X --- if_em.c~ 2015-09-28 06:29:35.000000000 +0000 > X +++ if_em.c 2015-10-18 18:49:36.876699000 +0000 > X @@ -609,8 +609,8 @@ > X em_tx_abs_int_delay_dflt); > X em_add_int_delay_sysctl(adapter, "itr", > X - "interrupt delay limit in usecs/4", > X + "interrupt delay limit in usecs", > X &adapter->tx_itr, > X E1000_REGISTER(hw, E1000_ITR), > X - DEFAULT_ITR); > X + 1000000 / MAX_INTS_PER_SEC); > X X /* Sysctl for limiting the amount of work done in the taskqueue */ > > "delay limit" is fairly good wording. Other parameters tend to give long > delays, but itr limits the longest delay due to interrupt moderation to > whatever the itr respresents. Everything in the last paragraph is backwards (inverted). Other parameters tend to give short delays. They should be set to small values to minimise latency. Then under load, itr limits the interrupt _rate_ from above. The interrupt delay is the inverse of the interrupt rate, so it is limited from below. So "delay limit" is fairly bad wording. Normally, limits are from above, but the inversion makes the itr limit from below. This is most easily understood by converting itr to a rate: itr = 125 means a rate limit of 8000 Hz. It doesn't quite mean that the latency is at least 125 usec. No one wants to ensure large latencies, and the itr setting only ensures a minimal average latency them under load. Bruce From owner-freebsd-net@freebsd.org Fri Oct 23 05:48:06 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 53587A1677D for ; Fri, 23 Oct 2015 05:48:06 +0000 (UTC) (envelope-from njwilliams@swin.edu.au) Received: from gpo1.cc.swin.edu.au (gpo1.cc.swin.edu.au [136.186.1.30]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C9633CF7 for ; Fri, 23 Oct 2015 05:48:05 +0000 (UTC) (envelope-from njwilliams@swin.edu.au) Received: from [136.186.229.154] (nwilliams-laptop.caia.swin.edu.au [136.186.229.154]) by gpo1.cc.swin.edu.au (8.14.3/8.14.3) with ESMTP id t9N5lwHf016645; Fri, 23 Oct 2015 16:47:59 +1100 Subject: Re: MPTCP for FreeBSD repository on BitBucket/v0.51 update To: George Neville-Neil References: <562466AA.7020707@swin.edu.au> <137EE92F-7611-402D-874F-37AA627F41CA@neville-neil.com> Cc: freebsd-net@freebsd.org From: Nigel Williams Message-ID: <5629C9BF.2030902@swin.edu.au> Date: Fri, 23 Oct 2015 16:46:39 +1100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <137EE92F-7611-402D-874F-37AA627F41CA@neville-neil.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Oct 2015 05:48:06 -0000 Hi George, > > Very nice! Just wondering how you're testing this out. I've been > working on a lot of networking > tests and I'm sure MPTCP introduces some interesting complications. > Right now the tests are quite simple - based off the shell scripts and topology here: https://bitbucket.org/nw-swin/caia-mptcp-files/src These are mostly quick and dirty tests to catch regressions (I'm still making some fairly broad changes to the implementation). They provide basic MP connections and some dummynet control to adjust the link characteristics. I don't yet have anything where I can easily script-in topology changes, bring links up and down mid-connection, introduce cross traffic, etc... but the plan is to move to that kind of set up. And it's probably worth mentioning I've just been running my tests in VMs, though I'd like to run tests on hardware once I squash some of the stability issues. cheers, nigel From owner-freebsd-net@freebsd.org Fri Oct 23 08:45:15 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E4E76A1AD16 for ; Fri, 23 Oct 2015 08:45:15 +0000 (UTC) (envelope-from alarig@swordarmor.fr) Received: from bulbizarre.swordarmor.fr (bulbizarre.swordarmor.fr [IPv6:2001:470:1f13:138:715d:2fa0:b591:532f]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0F4771F99 for ; Fri, 23 Oct 2015 08:45:14 +0000 (UTC) (envelope-from alarig@swordarmor.fr) Received: from drscott.swordarmor.fr (drscott-he.swordarmor.fr [IPv6:2001:470:1f13:138::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bulbizarre.swordarmor.fr (Postfix) with ESMTPS id 384CD4ED832 for ; Fri, 23 Oct 2015 10:45:11 +0200 (CEST) Authentication-Results: bulbizarre.swordarmor.fr; dkim=pass reason="1024-bit key" header.d=swordarmor.fr header.i=@swordarmor.fr header.b=jEgeH2Sx; dkim-adsp=pass; dkim-atps=neutral Received: from drscott.swordarmor.fr (localhost [127.0.0.1]); by drscott.swordarmor.fr (OpenSMTPD) with ESMTP id 26b26d7a; for ; Fri, 23 Oct 2015 08:45:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=simple; d=swordarmor.fr; h=date:from :to:subject:message-id:references:mime-version:content-type :in-reply-to; s=default; bh=nff+aR/JTyyAeyl/XjoCJaITR78=; b=jEge H2SxDDTJgQ32skbivphk5F2uIyHXKQd6d18U5PNLrOmx3GHdPGQYIw5M254VtJoS sHL+KyxccTEr41AHPIZC7a9xxO+2TkVTx5xQhm3isvXxVyPbU6IuqS+fZV1o4ah2 Pj8/497rBaU2xSlSIAOBhRZBw3xmJxUd9HCwwmA= DomainKey-Signature: a=rsa-sha1; c=simple; d=swordarmor.fr; h=date:from :to:subject:message-id:references:mime-version:content-type :in-reply-to; q=dns; s=default; b=kZxcg8kEf66SVKJRBy1SoYG/0xCzlm 4lHXsOqe4fkBQRsrN9VKqFzF5sL5BAihXPIpN8Kb/aU7MucznrHsjW58uGh7n6yt EidLc3vMUhl6u2V1Dq5mOG2FaS+qmT90aSGwheXL5W73a31XUWNWIPkvaLH0q0Ta NZNirfME9NfKY= Received: from drscott.swordarmor.fr (localhost [IPv6:::1]); by drscott.swordarmor.fr (OpenSMTPD) with ESMTP id 08ddd790; for ; Fri, 23 Oct 2015 08:45:10 +0000 (UTC) Date: Fri, 23 Oct 2015 10:45:10 +0200 From: Alarig Le Lay To: freebsd-net@freebsd.org Subject: Re: Cannot add an ipv6 route with -interface Message-ID: <20151023084510.GC22065@drscott.swordarmor.fr> References: <20151022230838.GA22065@drscott.swordarmor.fr> <20151023.090553.1908755081181502110.hrs@allbsd.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="FMJTF8LVhUQkvsEb" Content-Disposition: inline In-Reply-To: <20151023.090553.1908755081181502110.hrs@allbsd.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Oct 2015 08:45:16 -0000 --FMJTF8LVhUQkvsEb Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri Oct 23 09:05:53 2015, Hiroki Sato wrote: > Is 2a00:5884::1/64 configured on em1? An address within > 2a00:5884:8200::/40 is required on em1 in this case. em1 is configured with 2a00:5884::1/36 (which does not contain 2a00:5884:8200::/40), but physically this two networks are at the same place (i.e. in the same vlan on the same switch) so I don=E2=80=99t underst= and why I need to take an address from 2a00:5884:8200::/40. --=20 Alarig Le Lay --FMJTF8LVhUQkvsEb Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJWKfOWAAoJEK84SsFrICuIUqAIAKUm8LMgHHoi20AxihCXN6kJ LBBq+s64zMpaQhhYyeqVgD5hrvvv67vGCSa827u3U2ft2OmcQdBqDoIXtRyK8BcY YWEmZTYhmPZy2os/qODkYARogjhQCJ+9ry9LgExn/IfuDwrM4rpAzup8pUxAc4Ys XesgxwfRSD+k/JFOc7La5qhAF0D3pFDFIHvspK1rjXPYj0lSgmd1A6WkWzNHGTUS Q9EMRL3SZHzsuIwlamHbE2fZtBkhmwx+H8xjz+YmR1dDQjPSGLY1YpAvXACE1rnW n2G3OoZgUWtTEljR3RmBCaC5c1lURv1f/YDG4n6y5UrzZNUB6DUVDOrUKrtO+Y8= =gA9t -----END PGP SIGNATURE----- --FMJTF8LVhUQkvsEb-- From owner-freebsd-net@freebsd.org Fri Oct 23 18:11:54 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8D3D4A1DCF9 for ; Fri, 23 Oct 2015 18:11:54 +0000 (UTC) (envelope-from ricera10@gmail.com) Received: from mail-qk0-x231.google.com (mail-qk0-x231.google.com [IPv6:2607:f8b0:400d:c09::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 496CEE3C for ; Fri, 23 Oct 2015 18:11:54 +0000 (UTC) (envelope-from ricera10@gmail.com) Received: by qkbl190 with SMTP id l190so84590353qkb.2 for ; Fri, 23 Oct 2015 11:11:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-type; bh=VHieowAwTRKuwoIm+FKUzD8/GeIQGLwWnO5yoPjo3qw=; b=PpX2xHZTKMhykioqX8/Gf2uSJxBA7csDsUUU3rQ3hxE9IjzA/B29b8ORM5WfnNvxL7 rzU4K9y5OVgvLNc688KfjgcqvwR6IYmmu8pA1wB1ktACT2Ei3PhX1TeeZLtz7xW044Nd ZtmsRJtCu48bWle8ZbdjJgd7BlxcR0SQ57xrNI9MStK5oh7vSe5eMUvSJHzEIo62GlmZ ETszgMj7sQHlbvaz0VyBBvqbDhkgofaKULThut1HXdAGICgl1meljsESdCKXDOI7nxyM Tx1C7s9dz0Sf0R5HfNn07nnFEJs1Ks7K2nny96XFFqAYtipwovptP5EaIF+J7YkhLEHU fDnQ== X-Received: by 10.140.41.81 with SMTP id y75mr27250356qgy.5.1445623913272; Fri, 23 Oct 2015 11:11:53 -0700 (PDT) MIME-Version: 1.0 References: <561F6BFB.7080103@freebsd.org> <20151016154512.T15983@sola.nimnet.asn.au> In-Reply-To: From: Eric Joyner Date: Fri, 23 Oct 2015 18:11:43 +0000 Message-ID: Subject: Re: nice stuff from cloudflare (and, we need something like ethtool!) To: Sepherosa Ziehau , Jim Thompson Cc: "freebsd-net@freebsd.org" , Ian Smith Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Oct 2015 18:11:54 -0000 Bump -- I'd be okay with adding hooks to at least one Intel ethernet driver for whatever comes out of this. On Mon, Oct 19, 2015 at 1:47 AM Sepherosa Ziehau wrote: > On Fri, Oct 16, 2015 at 1:25 PM, Jim Thompson wrote: > > > > > > > >> On Oct 16, 2015, at 12:06 AM, Ian Smith wrote: > >> > >>> On Thu, 15 Oct 2015 17:03:55 +0800, Julian Elischer wrote: > >>>> On 10/10/15 10:59 PM, Luigi Rizzo wrote: > >>>> the nice folks at cloudflare implemented a nice feature > >>>> in netmap that puts some queues of the NIC in netmap mode > >>>> leaving others attached to the host stack > >>>> > >>>> > https://blog.cloudflare.com/single-rx-queue-kernel-bypass-with-netmap/ > >>>> > >>>> and use ethtool (and native NIC filters) to steer traffic around. > >>>> [FWIW, the chelsio native netmap driver is similar except that > >>>> the netmap queue has a different MAC address] > >>>> > >>>> While their code was developed on linux, it should run > >>>> almost unmodified on FreeBSD (and we plan to import it soon), > >>>> except for the fact that we don't have ethtool hence no > >>>> device-independent mechanism to configure traffic steering. > >>>> > >>>> We really need to address the latter. > >>> > >>> I suspect the answer may be a device dependent sysctl > >> > >> Interesting; care to flesh out your ideas a bit on how that might work? > >> > >> I've done nothing more than skim ethtool(8) on linuxcommand.org, and > >> wondered why its functionality wasn't incorporated into ifconfig, but > >> then ifconfig (on FreeBSD anyway) is tending towards obesity already > > > > Luigi already did netlink sockets for FreeBSD. > > > > https://github.com/luigirizzo/netlink-freebsd > > ha, the netlink for BSD, interesting :) > > -- > Tomorrow Will Never Die > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@freebsd.org Fri Oct 23 21:43:10 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 64A47A1C6EC for ; Fri, 23 Oct 2015 21:43:10 +0000 (UTC) (envelope-from ricera10@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 38C891E2D for ; Fri, 23 Oct 2015 21:43:10 +0000 (UTC) (envelope-from ricera10@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 38D29A1C6EB; Fri, 23 Oct 2015 21:43:10 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 385F2A1C6EA for ; Fri, 23 Oct 2015 21:43:10 +0000 (UTC) (envelope-from ricera10@gmail.com) Received: from mail-qg0-f53.google.com (mail-qg0-f53.google.com [209.85.192.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E649E1E2C; Fri, 23 Oct 2015 21:43:09 +0000 (UTC) (envelope-from ricera10@gmail.com) Received: by qgeo38 with SMTP id o38so78006919qge.0; Fri, 23 Oct 2015 14:43:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-type; bh=k6BJ7qKf2QAS+nkyxFPthAZjqSCrFDDfmyytETa1Bb8=; b=Jg7xb8MkMv3w6wWs4new7/OeNdZcpqBvM8ivRb+c9vw81h/eXlb0LgBELEmMIWLbcS hUwKDNOAAQH8UoNc/E1wfMz5ZZI86odo0CI92TyOgpTGrO733OtHFfe/Cc//dXvsqiAB pam954FX4Akpri6qjFB0+iCXfkrJwHW6i+cDMlzc6Gyxi9IWkUxcnHMMNoALEXUo1bzS CF9vo7qZ87wID/5HMSJsn0tCncbfU6wb6vyyPoZNc28WW2FGdArt3gwQhI/1qQ5sV2PE EQuPqgBxAUwx0hclz9pLKexI0U3mqRkH4YEhfzQa7B0ZErqYaIBjnXAekO9l+EzWfqoZ o2Iw== X-Received: by 10.140.146.13 with SMTP id 13mr29955233qhs.1.1445636181279; Fri, 23 Oct 2015 14:36:21 -0700 (PDT) Received: from mail-qg0-f41.google.com (mail-qg0-f41.google.com. [209.85.192.41]) by smtp.gmail.com with ESMTPSA id 42sm8233992qky.39.2015.10.23.14.36.20 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 23 Oct 2015 14:36:20 -0700 (PDT) Received: by qgad10 with SMTP id d10so77384685qga.3; Fri, 23 Oct 2015 14:36:20 -0700 (PDT) X-Received: by 10.140.94.55 with SMTP id f52mr27419721qge.0.1445636180376; Fri, 23 Oct 2015 14:36:20 -0700 (PDT) MIME-Version: 1.0 References: <79830D9D-94E6-47A9-92B9-D63DF5432272@netapp.com> <20151020190541.B15983@sola.nimnet.asn.au> <6CD6754D-FC0E-4B24-AAEC-7C9D68284141@netapp.com> <20151020232218.G1833@besplex.bde.org> <20151023162337.L1149@besplex.bde.org> In-Reply-To: <20151023162337.L1149@besplex.bde.org> From: Eric Joyner Date: Fri, 23 Oct 2015 21:36:10 +0000 Message-ID: Subject: Re: ixl 40G bad performance? To: Bruce Evans Cc: "Eggert, Lars" , Ian Smith , Kevin Oberman , "net@freebsd.org" , "jfv@FreeBSD.org" , Luigi Rizzo , Giuseppe Lettieri Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Oct 2015 21:43:10 -0000 Bruce mostly has it right -- ITR is the minimum latency between interrupts. But, it does actually guarantee a minimum period between interrupts. Though, Fortville actually is unique a little bit in that there is another ITR setting that can ensure a certain average number of interrupts per second (called Interrupt Rate Limiting), though, but I don't think this is used in the current version of the driver. I see that the sysctl does clobber the global value, but have you tried lowering the interval / raising the rate? You could try something like 10usecs, and see if that helps. We'll do some more investigation here -- 3Gb/s on a 40Gb/s using default settings is terrible, and we shouldn't let that be happening. - Eric On Thu, Oct 22, 2015 at 10:36 PM Bruce Evans wrote: > On Wed, 21 Oct 2015, Bruce Evans wrote: > > > Fix for em: > > > > X diff -u2 if_em.c~ if_em.c > > X --- if_em.c~ 2015-09-28 06:29:35.000000000 +0000 > > X +++ if_em.c 2015-10-18 18:49:36.876699000 +0000 > > X @@ -609,8 +609,8 @@ > > X em_tx_abs_int_delay_dflt); > > X em_add_int_delay_sysctl(adapter, "itr", > > X - "interrupt delay limit in usecs/4", > > X + "interrupt delay limit in usecs", > > X &adapter->tx_itr, > > X E1000_REGISTER(hw, E1000_ITR), > > X - DEFAULT_ITR); > > X + 1000000 / MAX_INTS_PER_SEC); > > X X /* Sysctl for limiting the amount of work done in the taskqueue */ > > > > "delay limit" is fairly good wording. Other parameters tend to give long > > delays, but itr limits the longest delay due to interrupt moderation to > > whatever the itr respresents. > > Everything in the last paragraph is backwards (inverted). Other > parameters tend to give short delays. They should be set to small > values to minimise latency. Then under load, itr limits the interrupt > _rate_ from above. The interrupt delay is the inverse of the interrupt > rate, so it is limited from below. So "delay limit" is fairly bad > wording. Normally, limits are from above, but the inversion makes > the itr limit from below. > > This is most easily understood by converting itr to a rate: itr = 125 > means a rate limit of 8000 Hz. It doesn't quite mean that the latency > is at least 125 usec. No one wants to ensure large latencies, and the > itr setting only ensures a minimal average latency them under load. > > Bruce > From owner-freebsd-net@freebsd.org Sat Oct 24 07:44:10 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DA032A1D3F2 for ; Sat, 24 Oct 2015 07:44:10 +0000 (UTC) (envelope-from lars@netapp.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id AD844EE4 for ; Sat, 24 Oct 2015 07:44:10 +0000 (UTC) (envelope-from lars@netapp.com) Received: by mailman.ysv.freebsd.org (Postfix) id ABE0FA1D3F1; Sat, 24 Oct 2015 07:44:10 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AB745A1D3F0 for ; Sat, 24 Oct 2015 07:44:10 +0000 (UTC) (envelope-from lars@netapp.com) Received: from mx142.netapp.com (mx142.netapp.com [216.240.21.19]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "mx142.netapp.com", Issuer "Symantec Class 3 Secure Server CA - G4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6C4C4EE3; Sat, 24 Oct 2015 07:44:09 +0000 (UTC) (envelope-from lars@netapp.com) X-IronPort-AV: E=Sophos;i="5.20,191,1444719600"; d="asc'?scan'208,217";a="72698554" Received: from hioexcmbx04-prd.hq.netapp.com ([10.122.105.37]) by mx142-out.netapp.com with ESMTP; 24 Oct 2015 00:43:06 -0700 Received: from HIOEXCMBX07-PRD.hq.netapp.com (10.122.105.40) by hioexcmbx04-prd.hq.netapp.com (10.122.105.37) with Microsoft SMTP Server (TLS) id 15.0.1104.5; Sat, 24 Oct 2015 00:43:06 -0700 Received: from HIOEXCMBX07-PRD.hq.netapp.com ([::1]) by hioexcmbx07-prd.hq.netapp.com ([fe80::e1d9:911e:3048:d510%21]) with mapi id 15.00.1104.000; Sat, 24 Oct 2015 00:43:06 -0700 From: "Eggert, Lars" To: Eric Joyner CC: Bruce Evans , Ian Smith , Kevin Oberman , "net@freebsd.org" , "jfv@FreeBSD.org" , Luigi Rizzo , Giuseppe Lettieri Subject: Re: ixl 40G bad performance? Thread-Topic: ixl 40G bad performance? Thread-Index: AQHRCnVeClivuZWSFkK4WCM/ynkiH55zUwOAgAAMKACAAAITgIAA5BEAgAA8ooCAACxPAIAAP6UAgAQb94CAAQw8AIAAqZUA Date: Sat, 24 Oct 2015 07:43:06 +0000 Message-ID: <0CFE1957-8FEB-4000-9E39-1F6A8983EFA6@netapp.com> References: <79830D9D-94E6-47A9-92B9-D63DF5432272@netapp.com> <20151020190541.B15983@sola.nimnet.asn.au> <6CD6754D-FC0E-4B24-AAEC-7C9D68284141@netapp.com> <20151020232218.G1833@besplex.bde.org> <20151023162337.L1149@besplex.bde.org> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-mailer: Apple Mail (2.3096.5) x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [10.120.60.34] Content-Type: multipart/signed; boundary="Apple-Mail=_C4A5AFE6-0FFD-49DB-AF8D-B08A2F3CBC4F"; protocol="application/pgp-signature"; micalg=pgp-sha256 MIME-Version: 1.0 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Oct 2015 07:44:10 -0000 --Apple-Mail=_C4A5AFE6-0FFD-49DB-AF8D-B08A2F3CBC4F Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii On 2015-10-23, at 23:36, Eric Joyner wrote: > I see that the sysctl does clobber the global value, but have you = tried lowering the interval / raising the rate? You could try something = like 10usecs, and see if that helps. We'll do some more investigation = here -- 3Gb/s on a 40Gb/s using default settings is terrible, and we = shouldn't let that be happening. I played with different settings, but I've never been able to get more = than 4Gb/s, whereas under Linux 4.2 without any special settings I get = 13. See my other email on TSO/LRO not looking to be effective; that would = certainly explain it. Plausible? Anything to try here? Lars --Apple-Mail=_C4A5AFE6-0FFD-49DB-AF8D-B08A2F3CBC4F Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="signature.asc" Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- iQCVAwUBVis2jNZcnpRveo1xAQg0ZAP+I2aX0iZnfKUtEzndV/LNYZThZ87CtQNM /VY+BWiV5ciCyKqbISlcSOCznkDN+i9oEIdKa59CdfGI/WjVWYFp3zd5AuwxLFKx 1hQlapfG91vraC1D3y8dCmiaI0LKXs+8BzKfzm0AQw1yHthUtsu7/wS7SLH312yl 8RYXPRLK2gY= =RruI -----END PGP SIGNATURE----- --Apple-Mail=_C4A5AFE6-0FFD-49DB-AF8D-B08A2F3CBC4F-- From owner-freebsd-net@freebsd.org Sat Oct 24 08:32:57 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AC680A1DE26 for ; Sat, 24 Oct 2015 08:32:57 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 84570C2 for ; Sat, 24 Oct 2015 08:32:57 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 7EC6FA1DE25; Sat, 24 Oct 2015 08:32:57 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7D707A1DE24 for ; Sat, 24 Oct 2015 08:32:57 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-yk0-x241.google.com (mail-yk0-x241.google.com [IPv6:2607:f8b0:4002:c07::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2CECEBF; Sat, 24 Oct 2015 08:32:57 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: by ykfw201 with SMTP id w201so11597735ykf.2; Sat, 24 Oct 2015 01:32:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=0p/Zagey6vp/YtiELlFcAeNaBUTKrel+VxW3wCq+v+A=; b=oVrdjnLh2OXdl2rNAVndCnQx9oF5QyPxUUB+NrCESsAd6eabAYIvggWECi8Un+cKLy 8QQYVICptrX4He1M6pqhNCcWwmgHrQdbJ6QKHmE2sUiw11o/11YNlaVUqAXeoVtdo67u FLY3b6I4L0BCCVZwZRUJx54UX2onm9g9SR9108BvdtrC/fEWIWlTD3D9psry9m5cADzG MgVR/Or/AWHU8ncBszNRLkLG2TO3IPZ3OWX+u/jj3j8AVXJlfyPX7P8ZIlTO37MjJxUj idkDJbfddoAapjDH7O3eG0SOWFCrI69TfdTHk4HFCLOhaxxZIt5IQ4gSeMoBFoXgue/V cgmA== MIME-Version: 1.0 X-Received: by 10.129.125.212 with SMTP id y203mr18438392ywc.182.1445675576374; Sat, 24 Oct 2015 01:32:56 -0700 (PDT) Received: by 10.37.36.204 with HTTP; Sat, 24 Oct 2015 01:32:56 -0700 (PDT) In-Reply-To: <0CFE1957-8FEB-4000-9E39-1F6A8983EFA6@netapp.com> References: <79830D9D-94E6-47A9-92B9-D63DF5432272@netapp.com> <20151020190541.B15983@sola.nimnet.asn.au> <6CD6754D-FC0E-4B24-AAEC-7C9D68284141@netapp.com> <20151020232218.G1833@besplex.bde.org> <20151023162337.L1149@besplex.bde.org> <0CFE1957-8FEB-4000-9E39-1F6A8983EFA6@netapp.com> Date: Sat, 24 Oct 2015 01:32:56 -0700 Message-ID: Subject: Re: ixl 40G bad performance? From: Jack Vogel To: "Eggert, Lars" Cc: Eric Joyner , Bruce Evans , Ian Smith , Kevin Oberman , "net@freebsd.org" , "jfv@FreeBSD.org" , Luigi Rizzo , Giuseppe Lettieri Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Oct 2015 08:32:57 -0000 13 on a 40G interface?? I don't think that's very good for Linux either, is this a 4x10 adapter? Maybe elaborating on the details of the hardware, you sure you don't have a bad PCI slot somewhere that might be throttling everything? Cheers, Jack On Sat, Oct 24, 2015 at 12:43 AM, Eggert, Lars wrote: > On 2015-10-23, at 23:36, Eric Joyner wrote: > > I see that the sysctl does clobber the global value, but have you tried > lowering the interval / raising the rate? You could try something like > 10usecs, and see if that helps. We'll do some more investigation here -- > 3Gb/s on a 40Gb/s using default settings is terrible, and we shouldn't let > that be happening. > > > I played with different settings, but I've never been able to get more > than 4Gb/s, whereas under Linux 4.2 without any special settings I get 13. > > See my other email on TSO/LRO not looking to be effective; that would > certainly explain it. Plausible? Anything to try here? > > Lars > > From owner-freebsd-net@freebsd.org Sat Oct 24 08:47:00 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5D0D7A1C0A2 for ; Sat, 24 Oct 2015 08:47:00 +0000 (UTC) (envelope-from lars@netapp.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 32CC17F5 for ; Sat, 24 Oct 2015 08:47:00 +0000 (UTC) (envelope-from lars@netapp.com) Received: by mailman.ysv.freebsd.org (Postfix) id 31E44A1C0A1; Sat, 24 Oct 2015 08:47:00 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 19196A1C0A0 for ; Sat, 24 Oct 2015 08:47:00 +0000 (UTC) (envelope-from lars@netapp.com) Received: from mx142.netapp.com (mx142.netapp.com [216.240.21.19]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "mx142.netapp.com", Issuer "Symantec Class 3 Secure Server CA - G4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D0EE77F4; Sat, 24 Oct 2015 08:46:59 +0000 (UTC) (envelope-from lars@netapp.com) X-IronPort-AV: E=Sophos;i="5.20,191,1444719600"; d="asc'?scan'208";a="72703441" Received: from hioexcmbx02-prd.hq.netapp.com ([10.122.105.35]) by mx142-out.netapp.com with ESMTP; 24 Oct 2015 01:45:54 -0700 Received: from HIOEXCMBX07-PRD.hq.netapp.com (10.122.105.40) by hioexcmbx02-prd.hq.netapp.com (10.122.105.35) with Microsoft SMTP Server (TLS) id 15.0.1104.5; Sat, 24 Oct 2015 01:45:54 -0700 Received: from HIOEXCMBX07-PRD.hq.netapp.com ([::1]) by hioexcmbx07-prd.hq.netapp.com ([fe80::e1d9:911e:3048:d510%21]) with mapi id 15.00.1104.000; Sat, 24 Oct 2015 01:45:54 -0700 From: "Eggert, Lars" To: Jack Vogel CC: Ian Smith , Kevin Oberman , "net@freebsd.org" , "jfv@FreeBSD.org" , Giuseppe Lettieri , Eric Joyner , Luigi Rizzo Subject: Re: ixl 40G bad performance? Thread-Topic: ixl 40G bad performance? Thread-Index: AQHRCnVeClivuZWSFkK4WCM/ynkiH55zUwOAgAAMKACAAAITgIAA5BEAgAA8ooCAACxPAIAAP6UAgAQb94CAAQw8AIAAqZUAgAAN6gCAAAOdAA== Date: Sat, 24 Oct 2015 08:45:53 +0000 Message-ID: <0998C63C-F7D9-44FE-A535-9E57EEF36C58@netapp.com> References: <79830D9D-94E6-47A9-92B9-D63DF5432272@netapp.com> <20151020190541.B15983@sola.nimnet.asn.au> <6CD6754D-FC0E-4B24-AAEC-7C9D68284141@netapp.com> <20151020232218.G1833@besplex.bde.org> <20151023162337.L1149@besplex.bde.org> <0CFE1957-8FEB-4000-9E39-1F6A8983EFA6@netapp.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-mailer: Apple Mail (2.3096.5) x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [10.120.60.34] Content-Type: multipart/signed; boundary="Apple-Mail=_4F86EABF-4DE1-40E3-B029-6FCBBAC8136F"; protocol="application/pgp-signature"; micalg=pgp-sha256 MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Oct 2015 08:47:00 -0000 --Apple-Mail=_4F86EABF-4DE1-40E3-B029-6FCBBAC8136F Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii On 2015-10-24, at 10:32, Jack Vogel wrote: > 13 on a 40G interface?? I don't think that's very good for Linux = either, is > this a 4x10 adapter? No, its's a 2x40. And I can get it into the high 30s with tuning. I just = mentioned the value to illustrate that something seems to be seriously = broken under FreeBSD. Lars > Maybe elaborating on the details of the hardware, you sure you don't = have a > bad PCI slot > somewhere that might be throttling everything? >=20 > Cheers, >=20 > Jack >=20 >=20 > On Sat, Oct 24, 2015 at 12:43 AM, Eggert, Lars = wrote: >=20 >> On 2015-10-23, at 23:36, Eric Joyner wrote: >>=20 >> I see that the sysctl does clobber the global value, but have you = tried >> lowering the interval / raising the rate? You could try something = like >> 10usecs, and see if that helps. We'll do some more investigation here = -- >> 3Gb/s on a 40Gb/s using default settings is terrible, and we = shouldn't let >> that be happening. >>=20 >>=20 >> I played with different settings, but I've never been able to get = more >> than 4Gb/s, whereas under Linux 4.2 without any special settings I = get 13. >>=20 >> See my other email on TSO/LRO not looking to be effective; that would >> certainly explain it. Plausible? Anything to try here? >>=20 >> Lars >>=20 >>=20 > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" --Apple-Mail=_4F86EABF-4DE1-40E3-B029-6FCBBAC8136F Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="signature.asc" Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- iQCVAwUBVitFQdZcnpRveo1xAQi+jwP/cfm/zGSqsTHrF5vkIswsdufNTQw85FE8 gJ9ScG/vw+S6yKZQY2rNpXmZqCESY2g6vuNUacmDtgKh7z4FVg2WXwvucmhZITu5 fpTzTNOTrfmoYHoxVSXV6EPoqoeL9xwShmcWKIgc2+FGGTlfWzuSYmsVHSuGPryV sZEsEr3F/IY= =S1D+ -----END PGP SIGNATURE----- --Apple-Mail=_4F86EABF-4DE1-40E3-B029-6FCBBAC8136F-- From owner-freebsd-net@freebsd.org Sat Oct 24 10:20:33 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EAC20A1B7EB for ; Sat, 24 Oct 2015 10:20:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D7AA41066 for ; Sat, 24 Oct 2015 10:20:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9OAKXFC020847 for ; Sat, 24 Oct 2015 10:20:33 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 170081] [fxp] pf/nat/jails not working if checksum offloading is enabled on fxp0 Date: Sat, 24 Oct 2015 10:20:34 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.1-PRERELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: alucard.ui@gmail.com X-Bugzilla-Status: In Progress X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Oct 2015 10:20:34 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=170081 alucard.ui@gmail.com changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |alucard.ui@gmail.com --- Comment #3 from alucard.ui@gmail.com --- I had the same problem. uname -rs FreeBSD 10.2-RELEASE jails and nat (pf) on fxp0 does not work. After disabling rx checksum ( ifconfig fxp0 -rxcsum ) it's working perfectly. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-net@freebsd.org Sat Oct 24 17:46:55 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2E5C3A1E8A7 for ; Sat, 24 Oct 2015 17:46:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1B610132F for ; Sat, 24 Oct 2015 17:46:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9OHksqm054603 for ; Sat, 24 Oct 2015 17:46:54 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 170081] [fxp] pf/nat/jails not working if checksum offloading is enabled on fxp0 Date: Sat, 24 Oct 2015 17:46:55 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.1-PRERELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: kp@freebsd.org X-Bugzilla-Status: Closed X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: kp@freebsd.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc bug_status assigned_to resolution Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Oct 2015 17:46:55 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=170081 Kristof Provost changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kp@freebsd.org Status|In Progress |Closed Assignee|freebsd-net@FreeBSD.org |kp@freebsd.org Resolution|--- |FIXED --- Comment #4 from Kristof Provost --- The fix for this issue went in in r289703 (for stable/10). See also 154428, 193579, 198868. The commit message, for reference: pf: Fix TSO issues In certain configurations (mostly but not exclusively as a VM on Xen) pf produced packets with an invalid TCP checksum. The problem was that pf could only handle packets with a full checksum. The FreeBSD IP stack produces TCP packets with a pseudo-header checksum (only addresses, length and protocol). Certain network interfaces expect to see the pseudo-header checksum, so they end up producing packets with invalid checksums. To fix this stop calculating the full checksum and teach pf to only update TCP checksums if TSO is disabled or the change affects the pseudo-header checksum. -- You are receiving this mail because: You are the assignee for the bug.