From owner-freebsd-net@FreeBSD.ORG Sun Nov 18 04:52:38 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 94A04793; Sun, 18 Nov 2012 04:52:38 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 551618FC13; Sun, 18 Nov 2012 04:52:38 +0000 (UTC) Received: by mail-pb0-f54.google.com with SMTP id wz12so2908015pbc.13 for ; Sat, 17 Nov 2012 20:52:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type :content-transfer-encoding; bh=JYBvCyZXTD+bjRmiuyUc4liSvg5C8bFB9v3vSI18vHg=; b=MZAkN1LQmBL5Ma6DZ2dtazdnuk9Roy8VkWUMra4nuHGZd9/xX7hVXx2Hdk7A5XkMXl u3ZcgVX4M9VDIMz5mrWKvJsHke1ErA7u89g157OJxsANr9GJaAL05pxIH8lBI+To4V3+ Ce8epHmV8iWA2RNzyvezH/A7xuLM11k4Xf5csBrqXjtbZQmAES6ic9+86U77Sxjqljxm kF99BBMOOnAiUbmkqy2ftpGlsP/UIihqM5kMhhIbXQEAi5UQ7VMsJmuZ9eoMfIRazTQI Up1EDINxvWbdNrZE8lZzCFMd3wg6opDhmgRUpCG6cHujxm9fXInj4toYebZSSYRL4Qww ycoA== MIME-Version: 1.0 Received: by 10.66.79.168 with SMTP id k8mr26249817pax.12.1353214352193; Sat, 17 Nov 2012 20:52:32 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.68.124.130 with HTTP; Sat, 17 Nov 2012 20:52:32 -0800 (PST) In-Reply-To: <201211180235.qAI2ZDql022583@freebsd-current.sentex.ca> References: <201211180235.qAI2ZDql022583@freebsd-current.sentex.ca> Date: Sat, 17 Nov 2012 20:52:32 -0800 X-Google-Sender-Auth: kwFBWrQJUzlVRn6jYuC6UmAa2zc Message-ID: Subject: Re: [head tinderbox] failure on mips64/mips From: Adrian Chadd To: freebsd-current , FreeBSD Net Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Nov 2012 04:52:38 -0000 On 17 November 2012 18:35, FreeBSD Tinderbox wrote: > cc -c -O -pipe -std=3Dc99 -g -Wall -Wredundant-decls -Wnested-externs -W= strict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qua= l -Wundef -Wno-pointer-sign -fformat-extensions -Wmissing-include-dirs -f= diagnostics-show-option -nostdinc -I. -I/src/sys -I/src/sys/contrib/altq= -I/src/sys/contrib/libfdt -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include = opt_global.h -fno-common -finline-limit=3D8000 --param inline-unit-growth= =3D10000 --param large-function-growth=3D100000 --param max-inline-insns-si= ngle=3D10000 -fno-pic -mno-abicalls -G0 -DKERNLOADADDR=3D0xffffffff8000100= 0 -mabi=3D64 -march=3Dmips64 -msoft-float -ffreestanding -Werror /src/sys/= net/rtsock.c > cc1: warnings being treated as errors > /src/sys/net/rtsock.c: In function 'sysctl_dumpentry': > /src/sys/net/rtsock.c:1577: warning: unused variable 'i' [-Wunused-variab= le] > *** [rtsock.o] Error code 1 Fixed. Damn those pesky non-IPV6 belivers. Adrian From owner-freebsd-net@FreeBSD.ORG Sun Nov 18 05:46:30 2012 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7AB2017C; Sun, 18 Nov 2012 05:46:30 +0000 (UTC) (envelope-from hrs@FreeBSD.org) Received: from mail.allbsd.org (gatekeeper.allbsd.org [IPv6:2001:2f0:104:e001::32]) by mx1.freebsd.org (Postfix) with ESMTP id ADDC08FC13; Sun, 18 Nov 2012 05:46:28 +0000 (UTC) Received: from alph.allbsd.org (p1137-ipbf1505funabasi.chiba.ocn.ne.jp [118.7.212.137]) (authenticated bits=128) by mail.allbsd.org (8.14.5/8.14.5) with ESMTP id qAI5kBGf037907 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 18 Nov 2012 14:46:21 +0900 (JST) (envelope-from hrs@FreeBSD.org) Received: from localhost (localhost [127.0.0.1]) (authenticated bits=0) by alph.allbsd.org (8.14.5/8.14.5) with ESMTP id qAI5k93r095634; Sun, 18 Nov 2012 14:46:11 +0900 (JST) (envelope-from hrs@FreeBSD.org) Date: Sun, 18 Nov 2012 14:45:59 +0900 (JST) Message-Id: <20121118.144559.684835936214218282.hrs@allbsd.org> To: adrian@FreeBSD.org Subject: Re: [head tinderbox] failure on mips64/mips From: Hiroki Sato In-Reply-To: References: <201211180235.qAI2ZDql022583@freebsd-current.sentex.ca> X-PGPkey-fingerprint: BDB3 443F A5DD B3D0 A530 FFD7 4F2C D3D8 2793 CF2D X-Mailer: Mew version 6.5 on Emacs 23.4 / Mule 6.0 (HANACHIRUSATO) Mime-Version: 1.0 Content-Type: Multipart/Signed; protocol="application/pgp-signature"; micalg=pgp-sha1; boundary="--Security_Multipart(Sun_Nov_18_14_45_59_2012_683)--" Content-Transfer-Encoding: 7bit X-Virus-Scanned: clamav-milter 0.97.4 at gatekeeper.allbsd.org X-Virus-Status: Clean X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (mail.allbsd.org [133.31.130.32]); Sun, 18 Nov 2012 14:46:21 +0900 (JST) X-Spam-Status: No, score=-97.9 required=13.0 tests=CONTENT_TYPE_PRESENT, ONLY1HOPDIRECT,QENCPTR1,SAMEHELOBY2HOP,USER_IN_WHITELIST autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on gatekeeper.allbsd.org Cc: freebsd-net@FreeBSD.org, freebsd-current@FreeBSD.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Nov 2012 05:46:30 -0000 ----Security_Multipart(Sun_Nov_18_14_45_59_2012_683)-- Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Adrian Chadd wrote in : ad> On 17 November 2012 18:35, FreeBSD Tinderbox wrote: ad> ad> > cc -c -O -pipe -std=c99 -g -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -Wmissing-include-dirs -fdiagnostics-show-option -nostdinc -I. -I/src/sys -I/src/sys/contrib/altq -I/src/sys/contrib/libfdt -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common -finline-limit=8000 --param inline-unit-growth=10000 --param large-function-growth=100000 --param max-inline-insns-single=10000 -fno-pic -mno-abicalls -G0 -DKERNLOADADDR=0xffffffff80001000 -mabi=64 -march=mips64 -msoft-float -ffreestanding -Werror /src/sys/net/rtsock.c ad> > cc1: warnings being treated as errors ad> > /src/sys/net/rtsock.c: In function 'sysctl_dumpentry': ad> > /src/sys/net/rtsock.c:1577: warning: unused variable 'i' [-Wunused-variable] ad> > *** [rtsock.o] Error code 1 ad> ad> Fixed. Damn those pesky non-IPV6 belivers. Sorry, I was careless about this part. -- Hiroki ----Security_Multipart(Sun_Nov_18_14_45_59_2012_683)-- Content-Type: application/pgp-signature Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iEYEABECAAYFAlCodhcACgkQTyzT2CeTzy2UdgCeMuJgRYyt0DfsYbZsFrZlSq25 thwAoKVcnz6ltjN9L5NaMCNkFxXsKMkX =bLAa -----END PGP SIGNATURE----- ----Security_Multipart(Sun_Nov_18_14_45_59_2012_683)---- From owner-freebsd-net@FreeBSD.ORG Sun Nov 18 08:10:56 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A167DB2D; Sun, 18 Nov 2012 08:10:56 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-da0-f54.google.com (mail-da0-f54.google.com [209.85.210.54]) by mx1.freebsd.org (Postfix) with ESMTP id 631CD8FC14; Sun, 18 Nov 2012 08:10:56 +0000 (UTC) Received: by mail-da0-f54.google.com with SMTP id z9so1841447dad.13 for ; Sun, 18 Nov 2012 00:10:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=gHYDOGP6DGWVWYueRBjeYS7NFocVfrkDja0zmotyGSA=; b=i0KHumUcR6mPDfCTxMpNhZE9+DsBqftmZMSTQPlO9uhqPjOjkOMRdQNhZx2SfF/B3u lB5FYHMskdgWcQcYjrfHGvmGDUELf9ZHXl4BsSsexFLwZiT7xMfMOnKTp+86C9EcpPj9 jy/eOWUQcDuPgnvAI1MnhIIGEyfm2lOYmXevQJabYJBHKL8JdvMQGbiKtMNrY4l3V/Hr MCv4QdBmU+7Tytc0OXpZXzme5rBFBx9ndjB+zSvIZCZX4MsYHJhMmBdVTv68RXd5ZAtH 7Ak6//rase8oCbuNhvyHU/FHnSOuvsJpC7UaUPwVe1srvtGdanW4IQr6xx9emdh56Jkd 1/FA== MIME-Version: 1.0 Received: by 10.66.75.101 with SMTP id b5mr26879964paw.45.1353226255760; Sun, 18 Nov 2012 00:10:55 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.68.124.130 with HTTP; Sun, 18 Nov 2012 00:10:55 -0800 (PST) In-Reply-To: <20121118.144559.684835936214218282.hrs@allbsd.org> References: <201211180235.qAI2ZDql022583@freebsd-current.sentex.ca> <20121118.144559.684835936214218282.hrs@allbsd.org> Date: Sun, 18 Nov 2012 00:10:55 -0800 X-Google-Sender-Auth: kYK4eb-IhqyrdDd4RVyFBpL4s2U Message-ID: Subject: Re: [head tinderbox] failure on mips64/mips From: Adrian Chadd To: Hiroki Sato Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-net@freebsd.org, freebsd-current@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Nov 2012 08:10:56 -0000 On 17 November 2012 21:45, Hiroki Sato wrote: > ad> Fixed. Damn those pesky non-IPV6 belivers. > > Sorry, I was careless about this part. It's fine. :-) God, I so can't grill anyone for breaking the build, given my track record! I hope you didn't mind me quickly fixing that up? Thanks, Adrian From owner-freebsd-net@FreeBSD.ORG Sun Nov 18 16:21:44 2012 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2B0E5824; Sun, 18 Nov 2012 16:21:44 +0000 (UTC) (envelope-from hrs@FreeBSD.org) Received: from mail.allbsd.org (gatekeeper.allbsd.org [IPv6:2001:2f0:104:e001::32]) by mx1.freebsd.org (Postfix) with ESMTP id 7ACC48FC13; Sun, 18 Nov 2012 16:21:43 +0000 (UTC) Received: from alph.allbsd.org (p1137-ipbf1505funabasi.chiba.ocn.ne.jp [118.7.212.137]) (authenticated bits=128) by mail.allbsd.org (8.14.5/8.14.5) with ESMTP id qAIGLQCI055073 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 19 Nov 2012 01:21:36 +0900 (JST) (envelope-from hrs@FreeBSD.org) Received: from localhost (localhost [127.0.0.1]) (authenticated bits=0) by alph.allbsd.org (8.14.5/8.14.5) with ESMTP id qAIGLP71017463; Mon, 19 Nov 2012 01:21:25 +0900 (JST) (envelope-from hrs@FreeBSD.org) Date: Mon, 19 Nov 2012 01:21:17 +0900 (JST) Message-Id: <20121119.012117.1135299527992880767.hrs@allbsd.org> To: adrian@FreeBSD.org Subject: Re: [head tinderbox] failure on mips64/mips From: Hiroki Sato In-Reply-To: References: <20121118.144559.684835936214218282.hrs@allbsd.org> X-PGPkey-fingerprint: BDB3 443F A5DD B3D0 A530 FFD7 4F2C D3D8 2793 CF2D X-Mailer: Mew version 6.5 on Emacs 23.4 / Mule 6.0 (HANACHIRUSATO) Mime-Version: 1.0 Content-Type: Multipart/Signed; protocol="application/pgp-signature"; micalg=pgp-sha1; boundary="--Security_Multipart(Mon_Nov_19_01_21_17_2012_132)--" Content-Transfer-Encoding: 7bit X-Virus-Scanned: clamav-milter 0.97.4 at gatekeeper.allbsd.org X-Virus-Status: Clean X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (mail.allbsd.org [133.31.130.32]); Mon, 19 Nov 2012 01:21:36 +0900 (JST) X-Spam-Status: No, score=-98.1 required=13.0 tests=CONTENT_TYPE_PRESENT, ONLY1HOPDIRECT,SAMEHELOBY2HOP,USER_IN_WHITELIST autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on gatekeeper.allbsd.org Cc: freebsd-net@FreeBSD.org, freebsd-current@FreeBSD.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Nov 2012 16:21:44 -0000 ----Security_Multipart(Mon_Nov_19_01_21_17_2012_132)-- Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Adrian Chadd wrote in : ad> On 17 November 2012 21:45, Hiroki Sato wrote: ad> ad> > ad> Fixed. Damn those pesky non-IPV6 belivers. ad> > ad> > Sorry, I was careless about this part. ad> ad> It's fine. :-) God, I so can't grill anyone for breaking the build, ad> given my track record! ad> ad> I hope you didn't mind me quickly fixing that up? I did not mind at all. Thank you for fixing it! -- Hiroki ----Security_Multipart(Mon_Nov_19_01_21_17_2012_132)-- Content-Type: application/pgp-signature Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iEYEABECAAYFAlCpCv0ACgkQTyzT2CeTzy3d4wCgzSZRxProKpE7ZPeIFZ/sKgKs iVsAoNDepgwpoFzUBpxynnQ8iq8wAfQD =0vU9 -----END PGP SIGNATURE----- ----Security_Multipart(Mon_Nov_19_01_21_17_2012_132)---- From owner-freebsd-net@FreeBSD.ORG Sun Nov 18 17:26:40 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A854C51A for ; Sun, 18 Nov 2012 17:26:40 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm4.bullet.mail.ne1.yahoo.com (nm4.bullet.mail.ne1.yahoo.com [98.138.90.67]) by mx1.freebsd.org (Postfix) with ESMTP id 55F458FC08 for ; Sun, 18 Nov 2012 17:26:40 +0000 (UTC) Received: from [98.138.90.52] by nm4.bullet.mail.ne1.yahoo.com with NNFMP; 18 Nov 2012 17:24:01 -0000 Received: from [98.138.89.168] by tm5.bullet.mail.ne1.yahoo.com with NNFMP; 18 Nov 2012 17:24:01 -0000 Received: from [127.0.0.1] by omp1024.mail.ne1.yahoo.com with NNFMP; 18 Nov 2012 17:24:01 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 861664.27620.bm@omp1024.mail.ne1.yahoo.com Received: (qmail 28544 invoked by uid 60001); 18 Nov 2012 17:24:01 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1353259441; bh=dAA35egQaYINnNnGSmDWGKPyL3Hn5a5GJa3GwDp6IQ0=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=kst5qxznPi+bHSVl1GY62Ocb6NiQLavgtUUnwXlHUa1Nl777lZg6spwJ4eqExzpd72gNY2cygL+IL54o5Nq4sXsAvTZpeApZyqZf/+XGu9lAQCImgt2KoRqjy+gDo0L9cYRH7Pr3y+NU2EOs7bfFfkD65iInEmHe6TTfqrrzq9I= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=ZU/5uXAshpbxQkWb+ffV/I5+0aIW4wNjQDlHJ5EvyYSiKecTLqi4SP9bnsHsIR/X04oSrKGeXP+x+NUo50FZkNA+Acqsrp1IcwT5nMutgQ4kjVpY1suLKUh7A+53tATSvtmJDZEEW9sB4GYcUN/ERj80ERHoT4O3a6fkci2H0y4=; X-YMail-OSG: 6OuxEckVM1mAZLx9L5yv4hW4Gzto.K09GuZclyfKSzXqQ2W cPGh_EWCyRnHqcYDuDkVGGV5QdBVgxorR3ZM9wV1v53RtJmvfjfGkK6bv2X6 qtovQ2k7VsC3DA.0mPHJBn3fBn3EKDJBbuHhvTXTapvyd8yAIQtQPro2K8Yt X1gavqeg9tFh9c_iOkVYf1Nl1aJBMzFph0CbpJS0A13xkBN.W6_UTZx5ZbjP f.Pq4akA.oNkKZdUevl2edxN9e22_a7EaUyuyvfJ_GMQ5vYga83TMn_JdxG3 JER9i3aOjyRYBBicOvhqoXoB_I0Tw7s5_vtgK9fyS9k65Vpt5Sp_5dmaGNR3 k8BQWDD2aJjGhVesAXH4WRsWj7cnYt7tzybXbEUh7Hpwtn9qckFfCc9vQB.w wZfDAGuNEW.U3GM3kuyRW3V_zyO4xYH66qVTqN5kfY2BrdmExNZdbPtInGxZ em_23SLimeVEbJwfPzDmEzNwZs_7Y5QrUTtjW_UXlrbCjzLvzcXHv4w.CbTm S0TdWP9d0.iDSYa3d5A-- Received: from [174.48.128.27] by web121605.mail.ne1.yahoo.com via HTTP; Sun, 18 Nov 2012 09:24:01 PST X-Rocket-MIMEInfo: 001.001, CgotLS0gT24gVGh1LCAxLzE5LzEyLCBKb2huIEJhbGR3aW4gPGpoYkBmcmVlYnNkLm9yZz4gd3JvdGU6Cgo.IEZyb206IEpvaG4gQmFsZHdpbiA8amhiQGZyZWVic2Qub3JnPgo.IFN1YmplY3Q6IExhdGVuY3kgaXNzdWVzIHdpdGggYnVmX3JpbmcKPiBUbzogbmV0QGZyZWVic2Qub3JnCj4gQ2M6ICJFZCBNYXN0ZSIgPGVtYXN0ZUBmcmVlYnNkLm9yZz4sICJOYXZkZWVwIFBhcmhhciIgPG5wQGZyZWVic2Qub3JnPgo.IERhdGU6IFRodXJzZGF5LCBKYW51YXJ5IDE5LCAyMDEyLCAxMTo0MSBBTQo.IFRoZSBjdXIBMAEBAQE- X-Mailer: YahooMailClassic/15.0.8 YahooMailWebService/0.8.123.460 Message-ID: <1353259441.19423.YahooMailClassic@web121605.mail.ne1.yahoo.com> Date: Sun, 18 Nov 2012 09:24:01 -0800 (PST) From: Barney Cordoba Subject: Re: Latency issues with buf_ring To: John Baldwin In-Reply-To: <201201191141.25998.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Nov 2012 17:26:40 -0000 =0A=0A--- On Thu, 1/19/12, John Baldwin wrote:=0A=0A> Fro= m: John Baldwin =0A> Subject: Latency issues with buf_ring= =0A> To: net@freebsd.org=0A> Cc: "Ed Maste" , "Navdeep = Parhar" =0A> Date: Thursday, January 19, 2012, 11:41 AM=0A>= The current buf_ring usage in various=0A> NIC drivers has a race that can= =0A> result in really high packet latencies in some cases.=A0=0A> Specifica= lly,=0A> the common pattern in an if_transmit routine is to use a=0A> try-l= ock on=0A> the queue and if that fails enqueue the packet in the=0A> buf_ri= ng and=0A> return.=A0 The race, of course, is that the thread=0A> holding t= he lock=0A> might have just finished checking the buf_ring and found it=0A>= empty and=0A> be in the process of releasing the lock when the original=0A= > thread fails=0A> the try lock.=A0 If this happens, then the packet queued= =0A> by the first=0A> thread will be stalled until another thread tries to= =0A> transmit packets=0A> for that queue.=A0 Some drivers attempt to handle= this=0A> race (igb(4)=0A> schedules a task to kick the transmit queue if t= he try lock=0A> fails) and=0A> others don't (cxgb(4) doesn't handle it at a= ll).=A0 At=0A> work this race=0A> was triggered very often after upgrading = from 7 to 8 with=0A> bursty=0A> traffic and caused numerous problems, so it= is not a rare=0A> occurrence=0A> and needs to be addressed.=0A> =0A> (Note= , all patches mentioned are against 8)=0A> =0A> The first hack I tried to u= se was to simply always lock the=0A> queue after=0A> the drbr enqueue if th= e try lock failed and then drain the=0A> queue if=0A> needed (www.freebsd.o= rg/~jhb/patches/try_fail.patch).=A0=0A> While this fixed=0A> my latency pro= blems, it would seem that this breaks other=0A> workloads=0A> that the drbr= design is trying to optimize.=0A> =0A> After further hacking what I came u= p with was a variant of=0A> drbr_enqueue()=0A> that would atomically set a = 'pending' flag.=A0 During the=0A> enqueue operation.=0A> The first thread t= o fail the try lock sets this flag (it is=0A> told that it=0A> set the flag= by a new return value (EINPROGRESS) from the=0A> enqueue call).=0A> The pe= nding thread then explicitly clears the flag once it=0A> acquires the=0A> q= ueue lock.=A0 This should prevent multiple threads from=0A> stacking up on = the=0A> queue lock so that if multiple threads are dumping packets=0A> into= the ring=0A> concurrently all but two (the one draining the queue=0A> curr= ently and the=0A> one waiting for the lock) can continue to drain the=0A> q= ueue.=A0 One downside=0A> of this approach though is that each driver has t= o be=0A> changed to make=0A> an explicit call to clear the pending flag aft= er grabbing=0A> the queue lock=0A> if the try lock fails.=A0 This is what I= am currently=0A> running in production=0A> (www.freebsd.org/~jhb/patches/t= ry_fail3.patch).=0A> =0A> However, this still results in a lot of duplicate= d code in=0A> each driver=0A> that wants to support multiq.=A0 Several folk= s have=0A> expressed a desire=0A> to move in a direction where the stack ha= s explicit=0A> knowledge of=0A> transmit queues allowing us to hoist some o= f this duplicated=0A> code out=0A> of the drivers and up into the calling l= ayer.=A0 After=0A> discussing this a=0A> bit with Navdeep (np@), the approa= ch I am looking at is to=0A> alter the=0A> buf_ring code flow a bit to more= closely model the older=0A> code-flow=0A> with IFQ and if_start methods.= =A0 That is, have the=0A> if_transmit methods=0A> always enqueue each packe= t that arrives to the buf_ring and=0A> then to=0A> call an if_start-like me= thod that drains a specific transmit=0A> queue.=0A> This approach simplifie= s a fair bit of driver code and means=0A> we can=0A> potentially move the e= nqueue, etc. bits up into the calling=0A> layer and=0A> instead have driver= s provide the per-transmit queue start=0A> routine as=0A> the direct functi= on pointer to the upper layers ala=0A> if_start.=0A> =0A> However, we would= still need a way to close the latency=0A> race.=A0 I've=0A> attempted to d= o that by inverting my previous 'thread=0A> pending' flag.=0A> Instead, I m= ake the buf_ring store a 'busy' flag.=A0 This=0A> flag is=0A> managed by th= e single-consumer buf_ring dequeue method=0A> (that=0A> drbr_dequeue() uses= ).=A0 It is set to true when a packet=0A> is removed from=0A> the queue whi= le there are more packets pending.=A0=0A> Conversely, if there=0A> are no o= ther pending packets then it is set to false.=A0=0A> The assumption=0A> is = that once a thread starts draining the queue, it will not=0A> stop=0A> unti= l the queue is empty (or if it has to stop for some=0A> other reason=0A> su= ch as the transmit ring being full, the driver will=0A> restart draining=0A= > of the queue until it is empty, e.g. after it receives a=0A> transmit=0A>= completion interrupt).=A0 Now when the if_transmit=0A> routine enqueues th= e=0A> packet, it will get either a real error, 0 if the packet was=0A> enqu= eued=0A> and the queue was not idle, or EINPROGRESS if the packet was=0A> e= nqueued=0A> and the queue was busy.=A0 For the EINPROGRESS case the=0A> if_= transmit=0A> routine just returns success.=A0 For the 0 case it does a=0A> = blocking lock=0A> on the queue lock and calls the queue's start routine (no= te=0A> that this=0A> means that the busy flag is similar to the old OACTIVE= =0A> interface=0A> flag).=A0 This does mean that in some cases you may have= =0A> one thread that=0A> is sending what was the last packet in the buf_rin= g holding=0A> the lock=0A> when another thread blocks, and that the first t= hread will=0A> see the new=0A> packet when it loops back around so that the= second thread=0A> is wasting=0A> it's time spinning, but in the common cas= e I believe it will=0A> give the=0A> same parallelism as the current code.= =A0 OTOH, there is=0A> nothing to=0A> prevent multiple threads from "stacki= ng up" in the new=0A> approach.=A0 At=0A> least the try_fail3 patch ensured= only one thread at a time=0A> would ever=0A> potentially block on the queu= e lock.=0A> =0A> Another approach might be to replace the 'busy' flag with= =0A> the 'thread=0A> pending' flag from try_fail3.patch, but to clear the '= thread=0A> pending'=0A> flag anytime the dequeue method is called rather th= an using=0A> an=0A> explicit 'clear pending' method.=A0 (Hadn't thought of= =0A> that until=0A> writing this e-mail.)=A0 That would prevent multiple=0A= > threads from=0A> waiting on the queue lock perhaps.=0A> =0A> Note that th= e 'busy' approach (or the modification I=0A> mentioned above)=0A> does rely= on the assumption I stated above, i.e. once a=0A> driver starts=0A> draini= ng a queue, it will drain it until empty unless it=0A> hits an=0A> "error" = condition (link went down, transmit ring full,=0A> etc.).=A0 If it=0A> hits= an "error" condition, the driver is responsible for=0A> restarting=0A> tra= nsmit when the condition clears.=A0 I believe our=0A> drivers already=0A> w= ork this way now.=0A> =0A> The 'busy' patch is at http://www.freebsd.org/~j= hb/patches/drbr.patch=0A> =0A> -- =0A> John Baldwin=0A=0AQ1: Has this been = corrected?=0A=0AQ2: Are there any case studies or benchmarks for buf_ring, = or it is just=0Ablindly being used because someone claimed it was better an= d offered it=0Afor free? One of the points of locking is to avoid race cond= itions, so the fact that you have races in a supposed lock-less scheme seem= s more than just ironic.=0A=0A=0ABC From owner-freebsd-net@FreeBSD.ORG Sun Nov 18 21:49:42 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6ADAB9BC for ; Sun, 18 Nov 2012 21:49:42 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id 971C58FC18 for ; Sun, 18 Nov 2012 21:49:41 +0000 (UTC) Received: (qmail 71061 invoked from network); 18 Nov 2012 23:22:53 -0000 Received: from c00l3r.networx.ch (HELO [127.0.0.1]) ([62.48.2.2]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 18 Nov 2012 23:22:53 -0000 Message-ID: <50A957F3.7020003@freebsd.org> Date: Sun, 18 Nov 2012 22:49:39 +0100 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20121026 Thunderbird/16.0.2 MIME-Version: 1.0 To: Barney Cordoba Subject: Re: Latency issues with buf_ring References: <1353259441.19423.YahooMailClassic@web121605.mail.ne1.yahoo.com> In-Reply-To: <1353259441.19423.YahooMailClassic@web121605.mail.ne1.yahoo.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, John Baldwin X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Nov 2012 21:49:42 -0000 On 18.11.2012 18:24, Barney Cordoba wrote: > > > --- On Thu, 1/19/12, John Baldwin wrote: > >> From: John Baldwin >> Subject: Latency issues with buf_ring >> To: net@freebsd.org >> Cc: "Ed Maste" , "Navdeep Parhar" >> Date: Thursday, January 19, 2012, 11:41 AM >> The current buf_ring usage in various >> NIC drivers has a race that can >> result in really high packet latencies in some cases. >> Specifically, >> the common pattern in an if_transmit routine is to use a >> try-lock on >> the queue and if that fails enqueue the packet in the >> buf_ring and >> return. The race, of course, is that the thread >> holding the lock >> might have just finished checking the buf_ring and found it >> empty and >> be in the process of releasing the lock when the original >> thread fails >> the try lock. If this happens, then the packet queued >> by the first >> thread will be stalled until another thread tries to >> transmit packets >> for that queue. Some drivers attempt to handle this >> race (igb(4) >> schedules a task to kick the transmit queue if the try lock >> fails) and >> others don't (cxgb(4) doesn't handle it at all). At >> work this race >> was triggered very often after upgrading from 7 to 8 with >> bursty >> traffic and caused numerous problems, so it is not a rare >> occurrence >> and needs to be addressed. >> >> (Note, all patches mentioned are against 8) >> >> The first hack I tried to use was to simply always lock the >> queue after >> the drbr enqueue if the try lock failed and then drain the >> queue if >> needed (www.freebsd.org/~jhb/patches/try_fail.patch). >> While this fixed >> my latency problems, it would seem that this breaks other >> workloads >> that the drbr design is trying to optimize. >> >> After further hacking what I came up with was a variant of >> drbr_enqueue() >> that would atomically set a 'pending' flag. During the >> enqueue operation. >> The first thread to fail the try lock sets this flag (it is >> told that it >> set the flag by a new return value (EINPROGRESS) from the >> enqueue call). >> The pending thread then explicitly clears the flag once it >> acquires the >> queue lock. This should prevent multiple threads from >> stacking up on the >> queue lock so that if multiple threads are dumping packets >> into the ring >> concurrently all but two (the one draining the queue >> currently and the >> one waiting for the lock) can continue to drain the >> queue. One downside >> of this approach though is that each driver has to be >> changed to make >> an explicit call to clear the pending flag after grabbing >> the queue lock >> if the try lock fails. This is what I am currently >> running in production >> (www.freebsd.org/~jhb/patches/try_fail3.patch). >> >> However, this still results in a lot of duplicated code in >> each driver >> that wants to support multiq. Several folks have >> expressed a desire >> to move in a direction where the stack has explicit >> knowledge of >> transmit queues allowing us to hoist some of this duplicated >> code out >> of the drivers and up into the calling layer. After >> discussing this a >> bit with Navdeep (np@), the approach I am looking at is to >> alter the >> buf_ring code flow a bit to more closely model the older >> code-flow >> with IFQ and if_start methods. That is, have the >> if_transmit methods >> always enqueue each packet that arrives to the buf_ring and >> then to >> call an if_start-like method that drains a specific transmit >> queue. >> This approach simplifies a fair bit of driver code and means >> we can >> potentially move the enqueue, etc. bits up into the calling >> layer and >> instead have drivers provide the per-transmit queue start >> routine as >> the direct function pointer to the upper layers ala >> if_start. >> >> However, we would still need a way to close the latency >> race. I've >> attempted to do that by inverting my previous 'thread >> pending' flag. >> Instead, I make the buf_ring store a 'busy' flag. This >> flag is >> managed by the single-consumer buf_ring dequeue method >> (that >> drbr_dequeue() uses). It is set to true when a packet >> is removed from >> the queue while there are more packets pending. >> Conversely, if there >> are no other pending packets then it is set to false. >> The assumption >> is that once a thread starts draining the queue, it will not >> stop >> until the queue is empty (or if it has to stop for some >> other reason >> such as the transmit ring being full, the driver will >> restart draining >> of the queue until it is empty, e.g. after it receives a >> transmit >> completion interrupt). Now when the if_transmit >> routine enqueues the >> packet, it will get either a real error, 0 if the packet was >> enqueued >> and the queue was not idle, or EINPROGRESS if the packet was >> enqueued >> and the queue was busy. For the EINPROGRESS case the >> if_transmit >> routine just returns success. For the 0 case it does a >> blocking lock >> on the queue lock and calls the queue's start routine (note >> that this >> means that the busy flag is similar to the old OACTIVE >> interface >> flag). This does mean that in some cases you may have >> one thread that >> is sending what was the last packet in the buf_ring holding >> the lock >> when another thread blocks, and that the first thread will >> see the new >> packet when it loops back around so that the second thread >> is wasting >> it's time spinning, but in the common case I believe it will >> give the >> same parallelism as the current code. OTOH, there is >> nothing to >> prevent multiple threads from "stacking up" in the new >> approach. At >> least the try_fail3 patch ensured only one thread at a time >> would ever >> potentially block on the queue lock. >> >> Another approach might be to replace the 'busy' flag with >> the 'thread >> pending' flag from try_fail3.patch, but to clear the 'thread >> pending' >> flag anytime the dequeue method is called rather than using >> an >> explicit 'clear pending' method. (Hadn't thought of >> that until >> writing this e-mail.) That would prevent multiple >> threads from >> waiting on the queue lock perhaps. >> >> Note that the 'busy' approach (or the modification I >> mentioned above) >> does rely on the assumption I stated above, i.e. once a >> driver starts >> draining a queue, it will drain it until empty unless it >> hits an >> "error" condition (link went down, transmit ring full, >> etc.). If it >> hits an "error" condition, the driver is responsible for >> restarting >> transmit when the condition clears. I believe our >> drivers already >> work this way now. >> >> The 'busy' patch is at http://www.freebsd.org/~jhb/patches/drbr.patch >> >> -- >> John Baldwin > > Q1: Has this been corrected? I don't know. > Q2: Are there any case studies or benchmarks for buf_ring, or it is just > blindly being used because someone claimed it was better and offered it > for free? One of the points of locking is to avoid race conditions, so > the fact that you have races in a supposed lock-less scheme seems more > than just ironic. At the moment our entire ifnet/driver handoff is not optimal. Especially the double buffering in the DMA rings and the IFQ (be it the old one or buf_ring) is troubling. See "Buffer Bloat" for one aspect of it. While hacking up a couple of drivers for automatic irq/polling experimenting I've come up with a couple of ideas in this area that I'm implementing right now. When it's ready meaningful benchmarks will be run to assess the impact. -- Andre From owner-freebsd-net@FreeBSD.ORG Mon Nov 19 11:06:48 2012 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id ACE7F20F for ; Mon, 19 Nov 2012 11:06:48 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 902638FC22 for ; Mon, 19 Nov 2012 11:06:48 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id qAJB6mae013376 for ; Mon, 19 Nov 2012 11:06:48 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id qAJB6mGo013374 for freebsd-net@FreeBSD.org; Mon, 19 Nov 2012 11:06:48 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 19 Nov 2012 11:06:48 GMT Message-Id: <201211191106.qAJB6mGo013374@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-net@FreeBSD.org Subject: Current problem reports assigned to freebsd-net@FreeBSD.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Nov 2012 11:06:48 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/173481 net [NFS] RH63 NFSv4 client does not reconnect to FreeBSD o kern/173479 net [nfs] chown and chgrp operations fail between FreeBSD o kern/173475 net [tun] tun(4) stays opened by PID after process is term o kern/173201 net [ixgbe] [patch] Missing / broken ixgbe sysctl's and tu o kern/173137 net [em] em(4) unable to run at gigabit with 9.1-RC2 o kern/173002 net [patch] data type size problem in if_spppsubr.c o kern/172985 net [patch] [ip6] lltable leak when adding and removing IP o kern/172895 net [ixgb] [ixgbe] do not properly determine link-state o kern/172683 net [ip6] Duplicate IPv6 Link Local Addresses o kern/172675 net [netinet] [patch] sysctl_tcp_hc_list (net.inet.tcp.hos o kern/172113 net [panic] [e1000] [patch] 9.1-RC1/amd64 panices in igb(4 o kern/171840 net [ip6] IPv6 packets transmitting only on queue 0 o kern/171838 net [oce] [patch] Possible lock reversal and duplicate loc o kern/171739 net [bce] [panic] bce related kernel panic o kern/171728 net [arp] arp issue o kern/171711 net [dummynet] [panic] Kernel panic in dummynet o kern/171697 net [ip6] [ndp] panic when changing routes o kern/171532 net [ndis] ndis(4) driver includes 'pccard'-specific code, o kern/171531 net [ndis] undocumented dependency for ndis(4) o kern/171524 net [ipmi] ipmi driver crashes kernel by reboot or shutdow s kern/171508 net [epair] [request] Add the ability to name epair device o kern/171228 net [re] [patch] if_re - eeprom write issues o kern/170701 net [ppp] killl ppp or reboot with active ppp connection c o kern/170267 net [ixgbe] IXGBE_LE32_TO_CPUS is probably an unintentiona o kern/170081 net [fxp] pf/nat/jails not working if checksum offloading o kern/169898 net ifconfig(8) fails to set MTU on multiple interfaces. o kern/169676 net [bge] [hang] system hangs, fully or partially after re o kern/169664 net [bgp] Wrongful replacement of interface connected net o kern/169620 net [ng] [pf] ng_l2tp incoming packet bypass pf firewall o kern/169459 net [ppp] umodem/ppp/3g stopped working after update from o kern/169438 net [ipsec] ipv4-in-ipv6 tunnel mode IPsec does not work p kern/168294 net [ixgbe] [patch] ixgbe driver compiled in kernel has no o kern/168246 net [em] Multiple em(4) not working with qemu o kern/168245 net [arp] [regression] Permanent ARP entry not deleted on o kern/168244 net [arp] [regression] Unable to manually remove permanent o kern/168183 net [bce] bce driver hang system o kern/167947 net [setfib] [patch] arpresolve checks only the default FI o kern/167603 net [ip] IP fragment reassembly's broken: file transfer ov o kern/167500 net [em] [panic] Kernel panics in em driver o kern/167325 net [netinet] [patch] sosend sometimes return EINVAL with o kern/167202 net [igmp]: Sending multiple IGMP packets crashes kernel o kern/167059 net [tcp] [panic] System does panic in in_pcbbind() and ha o kern/166940 net [ipfilter] [panic] Double fault in kern 8.2 o kern/166462 net [gre] gre(4) when using a tunnel source address from c o kern/166372 net [patch] ipfilter drops UDP packets with zero checksum o kern/166285 net [arp] FreeBSD v8.1 REL p8 arp: unknown hardware addres o kern/166255 net [net] [patch] It should be possible to disable "promis o kern/165963 net [panic] [ipf] ipfilter/nat NULL pointer deference o kern/165903 net mbuf leak o kern/165643 net [net] [patch] Missing vnet restores in net/if_ethersub o kern/165622 net [ndis][panic][patch] Unregistered use of FPU in kernel s kern/165562 net [request] add support for Intel i350 in FreeBSD 7.4 o kern/165526 net [bxe] UDP packets checksum calculation whithin if_bxe o kern/165488 net [ppp] [panic] Fatal trap 12 jails and ppp , kernel wit o kern/165305 net [ip6] [request] Feature parity between IP_TOS and IPV6 o kern/165296 net [vlan] [patch] Fix EVL_APPLY_VLID, update EVL_APPLY_PR o kern/165181 net [igb] igb freezes after about 2 weeks of uptime o kern/165174 net [patch] [tap] allow tap(4) to keep its address on clos o kern/165152 net [ip6] Does not work through the issue of ipv6 addresse o kern/164495 net [igb] connect double head igb to switch cause system t o kern/164490 net [pfil] Incorrect IP checksum on pfil pass from ip_outp o kern/164475 net [gre] gre misses RUNNING flag after a reboot o kern/164265 net [netinet] [patch] tcp_lro_rx computes wrong checksum i o kern/163903 net [igb] "igb0:tx(0)","bpf interface lock" v2.2.5 9-STABL o kern/163481 net freebsd do not add itself to ping route packet o kern/162927 net [tun] Modem-PPP error ppp[1538]: tun0: Phase: Clearing o kern/162926 net [ipfilter] Infinite loop in ipfilter with fragmented I o kern/162558 net [dummynet] [panic] seldom dummynet panics o kern/162153 net [em] intel em driver 7.2.4 don't compile o kern/162110 net [igb] [panic] RELENG_9 panics on boot in IGB driver - o kern/162028 net [ixgbe] [patch] misplaced #endif in ixgbe.c o kern/161277 net [em] [patch] BMC cannot receive IPMI traffic after loa o kern/160873 net [igb] igb(4) from HEAD fails to build on 7-STABLE o kern/160750 net Intel PRO/1000 connection breaks under load until rebo o kern/160693 net [gif] [em] Multicast packet are not passed from GIF0 t o kern/160293 net [ieee80211] ppanic] kernel panic during network setup o kern/160206 net [gif] gifX stops working after a while (IPv6 tunnel) o kern/159817 net [udp] write UDPv4: No buffer space available (code=55) o kern/159629 net [ipsec] [panic] kernel panic with IPsec in transport m o kern/159621 net [tcp] [panic] panic: soabort: so_count o kern/159603 net [netinet] [patch] in_ifscrubprefix() - network route c o kern/159601 net [netinet] [patch] in_scrubprefix() - loopback route re o kern/159294 net [em] em watchdog timeouts o kern/159203 net [wpi] Intel 3945ABG Wireless LAN not support IBSS o kern/158930 net [bpf] BPF element leak in ifp->bpf_if->bif_dlist o kern/158726 net [ip6] [patch] ICMPv6 Router Announcement flooding limi o kern/158694 net [ix] [lagg] ix0 is not working within lagg(4) o kern/158665 net [ip6] [panic] kernel pagefault in in6_setscope() o kern/158635 net [em] TSO breaks BPF packet captures with em driver f kern/157802 net [dummynet] [panic] kernel panic in dummynet o kern/157785 net amd64 + jail + ipfw + natd = very slow outbound traffi o kern/157418 net [em] em driver lockup during boot on Supermicro X9SCM- o kern/157410 net [ip6] IPv6 Router Advertisements Cause Excessive CPU U o kern/157287 net [re] [panic] INVARIANTS panic (Memory modified after f o kern/157209 net [ip6] [patch] locking error in rip6_input() (sys/netin o kern/157200 net [network.subr] [patch] stf(4) can not communicate betw o kern/157182 net [lagg] lagg interface not working together with epair o kern/156877 net [dummynet] [panic] dummynet move_pkt() null ptr derefe o kern/156667 net [em] em0 fails to init on CURRENT after March 17 o kern/156408 net [vlan] Routing failure when using VLANs vs. Physical e o kern/156328 net [icmp]: host can ping other subnet but no have IP from o kern/156317 net [ip6] Wrong order of IPv6 NS DAD/MLD Report o kern/156283 net [ip6] [patch] nd6_ns_input - rtalloc_mpath does not re o kern/156279 net [if_bridge][divert][ipfw] unable to correctly re-injec o kern/156226 net [lagg]: failover does not announce the failover to swi o kern/156030 net [ip6] [panic] Crash in nd6_dad_start() due to null ptr o kern/155772 net ifconfig(8): ioctl (SIOCAIFADDR): File exists on direc o kern/155680 net [multicast] problems with multicast s kern/155642 net [request] Add driver for Realtek RTL8191SE/RTL8192SE W o kern/155597 net [panic] Kernel panics with "sbdrop" message o kern/155420 net [vlan] adding vlan break existent vlan o kern/155177 net [route] [panic] Panic when inject routes in kernel p kern/155030 net [igb] igb(4) DEVICE_POLLING does not work with carp(4) o kern/155010 net [msk] ntfs-3g via iscsi using msk driver cause kernel o kern/154943 net [gif] ifconfig gifX create on existing gifX clears IP s kern/154851 net [request]: Port brcm80211 driver from Linux to FreeBSD o kern/154850 net [netgraph] [patch] ng_ether fails to name nodes when t o kern/154679 net [em] Fatal trap 12: "em1 taskq" only at startup (8.1-R o kern/154600 net [tcp] [panic] Random kernel panics on tcp_output o kern/154557 net [tcp] Freeze tcp-session of the clients, if in the gat o kern/154443 net [if_bridge] Kernel module bridgestp.ko missing after u o kern/154286 net [netgraph] [panic] 8.2-PRERELEASE panic in netgraph o kern/154255 net [nfs] NFS not responding o kern/154214 net [stf] [panic] Panic when creating stf interface o kern/154185 net race condition in mb_dupcl o kern/154169 net [multicast] [ip6] Node Information Query multicast add o kern/154134 net [ip6] stuck kernel state in LISTEN on ipv6 daemon whic o kern/154091 net [netgraph] [panic] netgraph, unaligned mbuf? o conf/154062 net [vlan] [patch] change to way of auto-generatation of v o kern/153937 net [ral] ralink panics the system (amd64 freeBSDD 8.X) wh o kern/153936 net [ixgbe] [patch] MPRC workaround incorrectly applied to o kern/153816 net [ixgbe] ixgbe doesn't work properly with the Intel 10g o kern/153772 net [ixgbe] [patch] sysctls reference wrong XON/XOFF varia o kern/153497 net [netgraph] netgraph panic due to race conditions o kern/153454 net [patch] [wlan] [urtw] Support ad-hoc and hostap modes o kern/153308 net [em] em interface use 100% cpu o kern/153244 net [em] em(4) fails to send UDP to port 0xffff o kern/152893 net [netgraph] [panic] 8.2-PRERELEASE panic in netgraph o kern/152853 net [em] tftpd (and likely other udp traffic) fails over e o kern/152828 net [em] poor performance on 8.1, 8.2-PRE o kern/152569 net [net]: Multiple ppp connections and routing table prob o kern/152235 net [arp] Permanent local ARP entries are not properly upd o kern/152141 net [vlan] [patch] encapsulate vlan in ng_ether before out o kern/152036 net [libc] getifaddrs(3) returns truncated sockaddrs for n o kern/151690 net [ep] network connectivity won't work until dhclient is o kern/151681 net [nfs] NFS mount via IPv6 leads to hang on client with o kern/151593 net [igb] [panic] Kernel panic when bringing up igb networ o kern/150920 net [ixgbe][igb] Panic when packets are dropped with heade o kern/150557 net [igb] igb0: Watchdog timeout -- resetting o kern/150251 net [patch] [ixgbe] Late cable insertion broken o kern/150249 net [ixgbe] Media type detection broken o bin/150224 net ppp(8) does not reassign static IP after kill -KILL co f kern/149969 net [wlan] [ral] ralink rt2661 fails to maintain connectio o kern/149937 net [ipfilter] [patch] kernel panic in ipfilter IP fragmen o kern/149643 net [rum] device not sending proper beacon frames in ap mo o kern/149609 net [panic] reboot after adding second default route o kern/149117 net [inet] [patch] in_pcbbind: redundant test o kern/149086 net [multicast] Generic multicast join failure in 8.1 o kern/148018 net [flowtable] flowtable crashes on ia64 o kern/147912 net [boot] FreeBSD 8 Beta won't boot on Thinkpad i1300 11 o kern/147894 net [ipsec] IPv6-in-IPv4 does not work inside an ESP-only o kern/147155 net [ip6] setfb not work with ipv6 o kern/146845 net [libc] close(2) returns error 54 (connection reset by f kern/146792 net [flowtable] flowcleaner 100% cpu's core load o kern/146719 net [pf] [panic] PF or dumynet kernel panic o kern/146534 net [icmp6] wrong source address in echo reply o kern/146427 net [mwl] Additional virtual access points don't work on m f kern/146394 net [vlan] IP source address for outgoing connections o bin/146377 net [ppp] [tun] Interface doesn't clear addresses when PPP o kern/146358 net [vlan] wrong destination MAC address o kern/146165 net [wlan] [panic] Setting bssid in adhoc mode causes pani o kern/146082 net [ng_l2tp] a false invaliant check was performed in ng_ o kern/146037 net [panic] mpd + CoA = kernel panic o kern/145825 net [panic] panic: soabort: so_count o kern/145728 net [lagg] Stops working lagg between two servers. p kern/145600 net TCP/ECN behaves different to CE/CWR than ns2 reference f kern/144917 net [flowtable] [panic] flowtable crashes system [regressi o kern/144882 net MacBookPro =>4.1 does not connect to BSD in hostap wit o kern/144874 net [if_bridge] [patch] if_bridge frees mbuf after pfil ho o conf/144700 net [rc.d] async dhclient breaks stuff for too many people o kern/144616 net [nat] [panic] ip_nat panic FreeBSD 7.2 f kern/144315 net [ipfw] [panic] freebsd 8-stable reboot after add ipfw o kern/144231 net bind/connect/sendto too strict about sockaddr length o kern/143846 net [gif] bringing gif3 tunnel down causes gif0 tunnel to s kern/143673 net [stf] [request] there should be a way to support multi s kern/143666 net [ip6] [request] PMTU black hole detection not implemen o kern/143622 net [pfil] [patch] unlock pfil lock while calling firewall o kern/143593 net [ipsec] When using IPSec, tcpdump doesn't show outgoin o kern/143591 net [ral] RT2561C-based DLink card (DWL-510) fails to work o kern/143208 net [ipsec] [gif] IPSec over gif interface not working o kern/143034 net [panic] system reboots itself in tcp code [regression] o kern/142877 net [hang] network-related repeatable 8.0-STABLE hard hang o kern/142774 net Problem with outgoing connections on interface with mu o kern/142772 net [libc] lla_lookup: new lle malloc failed f kern/142518 net [em] [lagg] Problem on 8.0-STABLE with em and lagg o kern/142018 net [iwi] [patch] Possibly wrong interpretation of beacon- o kern/141861 net [wi] data garbled with WEP and wi(4) with Prism 2.5 f kern/141741 net Etherlink III NIC won't work after upgrade to FBSD 8, o kern/140742 net rum(4) Two asus-WL167G adapters cannot talk to each ot o kern/140682 net [netgraph] [panic] random panic in netgraph f kern/140634 net [vlan] destroying if_lagg interface with if_vlan membe o kern/140619 net [ifnet] [patch] refine obsolete if_var.h comments desc o kern/140346 net [wlan] High bandwidth use causes loss of wlan connecti o kern/140142 net [ip6] [panic] FreeBSD 7.2-amd64 panic w/IPv6 o kern/140066 net [bwi] install report for 8.0 RC 2 (multiple problems) o kern/139565 net [ipfilter] ipfilter ioctl SIOCDELST broken o kern/139387 net [ipsec] Wrong lenth of PF_KEY messages in promiscuous o bin/139346 net [patch] arp(8) add option to remove static entries lis o kern/139268 net [if_bridge] [patch] allow if_bridge to forward just VL p kern/139204 net [arp] DHCP server replies rejected, ARP entry lost bef o kern/139117 net [lagg] + wlan boot timing (EBUSY) o kern/139058 net [ipfilter] mbuf cluster leak on FreeBSD 7.2 o kern/138850 net [dummynet] dummynet doesn't work correctly on a bridge o kern/138782 net [panic] sbflush_internal: cc 0 || mb 0xffffff004127b00 o kern/138688 net [rum] possibly broken on 8 Beta 4 amd64: able to wpa a o kern/138678 net [lo] FreeBSD does not assign linklocal address to loop o kern/138407 net [gre] gre(4) interface does not come up after reboot o kern/138332 net [tun] [lor] ifconfig tun0 destroy causes LOR if_adata/ o kern/138266 net [panic] kernel panic when udp benchmark test used as r o kern/138177 net [ipfilter] FreeBSD crashing repeatedly in ip_nat.c:257 f kern/138029 net [bpf] [panic] periodically kernel panic and reboot o kern/137881 net [netgraph] [panic] ng_pppoe fatal trap 12 p bin/137841 net [patch] wpa_supplicant(8) cannot verify SHA256 signed p kern/137776 net [rum] panic in rum(4) driver on 8.0-BETA2 o bin/137641 net ifconfig(8): various problems with "vlan_device.vlan_i o kern/137392 net [ip] [panic] crash in ip_nat.c line 2577 o kern/137372 net [ral] FreeBSD doesn't support wireless interface from o kern/137089 net [lagg] lagg falsely triggers IPv6 duplicate address de o bin/136994 net [patch] ifconfig(8) print carp mac address o kern/136911 net [netgraph] [panic] system panic on kldload ng_bpf.ko t o kern/136618 net [pf][stf] panic on cloning interface without unit numb o kern/135502 net [periodic] Warning message raised by rtfree function i o kern/134583 net [hang] Machine with jail freezes after random amount o o kern/134531 net [route] [panic] kernel crash related to routes/zebra o kern/134157 net [dummynet] dummynet loads cpu for 100% and make a syst o kern/133969 net [dummynet] [panic] Fatal trap 12: page fault while in o kern/133968 net [dummynet] [panic] dummynet kernel panic o kern/133736 net [udp] ip_id not protected ... o kern/133595 net [panic] Kernel Panic at pcpu.h:195 o kern/133572 net [ppp] [hang] incoming PPTP connection hangs the system o kern/133490 net [bpf] [panic] 'kmem_map too small' panic on Dell r900 o kern/133235 net [netinet] [patch] Process SIOCDLIFADDR command incorre f kern/133213 net arp and sshd errors on 7.1-PRERELEASE o kern/133060 net [ipsec] [pfsync] [panic] Kernel panic with ipsec + pfs o kern/132889 net [ndis] [panic] NDIS kernel crash on load BCM4321 AGN d o conf/132851 net [patch] rc.conf(5): allow to setfib(1) for service run o kern/132734 net [ifmib] [panic] panic in net/if_mib.c o kern/132705 net [libwrap] [patch] libwrap - infinite loop if hosts.all o kern/132672 net [ndis] [panic] ndis with rt2860.sys causes kernel pani o kern/132554 net [ipl] There is no ippool start script/ipfilter magic t o kern/132354 net [nat] Getting some packages to ipnat(8) causes crash o kern/132277 net [crypto] [ipsec] poor performance using cryptodevice f o kern/131781 net [ndis] ndis keeps dropping the link o kern/131776 net [wi] driver fails to init o kern/131753 net [altq] [panic] kernel panic in hfsc_dequeue o kern/131601 net [ipfilter] [panic] 7-STABLE panic in nat_finalise (tcp o bin/131567 net [socket] [patch] Update for regression/sockets/unix_cm o bin/131365 net route(8): route add changes interpretation of network f kern/130820 net [ndis] wpa_supplicant(8) returns 'no space on device' o kern/130628 net [nfs] NFS / rpc.lockd deadlock on 7.1-R o conf/130555 net [rc.d] [patch] No good way to set ipfilter variables a o kern/130525 net [ndis] [panic] 64 bit ar5008 ndisgen-erated driver cau o kern/130311 net [wlan_xauth] [panic] hostapd restart causing kernel pa o kern/130109 net [ipfw] Can not set fib for packets originated from loc f kern/130059 net [panic] Leaking 50k mbufs/hour f kern/129719 net [nfs] [panic] Panic during shutdown, tcp_ctloutput: in o kern/129517 net [ipsec] [panic] double fault / stack overflow f kern/129508 net [carp] [panic] Kernel panic with EtherIP (may be relat o kern/129219 net [ppp] Kernel panic when using kernel mode ppp o kern/129197 net [panic] 7.0 IP stack related panic o bin/128954 net ifconfig(8) deletes valid routes o bin/128602 net [an] wpa_supplicant(8) crashes with an(4) o kern/128448 net [nfs] 6.4-RC1 Boot Fails if NFS Hostname cannot be res o bin/128295 net [patch] ifconfig(8) does not print TOE4 or TOE6 capabi o bin/128001 net wpa_supplicant(8), wlan(4), and wi(4) issues o kern/127826 net [iwi] iwi0 driver has reduced performance and connecti o kern/127815 net [gif] [patch] if_gif does not set vlan attributes from o kern/127724 net [rtalloc] rtfree: 0xc5a8f870 has 1 refs f bin/127719 net [arp] arp: Segmentation fault (core dumped) f kern/127528 net [icmp]: icmp socket receives icmp replies not owned by p kern/127360 net [socket] TOE socket options missing from sosetopt() o bin/127192 net routed(8) removes the secondary alias IP of interface f kern/127145 net [wi]: prism (wi) driver crash at bigger traffic o kern/126895 net [patch] [ral] Add antenna selection (marked as TBD) o kern/126874 net [vlan]: Zebra problem if ifconfig vlanX destroy o kern/126695 net rtfree messages and network disruption upon use of if_ o kern/126339 net [ipw] ipw driver drops the connection o kern/126075 net [inet] [patch] internet control accesses beyond end of o bin/125922 net [patch] Deadlock in arp(8) o kern/125920 net [arp] Kernel Routing Table loses Ethernet Link status o kern/125845 net [netinet] [patch] tcp_lro_rx() should make use of hard o kern/125258 net [socket] socket's SO_REUSEADDR option does not work o kern/125239 net [gre] kernel crash when using gre o kern/124341 net [ral] promiscuous mode for wireless device ral0 looses o kern/124225 net [ndis] [patch] ndis network driver sometimes loses net o kern/124160 net [libc] connect(2) function loops indefinitely o kern/124021 net [ip6] [panic] page fault in nd6_output() o kern/123968 net [rum] [panic] rum driver causes kernel panic with WPA. o kern/123892 net [tap] [patch] No buffer space available o kern/123890 net [ppp] [panic] crash & reboot on work with PPP low-spee o kern/123858 net [stf] [patch] stf not usable behind a NAT o kern/123796 net [ipf] FreeBSD 6.1+VPN+ipnat+ipf: port mapping does not o kern/123758 net [panic] panic while restarting net/freenet6 o bin/123633 net ifconfig(8) doesn't set inet and ether address in one o kern/123559 net [iwi] iwi periodically disassociates/associates [regre o bin/123465 net [ip6] route(8): route add -inet6 -interfac o kern/123463 net [ipsec] [panic] repeatable crash related to ipsec-tool o conf/123330 net [nsswitch.conf] Enabling samba wins in nsswitch.conf c o kern/123160 net [ip] Panic and reboot at sysctl kern.polling.enable=0 o kern/122989 net [swi] [panic] 6.3 kernel panic in swi1: net o kern/122954 net [lagg] IPv6 EUI64 incorrectly chosen for lagg devices f kern/122780 net [lagg] tcpdump on lagg interface during high pps wedge o kern/122685 net It is not visible passing packets in tcpdump(1) o kern/122319 net [wi] imposible to enable ad-hoc demo mode with Orinoco o kern/122290 net [netgraph] [panic] Netgraph related "kmem_map too smal o kern/122252 net [ipmi] [bge] IPMI problem with BCM5704 (does not work o kern/122033 net [ral] [lor] Lock order reversal in ral0 at bootup ieee o bin/121895 net [patch] rtsol(8)/rtsold(8) doesn't handle managed netw s kern/121774 net [swi] [panic] 6.3 kernel panic in swi1: net o kern/121555 net [panic] Fatal trap 12: current process = 12 (swi1: net o kern/121443 net [gif] [lor] icmp6_input/nd6_lookup o kern/121437 net [vlan] Routing to layer-2 address does not work on VLA o bin/121359 net [patch] [security] ppp(8): fix local stack overflow in o kern/121257 net [tcp] TSO + natd -> slow outgoing tcp traffic o kern/121181 net [panic] Fatal trap 3: breakpoint instruction fault whi o kern/120966 net [rum] kernel panic with if_rum and WPA encryption o kern/120566 net [request]: ifconfig(8) make order of arguments more fr o kern/120304 net [netgraph] [patch] netgraph source assumes 32-bit time o kern/120266 net [udp] [panic] gnugk causes kernel panic when closing U o bin/120060 net routed(8) deletes link-level routes in the presence of o kern/119945 net [rum] [panic] rum device in hostap mode, cause kernel o kern/119791 net [nfs] UDP NFS mount of aliased IP addresses from a Sol o kern/119617 net [nfs] nfs error on wpa network when reseting/shutdown f kern/119516 net [ip6] [panic] _mtx_lock_sleep: recursed on non-recursi o kern/119432 net [arp] route add -host -iface causes arp e o kern/119225 net [wi] 7.0-RC1 no carrier with Prism 2.5 wifi card [regr o kern/118727 net [netgraph] [patch] [request] add new ng_pf module o kern/117423 net [vlan] Duplicate IP on different interfaces o bin/117339 net [patch] route(8): loading routing management commands o bin/116643 net [patch] [request] fstat(1): add INET/INET6 socket deta o kern/116185 net [iwi] if_iwi driver leads system to reboot o kern/115239 net [ipnat] panic with 'kmem_map too small' using ipnat o kern/115019 net [netgraph] ng_ether upper hook packet flow stops on ad o kern/115002 net [wi] if_wi timeout. failed allocation (busy bit). ifco o kern/114915 net [patch] [pcn] pcn (sys/pci/if_pcn.c) ethernet driver f o kern/113432 net [ucom] WARNING: attempt to net_add_domain(netgraph) af o kern/112722 net [ipsec] [udp] IP v4 udp fragmented packet reject o kern/112686 net [patm] patm driver freezes System (FreeBSD 6.2-p4) i38 o bin/112557 net [patch] ppp(8) lock file should not use symlink name o kern/112528 net [nfs] NFS over TCP under load hangs with "impossible p o kern/111537 net [inet6] [patch] ip6_input() treats mbuf cluster wrong o kern/111457 net [ral] ral(4) freeze o kern/110284 net [if_ethersubr] Invalid Assumption in SIOCSIFADDR in et o kern/110249 net [kernel] [regression] [patch] setsockopt() error regre o kern/109470 net [wi] Orinoco Classic Gold PC Card Can't Channel Hop o bin/108895 net pppd(8): PPPoE dead connections on 6.2 [regression] o kern/107944 net [wi] [patch] Forget to unlock mutex-locks o conf/107035 net [patch] bridge(8): bridge interface given in rc.conf n o kern/106444 net [netgraph] [panic] Kernel Panic on Binding to an ip to o kern/106316 net [dummynet] dummynet with multipass ipfw drops packets o kern/105945 net Address can disappear from network interface s kern/105943 net Network stack may modify read-only mbuf chain copies o bin/105925 net problems with ifconfig(8) and vlan(4) [regression] o kern/104851 net [inet6] [patch] On link routes not configured when usi o kern/104751 net [netgraph] kernel panic, when getting info about my tr o kern/103191 net Unpredictable reboot o kern/103135 net [ipsec] ipsec with ipfw divert (not NAT) encodes a pac o kern/102540 net [netgraph] [patch] supporting vlan(4) by ng_fec(4) o conf/102502 net [netgraph] [patch] ifconfig name does't rename netgrap o kern/102035 net [plip] plip networking disables parallel port printing o kern/101948 net [ipf] [panic] Kernel Panic Trap No 12 Page Fault - cau o kern/100709 net [libc] getaddrinfo(3) should return TTL info o kern/100519 net [netisr] suggestion to fix suboptimal network polling o kern/98978 net [ipf] [patch] ipfilter drops OOW packets under 6.1-Rel o kern/98597 net [inet6] Bug in FreeBSD 6.1 IPv6 link-local DAD procedu o bin/98218 net wpa_supplicant(8) blacklist not working o kern/97306 net [netgraph] NG_L2TP locks after connection with failed o conf/97014 net [gif] gifconfig_gif? in rc.conf does not recognize IPv f kern/96268 net [socket] TCP socket performance drops by 3000% if pack o kern/95519 net [ral] ral0 could not map mbuf o kern/95288 net [pppd] [tty] [panic] if_ppp panic in sys/kern/tty_subr o kern/95277 net [netinet] [patch] IP Encapsulation mask_match() return o kern/95267 net packet drops periodically appear f kern/93378 net [tcp] Slow data transfer in Postfix and Cyrus IMAP (wo o kern/93019 net [ppp] ppp and tunX problems: no traffic after restarti o kern/92880 net [libc] [patch] almost rewritten inet_network(3) functi s kern/92279 net [dc] Core faults everytime I reboot, possible NIC issu o kern/91859 net [ndis] if_ndis does not work with Asus WL-138 s kern/91777 net [ipf] [patch] wrong behaviour with skip rule inside an o kern/91364 net [ral] [wep] WF-511 RT2500 Card PCI and WEP o kern/91311 net [aue] aue interface hanging o kern/87521 net [ipf] [panic] using ipfilter "auth" keyword leads to k o kern/87421 net [netgraph] [panic]: ng_ether + ng_eiface + if_bridge o kern/86871 net [tcp] [patch] allocation logic for PCBs in TIME_WAIT s o kern/86427 net [lor] Deadlock with FASTIPSEC and nat o kern/86103 net [ipf] Illegal NAT Traversal in IPFilter o kern/85780 net 'panic: bogus refcnt 0' in routing/ipv6 o bin/85445 net ifconfig(8): deprecated keyword to ifconfig inoperativ p kern/85320 net [gre] [patch] possible depletion of kernel stack in ip o bin/82975 net route change does not parse classfull network as given o kern/82881 net [netgraph] [panic] ng_fec(4) causes kernel panic after o kern/82468 net Using 64MB tcp send/recv buffers, trafficflow stops, i o bin/82185 net [patch] ndp(8) can delete the incorrect entry o kern/81095 net IPsec connection stops working if associated network i o kern/78968 net FreeBSD freezes on mbufs exhaustion (network interface o kern/78090 net [ipf] ipf filtering on bridged packets doesn't work if o kern/77341 net [ip6] problems with IPV6 implementation s kern/77195 net [ipf] [patch] ipfilter ioctl SIOCGNATL does not match o kern/75873 net Usability problem with non-RFC-compliant IP spoof prot s kern/75407 net [an] an(4): no carrier after short time a kern/71474 net [route] route lookup does not skip interfaces marked d o kern/71469 net default route to internet magically disappears with mu o kern/70904 net [ipf] ipfilter ipnat problem with h323 proxy support o kern/68889 net [panic] m_copym, length > size of mbuf chain o kern/66225 net [netgraph] [patch] extend ng_eiface(4) control message o kern/65616 net IPSEC can't detunnel GRE packets after real ESP encryp s kern/60293 net [patch] FreeBSD arp poison patch a kern/56233 net IPsec tunnel (ESP) over IPv6: MTU computation is wrong s bin/41647 net ifconfig(8) doesn't accept lladdr along with inet addr o kern/39937 net ipstealth issue a kern/38554 net [patch] changing interface ipaddress doesn't seem to w o kern/34665 net [ipf] [hang] ipfilter rcmd proxy "hangs". o kern/31940 net ip queue length too short for >500kpps o kern/31647 net [libc] socket calls can return undocumented EINVAL o kern/30186 net [libc] getaddrinfo(3) does not handle incorrect servna o kern/27474 net [ipf] [ppp] Interactive use of user PPP and ipfilter c f kern/24959 net [patch] proper TCP_NOPUSH/TCP_CORK compatibility o conf/23063 net [arp] [patch] for static ARP tables in rc.network o kern/21998 net [socket] [patch] ident only for outgoing connections o kern/5877 net [socket] sb_cc counts control data as well as data dat 430 problems total. From owner-freebsd-net@FreeBSD.ORG Mon Nov 19 19:35:54 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0AC54B9D for ; Mon, 19 Nov 2012 19:35:54 +0000 (UTC) (envelope-from vijju.singh@gmail.com) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 921918FC08 for ; Mon, 19 Nov 2012 19:35:53 +0000 (UTC) Received: by mail-ee0-f54.google.com with SMTP id c13so3790676eek.13 for ; Mon, 19 Nov 2012 11:35:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=pv7dJ+HqTg340KwIr19bdTAr12zUc6nm6jOiqdY1Mw0=; b=TxDZUtfyW/L6znBs1W6kmUTIYH2p9NDPHpZmU2kACH24M/VoR82psbdyeXFTPZdxM0 KWSe3P71w7/murvkOHBr4CETeZ2LawRAW8YvYT+4UFEF27sM20v5C8ijYd0XNnrUckVq W13ki9pNKqxRMHQrf98KNKoEuDPWIvyS7azrFreA4fBM77x7WRcDDJxThjfifNrZ9hHv EbElwgys+ihYCkPZRDD1PH/8qDv/zTaf/KJm46Q/aj6Ev1By82NEJPlgpfP9vc/EBL3P f/fqop18sqUWQRnulBWeWhqnglhyVqhNsS+HhTeB6iXwhvMN6eiwGC14bdn1q5ASgB8E 15zA== MIME-Version: 1.0 Received: by 10.14.203.2 with SMTP id e2mr4209775eeo.20.1353353752234; Mon, 19 Nov 2012 11:35:52 -0800 (PST) Received: by 10.14.176.7 with HTTP; Mon, 19 Nov 2012 11:35:52 -0800 (PST) Date: Mon, 19 Nov 2012 11:35:52 -0800 Message-ID: Subject: Intel 1G (82571EB) link bouncing constantly From: Vijay Singh To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Nov 2012 19:35:54 -0000 Hi, I am on 8.2, and I see that the link on the 1G ports is constantly bouncing. I have verified cabling etc, and this is seen on multiple setups. The NIC exhibiting this is: em1@pci0:8:0:1: class=0x020000 card=0x11a48086 chip=0x10a48086 rev=0x06 hdr=0x00 vendor = 'Intel Corporation' device = '82571EB Gigabit Ethernet Controller' class = network subclass = ethernet em2@pci0:9:0:0: class=0x020000 card=0x11a48086 chip=0x10a48086 rev=0x06 hdr=0x00 vendor = 'Intel Corporation' device = '82571EB Gigabit Ethernet Controller' class = network subclass = ethernet ... got message of size 168 on Mon Nov 19 13:50:25 2012 RTM_IFINFO: iface status change: len 168, if# 12, link: up, flags: got message of size 168 on Mon Nov 19 13:50:25 2012 RTM_IFINFO: iface status change: len 168, if# 11, link: down, flags: got message of size 168 on Mon Nov 19 13:50:25 2012 RTM_IFINFO: iface status change: len 168, if# 12, link: up, flags: got message of size 168 on Mon Nov 19 13:50:25 2012 RTM_IFINFO: iface status change: len 168, if# 12, link: up, flags: got message of size 168 on Mon Nov 19 13:50:25 2012 RTM_IFINFO: iface status change: len 168, if# 12, link: down, flags: got message of size 168 on Mon Nov 19 13:50:25 2012 RTM_IFINFO: iface status change: len 168, if# 11, link: down, flags: ... Has anyone else seen this? -vijay From owner-freebsd-net@FreeBSD.ORG Mon Nov 19 19:57:34 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B72D55FF for ; Mon, 19 Nov 2012 19:57:34 +0000 (UTC) (envelope-from fodillemlinkarim@gmail.com) Received: from mail-ia0-f182.google.com (mail-ia0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 763068FC14 for ; Mon, 19 Nov 2012 19:57:34 +0000 (UTC) Received: by mail-ia0-f182.google.com with SMTP id x2so4651410iad.13 for ; Mon, 19 Nov 2012 11:57:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject :content-type; bh=HeUZu3FgEwMPfPbfEXqwmcboCkrnMeNM2IVeY7x5G+U=; b=G57051HJuNkKeOXWols8hCEK6azvRkfkjAHh+p16Z7YBHuLXH2NJ83KA+bR0djWnxe Qx7MWlqjPFlYYzyMnIyCmtZDL6c8/2KyxSWS1H8nJV/1A3TPLU5YLAp/WesCAWuPvzL3 btrxvzC0X+YRb6xFw5nhT27qrd+Wx1cV/YeIq7fWy4u+Ra5qCTXNbNbIDeMGJGm7dTWa N4KE/Q2UZfadW8xjqxqtIl3ucAaeThUUaqioMJ5ob9J0O0kwUtKh8RovjH3mJJVhFF3l Sb1X/sExxK18xK5jDjTxAIWAJpG8VsbGixLsFpduEE4E3O8ANkjXtkfOEhFmM6PI86Xs JB9w== Received: by 10.50.185.230 with SMTP id ff6mr7746798igc.7.1353355054008; Mon, 19 Nov 2012 11:57:34 -0800 (PST) Received: from [192.168.1.73] ([208.85.112.101]) by mx.google.com with ESMTPS id xn10sm7727530igb.4.2012.11.19.11.57.33 (version=SSLv3 cipher=OTHER); Mon, 19 Nov 2012 11:57:33 -0800 (PST) Message-ID: <50AA8F24.7080604@gmail.com> Date: Mon, 19 Nov 2012 14:57:24 -0500 From: Karim Fodil-Lemelin User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20121026 Thunderbird/16.0.2 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: igb diver crashes in head@241037 Content-Type: multipart/mixed; boundary="------------060400040902030200040703" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Nov 2012 19:57:34 -0000 This is a multi-part message in MIME format. --------------060400040902030200040703 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hello -net, While testing the latest igb driver in CURRENT I came across an issue with igb_mq_start(). More specifically this code: ... struct mbuf *pm = NULL; /* ** Try to queue first to avoid ** out-of-order delivery, but ** settle for it if that fails */ if (m && drbr_enqueue(ifp, txr->br, m)) pm = m; err = igb_mq_start_locked(ifp, txr, pm); ... The problem comes from the fact that drbr_enqueue() can return an error and delete the mbuf as seen in drbr_enqueue(): ... error = buf_ring_enqueue(br, m); if (error) m_freem(m); ... When this happens pm is set to m then igb_mq_start_locked() will enqueue an already freed mbuf with the outcome you can imagine. When I reverted only that part of r241037 that problem disappeared. I have attached a patch for those interested. Best regards, Karim. --------------060400040902030200040703 Content-Type: text/plain; charset=windows-1252; name="igb.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="igb.patch" diff --git a/sys/dev/e1000/if_igb.c b/sys/dev/e1000/if_igb.c index 1318910..be1719a 100644 --- a/sys/dev/e1000/if_igb.c +++ b/sys/dev/e1000/if_igb.c @@ -961,15 +961,7 @@ igb_mq_start(struct ifnet *ifp, struct mbuf *m) que = &adapter->queues[i]; if (((txr->queue_status & IGB_QUEUE_DEPLETED) == 0) && IGB_TX_TRYLOCK(txr)) { - struct mbuf *pm = NULL; - /* - ** Try to queue first to avoid - ** out-of-order delivery, but - ** settle for it if that fails - */ - if (m && drbr_enqueue(ifp, txr->br, m)) - pm = m; - err = igb_mq_start_locked(ifp, txr, pm); + err = igb_mq_start_locked(ifp, txr, m); IGB_TX_UNLOCK(txr); } else { err = drbr_enqueue(ifp, txr->br, m); --------------060400040902030200040703-- From owner-freebsd-net@FreeBSD.ORG Mon Nov 19 20:01:46 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E2F7A70E for ; Mon, 19 Nov 2012 20:01:46 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-vb0-f54.google.com (mail-vb0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id 922178FC15 for ; Mon, 19 Nov 2012 20:01:46 +0000 (UTC) Received: by mail-vb0-f54.google.com with SMTP id l1so7035939vba.13 for ; Mon, 19 Nov 2012 12:01:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=8xutW1KQA6B9Y+kgC/492GL3AOkhiwZiGr1G+dd3n4c=; b=syT8y+FSOi4Z2RmVVIJZu6NVnj9xmx+WMzNDAM3FC5LOV8HBDAEeDfpNLaKsXM+iBC PMkd93pP6jQMmzowrYzP41wI/OcEk6YxhfhsdL2XtCmi/rJ10B0q043+gPxinREob2iy ZAly96XYtKEl+Eb4x1k+OdScHjdMuMuhJ5KigjUULPCYuoQde7Dhgwo4tnOplO9pkWTS MapLPII02iAL0H68pc5gQDfvXaZWKaxRTzZszQPnXHBtOyXoE6ZkxbPHviaOPTsipPLg c24Cc5uJgrtau9fuHM9EGRsSTamDoMPkhj3NHv9tT/im+gWeVpHAdogl2mW9TioCJExs /Zxg== MIME-Version: 1.0 Received: by 10.220.228.2 with SMTP id jc2mr8871713vcb.32.1353355305456; Mon, 19 Nov 2012 12:01:45 -0800 (PST) Received: by 10.59.3.165 with HTTP; Mon, 19 Nov 2012 12:01:45 -0800 (PST) In-Reply-To: <50AA8F24.7080604@gmail.com> References: <50AA8F24.7080604@gmail.com> Date: Mon, 19 Nov 2012 12:01:45 -0800 Message-ID: Subject: Re: igb diver crashes in head@241037 From: Jack Vogel To: Karim Fodil-Lemelin Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Nov 2012 20:01:47 -0000 Indeed, I found this very same problem in internal testing, I took it out, but then have been working on the best way to keep the idea without the problems, I have code that will do that coming soon. Thanks for the report! Jack On Mon, Nov 19, 2012 at 11:57 AM, Karim Fodil-Lemelin < fodillemlinkarim@gmail.com> wrote: > Hello -net, > > While testing the latest igb driver in CURRENT I came across an issue with > igb_mq_start(). More specifically this code: > > ... > > struct mbuf *pm = NULL; > /* > ** Try to queue first to avoid > ** out-of-order delivery, but > ** settle for it if that fails > */ > if (m && drbr_enqueue(ifp, txr->br, m)) > pm = m; > err = igb_mq_start_locked(ifp, txr, pm); > > ... > > > The problem comes from the fact that drbr_enqueue() can return an error > and delete the mbuf as seen in drbr_enqueue(): > > ... > error = buf_ring_enqueue(br, m); > if (error) > m_freem(m); > ... > > When this happens pm is set to m then igb_mq_start_locked() will enqueue > an already freed mbuf with the outcome you can imagine. > > When I reverted only that part of r241037 that problem disappeared. I have > attached a patch for those interested. > > Best regards, > > Karim. > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Tue Nov 20 07:59:34 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C8230BA9 for ; Tue, 20 Nov 2012 07:59:34 +0000 (UTC) (envelope-from camden.lindsay@gmail.com) Received: from mail-oa0-f54.google.com (mail-oa0-f54.google.com [209.85.219.54]) by mx1.freebsd.org (Postfix) with ESMTP id 7EF0E8FC14 for ; Tue, 20 Nov 2012 07:59:34 +0000 (UTC) Received: by mail-oa0-f54.google.com with SMTP id n9so7226405oag.13 for ; Mon, 19 Nov 2012 23:59:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:cc:content-type; bh=jxmfqi95qZTlUMtdtQJ83qS7gMBQhSxXNmVGZq4iyrw=; b=tSJK2JBdL40EZrIh4FsygRgOwpoUDuio6FiNYg+bHi4JbJQqfRulzJ661AXLTWzurT zIisY9DEYc7oQ5ouHx1gHUBo15GtN+2vIXbU1MJGpCbXsx371I79V+alzGTYEhw5oB+X Olidx+yOEx947Aqkdf0/J5MLDPDFE/stWkR+nHEeyZHxYvTF6zKrpqkuj+eYSginct4K igQJJ4gZZWLPYV1lX2OKMV2gazwEwVUYYrQxToloEqTfv5m8USyObMUnqPhTEBGHUxlH 8cAfoHbyVm8iULFGe7jBS4IJ6nKPAzg+3RfDQzDNFhxGmyS1XgUBKFKvhoPj2lb7rAQu qDmA== Received: by 10.60.25.3 with SMTP id y3mr12641837oef.91.1353398373667; Mon, 19 Nov 2012 23:59:33 -0800 (PST) MIME-Version: 1.0 Sender: camden.lindsay@gmail.com Received: by 10.182.84.135 with HTTP; Mon, 19 Nov 2012 23:59:13 -0800 (PST) In-Reply-To: References: From: camden lindsay Date: Mon, 19 Nov 2012 23:59:13 -0800 X-Google-Sender-Auth: kiSSrfcfKwgv0SkwqdrDq3RL_8o Message-ID: Subject: bxe driver crash -- 'client ramrod halt failed!' To: freebsd-net@freebsd.org Content-Type: multipart/mixed; boundary=e89a8f503ba64d527204cee89d66 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Blake Anghilante X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 07:59:34 -0000 --e89a8f503ba64d527204cee89d66 Content-Type: text/plain; charset=ISO-8859-1 Hello- Please excuse if i am not following customary protocol, i've worked very little with bsd and linux mailing lists. My friend and I are trying to set up a Broadcom Netxtreme II NIC (57711 A0) in a BSD box, and we are having problems. Problem summary: BSD machine appears to boot healthily, and interface configures as expected using ifconfig BSD and Windows machines can ping each other, as expected BSD machine goes unreachable after first tcp packets arrive on interface, be it via SCP/SSH, HTTP/HTTPS or Iperf Ifconfig down after this hang leads to error 'client ramrod halt failed!'. Ifconfig up after ramrod failure bombards stdout with error messages (attached) seeming to indicate receive registers expected to be 0/empty have data in them. Identical error messages repeat until the device is once again ifconfig'd down UDP iperf works prior to any TCP traffic, albeit seems slower than should be. To verify hardware, the same card was tested in the same hardware environment using knoppix (7.0.4 DVD), and performed as expected using iperf with TCP. Environment: -Windows 7 Machine --Solar Flare 4000 -BSD Machine --Nas4Free 9.1.0.1 rev 358 (BSD 9.1 RC2) [blake@storage ~]$ uname -a FreeBSD storage.local 9.1-RC2 FreeBSD 9.1-RC2 #0: Mon Oct 8 03:51:59 JST 2012 aoyama@nas4free.local:/usr/obj/nas4free/usr/src/sys/NAS4FREE-amd64 amd64 and --BSD 9.1. RC2 vanilla (both display same symptoms) --Broadcom NetExtreme 57711 A0 blake@storage ~]$ dmesg | grep bxe bxe0: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A18E682C for ; Tue, 20 Nov 2012 09:50:03 +0000 (UTC) (envelope-from 3S1KrUAwHCygNIVVcgeemeelfkh.GSQJVIIFWH-RIXJVIIFWH.SVK@photos-server.bounces.google.com) Received: from mail-vc0-f202.google.com (mail-vc0-f202.google.com [209.85.220.202]) by mx1.freebsd.org (Postfix) with ESMTP id 59FBC8FC0C for ; Tue, 20 Nov 2012 09:50:03 +0000 (UTC) Received: by mail-vc0-f202.google.com with SMTP id m8so124193vcd.1 for ; Tue, 20 Nov 2012 01:50:03 -0800 (PST) MIME-Version: 1.0 Received: by 10.224.207.66 with SMTP id fx2mt24835826qab.7.1353405003036; Tue, 20 Nov 2012 01:50:03 -0800 (PST) Message-ID: <20cf3005ddae717aae04ceea2885@google.com> Date: Tue, 20 Nov 2012 09:50:03 +0000 Subject: USB flash diver Introduce for the NEW shared photos with you From: USB flash diver Introduce for the NEW To: freebsd-net@freebsd.org Content-Type: multipart/mixed; boundary=20cf3005ddae71df4704ceea2891 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: USB flash diver Introduce for the NEW List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 09:50:03 -0000 --20cf3005ddae71df4704ceea2891 Content-Type: text/plain; charset=ISO-8859-1; format=flowed; delsp=yes Dear Manager Have a nice day This is Jerry from Shenzhen of China, writing to you to enter into business relationship with you. We are the professional manufacturer with nearly 6 years of experience oversea business in supplying USB Flash Drives, Tablet PC, Keyboard, Mouse, MP3, Digital Photo frame, Laser Pointer, Sound recording pen. We can supply you the competitive price and best quality, Welcome to visit our website: http://pzz510127.cn.alibaba.com Meanwhile, we will offer positive cooperation and sincere service, also look foward to buliding up long cooperation and double-win situation between us, Do the promise as belows: 1: Samples are arranged only about 2-3 days 2: The goods is at the rate of 100% inspected and tested before delivery 3: Any style of LOGO is no problem for us per as your requirement to publicize your company 4: There is thousand kinds of USB Flash Drive for your choice, also you can select any type you like or open new mould for you 5: Our factory is located in Shenzhen, so our quotation is FOB shenzhen. If any interest or need, Please feel free to contact us without hesitating, we will do everything well for you. Your reply will be highly appreciated Thank you very much Jerry Telephone: 86-0755-83268990-802 Mobilephone : 086 18682315895 MSN:Jerry200808@hotmail.com Skype:jerry200836 Email :Jerry200836@yahoo.cn Company name : T-Jorda Electronics Co.,Ltd Website : http://pzz510127.cn.alibaba.com --20cf3005ddae71df4704ceea2891-- From owner-freebsd-net@FreeBSD.ORG Tue Nov 20 11:18:41 2012 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4C0E117E; Tue, 20 Nov 2012 11:18:41 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from cell.glebius.int.ru (glebius.int.ru [81.19.64.117]) by mx1.freebsd.org (Postfix) with ESMTP id 7D4C78FC08; Tue, 20 Nov 2012 11:18:40 +0000 (UTC) Received: from cell.glebius.int.ru (localhost [127.0.0.1]) by cell.glebius.int.ru (8.14.5/8.14.5) with ESMTP id qAKBIXfJ068342; Tue, 20 Nov 2012 15:18:33 +0400 (MSK) (envelope-from glebius@FreeBSD.org) Received: (from glebius@localhost) by cell.glebius.int.ru (8.14.5/8.14.5/Submit) id qAKBIXgU068341; Tue, 20 Nov 2012 15:18:33 +0400 (MSK) (envelope-from glebius@FreeBSD.org) X-Authentication-Warning: cell.glebius.int.ru: glebius set sender to glebius@FreeBSD.org using -f Date: Tue, 20 Nov 2012 15:18:33 +0400 From: Gleb Smirnoff To: Karim Fodil-Lemelin , jfv@FreeBSD.org Subject: Re: igb diver crashes in head@241037 Message-ID: <20121120111833.GC67660@FreeBSD.org> References: <50AA8F24.7080604@gmail.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="6sX45UoQRIJXqkqR" Content-Disposition: inline In-Reply-To: <50AA8F24.7080604@gmail.com> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-net@FreeBSD.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 11:18:41 -0000 --6sX45UoQRIJXqkqR Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Karim, On Mon, Nov 19, 2012 at 02:57:24PM -0500, Karim Fodil-Lemelin wrote: K> While testing the latest igb driver in CURRENT I came across an issue K> with igb_mq_start(). More specifically this code: K> K> ... K> K> struct mbuf *pm = NULL; K> /* K> ** Try to queue first to avoid K> ** out-of-order delivery, but K> ** settle for it if that fails K> */ K> if (m && drbr_enqueue(ifp, txr->br, m)) K> pm = m; K> err = igb_mq_start_locked(ifp, txr, pm); K> K> ... K> K> K> The problem comes from the fact that drbr_enqueue() can return an error K> and delete the mbuf as seen in drbr_enqueue(): K> K> ... K> error = buf_ring_enqueue(br, m); K> if (error) K> m_freem(m); K> ... Good catch! K> diff --git a/sys/dev/e1000/if_igb.c b/sys/dev/e1000/if_igb.c K> index 1318910..be1719a 100644 K> --- a/sys/dev/e1000/if_igb.c K> +++ b/sys/dev/e1000/if_igb.c K> @@ -961,15 +961,7 @@ igb_mq_start(struct ifnet *ifp, struct mbuf *m) K> que = &adapter->queues[i]; K> if (((txr->queue_status & IGB_QUEUE_DEPLETED) == 0) && K> IGB_TX_TRYLOCK(txr)) { K> - struct mbuf *pm = NULL; K> - /* K> - ** Try to queue first to avoid K> - ** out-of-order delivery, but K> - ** settle for it if that fails K> - */ K> - if (m && drbr_enqueue(ifp, txr->br, m)) K> - pm = m; K> - err = igb_mq_start_locked(ifp, txr, pm); K> + err = igb_mq_start_locked(ifp, txr, m); K> IGB_TX_UNLOCK(txr); K> } else { K> err = drbr_enqueue(ifp, txr->br, m); Well, the idea to prevent out-of-order delivery is important. TCP suffers a lot when this happens. I'd suggest the following code: if (m) drbr_enqueue(ifp, txr->br, m); err = igb_mq_start_locked(ifp, txr, NULL); Which eventually leads us to all invocations of igb_mq_start_locked() called with third argument as NULL. This allows us to simplify this function. Patch for review attached. -- Totus tuus, Glebius. --6sX45UoQRIJXqkqR Content-Type: text/x-diff; charset=koi8-r Content-Disposition: attachment; filename="if_igb.c.diff" Index: if_igb.c =================================================================== --- if_igb.c (revision 243329) +++ if_igb.c (working copy) @@ -181,8 +181,7 @@ static int igb_resume(device_t); #if __FreeBSD_version >= 800000 static int igb_mq_start(struct ifnet *, struct mbuf *); -static int igb_mq_start_locked(struct ifnet *, - struct tx_ring *, struct mbuf *); +static int igb_mq_start_locked(struct ifnet *, struct tx_ring *); static void igb_qflush(struct ifnet *); static void igb_deferred_mq_start(void *, int); #else @@ -845,7 +844,7 @@ /* Process the stack queue only if not depleted */ if (((txr->queue_status & IGB_QUEUE_DEPLETED) == 0) && !drbr_empty(ifp, txr->br)) - igb_mq_start_locked(ifp, txr, NULL); + igb_mq_start_locked(ifp, txr); #else if (!IFQ_DRV_IS_EMPTY(&ifp->if_snd)) igb_start_locked(txr, ifp); @@ -961,15 +960,14 @@ que = &adapter->queues[i]; if (((txr->queue_status & IGB_QUEUE_DEPLETED) == 0) && IGB_TX_TRYLOCK(txr)) { - struct mbuf *pm = NULL; /* ** Try to queue first to avoid ** out-of-order delivery, but ** settle for it if that fails */ - if (m && drbr_enqueue(ifp, txr->br, m)) - pm = m; - err = igb_mq_start_locked(ifp, txr, pm); + if (m) + drbr_enqueue(ifp, txr->br, m); + err = igb_mq_start_locked(ifp, txr); IGB_TX_UNLOCK(txr); } else { err = drbr_enqueue(ifp, txr->br, m); @@ -980,7 +978,7 @@ } static int -igb_mq_start_locked(struct ifnet *ifp, struct tx_ring *txr, struct mbuf *m) +igb_mq_start_locked(struct ifnet *ifp, struct tx_ring *txr) { struct adapter *adapter = txr->adapter; struct mbuf *next; @@ -990,24 +988,13 @@ if (((ifp->if_drv_flags & IFF_DRV_RUNNING) == 0) || (txr->queue_status & IGB_QUEUE_DEPLETED) || - adapter->link_active == 0) { - if (m != NULL) - err = drbr_enqueue(ifp, txr->br, m); + adapter->link_active == 0) return (err); - } enq = 0; - if (m == NULL) { - next = drbr_dequeue(ifp, txr->br); - } else if (drbr_needs_enqueue(ifp, txr->br)) { - if ((err = drbr_enqueue(ifp, txr->br, m)) != 0) - return (err); - next = drbr_dequeue(ifp, txr->br); - } else - next = m; /* Process the queue */ - while (next != NULL) { + while ((next = drbr_dequeue(ifp, txr->br)) != NULL) { if ((err = igb_xmit(txr, &next)) != 0) { if (next != NULL) err = drbr_enqueue(ifp, txr->br, next); @@ -1020,7 +1007,6 @@ ETHER_BPF_MTAP(ifp, next); if ((ifp->if_drv_flags & IFF_DRV_RUNNING) == 0) break; - next = drbr_dequeue(ifp, txr->br); } if (enq > 0) { /* Set the watchdog */ @@ -1046,7 +1032,7 @@ IGB_TX_LOCK(txr); if (!drbr_empty(ifp, txr->br)) - igb_mq_start_locked(ifp, txr, NULL); + igb_mq_start_locked(ifp, txr); IGB_TX_UNLOCK(txr); } @@ -1398,7 +1384,7 @@ /* Process the stack queue only if not depleted */ if (((txr->queue_status & IGB_QUEUE_DEPLETED) == 0) && !drbr_empty(ifp, txr->br)) - igb_mq_start_locked(ifp, txr, NULL); + igb_mq_start_locked(ifp, txr); #else if (!IFQ_DRV_IS_EMPTY(&ifp->if_snd)) igb_start_locked(txr, ifp); @@ -1449,7 +1435,7 @@ /* Process the stack queue only if not depleted */ if (((txr->queue_status & IGB_QUEUE_DEPLETED) == 0) && !drbr_empty(ifp, txr->br)) - igb_mq_start_locked(ifp, txr, NULL); + igb_mq_start_locked(ifp, txr); #else if (!IFQ_DRV_IS_EMPTY(&ifp->if_snd)) igb_start_locked(txr, ifp); @@ -1549,7 +1535,7 @@ } while (loop-- && more); #if __FreeBSD_version >= 800000 if (!drbr_empty(ifp, txr->br)) - igb_mq_start_locked(ifp, txr, NULL); + igb_mq_start_locked(ifp, txr); #else if (!IFQ_DRV_IS_EMPTY(&ifp->if_snd)) igb_start_locked(txr, ifp); @@ -1586,7 +1572,7 @@ /* Process the stack queue only if not depleted */ if (((txr->queue_status & IGB_QUEUE_DEPLETED) == 0) && !drbr_empty(ifp, txr->br)) - igb_mq_start_locked(ifp, txr, NULL); + igb_mq_start_locked(ifp, txr); #else if (!IFQ_DRV_IS_EMPTY(&ifp->if_snd)) igb_start_locked(txr, ifp); --6sX45UoQRIJXqkqR-- From owner-freebsd-net@FreeBSD.ORG Tue Nov 20 15:21:53 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3059C90A; Tue, 20 Nov 2012 15:21:53 +0000 (UTC) (envelope-from fodillemlinkarim@gmail.com) Received: from mail-ia0-f182.google.com (mail-ia0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id B5C1D8FC12; Tue, 20 Nov 2012 15:21:52 +0000 (UTC) Received: by mail-ia0-f182.google.com with SMTP id x2so5448827iad.13 for ; Tue, 20 Nov 2012 07:21:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=GhF0+nXmzhszx2tOXSWZdSrTOX+049W228Zpnt8GWKk=; b=WS1HDJSjSJnMtvsZs+i/W1cMQb4fssni2vanen2bgbihex/iXnl2zWLE7L5dXDupej JRLDPIrpFGc3FJgIkaLFUBoun+xS2ZX3BTz28klwLkidrKl98Wc1NUrEMKpCfZPBB79a djhmsmbrqkyle4LqMfc24kkSqtcjV8jjKneFonGnMVbMwGa23bmF7utup9oMnq9oiyd0 Gi4b9KwkETzgJ7oB65gKM1mVNkGo0kP6xlBJZyDl9/1THDxsD5abtaXNgEooTuJYa475 DnrrfhkLvTJ+8U6gwDT1Zi71c/QbPIbpmZM4xAMwBMejDXaEbeNzpnpBqCVKkRfRdoSa M8yA== Received: by 10.50.178.8 with SMTP id cu8mr10523759igc.5.1353424906190; Tue, 20 Nov 2012 07:21:46 -0800 (PST) Received: from [192.168.1.73] ([208.85.112.101]) by mx.google.com with ESMTPS id rd10sm9650361igb.1.2012.11.20.07.21.44 (version=SSLv3 cipher=OTHER); Tue, 20 Nov 2012 07:21:45 -0800 (PST) Message-ID: <50AB9FFE.70401@gmail.com> Date: Tue, 20 Nov 2012 10:21:34 -0500 From: Karim Fodil-Lemelin User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20121026 Thunderbird/16.0.2 MIME-Version: 1.0 To: Gleb Smirnoff Subject: Re: igb diver crashes in head@241037 References: <50AA8F24.7080604@gmail.com> <20121120111833.GC67660@FreeBSD.org> In-Reply-To: <20121120111833.GC67660@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Karim Fodil-Lemelin , jfv@FreeBSD.org, freebsd-net@FreeBSD.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 15:21:53 -0000 Gleb, On 20/11/2012 6:18 AM, Gleb Smirnoff wrote: > Karim, > > On Mon, Nov 19, 2012 at 02:57:24PM -0500, Karim Fodil-Lemelin wrote: > K> While testing the latest igb driver in CURRENT I came across an issue > K> with igb_mq_start(). More specifically this code: > K> > K> ... > K> > K> struct mbuf *pm = NULL; > K> /* > K> ** Try to queue first to avoid > K> ** out-of-order delivery, but > K> ** settle for it if that fails > K> */ > K> if (m && drbr_enqueue(ifp, txr->br, m)) > K> pm = m; > K> err = igb_mq_start_locked(ifp, txr, pm); > K> > K> ... > K> > K> > K> The problem comes from the fact that drbr_enqueue() can return an error > K> and delete the mbuf as seen in drbr_enqueue(): > K> > K> ... > K> error = buf_ring_enqueue(br, m); > K> if (error) > K> m_freem(m); > K> ... > > Good catch! > > K> diff --git a/sys/dev/e1000/if_igb.c b/sys/dev/e1000/if_igb.c > K> index 1318910..be1719a 100644 > K> --- a/sys/dev/e1000/if_igb.c > K> +++ b/sys/dev/e1000/if_igb.c > K> @@ -961,15 +961,7 @@ igb_mq_start(struct ifnet *ifp, struct mbuf *m) > K> que = &adapter->queues[i]; > K> if (((txr->queue_status & IGB_QUEUE_DEPLETED) == 0) && > K> IGB_TX_TRYLOCK(txr)) { > K> - struct mbuf *pm = NULL; > K> - /* > K> - ** Try to queue first to avoid > K> - ** out-of-order delivery, but > K> - ** settle for it if that fails > K> - */ > K> - if (m && drbr_enqueue(ifp, txr->br, m)) > K> - pm = m; > K> - err = igb_mq_start_locked(ifp, txr, pm); > K> + err = igb_mq_start_locked(ifp, txr, m); > K> IGB_TX_UNLOCK(txr); > K> } else { > K> err = drbr_enqueue(ifp, txr->br, m); > > Well, the idea to prevent out-of-order delivery is important. TCP > suffers a lot when this happens. > > I'd suggest the following code: > > if (m) > drbr_enqueue(ifp, txr->br, m); > err = igb_mq_start_locked(ifp, txr, NULL); > > Which eventually leads us to all invocations of igb_mq_start_locked() called > with third argument as NULL. This allows us to simplify this function. > > Patch for review attached. > > In my case I use ALTQ and in igb_mq_start_locked() it calls drbr_needs_enqueue() which always return 1 with ALTQ. So I did not see the out of order issue. I do think your patch makes sense though and I'm also thinking that em_mq_start() could also benefit from the same logic. Regards, Karim. From owner-freebsd-net@FreeBSD.ORG Tue Nov 20 17:19:56 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 26321CED; Tue, 20 Nov 2012 17:19:56 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-vb0-f54.google.com (mail-vb0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id 84D1F8FC08; Tue, 20 Nov 2012 17:19:55 +0000 (UTC) Received: by mail-vb0-f54.google.com with SMTP id l1so8352496vba.13 for ; Tue, 20 Nov 2012 09:19:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=6t8r0XDStFwZBMkkVt6lWYhDH7ysUPqe+ySsnmgQzWA=; b=g7J6ZLH3Qwk/u0bq1ViawEDaCIRJucaF86VKdTwfYk7C4SNXGzySmUo4McZI7WTcDr tDluzlQd4/qRoOFseLMaRzPehaPld8X6p/7JT5aKIsirLt75SoQoSnqsB8AIcDG/6j6a 18Vo4XY2iJxJp6wUSGT8vwDq4pzmUyQ0XwfC8AU2NOjxB5mkWLepIMw3GTiIt3xzXjot v0TNT/AFgHHKMoTEDd+ibqFzn9+XdsX+eifZ9+OwpLHFFgxCiLNoB1sgb5tqCfzZmjpc 1JLrNBzg1pDmYxN1AYWqtPHrjhXCw+ndv42y/vEz7969aa3mA+Yg0p7nXyxJvG9SSZyd Efjw== MIME-Version: 1.0 Received: by 10.52.180.40 with SMTP id dl8mr21174045vdc.51.1353431994328; Tue, 20 Nov 2012 09:19:54 -0800 (PST) Received: by 10.59.3.165 with HTTP; Tue, 20 Nov 2012 09:19:54 -0800 (PST) In-Reply-To: <20121120111833.GC67660@FreeBSD.org> References: <50AA8F24.7080604@gmail.com> <20121120111833.GC67660@FreeBSD.org> Date: Tue, 20 Nov 2012 09:19:54 -0800 Message-ID: Subject: Re: igb diver crashes in head@241037 From: Jack Vogel To: Gleb Smirnoff Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Karim Fodil-Lemelin , jfv@freebsd.org, freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 17:19:56 -0000 On Tue, Nov 20, 2012 at 3:18 AM, Gleb Smirnoff wrote: > Karim, > > On Mon, Nov 19, 2012 at 02:57:24PM -0500, Karim Fodil-Lemelin wrote: > K> While testing the latest igb driver in CURRENT I came across an issue > K> with igb_mq_start(). More specifically this code: > K> > K> ... > K> > K> struct mbuf *pm = NULL; > K> /* > K> ** Try to queue first to avoid > K> ** out-of-order delivery, but > K> ** settle for it if that fails > K> */ > K> if (m && drbr_enqueue(ifp, txr->br, m)) > K> pm = m; > K> err = igb_mq_start_locked(ifp, txr, pm); > K> > K> ... > K> > K> > K> The problem comes from the fact that drbr_enqueue() can return an error > K> and delete the mbuf as seen in drbr_enqueue(): > K> > K> ... > K> error = buf_ring_enqueue(br, m); > K> if (error) > K> m_freem(m); > K> ... > > Good catch! > > K> diff --git a/sys/dev/e1000/if_igb.c b/sys/dev/e1000/if_igb.c > K> index 1318910..be1719a 100644 > K> --- a/sys/dev/e1000/if_igb.c > K> +++ b/sys/dev/e1000/if_igb.c > K> @@ -961,15 +961,7 @@ igb_mq_start(struct ifnet *ifp, struct mbuf *m) > K> que = &adapter->queues[i]; > K> if (((txr->queue_status & IGB_QUEUE_DEPLETED) == 0) && > K> IGB_TX_TRYLOCK(txr)) { > K> - struct mbuf *pm = NULL; > K> - /* > K> - ** Try to queue first to avoid > K> - ** out-of-order delivery, but > K> - ** settle for it if that fails > K> - */ > K> - if (m && drbr_enqueue(ifp, txr->br, m)) > K> - pm = m; > K> - err = igb_mq_start_locked(ifp, txr, pm); > K> + err = igb_mq_start_locked(ifp, txr, m); > K> IGB_TX_UNLOCK(txr); > K> } else { > K> err = drbr_enqueue(ifp, txr->br, m); > > Well, the idea to prevent out-of-order delivery is important. TCP > suffers a lot when this happens. > > I'd suggest the following code: > > if (m) > drbr_enqueue(ifp, txr->br, m); > err = igb_mq_start_locked(ifp, txr, NULL); > > Which eventually leads us to all invocations of igb_mq_start_locked() > called > with third argument as NULL. This allows us to simplify this function. > > Patch for review attached. > > Yes Gleb, I already have code in my internal tree which simply removes an mbuf pointer form the start_locked call and ALWAYS does a dequeue, start similarly will always enqueue. I just have been busy with ixgbe for a bit and have not gotten it committed yet. Jack From owner-freebsd-net@FreeBSD.ORG Tue Nov 20 21:54:42 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 345249E9 for ; Tue, 20 Nov 2012 21:54:42 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm21-vm2.bullet.mail.ne1.yahoo.com (nm21-vm2.bullet.mail.ne1.yahoo.com [98.138.91.209]) by mx1.freebsd.org (Postfix) with ESMTP id D21148FC12 for ; Tue, 20 Nov 2012 21:54:41 +0000 (UTC) Received: from [98.138.90.54] by nm21.bullet.mail.ne1.yahoo.com with NNFMP; 20 Nov 2012 21:52:09 -0000 Received: from [98.138.89.194] by tm7.bullet.mail.ne1.yahoo.com with NNFMP; 20 Nov 2012 21:52:09 -0000 Received: from [127.0.0.1] by omp1052.mail.ne1.yahoo.com with NNFMP; 20 Nov 2012 21:52:09 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 605282.16429.bm@omp1052.mail.ne1.yahoo.com Received: (qmail 76446 invoked by uid 60001); 20 Nov 2012 21:52:09 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1353448329; bh=Rme4lfZn3rWhFkqTDhpG8vDYHLQDOpkFQBN4pVLzx6I=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=SP1gsxZNKaehmL8HXNFRO2NbfnCkIekpm8as8L4vLmHQcD7mTwSC6AKahmAl/ZW6LCqt8umsJIh3vrvz5YNbLe/Mz2HRwe5scPi6pXP/+g4Yak457Z0WSFyXyEQgq/0y+dtJbKg+qCBYLGnzZszhgNHrGoC3Aa/ZHXF743Kx9J0= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=UQ7oGg5c5sYtTLPj+/KiAIBN7ivXRUxOynOPSsVVIDmitnePN7T52GynHvAeGCJgVKMByxu0/X3dAO3q8QI4XK+8JaIFI49mtWW23ZV/c34XDLm0LkKT7VnjB+bcL/OOp0rsUEj+VUsUVFHqqlj2mTRNXPiG/19B8SgKZpzVrBs=; X-YMail-OSG: XMo5jnEVM1nXTyjRUoUn_pKR3HQXD3LqnZMVBbL0zjKyOUh GtNbLvGjmZZ3cH6C0IMxgn0DcNBkpsj5IcAKZnXF1iy4qUKLtvtNDYch3Jm3 eHsRDHGKWXOIt2NOcPL6a501ZxAslACdmj4cHauloSkMSbyEAwpPm..dZ4hF vogKD48GzOXk05U6qCanWul2k7p8CgemrJMQA.4RvPL0cENmEP7F7TpexeWS 03RBq_x7M5xHEgc3nw9qqsBHZvvDhF8dWARrlZkG_h6peohoHXr1zAUk2VkI 4fTCInbgJ8hKTRqe6TQbvw3IcEcuFK4G8K34T1wYEQEO2_WKKt7JK6lGDuAU U8lWRIiuHV.iv6iUX.tsfNvAFguNnCIRGpKywdGWKxdbfFX7bQJ_hkZoiY8y 8FP9cb_1vkO69PrUe.eXMN9TP5A1yHo5zFUQS_O9MymOT7wmi8VJrpU2okgK bxO8ouq2PeYYqu7_vbIFqFn_DyXCqCynibDoXJxNqN0qWnCx3uK2yrkhqCGa sXTqkcF2o69kHLSOWN17ectBjBvIq3g3L3l2P Received: from [174.48.128.27] by web121602.mail.ne1.yahoo.com via HTTP; Tue, 20 Nov 2012 13:52:08 PST X-Rocket-MIMEInfo: 001.001, QW55b25lIHdobyBldmVuIG1lbnRpb25zIHBvbGxpbmcgc2hvdWxkIGJlIGRpc2NvdW50ZWQgYWx0b2dldGhlci4gUG9sbGluZwpoYWQgdmFsdWUgd2hlbiB5b3UgY291bGRuJ3QgY29udHJvbCB0aGUgaW50ZXJydXB0IGRlbGF5czsgYnV0IGludGVycnVwdAptb2RlcmF0aW9uIGFsbG93cyB5b3UgdG8gcGFjZSB0aGUgaW50ZXJydXB0cyBhbnkgd2F5IHlvdSBsaWtlIHdpdGhvdXQKdGhlIGluZWZmaWNpZW5jaWVzIG9mIHBvbGxpbmcuCgpUaGUgaWRlYSB0aGF0IHBvbGxpbmcgdXNlcyBsZXNzIENQVSBpcyBjb20BMAEBAQE- X-Mailer: YahooMailClassic/15.0.8 YahooMailWebService/0.8.126.470 Message-ID: <1353448328.76219.YahooMailClassic@web121602.mail.ne1.yahoo.com> Date: Tue, 20 Nov 2012 13:52:08 -0800 (PST) From: Barney Cordoba Subject: Re: FreeBSD boxes as a 'router'... To: khatfield@socllc.net In-Reply-To: <1281530059.17550.1353434227450@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 21:54:42 -0000 Anyone who even mentions polling should be discounted altogether. Polling= =0Ahad value when you couldn't control the interrupt delays; but interrupt= =0Amoderation allows you to pace the interrupts any way you like without=0A= the inefficiencies of polling.=0A=0AThe idea that polling uses less CPU is = complete baloney; its exactly=0Athe same code as without polling. It's some= thing else that the CPU=0Ahas to do, so polling is a negative with almost a= ny modern NIC. With=0Aintel nics you can set whatever interrupt rate you wa= nt; setting your=0Aints/sec to whatever hz is (or whatever polling timer is= used) has=0Athe same effect without having to have hacked code running in = your driver.=0A=0A--- On Tue, 11/20/12, khatfield@socllc.net wrote:=0A=0A> From: khatfield@socllc.net =0A= > Subject: Re: FreeBSD boxes as a 'router'...=0A> To: "Victor Balada Diaz" = =0A> Cc: "freebsd-isp@freebsd.org" , "John Fretby" =0A> Date: Tuesday, November 20,= 2012, 12:57 PM=0A> One thing I have noticed is mixed=0A> results with poll= ing depending on the version.=0A> =0A> My experience with similar NICs was = that polling increased=0A> the PPS capabilities up to 7.4 but post 7.4 we h= ave seen=0A> most cases where polling caused either connectivity issues=0A>= or decreased overall performance.=0A> =0A> Now we were running full 1Gbps = in our tests. With only=0A> 140Mbps you should be able to handle this amoun= t without=0A> polling or additional kernel tweaks. Specifically with 9 - I= =0A> would recommend doing needed sysctl tweaks without polling=0A> and as = long as you are not receiving DDoS traffic then it=0A> should prove perfect= ly stable.=0A> =0A> =0A> =0A> On Nov 20, 2012, at 11:46 AM, "Victor Balada = Diaz" =0A> wrote:=0A> =0A> > On Tue, Nov 20, 2012 at 03:3= 5:13PM +0000, John Fretby=0A> wrote:=0A> >> Howdy all,=0A> >> =0A> >> We've= currently got an ageing HP DL360 running as a=0A> 'router' - it has=0A> >>= 100Mbit in/out onto our network, and has two 'bce'=0A> NIC's providing in/= out.=0A> >> It's running quite an old version of FreeBSD (6 I=0A> think) - = but works.=0A> >> =0A> >> As the network gets busier we've noticed the amou= nt=0A> of interrupt time on it=0A> >> is climbing (as you'd expect - i.e. e= sp. if many=0A> small packets are being=0A> >> forwarded). Many moons ago w= e did experiment with=0A> this box - and enabled=0A> >> device polling (inc= . upping the HZ on the box and=0A> recompiling the kernel=0A> >> etc). This= didn't work very well at the time=0A> (probably because it was in=0A> >> i= t's infancy) so we left it off in the end.=0A> >> =0A> >> If we were to rep= lace this box, with something new=0A> - say a SuperMicro based=0A> >> syste= m with two:=0A> >> =0A> >>=A0=A0=A0Intel 82574L's (em Driver Based)=0A> >> = =0A> >> And enable polling - is it likely to "just work"=0A> these days? Th= e current=0A> >> upstream is 100Mbit, we're looking to upgrade this=0A> to = 1Gbit in, but with=0A> >> say 200Mbit comitted on it (so shouldn't go above= =0A> 200Mbit).=0A> >> =0A> >> Is there anything that has to be done to enab= le=0A> polling - other than=0A> >> recompiling GENERIC to support it? - i.e= . no HZ=0A> hacks or anything needed on=0A> >> 'modern' machines (it's a qu= ad core Xeon).=0A> > =0A> > Hello John,=0A> > =0A> > You might find interes= ting to read this thread:=0A> > =0A> > http://lists.freebsd.org/pipermail/f= reebsd-current/2012-November/037590.html=0A=0A> > =0A> > In short: device p= olling can decrease performance on=0A> modern hardware.=0A> > =0A> > You mi= ght want to try upgrading to a new FreeBSD=0A> version and tuning it someho= w=0A> > before buying a new server. More info on tuning the=0A> network sta= ck:=0A> > =0A> > http://wiki.freebsd.org/NetworkPerformanceTuning=0A=0A> > = =0A> > Regards.=0A> > Victor.=0A> > -- =0A> > La prueba m=E1s fehaciente de= que existe vida=0A> inteligente en otros=0A> > planetas, es que no han int= entado contactar con=0A> nosotros. =0A> > _________________________________= ______________=0A> > freebsd-isp@freebsd.org=0A> mailing list=0A> > http://= lists.freebsd.org/mailman/listinfo/freebsd-isp=0A=0A> > To unsubscribe, sen= d any mail to "freebsd-isp-unsubscribe@freebsd.org"=0A> ___________________= ____________________________=0A> freebsd-isp@freebsd.org=0A> mailing list= =0A> http://lists.freebsd.org/mailman/listinfo/freebsd-isp=0A> To unsubscri= be, send any mail to "freebsd-isp-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Tue Nov 20 22:42:54 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id CE4E2DC0 for ; Tue, 20 Nov 2012 22:42:54 +0000 (UTC) (envelope-from jim@netgate.com) Received: from mail-gh0-f172.google.com (mail-gh0-f172.google.com [209.85.160.172]) by mx1.freebsd.org (Postfix) with ESMTP id 73B2C8FC0C for ; Tue, 20 Nov 2012 22:42:54 +0000 (UTC) Received: by mail-gh0-f172.google.com with SMTP id z22so120187ghb.17 for ; Tue, 20 Nov 2012 14:42:53 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to:x-mailer:x-gm-message-state; bh=zvFgsoA8QFsJwrBODJqwLLZm9UsrpofVIug2h7U3SSs=; b=TPOyS0hsL6cwFKJsFgI5jFP/QOWTea3f0RL3F2TkmQyeM5ZWZVjibkUsJ83zKQYwP0 fgjvnTgGnONxaHlOuPqz48CqnXRTAguEbWUlA+Bawi4XbbwChaQNer09aGb/rrdMb+3L czrmLcIFi1iE1S6Sb1pJO9imIfbwDwCZl6PIT7O+5j7zz+yph3A4NkwNzx70ZikExhEi MpC9OqJ2iCcCy28SQ8fhOZUq/iKNhGz7Vhs5HCIHU5O6NsIQ1oNupH3044OKgV+uGh0G tOBfs4yPL5Qnq931fPaeQQj8YrDiVmSeP1iW/h8N6+YWDh3JT1uUXp1gRYrEV2DI0foN PBtw== Received: by 10.236.74.106 with SMTP id w70mr16527440yhd.68.1353451373574; Tue, 20 Nov 2012 14:42:53 -0800 (PST) Received: from ?IPv6:2610:160:11:a000:41cc:64c0:3a45:aaf2? ([2610:160:11:a000:41cc:64c0:3a45:aaf2]) by mx.google.com with ESMTPS id m51sm14174137yhh.16.2012.11.20.14.42.51 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 20 Nov 2012 14:42:52 -0800 (PST) Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: FreeBSD boxes as a 'router'... From: Jim Thompson In-Reply-To: <1353448328.76219.YahooMailClassic@web121602.mail.ne1.yahoo.com> Date: Tue, 20 Nov 2012 16:42:48 -0600 Message-Id: References: <1353448328.76219.YahooMailClassic@web121602.mail.ne1.yahoo.com> To: Barney Cordoba X-Mailer: Apple Mail (2.1499) X-Gm-Message-State: ALoCoQmff1lrAQIgm+QEDBznf+WIsOK9aBjQWsf9vIttZdwy3+TvUm7THOzqUD7fcYPIGUyBNZic Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-net@freebsd.org, khatfield@socllc.net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 22:42:55 -0000 On Nov 20, 2012, at 3:52 PM, Barney Cordoba = wrote: > Anyone who even mentions polling should be discounted altogether. = Polling > had value when you couldn't control the interrupt delays; but = interrupt > moderation allows you to pace the interrupts any way you like without > the inefficiencies of polling. You're entitled to your opinion, but experimental results have tended to = show yours incorrect. Jim From owner-freebsd-net@FreeBSD.ORG Tue Nov 20 22:49:23 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 77435EDD for ; Tue, 20 Nov 2012 22:49:23 +0000 (UTC) (envelope-from bright@mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 551DF8FC08 for ; Tue, 20 Nov 2012 22:49:23 +0000 (UTC) Received: from Alfreds-MacBook-Pro-5.local (c-67-180-208-218.hsd1.ca.comcast.net [67.180.208.218]) by elvis.mu.org (Postfix) with ESMTPSA id C6B8C1A3C53; Tue, 20 Nov 2012 14:49:16 -0800 (PST) Message-ID: <50AC08EC.8070107@mu.org> Date: Tue, 20 Nov 2012 14:49:16 -0800 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:16.0) Gecko/20121026 Thunderbird/16.0.2 MIME-Version: 1.0 To: Jim Thompson Subject: Re: FreeBSD boxes as a 'router'... References: <1353448328.76219.YahooMailClassic@web121602.mail.ne1.yahoo.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Barney Cordoba , khatfield@socllc.net, freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 22:49:23 -0000 On 11/20/12 2:42 PM, Jim Thompson wrote: > On Nov 20, 2012, at 3:52 PM, Barney Cordoba wrote: > >> Anyone who even mentions polling should be discounted altogether. Polling >> had value when you couldn't control the interrupt delays; but interrupt >> moderation allows you to pace the interrupts any way you like without >> the inefficiencies of polling. > You're entitled to your opinion, but experimental results have tended to show yours incorrect. > > Jim Agree with Jim. If you want pure packet performance you burn a core to run a polling loop. -Alfred From owner-freebsd-net@FreeBSD.ORG Tue Nov 20 22:52:36 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D9334FA0 for ; Tue, 20 Nov 2012 22:52:36 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm9-vm2.bullet.mail.ne1.yahoo.com (nm9-vm2.bullet.mail.ne1.yahoo.com [98.138.90.157]) by mx1.freebsd.org (Postfix) with ESMTP id 81AB48FC08 for ; Tue, 20 Nov 2012 22:52:36 +0000 (UTC) Received: from [98.138.90.48] by nm9.bullet.mail.ne1.yahoo.com with NNFMP; 20 Nov 2012 22:49:42 -0000 Received: from [98.138.89.254] by tm1.bullet.mail.ne1.yahoo.com with NNFMP; 20 Nov 2012 22:49:41 -0000 Received: from [127.0.0.1] by omp1046.mail.ne1.yahoo.com with NNFMP; 20 Nov 2012 22:49:41 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 971595.71151.bm@omp1046.mail.ne1.yahoo.com Received: (qmail 18060 invoked by uid 60001); 20 Nov 2012 22:49:41 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1353451781; bh=phJrTASk8QjjPPq5ksQr9Bp3cU4mffb6tm1noqVk3A0=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=fs8mfzNFYj0npFW77OLIPaK1W8ApSBLEWRDWgICzorKGYRNoM9bJ8eaGHn3+UYnUm8K8djH8OHKJq0i59N9HOCbA/hpaR7Y3QMoHxunjpyYOsr/3R8Ba6hOPWV0sS9sdmETF/5JA987KbTkEbQFDqkI3g+LR51kEqmK/HR6NGGc= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=OElCFUuXOam338ktQ6mtc+r9uHA5yVM7RvwkEVvFqP1fGB1Jqi084uYyLiIi8s/TqTSWVgbiJ1kQnQvu+dEDyUDOgrY3IY3+ieRvHrBwXJoxA7AArYRsJnJhhgntZqp0cWIwLoole38wu0r0daOWv/RnMpd0gA8cuVAAHQi1Mwg=; X-YMail-OSG: mv6AVjYVM1kbqtW.TGi1vszKG2vzIqnj0ZsoheZHkGU3654 AbA.V4Rem19K84citAz8Uc0sQJRBZCbrheDo3ea9IMz10vAI2t.ciVQzbIZK n61r_u30MkYZVesbRLLuyYctONG7SWA_WJzCciBRd.NFcp.QkIpTnjgeIlJy j_la5UfCh2r6A8neU3iKlzq2J8J7GtE4zPp81zZCF0UaUt41s_HZc2MpHLy6 UlyoVE2FiUsloHwKEDV_HncR8H4bxrUBWvDrgqTHxz5s0My..MBT_.0uSh9g 53qwVjBPuO3N0ykGMMNFqv0m1N7vJSAaWgo9pHJStdj3EUURAeOQ2pebAP3a U5Y6vx7rDWTcxz1ljDMn0GE.2PBwyR.fq9Adj7CyJkp.f0vlkE2n5EfdsrcE BucOH864oJgyhxI1L22RMjFDNil32iZIQ28mTUO6dzyQXzW0xJadJTkRcTMO s2HlRKi6.uF9zl.J3ocrFUXyyWWw2h8zNCfvXgQuUwErqSWerlJyHA6kGjVt bvMucYSkLDz_GpyxbV1faA6hKpJuKKQ-- Received: from [174.48.128.27] by web121605.mail.ne1.yahoo.com via HTTP; Tue, 20 Nov 2012 14:49:41 PST X-Rocket-MIMEInfo: 001.001, SXQncyBub3QgYW4gIm9waW5pb24iOyBpdCBvYnZpb3VzIHRvIGFueW9uZSB3aG8gaGFkIGEgbWlsZCB1bmRlcnN0YW5kIG9mIHdoYXQgcG9sbGluZ2lzLiBZb3UndmUgbmV2ZXIgY29tcGFyZWQgaXQgdG8gbW9kZXJhdGlvbiwgd2hpY2ggaXMgd2hhdCB5b3Ugc2hvdWxkIGJlIHVzaW5nLCBiZWNhdXNleW91IGRvbid0IHVuZGVyc3RhbmQgd2hhdCB5b3UncmUgZG9pbmcuDQpJZiB5b3Ugc2V0IGludGVycnVwdCBtb2RlcmF0aW9uIHRvIDIwMDAgaW50cy9zZWMsIHlvdSdyZSBkb2luZyBleGFjdGx5IHRoZSBzYW0BMAEBAQE- X-Mailer: YahooMailClassic/15.0.8 YahooMailWebService/0.8.126.470 Message-ID: <1353451781.17468.YahooMailClassic@web121605.mail.ne1.yahoo.com> Date: Tue, 20 Nov 2012 14:49:41 -0800 (PST) From: Barney Cordoba Subject: Re: FreeBSD boxes as a 'router'... To: Jim Thompson In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 22:52:37 -0000 It's not an "opinion"; it obvious to anyone who had a mild understand of wh= at pollingis. You've never compared it to moderation, which is what you sho= uld be using, becauseyou don't understand what you're doing. If you set interrupt moderation to 2000 ints/sec, you're doing exactly the = same thingas polling without the overheard. You're comparing polling to random tuning. Which is why I say that anyone w= ho=A0recommends polling doesn't really understand what they're doing. --- On Tue, 11/20/12, Jim Thompson wrote: From: Jim Thompson Subject: Re: FreeBSD boxes as a 'router'... To: "Barney Cordoba" Cc: khatfield@socllc.net, freebsd-net@freebsd.org Date: Tuesday, November 20, 2012, 5:42 PM On Nov 20, 2012, at 3:52 PM, Barney Cordoba wrot= e: Anyone who even mentions polling should be discounted altogether. Polling had value when you couldn't control the interrupt delays; but interrupt moderation allows you to pace the interrupts any way you like without the inefficiencies of polling. You're entitled to your opinion, but experimental results have tended to sh= ow yours incorrect. Jim From owner-freebsd-net@FreeBSD.ORG Tue Nov 20 23:11:33 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BE90F630 for ; Tue, 20 Nov 2012 23:11:33 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id 0735D8FC12 for ; Tue, 20 Nov 2012 23:11:31 +0000 (UTC) Received: (qmail 23628 invoked from network); 21 Nov 2012 00:04:47 +0100 Received: from fw.xip.at (HELO ?127.0.0.1?) (89.207.145.147) by chile.gbit.at with SMTP; 21 Nov 2012 00:04:47 +0100 Message-ID: <50AC0C92.8080603@xip.at> Date: Wed, 21 Nov 2012 00:04:50 +0100 From: Ingo Flaschberger User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20121026 Thunderbird/16.0.2 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: Re: FreeBSD boxes as a 'router'... References: <1353448328.76219.YahooMailClassic@web121602.mail.ne1.yahoo.com> <50AC08EC.8070107@mu.org> In-Reply-To: <50AC08EC.8070107@mu.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 23:11:33 -0000 Am 20.11.2012 23:49, schrieb Alfred Perlstein: > On 11/20/12 2:42 PM, Jim Thompson wrote: >> On Nov 20, 2012, at 3:52 PM, Barney Cordoba >> wrote: >> >> You're entitled to your opinion, but experimental results have tended >> to show yours incorrect. >> >> Jim > Agree with Jim. If you want pure packet performance you burn a core > to run a polling loop. At new systems, without polling I had better performance and no live-locks, at old systems (Intel 82541GI) polling prevent live-locks. Best test: Loop a GigE Switch, inject a Packet and plug it into the test-box. Kind regards, Ingo Flaschberger From owner-freebsd-net@FreeBSD.ORG Tue Nov 20 23:30:23 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id F2B43B52 for ; Tue, 20 Nov 2012 23:30:22 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm15.bullet.mail.ne1.yahoo.com (nm15.bullet.mail.ne1.yahoo.com [98.138.90.78]) by mx1.freebsd.org (Postfix) with ESMTP id 9E8568FC13 for ; Tue, 20 Nov 2012 23:30:22 +0000 (UTC) Received: from [98.138.90.50] by nm15.bullet.mail.ne1.yahoo.com with NNFMP; 20 Nov 2012 23:30:15 -0000 Received: from [98.138.89.163] by tm3.bullet.mail.ne1.yahoo.com with NNFMP; 20 Nov 2012 23:30:15 -0000 Received: from [127.0.0.1] by omp1019.mail.ne1.yahoo.com with NNFMP; 20 Nov 2012 23:30:15 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 919819.81003.bm@omp1019.mail.ne1.yahoo.com Received: (qmail 21516 invoked by uid 60001); 20 Nov 2012 23:30:15 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1353454215; bh=UFN+Q2Pm+HvYxArpSLyvh9kCbu79RpS2E3JHZ0oE5hM=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=NDsm+cK1gRnjAHRIAqMu3tM5C6K2XQB8fszMHuXVSTKTLbebW+vntfosdxasV6qcqcLPwRsR8LJsHZr9egrz2KADms52WP403ZGEaRD9ijw2/5zvZZRuVoCIj8tI75CHU2B3v/bFW09WKmasyMU18zsGxQ9fsNXNwLBncciEzcA= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=Y4FY2qbDxxP8Gw+x0o70A7Bp7/1rTyLQx2Ol29F1d5z/pFqjGULnke+WQEqO3lapkTcm3sJu+kKPR2D3knyC+L7I251lXORhJy3USSkpEguPXZoV4/MT+Df39zMCnglqJnQBYkMosoCjjzOh78Py7SvbRVd9Fi/DXIJvDo1J2a8=; X-YMail-OSG: W72zAu4VM1lFcoyTs0fLc9fnlfGzNOp1hSHwPt6JZHNlaKy YP2qZbvB.MIU7bPgdtCxfPb3iHmWetde7czaRV.gDEgPkbPgLGOYqxAbfR.G ITvP9jVdPgbez2aLRed6KzziVftWU_AwRt.FAJMgDp8TtndAhBvLO619Wrk9 8LR7L.1Fu5xes.PKp2V4z3cfvqknks7nyeUSAcdAoND4if0MI9aJbLahUjnF k3nMglqK.Iakbk.95EkWqUi9CjIVAUyf4YX2GBy5FIXfFLcxTfr1FLmG03rO 9MpotBd0RRlQysDaG1kxG9y8eN7qqFn7VEoaHsdDoUwA76hpAgO0fc_jKtGZ A7Y1E7.GjHO12nIfBSrga.D6HsQ.8dNAhaTptBmcYfmnRtv0FUaMZ0UaJEWT dHQEUMozmMW_M5465N2rkqJPYCuoBWTHkavYu7tRIXPX3a65VrzPgAQy7tdp aUXKo4_R.Z.07pKVQUm3uBgp3mTPinQmyIIHunSuyrB8AG3rhEpJl_AJmEaf WQNmKBnHEziGZhC_D2Xsr..0ct0d4ag-- Received: from [174.48.128.27] by web121601.mail.ne1.yahoo.com via HTTP; Tue, 20 Nov 2012 15:30:15 PST X-Rocket-MIMEInfo: 001.001, CgotLS0gT24gVHVlLCAxMS8yMC8xMiwgSW5nbyBGbGFzY2hiZXJnZXIgPGlmQHhpcC5hdD4gd3JvdGU6Cgo.IEZyb206IEluZ28gRmxhc2NoYmVyZ2VyIDxpZkB4aXAuYXQ.Cj4gU3ViamVjdDogUmU6IEZyZWVCU0QgYm94ZXMgYXMgYSAncm91dGVyJy4uLgo.IFRvOiBmcmVlYnNkLW5ldEBmcmVlYnNkLm9yZwo.IERhdGU6IFR1ZXNkYXksIE5vdmVtYmVyIDIwLCAyMDEyLCA2OjA0IFBNCj4gQW0gMjAuMTEuMjAxMiAyMzo0OSwgc2NocmllYiBBbGZyZWQKPiBQZXJsc3RlaW46Cj4gPiBPbiAxMS8yMC8xMiAyOjQBMAEBAQE- X-Mailer: YahooMailClassic/15.0.8 YahooMailWebService/0.8.126.470 Message-ID: <1353454215.20382.YahooMailClassic@web121601.mail.ne1.yahoo.com> Date: Tue, 20 Nov 2012 15:30:15 -0800 (PST) From: Barney Cordoba Subject: Re: FreeBSD boxes as a 'router'... To: Ingo Flaschberger In-Reply-To: <50AC0C92.8080603@xip.at> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 23:30:23 -0000 =0A=0A--- On Tue, 11/20/12, Ingo Flaschberger wrote:=0A=0A> Fro= m: Ingo Flaschberger =0A> Subject: Re: FreeBSD boxes as a 'route= r'...=0A> To: freebsd-net@freebsd.org=0A> Date: Tuesday, November 20, 2012,= 6:04 PM=0A> Am 20.11.2012 23:49, schrieb Alfred=0A> Perlstein:=0A> > On 11= /20/12 2:42 PM, Jim Thompson wrote:=0A> >> On Nov 20, 2012, at 3:52 PM, Bar= ney Cordoba =0A> wrote:=0A> >> =0A> >> You're ent= itled to your opinion, but experimental=0A> results have tended to show you= rs incorrect.=0A> >> =0A> >> Jim=0A> > Agree with Jim.=A0 If you want pure = packet=0A> performance you burn a core to run a polling loop. =0A> =0A> At = new systems, without polling I had better performance and=0A> no live-locks= ,=0A> at old systems (Intel 82541GI) polling prevent live-locks.=0A> =0A> B= est test:=0A> Loop a GigE Switch, inject a Packet and plug it into the=0A> = test-box.=0A=0AYeah, thats a good real-world test.=0A=0ATo me "performance"= is not "burning a cpu" to get some extra pps. =0APerformance is not droppi= ng buckets of packets. Performance is using=0Aless cpu to do the same amoun= t of work. =0A=0AIs a machine that benchmarks at 998Mb/s at 95% cpu really = a "higher=0Aperformance" system than one that does 970Mb/s and uses 50% of = the cpu?=0A=0AThe measure of performance is to manage an entire load withou= t dropping=0Aany packets. If your machine goes into live-lock, then you nee= d more=0Amachine. Hacking it so that it drops packets is hardly a solution.= =0A=0ABC From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 01:15:34 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id ADD1E866 for ; Wed, 21 Nov 2012 01:15:34 +0000 (UTC) (envelope-from khatfield@socllc.net) Received: from smtp151.dfw.emailsrvr.com (smtp151.dfw.emailsrvr.com [67.192.241.151]) by mx1.freebsd.org (Postfix) with ESMTP id 697678FC08 for ; Wed, 21 Nov 2012 01:15:34 +0000 (UTC) Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp5.relay.dfw1a.emailsrvr.com (SMTP Server) with ESMTP id DE6821909BC; Tue, 20 Nov 2012 20:08:40 -0500 (EST) X-Virus-Scanned: OK Received: by smtp5.relay.dfw1a.emailsrvr.com (Authenticated sender: khatfield-AT-socllc.net) with ESMTPSA id 63A2D1909CC; Tue, 20 Nov 2012 20:08:40 -0500 (EST) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: Re: FreeBSD boxes as a 'router'... References: <1353448328.76219.YahooMailClassic@web121602.mail.ne1.yahoo.com> <50AC08EC.8070107@mu.org> From: khatfield@socllc.net Mime-Version: 1.0 In-Reply-To: <50AC08EC.8070107@mu.org> Message-Id: <832757660.33924.1353460119408@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> Date: Tue, 20 Nov 2012 19:08:33 -0600 To: Alfred Perlstein X-NS-Received: from Apple-iPhone5C2/1001.525(khatfield@socllc.net) SECURED(HTTPS); Wed, 21 Nov 2012 01:08:35 +0000 (UTC) Cc: Barney Cordoba , Jim Thompson , "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 01:15:34 -0000 Barney - I would certainly love to see some real evidence to backup such a = ridiculous claim. I agree here as well with Jim. I have a ton of experience with and without.= I haven't done enough testing with FreeBSD 9 to state 100% but I can state= that extensive testing and filtering traffic (specifically high PPS DDoS t= raffic) and polling is a requirement in certain situations.=20 It should not be required for normal traffic, certainly under 200Mbps but i= n no way should polling be discounted completely. Tuning Intel NICs works t= o an extent but offloading everything to the NIC without polling is a sure = fire way to live-lock a system in high PPS situations. So anyway, I stick to my original assessment that it can be iffy depending = on volume and scenario but I will also state that throwing polling out comp= letely discounts one of the strengths easily available on FreeBSD. That wou= ld be short-sighted, in my opinion. My recommendation is to use polling if you begin seeing lag or live-lock. I= n general use it isn't required but I assure you it can be extremely helpfu= l or detrimental. It all depends on the application of the system and the t= ype of workload it has. -Kevin On Nov 20, 2012, at 4:49 PM, "Alfred Perlstein" wrote: > On 11/20/12 2:42 PM, Jim Thompson wrote: >> On Nov 20, 2012, at 3:52 PM, Barney Cordoba w= rote: >>=20 >>> Anyone who even mentions polling should be discounted altogether. Polli= ng >>> had value when you couldn't control the interrupt delays; but interrupt >>> moderation allows you to pace the interrupts any way you like without >>> the inefficiencies of polling. >> You're entitled to your opinion, but experimental results have tended to= show yours incorrect. >>=20 >> Jim > Agree with Jim. If you want pure packet performance you burn a core to r= un a polling loop. >=20 > -Alfred From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 01:17:36 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id DB701919 for ; Wed, 21 Nov 2012 01:17:36 +0000 (UTC) (envelope-from khatfield@socllc.net) Received: from smtp151.dfw.emailsrvr.com (smtp151.dfw.emailsrvr.com [67.192.241.151]) by mx1.freebsd.org (Postfix) with ESMTP id A83BD8FC12 for ; Wed, 21 Nov 2012 01:17:36 +0000 (UTC) Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp5.relay.dfw1a.emailsrvr.com (SMTP Server) with ESMTP id EFAB859869; Tue, 20 Nov 2012 20:17:35 -0500 (EST) X-Virus-Scanned: OK Received: by smtp5.relay.dfw1a.emailsrvr.com (Authenticated sender: khatfield-AT-socllc.net) with ESMTPSA id 7208A59855; Tue, 20 Nov 2012 20:17:35 -0500 (EST) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: Re: FreeBSD boxes as a 'router'... References: <1353448328.76219.YahooMailClassic@web121602.mail.ne1.yahoo.com> <50AC08EC.8070107@mu.org> From: khatfield@socllc.net Mime-Version: 1.0 In-Reply-To: <50AC08EC.8070107@mu.org> Message-Id: <1236216591.34082.1353460654415@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> Date: Tue, 20 Nov 2012 19:17:30 -0600 To: Alfred Perlstein X-NS-Received: from Apple-iPhone5C2/1001.525(khatfield@socllc.net) SECURED(HTTPS); Wed, 21 Nov 2012 01:17:32 +0000 (UTC) Cc: Barney Cordoba , Jim Thompson , "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 01:17:37 -0000 I got side tracked and missed the new system question. In a newer system with an Intel em card then you likely won't need it. Howe= ver don't play with the HZ settings. If you still see the delays recompile = with polling on default. Verify on the NIC as well such as ifconfig em0 If it isn't on you can try to do it manually per NIC. We sometimes saw earl= y connection drops with new systems and polling enabled. (Could have also b= een a sysctl misconfiguration on our part) In either case, try the new system first and then one step at a time. Sorry for the double-posts. On Nov 20, 2012, at 4:49 PM, "Alfred Perlstein" wrote: > On 11/20/12 2:42 PM, Jim Thompson wrote: >> On Nov 20, 2012, at 3:52 PM, Barney Cordoba w= rote: >>=20 >>> Anyone who even mentions polling should be discounted altogether. Polli= ng >>> had value when you couldn't control the interrupt delays; but interrupt >>> moderation allows you to pace the interrupts any way you like without >>> the inefficiencies of polling. >> You're entitled to your opinion, but experimental results have tended to= show yours incorrect. >>=20 >> Jim > Agree with Jim. If you want pure packet performance you burn a core to r= un a polling loop. >=20 > -Alfred From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 01:19:14 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1028A9D1 for ; Wed, 21 Nov 2012 01:19:14 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-we0-f182.google.com (mail-we0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 8F0BE8FC13 for ; Wed, 21 Nov 2012 01:19:12 +0000 (UTC) Received: by mail-we0-f182.google.com with SMTP id u54so1888151wey.13 for ; Tue, 20 Nov 2012 17:19:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=vS84REOO+hCD1RWEnAOhAgkGhOnio++JC9Iyx10bED0=; b=Cc0ZEFoM/q/+xIMaTu03hsc2nVkWYskmY/oTASUz2DYJrr7fQqOK0oa5cFzbXljAM1 qpbOwI9EhYEO+JLc46qNGoD9mJLFmDOz8cllfljI1sxglaXa++V15HDDhAG78FAw5J7k jdaakBh+/FGe0gK2PNKQPWC7G8mQ09UlvPoGIGI6+T0/Fn3bXpu4c9SQp7TOs8xlaSRN TCzyD1y6Y8xwXSG/pfJesuQHINbTEGP5UbukPqeL1bGP57/uhUGqNE/NMFHTl29Q1DXh 3AhjtZHbJdWFdZ2bK0TWooB7sQhpBC8d1F9b//QmNeKhAU1BVSUaL5acrV8b1tG9obia /sMg== MIME-Version: 1.0 Received: by 10.216.200.160 with SMTP id z32mr72282wen.53.1353460752102; Tue, 20 Nov 2012 17:19:12 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.216.21.211 with HTTP; Tue, 20 Nov 2012 17:19:11 -0800 (PST) In-Reply-To: <832757660.33924.1353460119408@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> References: <1353448328.76219.YahooMailClassic@web121602.mail.ne1.yahoo.com> <50AC08EC.8070107@mu.org> <832757660.33924.1353460119408@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> Date: Tue, 20 Nov 2012 17:19:11 -0800 X-Google-Sender-Auth: jph2UMtMJBFGTexISmMzzmhYNmI Message-ID: Subject: Re: FreeBSD boxes as a 'router'... From: Adrian Chadd To: khatfield@socllc.net Content-Type: text/plain; charset=ISO-8859-1 Cc: Barney Cordoba , Jim Thompson , Alfred Perlstein , "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 01:19:14 -0000 Ok, so since people are talking about it, and i've been knee deep in at least the older intel gige interrupt moderation - at maximum pps, how exactly is the interrupt moderation giving you a livelock scenario? The biggest benefit I found when doing some forwarding work a few years ago was to write a little daemon that actually sat there and watched the interrupt rates and packet drop rates per-interface - and then tuned the interrupt moderation parameters to suit. So at the highest pps rates I wasn't swamped with interrupts. I think polling here is hiding some poor choices in driver design and network stack design.. adrian From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 02:16:57 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AFED440B for ; Wed, 21 Nov 2012 02:16:57 +0000 (UTC) (envelope-from khatfield@socllc.net) Received: from smtp151.dfw.emailsrvr.com (smtp151.dfw.emailsrvr.com [67.192.241.151]) by mx1.freebsd.org (Postfix) with ESMTP id 6631D8FC08 for ; Wed, 21 Nov 2012 02:16:57 +0000 (UTC) Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp5.relay.dfw1a.emailsrvr.com (SMTP Server) with ESMTP id A821159247; Tue, 20 Nov 2012 21:16:56 -0500 (EST) X-Virus-Scanned: OK Received: by smtp5.relay.dfw1a.emailsrvr.com (Authenticated sender: khatfield-AT-socllc.net) with ESMTPSA id EA75E59219; Tue, 20 Nov 2012 21:16:55 -0500 (EST) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: Re: FreeBSD boxes as a 'router'... References: <1353448328.76219.YahooMailClassic@web121602.mail.ne1.yahoo.com> <50AC08EC.8070107@mu.org> <832757660.33924.1353460119408@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> From: khatfield@socllc.net Mime-Version: 1.0 In-Reply-To: Message-Id: <250266404.35502.1353464214924@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> Date: Tue, 20 Nov 2012 20:16:49 -0600 To: Adrian Chadd X-NS-Received: from Apple-iPhone5C2/1001.525(khatfield@socllc.net) SECURED(HTTPS); Wed, 21 Nov 2012 02:16:53 +0000 (UTC) Cc: Barney Cordoba , Jim Thompson , Alfred Perlstein , "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 02:16:57 -0000 I may be misstating. Specifically under high burst floods either routed or being dropped by pf w= e would see the system go unresponsive to user-level applications / SSH for= example. The system would still function but it was inaccessible. To clarify as well= this was any number of floods or attacks to any ports, the behavior remain= ed. (These were not SSH ports being hit) Now we did a lot of sysctl resource tuning to correct this with some floods= but high rate would still cause the behavior. Other times the system would= simply drop all traffic (like a buffer filled or max connections) but it w= as not either case.=20 The attacks were also well within bandwidth capabilities for the pipe and n= etwork gear. All of these issues stopped upon adding polling or the overall threshold wa= s increased tremendously with polling. Yet, polling has some downsides not necessarily due to FreeBSD but applicat= ion issues. Haproxy is one example where we had handshake/premature connect= ions terminated with polling. Those issues were not present with polling di= sabled.=20 So that is my reasoning for saying that it was perfect for some things and = not for others. In the end, we spent years tinkering and it was always satisfactory but nev= er perfect. Finally we grew to the point of replacing the edge with MX80's = and left BSD to load balancing and the like. This finally resolved all issu= es for us. Albeit, we were a DDoS mitigation company running high PPS and lots of burs= ting. BSD was beautiful until we ended up needing 10Gps+ on the edge and it= was time to go Juniper. I still say BSD took us from nothing to a $30M company. So despite somethin= g's requiring tinkering with I think it is still worth the effort to put in= the testing to find what is best for your gear and environment. I got off-track but we did find one other thing. We found ipfw did seem to = reduce load on the interrupts (likely because we couldn't do near the scrub= bing with it vs pf) at any rate less filtering may also fix the issue with = the op.=20 Your forwarding - we found doing forwarding via a simple pf rule and a GRE = tunnel to an app server or by using a tool like haproxy on the router itsel= f seemed to reduce a large majority of our original stability issues (verse= s pure fw-based packet forwarding) *I also agree because as I mentioned in a previous email... (To me) our ove= rall PPS seemed to decrease from FBSD 7 to 9. No idea why but we seemed to = begin having less effect with polling as we seemed to get with polling on 7= .4. Not to say that this wasn't due to error on our part or some issue with th= e Juniper switches but we seemed to just run into more issues with newer re= leases when it came to performance with Intel 1Gbps NIC's. this later cause= d us to move more app servers to Linux because we never could get to the bo= ttom of some of those things. We do intend to revisit BSD with our new CDN = company to see if we can restandardize it for high volume traffic servers. Best, Kevin=20 On Nov 20, 2012, at 7:19 PM, "Adrian Chadd" wrote: > Ok, so since people are talking about it, and i've been knee deep in > at least the older intel gige interrupt moderation - at maximum pps, > how exactly is the interrupt moderation giving you a livelock > scenario? >=20 > The biggest benefit I found when doing some forwarding work a few > years ago was to write a little daemon that actually sat there and > watched the interrupt rates and packet drop rates per-interface - and > then tuned the interrupt moderation parameters to suit. So at the > highest pps rates I wasn't swamped with interrupts. >=20 > I think polling here is hiding some poor choices in driver design and > network stack design.. >=20 >=20 >=20 > adrian From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 02:23:28 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3FA135B8 for ; Wed, 21 Nov 2012 02:23:28 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-we0-f182.google.com (mail-we0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id BE7B18FC0C for ; Wed, 21 Nov 2012 02:23:27 +0000 (UTC) Received: by mail-we0-f182.google.com with SMTP id u54so1905515wey.13 for ; Tue, 20 Nov 2012 18:23:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=B9XiQ4U7CFgPh87/eg1YfbZKbLDWeSCsfswaZ6P4g0g=; b=y5wc9NnofBc3+qHcEpxZb3VC5Q7fnEETVu3xD5Q5PxO0wyG6PmS07nQsKprp1lxP4q 3Bo49qYD7XMLam6Rvm4K6IQBi7uPlBRi/1zUR8E04tbT3M4gwdoZUAHmyS95xH5R6xI3 Krzl1m2jTtBdzVUD84JhbPLvB9Uy1ZKuqDybgE0fWIpWZ9tZbhIFmJTmp8vaD14I7Vt+ fZXyYRiEjPxOgR7zYflHzSAbj2jr/om3IFDIXfQkSzUbz6gJ0XzwSBLo78n5g/UTp22y Ozpl7WP/6DOS7CZqSVsUhj8IH7oYPgy6A36Le927BngUpqWFMBDag5w0Msnpuwbd3jJH 3vBQ== MIME-Version: 1.0 Received: by 10.180.94.161 with SMTP id dd1mr17373835wib.17.1353464606453; Tue, 20 Nov 2012 18:23:26 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.216.21.211 with HTTP; Tue, 20 Nov 2012 18:23:26 -0800 (PST) In-Reply-To: <250266404.35502.1353464214924@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> References: <1353448328.76219.YahooMailClassic@web121602.mail.ne1.yahoo.com> <50AC08EC.8070107@mu.org> <832757660.33924.1353460119408@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> <250266404.35502.1353464214924@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> Date: Tue, 20 Nov 2012 18:23:26 -0800 X-Google-Sender-Auth: cZTpgQ457TdiP_aaTVhEaqklnYg Message-ID: Subject: Re: FreeBSD boxes as a 'router'... From: Adrian Chadd To: khatfield@socllc.net Content-Type: text/plain; charset=ISO-8859-1 Cc: Barney Cordoba , Jim Thompson , Alfred Perlstein , "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 02:23:28 -0000 The next time you have a chance to do high PPS tests, please compile in KTR and take a schedgraph trace or two. Read the schedgraph.py source in /usr/src/tools/ for a quick overview. It'd be nice to have a public record somewhere of this issue diagnosed. It may have just been some of the really odd edge case behaviour the intel 10GE NICs did (at least, before scottl and the netflix team were let at it.) Adrian From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 02:30:25 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0FC0F6FB for ; Wed, 21 Nov 2012 02:30:25 +0000 (UTC) (envelope-from khatfield@socllc.net) Received: from smtp151.dfw.emailsrvr.com (smtp151.dfw.emailsrvr.com [67.192.241.151]) by mx1.freebsd.org (Postfix) with ESMTP id CBF5D8FC18 for ; Wed, 21 Nov 2012 02:30:24 +0000 (UTC) Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp5.relay.dfw1a.emailsrvr.com (SMTP Server) with ESMTP id 2B3A058062; Tue, 20 Nov 2012 21:30:24 -0500 (EST) X-Virus-Scanned: OK Received: by smtp5.relay.dfw1a.emailsrvr.com (Authenticated sender: khatfield-AT-socllc.net) with ESMTPSA id 88C7B58099; Tue, 20 Nov 2012 21:30:23 -0500 (EST) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Subject: Re: FreeBSD boxes as a 'router'... References: <1353448328.76219.YahooMailClassic@web121602.mail.ne1.yahoo.com> <50AC08EC.8070107@mu.org> <832757660.33924.1353460119408@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> <250266404.35502.1353464214924@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> From: khatfield@socllc.net Mime-Version: 1.0 In-Reply-To: Message-Id: <1805311939.35859.1353465022550@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> Date: Tue, 20 Nov 2012 20:30:18 -0600 To: Adrian Chadd X-NS-Received: from Apple-iPhone5C2/1001.525(khatfield@socllc.net) SECURED(HTTPS); Wed, 21 Nov 2012 02:30:20 +0000 (UTC) Cc: Barney Cordoba , Jim Thompson , Alfred Perlstein , "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 02:30:25 -0000 Thank you, I certainly will. On Nov 20, 2012, at 8:23 PM, "Adrian Chadd" wrote: > The next time you have a chance to do high PPS tests, please compile > in KTR and take a schedgraph trace or two. > > Read the schedgraph.py source in /usr/src/tools/ for a quick overview. > > It'd be nice to have a public record somewhere of this issue > diagnosed. It may have just been some of the really odd edge case > behaviour the intel 10GE NICs did (at least, before scottl and the > netflix team were let at it.) > > > > Adrian From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 02:32:17 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 96E6B7A9 for ; Wed, 21 Nov 2012 02:32:17 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id DAFCC8FC08 for ; Wed, 21 Nov 2012 02:32:16 +0000 (UTC) Received: (qmail 3677 invoked from network); 21 Nov 2012 03:32:14 +0100 Received: from fw.xip.at (HELO ?127.0.0.1?) (89.207.145.147) by chile.gbit.at with SMTP; 21 Nov 2012 03:32:14 +0100 Message-ID: <50AC3D31.1070905@xip.at> Date: Wed, 21 Nov 2012 03:32:17 +0100 From: Ingo Flaschberger User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20121026 Thunderbird/16.0.2 MIME-Version: 1.0 To: Barney Cordoba Subject: Re: FreeBSD boxes as a 'router'... References: <1353454215.20382.YahooMailClassic@web121601.mail.ne1.yahoo.com> In-Reply-To: <1353454215.20382.YahooMailClassic@web121601.mail.ne1.yahoo.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 02:32:17 -0000 Am 21.11.2012 00:30, schrieb Barney Cordoba: > > --- On Tue, 11/20/12, Ingo Flaschberger wrote: > >> stems (Intel 82541GI) polling prevent live-locks. >> >> Best test: >> Loop a GigE Switch, inject a Packet and plug it into the >> test-box. > Yeah, thats a good real-world test. > > To me "performance" is not "burning a cpu" to get some extra pps. > Performance is not dropping buckets of packets. Performance is using > less cpu to do the same amount of work. > > Is a machine that benchmarks at 998Mb/s at 95% cpu really a "higher > performance" system than one that does 970Mb/s and uses 50% of the cpu? Talking about Mb/s is definitly the wrong way - forwarding performance is measured in pps. > The measure of performance is to manage an entire load without dropping > any packets. If your machine goes into live-lock, then you need more > machine. Hacking it so that it drops packets is hardly a solution. No, because standard internet traffic has not 100% 64b packets - but when a hacker attacks - it has (dos). Then it's important to know - who is the attacker and keep the box up even if it drops packets. If you don't like packet drops - go - buy some juniper (which also use FreeBSD). Kind regards, Ingo Flaschberger From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 03:08:10 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E1974D31 for ; Wed, 21 Nov 2012 03:08:10 +0000 (UTC) (envelope-from bright@mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id C27448FC16 for ; Wed, 21 Nov 2012 03:08:10 +0000 (UTC) Received: from Alfreds-MacBook-Pro-5.local (c-67-180-208-218.hsd1.ca.comcast.net [67.180.208.218]) by elvis.mu.org (Postfix) with ESMTPSA id 194161A3C1A; Tue, 20 Nov 2012 19:08:10 -0800 (PST) Message-ID: <50AC4599.7020407@mu.org> Date: Tue, 20 Nov 2012 19:08:09 -0800 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:16.0) Gecko/20121026 Thunderbird/16.0.2 MIME-Version: 1.0 To: Barney Cordoba Subject: Re: FreeBSD boxes as a 'router'... References: <1353454215.20382.YahooMailClassic@web121601.mail.ne1.yahoo.com> In-Reply-To: <1353454215.20382.YahooMailClassic@web121601.mail.ne1.yahoo.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, Ingo Flaschberger X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 03:08:11 -0000 On 11/20/12 3:30 PM, Barney Cordoba wrote: > > --- On Tue, 11/20/12, Ingo Flaschberger wrote: > >> From: Ingo Flaschberger >> Subject: Re: FreeBSD boxes as a 'router'... >> To: freebsd-net@freebsd.org >> Date: Tuesday, November 20, 2012, 6:04 PM >> Am 20.11.2012 23:49, schrieb Alfred >> Perlstein: >>> On 11/20/12 2:42 PM, Jim Thompson wrote: >>>> On Nov 20, 2012, at 3:52 PM, Barney Cordoba >> wrote: >>>> You're entitled to your opinion, but experimental >> results have tended to show yours incorrect. >>>> Jim >>> Agree with Jim. If you want pure packet >> performance you burn a core to run a polling loop. >> >> At new systems, without polling I had better performance and >> no live-locks, >> at old systems (Intel 82541GI) polling prevent live-locks. >> >> Best test: >> Loop a GigE Switch, inject a Packet and plug it into the >> test-box. > Yeah, thats a good real-world test. > > To me "performance" is not "burning a cpu" to get some extra pps. > Performance is not dropping buckets of packets. Performance is using > less cpu to do the same amount of work. > > Is a machine that benchmarks at 998Mb/s at 95% cpu really a "higher > performance" system than one that does 970Mb/s and uses 50% of the cpu? > > The measure of performance is to manage an entire load without dropping > any packets. If your machine goes into live-lock, then you need more > machine. Hacking it so that it drops packets is hardly a solution. > Any free CPU is wasted CPU. (unless you're concerned about power consumption, then it's debatable). -Alfred From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 06:26:33 2012 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BECEF904; Wed, 21 Nov 2012 06:26:33 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from cell.glebius.int.ru (glebius.int.ru [81.19.64.117]) by mx1.freebsd.org (Postfix) with ESMTP id 38A648FC0C; Wed, 21 Nov 2012 06:26:33 +0000 (UTC) Received: from cell.glebius.int.ru (localhost [127.0.0.1]) by cell.glebius.int.ru (8.14.5/8.14.5) with ESMTP id qAL6QVuv074945; Wed, 21 Nov 2012 10:26:31 +0400 (MSK) (envelope-from glebius@FreeBSD.org) Received: (from glebius@localhost) by cell.glebius.int.ru (8.14.5/8.14.5/Submit) id qAL6QVmx074944; Wed, 21 Nov 2012 10:26:31 +0400 (MSK) (envelope-from glebius@FreeBSD.org) X-Authentication-Warning: cell.glebius.int.ru: glebius set sender to glebius@FreeBSD.org using -f Date: Wed, 21 Nov 2012 10:26:31 +0400 From: Gleb Smirnoff To: Jack Vogel Subject: Re: igb diver crashes in head@241037 Message-ID: <20121121062631.GJ67660@glebius.int.ru> References: <50AA8F24.7080604@gmail.com> <20121120111833.GC67660@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Karim Fodil-Lemelin , jfv@FreeBSD.org, freebsd-net@FreeBSD.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 06:26:33 -0000 Jack, On Tue, Nov 20, 2012 at 09:19:54AM -0800, Jack Vogel wrote: J> > I'd suggest the following code: J> > J> > if (m) J> > drbr_enqueue(ifp, txr->br, m); J> > err = igb_mq_start_locked(ifp, txr, NULL); J> > J> > Which eventually leads us to all invocations of igb_mq_start_locked() J> > called J> > with third argument as NULL. This allows us to simplify this function. J> > J> > Patch for review attached. J> > J> > J> Yes Gleb, I already have code in my internal tree which simply removes an J> mbuf J> pointer form the start_locked call and ALWAYS does a dequeue, start J> similarly J> will always enqueue. I just have been busy with ixgbe for a bit and have J> not gotten J> it committed yet. Since ixgbe work is performance tuning and this patch closes a kernel crash, I'd ask to preempt the ixgbe job with this patch. :) Or you can approve my patch and I will check it in. -- Totus tuus, Glebius. From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 07:32:47 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 63578F20 for ; Wed, 21 Nov 2012 07:32:47 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id ACD4E8FC12 for ; Wed, 21 Nov 2012 07:32:46 +0000 (UTC) Received: (qmail 516 invoked from network); 21 Nov 2012 09:05:27 -0000 Received: from c00l3r.networx.ch (HELO [127.0.0.1]) ([62.48.2.2]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 21 Nov 2012 09:05:27 -0000 Message-ID: <50AC8393.3060001@freebsd.org> Date: Wed, 21 Nov 2012 08:32:35 +0100 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20121026 Thunderbird/16.0.2 MIME-Version: 1.0 To: khatfield@socllc.net Subject: Re: FreeBSD boxes as a 'router'... References: <1353448328.76219.YahooMailClassic@web121602.mail.ne1.yahoo.com> <50AC08EC.8070107@mu.org> <832757660.33924.1353460119408@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> <250266404.35502.1353464214924@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> In-Reply-To: <250266404.35502.1353464214924@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Barney Cordoba , Adrian Chadd , Alfred Perlstein , Jim Thompson , "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 07:32:47 -0000 On 21.11.2012 03:16, khatfield@socllc.net wrote: > I may be misstating. > > Specifically under high burst floods either routed or being dropped by pf we would see the system > go unresponsive to user-level applications / SSH for example. > > The system would still function but it was inaccessible. To clarify as well this was any number > of floods or attacks to any ports, the behavior remained. (These were not SSH ports being hit) I'm working on a hybrid interrupt/polling with live-lock prevention scheme in my svn branch. It works with a combination of disabling interrupts in interrupt context and then having an ithread loop over the RX DMA queue until it reaches the hardware and is done. Only then interrupts are re-enabled again. On a busy system it may never go back to interrupt. To prevent live-lock the ithread gives up the CPU after a normal quantum to let other threads/processes run as well. After that it gets immediately re-scheduled again with a sufficient high priority not get starved out by userspace. With multiple RX queues and MSI-X interrupts as many ithreads as available cores can be run and none of them will live-lock. I'm also looking at using the CoDel algorithm for totally maxed out systems to prevent long FIFO packet drop chains in the NIC. Think of it as RED queue management but for the input queue. That way we can use distributed single packet loss as a signalling mechanism for the senders to slow down. For a misbehaving sender blasting away this obviously doesn't help much. It improves the chance of good packets making it through though. While live-lock prevention is good you still won't be able to log in via ssh through an overloaded interface. Any other interface will work w/o packet loss instead. So far I've fully converted fxp(4) to this new scheme because it is one of the simpler drivers with sufficient documentation. And 100Mbit is easy to saturate. The bge(4) driver is mostly converted but not tested due to lack of hardware, which should arrive later this week though. The em(4), and with it due to similarity igb(4) and ixgbe(4) family, is in the works as well. Again hardware is on the way for testing. When this work has stabilized I'm looking for testers to put it through the paces. If you're interested and have a suitable test bed then drop me an email to get notified. -- Andre > Now we did a lot of sysctl resource tuning to correct this with some floods but high rate would > still cause the behavior. Other times the system would simply drop all traffic (like a buffer > filled or max connections) but it was not either case. > > The attacks were also well within bandwidth capabilities for the pipe and network gear. > > All of these issues stopped upon adding polling or the overall threshold was increased > tremendously with polling. > > Yet, polling has some downsides not necessarily due to FreeBSD but application issues. Haproxy is > one example where we had handshake/premature connections terminated with polling. Those issues > were not present with polling disabled. > > So that is my reasoning for saying that it was perfect for some things and not for others. > > In the end, we spent years tinkering and it was always satisfactory but never perfect. Finally we > grew to the point of replacing the edge with MX80's and left BSD to load balancing and the like. > This finally resolved all issues for us. > > Albeit, we were a DDoS mitigation company running high PPS and lots of bursting. BSD was > beautiful until we ended up needing 10Gps+ on the edge and it was time to go Juniper. > > I still say BSD took us from nothing to a $30M company. So despite something's requiring > tinkering with I think it is still worth the effort to put in the testing to find what is best > for your gear and environment. > > I got off-track but we did find one other thing. We found ipfw did seem to reduce load on the > interrupts (likely because we couldn't do near the scrubbing with it vs pf) at any rate less > filtering may also fix the issue with the op. > > Your forwarding - we found doing forwarding via a simple pf rule and a GRE tunnel to an app > server or by using a tool like haproxy on the router itself seemed to reduce a large majority of > our original stability issues (verses pure fw-based packet forwarding) > > *I also agree because as I mentioned in a previous email... (To me) our overall PPS seemed to > decrease from FBSD 7 to 9. No idea why but we seemed to begin having less effect with polling as > we seemed to get with polling on 7.4. > > Not to say that this wasn't due to error on our part or some issue with the Juniper switches but > we seemed to just run into more issues with newer releases when it came to performance with Intel > 1Gbps NIC's. this later caused us to move more app servers to Linux because we never could get to > the bottom of some of those things. We do intend to revisit BSD with our new CDN company to see > if we can restandardize it for high volume traffic servers. > > Best, Kevin > > > > On Nov 20, 2012, at 7:19 PM, "Adrian Chadd" wrote: > >> Ok, so since people are talking about it, and i've been knee deep in at least the older intel >> gige interrupt moderation - at maximum pps, how exactly is the interrupt moderation giving you >> a livelock scenario? >> >> The biggest benefit I found when doing some forwarding work a few years ago was to write a >> little daemon that actually sat there and watched the interrupt rates and packet drop rates >> per-interface - and then tuned the interrupt moderation parameters to suit. So at the highest >> pps rates I wasn't swamped with interrupts. >> >> I think polling here is hiding some poor choices in driver design and network stack design.. >> >> >> >> adrian > _______________________________________________ freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org" > > From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 07:55:08 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9C5498AD; Wed, 21 Nov 2012 07:55:08 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wi0-f178.google.com (mail-wi0-f178.google.com [209.85.212.178]) by mx1.freebsd.org (Postfix) with ESMTP id 01F578FC16; Wed, 21 Nov 2012 07:55:07 +0000 (UTC) Received: by mail-wi0-f178.google.com with SMTP id hm6so1361987wib.13 for ; Tue, 20 Nov 2012 23:55:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=Jx6HiNpcUXnha06xdduYuTZeI36zpAPNoYJ2tu4SCDM=; b=Q5b/hcfZEn1xGZDlHHqRKcdX6m9CJ7+jhC1QlRsqbn62jp3Kcw/FLLC6CIhxiBysH1 ZnHqK0uMmWxU7LgGis0ozvqiLUDWPuNotIaZqjSrbeO0rMAJypjEJ5KKuL4voaOHbcIP MnGNM7Z3mfCI3OAVVsX/CAK+3EDR5EGXCIojVGwrR2ydwohdVRB8+gG/0dob3hdtAp4G qNaRqLitCm0fT6moYrhmzgThDdAQl1s9T6wk+Rq2govaAT69/+IkMfzxux/nizKM8rA9 rA2GHzWtoRvHAHbUaOWzhxj8l1jwjOCCTCekiRiPaORv0L+aizf2QSISc2BzINqOT58s 5Q9g== MIME-Version: 1.0 Received: by 10.180.99.1 with SMTP id em1mr15497030wib.17.1353484501064; Tue, 20 Nov 2012 23:55:01 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.216.21.211 with HTTP; Tue, 20 Nov 2012 23:55:00 -0800 (PST) In-Reply-To: <50AC8393.3060001@freebsd.org> References: <1353448328.76219.YahooMailClassic@web121602.mail.ne1.yahoo.com> <50AC08EC.8070107@mu.org> <832757660.33924.1353460119408@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> <250266404.35502.1353464214924@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> <50AC8393.3060001@freebsd.org> Date: Tue, 20 Nov 2012 23:55:00 -0800 X-Google-Sender-Auth: EBsuC973devBRVcAu3widKNevJA Message-ID: Subject: Re: FreeBSD boxes as a 'router'... From: Adrian Chadd To: Andre Oppermann Content-Type: text/plain; charset=ISO-8859-1 Cc: Barney Cordoba , Jim Thompson , Alfred Perlstein , khatfield@socllc.net, "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 07:55:08 -0000 Something that has popped up a few times, even recently, is breaking out of an RX loop after you service a number of frames. During stupidly high levels of RX, you may find the NIC happily receiving frames faster than you can service the RX queue. If this occurs, you could end up just plain being stuck there. So what I've done in the past is to loop over a certain number of frames, then schedule a taskqueue to service whatever's left over. I've also had to do some proactive frame dropping at the driver layer when under ridiculous levels of RX load, but that's a different story. I wonder how many drivers break out of an RX loop after a fixed amount of time.. adrian From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 08:04:51 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E0E3136E; Wed, 21 Nov 2012 08:04:51 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-la0-f54.google.com (mail-la0-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id DF5C38FC15; Wed, 21 Nov 2012 08:04:50 +0000 (UTC) Received: by mail-la0-f54.google.com with SMTP id j13so6484203lah.13 for ; Wed, 21 Nov 2012 00:04:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=r6JJWu0MwFw+Qa66mxu8SAitLx+iDMVGk4ViN5CYjpI=; b=1H6jiOGnXFV2yoB19Tz92YuiYFpnLvmuR/I7RSaOhzUTheC65sU47w16ei/cpj4Eou jKG0VKL6CJoPcvN0fAGQN1NxVowTjWv6uQGK1fG85soq3xwFxfTdAOQAKKmuV45phfcU kyte/TBog9g09Aqk6GMpeelxAlkcjAVVYg3/Hee50TIHcdEJ1dJNxP/WT15sN2l8wnOJ J+Ft4fiPyi0TtHXIGmNibWCf0BJJ0W0qjLnfiAXN/YaLkeNcHFlZoK0zUIUmMw3BKVU4 qTHiNM/qAPSKlwpc9ZcHqmu9XV2yl7gs0Dbf4n0UPy/vKx1McnPeHrYK+BK7MUhf8Wyk 50sw== MIME-Version: 1.0 Received: by 10.112.38.226 with SMTP id j2mr7392434lbk.128.1353485089664; Wed, 21 Nov 2012 00:04:49 -0800 (PST) Sender: rizzo.unipi@gmail.com Received: by 10.112.20.131 with HTTP; Wed, 21 Nov 2012 00:04:49 -0800 (PST) In-Reply-To: <50AC8393.3060001@freebsd.org> References: <1353448328.76219.YahooMailClassic@web121602.mail.ne1.yahoo.com> <50AC08EC.8070107@mu.org> <832757660.33924.1353460119408@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> <250266404.35502.1353464214924@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> <50AC8393.3060001@freebsd.org> Date: Wed, 21 Nov 2012 00:04:49 -0800 X-Google-Sender-Auth: XnETzDF2PeezYxI6IrnSBRqw9Q8 Message-ID: Subject: Re: FreeBSD boxes as a 'router'... From: Luigi Rizzo To: Andre Oppermann Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Barney Cordoba , Adrian Chadd , khatfield@socllc.net, "freebsd-net@freebsd.org" , Jim Thompson , Alfred Perlstein X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 08:04:52 -0000 On Tue, Nov 20, 2012 at 11:32 PM, Andre Oppermann wrote: > On 21.11.2012 03:16, khatfield@socllc.net wrote: > >> I may be misstating. >> >> Specifically under high burst floods either routed or being dropped by pf >> we would see the system >> go unresponsive to user-level applications / SSH for example. >> >> The system would still function but it was inaccessible. To clarify as >> well this was any number >> of floods or attacks to any ports, the behavior remained. (These were not >> SSH ports being hit) >> > > I'm working on a hybrid interrupt/polling with live-lock prevention > scheme in my svn branch. It works with a combination of disabling > interrupts in interrupt context and then having an ithread loop over > the RX DMA queue until it reaches the hardware and is done. Only > then interrupts are re-enabled again. On a busy system it may never > go back to interrupt. To prevent live-lock the ithread gives up the > CPU after a normal quantum to let other threads/processes run as well. > After that it gets immediately re-scheduled again with a sufficient > high priority not get starved out by userspace. > > very cool. this seems similar to NAPI. The only adjustment i'd suggest to your scheme, if possible, is to add some control (as it existed in the old polling architecture) to make sure that userspace is not starved by the ithreads and other kernel tasks (otherwise you can have livelock, as it happens with NAPI). I am afraid that simple priorities do not work, you either need some kind of fair scheduler, or put some hard limit on the cpu fraction used by the kernel tasks when there are userspace processes around. cheers luigi With multiple RX queues and MSI-X interrupts as many ithreads as available > cores can be run and none of them will live-lock. I'm also looking at > using the CoDel algorithm for totally maxed out systems to prevent long > FIFO packet drop chains in the NIC. Think of it as RED queue management > but for the input queue. That way we can use distributed single packet > loss as a signalling mechanism for the senders to slow down. For a > misbehaving sender blasting away this obviously doesn't help much. It > improves the chance of good packets making it through though. > > While live-lock prevention is good you still won't be able to log in > via ssh through an overloaded interface. Any other interface will > work w/o packet loss instead. > > So far I've fully converted fxp(4) to this new scheme because it is one > of the simpler drivers with sufficient documentation. And 100Mbit is > easy to saturate. > > The bge(4) driver is mostly converted but not tested due to lack of > hardware, which should arrive later this week though. > > The em(4), and with it due to similarity igb(4) and ixgbe(4) family, > is in the works as well. Again hardware is on the way for testing. > > When this work has stabilized I'm looking for testers to put it through > the paces. If you're interested and have a suitable test bed then drop > me an email to get notified. > > -- > Andre > > > Now we did a lot of sysctl resource tuning to correct this with some >> floods but high rate would >> still cause the behavior. Other times the system would simply drop all >> traffic (like a buffer >> filled or max connections) but it was not either case. >> >> The attacks were also well within bandwidth capabilities for the pipe and >> network gear. >> >> All of these issues stopped upon adding polling or the overall threshold >> was increased >> tremendously with polling. >> >> Yet, polling has some downsides not necessarily due to FreeBSD but >> application issues. Haproxy is >> one example where we had handshake/premature connections terminated with >> polling. Those issues >> were not present with polling disabled. >> >> So that is my reasoning for saying that it was perfect for some things >> and not for others. >> >> In the end, we spent years tinkering and it was always satisfactory but >> never perfect. Finally we >> grew to the point of replacing the edge with MX80's and left BSD to load >> balancing and the like. >> This finally resolved all issues for us. >> >> Albeit, we were a DDoS mitigation company running high PPS and lots of >> bursting. BSD was >> beautiful until we ended up needing 10Gps+ on the edge and it was time to >> go Juniper. >> >> I still say BSD took us from nothing to a $30M company. So despite >> something's requiring >> tinkering with I think it is still worth the effort to put in the testing >> to find what is best >> for your gear and environment. >> >> I got off-track but we did find one other thing. We found ipfw did seem >> to reduce load on the >> interrupts (likely because we couldn't do near the scrubbing with it vs >> pf) at any rate less >> filtering may also fix the issue with the op. >> >> Your forwarding - we found doing forwarding via a simple pf rule and a >> GRE tunnel to an app >> server or by using a tool like haproxy on the router itself seemed to >> reduce a large majority of >> our original stability issues (verses pure fw-based packet forwarding) >> >> *I also agree because as I mentioned in a previous email... (To me) our >> overall PPS seemed to >> decrease from FBSD 7 to 9. No idea why but we seemed to begin having less >> effect with polling as >> we seemed to get with polling on 7.4. >> >> Not to say that this wasn't due to error on our part or some issue with >> the Juniper switches but >> we seemed to just run into more issues with newer releases when it came >> to performance with Intel >> 1Gbps NIC's. this later caused us to move more app servers to Linux >> because we never could get to >> the bottom of some of those things. We do intend to revisit BSD with our >> new CDN company to see >> if we can restandardize it for high volume traffic servers. >> >> Best, Kevin >> >> >> >> On Nov 20, 2012, at 7:19 PM, "Adrian Chadd" wrote: >> >> Ok, so since people are talking about it, and i've been knee deep in at >>> least the older intel >>> gige interrupt moderation - at maximum pps, how exactly is the interrupt >>> moderation giving you >>> a livelock scenario? >>> >>> The biggest benefit I found when doing some forwarding work a few years >>> ago was to write a >>> little daemon that actually sat there and watched the interrupt rates >>> and packet drop rates >>> per-interface - and then tuned the interrupt moderation parameters to >>> suit. So at the highest >>> pps rates I wasn't swamped with interrupts. >>> >>> I think polling here is hiding some poor choices in driver design and >>> network stack design.. >>> >>> >>> >>> adrian >>> >> ______________________________**_________________ freebsd-net@freebsd.orgmailing list >> http://lists.freebsd.org/**mailman/listinfo/freebsd-netTo unsubscribe, send any mail to >> "freebsd-net-unsubscribe@**freebsd.org >> " >> >> >> > ______________________________**_________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/**mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@**freebsd.org > " > -- -----------------------------------------+------------------------------- Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione http://www.iet.unipi.it/~luigi/ . Universita` di Pisa TEL +39-050-2211611 . via Diotisalvi 2 Mobile +39-338-6809875 . 56122 PISA (Italy) -----------------------------------------+------------------------------- From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 08:30:10 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 599ADD2F for ; Wed, 21 Nov 2012 08:30:10 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id AD3348FC19 for ; Wed, 21 Nov 2012 08:30:09 +0000 (UTC) Received: (qmail 783 invoked from network); 21 Nov 2012 10:02:55 -0000 Received: from c00l3r.networx.ch (HELO [127.0.0.1]) ([62.48.2.2]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 21 Nov 2012 10:02:55 -0000 Message-ID: <50AC910C.4030004@freebsd.org> Date: Wed, 21 Nov 2012 09:30:04 +0100 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20121026 Thunderbird/16.0.2 MIME-Version: 1.0 To: Adrian Chadd Subject: Re: FreeBSD boxes as a 'router'... References: <1353448328.76219.YahooMailClassic@web121602.mail.ne1.yahoo.com> <50AC08EC.8070107@mu.org> <832757660.33924.1353460119408@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> <250266404.35502.1353464214924@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> <50AC8393.3060001@freebsd.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Barney Cordoba , Jim Thompson , Alfred Perlstein , khatfield@socllc.net, "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 08:30:10 -0000 On 21.11.2012 08:55, Adrian Chadd wrote: > Something that has popped up a few times, even recently, is breaking > out of an RX loop after you service a number of frames. That is what I basically described. > During stupidly high levels of RX, you may find the NIC happily > receiving frames faster than you can service the RX queue. If this > occurs, you could end up just plain being stuck there. That's the live-lock. > So what I've done in the past is to loop over a certain number of > frames, then schedule a taskqueue to service whatever's left over. Taskqueue's shouldn't be used anymore. We've got ithreads now. Contrary to popular belief (and due to poor documentation) an ithread does not run at interrupt level. Only the fast interrupt handler does that. The ithread is a normal kernel thread tied to an fast interrupt handler and trailing it whenever it said INTR_SCHEDULE_ITHREAD. > I've also had to do some proactive frame dropping at the driver layer > when under ridiculous levels of RX load, but that's a different story. That's the CoDel stuff I mentioned where it tries to drop frames in a fair manner. > I wonder how many drivers break out of an RX loop after a fixed amount of time.. Most. The problem seems to be that the taskqueue runs at high priority effectively starving out other threads/processes again. -- Andre From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 08:36:55 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 998F6444 for ; Wed, 21 Nov 2012 08:36:55 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id E94578FC12 for ; Wed, 21 Nov 2012 08:36:54 +0000 (UTC) Received: (qmail 822 invoked from network); 21 Nov 2012 10:09:39 -0000 Received: from c00l3r.networx.ch (HELO [127.0.0.1]) ([62.48.2.2]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 21 Nov 2012 10:09:39 -0000 Message-ID: <50AC92A0.2000406@freebsd.org> Date: Wed, 21 Nov 2012 09:36:48 +0100 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20121026 Thunderbird/16.0.2 MIME-Version: 1.0 To: Luigi Rizzo Subject: Re: FreeBSD boxes as a 'router'... References: <1353448328.76219.YahooMailClassic@web121602.mail.ne1.yahoo.com> <50AC08EC.8070107@mu.org> <832757660.33924.1353460119408@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> <250266404.35502.1353464214924@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> <50AC8393.3060001@freebsd.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Barney Cordoba , Adrian Chadd , khatfield@socllc.net, "freebsd-net@freebsd.org" , Jim Thompson , Alfred Perlstein X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 08:36:55 -0000 On 21.11.2012 09:04, Luigi Rizzo wrote: > On Tue, Nov 20, 2012 at 11:32 PM, Andre Oppermann wrote: > >> On 21.11.2012 03:16, khatfield@socllc.net wrote: >> >>> I may be misstating. >>> >>> Specifically under high burst floods either routed or being dropped by pf >>> we would see the system >>> go unresponsive to user-level applications / SSH for example. >>> >>> The system would still function but it was inaccessible. To clarify as >>> well this was any number >>> of floods or attacks to any ports, the behavior remained. (These were not >>> SSH ports being hit) >>> >> >> I'm working on a hybrid interrupt/polling with live-lock prevention >> scheme in my svn branch. It works with a combination of disabling >> interrupts in interrupt context and then having an ithread loop over >> the RX DMA queue until it reaches the hardware and is done. Only >> then interrupts are re-enabled again. On a busy system it may never >> go back to interrupt. To prevent live-lock the ithread gives up the >> CPU after a normal quantum to let other threads/processes run as well. >> After that it gets immediately re-scheduled again with a sufficient >> high priority not get starved out by userspace. >> >> > very cool. this seems similar to NAPI. I've heard about NAPI but haven't looked at it. So I'm not sure how it works internally. In this case no special logic or kernel API support is required. Every driver can be converted and immediately use it. > The only adjustment i'd suggest to your scheme, if possible, is to add > some control (as it existed in the old polling architecture) to make sure > that userspace is not starved by the ithreads and other kernel tasks > (otherwise you can have livelock, as it happens with NAPI). > I am afraid that simple priorities do not work, you either need some > kind of fair scheduler, or put some hard limit on the cpu fraction used > by the kernel tasks when there are userspace processes around. That's the quantum stuff I talked about. When the ithread has used up its CPU quantum it gives up the CPU but schedules itself again at the same time. Also contrary to its name the ithread does not run in interrupt context but as a normal kernel thread with elevated priority. I have to thank Attilio though for a detailed explanation of what is going on behind the scenes in the interrupt and scheduler code to get a grip at the optimal approach for network drivers. The bus_setup_intr(9) and ithread documentation is very poor and misleading at the moment. I intend to fix that soon. -- Andre From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 08:53:21 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9D9E9960; Wed, 21 Nov 2012 08:53:21 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-la0-f54.google.com (mail-la0-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id 9A21A8FC0C; Wed, 21 Nov 2012 08:53:20 +0000 (UTC) Received: by mail-la0-f54.google.com with SMTP id j13so6520250lah.13 for ; Wed, 21 Nov 2012 00:53:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=YupkuT8hrD7LDNuloj7sGQggq6PCNn24n9sijcpejt0=; b=Ja1HAVWx+arqZlD/OlkJzU5tiOUSNFSVivUX+VONl9WWwLYG5aE4AIJkwXE9l7kj+c 68UpFJS9k9FGl07mZ5dukmtS3Vxj3xxQRKsJV4rmH8Q7fwWlQM+bwnaLFoB9rA2PIbfy Ev33GHkle3Zwohwij5M+sqBHFLxKuDsmLPsoWzKJaLuQOuGL1XdMxkaITe++DwRVQd9/ 3vJrOh4A3AxZE7KWmfhWZ89SC10iCyofjtWC4R49LUIu/HkGWFFdwtof0TuzFOzpXLGg Y9ePi7YS7UogxzK0tdMImIkIy6LcEGy2GVI9Jwy6ylfog2m6K4zSjJb1QJ5QKFUmq3rw BScA== MIME-Version: 1.0 Received: by 10.152.144.164 with SMTP id sn4mr11590636lab.57.1353487999147; Wed, 21 Nov 2012 00:53:19 -0800 (PST) Sender: rizzo.unipi@gmail.com Received: by 10.112.20.131 with HTTP; Wed, 21 Nov 2012 00:53:18 -0800 (PST) In-Reply-To: <50AC92A0.2000406@freebsd.org> References: <1353448328.76219.YahooMailClassic@web121602.mail.ne1.yahoo.com> <50AC08EC.8070107@mu.org> <832757660.33924.1353460119408@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> <250266404.35502.1353464214924@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> <50AC8393.3060001@freebsd.org> <50AC92A0.2000406@freebsd.org> Date: Wed, 21 Nov 2012 00:53:18 -0800 X-Google-Sender-Auth: 8SnuSXM9ORiJDucels07LsVt6JU Message-ID: Subject: Re: FreeBSD boxes as a 'router'... From: Luigi Rizzo To: Andre Oppermann Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Barney Cordoba , Adrian Chadd , khatfield@socllc.net, "freebsd-net@freebsd.org" , Jim Thompson , Alfred Perlstein X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 08:53:21 -0000 On Wed, Nov 21, 2012 at 12:36 AM, Andre Oppermann wrote: > On 21.11.2012 09:04, Luigi Rizzo wrote: > >> On Tue, Nov 20, 2012 at 11:32 PM, Andre Oppermann >> wrote: >> >>> ... >> >> very cool. this seems similar to NAPI. >> > > I've heard about NAPI but haven't looked at it. So I'm not sure how it > have a look at some of the intel nic drivers in linux, they are very similar to the bsd ones so you can easily compare. NAPI is basically the same thing that you describe, there is no special logic and the only kernel support is (i believe) some device independent code that probably you'll end up writing too after the first 2-3 drivers, to avoid boring replications. works internally. In this case no special logic or kernel API support > is required. Every driver can be converted and immediately use it. > > > The only adjustment i'd suggest to your scheme, if possible, is to add >> some control (as it existed in the old polling architecture) to make sure >> that userspace is not starved by the ithreads and other kernel tasks >> (otherwise you can have livelock, as it happens with NAPI). >> I am afraid that simple priorities do not work, you either need some >> kind of fair scheduler, or put some hard limit on the cpu fraction used >> by the kernel tasks when there are userspace processes around. >> > > That's the quantum stuff I talked about. When the ithread has used up > its CPU quantum it gives up the CPU but schedules itself again at the > same time. Also contrary to its name the ithread does not run in > interrupt context but as a normal kernel thread with elevated priority. > it is the elevated priority that worries me, because it has potential to preempt userspace and cause livelock. Again, lacking a proper fair process scheduler, i think the only reliable way is to try and track the (approximate) cpu cycles consumed overall by the various ithreads over, say, the current tick, and once you go above the threshold drop the priority of those threads just above IDLEPRI. Then when the next tick arrives you raise the priorities again. Compared to when i did polling in 2000 (when i wanted to support the i486 soekris), now we can probably assume that a cheap timecounter is available on all architectures (even ARM and MIPS should have some tsc-like thing, right ?) and the cycles used vs cycles per tick can be accounted with a relatively fine grain. cheers luigi From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 09:11:17 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id ED566F63 for ; Wed, 21 Nov 2012 09:11:17 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id 4350A8FC16 for ; Wed, 21 Nov 2012 09:11:17 +0000 (UTC) Received: (qmail 1000 invoked from network); 21 Nov 2012 10:44:02 -0000 Received: from c00l3r.networx.ch (HELO [127.0.0.1]) ([62.48.2.2]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 21 Nov 2012 10:44:02 -0000 Message-ID: <50AC9AAF.8040705@freebsd.org> Date: Wed, 21 Nov 2012 10:11:11 +0100 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20121026 Thunderbird/16.0.2 MIME-Version: 1.0 To: Luigi Rizzo Subject: Re: FreeBSD boxes as a 'router'... References: <1353448328.76219.YahooMailClassic@web121602.mail.ne1.yahoo.com> <50AC08EC.8070107@mu.org> <832757660.33924.1353460119408@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> <250266404.35502.1353464214924@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> <50AC8393.3060001@freebsd.org> <50AC92A0.2000406@freebsd.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Barney Cordoba , Adrian Chadd , khatfield@socllc.net, "freebsd-net@freebsd.org" , Jim Thompson , Alfred Perlstein X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 09:11:18 -0000 On 21.11.2012 09:53, Luigi Rizzo wrote: > On Wed, Nov 21, 2012 at 12:36 AM, Andre Oppermann wrote: >> On 21.11.2012 09:04, Luigi Rizzo wrote: >> The only adjustment i'd suggest to your scheme, if possible, is to add >>> some control (as it existed in the old polling architecture) to make sure >>> that userspace is not starved by the ithreads and other kernel tasks >>> (otherwise you can have livelock, as it happens with NAPI). >>> I am afraid that simple priorities do not work, you either need some >>> kind of fair scheduler, or put some hard limit on the cpu fraction used >>> by the kernel tasks when there are userspace processes around. >>> >> >> That's the quantum stuff I talked about. When the ithread has used up >> its CPU quantum it gives up the CPU but schedules itself again at the >> same time. Also contrary to its name the ithread does not run in >> interrupt context but as a normal kernel thread with elevated priority. >> > > it is the elevated priority that worries me, because it has potential > to preempt userspace and cause livelock. > Again, lacking a proper fair process scheduler, i think the only reliable > way is > to try and track the (approximate) cpu cycles consumed overall by the > various ithreads over, say, the current tick, and once you go above > the threshold drop the priority of those threads just above IDLEPRI. > Then when the next tick arrives you raise the priorities again. That is what happens. The first time the ithread is scheduled it is run at high priority. When it yields after consuming its quantum it drops much lower but above heavily nice'd processes. Have to verify the exact level though. > Compared to when i did polling in 2000 (when i wanted to support the i486 > soekris), > now we can probably assume that a cheap timecounter is available on > all architectures (even ARM and MIPS should have some tsc-like thing, right > ?) > and the cycles used vs cycles per tick can be accounted with a relatively > fine grain. I hope to use as much existing kernel infrastructure as possible and to keep it simple. I certainly don't want to do cross-core load analysis. -- Andre From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 12:10:08 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D62CC420 for ; Wed, 21 Nov 2012 12:10:08 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: from mail-qa0-f54.google.com (mail-qa0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id 84DF28FC13 for ; Wed, 21 Nov 2012 12:10:08 +0000 (UTC) Received: by mail-qa0-f54.google.com with SMTP id g24so1720028qab.13 for ; Wed, 21 Nov 2012 04:10:07 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=references:in-reply-to:mime-version:content-transfer-encoding :content-type:message-id:cc:x-mailer:from:subject:date:to :x-gm-message-state; bh=WrwjPlah3fDkuwnHTJE6nHpztkc6p97GhPBC2AM8jrc=; b=WOzllRRYgb1TdAZq+NfZruvvXmrEWN0bz6v/B8JqNTOxYiTqgDifiNromu11xnqB6E B0zTqG4r4d+UeP6i05gdcTqwQIFYcEymKCdpFr7uKriJENLYtuPSFgxetNwoSpamaLzA rjM+/RHJDXNi9zcSDXXgMmaq4X1kGjMis9GzuX/rBF5I0mcFx+zrINFey5m4Ja89wlL0 BcK7AWs2IkXqEaFDalNU3MMwnESs+moGTCSoDbHsvRmZSWDz3lFztDnFBedfHicyHATt nqXu3fdx2ZqjXHg26aqB6sBDfsiedj9ei3upXVrSKfrU54M9YOtzmP8F5a3NrvlglpVp 64Mw== Received: by 10.49.1.194 with SMTP id 2mr20512954qeo.56.1353499807655; Wed, 21 Nov 2012 04:10:07 -0800 (PST) Received: from [192.168.11.110] (ool-4a58bea2.dyn.optonline.net. [74.88.190.162]) by mx.google.com with ESMTPS id fl1sm130774qab.14.2012.11.21.04.10.06 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 21 Nov 2012 04:10:06 -0800 (PST) References: <50A372EC.5080002@freebsd.org> In-Reply-To: <50A372EC.5080002@freebsd.org> Mime-Version: 1.0 (1.0) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Message-Id: <2068A84F-0B1F-43B3-AAE8-5A8442C0B327@longcount.org> X-Mailer: iPhone Mail (9B206) From: Mark Saad Subject: Re: Looking for bge(4) , bce(4) and igb(4) cards Date: Wed, 21 Nov 2012 07:10:04 -0500 To: "andre@freebsd.org" X-Gm-Message-State: ALoCoQmaC6pzInBqgpXXhUlWi52+WeEQu4jsyEec5MOURcHZ5/GGIh9uHM3e2LwUtReBpHm+r2Mb Cc: "freebsd-net@freebsd.org" , "freebsd-current@freebsd.org" , "freebsd-stable@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 12:10:08 -0000 Andre I'll try to do it today or next monday when I get back from vacation . The= y are all hp branded nic's . I ordered them with in the last few years to us= e in place of bce nic's on the main boards of hp servers .=20 --- On Nov 14, 2012, at 5:31 AM, Andre Oppermann wrote: > Hello >=20 > I currently working on a number of drivers for popular network > cards and extend them with automatic hybrid interrupt/polling > ithread processing with life-lock prevention (so that the driver > can't consume all CPU when under heavy load or attack). >=20 > To properly test this I need the proper hardware as PCIe network > cards: >=20 > bge(4) Broadcom BCM57xx/BCM590x > bce(4) Broadcom NetXtreme II (BCM5706/5708/5709/5716) > igb(4) Intel PRO/1000 i82575, i82576, i82580, i210, i350 >=20 > If you have one of these and can spare it I'd be very glad if > you could send it to me. I'm located in Switzerland/Europe. > I can reply to you privately to give you my shipping address. >=20 >=20 > Of course if you have any other PCIe Gigabit Ethernet cards > with a driver in FreeBSD I'm interested in receiving one as > well. Of particular interest are: >=20 > em(4) Intel i82571 to i82573 > lem(4) Intel i82540 to i82546 > age(4) Atheros L1 GigE > ??? anything else 1GigE with PCIe >=20 >=20 > The same goes for 10 Gigabit Ethernet but the setup is a bit > more involved and I haven't done that yet, but will do soon > (the issue being expensive SPF+ optics): >=20 > bxe(4) Broadcom BCM5771x 10GigE > cxbge(4) Chelsio T4 10GigE > ixgbe(4) Intel i82598 and i82599 10GigE > mxge(4) Myricom Myri10G > qlxgb(4) QLogic 3200 and 8200 10GigE > sfxge(4) Solarflare >=20 > Many thanks for your support! >=20 > --=20 > Andre > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 15:48:44 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 44098BCE for ; Wed, 21 Nov 2012 15:48:44 +0000 (UTC) (envelope-from marc@mpeters.org) Received: from mail.mpeters.org (mail.mpeters.org [78.46.104.142]) by mx1.freebsd.org (Postfix) with ESMTP id A6BC58FC14 for ; Wed, 21 Nov 2012 15:48:43 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mail.mpeters.org (Postfix) with ESMTP id 4929113200F for ; Wed, 21 Nov 2012 16:41:45 +0100 (CET) X-Virus-Scanned: amavisd-new at mpeters.org Received: from mail.mpeters.org ([127.0.0.1]) by localhost (mail.mpeters.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RlO6qfZYLOm0 for ; Wed, 21 Nov 2012 16:41:33 +0100 (CET) Received: from [192.168.0.204] (unknown [62.159.86.18]) by mail.mpeters.org (Postfix) with ESMTPSA id F19C913200E for ; Wed, 21 Nov 2012 16:41:32 +0100 (CET) Message-ID: <50ACF62C.8000408@mpeters.org> Date: Wed, 21 Nov 2012 16:41:32 +0100 From: Marc Peters User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:16.0) Gecko/20121104 Thunderbird/16.0.1 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: Low Bandwidth on intercontinental connections X-Enigmail-Version: 1.5a1pre Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 15:48:44 -0000 Hi list, we are experiencing low throughput on interncontinental connections with our FreeBSD Servers. We made several tests and are wondering, why this would be. The first tests were on an IPSEC VPN between our datacenter in DE and Santa Clara, CA. We are connected with two gigabit uplinks in each DC. Pushing data by scp between our FreeBSD servers takes ages. Starting with several MB/s it drops to 60-70KB/s: [root@freebsd ~]# ls -alh test.tgz -rw-r----- 1 root wheel 58M Oct 5 2010 test.tgz [root@freebsd ~]# scp test.tgz 172.16.3.10:. Password: test.tgz 28% 17MB 75.3KB/s 09:32 ETA For comparision, we did a similiar test with Linux, which didn't show this behaviour: root@linux:~# scp jdk-6u33-linux-x64.bin 172.16.4.50: root@172.16.4.50's password: jdk-6u33-linux-x64.bin 100% 69MB 3.4MB/s 00:20 root@linux:~# Otherwise, the servers are really fast, when copying data to a machine nearby: [root@freebsd ~]# ls -alh test -rw-r--r-- 1 root wheel 1G Nov 21 13:43 test [root@freebsd ~]# scp test 172.16.3.11: Password: test 100% 1000MB 38.5MB/s 00:26 Intercontinental ftp downloads are the same: [root@freebsd ~]# fetch ftp://ftp1.us.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-bootonly.iso FreeBSD-9.1-RC3-amd64-bootonly.iso 100% of 146 MB 46 MBps [root@freebsd ~]# fetch ftp://ftp1.us.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso FreeBSD-9.1-RC3-amd64-disc1.iso 100% of 685 MB 36 MBps 00m00s [root@freebsd ~]# fetch ftp://ftp1.de.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso FreeBSD-9.1-RC3-amd64-disc1.iso 0% of 685 MB 13 kBps 14h49m^C Linux: root@linux:~# wget ftp://ftp1.de.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso --2012-11-21 15:07:57-- ftp://ftp1.de.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso => `FreeBSD-9.1-RC3-amd64-disc1.iso' Resolving ftp1.de.freebsd.org... 137.226.34.42 Connecting to ftp1.de.freebsd.org|137.226.34.42|:21... connected. Logging in as anonymous ... Logged in! ==> SYST ... done. ==> PWD ... done. ==> TYPE I ... done. ==> CWD (1) /pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1 ... done. ==> SIZE FreeBSD-9.1-RC3-amd64-disc1.iso ... 718800896 ==> PASV ... done. ==> RETR FreeBSD-9.1-RC3-amd64-disc1.iso ... done. Length: 718800896 (686M) (unauthoritative) 100%[=====================================================================>] 718,800,896 19.1M/s in 61s 2012-11-21 15:09:01 (11.2 MB/s) - `FreeBSD-9.1-RC3-amd64-disc1.iso' saved [718800896] Doing some googling brought up a lot of tuning hints, but nothing worked for us. We tweaked some sysctls: kern.ipc.maxsockbuf=16777216 net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.recvbuf_max=16777216 net.inet.tcp.sendbuf_inc=16384 net.inet.tcp.recvbuf_inc=524288 net.inet.tcp.hostcache.expire=1 but to no good. Disabling MSI and TSO4 for the card didn't change anything, too. The machines are all HP DL360G7 with bce cards (find dmesg, ifconfig and pciconf -lvc at the end of this mail). Can someone hit me with a cluestick to get the BSDs on speed? marc PS: The version is FreeBSD-RC2 amd64, because we need the patch for process migration on the CPUs which didn't make it 9.0 or an errata, as we were the only ones, hitting this bug (so kib@ said). ifconfig: [root@freebsd ~]# ifconfig bce0: flags=8843 metric 0 mtu 1500 options=c01bb ether ac:16:2d:b7:00:f4 inet 172.16.3.10 netmask 0xffffff00 broadcast 172.16.3.255 inet6 fe80::ae16:2dff:feb7:f4%bce0 prefixlen 64 scopeid 0x1 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active bce1: flags=8843 metric 0 mtu 1500 options=c01bb ether ac:16:2d:b7:00:f6 inet 172.17.3.10 netmask 0xfffff800 broadcast 172.17.7.255 inet6 fe80::ae16:2dff:feb7:f6%bce1 prefixlen 64 scopeid 0x2 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active bce2: flags=8802 metric 0 mtu 1500 options=c01bb ether ac:16:2d:b7:00:fc nd6 options=29 media: Ethernet autoselect bce3: flags=8802 metric 0 mtu 1500 options=c01bb ether ac:16:2d:b7:00:fe nd6 options=29 media: Ethernet autoselect lo0: flags=8049 metric 0 mtu 16384 options=600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0xb inet 127.0.0.1 netmask 0xff000000 nd6 options=21 pciconf -lvc: [root@freebsd ~]# pciconf -lvc hostb0@pci0:0:0:0: class=0x060000 card=0x330b103c chip=0x34068086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' device = '5520 I/O Hub to ESI Port' class = bridge subclass = HOST-PCI cap 05[60] = MSI supports 2 messages, vector masks cap 10[90] = PCI-Express 2 root port max data 128(128) link x4(x4) cap 01[e0] = powerspec 3 supports D0 D3 current D0 ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected ecap 000d[150] = unknown 1 ecap 000b[160] = unknown 0 pcib1@pci0:0:1:0: class=0x060400 card=0x330b103c chip=0x34088086 rev=0x13 hdr=0x01 vendor = 'Intel Corporation' device = '5520/5500/X58 I/O Hub PCI Express Root Port 1' class = bridge subclass = PCI-PCI cap 0d[40] = PCI Bridge card=0x330b103c cap 05[60] = MSI supports 2 messages, vector masks cap 10[90] = PCI-Express 2 root port max data 256(256) link x4(x4) cap 01[e0] = powerspec 3 supports D0 D3 current D0 ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected ecap 000d[150] = unknown 1 ecap 000b[160] = unknown 0 pcib2@pci0:0:2:0: class=0x060400 card=0x330b103c chip=0x34098086 rev=0x13 hdr=0x01 vendor = 'Intel Corporation' device = '5520/5500/X58 I/O Hub PCI Express Root Port 2' class = bridge subclass = PCI-PCI cap 0d[40] = PCI Bridge card=0x330b103c cap 05[60] = MSI supports 2 messages, vector masks cap 10[90] = PCI-Express 2 root port max data 256(256) link x0(x4) cap 01[e0] = powerspec 3 supports D0 D3 current D0 ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected ecap 000d[150] = unknown 1 pcib3@pci0:0:3:0: class=0x060400 card=0x330b103c chip=0x340a8086 rev=0x13 hdr=0x01 vendor = 'Intel Corporation' device = '5520/5500/X58 I/O Hub PCI Express Root Port 3' class = bridge subclass = PCI-PCI cap 0d[40] = PCI Bridge card=0x330b103c cap 05[60] = MSI supports 2 messages, vector masks cap 10[90] = PCI-Express 2 root port max data 256(256) link x0(x16) cap 01[e0] = powerspec 3 supports D0 D3 current D0 ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected ecap 000d[150] = unknown 1 ecap 000b[160] = unknown 0 pcib4@pci0:0:4:0: class=0x060400 card=0x330b103c chip=0x340b8086 rev=0x13 hdr=0x01 vendor = 'Intel Corporation' device = '5520/X58 I/O Hub PCI Express Root Port 4' class = bridge subclass = PCI-PCI cap 0d[40] = PCI Bridge card=0x330b103c cap 05[60] = MSI supports 2 messages, vector masks cap 10[90] = PCI-Express 2 root port max data 256(256) link x0(x4) cap 01[e0] = powerspec 3 supports D0 D3 current D0 ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected ecap 000d[150] = unknown 1 pcib5@pci0:0:5:0: class=0x060400 card=0x330b103c chip=0x340c8086 rev=0x13 hdr=0x01 vendor = 'Intel Corporation' device = '5520/X58 I/O Hub PCI Express Root Port 5' class = bridge subclass = PCI-PCI cap 0d[40] = PCI Bridge card=0x330b103c cap 05[60] = MSI supports 2 messages, vector masks cap 10[90] = PCI-Express 2 root port max data 256(256) link x0(x4) cap 01[e0] = powerspec 3 supports D0 D3 current D0 ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected ecap 000d[150] = unknown 1 pcib6@pci0:0:6:0: class=0x060400 card=0x330b103c chip=0x340d8086 rev=0x13 hdr=0x01 vendor = 'Intel Corporation' device = '5520/X58 I/O Hub PCI Express Root Port 6' class = bridge subclass = PCI-PCI cap 0d[40] = PCI Bridge card=0x330b103c cap 05[60] = MSI supports 2 messages, vector masks cap 10[90] = PCI-Express 2 root port max data 256(256) link x0(x4) cap 01[e0] = powerspec 3 supports D0 D3 current D0 ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected ecap 000d[150] = unknown 1 pcib7@pci0:0:7:0: class=0x060400 card=0x330b103c chip=0x340e8086 rev=0x13 hdr=0x01 vendor = 'Intel Corporation' device = '5520/5500/X58 I/O Hub PCI Express Root Port 7' class = bridge subclass = PCI-PCI cap 0d[40] = PCI Bridge card=0x330b103c cap 05[60] = MSI supports 2 messages, vector masks cap 10[90] = PCI-Express 2 root port max data 256(256) link x4(x4) cap 01[e0] = powerspec 3 supports D0 D3 current D0 ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected ecap 000d[150] = unknown 1 ecap 000b[160] = unknown 0 pcib8@pci0:0:8:0: class=0x060400 card=0x330b103c chip=0x340f8086 rev=0x13 hdr=0x01 vendor = 'Intel Corporation' device = '5520/5500/X58 I/O Hub PCI Express Root Port 8' class = bridge subclass = PCI-PCI cap 0d[40] = PCI Bridge card=0x330b103c cap 05[60] = MSI supports 2 messages, vector masks cap 10[90] = PCI-Express 2 root port max data 256(256) link x4(x4) cap 01[e0] = powerspec 3 supports D0 D3 current D0 ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected ecap 000d[150] = unknown 1 pcib9@pci0:0:9:0: class=0x060400 card=0x330b103c chip=0x34108086 rev=0x13 hdr=0x01 vendor = 'Intel Corporation' device = '5520/5500/X58 I/O Hub PCI Express Root Port 9' class = bridge subclass = PCI-PCI cap 0d[40] = PCI Bridge card=0x330b103c cap 05[60] = MSI supports 2 messages, vector masks cap 10[90] = PCI-Express 2 root port max data 256(256) link x0(x8) cap 01[e0] = powerspec 3 supports D0 D3 current D0 ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected ecap 000d[150] = unknown 1 pcib10@pci0:0:10:0: class=0x060400 card=0x330b103c chip=0x34118086 rev=0x13 hdr=0x01 vendor = 'Intel Corporation' device = '5520/5500/X58 I/O Hub PCI Express Root Port 10' class = bridge subclass = PCI-PCI cap 0d[40] = PCI Bridge card=0x330b103c cap 05[60] = MSI supports 2 messages, vector masks cap 10[90] = PCI-Express 2 root port max data 256(256) link x0(x4) cap 01[e0] = powerspec 3 supports D0 D3 current D0 ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected ecap 000d[150] = unknown 1 hostb1@pci0:0:13:0: class=0x060000 card=0x00000000 chip=0x343a8086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' class = bridge subclass = HOST-PCI cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) cap 00[60] = unknown ecap 000b[100] = unknown 0 ecap 000b[800] = unknown 0 hostb2@pci0:0:13:1: class=0x060000 card=0x00000000 chip=0x343b8086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' class = bridge subclass = HOST-PCI cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) cap 00[60] = unknown ecap 000b[100] = unknown 0 ecap 000b[800] = unknown 0 hostb3@pci0:0:13:2: class=0x060000 card=0x00000000 chip=0x343c8086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' class = bridge subclass = HOST-PCI cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) cap 00[60] = unknown hostb4@pci0:0:13:3: class=0x060000 card=0x00000000 chip=0x343d8086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' class = bridge subclass = HOST-PCI cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) cap 00[60] = unknown ecap 000b[100] = unknown 0 hostb5@pci0:0:13:4: class=0x060000 card=0x00000000 chip=0x34188086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' device = '5520/5500/X58 Physical Layer Port 0' class = bridge subclass = HOST-PCI cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) cap 00[60] = unknown hostb6@pci0:0:13:5: class=0x060000 card=0x00000000 chip=0x34198086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' device = '5520/5500 Physical Layer Port 1' class = bridge subclass = HOST-PCI cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) cap 00[60] = unknown hostb7@pci0:0:13:6: class=0x060000 card=0x00000000 chip=0x341a8086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' class = bridge subclass = HOST-PCI cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) cap 00[60] = unknown hostb8@pci0:0:14:0: class=0x060000 card=0x00000000 chip=0x341c8086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' class = bridge subclass = HOST-PCI cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) cap 00[60] = unknown ecap 000b[100] = unknown 0 hostb9@pci0:0:14:1: class=0x060000 card=0x00000000 chip=0x341d8086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' class = bridge subclass = HOST-PCI cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) cap 00[60] = unknown hostb10@pci0:0:14:2: class=0x060000 card=0x00000000 chip=0x341e8086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' class = bridge subclass = HOST-PCI cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) cap 00[60] = unknown hostb11@pci0:0:14:3: class=0x060000 card=0x00000000 chip=0x341f8086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' class = bridge subclass = HOST-PCI cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) cap 00[60] = unknown hostb12@pci0:0:14:4: class=0x060000 card=0x00000000 chip=0x34398086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' class = bridge subclass = HOST-PCI cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) none0@pci0:0:20:0: class=0x080000 card=0x000b003c chip=0x342e8086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' device = '5520/5500/X58 I/O Hub System Management Registers' class = base peripheral subclass = interrupt controller cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) none1@pci0:0:20:1: class=0x080000 card=0x000b003c chip=0x34228086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' device = '5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers' class = base peripheral subclass = interrupt controller cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) none2@pci0:0:20:2: class=0x080000 card=0x000b003c chip=0x34238086 rev=0x13 hdr=0x00 vendor = 'Intel Corporation' device = '5520/5500/X58 I/O Hub Control Status and RAS Registers' class = base peripheral subclass = interrupt controller cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) pcib11@pci0:0:28:0: class=0x060400 card=0x330d103c chip=0x3a408086 rev=0x00 hdr=0x01 vendor = 'Intel Corporation' device = '82801JI (ICH10 Family) PCI Express Root Port 1' class = bridge subclass = PCI-PCI cap 10[40] = PCI-Express 1 root port max data 128(128) link x0(x4) cap 05[80] = MSI supports 1 message cap 0d[90] = PCI Bridge card=0x330d103c cap 01[a0] = powerspec 2 supports D0 D3 current D0 ecap 0002[100] = VC 1 max VC0 ecap 0005[180] = unknown 1 pcib12@pci0:0:28:4: class=0x060400 card=0x330d103c chip=0x3a488086 rev=0x00 hdr=0x01 vendor = 'Intel Corporation' device = '82801JI (ICH10 Family) PCI Express Root Port 5' class = bridge subclass = PCI-PCI cap 10[40] = PCI-Express 1 root port max data 128(128) link x1(x1) cap 05[80] = MSI supports 1 message cap 0d[90] = PCI Bridge card=0x330d103c cap 01[a0] = powerspec 2 supports D0 D3 current D0 ecap 0002[100] = VC 1 max VC0 ecap 0005[180] = unknown 1 uhci1@pci0:0:29:0: class=0x0c0300 card=0x330d103c chip=0x3a348086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82801JI (ICH10 Family) USB UHCI Controller' class = serial bus subclass = USB cap 13[50] = PCI Advanced Features: FLR TP uhci2@pci0:0:29:1: class=0x0c0300 card=0x330d103c chip=0x3a358086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82801JI (ICH10 Family) USB UHCI Controller' class = serial bus subclass = USB cap 13[50] = PCI Advanced Features: FLR TP uhci3@pci0:0:29:2: class=0x0c0300 card=0x330d103c chip=0x3a368086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82801JI (ICH10 Family) USB UHCI Controller' class = serial bus subclass = USB cap 13[50] = PCI Advanced Features: FLR TP uhci4@pci0:0:29:3: class=0x0c0300 card=0x330d103c chip=0x3a398086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82801JI (ICH10 Family) USB UHCI Controller' class = serial bus subclass = USB cap 13[50] = PCI Advanced Features: FLR TP ehci0@pci0:0:29:7: class=0x0c0320 card=0x330d103c chip=0x3a3a8086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82801JI (ICH10 Family) USB2 EHCI Controller' class = serial bus subclass = USB cap 01[50] = powerspec 2 supports D0 D3 current D0 cap 0a[58] = EHCI Debug Port at offset 0xa0 in map 0x14 cap 13[98] = PCI Advanced Features: FLR TP pcib13@pci0:0:30:0: class=0x060401 card=0x330d103c chip=0x244e8086 rev=0x90 hdr=0x01 vendor = 'Intel Corporation' device = '82801 PCI Bridge' class = bridge subclass = PCI-PCI cap 0d[50] = PCI Bridge card=0x330d103c isab0@pci0:0:31:0: class=0x060100 card=0x00000000 chip=0x3a188086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82801JIB (ICH10) LPC Interface Controller' class = bridge subclass = PCI-ISA cap 09[e0] = vendor (length 12) Intel cap 1 version 0 features: Quick Resume, SATA RAID-5, 4 PCI-e x1 slots, SATA RAID-0/1/10 atapci0@pci0:0:31:2: class=0x01018f card=0x330d103c chip=0x3a208086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82801JI (ICH10 Family) 4 port SATA IDE Controller' class = mass storage subclass = ATA cap 01[70] = powerspec 3 supports D0 D3 current D0 cap 13[b0] = PCI Advanced Features: FLR TP ciss0@pci0:5:0:0: class=0x010400 card=0x3245103c chip=0x323a103c rev=0x01 hdr=0x00 vendor = 'Hewlett-Packard Company' device = 'Smart Array G6 controllers' class = mass storage subclass = RAID cap 01[40] = powerspec 3 supports D0 D3 current D0 cap 05[50] = MSI supports 1 message, 64 bit cap 10[70] = PCI-Express 2 endpoint max data 256(256) link x4(x8) cap 11[ac] = MSI-X supports 16 messages in map 0x10 enabled ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected bce0@pci0:3:0:0: class=0x020000 card=0x7055103c chip=0x163914e4 rev=0x20 hdr=0x00 vendor = 'Broadcom Corporation' device = 'NetXtreme II BCM5709 Gigabit Ethernet' class = network subclass = ethernet cap 01[48] = powerspec 3 supports D0 D3 current D0 cap 03[50] = VPD cap 05[58] = MSI supports 16 messages, 64 bit enabled with 1 message cap 11[a0] = MSI-X supports 9 messages in map 0x10 cap 10[ac] = PCI-Express 2 endpoint max data 256(512) link x4(x4) ecap 0003[100] = Serial 1 ac162dfffeb700f4 ecap 0001[110] = AER 1 0 fatal 1 non-fatal 1 corrected ecap 0004[150] = unknown 1 ecap 0002[160] = VC 1 max VC0 bce1@pci0:3:0:1: class=0x020000 card=0x7055103c chip=0x163914e4 rev=0x20 hdr=0x00 vendor = 'Broadcom Corporation' device = 'NetXtreme II BCM5709 Gigabit Ethernet' class = network subclass = ethernet cap 01[48] = powerspec 3 supports D0 D3 current D0 cap 03[50] = VPD cap 05[58] = MSI supports 16 messages, 64 bit enabled with 1 message cap 11[a0] = MSI-X supports 9 messages in map 0x10 cap 10[ac] = PCI-Express 2 endpoint max data 256(512) link x4(x4) ecap 0003[100] = Serial 1 ac162dfffeb700f6 ecap 0001[110] = AER 1 0 fatal 1 non-fatal 1 corrected ecap 0004[150] = unknown 1 ecap 0002[160] = VC 1 max VC0 bce2@pci0:4:0:0: class=0x020000 card=0x7055103c chip=0x163914e4 rev=0x20 hdr=0x00 vendor = 'Broadcom Corporation' device = 'NetXtreme II BCM5709 Gigabit Ethernet' class = network subclass = ethernet cap 01[48] = powerspec 3 supports D0 D3 current D0 cap 03[50] = VPD cap 05[58] = MSI supports 16 messages, 64 bit enabled with 1 message cap 11[a0] = MSI-X supports 9 messages in map 0x10 cap 10[ac] = PCI-Express 2 endpoint max data 256(512) link x4(x4) ecap 0003[100] = Serial 1 ac162dfffeb700fc ecap 0001[110] = AER 1 0 fatal 1 non-fatal 1 corrected ecap 0004[150] = unknown 1 ecap 0002[160] = VC 1 max VC0 bce3@pci0:4:0:1: class=0x020000 card=0x7055103c chip=0x163914e4 rev=0x20 hdr=0x00 vendor = 'Broadcom Corporation' device = 'NetXtreme II BCM5709 Gigabit Ethernet' class = network subclass = ethernet cap 01[48] = powerspec 3 supports D0 D3 current D0 cap 03[50] = VPD cap 05[58] = MSI supports 16 messages, 64 bit enabled with 1 message cap 11[a0] = MSI-X supports 9 messages in map 0x10 cap 10[ac] = PCI-Express 2 endpoint max data 256(512) link x4(x4) ecap 0003[100] = Serial 1 ac162dfffeb700fe ecap 0001[110] = AER 1 0 fatal 1 non-fatal 1 corrected ecap 0004[150] = unknown 1 ecap 0002[160] = VC 1 max VC0 none3@pci0:2:0:0: class=0x088000 card=0x3309103c chip=0x3306103c rev=0x04 hdr=0x00 vendor = 'Hewlett-Packard Company' device = 'Integrated Lights-Out Standard Slave Instrumentation & System Support' class = base peripheral cap 01[78] = powerspec 3 supports D0 D3 current D0 cap 05[b0] = MSI supports 1 message, 64 bit cap 10[c0] = PCI-Express 1 legacy endpoint max data 128(128) link x1(x1) none4@pci0:2:0:2: class=0x088000 card=0x3309103c chip=0x3307103c rev=0x04 hdr=0x00 vendor = 'Hewlett-Packard Company' device = 'Integrated Lights-Out Standard Management Processor Support and Messaging' class = base peripheral cap 01[78] = powerspec 3 supports D0 D3 current D0 cap 05[b0] = MSI supports 1 message, 64 bit cap 10[c0] = PCI-Express 1 legacy endpoint max data 128(128) link x1(x1) uhci0@pci0:2:0:4: class=0x0c0300 card=0x3309103c chip=0x3300103c rev=0x01 hdr=0x00 vendor = 'Hewlett-Packard Company' device = 'Integrated Lights-Out Standard Virtual USB Controller' class = serial bus subclass = USB cap 05[70] = MSI supports 1 message, 64 bit cap 10[80] = PCI-Express 1 legacy endpoint max data 128(128) link x1(x1) cap 01[f0] = powerspec 3 supports D0 D3 current D0 vgapci0@pci0:1:3:0: class=0x030000 card=0x31fb103c chip=0x515e1002 rev=0x02 hdr=0x00 vendor = 'ATI Technologies Inc' device = 'ES1000' class = display subclass = VGA cap 01[50] = powerspec 2 supports D0 D1 D2 D3 current D0 dmesg: [root@freebsd ~]# dmesg Copyright (c) 1992-2012 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.1-RC2 #0 r241106: Mon Oct 1 18:26:44 UTC 2012 root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 CPU: Intel(R) Xeon(R) CPU X5660 @ 2.80GHz (2800.16-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x206c2 Family = 6 Model = 2c Stepping = 2 Features=0xbfebfbff Features2=0x29ee3ff AMD Features=0x2c100800 AMD Features2=0x1 TSC: P-state invariant, performance statistics real memory = 34359738368 (32768 MB) avail memory = 33095315456 (31562 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 24 CPUs FreeBSD/SMP: 2 package(s) x 6 core(s) x 2 SMT threads cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 16 cpu7 (AP): APIC ID: 17 cpu8 (AP): APIC ID: 18 cpu9 (AP): APIC ID: 19 cpu10 (AP): APIC ID: 20 cpu11 (AP): APIC ID: 21 cpu12 (AP): APIC ID: 32 cpu13 (AP): APIC ID: 33 cpu14 (AP): APIC ID: 34 cpu15 (AP): APIC ID: 35 cpu16 (AP): APIC ID: 36 cpu17 (AP): APIC ID: 37 cpu18 (AP): APIC ID: 48 cpu19 (AP): APIC ID: 49 cpu20 (AP): APIC ID: 50 cpu21 (AP): APIC ID: 51 cpu22 (AP): APIC ID: 52 cpu23 (AP): APIC ID: 53 ACPI Warning: Invalid length for Pm1aControlBlock: 32, using default 16 (20110527/tbfadt-638) ACPI Warning: Invalid length for Pm2ControlBlock: 32, using default 8 (20110527/tbfadt-638) ioapic1 irqs 24-47 on motherboard ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 ctl: CAM Target Layer loaded acpi0: on motherboard acpi0: Power Button (fixed) cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cpu4: on acpi0 cpu5: on acpi0 cpu6: on acpi0 cpu7: on acpi0 cpu8: on acpi0 cpu9: on acpi0 cpu10: on acpi0 cpu11: on acpi0 cpu12: on acpi0 cpu13: on acpi0 cpu14: on acpi0 cpu15: on acpi0 cpu16: on acpi0 cpu17: on acpi0 cpu18: on acpi0 cpu19: on acpi0 cpu20: on acpi0 cpu21: on acpi0 cpu22: on acpi0 cpu23: on acpi0 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 450 Event timer "HPET1" frequency 14318180 Hz quality 440 Event timer "HPET2" frequency 14318180 Hz quality 440 Event timer "HPET3" frequency 14318180 Hz quality 440 atrtc0: port 0x70-0x71 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x908-0x90b on acpi0 pcib0: on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci5: on pcib1 ciss0: port 0x4000-0x40ff mem 0xfbe00000-0xfbffffff,0xfbdf0000-0xfbdf0fff irq 28 at device 0.0 on pci5 ciss0: PERFORMANT Transport pcib2: at device 2.0 on pci0 pci12: on pcib2 pcib3: at device 3.0 on pci0 pci9: on pcib3 pcib4: at device 4.0 on pci0 pci13: on pcib4 pcib5: at device 5.0 on pci0 pci14: on pcib5 pcib6: at device 6.0 on pci0 pci15: on pcib6 pcib7: at device 7.0 on pci0 pci3: on pcib7 bce0: mem 0xf4000000-0xf5ffffff irq 30 at device 0.0 on pci3 miibus0: on bce0 brgphy0: PHY 1 on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bce0: Ethernet address: ac:16:2d:b7:00:f4 bce0: ASIC (0x57092003); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.12) Coal (RX:6,6,18,18; TX:20,20,80,80) bce1: mem 0xf2000000-0xf3ffffff irq 37 at device 0.1 on pci3 miibus1: on bce1 brgphy1: PHY 1 on miibus1 brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bce1: Ethernet address: ac:16:2d:b7:00:f6 bce1: ASIC (0x57092003); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.12) Coal (RX:6,6,18,18; TX:20,20,80,80) pcib8: at device 8.0 on pci0 pci4: on pcib8 bce2: mem 0xf8000000-0xf9ffffff irq 31 at device 0.0 on pci4 miibus2: on bce2 brgphy2: PHY 1 on miibus2 brgphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bce2: Ethernet address: ac:16:2d:b7:00:fc bce2: ASIC (0x57092003); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.12) Coal (RX:6,6,18,18; TX:20,20,80,80) bce3: mem 0xf6000000-0xf7ffffff irq 39 at device 0.1 on pci4 miibus3: on bce3 brgphy3: PHY 1 on miibus3 brgphy3: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bce3: Ethernet address: ac:16:2d:b7:00:fe bce3: ASIC (0x57092003); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.12) Coal (RX:6,6,18,18; TX:20,20,80,80) pcib9: at device 9.0 on pci0 pci6: on pcib9 pcib10: at device 10.0 on pci0 pci16: on pcib10 pci0: at device 20.0 (no driver attached) pci0: at device 20.1 (no driver attached) pci0: at device 20.2 (no driver attached) pcib11: at device 28.0 on pci0 pci17: on pcib11 pcib12: at device 28.4 on pci0 pci2: on pcib12 pci2: at device 0.0 (no driver attached) pci2: at device 0.2 (no driver attached) uhci0: port 0x3c00-0x3c1f irq 17 at device 0.4 on pci2 usbus0 on uhci0 uhci1: port 0x1000-0x101f irq 20 at device 29.0 on pci0 usbus1 on uhci1 uhci2: port 0x1020-0x103f irq 23 at device 29.1 on pci0 usbus2 on uhci2 uhci3: port 0x1040-0x105f irq 22 at device 29.2 on pci0 usbus3 on uhci3 uhci4: port 0x1060-0x107f irq 23 at device 29.3 on pci0 usbus4 on uhci4 ehci0: mem 0xf1bf0000-0xf1bf03ff irq 20 at device 29.7 on pci0 usbus5: EHCI version 1.0 usbus5 on ehci0 pcib13: at device 30.0 on pci0 pci1: on pcib13 vgapci0: port 0x2000-0x20ff mem 0xe8000000-0xefffffff,0xf1cf0000-0xf1cfffff irq 23 at device 3.0 on pci1 isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1080-0x1087,0x1088-0x108b,0x1090-0x1097,0x1098-0x109b,0x10a0-0x10af,0x10b0-0x10bf irq 17 at device 31.2 on pci0 ata2: at channel 0 on atapci0 ata3: at channel 1 on atapci0 acpi_tz0: on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] uart0: port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 qpi0: on motherboard orm0: at iomem 0xc0000-0xcafff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ppc0: cannot reserve I/O port range uart1: at port 0x2f8-0x2ff irq 3 on isa0 est0: on cpu0 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est0 attach returned 6 p4tcc0: on cpu0 est1: on cpu1 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est1 attach returned 6 p4tcc1: on cpu1 est2: on cpu2 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est2 attach returned 6 p4tcc2: on cpu2 est3: on cpu3 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est3 attach returned 6 p4tcc3: on cpu3 est4: on cpu4 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est4 attach returned 6 p4tcc4: on cpu4 est5: on cpu5 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est5 attach returned 6 p4tcc5: on cpu5 est6: on cpu6 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est6 attach returned 6 p4tcc6: on cpu6 est7: on cpu7 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est7 attach returned 6 p4tcc7: on cpu7 est8: on cpu8 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est8 attach returned 6 p4tcc8: on cpu8 est9: on cpu9 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est9 attach returned 6 p4tcc9: on cpu9 est10: on cpu10 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est10 attach returned 6 p4tcc10: on cpu10 est11: on cpu11 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est11 attach returned 6 p4tcc11: on cpu11 est12: on cpu12 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est12 attach returned 6 p4tcc12: on cpu12 est13: on cpu13 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est13 attach returned 6 p4tcc13: on cpu13 est14: on cpu14 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est14 attach returned 6 p4tcc14: on cpu14 est15: on cpu15 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est15 attach returned 6 p4tcc15: on cpu15 est16: on cpu16 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est16 attach returned 6 p4tcc16: on cpu16 est17: on cpu17 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est17 attach returned 6 p4tcc17: on cpu17 est18: on cpu18 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est18 attach returned 6 p4tcc18: on cpu18 est19: on cpu19 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est19 attach returned 6 p4tcc19: on cpu19 est20: on cpu20 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est20 attach returned 6 p4tcc20: on cpu20 est21: on cpu21 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est21 attach returned 6 p4tcc21: on cpu21 est22: on cpu22 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est22 attach returned 6 p4tcc22: on cpu22 est23: on cpu23 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 16 device_attach: est23 attach returned 6 p4tcc23: on cpu23 ZFS filesystem version 5 ZFS storage pool version 28 Timecounters tick every 1.000 msec usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 12Mbps Full Speed USB v1.0 usbus4: 12Mbps Full Speed USB v1.0 usbus5: 480Mbps High Speed USB v2.0 ugen0.1: <0x103c> at usbus0 uhub0: <0x103c UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 ugen4.1: at usbus4 uhub4: on usbus4 ugen5.1: at usbus5 uhub5: on usbus5 uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered ugen0.2: at usbus0 ukbd0: on usbus0 kbd2 at ukbd0 ums0: on usbus0 (probe0:ciss0:0:0:0): REPORT LUNS. CDB: a0 0 0 0 0 0 0 0 0 10 0 0 (probe0:ciss0:0:0:0): CAM status: SCSI Status Error (probe0:ciss0:0:0:0): SCSI status: Check Condition (probe0:ciss0:0:0:0): SCSI sense: ILLEGAL REQUEST asc:20,0 (Invalid command operation code) (probe0:ciss0:0:0:0): Error 22, Unretryable error da0 at ciss0 bus 0 scbus0 target 0 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 135.168MB/s transfers da0: Command Queueing enabled da0: 286070MB (585871964 512 byte sectors: 255H 32S/T 65535C) SMP: AP CPU #1 Launched! cd0 at ata2 bus 0 scbus2 target 0 lun 0 cd0: Removable CD-ROM SCSI-0 device SMP: AP CPU #12 Launched! cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed SMP: AP CPU #11 Launched! SMP: AP CPU #19 Launched! SMP: AP CPU #10 Launched! SMP: AP CPU #22 Launched! SMP: AP CPU #8 Launched! SMP: AP CPU #16 Launched! SMP: AP CPU #9 Launched! SMP: AP CPU #23 Launched! SMP: AP CPU #21 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #14 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #17 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #20 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #13 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #18 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #15 Launched! Timecounter "TSC-low" frequency 10938126 Hz quality 1000 uhub5: 8 ports with 8 removable, self powered Trying to mount root from ufs:/dev/da0p2 [rw]... bce0: link state changed to UP bce0: Gigabit link up! bce0: Gigabit link up! bce0: Gigabit link up! bce1: link state changed to UP bce1: Gigabit link up! bce1: Gigabit link up! bce1: Gigabit link up! From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 16:47:25 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6B6E7C1B for ; Wed, 21 Nov 2012 16:47:25 +0000 (UTC) (envelope-from m.e.sanliturk@gmail.com) Received: from mail-vb0-f54.google.com (mail-vb0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id 195D28FC08 for ; Wed, 21 Nov 2012 16:47:24 +0000 (UTC) Received: by mail-vb0-f54.google.com with SMTP id l1so9765502vba.13 for ; Wed, 21 Nov 2012 08:47:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=yNX+W1U9inAzhWzMBu4kdNZ6kdpPBK37o5Sv40Js588=; b=civrOjajbP8/PBag5lmjWC1qLmlEUH+NYVqO+d2Xs1/R90JLm6MVWOcO/nc0oUjJW1 mOGFPBU/scEcUbgDjh823b/YZcGSChbDN4D1T1yXmzGDKvlHPg98ZOkFZGofaGORspuZ bmIeXubJ3kcXinfxTwAzx61hhHGAYEHcHtXvy27at0pWoe68msXrOOdwVvtpAZfYJroL 5SXr3X1bJ3zWwu9xiscJ8OSgSLhUJBNiZ0rB6fSI/fN49CUxWkA3W8FO/S20mjxzjDU1 x71yVHhyprdWgY3dFnwwiS/n4F+eqWaMVWDQ468INWXB0JkFVRjoTFYfOWfQAjDKrpdq Gk/w== MIME-Version: 1.0 Received: by 10.58.161.113 with SMTP id xr17mr27988330veb.3.1353516444049; Wed, 21 Nov 2012 08:47:24 -0800 (PST) Received: by 10.58.218.35 with HTTP; Wed, 21 Nov 2012 08:47:23 -0800 (PST) In-Reply-To: <50ACF62C.8000408@mpeters.org> References: <50ACF62C.8000408@mpeters.org> Date: Wed, 21 Nov 2012 08:47:23 -0800 Message-ID: Subject: Re: Low Bandwidth on intercontinental connections From: Mehmet Erol Sanliturk To: Marc Peters Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 16:47:25 -0000 On Wed, Nov 21, 2012 at 7:41 AM, Marc Peters wrote: > Hi list, > > we are experiencing low throughput on interncontinental connections with > our FreeBSD Servers. We made several tests and are wondering, why this > would be. The first tests were on an IPSEC VPN between our datacenter in > DE and Santa Clara, CA. We are connected with two gigabit uplinks in > each DC. Pushing data by scp between our FreeBSD servers takes ages. > Starting with several MB/s it drops to 60-70KB/s: > ..... I do not have any answer to your question , but I want to share one my experiences . I Linux ( KDE ) I was copying a hard disk contents to another drive by using Dolphin . At the beginning it was very fast , but over time its speed reduced to a few kilobytes per second . It listed completion time left as months . I inspected why this is the case . The reason was the following : On each file it is copied , the Dolphin was producing approximately 1 Kilobyte memory leak . After copying more than one million file , all of the memory exhausted and it started to swap memory to hard disk swap space which reduced copy speed to a few kilobytes per second . I stopped the Dolphin and copied small directory groups by restarting the Dolphin . This cured the problem because on each exit , all of the leaked memory by Dolphin has been disposed ( where "Undo" item of Dolphin menu was disabled means memory is not reserved for undo ). Please study your data transfer software for such a possibility . It may not be problematic in Linux but FreeBSD version may have some trouble points . There is another possibility : Graceful degradation . http://en.wikipedia.org/wiki/Graceful_degradation http://en.wikipedia.org/wiki/Fail_soft A program part may produce graceful degradation over time or processed data : For example , assume a list is searched by sequentially . When list length grows , search times also grows linearly and produces a degradation although there is no any error in the process . You may study your system with respect to such a process . These are the possibilities which come to my mind . Thank you very much . Mehmet Erol Sanliturk From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 16:59:43 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8C7DF383 for ; Wed, 21 Nov 2012 16:59:43 +0000 (UTC) (envelope-from benjamin.villain@ucopia.fr) Received: from mail-bk0-f54.google.com (mail-bk0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id 115CA8FC16 for ; Wed, 21 Nov 2012 16:59:42 +0000 (UTC) Received: by mail-bk0-f54.google.com with SMTP id je9so2503938bkc.13 for ; Wed, 21 Nov 2012 08:59:41 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=references:message-id:x-mailer:from:to:cc:subject:date:mime-version :content-type:content-disposition:content-transfer-encoding :x-gm-message-state; bh=xqVfiIG9LfKkgA2OjaY7G5KEuBVT91mTG96GEs4+T3E=; b=CoPGDIbUWRS+PioEopNkbF2a5p/cvgs902HpgPxzdzy4FEHi2otrSSeydhgr6xALG2 JQzrXn/+JJQCmF4Z1Z6vpB8jrAu4pGNMvktPcAUC89vD8E4rFacqhxPUjKskeFctab9G DUivMcf9uzaqniM3Z/O84ifNH1xqH+FnomAWYSDOtub0uSCBil53aF7bGRTT9gbjQvEt GBeyAN54RhQ8qEpgyU46FzFf2D775owY4Ceek2R+SmzptA+VRXnPS2L45XaqbAxuMyet jhGARQpz23zp8dr2HDHUGgCEN+FG8mEFHU5QENNlP3qSVrTXvSdCH6bpTx+tMfn5Q6dv Cr5g== Received: by 10.204.7.78 with SMTP id c14mr2773637bkc.100.1353517181740; Wed, 21 Nov 2012 08:59:41 -0800 (PST) Received: from ([195.68.4.238]) by mx.google.com with ESMTPS id f24sm710821bkv.7.2012.11.21.08.59.40 (version=SSLv3 cipher=AES128-SHA); Wed, 21 Nov 2012 08:59:41 -0800 (PST) References: <50ACF62C.8000408@mpeters.org> Message-ID: <50ad087d.1892cc0a.2cce.3bf2@mx.google.com> X-Mailer: http://www.courier-mta.org/cone/ From: Benjamin Villain To: Mehmet Erol Sanliturk Subject: Re: Low Bandwidth on intercontinental connections Date: Wed, 21 Nov 2012 17:58:57 +0100 Mime-Version: 1.0 Content-Type: text/plain; format=flowed; delsp=yes; charset="iso-8859-1" Content-Disposition: inline Content-Transfer-Encoding: 7bit X-Gm-Message-State: ALoCoQlnet3Y2h8LPCTkqIRkE70cJPZtZovmV/hM+sKlTf0lJWKFxnRWz50y+UPilgqlYei6nj9U Cc: freebsd-net@freebsd.org, Marc Peters X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 16:59:43 -0000 I don't think this is about disk or memory leak as transfering files locally seem to work fine. Can you test transferring files from (and to) your Linux boxes to (and from) the FreeBSD servers to check that it is not a network issue inside your DCs. King regards, -- Ben Mehmet Erol Sanliturk writes: > On Wed, Nov 21, 2012 at 7:41 AM, Marc Peters wrote: > > > Hi list, > > > > we are experiencing low throughput on interncontinental connections with > > our FreeBSD Servers. We made several tests and are wondering, why this > > would be. The first tests were on an IPSEC VPN between our datacenter in > > DE and Santa Clara, CA. We are connected with two gigabit uplinks in > > each DC. Pushing data by scp between our FreeBSD servers takes ages. > > Starting with several MB/s it drops to 60-70KB/s: > > > > > > ..... > > > I do not have any answer to your question , but I want to share one my > experiences . > > I Linux ( KDE ) I was copying a hard disk contents to another drive by > using Dolphin . > At the beginning it was very fast , but over time its speed reduced to a > few kilobytes per second . > It listed completion time left as months . > > I inspected why this is the case . > > The reason was the following : > > On each file it is copied , the Dolphin was producing approximately 1 > Kilobyte memory leak . > After copying more than one million file , all of the memory exhausted and > it started to swap > memory to hard disk swap space which reduced copy speed to a few kilobytes > per second . > > > I stopped the Dolphin and copied small directory groups by restarting the > Dolphin . This cured the problem because on each exit , all of the leaked > memory by Dolphin has been disposed ( where "Undo" item of Dolphin menu was > disabled means memory is not reserved for undo ). > > > Please study your data transfer software for such a possibility . It may > not be problematic in Linux but FreeBSD version may have some trouble > points . > > > There is another possibility : Graceful degradation . > > http://en.wikipedia.org/wiki/Graceful_degradation > http://en.wikipedia.org/wiki/Fail_soft > > A program part may produce graceful degradation over time or processed data > : > > For example , assume a list is searched by sequentially . When list length > grows , search times > also grows linearly and produces a degradation although there is no any > error in the process . > > You may study your system with respect to such a process . > > > These are the possibilities which come to my mind . > > > Thank you very much . > > Mehmet Erol Sanliturk > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 17:03:38 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 02C5F626; Wed, 21 Nov 2012 17:03:38 +0000 (UTC) (envelope-from fodillemlinkarim@gmail.com) Received: from mail-ia0-f182.google.com (mail-ia0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 99EB08FC17; Wed, 21 Nov 2012 17:03:37 +0000 (UTC) Received: by mail-ia0-f182.google.com with SMTP id x2so6556193iad.13 for ; Wed, 21 Nov 2012 09:03:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=04uYjvrz9/krDuT3ZrC4PCTIR7L0onMIDQxpK8PAMuY=; b=qS+d6MHPE/NzV/xVhIDSPgzRnG4Dj45wi9oXNmqDA5RuJ64HORtOcRzZRGRWlAXCJj MXR1BWGUbGKgE+m7eq0sjAgAt2KBe38jStldF3WloLXbGDP1nulUA+ejz5su+4gdq3lI MMT0yeG8yvY+xYjAvko+84ZPH3C7eC3R1BY2xcEvc2LSGO3X//rIYrB3x3PPjEDKj06r d85O3cRj06po2QaZ7yj1GEZfmCBUG9MihYkuvOqOx3K2EwJgGmvyrJG+qC2zm+GNi4XS 2Fdd7d7EH3/W6XkMBCTSKEgcPcojCfnr9QOK9UyOrtAj+0PtC5y1QHu1qtpNqNMK0wZM wZow== Received: by 10.50.57.225 with SMTP id l1mr163909igq.37.1353517411578; Wed, 21 Nov 2012 09:03:31 -0800 (PST) Received: from [192.168.1.73] ([208.85.112.101]) by mx.google.com with ESMTPS id gk1sm76770igc.5.2012.11.21.09.03.29 (version=SSLv3 cipher=OTHER); Wed, 21 Nov 2012 09:03:30 -0800 (PST) Message-ID: <50AD0955.1040600@gmail.com> Date: Wed, 21 Nov 2012 12:03:17 -0500 From: Karim Fodil-Lemelin User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Gleb Smirnoff Subject: Re: igb diver crashes in head@241037 References: <50AA8F24.7080604@gmail.com> <20121120111833.GC67660@FreeBSD.org> <20121121062631.GJ67660@glebius.int.ru> In-Reply-To: <20121121062631.GJ67660@glebius.int.ru> Content-Type: text/plain; charset=KOI8-R; format=flowed Content-Transfer-Encoding: 7bit Cc: jfv@FreeBSD.org, freebsd-net@FreeBSD.org, Jack Vogel X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 17:03:38 -0000 Hi Gleb, On 21/11/2012 1:26 AM, Gleb Smirnoff wrote: > Jack, > > On Tue, Nov 20, 2012 at 09:19:54AM -0800, Jack Vogel wrote: > J> > I'd suggest the following code: > J> > > J> > if (m) > J> > drbr_enqueue(ifp, txr->br, m); > J> > err = igb_mq_start_locked(ifp, txr, NULL); > J> > > J> > Which eventually leads us to all invocations of igb_mq_start_locked() > J> > called > J> > with third argument as NULL. This allows us to simplify this function. > J> > > J> > Patch for review attached. > J> > > J> > > J> Yes Gleb, I already have code in my internal tree which simply removes an > J> mbuf > J> pointer form the start_locked call and ALWAYS does a dequeue, start > J> similarly > J> will always enqueue. I just have been busy with ixgbe for a bit and have > J> not gotten > J> it committed yet. > > Since ixgbe work is performance tuning and this patch closes a kernel crash, > I'd ask to preempt the ixgbe job with this patch. :) > > Or you can approve my patch and I will check it in. > What about protecting the em driver from the same out of order problem that was fixed in igb? Wouldn't make sense to commit a fix for both drivers? Thanks, Karim. From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 17:20:55 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2C017FA6 for ; Wed, 21 Nov 2012 17:20:55 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mail-ea0-f182.google.com (mail-ea0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id 9CE7D8FC17 for ; Wed, 21 Nov 2012 17:20:54 +0000 (UTC) Received: by mail-ea0-f182.google.com with SMTP id a14so1693001eaa.13 for ; Wed, 21 Nov 2012 09:20:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=vBzcyll4caP3d4PYBlJAHg3tJyn8YPjh7vl44CnHwKI=; b=Dn2tLBAk+ddapEkK5fXmCvZtW5lMymNkonZWMCLgkjg/Sc8EBbYUMgUSnDM3GnZFY4 zb+e+zdmvIN7E5UlJuDpPjZX6tr5ozq0f8FGhQf/bqwpzwiG3A8zzsqsML52eaTiEFKM 3EqPzrKlUd5TvA1zp5mHSj0RwB4a5vIXGgF2n7GJ/ceBdbQVQFonmC2885OpvvOvK+Y1 m+0hBov7CtX0THr9OdoPPnrDmwlJKE5sOQXHNAiJQ1TcCISBcYnHNj7b/HiCWUYPixjD /fWatiyAt/7v+Ldgo8rGVBf+eHHC1RWz1kmqhkBLP/EuXEso6fdVmaiS0Nyd7xmaRCXo 5W+Q== MIME-Version: 1.0 Received: by 10.14.175.198 with SMTP id z46mr47503956eel.26.1353518453598; Wed, 21 Nov 2012 09:20:53 -0800 (PST) Received: by 10.223.71.199 with HTTP; Wed, 21 Nov 2012 09:20:53 -0800 (PST) In-Reply-To: <50ad087d.1892cc0a.2cce.3bf2@mx.google.com> References: <50ACF62C.8000408@mpeters.org> <50ad087d.1892cc0a.2cce.3bf2@mx.google.com> Date: Wed, 21 Nov 2012 09:20:53 -0800 Message-ID: Subject: Re: Low Bandwidth on intercontinental connections From: Kevin Oberman To: Benjamin Villain Content-Type: text/plain; charset=UTF-8 Cc: freebsd-net@freebsd.org, Marc Peters X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 17:20:55 -0000 On Wed, Nov 21, 2012 at 8:58 AM, Benjamin Villain wrote: > I don't think this is about disk or memory leak as transfering files locally > seem to work fine. > > Can you test transferring files from (and to) your Linux boxes to (and from) > the FreeBSD servers to check that it is not a network issue inside your DCs. > > King regards, > > -- > Ben > > > Mehmet Erol Sanliturk writes: > >> On Wed, Nov 21, 2012 at 7:41 AM, Marc Peters wrote: >> >> > Hi list, >> > >> > we are experiencing low throughput on interncontinental connections with >> > our FreeBSD Servers. We made several tests and are wondering, why this >> > would be. The first tests were on an IPSEC VPN between our datacenter in >> > DE and Santa Clara, CA. We are connected with two gigabit uplinks in >> > each DC. Pushing data by scp between our FreeBSD servers takes ages. >> > Starting with several MB/s it drops to 60-70KB/s: >> > >> >> >> >> ..... >> >> >> I do not have any answer to your question , but I want to share one my >> experiences . >> >> I Linux ( KDE ) I was copying a hard disk contents to another drive by >> using Dolphin . >> At the beginning it was very fast , but over time its speed reduced to a >> few kilobytes per second . >> It listed completion time left as months . >> >> I inspected why this is the case . >> >> The reason was the following : >> >> On each file it is copied , the Dolphin was producing approximately 1 >> Kilobyte memory leak . >> After copying more than one million file , all of the memory exhausted and >> it started to swap >> memory to hard disk swap space which reduced copy speed to a few kilobytes >> per second . >> >> >> I stopped the Dolphin and copied small directory groups by restarting the >> Dolphin . This cured the problem because on each exit , all of the leaked >> memory by Dolphin has been disposed ( where "Undo" item of Dolphin menu >> was >> disabled means memory is not reserved for undo ). >> >> >> Please study your data transfer software for such a possibility . It may >> not be problematic in Linux but FreeBSD version may have some trouble >> points . >> >> >> There is another possibility : Graceful degradation . >> >> http://en.wikipedia.org/wiki/Graceful_degradation >> http://en.wikipedia.org/wiki/Fail_soft >> >> A program part may produce graceful degradation over time or processed >> data >> : >> >> For example , assume a list is searched by sequentially . When list length >> grows , search times >> also grows linearly and produces a degradation although there is no any >> error in the process . >> >> You may study your system with respect to such a process . >> >> >> These are the possibilities which come to my mind . If you have not done so, I suggest you use SIFTR to capture data on what is happening in TCP. It can often tell you a great deal and is very easy to work with. Just load the kernel module and use sysctls to control it. I have used it in conjunction with tcpdump and wireshark to find performance problems. Also, for high performance on bulk data transfers over long, fat pipes, take a look at http://fasterdata.es.net. It is a detailed guide on moving data developed by the people who have to deal with the huge volumes of Large Hadron Collider data moving across the Atlantic from CERN to researchers in the US. (Note that this is not FreeBSD specific.) -- R. Kevin Oberman, Network Engineer E-mail: kob6558@gmail.com From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 17:32:06 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 655DF5F0 for ; Wed, 21 Nov 2012 17:32:06 +0000 (UTC) (envelope-from marc@mpeters.org) Received: from mail.mpeters.org (mail.mpeters.org [78.46.104.142]) by mx1.freebsd.org (Postfix) with ESMTP id F39648FC12 for ; Wed, 21 Nov 2012 17:32:05 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mail.mpeters.org (Postfix) with ESMTP id D1AC213200F; Wed, 21 Nov 2012 18:32:04 +0100 (CET) X-Virus-Scanned: amavisd-new at mpeters.org Received: from mail.mpeters.org ([127.0.0.1]) by localhost (mail.mpeters.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jtrZpvUxrXoI; Wed, 21 Nov 2012 18:32:02 +0100 (CET) Received: from [192.168.0.204] (unknown [62.159.86.18]) by mail.mpeters.org (Postfix) with ESMTPSA id A648B13200E; Wed, 21 Nov 2012 18:32:02 +0100 (CET) Message-ID: <50AD1012.7020209@mpeters.org> Date: Wed, 21 Nov 2012 18:32:02 +0100 From: Marc Peters User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:16.0) Gecko/20121104 Thunderbird/16.0.1 MIME-Version: 1.0 To: Benjamin Villain Subject: Re: Low Bandwidth on intercontinental connections References: <50ACF62C.8000408@mpeters.org> <50ad087d.1892cc0a.2cce.3bf2@mx.google.com> In-Reply-To: <50ad087d.1892cc0a.2cce.3bf2@mx.google.com> X-Enigmail-Version: 1.5a1pre Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 17:32:06 -0000 On 11/21/2012 05:58 PM, Benjamin Villain wrote: > I don't think this is about disk or memory leak as transfering files > locally seem to work fine. > > Can you test transferring files from (and to) your Linux boxes to (and > from) the FreeBSD servers to check that it is not a network issue inside > your DCs. > > King regards, > > -- > Ben Hi Ben, i don't think this is memory related, too. We used plain CLI scp ot ftp from base, both times. Here is the requested data: Linux ---> FreeBSD: root@linux:~# scp jdk-6u33-linux-x64.bin 172.16.3.10: Password: jdk-6u33-linux-x64.bin 89% 61MB 59.0KB/s FreeBSD ---> Linux: [root@freebsd ~]# scp test.tgz 172.16.4.50: Password: test.tgz 100% 59MB 1.1MB/s 00:55 [root@freebsd ~]# >From BSD to Linux is not as fast as L <--> L. I don't think, this is network related in some sort. Marc > > Mehmet Erol Sanliturk writes: > >> On Wed, Nov 21, 2012 at 7:41 AM, Marc Peters wrote: >> >> > Hi list, >> > >> > we are experiencing low throughput on interncontinental connections >> with >> > our FreeBSD Servers. We made several tests and are wondering, why this >> > would be. The first tests were on an IPSEC VPN between our >> datacenter in >> > DE and Santa Clara, CA. We are connected with two gigabit uplinks in >> > each DC. Pushing data by scp between our FreeBSD servers takes ages. >> > Starting with several MB/s it drops to 60-70KB/s: >> > >> >> >> >> ..... >> >> >> I do not have any answer to your question , but I want to share one my >> experiences . >> >> I Linux ( KDE ) I was copying a hard disk contents to another drive by >> using Dolphin . >> At the beginning it was very fast , but over time its speed reduced to a >> few kilobytes per second . >> It listed completion time left as months . >> >> I inspected why this is the case . >> >> The reason was the following : >> >> On each file it is copied , the Dolphin was producing approximately 1 >> Kilobyte memory leak . >> After copying more than one million file , all of the memory exhausted >> and >> it started to swap >> memory to hard disk swap space which reduced copy speed to a few >> kilobytes >> per second . >> >> >> I stopped the Dolphin and copied small directory groups by restarting the >> Dolphin . This cured the problem because on each exit , all of the leaked >> memory by Dolphin has been disposed ( where "Undo" item of Dolphin >> menu was >> disabled means memory is not reserved for undo ). >> >> >> Please study your data transfer software for such a possibility . It may >> not be problematic in Linux but FreeBSD version may have some trouble >> points . >> >> >> There is another possibility : Graceful degradation . >> >> http://en.wikipedia.org/wiki/Graceful_degradation >> http://en.wikipedia.org/wiki/Fail_soft >> >> A program part may produce graceful degradation over time or processed >> data >> : >> >> For example , assume a list is searched by sequentially . When list >> length >> grows , search times >> also grows linearly and produces a degradation although there is no any >> error in the process . >> >> You may study your system with respect to such a process . >> >> >> These are the possibilities which come to my mind . >> >> >> Thank you very much . >> >> Mehmet Erol Sanliturk >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 17:32:44 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C9E5668F for ; Wed, 21 Nov 2012 17:32:44 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm16-vm1.bullet.mail.ne1.yahoo.com (nm16-vm1.bullet.mail.ne1.yahoo.com [98.138.91.47]) by mx1.freebsd.org (Postfix) with ESMTP id 76B708FC1D for ; Wed, 21 Nov 2012 17:32:44 +0000 (UTC) Received: from [98.138.90.51] by nm16.bullet.mail.ne1.yahoo.com with NNFMP; 21 Nov 2012 17:32:37 -0000 Received: from [98.138.89.244] by tm4.bullet.mail.ne1.yahoo.com with NNFMP; 21 Nov 2012 17:32:37 -0000 Received: from [127.0.0.1] by omp1058.mail.ne1.yahoo.com with NNFMP; 21 Nov 2012 17:32:37 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 670976.77800.bm@omp1058.mail.ne1.yahoo.com Received: (qmail 84192 invoked by uid 60001); 21 Nov 2012 17:32:37 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1353519157; bh=GJ05ix2XNfPpal05Aq29cVwSlO6n7csQDUEOSmwxHMc=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=06KDAD4ZDexfLQN6bhA8ens66/gwgzvmz3vn9qeHF6k7R90Zt6uhN2Li1VtLt6WtXVTIWV7cs5V3RxuJ+0YgTQ8tLhrIAO434r8IxZ9ElnHBMODRcAhBrMeMkiwm1H8g7X0XGYeVZPM3rGMVr3iHPTTqEcMl4Xf9ZHbbdrc14vo= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=hP8x0iGmniNHWr02a+ZfRiZgdbtzWGTX8fPPXdx0l6nWL+FHFzGhE5cAEoZMISrpqijWxc1RHDhoRlrzxl81yilom6k58zJ/oFiJ6wdflvtv1mqbPNyBbLdhzFWBnJT/1aDyUsXjgzD3dpDcel+G+RK/1lsLEOCx9pJ/ZX/Ff6I=; X-YMail-OSG: 1j.5n5UVM1lskrkSclSNxZwz3fNXnQZGC0zRl5UXkFm8NtI xynSDKzL0MIzNQ3Vt5yD29DHE.goe5t5NCGNhepr0SR6YYAxAhXd7xZC8ThO 5SuW2tUad_NfV5Yn1_lCLjez79U2YNUf6IMchxi6ABDQJce9tTqfbkWF.5Bb L5isJxaO2pq2VFTyMRDWwjnuPdFQGhrFv5X8UQH_VcEpmEtLQUPsutLRI6FM 3qtqu90vdS7fh7kQaa_tvD4pGdP96vbosNl4l7f1GN1uv.F6pJGRDmXHn6YG flphEP73fJmxjomGUO3ijsAN8lYKCS7O849Ya7KE08Lwy4bVXvdcH6Pq1WDU YelktISsctdjXr1pHUop6ywsr71zqWUCUBnmMnzc39PV_C7d6entD1AX0K3x OR28Vy_tY4eJZFHu5ccCR9VNKVNimheZ3HqXEB3.R.NQZ76TpHX_3U_1K7yj yn5q1vMbU0a81ORiqJJiO4Xzs7BkPuSM0.PgUXAQYkVUDs2DrsENZuk_jZn8 E1B0muHb0.Dyl6p6A4RrirZrw7qdZNyV4czupTZV9pmeIhvDIMv_M45dTInE UfN1SSCtr5aMiLPEdhmH8NIJ.bE6rbddRpTHrud9ClLLm4MNeOwVlWREqao0 pHssx Received: from [174.48.128.27] by web121605.mail.ne1.yahoo.com via HTTP; Wed, 21 Nov 2012 09:32:37 PST X-Rocket-MIMEInfo: 001.001, CgotLS0gT24gV2VkLCAxMS8yMS8xMiwgSm9obiBGcmV0YnkgPGpmcmV0YnlAZ29vZ2xlbWFpbC5jb20.IHdyb3RlOgoKPiBGcm9tOiBKb2huIEZyZXRieSA8amZyZXRieUBnb29nbGVtYWlsLmNvbT4KPiBTdWJqZWN0OiBSZTogRnJlZUJTRCBib3hlcyBhcyBhICdyb3V0ZXInLi4uCj4gVG86ICJWaWN0b3IgQmFsYWRhIERpYXoiIDx2aWN0b3JAYnNkZXMubmV0Pgo.IENjOiBmcmVlYnNkLWlzcEBmcmVlYnNkLm9yZwo.IERhdGU6IFdlZG5lc2RheSwgTm92ZW1iZXIgMjEsIDIwMTIsIDExOjQwIEFNCj4gT24gMjEBMAEBAQE- X-Mailer: YahooMailClassic/15.0.8 YahooMailWebService/0.8.123.460 Message-ID: <1353519157.76595.YahooMailClassic@web121605.mail.ne1.yahoo.com> Date: Wed, 21 Nov 2012 09:32:37 -0800 (PST) From: Barney Cordoba Subject: Re: FreeBSD boxes as a 'router'... To: John Fretby In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 17:32:44 -0000 --- On Wed, 11/21/12, John Fretby wrote: > From: John Fretby > Subject: Re: FreeBSD boxes as a 'router'... > To: "Victor Balada Diaz" > Cc: freebsd-isp@freebsd.org > Date: Wednesday, November 21, 2012, 11:40 AM > On 21 November 2012 14:57, Victor > Balada Diaz > wrote: > > > > I think you forgot to CC the list. I'll add it so you > can get > > more answers. > > > > I did forget, thanks for that! :) > > > > em(4) and igb(4) are both drivers for Intel NICs. They > just have > > different capabilities. The sysctl you're asking for > controls behavior > > of adaptive interrupt moderation. It's a recommended > tuning for end hosts > > more than routers. You can read more about interrupt > moderation on this > > document: > > > > http://www.intel.com/design/network/applnots/ap450.htm > > > > em(4) NICs don't have all the capabilities of igb(4) > ones. Some em(4) NICs > > have > > interrupt moderation (eg: 82574L) but not all of them > do. If your em(4) > > card does > > have interrupt moderation you can tune it with: > > > > hw.em.rx_int_delay > > hw.em.rx_abs_int_delay > > hw.em.tx_int_delay > > hw.em.tx_abs_int_delay > > > > Exchanging latency to get more throughput. > > > > You can take a look at this document explaining > capabilities of different > > NICs: > > > > > > http://www.intel.com/content/dam/doc/brochure/ethernet-controllers-phys-brochure.pdf > > > > You should ask supermicro what's the exact model > they'll put on your server > > and then decide if it's OK for you. > > > They are apparently: > > em0: port > 0xf020-0xf03f mem > 0xdfa00000-0xdfa1ffff,0xdfa25000-0xdfa25fff irq 20 at device > 25.0 on pci0 > em0: Using an MSI interrupt > ... > em0: flags=8c02 > metric 0 mtu 1500 > options=4219b > > > > About the interrupt storm: We've had various interrupt > storms that were > > caused by > > different problems. The most common was a software bug > with interrupts. > > After > > reporting on the lists it was fixed and we didn't have > problems again. > > > > If you have a problem with high interrupts because too > many small packets > > (eg a DoS), > > getting a card with interrupt moderation should help a > lot. Most probably > > your problem > > with interrupt storms was caused by something else like > a shared interrupt > > with other > > device or software bug. Without more analysis it's > impossible to really > > say. > > > > I have some details from when it happened - it doesn't look > like it was a > shared interrupt issue - it just literally looks like the > host came up, > with a stampeding hurd of "other" hosts hitting it for > services that > weren't yet running, and it folded :( > > That's why I was wondering if there was a similar sysctl for > the em driver > - in order to raise the number of interrupts the system > allows, before > declaring it "a storm". > > > > > > Keep in mind that i'm not an expert on this area, so > you might get better > > answers > > on frebsd-net@ :) > > > > Hope it helps. > > > > It has - half the problem is there are *so* many options, > combinations - > and no matter what you pick, if you look them up enough > you'll find someone > finding fault with them, or casting doubts on their > performance. > > Doesn't really help when all you want is something that has > a good chance > of "working" :) > The road to mediocrity is listening to people who are not experts. The admission that you dont understand something is a good disclosure. But the argument than understanding how to tune a system is too complicated only means that I don't want to listen to what you have to say. Polling implies that there are unnatural intervals between processing packets. So you're introducing delay into your network. The more often you poll, the less the gap. Suppose you poll every 1ms. Each received packet will have from 0-1ms of delay. Packets that are arriving as you ended you last poll will be delayed 1ms. The more knobs you have, the more you can determine what the system can do. An interrupt for every packet would have the least delay, but that's not practical for systems managing 100s of 1000s of pps. You make trade offs between delays and cpu usage. On a small network with a fast cpu, you can process every packet. On a very large network you cant. The idea that there is one way to tune a system for every environment is simply wrong. A bridge/filter has completely different requirements than a "router". And a "server" which only uses 1 NIC has different requirements than a system that forwards traffic and has to manage more locks at the hardware level. BC From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 17:43:38 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E5D3A8E0 for ; Wed, 21 Nov 2012 17:43:37 +0000 (UTC) (envelope-from m.e.sanliturk@gmail.com) Received: from mail-vb0-f54.google.com (mail-vb0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id 9242F8FC16 for ; Wed, 21 Nov 2012 17:43:37 +0000 (UTC) Received: by mail-vb0-f54.google.com with SMTP id l1so9853537vba.13 for ; Wed, 21 Nov 2012 09:43:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=r62+ltcWOEgFCF4x/Mz6GL6YxOTC4IoDqyqNlfUakdI=; b=bkMEYhRPL5yTn6ZdR1x3canVIXxRwEwS0aTUCMQaKfNCEKWkQV1r8du6kFwM6v2lIp 8eTHZ0AnJPtUcZrbjhu+qUtCGWuCj9kBUXuuY+ST9I7n9gSMsRIRWGxggQKVITNp38fo NEgxOXZblTfAg3I51DyoxbxACYmrSlTNZNTc8AU+jKRv1edoX+7FFSMASBg7ina2nsl5 9ebCCNvcqTJGMoyIqYU6y+3eUq3JjgqLxJnjnzPvgj6KuDAx5kZ3BwlmUaqfzirvFPWB rgKuTiWlPKKM6ZdFu5T13DKrIxhkSB7CSxTxg4uwQve9N7PyO0WVjeoGfdTgVpwz53h3 PeWA== MIME-Version: 1.0 Received: by 10.52.90.212 with SMTP id by20mr26658330vdb.118.1353519817052; Wed, 21 Nov 2012 09:43:37 -0800 (PST) Received: by 10.58.218.35 with HTTP; Wed, 21 Nov 2012 09:43:36 -0800 (PST) In-Reply-To: References: <50ACF62C.8000408@mpeters.org> <50ad087d.1892cc0a.2cce.3bf2@mx.google.com> Date: Wed, 21 Nov 2012 09:43:36 -0800 Message-ID: Subject: Re: Low Bandwidth on intercontinental connections From: Mehmet Erol Sanliturk To: Kevin Oberman Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-net@freebsd.org, Benjamin Villain , Marc Peters X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 17:43:38 -0000 On Wed, Nov 21, 2012 at 9:20 AM, Kevin Oberman wrote: > On Wed, Nov 21, 2012 at 8:58 AM, Benjamin Villain > wrote: > > I don't think this is about disk or memory leak as transfering files > locally > > seem to work fine. > > > > Can you test transferring files from (and to) your Linux boxes to (and > from) > > the FreeBSD servers to check that it is not a network issue inside your > DCs. > > > > King regards, > > > > -- > > Ben > > > > > > Mehmet Erol Sanliturk writes: > > > >> On Wed, Nov 21, 2012 at 7:41 AM, Marc Peters wrote: > >> > >> > Hi list, > >> > > >> > we are experiencing low throughput on interncontinental connections > with > >> > our FreeBSD Servers. We made several tests and are wondering, why this > >> > would be. The first tests were on an IPSEC VPN between our datacenter > in > >> > DE and Santa Clara, CA. We are connected with two gigabit uplinks in > >> > each DC. Pushing data by scp between our FreeBSD servers takes ages. > >> > Starting with several MB/s it drops to 60-70KB/s: > >> > > >> > >> > >> > >> ..... > >> > >> > >> I do not have any answer to your question , but I want to share one my > >> experiences . > >> > >> I Linux ( KDE ) I was copying a hard disk contents to another drive by > >> using Dolphin . > >> At the beginning it was very fast , but over time its speed reduced to a > >> few kilobytes per second . > >> It listed completion time left as months . > >> > >> I inspected why this is the case . > >> > >> The reason was the following : > >> > >> On each file it is copied , the Dolphin was producing approximately 1 > >> Kilobyte memory leak . > >> After copying more than one million file , all of the memory exhausted > and > >> it started to swap > >> memory to hard disk swap space which reduced copy speed to a few > kilobytes > >> per second . > >> > >> > >> I stopped the Dolphin and copied small directory groups by restarting > the > >> Dolphin . This cured the problem because on each exit , all of the > leaked > >> memory by Dolphin has been disposed ( where "Undo" item of Dolphin menu > >> was > >> disabled means memory is not reserved for undo ). > >> > >> > >> Please study your data transfer software for such a possibility . It may > >> not be problematic in Linux but FreeBSD version may have some trouble > >> points . > >> > >> > >> There is another possibility : Graceful degradation . > >> > >> http://en.wikipedia.org/wiki/Graceful_degradation > >> http://en.wikipedia.org/wiki/Fail_soft > >> > >> A program part may produce graceful degradation over time or processed > >> data > >> : > >> > >> For example , assume a list is searched by sequentially . When list > length > >> grows , search times > >> also grows linearly and produces a degradation although there is no any > >> error in the process . > >> > >> You may study your system with respect to such a process . > >> > >> > >> These are the possibilities which come to my mind . > > If you have not done so, I suggest you use SIFTR to capture data on > what is happening in TCP. It can often tell you a great deal and is > very easy to work with. Just load the kernel module and use sysctls to > control it. I have used it in conjunction with tcpdump and wireshark > to find performance problems. > > Also, for high performance on bulk data transfers over long, fat > pipes, take a look at http://fasterdata.es.net. It is a detailed guide > on moving data developed by the people who have to deal with the huge > volumes of Large Hadron Collider data moving across the Atlantic from > CERN to researchers in the US. (Note that this is not FreeBSD > specific.) > -- > R. Kevin Oberman, Network Engineer > E-mail: kob6558@gmail.com > A very good link . In the above site , please see the following especially : http://fasterdata.es.net/data-transfer-tools/say-no-to-scp/ Say No to scp Why you should avoid scp over a WAN and http://fasterdata.es.net/data-transfer-tools/scp-and-sftp/ scp and sftp Thank you very much . Mehmet Erol Sanliturk From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 17:52:56 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 708FBCCD for ; Wed, 21 Nov 2012 17:52:56 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id B1B748FC1A for ; Wed, 21 Nov 2012 17:52:54 +0000 (UTC) Received: (qmail 11889 invoked from network); 21 Nov 2012 18:52:52 +0100 Received: from fw.xip.at (HELO ?127.0.0.1?) (89.207.145.147) by chile.gbit.at with SMTP; 21 Nov 2012 18:52:52 +0100 Message-ID: <50AD14F8.8050001@xip.at> Date: Wed, 21 Nov 2012 18:52:56 +0100 From: Ingo Flaschberger User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: Re: Low Bandwidth on intercontinental connections References: <50ACF62C.8000408@mpeters.org> <50ad087d.1892cc0a.2cce.3bf2@mx.google.com> <50AD1012.7020209@mpeters.org> In-Reply-To: <50AD1012.7020209@mpeters.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 17:52:56 -0000 Am 21.11.2012 18:32, schrieb Marc Peters: > Hi Ben, > > i don't think this is memory related, too. We used plain CLI scp ot ftp > from base, both times. > > Here is the requested data: > > Linux ---> FreeBSD: > > root@linux:~# scp jdk-6u33-linux-x64.bin 172.16.3.10: > Password: > jdk-6u33-linux-x64.bin 89% 61MB 59.0KB/s > > FreeBSD ---> Linux: > > [root@freebsd ~]# scp test.tgz 172.16.4.50: > Password: > test.tgz 100% 59MB 1.1MB/s 00:55 > [root@freebsd ~]# > > From BSD to Linux is not as fast as L <--> L. > > I don't think, this is network related in some sort. hm - sounds like a duplex problem? *) whats the distance between Linux and FreeBSD box? *) check network counters: linux: ifconfig FreeBSD: netstat -nia look for errors check switches between (or as far as possible) for full duplex, also FreeBSD (ifconfig) *) check and compare tcpdump Kind regards, Ingo Flaschberger From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 18:00:28 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BAEC71BA for ; Wed, 21 Nov 2012 18:00:28 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from mail-oa0-f54.google.com (mail-oa0-f54.google.com [209.85.219.54]) by mx1.freebsd.org (Postfix) with ESMTP id 7DAB48FC29 for ; Wed, 21 Nov 2012 18:00:28 +0000 (UTC) Received: by mail-oa0-f54.google.com with SMTP id n9so9408533oag.13 for ; Wed, 21 Nov 2012 10:00:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=JH8rWIJBwJwJvufJpMb9UpEd+/crINTkElDI62aOu3w=; b=iyRcSdnrav2k6TwiU0O6qL5568xE/TLLSQAj8V45KrQ+I5SjCBj4SebLVrpmuYCKoD wwpTxbZy9FN9eDyi16dNEajPjX+rBkkyHw+GkVj2usxxwdAmOnqR12s5SoNhBoaT4Z/l a8v8Rfl8OmiwOPh+Txw71NbABAAzxwox11Kiicgf9yN/OfrtkxWPWXO3AX0umsKoMbUL 8ftO9Dsl0nUBz9/VD+mgfGn5fmA0IOUdWqDTI24EWPuUUw+6efRGtr7/OwbsYI1VXXdT yHDan4QvRCwpZ8704UGewx2QKLlVXk48ntOXwttlWfFN6LjP1tfpLCUUdQnqUo3JXIeF utVA== MIME-Version: 1.0 Received: by 10.60.172.178 with SMTP id bd18mr8384541oec.133.1353520827771; Wed, 21 Nov 2012 10:00:27 -0800 (PST) Received: by 10.76.143.33 with HTTP; Wed, 21 Nov 2012 10:00:27 -0800 (PST) Date: Wed, 21 Nov 2012 10:00:27 -0800 Message-ID: Subject: net.inet6.icmp6.nd6_useloopback - what is it supposed to do? From: Garrett Cooper To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 18:00:28 -0000 Hi, I've been TAHI testing FreeBSD 7.x sources for the past couple months and over the course of my testing via the TAHI IPv6 conformance test, I changed the knob value from net.inet6.icmp6.nd6_useloopback=1 -> net.inet6.icmp6.nd6_useloopback=0 and ran into a slew of errors with the addr.p2 phase-1 TAHI tests. I was wondering if someone could describe what the beforementioned sysctl is supposed to do from a functional perspective, how it might tie into other IPv6 RFCs (if applicable), and if disabled how it would affect a system with IPv6 enabled. Thanks, -Garrett From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 18:26:03 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id DC45BF28; Wed, 21 Nov 2012 18:26:03 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wg0-f50.google.com (mail-wg0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 312338FC1C; Wed, 21 Nov 2012 18:26:02 +0000 (UTC) Received: by mail-wg0-f50.google.com with SMTP id 12so3408259wgr.31 for ; Wed, 21 Nov 2012 10:26:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=AMDGOmDO8D78FRZkd9szZv2BScP13higpgP/6Mg99zc=; b=fbrwBpw4jmpwVwkDyOP5d7Dm2s4dAsM6oR25tdh18/8lRGePtarpMKFfb3RU4MGWC/ GEr9VT/GtKso+sSV537m2QnkbMmviFjjwlv+NyWAVTHjzqo5eNEDalVaCt6j+damkFnn aIw62ohEMW2UddAZv0hYw63qFmkZ06+hjsZ4qVjueiJWQd9Nq3+bN0gMqBV9HcqNaxAn 26mvnf2lHvEkVOiYPhwcFaBzaDOQnNukIf1dA/U3ng4/NIRoYptncYFuZvgyyLGLp5nn zu+l85/WIBSE/y0BgRztIfBCJRNWqxqa3ol0d5ZqlVLSlxeAwief438n8jkpozBH6/ql F2EQ== MIME-Version: 1.0 Received: by 10.216.74.85 with SMTP id w63mr8125587wed.212.1353522361826; Wed, 21 Nov 2012 10:26:01 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.216.21.211 with HTTP; Wed, 21 Nov 2012 10:26:01 -0800 (PST) In-Reply-To: <50AC910C.4030004@freebsd.org> References: <1353448328.76219.YahooMailClassic@web121602.mail.ne1.yahoo.com> <50AC08EC.8070107@mu.org> <832757660.33924.1353460119408@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> <250266404.35502.1353464214924@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> <50AC8393.3060001@freebsd.org> <50AC910C.4030004@freebsd.org> Date: Wed, 21 Nov 2012 10:26:01 -0800 X-Google-Sender-Auth: M8peE6LifayZc0H9a60xj8NqK44 Message-ID: Subject: Re: FreeBSD boxes as a 'router'... From: Adrian Chadd To: Andre Oppermann Content-Type: text/plain; charset=ISO-8859-1 Cc: Barney Cordoba , Jim Thompson , Alfred Perlstein , khatfield@socllc.net, "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 18:26:04 -0000 On 21 November 2012 00:30, Andre Oppermann wrote: > On 21.11.2012 08:55, Adrian Chadd wrote: >> >> Something that has popped up a few times, even recently, is breaking >> out of an RX loop after you service a number of frames. > > That is what I basically described. Right, and this can be done right now without too much reworking, right? I mean, people could begin by doing a drive-by on drivers for this. The RX path for a driver shouldn't be too difficult to do; the TX path is the racy one. >> During stupidly high levels of RX, you may find the NIC happily >> receiving frames faster than you can service the RX queue. If this >> occurs, you could end up just plain being stuck there. > That's the live-lock. And again you can solve this without having to devolve into polling. Again, polling to me feels like a bludgeon beating around a system that isn't really designed for the extreme cases it's facing. Maybe your work in the tcp_taskqueue branch addresses the larger scale issues here, but I've solved this relatively easily in the past. >> So what I've done in the past is to loop over a certain number of >> frames, then schedule a taskqueue to service whatever's left over. > Taskqueue's shouldn't be used anymore. We've got ithreads now. > Contrary to popular belief (and due to poor documentation) an > ithread does not run at interrupt level. Only the fast interrupt > handler does that. The ithread is a normal kernel thread tied to > an fast interrupt handler and trailing it whenever it said > INTR_SCHEDULE_ITHREAD. Sure, but taskqueues are still useful if you want to serialise access without relying on mutexes wrapping large parts of the packet handling code to enforce said order. Yes, normal ithreads don't run at interrupt level. And we can change the priority of taskqueues in each driver, right? And/or we could change the behaviour of driver ithreads/taskqueues to be automatically reniced? I'm not knocking your work here, I'm just trying to understand whether we can do this stuff as small individual pieces of work rather than one big subsystem overhaul. And CoDel is interesting as a concept, but it's certainly not new. But again, if you don't drop the frames during the driver receive path (and try to do it higher up in the stack, eg as part of some firewall rule) you still risk reaching a stable state where the CPU is 100% pinned because you've wasted cycles pushing those frames into the queue only to be dropped. What _I_ had to do there was have a quick gate to look up if a frame was part of an active session in ipfw and if it was, let it be queued to the driver. I also had a second gate in the driver for new TCP connections, but that was a separate hack. Anything else was dropped. In any case, what I'm trying to say is this - when I was last doing this kind of stuff, I didn't just subscribe to "polling will fix all." I spent a few months knee deep in the public intel e1000 documentation and tuning guide, the em driver and the queue/firewall code, in order to figure out how to attack this without using polling. And yes, you've also just described NAPI. :-) Adrian From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 18:29:34 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 06E5227A for ; Wed, 21 Nov 2012 18:29:34 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wi0-f178.google.com (mail-wi0-f178.google.com [209.85.212.178]) by mx1.freebsd.org (Postfix) with ESMTP id 52F518FC13 for ; Wed, 21 Nov 2012 18:29:32 +0000 (UTC) Received: by mail-wi0-f178.google.com with SMTP id hm6so85984wib.13 for ; Wed, 21 Nov 2012 10:29:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=FacxZCg5JdI+aC3/rdWrjeHLvupFgCfhfpYY5DyCKOs=; b=rxl6VA2pX9WMDgsT9lnYKVdN4+2MZni8BqDOoIjSmWt0g2E7GQJdfUeuQPt4h9i53+ hWgqhZKui/n4RtJGpRveMp/QGBbynY+kUHWGT2sFsRAOO6bVgaAUW0tAdg2jCE2K/HmY pk/TDtj6Gk0Zs68lAB1RfqdKQ+UxkU7lQTHJqc7tbyyOhpdpcjvDGP50bqfIeazezwrS AGyfCdfxmQXqz12VWNba5TbdzUQWQ1pSS7q1gBvD6cg5efZzRKZGy8z1x24jF9loI8Hi QZLcJb1uazsjoYSulfSKMT6+N/IIqJc6bgnF5WFOxVOeIYxPSGeZhFFbEqrRJ9rOg8lZ eFMQ== MIME-Version: 1.0 Received: by 10.180.7.197 with SMTP id l5mr721051wia.13.1353522571856; Wed, 21 Nov 2012 10:29:31 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.216.21.211 with HTTP; Wed, 21 Nov 2012 10:29:31 -0800 (PST) In-Reply-To: <50ACF62C.8000408@mpeters.org> References: <50ACF62C.8000408@mpeters.org> Date: Wed, 21 Nov 2012 10:29:31 -0800 X-Google-Sender-Auth: w-Fk3q2UsDhmJNBnGZ96SBRf3JY Message-ID: Subject: Re: Low Bandwidth on intercontinental connections From: Adrian Chadd To: Marc Peters Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 18:29:34 -0000 Hi! Firstly - please file a PR. Secondly - there's some great tcp counters in 'netstat -sp tcp' (and ip, and udp, and icmp.) You can zero them; netstat -sp tcp -z. I suggest dumping them before/after and compare the values. Adrian On 21 November 2012 07:41, Marc Peters wrote: > Hi list, > > we are experiencing low throughput on interncontinental connections with > our FreeBSD Servers. We made several tests and are wondering, why this > would be. The first tests were on an IPSEC VPN between our datacenter in > DE and Santa Clara, CA. We are connected with two gigabit uplinks in > each DC. Pushing data by scp between our FreeBSD servers takes ages. > Starting with several MB/s it drops to 60-70KB/s: > > [root@freebsd ~]# ls -alh test.tgz > -rw-r----- 1 root wheel 58M Oct 5 2010 test.tgz > [root@freebsd ~]# scp test.tgz 172.16.3.10:. > Password: > test.tgz 28% 17MB 75.3KB/s 09:32 ETA > > > For comparision, we did a similiar test with Linux, which didn't show > this behaviour: > > root@linux:~# scp jdk-6u33-linux-x64.bin 172.16.4.50: > root@172.16.4.50's password: > jdk-6u33-linux-x64.bin 100% > 69MB 3.4MB/s 00:20 > root@linux:~# > > > Otherwise, the servers are really fast, when copying data to a machine > nearby: > > [root@freebsd ~]# ls -alh test > -rw-r--r-- 1 root wheel 1G Nov 21 13:43 test > [root@freebsd ~]# scp test 172.16.3.11: > Password: > test 100% 1000MB 38.5MB/s 00:26 > > > Intercontinental ftp downloads are the same: > > [root@freebsd ~]# fetch > ftp://ftp1.us.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-bootonly.iso > FreeBSD-9.1-RC3-amd64-bootonly.iso 100% of 146 MB 46 MBps > > [root@freebsd ~]# fetch > ftp://ftp1.us.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso > > FreeBSD-9.1-RC3-amd64-disc1.iso 100% of 685 MB 36 MBps 00m00s > > [root@freebsd ~]# fetch > ftp://ftp1.de.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso > FreeBSD-9.1-RC3-amd64-disc1.iso 0% of 685 MB 13 kBps 14h49m^C > > > Linux: > > root@linux:~# wget > ftp://ftp1.de.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso > --2012-11-21 15:07:57-- > ftp://ftp1.de.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso > => `FreeBSD-9.1-RC3-amd64-disc1.iso' > Resolving ftp1.de.freebsd.org... 137.226.34.42 > Connecting to ftp1.de.freebsd.org|137.226.34.42|:21... connected. > Logging in as anonymous ... Logged in! > ==> SYST ... done. ==> PWD ... done. > ==> TYPE I ... done. ==> CWD (1) > /pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1 ... done. > ==> SIZE FreeBSD-9.1-RC3-amd64-disc1.iso ... 718800896 > ==> PASV ... done. ==> RETR FreeBSD-9.1-RC3-amd64-disc1.iso ... done. > Length: 718800896 (686M) (unauthoritative) > > 100%[=====================================================================>] > 718,800,896 19.1M/s in 61s > > 2012-11-21 15:09:01 (11.2 MB/s) - `FreeBSD-9.1-RC3-amd64-disc1.iso' > saved [718800896] > > > Doing some googling brought up a lot of tuning hints, but nothing worked > for us. We tweaked some sysctls: > > kern.ipc.maxsockbuf=16777216 > net.inet.tcp.sendbuf_max=16777216 > net.inet.tcp.recvbuf_max=16777216 > net.inet.tcp.sendbuf_inc=16384 > net.inet.tcp.recvbuf_inc=524288 > net.inet.tcp.hostcache.expire=1 > > but to no good. Disabling MSI and TSO4 for the card didn't change > anything, too. > > The machines are all HP DL360G7 with bce cards (find dmesg, ifconfig and > pciconf -lvc at the end of this mail). > > Can someone hit me with a cluestick to get the BSDs on speed? > > marc > > PS: The version is FreeBSD-RC2 amd64, because we need the patch for > process migration on the CPUs which didn't make it 9.0 or an errata, as > we were the only ones, hitting this bug (so kib@ said). > > ifconfig: > [root@freebsd ~]# ifconfig > bce0: flags=8843 metric 0 mtu 1500 > options=c01bb > ether ac:16:2d:b7:00:f4 > inet 172.16.3.10 netmask 0xffffff00 broadcast 172.16.3.255 > inet6 fe80::ae16:2dff:feb7:f4%bce0 prefixlen 64 scopeid 0x1 > nd6 options=29 > media: Ethernet autoselect (1000baseT ) > status: active > bce1: flags=8843 metric 0 mtu 1500 > options=c01bb > ether ac:16:2d:b7:00:f6 > inet 172.17.3.10 netmask 0xfffff800 broadcast 172.17.7.255 > inet6 fe80::ae16:2dff:feb7:f6%bce1 prefixlen 64 scopeid 0x2 > nd6 options=29 > media: Ethernet autoselect (1000baseT ) > status: active > bce2: flags=8802 metric 0 mtu 1500 > options=c01bb > ether ac:16:2d:b7:00:fc > nd6 options=29 > media: Ethernet autoselect > bce3: flags=8802 metric 0 mtu 1500 > options=c01bb > ether ac:16:2d:b7:00:fe > nd6 options=29 > media: Ethernet autoselect > lo0: flags=8049 metric 0 mtu 16384 > options=600003 > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0xb > inet 127.0.0.1 netmask 0xff000000 > nd6 options=21 > > pciconf -lvc: > [root@freebsd ~]# pciconf -lvc > hostb0@pci0:0:0:0: class=0x060000 card=0x330b103c chip=0x34068086 > rev=0x13 hdr=0x00 > vendor = 'Intel Corporation' > device = '5520 I/O Hub to ESI Port' > class = bridge > subclass = HOST-PCI > cap 05[60] = MSI supports 2 messages, vector masks > cap 10[90] = PCI-Express 2 root port max data 128(128) link x4(x4) > cap 01[e0] = powerspec 3 supports D0 D3 current D0 > ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected > ecap 000d[150] = unknown 1 > ecap 000b[160] = unknown 0 > pcib1@pci0:0:1:0: class=0x060400 card=0x330b103c chip=0x34088086 > rev=0x13 hdr=0x01 > vendor = 'Intel Corporation' > device = '5520/5500/X58 I/O Hub PCI Express Root Port 1' > class = bridge > subclass = PCI-PCI > cap 0d[40] = PCI Bridge card=0x330b103c > cap 05[60] = MSI supports 2 messages, vector masks > cap 10[90] = PCI-Express 2 root port max data 256(256) link x4(x4) > cap 01[e0] = powerspec 3 supports D0 D3 current D0 > ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected > ecap 000d[150] = unknown 1 > ecap 000b[160] = unknown 0 > pcib2@pci0:0:2:0: class=0x060400 card=0x330b103c chip=0x34098086 > rev=0x13 hdr=0x01 > vendor = 'Intel Corporation' > device = '5520/5500/X58 I/O Hub PCI Express Root Port 2' > class = bridge > subclass = PCI-PCI > cap 0d[40] = PCI Bridge card=0x330b103c > cap 05[60] = MSI supports 2 messages, vector masks > cap 10[90] = PCI-Express 2 root port max data 256(256) link x0(x4) > cap 01[e0] = powerspec 3 supports D0 D3 current D0 > ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected > ecap 000d[150] = unknown 1 > pcib3@pci0:0:3:0: class=0x060400 card=0x330b103c chip=0x340a8086 > rev=0x13 hdr=0x01 > vendor = 'Intel Corporation' > device = '5520/5500/X58 I/O Hub PCI Express Root Port 3' > class = bridge > subclass = PCI-PCI > cap 0d[40] = PCI Bridge card=0x330b103c > cap 05[60] = MSI supports 2 messages, vector masks > cap 10[90] = PCI-Express 2 root port max data 256(256) link x0(x16) > cap 01[e0] = powerspec 3 supports D0 D3 current D0 > ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected > ecap 000d[150] = unknown 1 > ecap 000b[160] = unknown 0 > pcib4@pci0:0:4:0: class=0x060400 card=0x330b103c chip=0x340b8086 > rev=0x13 hdr=0x01 > vendor = 'Intel Corporation' > device = '5520/X58 I/O Hub PCI Express Root Port 4' > class = bridge > subclass = PCI-PCI > cap 0d[40] = PCI Bridge card=0x330b103c > cap 05[60] = MSI supports 2 messages, vector masks > cap 10[90] = PCI-Express 2 root port max data 256(256) link x0(x4) > cap 01[e0] = powerspec 3 supports D0 D3 current D0 > ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected > ecap 000d[150] = unknown 1 > pcib5@pci0:0:5:0: class=0x060400 card=0x330b103c chip=0x340c8086 > rev=0x13 hdr=0x01 > vendor = 'Intel Corporation' > device = '5520/X58 I/O Hub PCI Express Root Port 5' > class = bridge > subclass = PCI-PCI > cap 0d[40] = PCI Bridge card=0x330b103c > cap 05[60] = MSI supports 2 messages, vector masks > cap 10[90] = PCI-Express 2 root port max data 256(256) link x0(x4) > cap 01[e0] = powerspec 3 supports D0 D3 current D0 > ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected > ecap 000d[150] = unknown 1 > pcib6@pci0:0:6:0: class=0x060400 card=0x330b103c chip=0x340d8086 > rev=0x13 hdr=0x01 > vendor = 'Intel Corporation' > device = '5520/X58 I/O Hub PCI Express Root Port 6' > class = bridge > subclass = PCI-PCI > cap 0d[40] = PCI Bridge card=0x330b103c > cap 05[60] = MSI supports 2 messages, vector masks > cap 10[90] = PCI-Express 2 root port max data 256(256) link x0(x4) > cap 01[e0] = powerspec 3 supports D0 D3 current D0 > ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected > ecap 000d[150] = unknown 1 > pcib7@pci0:0:7:0: class=0x060400 card=0x330b103c chip=0x340e8086 > rev=0x13 hdr=0x01 > vendor = 'Intel Corporation' > device = '5520/5500/X58 I/O Hub PCI Express Root Port 7' > class = bridge > subclass = PCI-PCI > cap 0d[40] = PCI Bridge card=0x330b103c > cap 05[60] = MSI supports 2 messages, vector masks > cap 10[90] = PCI-Express 2 root port max data 256(256) link x4(x4) > cap 01[e0] = powerspec 3 supports D0 D3 current D0 > ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected > ecap 000d[150] = unknown 1 > ecap 000b[160] = unknown 0 > pcib8@pci0:0:8:0: class=0x060400 card=0x330b103c chip=0x340f8086 > rev=0x13 hdr=0x01 > vendor = 'Intel Corporation' > device = '5520/5500/X58 I/O Hub PCI Express Root Port 8' > class = bridge > subclass = PCI-PCI > cap 0d[40] = PCI Bridge card=0x330b103c > cap 05[60] = MSI supports 2 messages, vector masks > cap 10[90] = PCI-Express 2 root port max data 256(256) link x4(x4) > cap 01[e0] = powerspec 3 supports D0 D3 current D0 > ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected > ecap 000d[150] = unknown 1 > pcib9@pci0:0:9:0: class=0x060400 card=0x330b103c chip=0x34108086 > rev=0x13 hdr=0x01 > vendor = 'Intel Corporation' > device = '5520/5500/X58 I/O Hub PCI Express Root Port 9' > class = bridge > subclass = PCI-PCI > cap 0d[40] = PCI Bridge card=0x330b103c > cap 05[60] = MSI supports 2 messages, vector masks > cap 10[90] = PCI-Express 2 root port max data 256(256) link x0(x8) > cap 01[e0] = powerspec 3 supports D0 D3 current D0 > ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected > ecap 000d[150] = unknown 1 > pcib10@pci0:0:10:0: class=0x060400 card=0x330b103c chip=0x34118086 > rev=0x13 hdr=0x01 > vendor = 'Intel Corporation' > device = '5520/5500/X58 I/O Hub PCI Express Root Port 10' > class = bridge > subclass = PCI-PCI > cap 0d[40] = PCI Bridge card=0x330b103c > cap 05[60] = MSI supports 2 messages, vector masks > cap 10[90] = PCI-Express 2 root port max data 256(256) link x0(x4) > cap 01[e0] = powerspec 3 supports D0 D3 current D0 > ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected > ecap 000d[150] = unknown 1 > hostb1@pci0:0:13:0: class=0x060000 card=0x00000000 chip=0x343a8086 > rev=0x13 hdr=0x00 > vendor = 'Intel Corporation' > class = bridge > subclass = HOST-PCI > cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) > cap 00[60] = unknown > ecap 000b[100] = unknown 0 > ecap 000b[800] = unknown 0 > hostb2@pci0:0:13:1: class=0x060000 card=0x00000000 chip=0x343b8086 > rev=0x13 hdr=0x00 > vendor = 'Intel Corporation' > class = bridge > subclass = HOST-PCI > cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) > cap 00[60] = unknown > ecap 000b[100] = unknown 0 > ecap 000b[800] = unknown 0 > hostb3@pci0:0:13:2: class=0x060000 card=0x00000000 chip=0x343c8086 > rev=0x13 hdr=0x00 > vendor = 'Intel Corporation' > class = bridge > subclass = HOST-PCI > cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) > cap 00[60] = unknown > hostb4@pci0:0:13:3: class=0x060000 card=0x00000000 chip=0x343d8086 > rev=0x13 hdr=0x00 > vendor = 'Intel Corporation' > class = bridge > subclass = HOST-PCI > cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) > cap 00[60] = unknown > ecap 000b[100] = unknown 0 > hostb5@pci0:0:13:4: class=0x060000 card=0x00000000 chip=0x34188086 > rev=0x13 hdr=0x00 > vendor = 'Intel Corporation' > device = '5520/5500/X58 Physical Layer Port 0' > class = bridge > subclass = HOST-PCI > cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) > cap 00[60] = unknown > hostb6@pci0:0:13:5: class=0x060000 card=0x00000000 chip=0x34198086 > rev=0x13 hdr=0x00 > vendor = 'Intel Corporation' > device = '5520/5500 Physical Layer Port 1' > class = bridge > subclass = HOST-PCI > cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) > cap 00[60] = unknown > hostb7@pci0:0:13:6: class=0x060000 card=0x00000000 chip=0x341a8086 > rev=0x13 hdr=0x00 > vendor = 'Intel Corporation' > class = bridge > subclass = HOST-PCI > cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) > cap 00[60] = unknown > hostb8@pci0:0:14:0: class=0x060000 card=0x00000000 chip=0x341c8086 > rev=0x13 hdr=0x00 > vendor = 'Intel Corporation' > class = bridge > subclass = HOST-PCI > cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) > cap 00[60] = unknown > ecap 000b[100] = unknown 0 > hostb9@pci0:0:14:1: class=0x060000 card=0x00000000 chip=0x341d8086 > rev=0x13 hdr=0x00 > vendor = 'Intel Corporation' > class = bridge > subclass = HOST-PCI > cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) > cap 00[60] = unknown > hostb10@pci0:0:14:2: class=0x060000 card=0x00000000 chip=0x341e8086 > rev=0x13 hdr=0x00 > vendor = 'Intel Corporation' > class = bridge > subclass = HOST-PCI > cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) > cap 00[60] = unknown > hostb11@pci0:0:14:3: class=0x060000 card=0x00000000 chip=0x341f8086 > rev=0x13 hdr=0x00 > vendor = 'Intel Corporation' > class = bridge > subclass = HOST-PCI > cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) > cap 00[60] = unknown > hostb12@pci0:0:14:4: class=0x060000 card=0x00000000 chip=0x34398086 > rev=0x13 hdr=0x00 > vendor = 'Intel Corporation' > class = bridge > subclass = HOST-PCI > cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) > none0@pci0:0:20:0: class=0x080000 card=0x000b003c chip=0x342e8086 > rev=0x13 hdr=0x00 > vendor = 'Intel Corporation' > device = '5520/5500/X58 I/O Hub System Management Registers' > class = base peripheral > subclass = interrupt controller > cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) > none1@pci0:0:20:1: class=0x080000 card=0x000b003c chip=0x34228086 > rev=0x13 hdr=0x00 > vendor = 'Intel Corporation' > device = '5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers' > class = base peripheral > subclass = interrupt controller > cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) > none2@pci0:0:20:2: class=0x080000 card=0x000b003c chip=0x34238086 > rev=0x13 hdr=0x00 > vendor = 'Intel Corporation' > device = '5520/5500/X58 I/O Hub Control Status and RAS Registers' > class = base peripheral > subclass = interrupt controller > cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0) > pcib11@pci0:0:28:0: class=0x060400 card=0x330d103c chip=0x3a408086 > rev=0x00 hdr=0x01 > vendor = 'Intel Corporation' > device = '82801JI (ICH10 Family) PCI Express Root Port 1' > class = bridge > subclass = PCI-PCI > cap 10[40] = PCI-Express 1 root port max data 128(128) link x0(x4) > cap 05[80] = MSI supports 1 message > cap 0d[90] = PCI Bridge card=0x330d103c > cap 01[a0] = powerspec 2 supports D0 D3 current D0 > ecap 0002[100] = VC 1 max VC0 > ecap 0005[180] = unknown 1 > pcib12@pci0:0:28:4: class=0x060400 card=0x330d103c chip=0x3a488086 > rev=0x00 hdr=0x01 > vendor = 'Intel Corporation' > device = '82801JI (ICH10 Family) PCI Express Root Port 5' > class = bridge > subclass = PCI-PCI > cap 10[40] = PCI-Express 1 root port max data 128(128) link x1(x1) > cap 05[80] = MSI supports 1 message > cap 0d[90] = PCI Bridge card=0x330d103c > cap 01[a0] = powerspec 2 supports D0 D3 current D0 > ecap 0002[100] = VC 1 max VC0 > ecap 0005[180] = unknown 1 > uhci1@pci0:0:29:0: class=0x0c0300 card=0x330d103c chip=0x3a348086 > rev=0x00 hdr=0x00 > vendor = 'Intel Corporation' > device = '82801JI (ICH10 Family) USB UHCI Controller' > class = serial bus > subclass = USB > cap 13[50] = PCI Advanced Features: FLR TP > uhci2@pci0:0:29:1: class=0x0c0300 card=0x330d103c chip=0x3a358086 > rev=0x00 hdr=0x00 > vendor = 'Intel Corporation' > device = '82801JI (ICH10 Family) USB UHCI Controller' > class = serial bus > subclass = USB > cap 13[50] = PCI Advanced Features: FLR TP > uhci3@pci0:0:29:2: class=0x0c0300 card=0x330d103c chip=0x3a368086 > rev=0x00 hdr=0x00 > vendor = 'Intel Corporation' > device = '82801JI (ICH10 Family) USB UHCI Controller' > class = serial bus > subclass = USB > cap 13[50] = PCI Advanced Features: FLR TP > uhci4@pci0:0:29:3: class=0x0c0300 card=0x330d103c chip=0x3a398086 > rev=0x00 hdr=0x00 > vendor = 'Intel Corporation' > device = '82801JI (ICH10 Family) USB UHCI Controller' > class = serial bus > subclass = USB > cap 13[50] = PCI Advanced Features: FLR TP > ehci0@pci0:0:29:7: class=0x0c0320 card=0x330d103c chip=0x3a3a8086 > rev=0x00 hdr=0x00 > vendor = 'Intel Corporation' > device = '82801JI (ICH10 Family) USB2 EHCI Controller' > class = serial bus > subclass = USB > cap 01[50] = powerspec 2 supports D0 D3 current D0 > cap 0a[58] = EHCI Debug Port at offset 0xa0 in map 0x14 > cap 13[98] = PCI Advanced Features: FLR TP > pcib13@pci0:0:30:0: class=0x060401 card=0x330d103c chip=0x244e8086 > rev=0x90 hdr=0x01 > vendor = 'Intel Corporation' > device = '82801 PCI Bridge' > class = bridge > subclass = PCI-PCI > cap 0d[50] = PCI Bridge card=0x330d103c > isab0@pci0:0:31:0: class=0x060100 card=0x00000000 chip=0x3a188086 > rev=0x00 hdr=0x00 > vendor = 'Intel Corporation' > device = '82801JIB (ICH10) LPC Interface Controller' > class = bridge > subclass = PCI-ISA > cap 09[e0] = vendor (length 12) Intel cap 1 version 0 > features: Quick Resume, SATA RAID-5, 4 PCI-e x1 slots, SATA RAID-0/1/10 > atapci0@pci0:0:31:2: class=0x01018f card=0x330d103c chip=0x3a208086 > rev=0x00 hdr=0x00 > vendor = 'Intel Corporation' > device = '82801JI (ICH10 Family) 4 port SATA IDE Controller' > class = mass storage > subclass = ATA > cap 01[70] = powerspec 3 supports D0 D3 current D0 > cap 13[b0] = PCI Advanced Features: FLR TP > ciss0@pci0:5:0:0: class=0x010400 card=0x3245103c chip=0x323a103c > rev=0x01 hdr=0x00 > vendor = 'Hewlett-Packard Company' > device = 'Smart Array G6 controllers' > class = mass storage > subclass = RAID > cap 01[40] = powerspec 3 supports D0 D3 current D0 > cap 05[50] = MSI supports 1 message, 64 bit > cap 10[70] = PCI-Express 2 endpoint max data 256(256) link x4(x8) > cap 11[ac] = MSI-X supports 16 messages in map 0x10 enabled > ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected > bce0@pci0:3:0:0: class=0x020000 card=0x7055103c chip=0x163914e4 rev=0x20 > hdr=0x00 > vendor = 'Broadcom Corporation' > device = 'NetXtreme II BCM5709 Gigabit Ethernet' > class = network > subclass = ethernet > cap 01[48] = powerspec 3 supports D0 D3 current D0 > cap 03[50] = VPD > cap 05[58] = MSI supports 16 messages, 64 bit enabled with 1 message > cap 11[a0] = MSI-X supports 9 messages in map 0x10 > cap 10[ac] = PCI-Express 2 endpoint max data 256(512) link x4(x4) > ecap 0003[100] = Serial 1 ac162dfffeb700f4 > ecap 0001[110] = AER 1 0 fatal 1 non-fatal 1 corrected > ecap 0004[150] = unknown 1 > ecap 0002[160] = VC 1 max VC0 > bce1@pci0:3:0:1: class=0x020000 card=0x7055103c chip=0x163914e4 rev=0x20 > hdr=0x00 > vendor = 'Broadcom Corporation' > device = 'NetXtreme II BCM5709 Gigabit Ethernet' > class = network > subclass = ethernet > cap 01[48] = powerspec 3 supports D0 D3 current D0 > cap 03[50] = VPD > cap 05[58] = MSI supports 16 messages, 64 bit enabled with 1 message > cap 11[a0] = MSI-X supports 9 messages in map 0x10 > cap 10[ac] = PCI-Express 2 endpoint max data 256(512) link x4(x4) > ecap 0003[100] = Serial 1 ac162dfffeb700f6 > ecap 0001[110] = AER 1 0 fatal 1 non-fatal 1 corrected > ecap 0004[150] = unknown 1 > ecap 0002[160] = VC 1 max VC0 > bce2@pci0:4:0:0: class=0x020000 card=0x7055103c chip=0x163914e4 rev=0x20 > hdr=0x00 > vendor = 'Broadcom Corporation' > device = 'NetXtreme II BCM5709 Gigabit Ethernet' > class = network > subclass = ethernet > cap 01[48] = powerspec 3 supports D0 D3 current D0 > cap 03[50] = VPD > cap 05[58] = MSI supports 16 messages, 64 bit enabled with 1 message > cap 11[a0] = MSI-X supports 9 messages in map 0x10 > cap 10[ac] = PCI-Express 2 endpoint max data 256(512) link x4(x4) > ecap 0003[100] = Serial 1 ac162dfffeb700fc > ecap 0001[110] = AER 1 0 fatal 1 non-fatal 1 corrected > ecap 0004[150] = unknown 1 > ecap 0002[160] = VC 1 max VC0 > bce3@pci0:4:0:1: class=0x020000 card=0x7055103c chip=0x163914e4 rev=0x20 > hdr=0x00 > vendor = 'Broadcom Corporation' > device = 'NetXtreme II BCM5709 Gigabit Ethernet' > class = network > subclass = ethernet > cap 01[48] = powerspec 3 supports D0 D3 current D0 > cap 03[50] = VPD > cap 05[58] = MSI supports 16 messages, 64 bit enabled with 1 message > cap 11[a0] = MSI-X supports 9 messages in map 0x10 > cap 10[ac] = PCI-Express 2 endpoint max data 256(512) link x4(x4) > ecap 0003[100] = Serial 1 ac162dfffeb700fe > ecap 0001[110] = AER 1 0 fatal 1 non-fatal 1 corrected > ecap 0004[150] = unknown 1 > ecap 0002[160] = VC 1 max VC0 > none3@pci0:2:0:0: class=0x088000 card=0x3309103c chip=0x3306103c > rev=0x04 hdr=0x00 > vendor = 'Hewlett-Packard Company' > device = 'Integrated Lights-Out Standard Slave Instrumentation & > System Support' > class = base peripheral > cap 01[78] = powerspec 3 supports D0 D3 current D0 > cap 05[b0] = MSI supports 1 message, 64 bit > cap 10[c0] = PCI-Express 1 legacy endpoint max data 128(128) link x1(x1) > none4@pci0:2:0:2: class=0x088000 card=0x3309103c chip=0x3307103c > rev=0x04 hdr=0x00 > vendor = 'Hewlett-Packard Company' > device = 'Integrated Lights-Out Standard Management Processor > Support and Messaging' > class = base peripheral > cap 01[78] = powerspec 3 supports D0 D3 current D0 > cap 05[b0] = MSI supports 1 message, 64 bit > cap 10[c0] = PCI-Express 1 legacy endpoint max data 128(128) link x1(x1) > uhci0@pci0:2:0:4: class=0x0c0300 card=0x3309103c chip=0x3300103c > rev=0x01 hdr=0x00 > vendor = 'Hewlett-Packard Company' > device = 'Integrated Lights-Out Standard Virtual USB Controller' > class = serial bus > subclass = USB > cap 05[70] = MSI supports 1 message, 64 bit > cap 10[80] = PCI-Express 1 legacy endpoint max data 128(128) link x1(x1) > cap 01[f0] = powerspec 3 supports D0 D3 current D0 > vgapci0@pci0:1:3:0: class=0x030000 card=0x31fb103c chip=0x515e1002 > rev=0x02 hdr=0x00 > vendor = 'ATI Technologies Inc' > device = 'ES1000' > class = display > subclass = VGA > cap 01[50] = powerspec 2 supports D0 D1 D2 D3 current D0 > > dmesg: > [root@freebsd ~]# dmesg > Copyright (c) 1992-2012 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 9.1-RC2 #0 r241106: Mon Oct 1 18:26:44 UTC 2012 > root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 > CPU: Intel(R) Xeon(R) CPU X5660 @ 2.80GHz (2800.16-MHz > K8-class CPU) > Origin = "GenuineIntel" Id = 0x206c2 Family = 6 Model = 2c > Stepping = 2 > > Features=0xbfebfbff > > Features2=0x29ee3ff > AMD Features=0x2c100800 > AMD Features2=0x1 > TSC: P-state invariant, performance statistics > real memory = 34359738368 (32768 MB) > avail memory = 33095315456 (31562 MB) > Event timer "LAPIC" quality 600 > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 24 CPUs > FreeBSD/SMP: 2 package(s) x 6 core(s) x 2 SMT threads > cpu0 (BSP): APIC ID: 0 > cpu1 (AP): APIC ID: 1 > cpu2 (AP): APIC ID: 2 > cpu3 (AP): APIC ID: 3 > cpu4 (AP): APIC ID: 4 > cpu5 (AP): APIC ID: 5 > cpu6 (AP): APIC ID: 16 > cpu7 (AP): APIC ID: 17 > cpu8 (AP): APIC ID: 18 > cpu9 (AP): APIC ID: 19 > cpu10 (AP): APIC ID: 20 > cpu11 (AP): APIC ID: 21 > cpu12 (AP): APIC ID: 32 > cpu13 (AP): APIC ID: 33 > cpu14 (AP): APIC ID: 34 > cpu15 (AP): APIC ID: 35 > cpu16 (AP): APIC ID: 36 > cpu17 (AP): APIC ID: 37 > cpu18 (AP): APIC ID: 48 > cpu19 (AP): APIC ID: 49 > cpu20 (AP): APIC ID: 50 > cpu21 (AP): APIC ID: 51 > cpu22 (AP): APIC ID: 52 > cpu23 (AP): APIC ID: 53 > ACPI Warning: Invalid length for Pm1aControlBlock: 32, using default 16 > (20110527/tbfadt-638) > ACPI Warning: Invalid length for Pm2ControlBlock: 32, using default 8 > (20110527/tbfadt-638) > ioapic1 irqs 24-47 on motherboard > ioapic0 irqs 0-23 on motherboard > kbd1 at kbdmux0 > ctl: CAM Target Layer loaded > acpi0: on motherboard > acpi0: Power Button (fixed) > cpu0: on acpi0 > cpu1: on acpi0 > cpu2: on acpi0 > cpu3: on acpi0 > cpu4: on acpi0 > cpu5: on acpi0 > cpu6: on acpi0 > cpu7: on acpi0 > cpu8: on acpi0 > cpu9: on acpi0 > cpu10: on acpi0 > cpu11: on acpi0 > cpu12: on acpi0 > cpu13: on acpi0 > cpu14: on acpi0 > cpu15: on acpi0 > cpu16: on acpi0 > cpu17: on acpi0 > cpu18: on acpi0 > cpu19: on acpi0 > cpu20: on acpi0 > cpu21: on acpi0 > cpu22: on acpi0 > cpu23: on acpi0 > attimer0: port 0x40-0x43 irq 0 on acpi0 > Timecounter "i8254" frequency 1193182 Hz quality 0 > Event timer "i8254" frequency 1193182 Hz quality 100 > hpet0: iomem 0xfed00000-0xfed003ff on acpi0 > Timecounter "HPET" frequency 14318180 Hz quality 950 > Event timer "HPET" frequency 14318180 Hz quality 450 > Event timer "HPET1" frequency 14318180 Hz quality 440 > Event timer "HPET2" frequency 14318180 Hz quality 440 > Event timer "HPET3" frequency 14318180 Hz quality 440 > atrtc0: port 0x70-0x71 on acpi0 > Event timer "RTC" frequency 32768 Hz quality 0 > Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x908-0x90b on acpi0 > pcib0: on acpi0 > pci0: on pcib0 > pcib1: at device 1.0 on pci0 > pci5: on pcib1 > ciss0: port 0x4000-0x40ff mem > 0xfbe00000-0xfbffffff,0xfbdf0000-0xfbdf0fff irq 28 at device 0.0 on pci5 > ciss0: PERFORMANT Transport > pcib2: at device 2.0 on pci0 > pci12: on pcib2 > pcib3: at device 3.0 on pci0 > pci9: on pcib3 > pcib4: at device 4.0 on pci0 > pci13: on pcib4 > pcib5: at device 5.0 on pci0 > pci14: on pcib5 > pcib6: at device 6.0 on pci0 > pci15: on pcib6 > pcib7: at device 7.0 on pci0 > pci3: on pcib7 > bce0: mem > 0xf4000000-0xf5ffffff irq 30 at device 0.0 on pci3 > miibus0: on bce0 > brgphy0: PHY 1 on miibus0 > brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, > 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow > bce0: Ethernet address: ac:16:2d:b7:00:f4 > bce0: ASIC (0x57092003); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.3); > Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.12) > Coal (RX:6,6,18,18; TX:20,20,80,80) > bce1: mem > 0xf2000000-0xf3ffffff irq 37 at device 0.1 on pci3 > miibus1: on bce1 > brgphy1: PHY 1 on miibus1 > brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, > 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow > bce1: Ethernet address: ac:16:2d:b7:00:f6 > bce1: ASIC (0x57092003); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.3); > Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.12) > Coal (RX:6,6,18,18; TX:20,20,80,80) > pcib8: at device 8.0 on pci0 > pci4: on pcib8 > bce2: mem > 0xf8000000-0xf9ffffff irq 31 at device 0.0 on pci4 > miibus2: on bce2 > brgphy2: PHY 1 on miibus2 > brgphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, > 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow > bce2: Ethernet address: ac:16:2d:b7:00:fc > bce2: ASIC (0x57092003); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.3); > Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.12) > Coal (RX:6,6,18,18; TX:20,20,80,80) > bce3: mem > 0xf6000000-0xf7ffffff irq 39 at device 0.1 on pci4 > miibus3: on bce3 > brgphy3: PHY 1 on miibus3 > brgphy3: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, > 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow > bce3: Ethernet address: ac:16:2d:b7:00:fe > bce3: ASIC (0x57092003); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.3); > Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.12) > Coal (RX:6,6,18,18; TX:20,20,80,80) > pcib9: at device 9.0 on pci0 > pci6: on pcib9 > pcib10: at device 10.0 on pci0 > pci16: on pcib10 > pci0: at device 20.0 (no driver > attached) > pci0: at device 20.1 (no driver > attached) > pci0: at device 20.2 (no driver > attached) > pcib11: at device 28.0 on pci0 > pci17: on pcib11 > pcib12: at device 28.4 on pci0 > pci2: on pcib12 > pci2: at device 0.0 (no driver attached) > pci2: at device 0.2 (no driver attached) > uhci0: port 0x3c00-0x3c1f irq 17 at > device 0.4 on pci2 > usbus0 on uhci0 > uhci1: port 0x1000-0x101f > irq 20 at device 29.0 on pci0 > usbus1 on uhci1 > uhci2: port 0x1020-0x103f > irq 23 at device 29.1 on pci0 > usbus2 on uhci2 > uhci3: port 0x1040-0x105f > irq 22 at device 29.2 on pci0 > usbus3 on uhci3 > uhci4: port 0x1060-0x107f > irq 23 at device 29.3 on pci0 > usbus4 on uhci4 > ehci0: mem > 0xf1bf0000-0xf1bf03ff irq 20 at device 29.7 on pci0 > usbus5: EHCI version 1.0 > usbus5 on ehci0 > pcib13: at device 30.0 on pci0 > pci1: on pcib13 > vgapci0: port 0x2000-0x20ff mem > 0xe8000000-0xefffffff,0xf1cf0000-0xf1cfffff irq 23 at device 3.0 on pci1 > isab0: at device 31.0 on pci0 > isa0: on isab0 > atapci0: port > 0x1080-0x1087,0x1088-0x108b,0x1090-0x1097,0x1098-0x109b,0x10a0-0x10af,0x10b0-0x10bf > irq 17 at device 31.2 on pci0 > ata2: at channel 0 on atapci0 > ata3: at channel 1 on atapci0 > acpi_tz0: on acpi0 > atkbdc0: port 0x60,0x64 irq 1 on acpi0 > atkbd0: irq 1 on atkbdc0 > kbd0 at atkbd0 > atkbd0: [GIANT-LOCKED] > uart0: port 0x3f8-0x3ff irq > 4 flags 0x10 on acpi0 > qpi0: on motherboard > orm0: at iomem 0xc0000-0xcafff on isa0 > sc0: at flags 0x100 on isa0 > sc0: VGA <16 virtual consoles, flags=0x300> > vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 > ppc0: cannot reserve I/O port range > uart1: at port 0x2f8-0x2ff > irq 3 on isa0 > est0: on cpu0 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 16 > device_attach: est0 attach returned 6 > p4tcc0: on cpu0 > est1: on cpu1 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 16 > device_attach: est1 attach returned 6 > p4tcc1: on cpu1 > est2: on cpu2 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 16 > device_attach: est2 attach returned 6 > p4tcc2: on cpu2 > est3: on cpu3 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 16 > device_attach: est3 attach returned 6 > p4tcc3: on cpu3 > est4: on cpu4 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 16 > device_attach: est4 attach returned 6 > p4tcc4: on cpu4 > est5: on cpu5 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 16 > device_attach: est5 attach returned 6 > p4tcc5: on cpu5 > est6: on cpu6 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 16 > device_attach: est6 attach returned 6 > p4tcc6: on cpu6 > est7: on cpu7 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 16 > device_attach: est7 attach returned 6 > p4tcc7: on cpu7 > est8: on cpu8 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 16 > device_attach: est8 attach returned 6 > p4tcc8: on cpu8 > est9: on cpu9 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 16 > device_attach: est9 attach returned 6 > p4tcc9: on cpu9 > est10: on cpu10 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 16 > device_attach: est10 attach returned 6 > p4tcc10: on cpu10 > est11: on cpu11 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 16 > device_attach: est11 attach returned 6 > p4tcc11: on cpu11 > est12: on cpu12 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 16 > device_attach: est12 attach returned 6 > p4tcc12: on cpu12 > est13: on cpu13 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 16 > device_attach: est13 attach returned 6 > p4tcc13: on cpu13 > est14: on cpu14 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 16 > device_attach: est14 attach returned 6 > p4tcc14: on cpu14 > est15: on cpu15 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 16 > device_attach: est15 attach returned 6 > p4tcc15: on cpu15 > est16: on cpu16 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 16 > device_attach: est16 attach returned 6 > p4tcc16: on cpu16 > est17: on cpu17 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 16 > device_attach: est17 attach returned 6 > p4tcc17: on cpu17 > est18: on cpu18 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 16 > device_attach: est18 attach returned 6 > p4tcc18: on cpu18 > est19: on cpu19 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 16 > device_attach: est19 attach returned 6 > p4tcc19: on cpu19 > est20: on cpu20 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 16 > device_attach: est20 attach returned 6 > p4tcc20: on cpu20 > est21: on cpu21 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 16 > device_attach: est21 attach returned 6 > p4tcc21: on cpu21 > est22: on cpu22 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 16 > device_attach: est22 attach returned 6 > p4tcc22: on cpu22 > est23: on cpu23 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 16 > device_attach: est23 attach returned 6 > p4tcc23: on cpu23 > ZFS filesystem version 5 > ZFS storage pool version 28 > Timecounters tick every 1.000 msec > usbus0: 12Mbps Full Speed USB v1.0 > usbus1: 12Mbps Full Speed USB v1.0 > usbus2: 12Mbps Full Speed USB v1.0 > usbus3: 12Mbps Full Speed USB v1.0 > usbus4: 12Mbps Full Speed USB v1.0 > usbus5: 480Mbps High Speed USB v2.0 > ugen0.1: <0x103c> at usbus0 > uhub0: <0x103c UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0 > ugen1.1: at usbus1 > uhub1: on usbus1 > ugen2.1: at usbus2 > uhub2: on usbus2 > ugen3.1: at usbus3 > uhub3: on usbus3 > ugen4.1: at usbus4 > uhub4: on usbus4 > ugen5.1: at usbus5 > uhub5: on usbus5 > uhub0: 2 ports with 2 removable, self powered > uhub1: 2 ports with 2 removable, self powered > uhub2: 2 ports with 2 removable, self powered > uhub3: 2 ports with 2 removable, self powered > uhub4: 2 ports with 2 removable, self powered > ugen0.2: at usbus0 > ukbd0: on usbus0 > kbd2 at ukbd0 > ums0: on usbus0 > (probe0:ciss0:0:0:0): REPORT LUNS. CDB: a0 0 0 0 0 0 0 0 0 10 0 0 > (probe0:ciss0:0:0:0): CAM status: SCSI Status Error > (probe0:ciss0:0:0:0): SCSI status: Check Condition > (probe0:ciss0:0:0:0): SCSI sense: ILLEGAL REQUEST asc:20,0 (Invalid > command operation code) > (probe0:ciss0:0:0:0): Error 22, Unretryable error > da0 at ciss0 bus 0 scbus0 target 0 lun 0 > da0: Fixed Direct Access SCSI-5 device > da0: 135.168MB/s transfers > da0: Command Queueing enabled > da0: 286070MB (585871964 512 byte sectors: 255H 32S/T 65535C) > SMP: AP CPU #1 Launched! > cd0 at ata2 bus 0 scbus2 target 0 lun 0 > cd0: Removable CD-ROM SCSI-0 device > SMP: AP CPU #12 Launched! > cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes) > cd0: Attempt to query device size failed: NOT READY, Medium not present > - tray closed > SMP: AP CPU #11 Launched! > SMP: AP CPU #19 Launched! > SMP: AP CPU #10 Launched! > SMP: AP CPU #22 Launched! > SMP: AP CPU #8 Launched! > SMP: AP CPU #16 Launched! > SMP: AP CPU #9 Launched! > SMP: AP CPU #23 Launched! > SMP: AP CPU #21 Launched! > SMP: AP CPU #3 Launched! > SMP: AP CPU #14 Launched! > SMP: AP CPU #6 Launched! > SMP: AP CPU #17 Launched! > SMP: AP CPU #7 Launched! > SMP: AP CPU #20 Launched! > SMP: AP CPU #5 Launched! > SMP: AP CPU #13 Launched! > SMP: AP CPU #2 Launched! > SMP: AP CPU #18 Launched! > SMP: AP CPU #4 Launched! > SMP: AP CPU #15 Launched! > Timecounter "TSC-low" frequency 10938126 Hz quality 1000 > uhub5: 8 ports with 8 removable, self powered > Trying to mount root from ufs:/dev/da0p2 [rw]... > bce0: link state changed to UP > bce0: Gigabit link up! > bce0: Gigabit link up! > bce0: Gigabit link up! > bce1: link state changed to UP > bce1: Gigabit link up! > bce1: Gigabit link up! > bce1: Gigabit link up! > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 18:39:48 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C3ED5692 for ; Wed, 21 Nov 2012 18:39:48 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 86ADA8FC0C for ; Wed, 21 Nov 2012 18:39:48 +0000 (UTC) Received: by mail-ob0-f182.google.com with SMTP id 16so9464395obc.13 for ; Wed, 21 Nov 2012 10:39:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=tTOLAvnmqavixqOlZkdEGZRMQjoniOLUEiyLjlRb8k0=; b=lnuLYrcZrIHV5l8IwyNQad46EnJad/wE6CV7VGiDdOaWWNcvppZruGqtL6eaD2A/Pk aMafLzC3vGE9UY5XHUJaiBg2ufJ7Xchc48mCQcQGTdEc6HZD4OuXUvapmhDxf1Q3ucuD PbrVbNok/hooH6EdEb54KMHWxkd5xKbpyCkrhM6KpcKCBV3wBlomlMnargZw0XnoBUGa iMRkwss25dDP4ioESRUyVFtOaaoTg2y0pOdvnJGxJq3IA0Sq1DIRbqATUxyE8PYTqnaN zQbFW6vbR15b9g0eBRfJ7TuS0L9n7htqjGOsGYSACsp3FuIQRWHl75l6UpnW6uN92k4m vLJA== MIME-Version: 1.0 Received: by 10.60.25.104 with SMTP id b8mr16736915oeg.18.1353523187976; Wed, 21 Nov 2012 10:39:47 -0800 (PST) Received: by 10.76.143.33 with HTTP; Wed, 21 Nov 2012 10:39:47 -0800 (PST) Date: Wed, 21 Nov 2012 10:39:47 -0800 Message-ID: Subject: [RFC] Prune net.inet6.ip6.rr_prune? From: Garrett Cooper To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 18:39:48 -0000 While going through the tree trying to document all of our net.inet6 sysctls, I noticed that net.inet6.ip6.rr_prune is defined, but not actually used anywhere in the stack: netinet6/ip6_var.h:VNET_DECLARE(int, ip6_rr_prune); /* router renumbering prefix netinet6/ip6_var.h:#define V_ip6_rr_prune VNET(ip6_rr_prune) netinet6/in6_proto.c:VNET_DEFINE(int, ip6_rr_prune) = 5; /* router renumbering prefix netinet6/in6_proto.c:SYSCTL_VNET_INT(_net_inet6_ip6, IPV6CTL_RR_PRUNE, rr_prune, CTLFLAG_RW, netinet6/in6_proto.c: &VNET_NAME(ip6_rr_prune), 0, The knob was declared in r181803 and shuffled around a few times, but isn't in use anywhere (either then or now). Should I send out a PR to remove it (or am I missing some context)? Thanks, -Garrett From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 18:50:06 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1F6FDE7B for ; Wed, 21 Nov 2012 18:50:06 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) by mx1.freebsd.org (Postfix) with ESMTP id EC1A88FC0C for ; Wed, 21 Nov 2012 18:50:05 +0000 (UTC) Received: from JRE-MBP-2.local (c-50-143-149-146.hsd1.ca.comcast.net [50.143.149.146]) (authenticated bits=0) by vps1.elischer.org (8.14.5/8.14.5) with ESMTP id qALInwIh083667 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Wed, 21 Nov 2012 10:49:58 -0800 (PST) (envelope-from julian@freebsd.org) Message-ID: <50AD2251.3040904@freebsd.org> Date: Wed, 21 Nov 2012 10:49:53 -0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:16.0) Gecko/20121026 Thunderbird/16.0.2 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: Re: Low Bandwidth on intercontinental connections References: <50ACF62C.8000408@mpeters.org> In-Reply-To: <50ACF62C.8000408@mpeters.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 18:50:06 -0000 On 11/21/12 7:41 AM, Marc Peters wrote: > Hi list, > > we are experiencing low throughput on interncontinental connections with > our FreeBSD Servers. We made several tests and are wondering, why this > would be. The first tests were on an IPSEC VPN between our datacenter in > DE and Santa Clara, CA. We are connected with two gigabit uplinks in > each DC. Pushing data by scp between our FreeBSD servers takes ages. > Starting with several MB/s it drops to 60-70KB/s: > > [root@freebsd ~]# ls -alh test.tgz > -rw-r----- 1 root wheel 58M Oct 5 2010 test.tgz > [root@freebsd ~]# scp test.tgz 172.16.3.10:. > Password: > test.tgz 28% 17MB 75.3KB/s 09:32 ETA > > > For comparision, we did a similiar test with Linux, which didn't show > this behaviour: > > root@linux:~# scp jdk-6u33-linux-x64.bin 172.16.4.50: > root@172.16.4.50's password: > jdk-6u33-linux-x64.bin 100% > 69MB 3.4MB/s 00:20 > root@linux:~# > > > Otherwise, the servers are really fast, when copying data to a machine > nearby: > > [root@freebsd ~]# ls -alh test > -rw-r--r-- 1 root wheel 1G Nov 21 13:43 test > [root@freebsd ~]# scp test 172.16.3.11: > Password: > test 100% 1000MB 38.5MB/s 00:26 > > > Intercontinental ftp downloads are the same: > > [root@freebsd ~]# fetch > ftp://ftp1.us.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-bootonly.iso > FreeBSD-9.1-RC3-amd64-bootonly.iso 100% of 146 MB 46 MBps > > [root@freebsd ~]# fetch > ftp://ftp1.us.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso > > FreeBSD-9.1-RC3-amd64-disc1.iso 100% of 685 MB 36 MBps 00m00s > > [root@freebsd ~]# fetch > ftp://ftp1.de.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso > FreeBSD-9.1-RC3-amd64-disc1.iso 0% of 685 MB 13 kBps 14h49m^C > > > Linux: > > root@linux:~# wget > ftp://ftp1.de.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso > --2012-11-21 15:07:57-- > ftp://ftp1.de.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso > => `FreeBSD-9.1-RC3-amd64-disc1.iso' > Resolving ftp1.de.freebsd.org... 137.226.34.42 > Connecting to ftp1.de.freebsd.org|137.226.34.42|:21... connected. > Logging in as anonymous ... Logged in! > ==> SYST ... done. ==> PWD ... done. > ==> TYPE I ... done. ==> CWD (1) > /pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1 ... done. > ==> SIZE FreeBSD-9.1-RC3-amd64-disc1.iso ... 718800896 > ==> PASV ... done. ==> RETR FreeBSD-9.1-RC3-amd64-disc1.iso ... done. > Length: 718800896 (686M) (unauthoritative) > > 100%[=====================================================================>] > 718,800,896 19.1M/s in 61s > > 2012-11-21 15:09:01 (11.2 MB/s) - `FreeBSD-9.1-RC3-amd64-disc1.iso' > saved [718800896] > > > Doing some googling brought up a lot of tuning hints, but nothing worked > for us. We tweaked some sysctls: > > kern.ipc.maxsockbuf=16777216 > net.inet.tcp.sendbuf_max=16777216 > net.inet.tcp.recvbuf_max=16777216 > net.inet.tcp.sendbuf_inc=16384 > net.inet.tcp.recvbuf_inc=524288 > net.inet.tcp.hostcache.expire=1 > > but to no good. Disabling MSI and TSO4 for the card didn't change > anything, too. > > The machines are all HP DL360G7 with bce cards (find dmesg, ifconfig and > pciconf -lvc at the end of this mail). > > Can someone hit me with a cluestick to get the BSDs on speed? you really do need to get a tcpdump of the transfer under slow conditions and a SIFTR output to match. What is the ping time between the hosts. that will allow you to work out how large a window you should have. > > marc > > PS: The version is FreeBSD-RC2 amd64, because we need the patch for > process migration on the CPUs which didn't make it 9.0 or an errata, as > we were the only ones, hitting this bug (so kib@ said). > From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 19:14:25 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 63962678; Wed, 21 Nov 2012 19:14:25 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-vb0-f54.google.com (mail-vb0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id D6B098FC12; Wed, 21 Nov 2012 19:14:24 +0000 (UTC) Received: by mail-vb0-f54.google.com with SMTP id l1so9978522vba.13 for ; Wed, 21 Nov 2012 11:14:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=3UAZzvL9bkhCt7a/a3nY6YBY3pplxN4zK+vCsCm3Ht0=; b=Zk1qPoqX1U6cu4H8X52jiHSLCIbwrpbjZCjkv++AqfbA7laAZuLG+Mgo01Now8VAIi BMlD4tmOulDH24ik7nUpjcJ4j9YkKFszX8Wfh9Lz/RtVAVz4QqE9UexXc+Ft8IeHml2r att+kmZQXvIdbrMQIGj7EP44ZRhFZop3Dkhadc5cNmMdL6OylVeNc8F+AYzDf+Q5lTAp qxMCRIdxTFgfqbHp93Bx4BEzm0M2XQkDMwXonkgOqzKY4H7C2hHeMjwRjQrhBoUpgFT2 i+WHCw/4eWdS+hbk5kpi9S335lT9NvPkT7mDYOcOLNV3lx2+CJGcxru1kC/pWgzOJnT9 R78g== MIME-Version: 1.0 Received: by 10.52.100.5 with SMTP id eu5mr25952943vdb.34.1353525263965; Wed, 21 Nov 2012 11:14:23 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.58.2.5 with HTTP; Wed, 21 Nov 2012 11:14:23 -0800 (PST) In-Reply-To: <50AD2251.3040904@freebsd.org> References: <50ACF62C.8000408@mpeters.org> <50AD2251.3040904@freebsd.org> Date: Wed, 21 Nov 2012 11:14:23 -0800 X-Google-Sender-Auth: 0DDxggyEBz-DrWgV-J5AzUoShLI Message-ID: Subject: Re: Low Bandwidth on intercontinental connections From: Adrian Chadd To: Julian Elischer Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 19:14:25 -0000 .. and there's also some SACK stuff and RTT prediction that you may be totally running afoul of over that high latency link? (I thought this stuff was fixed in -HEAD and -9?) Adrian From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 19:44:59 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8A2BF22A; Wed, 21 Nov 2012 19:44:59 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-vb0-f54.google.com (mail-vb0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id 057498FC12; Wed, 21 Nov 2012 19:44:58 +0000 (UTC) Received: by mail-vb0-f54.google.com with SMTP id l1so10015476vba.13 for ; Wed, 21 Nov 2012 11:44:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Z6wmVCN3t8zQqEuPS74wjLm6snhh0Dvu1b/hxqYvUb0=; b=QnuPmNx1nnYiMynILbZOFS4CilquakkBnYnp2ljBwPgwD3Q1Zdo2D+uf6x0QFK9Mn1 qtxLSWs1UvWTMshLhWsvjdMzGjbsFsuK+pOX8iUXGQS/CT81MLjfwmhSm7GjxA241uiS lHwKX4O1/EhEjNTgaPtlzqXCsGhF+s1dvgjO77lBWdXxGz1y3LhIBol1FOYoeh2UKvZ4 S6qK7HzOKvRm0jzyBUQK+c/wUBWiHqixjpqfwXtHrVqUzhcqtYa1Mq5PkxhKbUoWBagm uGr4H6WJKbV6WnUWDFkaERM+4xPtCjuwyzeooVaDIZkwTdqEYrC2XiGBWtZqWrpPcPg0 f7hQ== MIME-Version: 1.0 Received: by 10.58.15.227 with SMTP id a3mr28889362ved.38.1353527097931; Wed, 21 Nov 2012 11:44:57 -0800 (PST) Received: by 10.59.3.165 with HTTP; Wed, 21 Nov 2012 11:44:57 -0800 (PST) In-Reply-To: <20121121062631.GJ67660@glebius.int.ru> References: <50AA8F24.7080604@gmail.com> <20121120111833.GC67660@FreeBSD.org> <20121121062631.GJ67660@glebius.int.ru> Date: Wed, 21 Nov 2012 11:44:57 -0800 Message-ID: Subject: Re: igb diver crashes in head@241037 From: Jack Vogel To: Gleb Smirnoff Content-Type: multipart/mixed; boundary=047d7b5daf50ddb67504cf069572 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Karim Fodil-Lemelin , jfv@freebsd.org, freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 19:44:59 -0000 --047d7b5daf50ddb67504cf069572 Content-Type: text/plain; charset=ISO-8859-1 Gleb, Here is a patch based on my latest igb internal code, I had not yet committed this as I was not completely confident about the start/queueing changes, I would love to have a wider testing base, so anyone that wishes to test this... Its against HEAD. It does a few things: change mq_start to ALWAYS queue, hence mq_start_locked no longer takes an mbuf pointer arg. Second, it gets rid of OACTIVE as far as the queues go, its still used only in a device wide up/down sense. Last, there is a flow control display added, this follows what our linux driver does, it gives you the current flow control state when a link up event happens. I was asked to do this by my validation group, and it seemed kinda handy... Let me know what you think, Jack On Tue, Nov 20, 2012 at 10:26 PM, Gleb Smirnoff wrote: > Jack, > > On Tue, Nov 20, 2012 at 09:19:54AM -0800, Jack Vogel wrote: > J> > I'd suggest the following code: > J> > > J> > if (m) > J> > drbr_enqueue(ifp, txr->br, m); > J> > err = igb_mq_start_locked(ifp, txr, NULL); > J> > > J> > Which eventually leads us to all invocations of igb_mq_start_locked() > J> > called > J> > with third argument as NULL. This allows us to simplify this function. > J> > > J> > Patch for review attached. > J> > > J> > > J> Yes Gleb, I already have code in my internal tree which simply removes > an > J> mbuf > J> pointer form the start_locked call and ALWAYS does a dequeue, start > J> similarly > J> will always enqueue. I just have been busy with ixgbe for a bit and have > J> not gotten > J> it committed yet. > > Since ixgbe work is performance tuning and this patch closes a kernel > crash, > I'd ask to preempt the ixgbe job with this patch. :) > > Or you can approve my patch and I will check it in. > > -- > Totus tuus, Glebius. > --047d7b5daf50ddb67504cf069572 Content-Type: application/octet-stream; name="if_igb.patch" Content-Disposition: attachment; filename="if_igb.patch" Content-Transfer-Encoding: base64 X-Attachment-Id: f_h9sv1y400 LS0tIGlmX2lnYi5jCTIwMTItMTAtMzAgMDk6NDY6NTIuNDc4MTQ3MjIxIC0wNzAwCisrKyBpZl9p Z2IubmV3LmMJMjAxMi0xMS0yMSAxMTozMjowMS4xNzU2MDU0NjEgLTA4MDAKQEAgLTEwMCw3ICsx MDAsNyBAQAogLyoqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKgogICogIERyaXZlciB2ZXJzaW9uOgogICoqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq Ki8KLWNoYXIgaWdiX2RyaXZlcl92ZXJzaW9uW10gPSAidmVyc2lvbiAtIDIuMy41IjsKK2NoYXIg aWdiX2RyaXZlcl92ZXJzaW9uW10gPSAidmVyc2lvbiAtIDIuMy44IjsKIAogCiAvKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqCkBAIC0xODEsOCArMTgxLDcgQEAKIHN0YXRpYyBpbnQJaWdiX3Jlc3VtZShkZXZpY2VfdCk7 CiAjaWYgX19GcmVlQlNEX3ZlcnNpb24gPj0gODAwMDAwCiBzdGF0aWMgaW50CWlnYl9tcV9zdGFy dChzdHJ1Y3QgaWZuZXQgKiwgc3RydWN0IG1idWYgKik7Ci1zdGF0aWMgaW50CWlnYl9tcV9zdGFy dF9sb2NrZWQoc3RydWN0IGlmbmV0ICosCi0JCSAgICBzdHJ1Y3QgdHhfcmluZyAqLCBzdHJ1Y3Qg bWJ1ZiAqKTsKK3N0YXRpYyBpbnQJaWdiX21xX3N0YXJ0X2xvY2tlZChzdHJ1Y3QgaWZuZXQgKiwg c3RydWN0IHR4X3JpbmcgKik7CiBzdGF0aWMgdm9pZAlpZ2JfcWZsdXNoKHN0cnVjdCBpZm5ldCAq KTsKIHN0YXRpYyB2b2lkCWlnYl9kZWZlcnJlZF9tcV9zdGFydCh2b2lkICosIGludCk7CiAjZWxz ZQpAQCAtODQ1LDcgKzg0NCw3IEBACiAJCQkvKiBQcm9jZXNzIHRoZSBzdGFjayBxdWV1ZSBvbmx5 IGlmIG5vdCBkZXBsZXRlZCAqLwogCQkJaWYgKCgodHhyLT5xdWV1ZV9zdGF0dXMgJiBJR0JfUVVF VUVfREVQTEVURUQpID09IDApICYmCiAJCQkgICAgIWRyYnJfZW1wdHkoaWZwLCB0eHItPmJyKSkK LQkJCQlpZ2JfbXFfc3RhcnRfbG9ja2VkKGlmcCwgdHhyLCBOVUxMKTsKKwkJCQlpZ2JfbXFfc3Rh cnRfbG9ja2VkKGlmcCwgdHhyKTsKICNlbHNlCiAJCQlpZiAoIUlGUV9EUlZfSVNfRU1QVFkoJmlm cC0+aWZfc25kKSkKIAkJCQlpZ2Jfc3RhcnRfbG9ja2VkKHR4ciwgaWZwKTsKQEAgLTk1OSw2OCAr OTU4LDQ3IEBACiAKIAl0eHIgPSAmYWRhcHRlci0+dHhfcmluZ3NbaV07CiAJcXVlID0gJmFkYXB0 ZXItPnF1ZXVlc1tpXTsKLQlpZiAoKCh0eHItPnF1ZXVlX3N0YXR1cyAmIElHQl9RVUVVRV9ERVBM RVRFRCkgPT0gMCkgJiYKLQkgICAgSUdCX1RYX1RSWUxPQ0sodHhyKSkgewotCQlzdHJ1Y3QgbWJ1 ZiAqcG0gPSBOVUxMOwotCQkvKgotCQkqKiBUcnkgdG8gcXVldWUgZmlyc3QgdG8gYXZvaWQKLQkJ Kiogb3V0LW9mLW9yZGVyIGRlbGl2ZXJ5LCBidXQgCi0JCSoqIHNldHRsZSBmb3IgaXQgaWYgdGhh dCBmYWlscwotCQkqLwotCQlpZiAobSAmJiBkcmJyX2VucXVldWUoaWZwLCB0eHItPmJyLCBtKSkK LQkJCXBtID0gbTsKLQkJZXJyID0gaWdiX21xX3N0YXJ0X2xvY2tlZChpZnAsIHR4ciwgcG0pOwot CQlJR0JfVFhfVU5MT0NLKHR4cik7Ci0JfSBlbHNlIHsKLQkJZXJyID0gZHJicl9lbnF1ZXVlKGlm cCwgdHhyLT5iciwgbSk7Ci0JCXRhc2txdWV1ZV9lbnF1ZXVlKHF1ZS0+dHEsICZ0eHItPnR4cV90 YXNrKTsKLQl9CisJdGFza3F1ZXVlX2VucXVldWUocXVlLT50cSwgJnR4ci0+dHhxX3Rhc2spOwog CiAJcmV0dXJuIChlcnIpOwogfQogCiBzdGF0aWMgaW50Ci1pZ2JfbXFfc3RhcnRfbG9ja2VkKHN0 cnVjdCBpZm5ldCAqaWZwLCBzdHJ1Y3QgdHhfcmluZyAqdHhyLCBzdHJ1Y3QgbWJ1ZiAqbSkKK2ln Yl9tcV9zdGFydF9sb2NrZWQoc3RydWN0IGlmbmV0ICppZnAsIHN0cnVjdCB0eF9yaW5nICp0eHIp CiB7CiAJc3RydWN0IGFkYXB0ZXIgICphZGFwdGVyID0gdHhyLT5hZGFwdGVyOwotICAgICAgICBz dHJ1Y3QgbWJ1ZiAgICAgKm5leHQ7Ci0gICAgICAgIGludCAgICAgICAgICAgICBlcnIgPSAwLCBl bnE7CisgICAgICAgIHN0cnVjdCBtYnVmICAgICAqbTsKKyAgICAgICAgaW50ICAgICAgICAgICAg IGVyciA9IDAsIGVucSA9IDA7CiAKIAlJR0JfVFhfTE9DS19BU1NFUlQodHhyKTsKIAogCWlmICgo KGlmcC0+aWZfZHJ2X2ZsYWdzICYgSUZGX0RSVl9SVU5OSU5HKSA9PSAwKSB8fAotCSAgICAodHhy LT5xdWV1ZV9zdGF0dXMgJiBJR0JfUVVFVUVfREVQTEVURUQpIHx8Ci0JICAgIGFkYXB0ZXItPmxp bmtfYWN0aXZlID09IDApIHsKLQkJaWYgKG0gIT0gTlVMTCkKLQkJCWVyciA9IGRyYnJfZW5xdWV1 ZShpZnAsIHR4ci0+YnIsIG0pOwotCQlyZXR1cm4gKGVycik7Ci0JfQorCSAgICBhZGFwdGVyLT5s aW5rX2FjdGl2ZSA9PSAwKQorCQlyZXR1cm4gKEVORVRET1dOKTsKIAotCWVucSA9IDA7Ci0JaWYg KG0gPT0gTlVMTCkgewotCQluZXh0ID0gZHJicl9kZXF1ZXVlKGlmcCwgdHhyLT5icik7Ci0JfSBl bHNlIGlmIChkcmJyX25lZWRzX2VucXVldWUoaWZwLCB0eHItPmJyKSkgewotCQlpZiAoKGVyciA9 IGRyYnJfZW5xdWV1ZShpZnAsIHR4ci0+YnIsIG0pKSAhPSAwKQotCQkJcmV0dXJuIChlcnIpOwot CQluZXh0ID0gZHJicl9kZXF1ZXVlKGlmcCwgdHhyLT5icik7Ci0JfSBlbHNlCi0JCW5leHQgPSBt OworCWlmICh0eHItPnF1ZXVlX3N0YXR1cyAmIElHQl9RVUVVRV9ERVBMRVRFRCkKKwkJaWdiX3R4 ZW9mKHR4cik7CisJaWYgKHR4ci0+dHhfYXZhaWwgPj0gSUdCX01BWF9TQ0FUVEVSKQorICAgICAg ICAgICAgICAgIHR4ci0+cXVldWVfc3RhdHVzICY9IH5JR0JfUVVFVUVfREVQTEVURUQ7CisJZWxz ZQorCQlyZXR1cm4gKEVJTyk7CiAKIAkvKiBQcm9jZXNzIHRoZSBxdWV1ZSAqLwotCXdoaWxlIChu ZXh0ICE9IE5VTEwpIHsKLQkJaWYgKChlcnIgPSBpZ2JfeG1pdCh0eHIsICZuZXh0KSkgIT0gMCkg ewotCQkJaWYgKG5leHQgIT0gTlVMTCkKLQkJCQllcnIgPSBkcmJyX2VucXVldWUoaWZwLCB0eHIt PmJyLCBuZXh0KTsKKwltID0gZHJicl9kZXF1ZXVlKGlmcCwgdHhyLT5icik7CisJd2hpbGUgKG0g IT0gTlVMTCkgeworCQlpZiAoKGVyciA9IGlnYl94bWl0KHR4ciwgJm0pKSAhPSAwKSB7CisJCQlp ZiAobSAhPSBOVUxMKQorCQkJCWVyciA9IGRyYnJfZW5xdWV1ZShpZnAsIHR4ci0+YnIsIG0pOwog CQkJYnJlYWs7CiAJCX0KIAkJZW5xKys7Ci0JCWlmcC0+aWZfb2J5dGVzICs9IG5leHQtPm1fcGt0 aGRyLmxlbjsKLQkJaWYgKG5leHQtPm1fZmxhZ3MgJiBNX01DQVNUKQorCQlpZnAtPmlmX29ieXRl cyArPSBtLT5tX3BrdGhkci5sZW47CisJCWlmIChtLT5tX2ZsYWdzICYgTV9NQ0FTVCkKIAkJCWlm cC0+aWZfb21jYXN0cysrOwotCQlFVEhFUl9CUEZfTVRBUChpZnAsIG5leHQpOworCQlFVEhFUl9C UEZfTVRBUChpZnAsIG0pOwogCQlpZiAoKGlmcC0+aWZfZHJ2X2ZsYWdzICYgSUZGX0RSVl9SVU5O SU5HKSA9PSAwKQogCQkJYnJlYWs7Ci0JCW5leHQgPSBkcmJyX2RlcXVldWUoaWZwLCB0eHItPmJy KTsKKwkJbSA9IGRyYnJfZGVxdWV1ZShpZnAsIHR4ci0+YnIpOwogCX0KIAlpZiAoZW5xID4gMCkg ewogCQkvKiBTZXQgdGhlIHdhdGNoZG9nICovCkBAIC0xMDQ2LDcgKzEwMjQsNyBAQAogCiAJSUdC X1RYX0xPQ0sodHhyKTsKIAlpZiAoIWRyYnJfZW1wdHkoaWZwLCB0eHItPmJyKSkKLQkJaWdiX21x X3N0YXJ0X2xvY2tlZChpZnAsIHR4ciwgTlVMTCk7CisJCWlnYl9tcV9zdGFydF9sb2NrZWQoaWZw LCB0eHIpOwogCUlHQl9UWF9VTkxPQ0sodHhyKTsKIH0KIApAQCAtMTM5OCw3ICsxMzc2LDcgQEAK IAkJLyogUHJvY2VzcyB0aGUgc3RhY2sgcXVldWUgb25seSBpZiBub3QgZGVwbGV0ZWQgKi8KIAkJ aWYgKCgodHhyLT5xdWV1ZV9zdGF0dXMgJiBJR0JfUVVFVUVfREVQTEVURUQpID09IDApICYmCiAJ CSAgICAhZHJicl9lbXB0eShpZnAsIHR4ci0+YnIpKQotCQkJaWdiX21xX3N0YXJ0X2xvY2tlZChp ZnAsIHR4ciwgTlVMTCk7CisJCQlpZ2JfbXFfc3RhcnRfbG9ja2VkKGlmcCwgdHhyKTsKICNlbHNl CiAJCWlmICghSUZRX0RSVl9JU19FTVBUWSgmaWZwLT5pZl9zbmQpKQogCQkJaWdiX3N0YXJ0X2xv Y2tlZCh0eHIsIGlmcCk7CkBAIC0xNDQ5LDcgKzE0MjcsNyBAQAogCQkJLyogUHJvY2VzcyB0aGUg c3RhY2sgcXVldWUgb25seSBpZiBub3QgZGVwbGV0ZWQgKi8KIAkJCWlmICgoKHR4ci0+cXVldWVf c3RhdHVzICYgSUdCX1FVRVVFX0RFUExFVEVEKSA9PSAwKSAmJgogCQkJICAgICFkcmJyX2VtcHR5 KGlmcCwgdHhyLT5icikpCi0JCQkJaWdiX21xX3N0YXJ0X2xvY2tlZChpZnAsIHR4ciwgTlVMTCk7 CisJCQkJaWdiX21xX3N0YXJ0X2xvY2tlZChpZnAsIHR4cik7CiAjZWxzZQogCQkJaWYgKCFJRlFf RFJWX0lTX0VNUFRZKCZpZnAtPmlmX3NuZCkpCiAJCQkJaWdiX3N0YXJ0X2xvY2tlZCh0eHIsIGlm cCk7CkBAIC0xNTQ5LDcgKzE1MjcsNyBAQAogCQl9IHdoaWxlIChsb29wLS0gJiYgbW9yZSk7CiAj aWYgX19GcmVlQlNEX3ZlcnNpb24gPj0gODAwMDAwCiAJCWlmICghZHJicl9lbXB0eShpZnAsIHR4 ci0+YnIpKQotCQkJaWdiX21xX3N0YXJ0X2xvY2tlZChpZnAsIHR4ciwgTlVMTCk7CisJCQlpZ2Jf bXFfc3RhcnRfbG9ja2VkKGlmcCwgdHhyKTsKICNlbHNlCiAJCWlmICghSUZRX0RSVl9JU19FTVBU WSgmaWZwLT5pZl9zbmQpKQogCQkJaWdiX3N0YXJ0X2xvY2tlZCh0eHIsIGlmcCk7CkBAIC0xNTg2 LDcgKzE1NjQsNyBAQAogCS8qIFByb2Nlc3MgdGhlIHN0YWNrIHF1ZXVlIG9ubHkgaWYgbm90IGRl cGxldGVkICovCiAJaWYgKCgodHhyLT5xdWV1ZV9zdGF0dXMgJiBJR0JfUVVFVUVfREVQTEVURUQp ID09IDApICYmCiAJICAgICFkcmJyX2VtcHR5KGlmcCwgdHhyLT5icikpCi0JCWlnYl9tcV9zdGFy dF9sb2NrZWQoaWZwLCB0eHIsIE5VTEwpOworCQlpZ2JfbXFfc3RhcnRfbG9ja2VkKGlmcCwgdHhy KTsKICNlbHNlCiAJaWYgKCFJRlFfRFJWX0lTX0VNUFRZKCZpZnAtPmlmX3NuZCkpCiAJCWlnYl9z dGFydF9sb2NrZWQodHhyLCBpZnApOwpAQCAtMTY5NSw3ICsxNjczLDYgQEAKIGlnYl9tZWRpYV9z dGF0dXMoc3RydWN0IGlmbmV0ICppZnAsIHN0cnVjdCBpZm1lZGlhcmVxICppZm1yKQogewogCXN0 cnVjdCBhZGFwdGVyICphZGFwdGVyID0gaWZwLT5pZl9zb2Z0YzsKLQl1X2NoYXIgZmliZXJfdHlw ZSA9IElGTV8xMDAwX1NYOwogCiAJSU5JVF9ERUJVR09VVCgiaWdiX21lZGlhX3N0YXR1czogYmVn aW4iKTsKIApAQCAtMTcxMiwyNiArMTY4OSwzMSBAQAogCiAJaWZtci0+aWZtX3N0YXR1cyB8PSBJ Rk1fQUNUSVZFOwogCi0JaWYgKChhZGFwdGVyLT5ody5waHkubWVkaWFfdHlwZSA9PSBlMTAwMF9t ZWRpYV90eXBlX2ZpYmVyKSB8fAotCSAgICAoYWRhcHRlci0+aHcucGh5Lm1lZGlhX3R5cGUgPT0g ZTEwMDBfbWVkaWFfdHlwZV9pbnRlcm5hbF9zZXJkZXMpKQotCQlpZm1yLT5pZm1fYWN0aXZlIHw9 IGZpYmVyX3R5cGUgfCBJRk1fRkRYOwotCWVsc2UgewotCQlzd2l0Y2ggKGFkYXB0ZXItPmxpbmtf c3BlZWQpIHsKLQkJY2FzZSAxMDoKLQkJCWlmbXItPmlmbV9hY3RpdmUgfD0gSUZNXzEwX1Q7Ci0J CQlicmVhazsKLQkJY2FzZSAxMDA6Ci0JCQlpZm1yLT5pZm1fYWN0aXZlIHw9IElGTV8xMDBfVFg7 Ci0JCQlicmVhazsKLQkJY2FzZSAxMDAwOgotCQkJaWZtci0+aWZtX2FjdGl2ZSB8PSBJRk1fMTAw MF9UOwotCQkJYnJlYWs7Ci0JCX0KLQkJaWYgKGFkYXB0ZXItPmxpbmtfZHVwbGV4ID09IEZVTExf RFVQTEVYKQotCQkJaWZtci0+aWZtX2FjdGl2ZSB8PSBJRk1fRkRYOworCXN3aXRjaCAoYWRhcHRl ci0+bGlua19zcGVlZCkgeworCWNhc2UgMTA6CisJCWlmbXItPmlmbV9hY3RpdmUgfD0gSUZNXzEw X1Q7CisJCWJyZWFrOworCWNhc2UgMTAwOgorCQkvKgorCQkqKiBTdXBwb3J0IGZvciAxMDBNYiBT RlAgLSB0aGVzZSBhcmUgRmliZXIgCisJCSoqIGJ1dCB0aGUgbWVkaWEgdHlwZSBhcHBlYXJzIGFz IHNlcmRlcworCQkqLworCQlpZiAoYWRhcHRlci0+aHcucGh5Lm1lZGlhX3R5cGUgPT0KKwkJICAg IGUxMDAwX21lZGlhX3R5cGVfaW50ZXJuYWxfc2VyZGVzKQorCQkJaWZtci0+aWZtX2FjdGl2ZSB8 PSBJRk1fMTAwX0ZYOwogCQllbHNlCi0JCQlpZm1yLT5pZm1fYWN0aXZlIHw9IElGTV9IRFg7CisJ CQlpZm1yLT5pZm1fYWN0aXZlIHw9IElGTV8xMDBfVFg7CisJCWJyZWFrOworCWNhc2UgMTAwMDoK KwkJaWZtci0+aWZtX2FjdGl2ZSB8PSBJRk1fMTAwMF9UOworCQlicmVhazsKIAl9CisKKwlpZiAo YWRhcHRlci0+bGlua19kdXBsZXggPT0gRlVMTF9EVVBMRVgpCisJCWlmbXItPmlmbV9hY3RpdmUg fD0gSUZNX0ZEWDsKKwllbHNlCisJCWlmbXItPmlmbV9hY3RpdmUgfD0gSUZNX0hEWDsKKwogCUlH Ql9DT1JFX1VOTE9DSyhhZGFwdGVyKTsKIH0KIApAQCAtMjE4MCw3ICsyMTYyLDcgQEAKIAlzdHJ1 Y3QgaWZuZXQJCSppZnAgPSBhZGFwdGVyLT5pZnA7CiAJc3RydWN0IHR4X3JpbmcJCSp0eHIgPSBh ZGFwdGVyLT50eF9yaW5nczsKIAlzdHJ1Y3QgaWdiX3F1ZXVlCSpxdWUgPSBhZGFwdGVyLT5xdWV1 ZXM7Ci0JaW50CQkJaHVuZyA9IDAsIGJ1c3kgPSAwOworCWludAkJCWh1bmcgPSAwOwogCiAKIAlJ R0JfQ09SRV9MT0NLX0FTU0VSVChhZGFwdGVyKTsKQEAgLTIxOTAsMjYgKzIxNzIsMTYgQEAKIAog ICAgICAgICAvKgogICAgICAgICAqKiBDaGVjayB0aGUgVFggcXVldWVzIHN0YXR1cwotCSoqCS0g Y2VudHJhbCBsb2NrZWQgaGFuZGxpbmcgb2YgT0FDVElWRQotCSoqCS0gd2F0Y2hkb2cgb25seSBp ZiBhbGwgcXVldWVzIHNob3cgaHVuZwogICAgICAgICAqLwogCWZvciAoaW50IGkgPSAwOyBpIDwg YWRhcHRlci0+bnVtX3F1ZXVlczsgaSsrLCBxdWUrKywgdHhyKyspIHsKIAkJaWYgKCh0eHItPnF1 ZXVlX3N0YXR1cyAmIElHQl9RVUVVRV9IVU5HKSAmJgogCQkgICAgKGFkYXB0ZXItPnBhdXNlX2Zy YW1lcyA9PSAwKSkKIAkJCSsraHVuZzsKLQkJaWYgKHR4ci0+cXVldWVfc3RhdHVzICYgSUdCX1FV RVVFX0RFUExFVEVEKQotCQkJKytidXN5OwogCQlpZiAoKHR4ci0+cXVldWVfc3RhdHVzICYgSUdC X1FVRVVFX0lETEUpID09IDApCiAJCQl0YXNrcXVldWVfZW5xdWV1ZShxdWUtPnRxLCAmcXVlLT5x dWVfdGFzayk7CiAJfQogCWlmIChodW5nID09IGFkYXB0ZXItPm51bV9xdWV1ZXMpCiAJCWdvdG8g dGltZW91dDsKLQlpZiAoYnVzeSA9PSBhZGFwdGVyLT5udW1fcXVldWVzKQotCQlpZnAtPmlmX2Ry dl9mbGFncyB8PSBJRkZfRFJWX09BQ1RJVkU7Ci0JZWxzZSBpZiAoKGlmcC0+aWZfZHJ2X2ZsYWdz ICYgSUZGX0RSVl9PQUNUSVZFKSAmJgotCSAgICAoYnVzeSA8IGFkYXB0ZXItPm51bV9xdWV1ZXMp KQotCQlpZnAtPmlmX2Rydl9mbGFncyAmPSB+SUZGX0RSVl9PQUNUSVZFOwotCiAJYWRhcHRlci0+ cGF1c2VfZnJhbWVzID0gMDsKIAljYWxsb3V0X3Jlc2V0KCZhZGFwdGVyLT50aW1lciwgaHosIGln Yl9sb2NhbF90aW1lciwgYWRhcHRlcik7CiAjaWZuZGVmIERFVklDRV9QT0xMSU5HCkBAIC0yMjM0 LDExICsyMjA2LDEzIEBACiBzdGF0aWMgdm9pZAogaWdiX3VwZGF0ZV9saW5rX3N0YXR1cyhzdHJ1 Y3QgYWRhcHRlciAqYWRhcHRlcikKIHsKLQlzdHJ1Y3QgZTEwMDBfaHcgKmh3ID0gJmFkYXB0ZXIt Pmh3OwotCXN0cnVjdCBpZm5ldCAqaWZwID0gYWRhcHRlci0+aWZwOwotCWRldmljZV90IGRldiA9 IGFkYXB0ZXItPmRldjsKLQlzdHJ1Y3QgdHhfcmluZyAqdHhyID0gYWRhcHRlci0+dHhfcmluZ3M7 Ci0JdTMyIGxpbmtfY2hlY2ssIHRoc3RhdCwgY3RybDsKKwlzdHJ1Y3QgZTEwMDBfaHcJCSpodyA9 ICZhZGFwdGVyLT5odzsKKwlzdHJ1Y3QgZTEwMDBfZmNfaW5mbwkqZmMgPSAmaHctPmZjOworCXN0 cnVjdCBpZm5ldAkJKmlmcCA9IGFkYXB0ZXItPmlmcDsKKwlkZXZpY2VfdAkJZGV2ID0gYWRhcHRl ci0+ZGV2OworCXN0cnVjdCB0eF9yaW5nCQkqdHhyID0gYWRhcHRlci0+dHhfcmluZ3M7CisJdTMy CQkJbGlua19jaGVjaywgdGhzdGF0LCBjdHJsOworCWNoYXIJCQkqZmxvd2N0bCA9IE5VTEw7CiAK IAlsaW5rX2NoZWNrID0gdGhzdGF0ID0gY3RybCA9IDA7CiAKQEAgLTIyNzYsMTUgKzIyNTAsMzMg QEAKIAkJY3RybCA9IEUxMDAwX1JFQURfUkVHKGh3LCBFMTAwMF9DVFJMX0VYVCk7CiAJfQogCisJ LyogR2V0IHRoZSBmbG93IGNvbnRyb2wgZm9yIGRpc3BsYXkgKi8KKwlzd2l0Y2ggKGZjLT5jdXJy ZW50X21vZGUpIHsKKwljYXNlIGUxMDAwX2ZjX3J4X3BhdXNlOgorCQlmbG93Y3RsID0gIlJYIjsK KwkJYnJlYWs7CQorCWNhc2UgZTEwMDBfZmNfdHhfcGF1c2U6CisJCWZsb3djdGwgPSAiVFgiOwor CQlicmVhazsJCisJY2FzZSBlMTAwMF9mY19mdWxsOgorCQlmbG93Y3RsID0gIkZ1bGwiOworCQli cmVhazsJCisJY2FzZSBlMTAwMF9mY19ub25lOgorCWRlZmF1bHQ6CisJCWZsb3djdGwgPSAiTm9u ZSI7CisJCWJyZWFrOwkKKwl9CisKIAkvKiBOb3cgd2UgY2hlY2sgaWYgYSB0cmFuc2l0aW9uIGhh cyBoYXBwZW5lZCAqLwogCWlmIChsaW5rX2NoZWNrICYmIChhZGFwdGVyLT5saW5rX2FjdGl2ZSA9 PSAwKSkgewogCQllMTAwMF9nZXRfc3BlZWRfYW5kX2R1cGxleCgmYWRhcHRlci0+aHcsIAogCQkg ICAgJmFkYXB0ZXItPmxpbmtfc3BlZWQsICZhZGFwdGVyLT5saW5rX2R1cGxleCk7CiAJCWlmIChi b290dmVyYm9zZSkKLQkJCWRldmljZV9wcmludGYoZGV2LCAiTGluayBpcyB1cCAlZCBNYnBzICVz XG4iLAorCQkJZGV2aWNlX3ByaW50ZihkZXYsICJMaW5rIGlzIHVwICVkIE1icHMgJXMsIgorCQkJ ICAgICIgRmxvdyBDb250cm9sOiAlc1xuIiwKIAkJCSAgICBhZGFwdGVyLT5saW5rX3NwZWVkLAog CQkJICAgICgoYWRhcHRlci0+bGlua19kdXBsZXggPT0gRlVMTF9EVVBMRVgpID8KLQkJCSAgICAi RnVsbCBEdXBsZXgiIDogIkhhbGYgRHVwbGV4IikpOworCQkJICAgICJGdWxsIER1cGxleCIgOiAi SGFsZiBEdXBsZXgiKSwgZmxvd2N0bCk7CiAJCWFkYXB0ZXItPmxpbmtfYWN0aXZlID0gMTsKIAkJ aWZwLT5pZl9iYXVkcmF0ZSA9IGFkYXB0ZXItPmxpbmtfc3BlZWQgKiAxMDAwMDAwOwogCQlpZiAo KGN0cmwgJiBFMTAwMF9DVFJMX0VYVF9MSU5LX01PREVfR01JSSkgJiYK --047d7b5daf50ddb67504cf069572-- From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 23:07:39 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 97B616E2 for ; Wed, 21 Nov 2012 23:07:39 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: from mail-qa0-f54.google.com (mail-qa0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id E69E18FC08 for ; Wed, 21 Nov 2012 23:07:38 +0000 (UTC) Received: by mail-qa0-f54.google.com with SMTP id g24so470854qab.13 for ; Wed, 21 Nov 2012 15:07:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=VqbWXdFKcC1epc0gcUCQgOTpzWSD4q9VrLasK0PO9/Y=; b=Prvk3U6YIj32SluzemSjSljaLTjZU4jEJFO/t2MrHFW29CPhiaK8FiY8BLTYKXln5c bB8XHEpfvAvy/VtSIEVOuP3/d4gbbGr3H9hv/zNSRTh+rhUYdzDU5vObZfUcLCdDQVT2 GxXataCupfDSFmQkU9G4KyGTszDgYMU96lC5Xv1GEIZP7DOqpcbTxQmyEnqu3ss5dTR6 gjp/VoIR415fSArG3V9+5mYUs+wH5yl4J5WlXbfrw7gruJULE9g+fYZOVPWOGLUFZ7sG dluw/ALsHGL8tjMORf2yVU7JXdzka4GbPW+q2r/2krMw83ywZN/S17ig7rIVKffgPrCR o4vA== MIME-Version: 1.0 Received: by 10.49.30.65 with SMTP id q1mr20014991qeh.46.1353539257854; Wed, 21 Nov 2012 15:07:37 -0800 (PST) Received: by 10.229.78.96 with HTTP; Wed, 21 Nov 2012 15:07:37 -0800 (PST) In-Reply-To: References: Date: Thu, 22 Nov 2012 02:07:37 +0300 Message-ID: Subject: Re: [RFC] Prune net.inet6.ip6.rr_prune? From: Sergey Kandaurov To: Garrett Cooper Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 23:07:39 -0000 On 21 November 2012 22:39, Garrett Cooper wrote: > While going through the tree trying to document all of our > net.inet6 sysctls, I noticed that net.inet6.ip6.rr_prune is defined, > but not actually used anywhere in the stack: > > netinet6/ip6_var.h:VNET_DECLARE(int, ip6_rr_prune); /* router > renumbering prefix > netinet6/ip6_var.h:#define V_ip6_rr_prune VNET(ip6_rr_prune) > netinet6/in6_proto.c:VNET_DEFINE(int, ip6_rr_prune) = 5; /* router > renumbering prefix > netinet6/in6_proto.c:SYSCTL_VNET_INT(_net_inet6_ip6, IPV6CTL_RR_PRUNE, > rr_prune, CTLFLAG_RW, > netinet6/in6_proto.c: &VNET_NAME(ip6_rr_prune), 0, > > The knob was declared in r181803 and shuffled around a few times, > but isn't in use anywhere (either then or now). > Should I send out a PR to remove it (or am I missing some context)? I believe this knob became unused with invalidation of the prefix manipulation mechanism (including prefix or router renumbering, rfc2894) at KAME about 11 years ago. It was intended to schedule in6_rr_timer() callout every ip6_rr_prune seconds to check for expired prefixes and delete the associated addresses from interface. Last bits of old ipv6 prefix management stuff cleaned up in r231229. -- wbr, pluknet From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 23:12:34 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id F0FC9A6E for ; Wed, 21 Nov 2012 23:12:34 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id AF3318FC14 for ; Wed, 21 Nov 2012 23:12:34 +0000 (UTC) Received: by mail-ob0-f182.google.com with SMTP id 16so9756977obc.13 for ; Wed, 21 Nov 2012 15:12:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Sfk/jf5VOph62B2ERTl9MLYjezOs92hTBmrMYFicQCE=; b=MD8q/056r+RWwjaM3ONjke2aHXqE0MIEkZop2byL62kAsJ7zqYUuAzlwADmUpkzT4p Y/VJwEpZQMNafdSPO6CdX9NW1ompJhu0w3BA3FKd/x4WVI43S4+jgxhIF46nr44/sKij b6FcQuZtTbzAognmFdQD6CmeYkTk70Y/xtp5CjfKdtvpMB704o31r6qADs33dVs9cAlD Y5fvQy/h6WvcfQ6l0KkEyZA7W8cxJBas5squdEE+YcLNqA83kHgQNfnHn5lN4UX81h7a EJbdi7uvaoOHtD7ncUCv9Aje+HFifOukAviOR0dUFYl6oCd0VbnU0Ed8ZPmHhXZ1ZiV3 wD7A== MIME-Version: 1.0 Received: by 10.182.95.205 with SMTP id dm13mr17914629obb.9.1353539553733; Wed, 21 Nov 2012 15:12:33 -0800 (PST) Received: by 10.76.143.33 with HTTP; Wed, 21 Nov 2012 15:12:33 -0800 (PST) In-Reply-To: References: Date: Wed, 21 Nov 2012 15:12:33 -0800 Message-ID: Subject: Re: [RFC] Prune net.inet6.ip6.rr_prune? From: Garrett Cooper To: Sergey Kandaurov Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 23:12:35 -0000 On Wed, Nov 21, 2012 at 3:07 PM, Sergey Kandaurov wrote: > On 21 November 2012 22:39, Garrett Cooper wrote: >> While going through the tree trying to document all of our >> net.inet6 sysctls, I noticed that net.inet6.ip6.rr_prune is defined, >> but not actually used anywhere in the stack: >> >> netinet6/ip6_var.h:VNET_DECLARE(int, ip6_rr_prune); /* router >> renumbering prefix >> netinet6/ip6_var.h:#define V_ip6_rr_prune VNET(ip6_rr_prune) >> netinet6/in6_proto.c:VNET_DEFINE(int, ip6_rr_prune) = 5; /* router >> renumbering prefix >> netinet6/in6_proto.c:SYSCTL_VNET_INT(_net_inet6_ip6, IPV6CTL_RR_PRUNE, >> rr_prune, CTLFLAG_RW, >> netinet6/in6_proto.c: &VNET_NAME(ip6_rr_prune), 0, >> >> The knob was declared in r181803 and shuffled around a few times, >> but isn't in use anywhere (either then or now). >> Should I send out a PR to remove it (or am I missing some context)? > > I believe this knob became unused with invalidation of the prefix > manipulation mechanism (including prefix or router renumbering, rfc2894) > at KAME about 11 years ago. It was intended to schedule in6_rr_timer() > callout every ip6_rr_prune seconds to check for expired prefixes and > delete the associated addresses from interface. > Last bits of old ipv6 prefix management stuff cleaned up in r231229. Interesting. If I don't get any negative feedback, I'll roll it into the patch I was going to submit better documenting the sysctls that I was going to submit via a PR. Thanks! -Garrett From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 23:23:05 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C4FB9C91 for ; Wed, 21 Nov 2012 23:23:05 +0000 (UTC) (envelope-from oppermann@networx.ch) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id 547B78FC13 for ; Wed, 21 Nov 2012 23:23:04 +0000 (UTC) Received: (qmail 4025 invoked from network); 22 Nov 2012 00:55:44 -0000 Received: from c00l3r.networx.ch (HELO [127.0.0.1]) ([62.48.2.2]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 22 Nov 2012 00:55:44 -0000 Message-ID: <50AD6252.6030806@networx.ch> Date: Thu, 22 Nov 2012 00:22:58 +0100 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20121026 Thunderbird/16.0.2 MIME-Version: 1.0 To: Marc Peters Subject: Re: Low Bandwidth on intercontinental connections References: <50ACF62C.8000408@mpeters.org> In-Reply-To: <50ACF62C.8000408@mpeters.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 23:23:06 -0000 On 21.11.2012 16:41, Marc Peters wrote: > Hi list, > -snip- > Doing some googling brought up a lot of tuning hints, but nothing worked > for us. We tweaked some sysctls: > > kern.ipc.maxsockbuf=16777216 > net.inet.tcp.sendbuf_max=16777216 > net.inet.tcp.recvbuf_max=16777216 > net.inet.tcp.sendbuf_inc=16384 > net.inet.tcp.recvbuf_inc=524288 > net.inet.tcp.hostcache.expire=1 This doesn't help. Please revert it to the default values. -- Andre From owner-freebsd-net@FreeBSD.ORG Thu Nov 22 00:31:49 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id F20D8D42; Thu, 22 Nov 2012 00:31:48 +0000 (UTC) (envelope-from rpaulo@FreeBSD.org) Received: from felyko.com (felyko.com [IPv6:2607:f2f8:a528::3:1337:ca7]) by mx1.freebsd.org (Postfix) with ESMTP id CF2E48FC0C; Thu, 22 Nov 2012 00:31:48 +0000 (UTC) Received: from [IPv6:2601:9:4d00:85:ac6f:ff48:5b36:8dc4] (unknown [IPv6:2601:9:4d00:85:ac6f:ff48:5b36:8dc4]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by felyko.com (Postfix) with ESMTPSA id 4A9C13981E; Wed, 21 Nov 2012 16:31:48 -0800 (PST) From: Rui Paulo Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: LOR in rtsock/ifnet Date: Wed, 21 Nov 2012 16:31:47 -0800 Message-Id: <7C5421B8-9D32-4771-B453-151D9D07A214@FreeBSD.org> To: "" Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) X-Mailer: Apple Mail (2.1499) Cc: Hiroki Sato , "Andrey V. Elsukov" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2012 00:31:49 -0000 I just started seeing this on r243286. lock order reversal: 1st 0xfffffe0001b40400 if_addr_lock (if_addr_lock) @ = /usr/home/rpaulo/freebsd/head/sys/net/rtsock.c:1818 2nd 0xffffffff80c693f8 ifnet_rw (ifnet_rw) @ = /usr/home/rpaulo/freebsd/head/sys/net/if.c:241 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b kdb_backtrace() at kdb_backtrace+0x39 witness_checkorder() at witness_checkorder+0xc37 __rw_rlock() at __rw_rlock+0x8c ifnet_byindex() at ifnet_byindex+0x22 sa6_recoverscope() at sa6_recoverscope+0x7b rt_msg2() at rt_msg2+0x1a2 sysctl_rtsock() at sysctl_rtsock+0x68c sysctl_root() at sysctl_root+0x1d7 userland_sysctl() at userland_sysctl+0x192 sys___sysctl() at sys___sysctl+0x74 amd64_syscall() at amd64_syscall+0x265 Xfast_syscall() at Xfast_syscall+0xfb --- syscall (202, FreeBSD ELF64, sys___sysctl), rip =3D 0x8011813ea, rsp = =3D 0x7fffffffd408, rbp =3D 0x7fffffffd440 --- Regards, -- Rui Paulo From owner-freebsd-net@FreeBSD.ORG Thu Nov 22 01:41:17 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id DD7845BE for ; Thu, 22 Nov 2012 01:41:17 +0000 (UTC) (envelope-from chrish@ultimateDNS.NET) Received: from udns.ultimateDNS.NET (24-113-118-203.wavecable.com [24.113.118.203]) by mx1.freebsd.org (Postfix) with ESMTP id B2DEB8FC08 for ; Thu, 22 Nov 2012 01:41:16 +0000 (UTC) Received: from ultimatedns.net (localhost [IPv6:::1]) by udns.ultimateDNS.NET (8.14.5/8.14.5) with SMTP id qAM1L1Ux003750 for ; Wed, 21 Nov 2012 17:21:07 -0800 (PST) (envelope-from chrish@ultimateDNS.NET) Received: from 24.113.118.203 (auth. user chrish@localhost) by ultimatedns.net with HTTP; Wed, 21 Nov 2012 19:20:56 -0600 To: freebsd-net@freebsd.org Subject: USB ethernet support - can't complete CD install Date: Wed, 21 Nov 2012 19:20:56 -0600 X-Mailer: UDNS Messaging System/0.9.5 (On: ultimatedns.net) Message-ID: From: Chris Bounce-To: Chris Errors-To: Chris MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Chris List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2012 01:41:17 -0000 Greetings, I just attempted a CD install of RELENG_8 on an AMD box. The install went nearly as expected, but upon rebooting into the new install. I ran into trouble -- cdce0 faking MAC address. CRAP! this'll never work. My ISP leases the DHCP by MAC address. Now I won't be able to make an attempt to gain internet access for another ~24hrs. I didn't think I'd need to parse the log during install to copy the MAC address for later use. As I see I'll need to maintain a _single_ "faked" MAC address. Can anyone assist me in effectively utilizing it? relevent info: nVidia nForce2 USB2.0 controller on ehci0 -- simply put; a USB cable attached from the USB port on the AMD box, to the USB port on the modem. What is the proper usage in rc.conf(5)? I'm attempting the following: ifconfig_ue0="ether xx:xx:xx:xx:xx:xx DHCP" Thank you for all your time, and consideration. --Chris From owner-freebsd-net@FreeBSD.ORG Thu Nov 22 07:09:04 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9B3C3533; Thu, 22 Nov 2012 07:09:04 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wi0-f174.google.com (mail-wi0-f174.google.com [209.85.212.174]) by mx1.freebsd.org (Postfix) with ESMTP id C19A58FC14; Thu, 22 Nov 2012 07:09:03 +0000 (UTC) Received: by mail-wi0-f174.google.com with SMTP id hm9so416016wib.13 for ; Wed, 21 Nov 2012 23:09:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=7v/wq/wKjRRTrYI3ngcbQd9AUySHOV5XXJOiIb4zrzo=; b=cWzeXWoGh1Ux4/CeiWH6FS1kXaRZrMfcdoIicRTDjZkhFNdgnBeef8UjEe28MuWtXo i8WClZM3V4dpII0f0WfrgcwG3r+GloKCq8DXc4xZM26DdlVDrRwco42W8P4JE67IX4fw SdPh4rJqAO0+3HQpQuRQWH6qgFTClwgMXkrZBnPMjP/KuUodFzE6TwCpr3Lc0oNCIpwY SDJZI6Ip0n/xxXlKlv/8QzrSnlRMGJ+wNYR9luvvmYh0ygLTUA56wLbpxX9CMBHtL3mz KG2z00Ik5hVX96fOEoki3/0QupLZgTtYCM/sAAYM9jzHkZhkSJBJcwxH0LMGoy4oDYUY aJVg== MIME-Version: 1.0 Received: by 10.180.99.1 with SMTP id em1mr2789824wib.17.1353568142474; Wed, 21 Nov 2012 23:09:02 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.217.57.9 with HTTP; Wed, 21 Nov 2012 23:09:02 -0800 (PST) In-Reply-To: <7C5421B8-9D32-4771-B453-151D9D07A214@FreeBSD.org> References: <7C5421B8-9D32-4771-B453-151D9D07A214@FreeBSD.org> Date: Wed, 21 Nov 2012 23:09:02 -0800 X-Google-Sender-Auth: 6cKJ4ABZvvXO6WZNKiOJpuoMY9w Message-ID: Subject: Re: LOR in rtsock/ifnet From: Adrian Chadd To: Rui Paulo Content-Type: text/plain; charset=ISO-8859-1 Cc: "" , "Andrey V. Elsukov" , Hiroki Sato X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2012 07:09:04 -0000 I've started seeing this too. We're both running nat/bridge gateways of some sort. Adrian On 21 November 2012 16:31, Rui Paulo wrote: > I just started seeing this on r243286. > > lock order reversal: > 1st 0xfffffe0001b40400 if_addr_lock (if_addr_lock) @ /usr/home/rpaulo/freebsd/head/sys/net/rtsock.c:1818 > 2nd 0xffffffff80c693f8 ifnet_rw (ifnet_rw) @ /usr/home/rpaulo/freebsd/head/sys/net/if.c:241 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b > kdb_backtrace() at kdb_backtrace+0x39 > witness_checkorder() at witness_checkorder+0xc37 > __rw_rlock() at __rw_rlock+0x8c > ifnet_byindex() at ifnet_byindex+0x22 > sa6_recoverscope() at sa6_recoverscope+0x7b > rt_msg2() at rt_msg2+0x1a2 > sysctl_rtsock() at sysctl_rtsock+0x68c > sysctl_root() at sysctl_root+0x1d7 > userland_sysctl() at userland_sysctl+0x192 > sys___sysctl() at sys___sysctl+0x74 > amd64_syscall() at amd64_syscall+0x265 > Xfast_syscall() at Xfast_syscall+0xfb > --- syscall (202, FreeBSD ELF64, sys___sysctl), rip = 0x8011813ea, rsp = 0x7fffffffd408, rbp = 0x7fffffffd440 --- > > Regards, > -- > Rui Paulo > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Thu Nov 22 08:03:18 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 17068D88 for ; Thu, 22 Nov 2012 08:03:18 +0000 (UTC) (envelope-from marc@mpeters.org) Received: from mail.mpeters.org (mail.mpeters.org [78.46.104.142]) by mx1.freebsd.org (Postfix) with ESMTP id BBC5F8FC08 for ; Thu, 22 Nov 2012 08:03:17 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mail.mpeters.org (Postfix) with ESMTP id C261813200F for ; Thu, 22 Nov 2012 09:03:16 +0100 (CET) X-Virus-Scanned: amavisd-new at mpeters.org Received: from mail.mpeters.org ([127.0.0.1]) by localhost (mail.mpeters.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2US10lNTEG-g for ; Thu, 22 Nov 2012 09:03:15 +0100 (CET) Received: from [192.168.0.204] (unknown [62.159.86.18]) by mail.mpeters.org (Postfix) with ESMTPSA id D768213200E for ; Thu, 22 Nov 2012 09:03:15 +0100 (CET) Message-ID: <50ADDC43.1030004@mpeters.org> Date: Thu, 22 Nov 2012 09:03:15 +0100 From: Marc Peters User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:16.0) Gecko/20121104 Thunderbird/16.0.1 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: Re: Low Bandwidth on intercontinental connections References: <50ACF62C.8000408@mpeters.org> <50AD6252.6030806@networx.ch> In-Reply-To: <50AD6252.6030806@networx.ch> X-Enigmail-Version: 1.5a1pre Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2012 08:03:18 -0000 On 11/22/2012 12:22 AM, Andre Oppermann wrote: > On 21.11.2012 16:41, Marc Peters wrote: >> Hi list, >> > -snip- >> Doing some googling brought up a lot of tuning hints, but nothing worked >> for us. We tweaked some sysctls: >> >> kern.ipc.maxsockbuf=16777216 >> net.inet.tcp.sendbuf_max=16777216 >> net.inet.tcp.recvbuf_max=16777216 >> net.inet.tcp.sendbuf_inc=16384 >> net.inet.tcp.recvbuf_inc=524288 >> net.inet.tcp.hostcache.expire=1 > > This doesn't help. Please revert it to the default values. > Yeah, as i saw no change, i reverted it already. From owner-freebsd-net@FreeBSD.ORG Thu Nov 22 08:44:23 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6710911E for ; Thu, 22 Nov 2012 08:44:23 +0000 (UTC) (envelope-from marc@mpeters.org) Received: from mail.mpeters.org (mail.mpeters.org [78.46.104.142]) by mx1.freebsd.org (Postfix) with ESMTP id 187988FC08 for ; Thu, 22 Nov 2012 08:44:22 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mail.mpeters.org (Postfix) with ESMTP id 0154713200F for ; Thu, 22 Nov 2012 09:44:22 +0100 (CET) X-Virus-Scanned: amavisd-new at mpeters.org Received: from mail.mpeters.org ([127.0.0.1]) by localhost (mail.mpeters.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1v6ZIf8j7nR2 for ; Thu, 22 Nov 2012 09:44:20 +0100 (CET) Received: from [192.168.0.204] (unknown [62.159.86.18]) by mail.mpeters.org (Postfix) with ESMTPSA id 8C4DC13200E for ; Thu, 22 Nov 2012 09:44:20 +0100 (CET) Message-ID: <50ADE5E4.9090708@mpeters.org> Date: Thu, 22 Nov 2012 09:44:20 +0100 From: Marc Peters User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:16.0) Gecko/20121104 Thunderbird/16.0.1 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: Re: Low Bandwidth on intercontinental connections References: <50ACF62C.8000408@mpeters.org> <50ad087d.1892cc0a.2cce.3bf2@mx.google.com> <50AD1012.7020209@mpeters.org> <50AD14F8.8050001@xip.at> In-Reply-To: <50AD14F8.8050001@xip.at> X-Enigmail-Version: 1.5a1pre Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2012 08:44:23 -0000 On 11/21/2012 06:52 PM, Ingo Flaschberger wrote: > Am 21.11.2012 18:32, schrieb Marc Peters: >> Hi Ben, >> >> i don't think this is memory related, too. We used plain CLI scp ot ftp >> from base, both times. >> >> Here is the requested data: >> >> Linux ---> FreeBSD: >> >> root@linux:~# scp jdk-6u33-linux-x64.bin 172.16.3.10: >> Password: >> jdk-6u33-linux-x64.bin 89% 61MB 59.0KB/s >> >> FreeBSD ---> Linux: >> >> [root@freebsd ~]# scp test.tgz 172.16.4.50: >> Password: >> test.tgz 100% 59MB 1.1MB/s 00:55 >> [root@freebsd ~]# >> >> From BSD to Linux is not as fast as L <--> L. >> >> I don't think, this is network related in some sort. > > hm - sounds like a duplex problem? duplex is fine (as showed by ifconfig in the first post). The switches also show full duplex on a Gig link for this. > > *) whats the distance between Linux and FreeBSD box? some 10k kilometers. One of our DCs is in Germany, the other in the US. > *) check network counters: > linux: ifconfig > FreeBSD: netstat -nia > look for errors no errors and no collisions on any system > check switches between (or as far as possible) for full duplex, also > FreeBSD (ifconfig) the internal switches are working properly. > *) check and compare tcpdump for the FreeBSD hosts on the receiver side, it showed a lot of window size changes and from time to time a lot of duplicate ACKs. i will file a PR (as Adrian asked) and see to get a matching tcpdump and SIFTR output. > > Kind regards, > Ingo Flaschberger > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Thu Nov 22 09:07:44 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E6ED3760 for ; Thu, 22 Nov 2012 09:07:44 +0000 (UTC) (envelope-from marc@mpeters.org) Received: from mail.mpeters.org (mail.mpeters.org [78.46.104.142]) by mx1.freebsd.org (Postfix) with ESMTP id 66F398FC15 for ; Thu, 22 Nov 2012 09:07:44 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mail.mpeters.org (Postfix) with ESMTP id 185F113200F for ; Thu, 22 Nov 2012 10:07:43 +0100 (CET) X-Virus-Scanned: amavisd-new at mpeters.org Received: from mail.mpeters.org ([127.0.0.1]) by localhost (mail.mpeters.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PvwONG0F50oH for ; Thu, 22 Nov 2012 10:07:41 +0100 (CET) Received: from [192.168.0.204] (unknown [62.159.86.18]) by mail.mpeters.org (Postfix) with ESMTPSA id E3A9513200E for ; Thu, 22 Nov 2012 10:07:40 +0100 (CET) Message-ID: <50ADEB5C.6060401@mpeters.org> Date: Thu, 22 Nov 2012 10:07:40 +0100 From: Marc Peters User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:16.0) Gecko/20121104 Thunderbird/16.0.1 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: Re: Low Bandwidth on intercontinental connections References: <50ACF62C.8000408@mpeters.org> <50AD2251.3040904@freebsd.org> In-Reply-To: <50AD2251.3040904@freebsd.org> X-Enigmail-Version: 1.5a1pre Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2012 09:07:45 -0000 On 11/21/2012 07:49 PM, Julian Elischer wrote: > On 11/21/12 7:41 AM, Marc Peters wrote: >> Hi list, >> >> we are experiencing low throughput on interncontinental connections with >> our FreeBSD Servers. We made several tests and are wondering, why this >> would be. The first tests were on an IPSEC VPN between our datacenter in >> DE and Santa Clara, CA. We are connected with two gigabit uplinks in >> each DC. Pushing data by scp between our FreeBSD servers takes ages. >> Starting with several MB/s it drops to 60-70KB/s: >> >> [root@freebsd ~]# ls -alh test.tgz >> -rw-r----- 1 root wheel 58M Oct 5 2010 test.tgz >> [root@freebsd ~]# scp test.tgz 172.16.3.10:. >> Password: >> test.tgz 28% 17MB 75.3KB/s 09:32 ETA >> >> >> For comparision, we did a similiar test with Linux, which didn't show >> this behaviour: >> >> root@linux:~# scp jdk-6u33-linux-x64.bin 172.16.4.50: >> root@172.16.4.50's password: >> jdk-6u33-linux-x64.bin 100% >> 69MB 3.4MB/s 00:20 >> root@linux:~# >> >> >> Otherwise, the servers are really fast, when copying data to a machine >> nearby: >> >> [root@freebsd ~]# ls -alh test >> -rw-r--r-- 1 root wheel 1G Nov 21 13:43 test >> [root@freebsd ~]# scp test 172.16.3.11: >> Password: >> test 100% 1000MB 38.5MB/s 00:26 >> >> >> Intercontinental ftp downloads are the same: >> >> [root@freebsd ~]# fetch >> ftp://ftp1.us.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-bootonly.iso >> >> FreeBSD-9.1-RC3-amd64-bootonly.iso 100% of 146 MB 46 MBps >> >> [root@freebsd ~]# fetch >> ftp://ftp1.us.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso >> >> >> FreeBSD-9.1-RC3-amd64-disc1.iso 100% of 685 MB 36 MBps 00m00s >> >> [root@freebsd ~]# fetch >> ftp://ftp1.de.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso >> >> FreeBSD-9.1-RC3-amd64-disc1.iso 0% of 685 MB 13 kBps 14h49m^C >> >> >> Linux: >> >> root@linux:~# wget >> ftp://ftp1.de.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso >> >> --2012-11-21 15:07:57-- >> ftp://ftp1.de.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso >> >> => `FreeBSD-9.1-RC3-amd64-disc1.iso' >> Resolving ftp1.de.freebsd.org... 137.226.34.42 >> Connecting to ftp1.de.freebsd.org|137.226.34.42|:21... connected. >> Logging in as anonymous ... Logged in! >> ==> SYST ... done. ==> PWD ... done. >> ==> TYPE I ... done. ==> CWD (1) >> /pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1 ... done. >> ==> SIZE FreeBSD-9.1-RC3-amd64-disc1.iso ... 718800896 >> ==> PASV ... done. ==> RETR FreeBSD-9.1-RC3-amd64-disc1.iso ... done. >> Length: 718800896 (686M) (unauthoritative) >> >> 100%[=====================================================================>] >> >> 718,800,896 19.1M/s in 61s >> >> 2012-11-21 15:09:01 (11.2 MB/s) - `FreeBSD-9.1-RC3-amd64-disc1.iso' >> saved [718800896] >> >> >> Doing some googling brought up a lot of tuning hints, but nothing worked >> for us. We tweaked some sysctls: >> >> kern.ipc.maxsockbuf=16777216 >> net.inet.tcp.sendbuf_max=16777216 >> net.inet.tcp.recvbuf_max=16777216 >> net.inet.tcp.sendbuf_inc=16384 >> net.inet.tcp.recvbuf_inc=524288 >> net.inet.tcp.hostcache.expire=1 >> >> but to no good. Disabling MSI and TSO4 for the card didn't change >> anything, too. >> >> The machines are all HP DL360G7 with bce cards (find dmesg, ifconfig and >> pciconf -lvc at the end of this mail). >> >> Can someone hit me with a cluestick to get the BSDs on speed? > you really do need to get a tcpdump of the transfer under slow > conditions and a SIFTR output to match. > What is the ping time between the hosts. The ping times are okay, as for the distance (DE to US): ping 172.16.3.10 PING 172.16.3.10 (172.16.3.10) 56(84) bytes of data. 64 bytes from 172.16.3.10: icmp_req=1 ttl=62 time=155 ms 64 bytes from 172.16.3.10: icmp_req=2 ttl=62 time=156 ms 64 bytes from 172.16.3.10: icmp_req=3 ttl=62 time=155 ms 64 bytes from 172.16.3.10: icmp_req=4 ttl=62 time=156 ms 64 bytes from 172.16.3.10: icmp_req=5 ttl=62 time=156 ms > that will allow you to work out > how large a window you should have. What exactly do you mean by that? The MTU, we should use on the hosts? >> >> marc >> >> PS: The version is FreeBSD-RC2 amd64, because we need the patch for >> process migration on the CPUs which didn't make it 9.0 or an errata, as >> we were the only ones, hitting this bug (so kib@ said). >> > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Thu Nov 22 10:44:09 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6910E1CE for ; Thu, 22 Nov 2012 10:44:09 +0000 (UTC) (envelope-from marc@mpeters.org) Received: from mail.mpeters.org (mail.mpeters.org [78.46.104.142]) by mx1.freebsd.org (Postfix) with ESMTP id 1DFBA8FC08 for ; Thu, 22 Nov 2012 10:44:08 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mail.mpeters.org (Postfix) with ESMTP id 1391013200F; Thu, 22 Nov 2012 11:44:08 +0100 (CET) X-Virus-Scanned: amavisd-new at mpeters.org Received: from mail.mpeters.org ([127.0.0.1]) by localhost (mail.mpeters.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fWbwBT4N58Mt; Thu, 22 Nov 2012 11:44:07 +0100 (CET) Received: from [192.168.0.204] (unknown [62.159.86.18]) by mail.mpeters.org (Postfix) with ESMTPSA id E18CC13200E; Thu, 22 Nov 2012 11:44:06 +0100 (CET) Message-ID: <50AE01F6.8030808@mpeters.org> Date: Thu, 22 Nov 2012 11:44:06 +0100 From: Marc Peters User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:16.0) Gecko/20121104 Thunderbird/16.0.1 MIME-Version: 1.0 To: wishmaster Subject: Re: Low Bandwidth on intercontinental connections References: <50ADEB5C.6060401@mpeters.org> <50ACF62C.8000408@mpeters.org> <50AD2251.3040904@freebsd.org> <11556.1353578231.3984630695456866304@ffe8.ukr.net> In-Reply-To: <11556.1353578231.3984630695456866304@ffe8.ukr.net> X-Enigmail-Version: 1.5a1pre Content-Type: text/plain; charset=windows-1251 Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2012 10:44:09 -0000 On 11/22/2012 10:57 AM, wishmaster wrote: > >> The ping times are okay, as for the distance (DE to US): >> >> ping 172.16.3.10 >> PING 172.16.3.10 (172.16.3.10) 56(84) bytes of data. >> 64 bytes from 172.16.3.10: icmp_req=1 ttl=62 time=155 ms >> 64 bytes from 172.16.3.10: icmp_req=2 ttl=62 time=156 ms >> 64 bytes from 172.16.3.10: icmp_req=3 ttl=62 time=155 ms >> 64 bytes from 172.16.3.10: icmp_req=4 ttl=62 time=156 ms >> 64 bytes from 172.16.3.10: icmp_req=5 ttl=62 time=156 ms >> > Can you ping host not in IPSec channel? Ping external public IP second point. > The times are the same: root@linux:~# ping 8.35.XXX.XXX PING 8.35.XXX.XXX (8.35.XXX.XXX) 56(84) bytes of data. 64 bytes from 8.35.XXX.XXX: icmp_req=1 ttl=47 time=163 ms 64 bytes from 8.35.XXX.XXX: icmp_req=2 ttl=47 time=159 ms 64 bytes from 8.35.XXX.XXX: icmp_req=3 ttl=47 time=149 ms 64 bytes from 8.35.XXX.XXX: icmp_req=4 ttl=47 time=154 ms 64 bytes from 8.35.XXX.XXX: icmp_req=5 ttl=47 time=156 ms 64 bytes from 8.35.XXX.XXX: icmp_req=6 ttl=47 time=153 ms 64 bytes from 8.35.XXX.XXX: icmp_req=7 ttl=47 time=155 ms From owner-freebsd-net@FreeBSD.ORG Thu Nov 22 11:23:03 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AB816DDB for ; Thu, 22 Nov 2012 11:23:03 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id E93938FC12 for ; Thu, 22 Nov 2012 11:23:02 +0000 (UTC) Received: (qmail 5530 invoked from network); 22 Nov 2012 12:22:54 +0100 Received: from fw.xip.at (HELO ?127.0.0.1?) (89.207.145.147) by chile.gbit.at with SMTP; 22 Nov 2012 12:22:54 +0100 Message-ID: <50AE0B12.8000309@xip.at> Date: Thu, 22 Nov 2012 12:22:58 +0100 From: Ingo Flaschberger User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Marc Peters Subject: Re: Low Bandwidth on intercontinental connections References: <50ACF62C.8000408@mpeters.org> <50ad087d.1892cc0a.2cce.3bf2@mx.google.com> <50AD1012.7020209@mpeters.org> <50AD14F8.8050001@xip.at> <50ADE5E4.9090708@mpeters.org> In-Reply-To: <50ADE5E4.9090708@mpeters.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2012 11:23:03 -0000 >> *) check and compare tcpdump > for the FreeBSD hosts on the receiver side, it showed a lot of window > size changes and from time to time a lot of duplicate ACKs. i will file > a PR (as Adrian asked) and see to get a matching tcpdump and SIFTR output. *) can you check which ping-sizes work? ping -s 1472 ping -D -s 1472 (should work if you have a mtu of 1500 all over the way) *) any offloading/supported used at the network-card? *) try a rate-shaping queue outgoing (not really good - as shaping works best on incomming interfaces): you need dummynet (and ipfw for this example): ipfw add pipe 1 all from .... ipfw pipe 1 config bw 10Mbit/s queue 50Kbytes (adjust queue size ~40ms at rated speed) Kind regards, Ingo Flaschberger From owner-freebsd-net@FreeBSD.ORG Thu Nov 22 12:38:40 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 174207D5 for ; Thu, 22 Nov 2012 12:38:40 +0000 (UTC) (envelope-from marc@mpeters.org) Received: from mail.mpeters.org (mail.mpeters.org [78.46.104.142]) by mx1.freebsd.org (Postfix) with ESMTP id BE1308FC08 for ; Thu, 22 Nov 2012 12:38:39 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mail.mpeters.org (Postfix) with ESMTP id 9F572132013; Thu, 22 Nov 2012 13:38:38 +0100 (CET) X-Virus-Scanned: amavisd-new at mpeters.org Received: from mail.mpeters.org ([127.0.0.1]) by localhost (mail.mpeters.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id I1mWz2-3bin4; Thu, 22 Nov 2012 13:38:37 +0100 (CET) Received: from [192.168.0.204] (unknown [62.159.86.18]) by mail.mpeters.org (Postfix) with ESMTPSA id 3362C13200E; Thu, 22 Nov 2012 13:38:36 +0100 (CET) Message-ID: <50AE1CCC.7080706@mpeters.org> Date: Thu, 22 Nov 2012 13:38:36 +0100 From: Marc Peters User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:16.0) Gecko/20121104 Thunderbird/16.0.1 MIME-Version: 1.0 To: Ingo Flaschberger Subject: Re: Low Bandwidth on intercontinental connections References: <50ACF62C.8000408@mpeters.org> <50ad087d.1892cc0a.2cce.3bf2@mx.google.com> <50AD1012.7020209@mpeters.org> <50AD14F8.8050001@xip.at> <50ADE5E4.9090708@mpeters.org> <50AE0B12.8000309@xip.at> In-Reply-To: <50AE0B12.8000309@xip.at> X-Enigmail-Version: 1.5a1pre Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2012 12:38:40 -0000 On 11/22/2012 12:22 PM, Ingo Flaschberger wrote: > >>> *) check and compare tcpdump >> for the FreeBSD hosts on the receiver side, it showed a lot of window >> size changes and from time to time a lot of duplicate ACKs. i will file >> a PR (as Adrian asked) and see to get a matching tcpdump and SIFTR >> output. > > *) can you check which ping-sizes work? > ping -s 1472 > ping -D -s 1472 (should work if you have a mtu of 1500 all over the > way) interesting, the MTU is way lower, than i expected. Through the VPN tunnel, only 1322 bytes are possible without fragmentation. ScreenOS adds 42 additional bytes per paket and the FreeBSD box is receiving 1364 bytes, according to tcpdump. From the outside (only one Netscreen on the way), 1472 is the maximum possible size to send pakets without fragmentation (-D). Which MTU would you suggest to use? Shouldn't the MTU discovery of FreeBSD handle this correct? > > *) any offloading/supported used at the network-card? Yes: bce0: flags=8843 metric 0 mtu 1500 options=c01bb ether ac:16:2d:b7:00:f4 inet 172.16.3.10 netmask 0xffffff00 broadcast 172.16.3.255 inet6 fe80::ae16:2dff:feb7:f4%bce0 prefixlen 64 scopeid 0x1 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active > *) try a rate-shaping queue outgoing (not really good - as shaping works > best on incomming interfaces): > you need dummynet (and ipfw for this example): > ipfw add pipe 1 all from .... > ipfw pipe 1 config bw 10Mbit/s queue 50Kbytes > (adjust queue size ~40ms at rated speed) no paketfiltering on the host itself is intended and i don't know anything of ipfw for a simple setup, sorry. marc > > Kind regards, > Ingo Flaschberger > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Thu Nov 22 13:16:48 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6FE0F374 for ; Thu, 22 Nov 2012 13:16:48 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id CB2658FC08 for ; Thu, 22 Nov 2012 13:16:46 +0000 (UTC) Received: (qmail 16023 invoked from network); 22 Nov 2012 14:16:45 +0100 Received: from fw.xip.at (HELO ?127.0.0.1?) (89.207.145.147) by chile.gbit.at with SMTP; 22 Nov 2012 14:16:45 +0100 Message-ID: <50AE25C1.7050208@xip.at> Date: Thu, 22 Nov 2012 14:16:49 +0100 From: Ingo Flaschberger User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: Re: Low Bandwidth on intercontinental connections References: <50ACF62C.8000408@mpeters.org> <50ad087d.1892cc0a.2cce.3bf2@mx.google.com> <50AD1012.7020209@mpeters.org> <50AD14F8.8050001@xip.at> <50ADE5E4.9090708@mpeters.org> <50AE0B12.8000309@xip.at> <50AE1CCC.7080706@mpeters.org> In-Reply-To: <50AE1CCC.7080706@mpeters.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2012 13:16:48 -0000 Am 22.11.2012 13:38, schrieb Marc Peters: > interesting, the MTU is way lower, than i expected. Through the VPN > tunnel, only 1322 bytes are possible without fragmentation. ScreenOS > adds 42 additional bytes per paket and the FreeBSD box is receiving 1364 > bytes, according to tcpdump. From the outside (only one Netscreen on the > way), 1472 is the maximum possible size to send pakets without > fragmentation (-D). > > Which MTU would you suggest to use? Shouldn't the MTU discovery of > FreeBSD handle this correct? should handle, yes. Any icmp firewalls in between? you can also try tcp mss clamping (via firewall, infos via google) >> *) any offloading/supported used at the network-card? > Yes: > bce0: flags=8843 metric 0 mtu 1500 > options=c01bb > ether ac:16:2d:b7:00:f4 > inet 172.16.3.10 netmask 0xffffff00 broadcast 172.16.3.255 > inet6 fe80::ae16:2dff:feb7:f4%bce0 prefixlen 64 scopeid 0x1 > nd6 options=29 > media: Ethernet autoselect (1000baseT ) > status: active try to deactivate JUMBO_MTU and TSO4 >> *) try a rate-shaping queue outgoing (not really good - as shaping works >> best on incomming interfaces): >> you need dummynet (and ipfw for this example): >> ipfw add pipe 1 all from .... >> ipfw pipe 1 config bw 10Mbit/s queue 50Kbytes >> (adjust queue size ~40ms at rated speed) > no paketfiltering on the host itself is intended and i don't know > anything of ipfw for a simple setup, sorry. > ipfw add pipe 1 all from thishostip to destinationhostip ipfw pipe 1 config bw 10Mbit/s queue 50Kbytes Kind regards, Ingo Flaschberger From owner-freebsd-net@FreeBSD.ORG Thu Nov 22 13:20:04 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 884E8530 for ; Thu, 22 Nov 2012 13:20:04 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id C9E158FC12 for ; Thu, 22 Nov 2012 13:20:03 +0000 (UTC) Received: (qmail 16205 invoked from network); 22 Nov 2012 14:20:02 +0100 Received: from fw.xip.at (HELO ?127.0.0.1?) (89.207.145.147) by chile.gbit.at with SMTP; 22 Nov 2012 14:20:02 +0100 Message-ID: <50AE2686.8070007@xip.at> Date: Thu, 22 Nov 2012 14:20:06 +0100 From: Ingo Flaschberger User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: Re: Low Bandwidth on intercontinental connections References: <50ACF62C.8000408@mpeters.org> <50ad087d.1892cc0a.2cce.3bf2@mx.google.com> <50AD1012.7020209@mpeters.org> <50AD14F8.8050001@xip.at> <50ADE5E4.9090708@mpeters.org> <50AE0B12.8000309@xip.at> <50AE1CCC.7080706@mpeters.org> In-Reply-To: <50AE1CCC.7080706@mpeters.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2012 13:20:04 -0000 Am 22.11.2012 13:38, schrieb Marc Peters: > interesting, the MTU is way lower, than i expected. Through the VPN > tunnel, only 1322 bytes are possible without fragmentation. ScreenOS > adds 42 additional bytes per paket and the FreeBSD box is receiving > 1364 bytes, according to tcpdump. From the outside (only one Netscreen > on the way), 1472 is the maximum possible size to send pakets without > fragmentation (-D). Which MTU would you suggest to use? Shouldn't the > MTU discovery of FreeBSD handle this correct? do you see fragmented tcp packets on the receiving site in tcpdump? When you load the tcpdump data (tcpdump -s 1500 -w filename ...) into wireshark, you can graph the speed (bit/sec, packets/sec) and do some more tcp analysis. Kind regards, Ingo Flaschberger From owner-freebsd-net@FreeBSD.ORG Thu Nov 22 13:27:35 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D76ED66D for ; Thu, 22 Nov 2012 13:27:35 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id 226A58FC08 for ; Thu, 22 Nov 2012 13:27:34 +0000 (UTC) Received: (qmail 17690 invoked from network); 22 Nov 2012 14:27:33 +0100 Received: from fw.xip.at (HELO ?127.0.0.1?) (89.207.145.147) by chile.gbit.at with SMTP; 22 Nov 2012 14:27:33 +0100 Message-ID: <50AE2849.6020004@xip.at> Date: Thu, 22 Nov 2012 14:27:37 +0100 From: Ingo Flaschberger User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: Re: Low Bandwidth on intercontinental connections References: <50ACF62C.8000408@mpeters.org> <50ad087d.1892cc0a.2cce.3bf2@mx.google.com> <50AD1012.7020209@mpeters.org> <50AD14F8.8050001@xip.at> <50ADE5E4.9090708@mpeters.org> <50AE0B12.8000309@xip.at> <50AE1CCC.7080706@mpeters.org> <50AE25C1.7050208@xip.at> In-Reply-To: <50AE25C1.7050208@xip.at> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2012 13:27:36 -0000 Am 22.11.2012 14:16, schrieb Ingo Flaschberger: > > >>> *) try a rate-shaping queue outgoing (not really good - as shaping >>> works >>> best on incomming interfaces): sorry - told something wrong. shapeing works best on outgoing interfaces (not incomming) >>> you need dummynet (and ipfw for this example): >>> ipfw add pipe 1 all from .... >>> ipfw pipe 1 config bw 10Mbit/s queue 50Kbytes >>> (adjust queue size ~40ms at rated speed) >> no paketfiltering on the host itself is intended and i don't know >> anything of ipfw for a simple setup, sorry. right rules are: ipfw add pipe 1 all from thishostip to destinationhostip out ipfw pipe 1 config bw 10Mbit/s queue 50Kbytes Kind regards, Ingo Flaschberger From owner-freebsd-net@FreeBSD.ORG Thu Nov 22 14:14:14 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 10FFB9D for ; Thu, 22 Nov 2012 14:14:14 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm36-vm0.bullet.mail.ne1.yahoo.com (nm36-vm0.bullet.mail.ne1.yahoo.com [98.138.229.112]) by mx1.freebsd.org (Postfix) with ESMTP id 7CDC28FC0C for ; Thu, 22 Nov 2012 14:14:13 +0000 (UTC) Received: from [98.138.226.178] by nm36.bullet.mail.ne1.yahoo.com with NNFMP; 22 Nov 2012 14:11:03 -0000 Received: from [98.138.89.244] by tm13.bullet.mail.ne1.yahoo.com with NNFMP; 22 Nov 2012 14:11:03 -0000 Received: from [127.0.0.1] by omp1058.mail.ne1.yahoo.com with NNFMP; 22 Nov 2012 14:11:03 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 634641.87417.bm@omp1058.mail.ne1.yahoo.com Received: (qmail 18851 invoked by uid 60001); 22 Nov 2012 14:11:03 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1353593463; bh=0if7lDaF0RaCATGIX9JZUAYAq/MlC1GCdLqhsOK09xg=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=jhS+tQ5pchflEl2ldTDeEwmbcaK8z9G8+zBVUd829wuW59rJEhuYPW8ceYbtnQczzAuEWZI1qT3uHZVC9LMtg5I0GwzWYXkQJpyS0+GKoA5dhaNJzRFZ2o5HxLVQO2J1WZ9wBL+VDnV5MQ9pMU0EeAEu94TGIcid+OTh00RUUGI= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=0gBateqoceRr3rnrY3OszMnTz4beJJC0SkO9siVU/noEwdGA9MFADMJQPAJrX7wNOHRI5AkPtYKLnjGKWmKwXREkL5T63z88fHowRDAWKmxC1c4PmZK3DgwsSX2BVjSXpKjdj8NdzY9fpIRaMHlmc8C63z+SBNMn51EN6xdaZWQ=; X-YMail-OSG: hZk9OesVM1mhBJ3_77k2S1DKwZ7ZCMp7pRQsbVq68sUORcO 4ym9JkLoiJ8uMncxwoTN0HTxiZLlmrLzre.LiSYJ1jq1Nj3dmqutdylbDROC R4VaHb92moPCIzRsH57m069ktkJlbC7yA4xS2rc8SG2J5hbmnPh2EvhNh78T GYFCPzoPVZGFyU2gQIA_52EDEYj9MwDmZ1qyaQBZdZkX0fzr0CoomEzHRG3q mdmJ0lkcG.o9bj3BgqA721NXcGEOhEeHH8cLsYDR1M7V3SxFKrC9Vpw.xiiE UYkAQcSNzx.xqhAFHEFaoTFcS5E6tqqeBTzyT2Kk8NmxsC2dh_x3f6W9akuT 6K2Fxa.EmqbwEGCxLqHCIlI05JxHOTZs9bi6w4FqhsgIA4j0.q2npW_eRZiQ PVohfoPD2bcSEHo3eUvVgjigg2xDXCT7NX2zMK7dTeOg2kQpnWdM2encgz6X h70uhw1tF5lsIsa0oUZlR7fbKOYp7wN8eQCsthDLzBgHhFgHCpRsOh6znLKQ AM2sTLo3vofSJXmAotA-- Received: from [174.48.128.27] by web121601.mail.ne1.yahoo.com via HTTP; Thu, 22 Nov 2012 06:11:02 PST X-Rocket-MIMEInfo: 001.001, CgotLS0gT24gV2VkLCAxMS8yMS8xMiwgQWRyaWFuIENoYWRkIDxhZHJpYW5AZnJlZWJzZC5vcmc.IHdyb3RlOgoKPiBGcm9tOiBBZHJpYW4gQ2hhZGQgPGFkcmlhbkBmcmVlYnNkLm9yZz4KPiBTdWJqZWN0OiBSZTogRnJlZUJTRCBib3hlcyBhcyBhICdyb3V0ZXInLi4uCj4gVG86ICJBbmRyZSBPcHBlcm1hbm4iIDxhbmRyZUBmcmVlYnNkLm9yZz4KPiBDYzogIkJhcm5leSBDb3Jkb2JhIiA8YmFybmV5X2NvcmRvYmFAeWFob28uY29tPiwgIkppbSBUaG9tcHNvbiIgPGppbUBuZXRnYXRlLmNvbT4sICJBbGZyZWQBMAEBAQE- X-Mailer: YahooMailClassic/15.0.8 YahooMailWebService/0.8.123.460 Message-ID: <1353593462.18740.YahooMailClassic@web121601.mail.ne1.yahoo.com> Date: Thu, 22 Nov 2012 06:11:02 -0800 (PST) From: Barney Cordoba Subject: Re: FreeBSD boxes as a 'router'... To: Andre Oppermann , Adrian Chadd In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-net@freebsd.org" , Jim Thompson , Alfred Perlstein , khatfield@socllc.net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2012 14:14:14 -0000 =0A=0A--- On Wed, 11/21/12, Adrian Chadd wrote:=0A=0A>= From: Adrian Chadd =0A> Subject: Re: FreeBSD boxes as = a 'router'...=0A> To: "Andre Oppermann" =0A> Cc: "Barney= Cordoba" , "Jim Thompson" , "Al= fred Perlstein" , khatfield@socllc.net, "freebsd-net@freebsd= .org" =0A> Date: Wednesday, November 21, 2012, 1:2= 6 PM=0A> On 21 November 2012 00:30, Andre=0A> Oppermann = =0A> wrote:=0A> > On 21.11.2012 08:55, Adrian Chadd wrote:=0A> >>=0A> >> So= mething that has popped up a few times, even=0A> recently, is breaking=0A> = >> out of an RX loop after you service a number of=0A> frames.=0A> >=0A> > = That is what I basically described.=0A> =0A> Right, and this can be done ri= ght now without too much=0A> reworking,=0A> right? I mean, people could beg= in by doing a drive-by on=0A> drivers for=0A> this.=0A> The RX path for a d= river shouldn't be too difficult to do;=0A> the TX path=0A> is the racy one= .=0A> =0A> >> During stupidly high levels of RX, you may find the=0A> NIC h= appily=0A> >> receiving frames faster than you can service the RX=0A> queue= . If this=0A> >> occurs, you could end up just plain being stuck=0A> there.= =0A> =0A> > That's the live-lock.=0A> =0A> And again you can solve this wit= hout having to devolve into=0A> polling.=0A> Again, polling to me feels lik= e a bludgeon beating around a=0A> system=0A> that isn't really designed for= the extreme cases it's=0A> facing.=0A> Maybe your work in the tcp_taskqueu= e branch addresses the=0A> larger scale=0A> issues here, but I've solved th= is relatively easily in the=0A> past.=0A> =0A> >> So what I've done in the = past is to loop over a=0A> certain number of=0A> >> frames, then schedule a= taskqueue to service=0A> whatever's left over.=0A> =0A> > Taskqueue's shou= ldn't be used anymore.=A0 We've got=0A> ithreads now.=0A> > Contrary to pop= ular belief (and due to poor=0A> documentation) an=0A> > ithread does not r= un at interrupt level.=A0 Only the=0A> fast interrupt=0A> > handler does th= at.=A0 The ithread is a normal kernel=0A> thread tied to=0A> > an fast inte= rrupt handler and trailing it whenever it=0A> said=0A> > INTR_SCHEDULE_ITHR= EAD.=0A> =0A> Sure, but taskqueues are still useful if you want to=0A> seri= alise access=0A> without relying on mutexes wrapping large parts of the=0A>= packet handling=0A> code to enforce said order.=0A> =0A> Yes, normal ithre= ads don't run at interrupt level.=0A> =0A> And we can change the priority o= f taskqueues in each driver,=0A> right?=0A> And/or we could change the beha= viour of driver=0A> ithreads/taskqueues to=0A> be automatically reniced?=0A= =0AWhy schedule a taskqueue? You're just adding more work to a system=0Atha= t's already overloaded. You'll get another interrupt soon enough.=0AYou can= control the delay, to simulate a "poll" without having to =0Aadd yet-anoth= er task to the system.=0A=0AThe idea that you're getting so many packets th= at the system can't handle=0Ait, and that you have to schedule a task becau= se you might not get =0Aanother interrupt is just bad thinking for anything= other than an end=0Auser application, in which case this conversation isn'= t relevant. =0A> =0A> I'm not knocking your work here, I'm just trying to= =0A> understand whether=0A> we can do this stuff as small individual pieces= of work=0A> rather than=0A> one big subsystem overhaul.=0A> =0A> And CoDel= is interesting as a concept, but it's certainly=0A> not new. But=0A> again= , if you don't drop the frames during the driver=0A> receive path=0A> (and = try to do it higher up in the stack, eg as part of some=0A> firewall=0A> ru= le) you still risk reaching a stable state where the CPU=0A> is 100%=0A> pi= nned because you've wasted cycles pushing those frames=0A> into the=0A> que= ue only to be dropped.=0A=0A=0AQueue althorithms that assume that all netwo= rk applications are the=0Asame are to be put on the heap with isdn and ATM = and other stupid ideas=0Adesigned by IETF "thinkers". =0A=0AThe design goal= should be to avoid queuing; and drop events are usually=0Anot part of a no= rmal flow; the concept that you can have a nice algorithm=0Ato handle it as= sumes that you are trying to do too much with a too slow=0Acpu. Crap design= ed by Cisco exists only because their hardware never had=0Aenough CPU to do= the work needed to be done. =0A=0A=0A> What _I_ had to do there was have a= quick gate to look up if=0A> a frame=0A> was part of an active session in = ipfw and if it was, let it=0A> be queued=0A> to the driver. I also had a se= cond gate in the driver for=0A> new TCP=0A> connections, but that was a sep= arate hack. Anything else was=0A> dropped.=0A> =0A> In any case, what I'm t= rying to say is this - when I was=0A> last doing=0A> this kind of stuff, I = didn't just subscribe to "polling will=0A> fix all."=0A> I spent a few mont= hs knee deep in the public intel e1000=0A> documentation=0A> and tuning gui= de, the em driver and the queue/firewall code,=0A> in order=0A> to figure o= ut how to attack this without using polling.=0A> =0A> And yes, you've also = just described NAPI. :-)=0A> =0A=0AIm not sure what you're doing that would= cause packets to come in faster=0Athan you can service them, unless you're= running on an old XT or something.=0AA modern $300 cpu can manage an awful= lot of packets, depending on your=0Aapplication.=0A=0APackets are like cus= tomers. Sometimes you have to let them go. Its fairly=0Aeasy to determine w= hat a given system running a given application can =0Ahandle. If you get mo= re than that you have little chance of figuring out=0Aa scheme to manage it= .=0A=0AIf you're running an embedded app and you dont have the option of si= mply=0Agetting a faster machine, then you just have to set a threshold and = deal=0Awith it. You can try to be "smart" and peek at packets and drop "les= s=0Aimportant" packets, but in my experience the smarter you try to be, the= =0Adumber you turn out to be.=0A=0Awith modern cpus with big caches, the b= ottlenecks are almost always locking=0Aand not queuing or memory shuffling,= assuming you're not running on a=0Asingle core system. So design according= ly.=0A=0AUnfortunately the multiqueue drivers in FreeBSD aren't usable, so = until=0Asomeone figures out a proper design that just doesn't suck up more = cores=0Awith marginal if any gains in capacity, you're stuck =0A=0ABC=0A=0A From owner-freebsd-net@FreeBSD.ORG Thu Nov 22 14:32:29 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 73A674CA for ; Thu, 22 Nov 2012 14:32:29 +0000 (UTC) (envelope-from artemrts@ukr.net) Received: from ffe17.ukr.net (ffe17.ukr.net [195.214.192.83]) by mx1.freebsd.org (Postfix) with ESMTP id 195258FC08 for ; Thu, 22 Nov 2012 14:32:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=ffe; h=Date:Message-Id:From:To:References:In-Reply-To:Subject:Cc:Content-Type:Content-Transfer-Encoding:MIME-Version; bh=mGFsww72y8868q7ORo0+2dtK0QmnCx+zm6E+ZICLi0I=; b=M607Mz6NE32qVVOPniJbos+uLEsh17X3km4iaf4N4Hf7wGOLUPd6iWHCKjFzlQbSjM4MZqA3T70L/IYYy/35TX2o3ZCmKjs5DcOEGLO0un1aZ4RmbVOXoIEaJA93ARjtwt2Brf/aNflNAfOeVrE7JmYNwFlFO0IdDKRIm9uLjfs=; Received: from mail by ffe17.ukr.net with local ID 1TbXUx-000Faf-Vq ; Thu, 22 Nov 2012 16:11:24 +0200 MIME-Version: 1.0 Content-Disposition: inline Content-Transfer-Encoding: binary Content-Type: text/plain; charset="windows-1251" Subject: Re[2]: Low Bandwidth on intercontinental connections In-Reply-To: <50AE2849.6020004@xip.at> References: <50AE0B12.8000309@xip.at> <50AE1CCC.7080706@mpeters.org> <50AE2849.6020004@xip.at> <50AD14F8.8050001@xip.at> <50ADE5E4.9090708@mpeters.org> <50AE25C1.7050208@xip.at> <50ACF62C.8000408@mpeters.org> <50ad087d.1892cc0a.2cce.3bf2@mx.google.com> <50AD1012.7020209@mpeters.org> To: "Ingo Flaschberger" From: "wishmaster" X-Mailer: freemail.ukr.net 4.0 Message-Id: <59072.1353593483.18207769361274765312@ffe17.ukr.net> X-Browser: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Firefox/17.0 Date: Thu, 22 Nov 2012 16:11:23 +0200 Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2012 14:32:29 -0000 --- Original message --- From: "Ingo Flaschberger" To: freebsd-net@freebsd.org Date: 22 November 2012, 15:27:48 Subject: Re: Low Bandwidth on intercontinental connections > Am 22.11.2012 14:16, schrieb Ingo Flaschberger: > > > > > >>> *) try a rate-shaping queue outgoing (not really good - as shaping > >>> works > >>> best on incomming interfaces): > sorry - told something wrong. > shapeing works best on outgoing interfaces (not incomming) > >>> you need dummynet (and ipfw for this example): > >>> ipfw add pipe 1 all from .... > >>> ipfw pipe 1 config bw 10Mbit/s queue 50Kbytes > >>> (adjust queue size ~40ms at rated speed) > >> no paketfiltering on the host itself is intended and i don't know > >> anything of ipfw for a simple setup, sorry. > right rules are: > ipfw add pipe 1 all from thishostip to destinationhostip out > ipfw pipe 1 config bw 10Mbit/s queue 50Kbytes You have forgotten kldload ipfw && kldload dummynet before adding this rules :) From owner-freebsd-net@FreeBSD.ORG Thu Nov 22 15:42:22 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1E434C2F for ; Thu, 22 Nov 2012 15:42:22 +0000 (UTC) (envelope-from marc@mpeters.org) Received: from mail.mpeters.org (mail.mpeters.org [78.46.104.142]) by mx1.freebsd.org (Postfix) with ESMTP id C193F8FC17 for ; Thu, 22 Nov 2012 15:42:21 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mail.mpeters.org (Postfix) with ESMTP id DC256132011 for ; Thu, 22 Nov 2012 16:42:14 +0100 (CET) X-Virus-Scanned: amavisd-new at mpeters.org Received: from mail.mpeters.org ([127.0.0.1]) by localhost (mail.mpeters.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kPC2ysm4I6hR for ; Thu, 22 Nov 2012 16:42:13 +0100 (CET) Received: from [192.168.0.204] (unknown [62.159.86.18]) by mail.mpeters.org (Postfix) with ESMTPSA id 02C1513200E for ; Thu, 22 Nov 2012 16:42:12 +0100 (CET) Message-ID: <50AE47D4.7080608@mpeters.org> Date: Thu, 22 Nov 2012 16:42:12 +0100 From: Marc Peters User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:16.0) Gecko/20121104 Thunderbird/16.0.1 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: Re: Low Bandwidth on intercontinental connections References: <50ACF62C.8000408@mpeters.org> <50ad087d.1892cc0a.2cce.3bf2@mx.google.com> <50AD1012.7020209@mpeters.org> <50AD14F8.8050001@xip.at> <50ADE5E4.9090708@mpeters.org> <50AE0B12.8000309@xip.at> <50AE1CCC.7080706@mpeters.org> <50AE2686.8070007@xip.at> In-Reply-To: <50AE2686.8070007@xip.at> X-Enigmail-Version: 1.5a1pre Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2012 15:42:22 -0000 On 11/22/2012 02:20 PM, Ingo Flaschberger wrote: > Am 22.11.2012 13:38, schrieb Marc Peters: >> interesting, the MTU is way lower, than i expected. Through the VPN >> tunnel, only 1322 bytes are possible without fragmentation. ScreenOS >> adds 42 additional bytes per paket and the FreeBSD box is receiving >> 1364 bytes, according to tcpdump. From the outside (only one Netscreen >> on the way), 1472 is the maximum possible size to send pakets without >> fragmentation (-D). Which MTU would you suggest to use? Shouldn't the >> MTU discovery of FreeBSD handle this correct? > > do you see fragmented tcp packets on the receiving site in tcpdump? nearly every packet is fragmented, if i read th [TCP segment of a reassembled PDU] correct. Those have a length of 1364. Thera are also lots of [TCP Window Update] (every two to three acks from the receiving host, where the tcpdump took place). After some time, there were lots of [TCP Dup ACK] from the receiving host and the throughput went to a crawl and had lots of retransmissions. After 30 sec. everything went back to "normal" as of transmitting pakets without dubs and retransmissions. The same tcpdump collect on the Linux hosts, the packets had a length of 1514 and no dubs or retransmissions. I didn't get the option JUMBO_MTU removed, but removing TSO4 didn't changed anything. Adding packetfiltering and the pipe didn't change anything, too. > > When you load the tcpdump data (tcpdump -s 1500 -w filename ...) into > wireshark, you can graph the speed (bit/sec, packets/sec) and do some > more tcp analysis. > > Kind regards, > Ingo Flaschberger > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Thu Nov 22 17:09:07 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id CFAAC793 for ; Thu, 22 Nov 2012 17:09:07 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-we0-f182.google.com (mail-we0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 56E138FC08 for ; Thu, 22 Nov 2012 17:09:07 +0000 (UTC) Received: by mail-we0-f182.google.com with SMTP id u54so2882376wey.13 for ; Thu, 22 Nov 2012 09:09:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=DzNVzNauWYd0DC2MgK6W/L6tC93GDTeqYr/zDziTMBw=; b=xT7gxgbsj2FY3mqHAffCV03vVrdPF8qrQYjTlCa+9NaYaLRgPfrdy6QZXfmufzUiVA D+xOhLtW95xhAp7AJB14v5MUsBHYJVkNpgfoZCBqztifeSAN/7q6QBFRJ0csxCFbz0gL VqZV+m2sGce1N2Hyj9oFBiCMLdEj2t6ElvUBswN/Wq04FlEwO7r+RLbV3rEHL1cp9DcF CcVsLCSa8QNnGn1AKiBQZnr6CoJb7EKQiuHlJjoQJ+vSybjdJOdpIGghYSLpACIQgYnQ vihskY6liMODGb0zhcVc0dF8/QqhUUws0I9SpckjPAuu9zrdfAtfzttwyHlJggNQPYMB 2uFA== MIME-Version: 1.0 Received: by 10.216.139.140 with SMTP id c12mr612842wej.46.1353604146285; Thu, 22 Nov 2012 09:09:06 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.217.57.9 with HTTP; Thu, 22 Nov 2012 09:09:06 -0800 (PST) In-Reply-To: <50AE47D4.7080608@mpeters.org> References: <50ACF62C.8000408@mpeters.org> <50ad087d.1892cc0a.2cce.3bf2@mx.google.com> <50AD1012.7020209@mpeters.org> <50AD14F8.8050001@xip.at> <50ADE5E4.9090708@mpeters.org> <50AE0B12.8000309@xip.at> <50AE1CCC.7080706@mpeters.org> <50AE2686.8070007@xip.at> <50AE47D4.7080608@mpeters.org> Date: Thu, 22 Nov 2012 09:09:06 -0800 X-Google-Sender-Auth: UG5rgBqS09V32q3eHUSOYkyrotc Message-ID: Subject: Re: Low Bandwidth on intercontinental connections From: Adrian Chadd To: Marc Peters Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2012 17:09:08 -0000 Hi Marc, You definitely have enough information now for a PR. Would you please file all of this into a PR so one of the IP stack people can take a look? Thanks, Adrian On 22 November 2012 07:42, Marc Peters wrote: > On 11/22/2012 02:20 PM, Ingo Flaschberger wrote: >> Am 22.11.2012 13:38, schrieb Marc Peters: >>> interesting, the MTU is way lower, than i expected. Through the VPN >>> tunnel, only 1322 bytes are possible without fragmentation. ScreenOS >>> adds 42 additional bytes per paket and the FreeBSD box is receiving >>> 1364 bytes, according to tcpdump. From the outside (only one Netscreen >>> on the way), 1472 is the maximum possible size to send pakets without >>> fragmentation (-D). Which MTU would you suggest to use? Shouldn't the >>> MTU discovery of FreeBSD handle this correct? >> >> do you see fragmented tcp packets on the receiving site in tcpdump? > > nearly every packet is fragmented, if i read th [TCP segment of a > reassembled PDU] correct. Those have a length of 1364. Thera are also > lots of [TCP Window Update] (every two to three acks from the receiving > host, where the tcpdump took place). After some time, there were lots of > [TCP Dup ACK] from the receiving host and the throughput went to a crawl > and had lots of retransmissions. After 30 sec. everything went back to > "normal" as of transmitting pakets without dubs and retransmissions. > > The same tcpdump collect on the Linux hosts, the packets had a length of > 1514 and no dubs or retransmissions. > > I didn't get the option JUMBO_MTU removed, but removing TSO4 didn't > changed anything. > > Adding packetfiltering and the pipe didn't change anything, too. > >> >> When you load the tcpdump data (tcpdump -s 1500 -w filename ...) into >> wireshark, you can graph the speed (bit/sec, packets/sec) and do some >> more tcp analysis. >> >> Kind regards, >> Ingo Flaschberger >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Thu Nov 22 23:07:14 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 19B4B76D for ; Thu, 22 Nov 2012 23:07:14 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id 58BEC8FC08 for ; Thu, 22 Nov 2012 23:07:12 +0000 (UTC) Received: (qmail 25672 invoked from network); 23 Nov 2012 00:07:10 +0100 Received: from fw.xip.at (HELO ?127.0.0.1?) (89.207.145.147) by chile.gbit.at with SMTP; 23 Nov 2012 00:07:10 +0100 Message-ID: <50AEB022.6050608@xip.at> Date: Fri, 23 Nov 2012 00:07:14 +0100 From: Ingo Flaschberger User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: Re: Low Bandwidth on intercontinental connections References: <50ACF62C.8000408@mpeters.org> <50ad087d.1892cc0a.2cce.3bf2@mx.google.com> <50AD1012.7020209@mpeters.org> <50AD14F8.8050001@xip.at> <50ADE5E4.9090708@mpeters.org> <50AE0B12.8000309@xip.at> <50AE1CCC.7080706@mpeters.org> <50AE2686.8070007@xip.at> <50AE47D4.7080608@mpeters.org> In-Reply-To: <50AE47D4.7080608@mpeters.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2012 23:07:14 -0000 > nearly every packet is fragmented, if i read th [TCP segment of a > reassembled PDU] correct. Those have a length of 1364. Thera are also > lots of [TCP Window Update] (every two to three acks from the receiving > host, where the tcpdump took place). After some time, there were lots of > [TCP Dup ACK] from the receiving host and the throughput went to a crawl > and had lots of retransmissions. After 30 sec. everything went back to > "normal" as of transmitting pakets without dubs and retransmissions. how you can detect fragmented packets: tcpdump at sender and at receiver site. If you send 1500 byte packets but receive 2 differnt sized packets (a bigger and a smaller one) at the other site - than there is fragmentation. 1364 packet length seems strage, as your max icmp packet is 1322 byte + 20byte IP header + 8byte icmp = mtu 1350? > The same tcpdump collect on the Linux hosts, the packets had a length of > 1514 and no dubs or retransmissions. 1514? max mtu on ethernet is 1500 can you send me (directly) a send and receive site tcpdump from linux and freebsd for a whole data-transfer? Kind regards, Ingo Flaschberger From owner-freebsd-net@FreeBSD.ORG Fri Nov 23 12:47:48 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 68A86AC7 for ; Fri, 23 Nov 2012 12:47:48 +0000 (UTC) (envelope-from marc@mpeters.org) Received: from mail.mpeters.org (mail.mpeters.org [78.46.104.142]) by mx1.freebsd.org (Postfix) with ESMTP id 157938FC13 for ; Fri, 23 Nov 2012 12:47:47 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mail.mpeters.org (Postfix) with ESMTP id C49EA13200F for ; Fri, 23 Nov 2012 13:47:46 +0100 (CET) X-Virus-Scanned: amavisd-new at mpeters.org Received: from mail.mpeters.org ([127.0.0.1]) by localhost (mail.mpeters.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7wVIhTW9MCNG for ; Fri, 23 Nov 2012 13:47:45 +0100 (CET) Received: from [192.168.0.204] (unknown [62.159.86.18]) by mail.mpeters.org (Postfix) with ESMTPSA id 7BF3313200E for ; Fri, 23 Nov 2012 13:47:45 +0100 (CET) Message-ID: <50AF7070.50206@mpeters.org> Date: Fri, 23 Nov 2012 13:47:44 +0100 From: Marc Peters User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:16.0) Gecko/20121104 Thunderbird/16.0.1 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: Re: Low Bandwidth on intercontinental connections References: <50ACF62C.8000408@mpeters.org> <50ad087d.1892cc0a.2cce.3bf2@mx.google.com> <50AD1012.7020209@mpeters.org> <50AD14F8.8050001@xip.at> <50ADE5E4.9090708@mpeters.org> <50AE0B12.8000309@xip.at> <50AE1CCC.7080706@mpeters.org> <50AE2686.8070007@xip.at> <50AE47D4.7080608@mpeters.org> In-Reply-To: X-Enigmail-Version: 1.5a1pre Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2012 12:47:48 -0000 On 11/22/2012 06:09 PM, Adrian Chadd wrote: > Hi Marc, > > You definitely have enough information now for a PR. Would you please > file all of this into a PR so one of the IP stack people can take a > look? PR filed: http://www.freebsd.org/cgi/query-pr.cgi?pr=173859 > > Thanks, > > > Adrian From owner-freebsd-net@FreeBSD.ORG Fri Nov 23 13:07:00 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 05B7CF4F for ; Fri, 23 Nov 2012 13:07:00 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wg0-f50.google.com (mail-wg0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 7FFA98FC0C for ; Fri, 23 Nov 2012 13:06:59 +0000 (UTC) Received: by mail-wg0-f50.google.com with SMTP id 12so4557582wgr.31 for ; Fri, 23 Nov 2012 05:06:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=FRxhs19lt6kDRBJUv9VvOdKN+71gM5N4L+lGb8JI0Ns=; b=GmASw/9t6GZliuFWmECXDmbBAW0CM2lGQAuKb5jdD6wrm9j37NNqFfGfk3d803yzXH DhokBbQ317rVLexnTCyX9iRQeAT3M5mi9PlL/vEzHjUlVtr6mLpWxIA6PhRLU0aj3RhM o81bZSV0nTYVfjU/L4O3jpzXPaxGLHggD2OD4VMALTcOm2k4fja1YRNVoUgAi3Jw/Jp9 VdFKYfA/0tjEjRpz0VLPQSaaEEl1sAy/O64r7ynYUlx14kz41xx+oqLWx4tfnMqJ0Owo vcQ/fBAYbdPEB9X7k7Ar65QHK7s8x8l8p6tCZvra2T7DWGpk9XQv8EJJVcQUItRfDuuj u41A== MIME-Version: 1.0 Received: by 10.180.97.137 with SMTP id ea9mr1565941wib.13.1353676018075; Fri, 23 Nov 2012 05:06:58 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.217.57.9 with HTTP; Fri, 23 Nov 2012 05:06:57 -0800 (PST) In-Reply-To: <50AF7070.50206@mpeters.org> References: <50ACF62C.8000408@mpeters.org> <50ad087d.1892cc0a.2cce.3bf2@mx.google.com> <50AD1012.7020209@mpeters.org> <50AD14F8.8050001@xip.at> <50ADE5E4.9090708@mpeters.org> <50AE0B12.8000309@xip.at> <50AE1CCC.7080706@mpeters.org> <50AE2686.8070007@xip.at> <50AE47D4.7080608@mpeters.org> <50AF7070.50206@mpeters.org> Date: Fri, 23 Nov 2012 05:06:57 -0800 X-Google-Sender-Auth: HuZzjw4mUpE4la2QOWzIlT0LCmY Message-ID: Subject: Re: Low Bandwidth on intercontinental connections From: Adrian Chadd To: Marc Peters Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2012 13:07:00 -0000 On 23 November 2012 04:47, Marc Peters wrote: > On 11/22/2012 06:09 PM, Adrian Chadd wrote: >> Hi Marc, >> >> You definitely have enough information now for a PR. Would you please >> file all of this into a PR so one of the IP stack people can take a >> look? > > PR filed: > > http://www.freebsd.org/cgi/query-pr.cgi?pr=173859 Cool, thanks. Now, can you please add the output of netstat -sp tcp before and after you do a test? That includes doing a 'zero' of the stats first - netstat -sp tcp -z (then netstat -sp tcp to make sure they're zeroed.) Thanks! Adrian From owner-freebsd-net@FreeBSD.ORG Fri Nov 23 14:46:10 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 759F9A00 for ; Fri, 23 Nov 2012 14:46:10 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id B70008FC17 for ; Fri, 23 Nov 2012 14:46:09 +0000 (UTC) Received: (qmail 27870 invoked from network); 23 Nov 2012 15:46:00 +0100 Received: from fw.xip.at (HELO ?127.0.0.1?) (89.207.145.147) by chile.gbit.at with SMTP; 23 Nov 2012 15:46:00 +0100 Message-ID: <50AF8C2A.7010006@xip.at> Date: Fri, 23 Nov 2012 15:46:02 +0100 From: Ingo Flaschberger User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: Re: Low Bandwidth on intercontinental connections References: <50ACF62C.8000408@mpeters.org> <50ad087d.1892cc0a.2cce.3bf2@mx.google.com> <50AD1012.7020209@mpeters.org> <50AD14F8.8050001@xip.at> <50ADE5E4.9090708@mpeters.org> <50AE0B12.8000309@xip.at> <50AE1CCC.7080706@mpeters.org> <50AE2686.8070007@xip.at> <50AE47D4.7080608@mpeters.org> <50AF7070.50206@mpeters.org> In-Reply-To: <50AF7070.50206@mpeters.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2012 14:46:10 -0000 Am 23.11.2012 13:47, schrieb Marc Peters: > PR filed: http://www.freebsd.org/cgi/query-pr.cgi?pr=173859 Downloads from ftp1.us.freebsd.org to Europe (ping time ~170ms), I see up & down ramping of transfer speed (600kb/sec - 50kb/sec) over serverall releases: FreeBSD 6.3-PRERELEASE FreeBSD 7.2-STABLE FreeBSD 8.2-STABLE FreeBSD 9.1-PRERELEASE Linux performs much better, max speed was 1.2Mb/sec, and is not as much up & down ramping as freebsd. Can anyone compare how other *BSD performs? Kind regards, Ingo Flaschberger From owner-freebsd-net@FreeBSD.ORG Fri Nov 23 15:06:02 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 443B1166 for ; Fri, 23 Nov 2012 15:06:02 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id A19798FC13 for ; Fri, 23 Nov 2012 15:06:01 +0000 (UTC) Received: (qmail 16901 invoked from network); 23 Nov 2012 16:38:22 -0000 Received: from c00l3r.networx.ch (HELO [127.0.0.1]) ([62.48.2.2]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 23 Nov 2012 16:38:22 -0000 Message-ID: <50AF90D1.4020404@freebsd.org> Date: Fri, 23 Nov 2012 16:05:53 +0100 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20121026 Thunderbird/16.0.2 MIME-Version: 1.0 To: Ingo Flaschberger Subject: Re: Low Bandwidth on intercontinental connections References: <50ACF62C.8000408@mpeters.org> <50ad087d.1892cc0a.2cce.3bf2@mx.google.com> <50AD1012.7020209@mpeters.org> <50AD14F8.8050001@xip.at> <50ADE5E4.9090708@mpeters.org> <50AE0B12.8000309@xip.at> <50AE1CCC.7080706@mpeters.org> <50AE2686.8070007@xip.at> <50AE47D4.7080608@mpeters.org> <50AF7070.50206@mpeters.org> <50AF8C2A.7010006@xip.at> In-Reply-To: <50AF8C2A.7010006@xip.at> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2012 15:06:02 -0000 On 23.11.2012 15:46, Ingo Flaschberger wrote: > Am 23.11.2012 13:47, schrieb Marc Peters: >> PR filed: http://www.freebsd.org/cgi/query-pr.cgi?pr=173859 > > Downloads from ftp1.us.freebsd.org to Europe (ping time ~170ms), I see up & down ramping of transfer > speed (600kb/sec - 50kb/sec) > over serverall releases: > FreeBSD 6.3-PRERELEASE > FreeBSD 7.2-STABLE > FreeBSD 8.2-STABLE > FreeBSD 9.1-PRERELEASE > > Linux performs much better, max speed was 1.2Mb/sec, and is not as much up & down ramping as freebsd. It's about 168ms for me and I don't see the problem here. Speed is about 18.4Mbps. -- Andre From owner-freebsd-net@FreeBSD.ORG Fri Nov 23 15:29:38 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AFA769B2; Fri, 23 Nov 2012 15:29:38 +0000 (UTC) (envelope-from marc@mpeters.org) Received: from mail.mpeters.org (mail.mpeters.org [78.46.104.142]) by mx1.freebsd.org (Postfix) with ESMTP id 5EF858FC0C; Fri, 23 Nov 2012 15:29:38 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mail.mpeters.org (Postfix) with ESMTP id 7937C13200F; Fri, 23 Nov 2012 16:29:37 +0100 (CET) X-Virus-Scanned: amavisd-new at mpeters.org Received: from mail.mpeters.org ([127.0.0.1]) by localhost (mail.mpeters.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tfu101wFpq_j; Fri, 23 Nov 2012 16:29:36 +0100 (CET) Received: from [192.168.0.204] (unknown [62.159.86.18]) by mail.mpeters.org (Postfix) with ESMTPSA id 1279C13200E; Fri, 23 Nov 2012 16:29:35 +0100 (CET) Message-ID: <50AF965D.5050301@mpeters.org> Date: Fri, 23 Nov 2012 16:29:33 +0100 From: Marc Peters User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:16.0) Gecko/20121104 Thunderbird/16.0.1 MIME-Version: 1.0 To: Ingo Flaschberger , Andre Opperman Subject: Re: Low Bandwidth on intercontinental connections References: <50ACF62C.8000408@mpeters.org> <50ad087d.1892cc0a.2cce.3bf2@mx.google.com> <50AD1012.7020209@mpeters.org> <50AD14F8.8050001@xip.at> <50ADE5E4.9090708@mpeters.org> <50AE0B12.8000309@xip.at> <50AE1CCC.7080706@mpeters.org> <50AE2686.8070007@xip.at> <50AE47D4.7080608@mpeters.org> <50AF7070.50206@mpeters.org> <50AF8C2A.7010006@xip.at> In-Reply-To: <50AF8C2A.7010006@xip.at> X-Enigmail-Version: 1.5a1pre Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2012 15:29:38 -0000 On 11/23/2012 03:46 PM, Ingo Flaschberger wrote: > Am 23.11.2012 13:47, schrieb Marc Peters: >> PR filed: http://www.freebsd.org/cgi/query-pr.cgi?pr=173859 > > Downloads from ftp1.us.freebsd.org to Europe (ping time ~170ms), I see > up & down ramping of transfer speed (600kb/sec - 50kb/sec) > over serverall releases: > FreeBSD 6.3-PRERELEASE > FreeBSD 7.2-STABLE > FreeBSD 8.2-STABLE > FreeBSD 9.1-PRERELEASE > > Linux performs much better, max speed was 1.2Mb/sec, and is not as much > up & down ramping as freebsd. > > Can anyone compare how other *BSD performs? OpenBSD 5.2 also ramping between 600K/s at lowest and 1.5M/s. @Andre: Which version are you using? > > Kind regards, > Ingo Flaschberger > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Fri Nov 23 15:52:58 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B89A7431 for ; Fri, 23 Nov 2012 15:52:58 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm20.bullet.mail.ne1.yahoo.com (nm20.bullet.mail.ne1.yahoo.com [98.138.90.83]) by mx1.freebsd.org (Postfix) with ESMTP id 722E28FC15 for ; Fri, 23 Nov 2012 15:52:58 +0000 (UTC) Received: from [98.138.226.179] by nm20.bullet.mail.ne1.yahoo.com with NNFMP; 23 Nov 2012 15:52:52 -0000 Received: from [98.138.89.166] by tm14.bullet.mail.ne1.yahoo.com with NNFMP; 23 Nov 2012 15:52:52 -0000 Received: from [127.0.0.1] by omp1022.mail.ne1.yahoo.com with NNFMP; 23 Nov 2012 15:52:52 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 548290.49123.bm@omp1022.mail.ne1.yahoo.com Received: (qmail 92097 invoked by uid 60001); 23 Nov 2012 15:52:52 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1353685972; bh=qwORA9kHLQJl/TZUcdLgPs7pmVb9O2S83kSAGT7tcS4=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=sZZW/UnkV9U6e10aMXEOU+lexVbS8xf08eG6oOAurSV6/KmqyHIgEgmIK1tshw2Dyl3tDnDVqVILn18IR+RHbJOLQs/FCQ2qd6DSNt42SaV9P1SF6iL7JVv1p3FjF3qb/D0oI5O9c3G3W3VMBG28kxHGqtYNkpsxnChlpbpwzBY= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=vyys9Gx4NZYnKxpmetrrItVXNpjv6ixY/WqFXQi+vEF17jl5b9SeNp5aC5Zwlp5n4nfZoO55hiV80PN4zffbCvyYvxRxXBW6E2Ey24Tvht6u/M3X1NFLn2/g4b3qJha+Sprql10MpnRubOHL2/vwHQr/y83X5NwN8Ky3vQacbB4=; X-YMail-OSG: Yeg19q8VM1l49LVs.AtVe2G96HwAKRr_W4XOh1AfvjAQlI2 7xugjAxa3qlAwIe4yfPB0skPoZAA2xyKuwM1aUlS06Hb6gQV3_OTMoL2tM63 G3aWUdi1Ma21.ju420OnkhIimwefmUYuXlJO3VqzLuroeGDh9Kq9mZCEzcc4 NNXoHfOhfQcDoYaG7aDGjpnzmwzuzwA2IbhqbZ3.DLURzVqBZGTwArmDn3yu lKcK9pNRC1vw_f7N47_tQw5sS0U9efpGFuXEvWDJrNbog6TCyfqA33bhIF8Q iDbNZ.U51YZvwrfMPh4akfvPfEslBzG01Gbl01MCoRXTN5PPmTaIfcFDqxVQ w69ABHWTVg8P4eV26SYP2WWxLUa8flFQjomMclGgMo5gqO_iN7IUK6KDkxl8 dji4IEoE9oCxUuzP7qwTKkzMvifOKOEsILmMMAd8pLN7m94tlNr9SuyaBq_z axJYZcpohYPhDTJZaS2.8AoPgwuuAygS1d4AzMJBWht4HKlVtiXlA1xLvj7Y c.x5s__ZgNo23_j0jBw-- Received: from [174.48.128.27] by web121601.mail.ne1.yahoo.com via HTTP; Fri, 23 Nov 2012 07:52:51 PST X-Rocket-MIMEInfo: 001.001, CgotLS0gT24gRnJpLCAxMS8yMy8xMiwgQW5kcmUgT3BwZXJtYW5uIDxhbmRyZUBmcmVlYnNkLm9yZz4gd3JvdGU6Cgo.IEZyb206IEFuZHJlIE9wcGVybWFubiA8YW5kcmVAZnJlZWJzZC5vcmc.Cj4gU3ViamVjdDogUmU6IExvdyBCYW5kd2lkdGggb24gaW50ZXJjb250aW5lbnRhbCBjb25uZWN0aW9ucwo.IFRvOiAiSW5nbyBGbGFzY2hiZXJnZXIiIDxpZkB4aXAuYXQ.Cj4gQ2M6IGZyZWVic2QtbmV0QGZyZWVic2Qub3JnCj4gRGF0ZTogRnJpZGF5LCBOb3ZlbWJlciAyMywgMjAxMiwgMTA6MDUgQU0KPiBPbiABMAEBAQE- X-Mailer: YahooMailClassic/15.0.8 YahooMailWebService/0.8.123.460 Message-ID: <1353685971.91981.YahooMailClassic@web121601.mail.ne1.yahoo.com> Date: Fri, 23 Nov 2012 07:52:51 -0800 (PST) From: Barney Cordoba Subject: Re: Low Bandwidth on intercontinental connections To: Ingo Flaschberger , Andre Oppermann In-Reply-To: <50AF90D1.4020404@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2012 15:52:58 -0000 --- On Fri, 11/23/12, Andre Oppermann wrote: > From: Andre Oppermann > Subject: Re: Low Bandwidth on intercontinental connections > To: "Ingo Flaschberger" > Cc: freebsd-net@freebsd.org > Date: Friday, November 23, 2012, 10:05 AM > On 23.11.2012 15:46, Ingo > Flaschberger wrote: > > Am 23.11.2012 13:47, schrieb Marc Peters: > >> PR filed: http://www.freebsd.org/cgi/query-pr.cgi?pr=173859 > > > > Downloads from ftp1.us.freebsd.org to Europe (ping time > ~170ms), I see up & down ramping of transfer > > speed (600kb/sec - 50kb/sec) > > over serverall releases: > > FreeBSD 6.3-PRERELEASE > > FreeBSD 7.2-STABLE > > FreeBSD 8.2-STABLE > > FreeBSD 9.1-PRERELEASE > > > > Linux performs much better, max speed was 1.2Mb/sec, > and is not as much up & down ramping as freebsd. > > It's about 168ms for me and I don't see the problem here. > Speed is about 18.4Mbps. > > -- > Andre It's probably something with the window timeout. On a local connection, the acks come in before the window is fully sent (or very nearly) while on a distant connection the window is sent and then there is a longish delay/wait. This used to be a big issue on ftp servers back in the day. The algorithms for how to send stuff when the window left was less than a full packet were poorly done. BC From owner-freebsd-net@FreeBSD.ORG Fri Nov 23 16:03:46 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5C327770 for ; Fri, 23 Nov 2012 16:03:46 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id 9B2358FC0C for ; Fri, 23 Nov 2012 16:03:44 +0000 (UTC) Received: (qmail 1725 invoked from network); 23 Nov 2012 17:03:43 +0100 Received: from fw.xip.at (HELO ?127.0.0.1?) (89.207.145.147) by chile.gbit.at with SMTP; 23 Nov 2012 17:03:43 +0100 Message-ID: <50AF9E64.60405@xip.at> Date: Fri, 23 Nov 2012 17:03:48 +0100 From: Ingo Flaschberger User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: Re: Low Bandwidth on intercontinental connections References: <50ACF62C.8000408@mpeters.org> <50ad087d.1892cc0a.2cce.3bf2@mx.google.com> <50AD1012.7020209@mpeters.org> <50AD14F8.8050001@xip.at> <50ADE5E4.9090708@mpeters.org> <50AE0B12.8000309@xip.at> <50AE1CCC.7080706@mpeters.org> <50AE2686.8070007@xip.at> <50AE47D4.7080608@mpeters.org> <50AF7070.50206@mpeters.org> <50AF8C2A.7010006@xip.at> <50AF965D.5050301@mpeters.org> In-Reply-To: <50AF965D.5050301@mpeters.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2012 16:03:46 -0000 The sender dump shows, that the network-card does tso. From Marc's dump at receiver site: network io bits/sec: FreeBSD: http://devel.crossip.net/freebsd_receiver.jpg Linux: http://devel.crossip.net/linux_receiver.jpg During the "breaks" between the transfer, I see tcp retransmissions (but slow). window scaling: FreeBSD: http://devel.crossip.net/freebsd_window_size.jpg Linux: http://devel.crossip.net/linux_window_size.jpg But window scaling keeps time sequence graph (stevens): FreeBSD: http://devel.crossip.net/freebsd_time_sequence_graph_stevens.jpg Linux: http://devel.crossip.net/linux_time_sequence_graph_stevens.jpg round trip time: FreeBSD: http://devel.crossip.net/freebsd_round_trip_time.jpg Linux: http://devel.crossip.net/linux_round_trip_time.jpg Kind regards, Ingo Flaschberger From owner-freebsd-net@FreeBSD.ORG Fri Nov 23 18:52:49 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C50D94D4 for ; Fri, 23 Nov 2012 18:52:49 +0000 (UTC) (envelope-from jason.lee.campbell@gmail.com) Received: from mail-ia0-f182.google.com (mail-ia0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 8C6518FC0C for ; Fri, 23 Nov 2012 18:52:49 +0000 (UTC) Received: by mail-ia0-f182.google.com with SMTP id x2so8364920iad.13 for ; Fri, 23 Nov 2012 10:52:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=7l8vYgVWB3sqgLavAu8A/XCw3Cvo5/HGOhgB6u/baUQ=; b=jUdepxASVZAZaDSkUbmqlCB6fvW+VvwHB8wxKdBfgRUj+zuOC7PuSgXu3ZV83AhRzt OeHGmAOyH1MOjzXSuMOW1yTBVUJhV5A87zmLysCxUxkLf/FbR1yCUSeShAXDBEpW++/p stxeFy9GJ4sIqbjcjBKTGlKHlq72b47IkocbdzvVGdKgPkMr8+EeikYsUuOqzgFy58Em 4Ni0EOGMUB7oJoMFLfH+ZhH5nex+dV9c2FUm5vI+qDxbEqQLSJeCig4nOb4HdSlexfoj Ovhwo5Fw+9yoNs+UpMMUd9VEmNHhK7wGJlJSVdUTxjn/JX2v071cDyC9syn+m8RRrzA1 s+cg== MIME-Version: 1.0 Received: by 10.43.14.135 with SMTP id pq7mr3877221icb.8.1353696768803; Fri, 23 Nov 2012 10:52:48 -0800 (PST) Received: by 10.64.50.103 with HTTP; Fri, 23 Nov 2012 10:52:48 -0800 (PST) Date: Fri, 23 Nov 2012 13:52:48 -0500 Message-ID: Subject: FreeBSD 9.1: RFC 5569 (6rd) support in stf(4) From: Jason Campbell To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2012 18:52:49 -0000 I just noticed my ISP appears to support IPv6 via 6rd. Using FreeBSD 9.1-PRERELEASE as my router, so need 6rd support. I saw port net/stf-6rd-kmod, but says it only supports 8. I'm thinking it would work for 9 also if the makefile was patched to support 9. However, found a patch at http://people.allbsd.org/~hrs/FreeBSD/stf_6rd_20100923-1.diff and applied it, with only one failure (line 30ish of man/stf.4, the date line). Questions for the more experienced, which is the better approach, try the port or apply the patch, or are they pretty much the same? And, will 6rd be added into the main source? Or have I totally messed things up by applying a patch that is over 2 years old? Jason From owner-freebsd-net@FreeBSD.ORG Sat Nov 24 02:09:47 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A72871C4 for ; Sat, 24 Nov 2012 02:09:47 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id E2B978FC16 for ; Sat, 24 Nov 2012 02:09:45 +0000 (UTC) Received: (qmail 19767 invoked from network); 24 Nov 2012 03:09:43 +0100 Received: from fw.xip.at (HELO ?127.0.0.1?) (89.207.145.147) by chile.gbit.at with SMTP; 24 Nov 2012 03:09:43 +0100 Message-ID: <50B02C6C.9090507@xip.at> Date: Sat, 24 Nov 2012 03:09:48 +0100 From: Ingo Flaschberger User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: Re: Low Bandwidth on intercontinental connections References: <50ACF62C.8000408@mpeters.org> <50ad087d.1892cc0a.2cce.3bf2@mx.google.com> <50AD1012.7020209@mpeters.org> <50AD14F8.8050001@xip.at> <50ADE5E4.9090708@mpeters.org> <50AE0B12.8000309@xip.at> <50AE1CCC.7080706@mpeters.org> <50AE2686.8070007@xip.at> <50AE47D4.7080608@mpeters.org> <50AF7070.50206@mpeters.org> <50AF8C2A.7010006@xip.at> <50AF965D.5050301@mpeters.org> <50AF9E64.60405@xip.at> In-Reply-To: <50AF9E64.60405@xip.at> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Nov 2012 02:09:47 -0000 The last linux graphs are wrong - sorry! (Thanks for the hint, much to low RTT and small window) I graphed the wrong direction of the session. Now updated graphs: network io bits/sec: FreeBSD: http://devel.crossip.net/freebsd_receiver.jpg Linux: http://devel.crossip.net/linux_receiver.jpg window scaling: FreeBSD: http://devel.crossip.net/freebsd_window_size.jpg Linux: http://devel.crossip.net/linux_window_size.jpg time sequence graph (stevens): FreeBSD: http://devel.crossip.net/freebsd_time_sequence_graph_stevens.jpg Linux: http://devel.crossip.net/linux_time_sequence_graph_stevens.jpg round trip time: FreeBSD: http://devel.crossip.net/freebsd_round_trip_time.jpg Linux: http://devel.crossip.net/linux_round_trip_time.jpg Kind regards, Ingo Flaschberger