From owner-freebsd-net@freebsd.org Thu Feb 4 13:50:17 2016 Return-Path: <owner-freebsd-net@freebsd.org> Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 59D7FA9C6A2; Thu, 4 Feb 2016 13:50:17 +0000 (UTC) (envelope-from honzhan@microsoft.com) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1on0138.outbound.protection.outlook.com [157.56.110.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A53E51D6; Thu, 4 Feb 2016 13:50:15 +0000 (UTC) (envelope-from honzhan@microsoft.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector1; h=From:To:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=KUyW6GgtJ2QiBWtpqKpCa2vK5VTlhJsDOdgaXOcbTM4=; b=lInxb4OiHPqEe41gHOhhHGTeiNQgrvzY+0oCHcix1wDreEXmHber3gL/8mfb/arp4qQNQbLGB7SD8YhmjiqfMq8cp1umHYUHsZUmv8CyMb2My2ZG3QAvgWSaFZL1h+c8sQa176o0wyvT3y4zsBf/ntdrvUr9iQRF5X68CA6fWKQ= Received: from BY2PR03CA007.namprd03.prod.outlook.com (10.255.93.24) by BLUPR0301MB1540.namprd03.prod.outlook.com (10.162.213.158) with Microsoft SMTP Server (TLS) id 15.1.396.15; Thu, 4 Feb 2016 13:35:08 +0000 Received: from BL2FFO11FD029.protection.gbl (10.255.93.4) by BY2PR03CA007.outlook.office365.com (10.255.93.24) with Microsoft SMTP Server (TLS) id 15.1.403.16 via Frontend Transport; Thu, 4 Feb 2016 13:35:07 +0000 Authentication-Results: spf=pass (sender IP is 206.191.228.180) smtp.mailfrom=microsoft.com; hob.de; dkim=none (message not signed) header.d=none;hob.de; dmarc=pass action=none header.from=microsoft.com; Received-SPF: Pass (protection.outlook.com: domain of microsoft.com designates 206.191.228.180 as permitted sender) receiver=protection.outlook.com; client-ip=206.191.228.180; helo=064-smtp-out.microsoft.com; Received: from 064-smtp-out.microsoft.com (206.191.228.180) by BL2FFO11FD029.mail.protection.outlook.com (10.173.160.69) with Microsoft SMTP Server (TLS) id 15.1.409.7 via Frontend Transport; Thu, 4 Feb 2016 13:35:05 +0000 Received: from SG2PR3002MB0106.064d.mgd.msft.net (141.251.56.18) by SG2PR3002MB0107.064d.mgd.msft.net (141.251.56.19) with Microsoft SMTP Server (TLS) id 15.1.403.10; Thu, 4 Feb 2016 13:35:02 +0000 Received: from SG2PR3002MB0106.064d.mgd.msft.net ([141.251.56.18]) by SG2PR3002MB0106.064d.mgd.msft.net ([141.251.56.18]) with mapi id 15.01.0403.016; Thu, 4 Feb 2016 13:35:02 +0000 From: Hongjiang Zhang <honzhan@microsoft.com> To: "Meyer, Wolfgang" <wolfgang.meyer@hob.de>, "'freebsd-net@FreeBSD.org'" <freebsd-net@FreeBSD.org> CC: "'freebsd-performance@FreeBSD.org'" <freebsd-performance@FreeBSD.org> Subject: RE: ixgbe: Network performance tuning (#TCP connections) Thread-Topic: ixgbe: Network performance tuning (#TCP connections) Thread-Index: AdFefq3DO+J8ScSLR4uotZFUcfP67wAG7Oyw Date: Thu, 4 Feb 2016 13:35:02 +0000 Message-ID: <af25d963881e4080ab5a946c05aee077@SG2PR3002MB0106.064d.mgd.msft.net> References: <EC88118611AE564AB0B10C6A4569004D0137D57AEB@HOBEX11.hob.de> In-Reply-To: <EC88118611AE564AB0B10C6A4569004D0137D57AEB@HOBEX11.hob.de> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [180.170.80.203] X-MS-Office365-Filtering-Correlation-Id: f5ac5343-39fd-4d76-0e69-08d32d67fe55 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-Microsoft-Exchange-Diagnostics: 1; BL2FFO11FD029; 1:vUwuqiTdPhCBH/XtzJYIA9Lcwlqcx4J+2VfiOOZbCWa16t//G/SVZY1ge4wvI+b/7quVVogkTm8KTaS747XU/pBPb0umas9S4ItG0wBr5CEfmBYQsZA43ZKdYJLGe8jIVknyO1uS4GIs0vqaKFhLp8tH/+Vx6Jkrk0pSGV3Yj8NBjC5uAMHoX24864ovKA4lsKATLMf4cmjLEc5Ghu1z1n01X+C1RAhoWlEflcGMAkWmj9Y+4pLkxvjwuNUs2gpu12j2mi9At2W0OXJnfIBBVuN1TjA5Htt8MT3SFPZJesugRqC6Ni/hGMpYSytAK+6yCFywU4d+GbnxWNl/rbxWqA== X-Forefront-Antispam-Report: CIP:206.191.228.180; CTRY:US; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(10019020)(6009001)(2980300002)(438002)(13464003)(199003)(3905003)(189002)(377454003)(16796002)(50466002)(102836003)(23726003)(1220700001)(1096002)(586003)(3846002)(6116002)(24736003)(10290500002)(10400500002)(189998001)(5001960100002)(5005710100001)(5001770100001)(86612001)(106466001)(97756001)(6806005)(5008740100001)(92566002)(19580395003)(46406003)(108616004)(19580405001)(86146001)(11100500001)(47776003)(87936001)(54356999)(575784001)(86362001)(76176999)(2950100001)(33646002)(66066001)(2900100001)(2906002)(4326007)(5003600100002)(15975445007)(50986999)(491001); DIR:OUT; SFP:1102; SCL:1; SRVR:BLUPR0301MB1540; H:064-smtp-out.microsoft.com; FPR:; SPF:Pass; MLV:sfv; A:1; MX:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; BLUPR0301MB1540; 2:fTAqvAk/I2b1QK63qoglNfRDa+RFRPq1+cOHwzc74/AqmuNyAypB3oLDIanl/sxQX/0sTg1iiFXsmCLtsr2WcyTcZ8B4gRg9ShZOZURo5feR4QLCt4YDGwj0XW4XP9RmxOtq6qc2IdT9BddoyaErR6QMBUUgkAGnJ0W/+Jj0OaZODhNFeuAd3Db65+DGoCG3; 3:ttkFRxccojWw+0YPG/S6P4geHLpqOpTxjczPntTt5qw1SfZQLU35dSkIcXiSuPAl+gdM/emDwZDXS5dBRTkqZgz1tUyj97HBBogqOxb4CksHa9yIWjGw5tVcXyWbKJNVQIjyuRyPYoQp/g5IqY3TPlI9ybs7Wmyx5ByYBC75ZbK/RrccxI7NuRsFDqDwH6f5IVeuiHpXuVoQ7Yxk/VFCNN0ggCCKA4etuq1pmjSTFoM8Ig4twccfckVqXFHWbvvaezEN3tBQwvTz3SBtCkUP1A==; 25:TZadMu1vzuSyQyHqrEZ4kuPsNeI6Sgaa0Fm19J7S9vGFm8+IkTX1Av+EMGvEeFtRtFE0dcokcoEdFvGsEzhVIexwOPoC4G3oJO0QRBwwMgL2lqgO7x78Ez5uKrWTa9v0Y56vROGex4f76vrdDSpiMWJ1CBP48JGZhJSncP/e+TfCsQI5Q4SAQV61TU/sk46k2FEY4OStE4U8TtS0IFOm9Rkzs0hWnDiWlwsi/2IpJD6MoGX4/QYPA/exgVMsbhHUW+1cSW2UbAyGmFQNFqbveDGcpA8jDGWTT8pUt7i/Nz4YduOsmHCVHwUN0I0N6LW4q0jQef/2NI3DmTHxD2uwTw== X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(8251501002); SRVR:BLUPR0301MB1540; X-O365EOP-Header: O365_EOP: AllowList from IP - set SCL to -1 X-Microsoft-Exchange-Diagnostics: 1; BLUPR0301MB1540; 20:TztFL3ECnvk8h05f4nG4adDZnerx5rWM5Gk4kHCE8lDJsArnq2zAWGNWLwLmQmQYyIrWjHkUQ//HvRZUdTHO0hZO6yiQNKgP05Il41wZk0RiaK06hghnAbIyC270qW0o8WL5A0lEM5QDMK7htD/MqRIQ6xrqA7UjOsmSDrvP9U9QvIZRcLgZA8+LkNLRMKlceuLNNzNFtodguD0R8d1r55DHRPPiVWHEJDavChiiZ8ZxfFb8u13J+ucCuHsG0Nkqr5+Dl9Ssdumi3ql5tYxXhSEamZzH+NTy1JzaqIv/JAPgdkOEspzf5Y3e+aOj09a4k1urDAfwyVz1+7ZvxCdLIlWY2zG7hjXmdq2PxmMrU4+beMylHNNerdSJE6Gw3is0J0Hgzu5NGqwpox+otqzhEeOkmpKWafec8624eHsYndORIOcbGCzI/ESQ8qGp1b2tM82528NG+igOUFKW1Jakn4dVKxM5ZAXUedXarwQlZ7nKZmrvjSWJMY8aL6eSrb+6ro1ExHvQeJsMWAfRS0FuX/K6pgKcws96BsRimTEYgqpNbegF25UITHBJa30RVgjTx6x7NZuKa2Q4KwzV2XBWnhQOq1EtGTtD1af77G71WzU= X-Microsoft-Antispam-PRVS: <BLUPR0301MB154075369F6D66FB4791C6E3B5D10@BLUPR0301MB1540.namprd03.prod.outlook.com> X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(61425038)(601004)(2401047)(5005006)(13016025)(8121501046)(13018025)(10201501046)(3002001)(61426038)(61427038); SRVR:BLUPR0301MB1540; BCL:0; PCL:0; RULEID:; SRVR:BLUPR0301MB1540; X-Microsoft-Exchange-Diagnostics: 1; BLUPR0301MB1540; 4:f36l2FIMlY0gmFvIU/4/RImzjeCOgrW58HHFWCS0ClQfS/PKBZF7lnYLRT8GGBEP/P/zKP6XNhixcwKEYVmt+Squ+9sr1hxwcQfcFNkdWOTrEmaIjgcxuyH8RQErj/Se77KnVcvWdU66TXk3EqrVkQRTj0fcAlAIZgpQm/zYSD/iShmMc0fHb93SNP5Mw0YrIQ3Rk4XgebsxHhET9wTziRPpg6GmP1GJGdIShju+MzFRAFw/WTrCGIoyq56mFOmqd6Mhg5uoaMW1WKa4LC0qgLxqGqujJpOQEG0OJujparwNcx1oaECnB1hEMmsHC7C64vqHbB9fM7Gy16Nmp5Yd0eieQusincmrHHn0Hq15ckOjb9lSx4aYdd+PZC5VBhvuzGw/vl1gqg3Te3B7jNUbRy24/3IQIe6VIxBYLz80T/D9+d/NBc4WQls8FMTJH8SJTcaWzHYpOclXGi91glPY5Q== X-Forefront-PRVS: 084285FC5C X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BLUPR0301MB1540; 23:7DVRo5D31+pBpdQgTaqjuJd5ZScbYpVuyromR38?= =?us-ascii?Q?MuQ2X4rCNgulBPoTUwV27n1OABmzNKb2C35KJfDhg3tEkJE0AKTse4UxecN0?= =?us-ascii?Q?UUEH4TBf8aD/+4gTJpsSz6G6Y1SaEpF0bIeFOnVec9TGCr7hYvZy+elf5op/?= =?us-ascii?Q?n+MpizpUG2QW8B9NmKEoO0tHcxx79p/4ZiprSqZQqScqRvFciGTfRHIZaHUG?= =?us-ascii?Q?UNCZvktZvGWjIww0LG2BzgWT2zFsdUUHR+7JyoWHTUTz8yFrtbIW8ZNgD6Ep?= =?us-ascii?Q?D3ET9YOOihhu1moE3luD+H6jlj6+cG4OodMH+OcAyxDqPOLXu0iuflfOxY3Y?= =?us-ascii?Q?mYMcPSJlQaOpkLln+OhPOHugUfrDkOBpnf9Bay6+O6in59YxVEh3rfJWO5vX?= =?us-ascii?Q?VptNAT9POb0wXlwaa1A2f2OAIb7fw4Vp/uuF6gYvRX1FHR4ybR0AfkYlAoNK?= =?us-ascii?Q?PtA7VwFdW/gq6mJN/jrvN2vyoXwh9OVGixInAFwGwrUnZLyV/OYcvIbZrfFD?= =?us-ascii?Q?p6uLQmfiyC/uNwrPkenQPjKXT7f/4HgAwJ7yS5dPazSMwLOJracwtaDdDaRL?= =?us-ascii?Q?38eAAslWepgNnjBbJu/kb5lnuoCletzHJCFJhKKzboGMa3Lz1PT+ssNpmo8l?= =?us-ascii?Q?rjJkPBTKY7M80qbkVO/94TuzMOK6yyPflZfR1zfsJEMcNa8yL8755hHkVDLV?= =?us-ascii?Q?gYvYVkOx/nBaRp9Et4DS0EwHIKePPFJUXoWCJbj4ghh8iS+UL1avv7dgaCL0?= =?us-ascii?Q?IzOK2H2JNsVssAnZi+/gP5HzMPGPh55Ot4IXbjSSJ4CETWIBDKSuIKHuBgBa?= =?us-ascii?Q?61i47BVWV/rXmQL9wAPZj2yGbGCyysuIpYfS60Buw1hgNTOzQxJeybTGDjcZ?= =?us-ascii?Q?sFsy7Xdas0B1ScObeUSkzyYDcZgi5+L6j0xG1DfmwBCnGpWgEQutQhaSng/4?= =?us-ascii?Q?pOPaZfX82qsRPaXF9AFRpGIikwMHqoku73ISLNmo7hFXH7sU/AimbmjhNVjT?= =?us-ascii?Q?3iNac3Xyw1U/4QGI2lmJrCS/lheQgKIMMFY9ImEO9U6sBzBMdLXqirOJuTmh?= =?us-ascii?Q?9WudF1Pc7Z/Pgkdt2spw4D8Y/APp7qmEzUPtkymTQn+k6JG4bcUbNMuHjrHE?= =?us-ascii?Q?2erD+lBkMnKzFXUnciSxV7vNwiQ8OTdFyiSKfrlcbyDkquOan7St3zgjpkej?= =?us-ascii?Q?o0qoMQY+ivLSflYTErJGA9qWX1UO3VZeVXIQCujjotTa94HZy7L5oIF1oY4J?= =?us-ascii?Q?yC2Uk1XOv2wPNSLH/tr/d88upGMn7uLjAHHPwoze8mfn8gPo4kbc7Or4h84G?= =?us-ascii?Q?010ljcaNa5Nl3hVZXKWpZd8Iq7QpR2QvUtJLo524Xvc88?= X-Microsoft-Exchange-Diagnostics: 1; BLUPR0301MB1540; 5:eDsOs3g22CnTop9Z/8cL1SXS8cRoBVrP6Pz7L/c3NCn1+b7r0FEpKJNP8QfRmcRWfzcj8WR64AX8SbDprAkUbGQAE0KTT/3uQEORhw3PcMhGgq9lu2PCT56URc1cx9vMHN2DtJ8dwyQNHyGFYbmA/g==; 24:jj6/YCkK79ROyDQLGNQ8t4ibspl5IgAlY0JLig/OkXBvGN5vSHdAIKV5QrrfB+YAcOM1GygPIJBeThwEoeU59G/KgAKi9k3u+zbytfrGEFA= SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Feb 2016 13:35:05.6716 (UTC) X-MS-Exchange-CrossTenant-Id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=72f988bf-86f1-41af-91ab-2d7cd011db47; Ip=[206.191.228.180]; Helo=[064-smtp-out.microsoft.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLUPR0301MB1540 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org> List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/> List-Post: <mailto:freebsd-net@freebsd.org> List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help> List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=subscribe> X-List-Received-Date: Thu, 04 Feb 2016 13:50:17 -0000 Did you enable LRO on FreeBSD side (check 'ifconfig' output)? Linux default= enables GRO (see the output of 'ethtool -k eth0'). Thanks Hongjiang Zhang -----Original Message----- From: owner-freebsd-net@freebsd.org [mailto:owner-freebsd-net@freebsd.org] = On Behalf Of Meyer, Wolfgang Sent: Wednesday, February 3, 2016 9:37 PM To: 'freebsd-net@FreeBSD.org' <freebsd-net@FreeBSD.org> Cc: 'freebsd-performance@FreeBSD.org' <freebsd-performance@FreeBSD.org> Subject: ixgbe: Network performance tuning (#TCP connections) Hello, we are evaluating network performance on a DELL-Server (PowerEdge R930 with= 4 Sockets, hw.model: Intel(R) Xeon(R) CPU E7-8891 v3 @ 2.80GHz) with 10 Gb= E-Cards. We use programs that on server side accepts connections on a IP-ad= dress+port from the client side and after establishing the connection data = is sent in turns between server and client in a predefined pattern (server = side sends more data than client side) with sleeps in between the send phas= es. The test set-up is chosen in such way that every client process initiat= es 500 connections handled in threads and on the server side each process r= epresenting an IP/Port pair also handles 500 connections in threads. The number of connections is then increased and the overall network througp= ut is observed using nload. On FreeBSD (on server side) roughly at 50,000 c= onnections errors begin to occur and the overall throughput won't increase = further with more connections. With Linux on the server side it is possible= to establish more than 120,000 connections and at 50,000 connections the o= verall throughput ist double that of FreeBSD with the same sending pattern.= Furthermore system load on FreeBSD is much higher with 50 % system usage o= n each core and 80 % interrupt usage on the 8 cores handling the interrupt = queues for the NIC. In comparison Linux has <10 % system usage, <10 % user = usage and about 15 % interrupt usage on the 16 cores handling the network i= nterrupts for 50,000 connections. Varying the numbers for the NIC interrupt queues won't change the performan= ce (rather worsens the situation). Disabling Hyperthreading (utilising 40 c= ores) degrades the performance. Increasing MAXCPU to utilise all 80 cores w= on't improve compared to 64 cores, atkbd and uart had to be disabled to avo= id kernel panics with increased MAXCPU (thanks to Andre Oppermann for inves= tigating this). Initiallly the tests were made on 10.2 Release, later I swi= tched to 10 Stable (later with ixgbe driver version 3.1.0) but that didn't = change the numbers. Some sysctl configurables were modified along the network performance guide= lines found on the net (e.g. https://na01.safelinks.protection.outlook.com/= ?url=3Dhttps%3a%2f%2fcalomel.org%2ffreebsd_network_tuning.html%2c&data=3D01= %7c01%7chonzhan%40064d.mgd.microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12= %7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=3DxsMoC%2b1ZcnoHBnPqhLUMDIr8V= LBcLejnrXgkRyDWzYc%3d https://na01.safelinks.protection.outlook.com/?url=3D= https%3a%2f%2fwww.freebsd.org%2fdoc%2fhandbook%2fconfigtuning-kernel-limits= .html%2c&data=3D01%7c01%7chonzhan%40064d.mgd.microsoft.com%7cf827a05328ca4c= a9781608d32c9f5b12%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=3DXNqvrYfTN= zfe2btrip%2f5FoX3iTTpTSbNrDjbhtVBevo%3d https://na01.safelinks.protection.o= utlook.com/?url=3Dhttps%3a%2f%2fpleiades.ucsc.edu%2fhyades%2fFreeBSD_Networ= k_Tuning&data=3D01%7c01%7chonzhan%40064d.mgd.microsoft.com%7cf827a05328ca4c= a9781608d32c9f5b12%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=3D%2bQ66X%2= frnqNakX%2fSGcK08QTTrsDjUUWBHOXu6%2fOBIBNQ%3d) but most of them didn't have= any measuarable impact. Final sysctl.conf and loader.conf settings see bel= ow. Actually the only tunables that provided any improvement were identifie= d to be hw.ix.txd, and hw.ix.rxd that were reduced (!) to the minimum value= of 64 and hw.ix.tx_process_limit and hw.ix.rx_process_limit that were set = to -1. Any ideas what tunables might be changed to get a higher number of TCP conn= ections (it's not a question of the overall throughput as changing the send= ing pattern allows me to fully utilise the 10Gb bandwidth)? How can I deter= mine where the kernel is spending its time that causes the high CPU load? A= ny pointers are highly appreciated, I can't believe that there is such a bl= atant difference in network performance compared to Linux. Regards, Wolfgang <loader.conf>: cc_htcp_load=3D"YES" hw.ix.txd=3D"64" hw.ix.rxd=3D"64" hw.ix.tx_process_limit=3D"-1" hw.ix.rx_process_limit=3D"-1" hw.ix.num_queues=3D"8" #hw.ix.enable_aim=3D"0" #hw.ix.max_interrupt_rate=3D"31250" #net.isr.maxthreads=3D"16" <sysctl.conf>: kern.ipc.soacceptqueue=3D1024 kern.ipc.maxsockbuf=3D16777216 net.inet.tcp.sendbuf_max=3D16777216 net.inet.tcp.recvbuf_max=3D16777216 net.inet.tcp.tso=3D0 net.inet.tcp.mssdflt=3D1460 net.inet.tcp.minmss=3D1300 net.inet.tcp.nolocaltimewait=3D1 net.inet.tcp.syncache.rexmtlimit=3D0 #net.inet.tcp.syncookies=3D0 net.inet.tcp.drop_synfin=3D1 net.inet.tcp.fast_finwait2_recycle=3D1 net.inet.tcp.icmp_may_rst=3D0 net.inet.tcp.msl=3D5000 net.inet.tcp.path_mtu_discovery=3D0 net.inet.tcp.blackhole=3D1 net.inet.udp.blackhole=3D1 net.inet.tcp.cc.algorithm=3Dhtcp net.inet.tcp.cc.htcp.adaptive_backoff=3D1 net.inet.tcp.cc.htcp.rtt_scaling=3D1 net.inet.ip.forwarding=3D1 net.inet.ip.fastforwarding=3D1 net.inet.ip.rtexpire=3D1 net.inet.ip.rtminexpire=3D1 ________________________________ Follow HOB: - HOB: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f%2fww= w.hob.de%2fredirect%2fhob.html&data=3D01%7c01%7chonzhan%40064d.mgd.microsof= t.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d7cd011db47= %7c1&sdata=3D%2fenM%2fs72BvfP9H6CnrFeXqhrZoetovqoIMB%2bk0RFfQM%3d - Xing: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f%2fw= ww.hob.de%2fredirect%2fxing.html&data=3D01%7c01%7chonzhan%40064d.mgd.micros= oft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d7cd011db= 47%7c1&sdata=3DSi0aQmWV%2ba%2fWx5%2f6tZtOWQDj5cmq9t57DhA2h0qsa7Y%3d - LinkedIn: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f= %2fwww.hob.de%2fredirect%2flinkedin.html&data=3D01%7c01%7chonzhan%40064d.mg= d.microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d= 7cd011db47%7c1&sdata=3DBc0iX1u2twaazr%2fzi2wuTRqTZOk2rwtu61lNqJrui14%3d - HOBLink Mobile: https://na01.safelinks.protection.outlook.com/?url=3Dhttp= %3a%2f%2fwww.hob.de%2fredirect%2fhoblinkmobile.html&data=3D01%7c01%7chonzha= n%40064d.mgd.microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f= 141af91ab2d7cd011db47%7c1&sdata=3DI1ViplWsQ2HMWM%2fLnhWRKseziJBoNlyVBj2wiWF= x1wM%3d - Facebook: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f= %2fwww.hob.de%2fredirect%2ffacebook.html&data=3D01%7c01%7chonzhan%40064d.mg= d.microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d= 7cd011db47%7c1&sdata=3D1WliqTPX5YN3AdvK6DzAB7yDkQmjyC3jh%2f47PZ2uU7Y%3d - Twitter: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f%= 2fwww.hob.de%2fredirect%2ftwitter.html&data=3D01%7c01%7chonzhan%40064d.mgd.= microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d7c= d011db47%7c1&sdata=3DgaLO9OTaBis4F3IGoFY55nwMIOGPaZ0ri%2fK7N7nx7kI%3d - YouTube: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f%= 2fwww.hob.de%2fredirect%2fyoutube.html&data=3D01%7c01%7chonzhan%40064d.mgd.= microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d7c= d011db47%7c1&sdata=3DT%2fhNogZgLHoTCOj%2fKjsSJhPDlBqyxCAD8tJj5fueiqw%3d - E-Mail: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f%2= fwww.hob.de%2fredirect%2fmail.html&data=3D01%7c01%7chonzhan%40064d.mgd.micr= osoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d7cd011= db47%7c1&sdata=3DQnWAw2culSVFforLRWYgXfyHaj4PyB4Hn1rj4jVQvjU%3d HOB GmbH & Co. KG Schwadermuehlstr. 3 D-90556 Cadolzburg Geschaeftsfuehrung: Klaus Brandstaetter, Zoran Adamovic AG Fuerth, HRA 5180 Steuer-Nr. 218/163/00107 USt-ID-Nr. DE 132747002 Komplementaerin HOB electronic Beteiligungs GmbH AG Fuerth, HRB 3416 ______= _________________________________________ freebsd-net@freebsd.org mailing list https://na01.safelinks.protection.outlook.com/?url=3Dhttps%3a%2f%2flists.fr= eebsd.org%2fmailman%2flistinfo%2ffreebsd-net&data=3D01%7c01%7chonzhan%40064= d.mgd.microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91= ab2d7cd011db47%7c1&sdata=3DGU28hzKU5SSyA3hbP1PNtUNh7G5ut%2fMD6mdRmuJNkEs%3d To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"