From owner-freebsd-hackers@FreeBSD.ORG Wed Aug 11 15:25:43 2004 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 897DD16A4CE for ; Wed, 11 Aug 2004 15:25:43 +0000 (GMT) Received: from ktts.kharkov.ua (ktts.kharkov.ua [193.124.57.65]) by mx1.FreeBSD.org (Postfix) with ESMTP id 428A343D3F for ; Wed, 11 Aug 2004 15:25:42 +0000 (GMT) (envelope-from igor@ktts.kharkov.ua) Received: from ktts.kharkov.ua (smmsp@localhost [127.0.0.1]) by ktts.kharkov.ua (8.12.10/8.12.9) with ESMTP id i7BFPeTY014817 for ; Wed, 11 Aug 2004 18:25:41 +0300 (EEST) Received: (from igor@localhost) by ktts.kharkov.ua (8.12.10/8.12.9/Submit) id i7BFPdUf014815 for freebsd-hackers@freebsd.org; Wed, 11 Aug 2004 18:25:39 +0300 (EEST) Message-Id: <200408111525.i7BFPdUf014815@ktts.kharkov.ua> To: freebsd-hackers@freebsd.org Date: Wed, 11 Aug 2004 18:25:39 +0300 (EEST) From: igor@ktts.kharkov.ua X-Flames-To: /dev/null X-Header: /usr/include gives great headers X-Mailer: ELM [version 2.5 PL3] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Subject: 5.2.1 performance problem X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: igor@ktts.kharkov.ua List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Aug 2004 15:25:43 -0000 Folks, My box is Dell PowerEdge 4600 with: Dual Xeon 2.4GHz aac0 Dell PERC 3/Di with 10x 70Gb HDDs, em0 82544EI, bge0, fxp0, 6GB RAM of which only 4GB being used, This is a Usenet server running INN 2.4.1 (release) connected to Cisco GSR 12416 with Gigabit Ethernet em0; physical connection is 1000SX Full-Duplex. Yesterday I upgraded my news server from 5.1-p17 to 5.2.1-p9 and ran into a performance problem. Want to say from the beginning that during upgrade I transferred all my sysctl and other settings from 5.1 into 5.2.1 This server has about 50 NNTP peers (about 400 TCP connections) and traffic flow used to be at least 100mbit/160mbit in/out repectively. However after upgrade these number dropped down to 90/80mbit (input/output respectively), which is not normal for a Usenet server - now traffic is too low and output traffic must be about as twice higher than input traffic. I tried to do some FTP with nearby servers - along with news traffic it gave something like 380kbytes/sec transfer rate, so total output traffic doesn't exceed 100mbit/sec. I noticed that box started getting input errors on em0. In order to fix that I increased EM_MAX_TXD from 256 to 1024 and EM_MAX_RXD from 256 to 2048. After that Input errors disappeared but I continued getting intr_queue_drops counter increasing. I also observe full Send-Q for many TCP connections. I increased intr_queue_maxlen several times, from 400 up to 1200 with 200 increments, but it's still increasing. I changed kern.hz from 100 to 1000; also played with hw.em0.{tx,rx}_int_delay - neither one helped. Here are my modified sysctl values: kern.ipc.maxsockbuf=524288 net.inet.ip.intr_queue_maxlen=900 net.inet.tcp.sendspace=65536 net.inet.tcp.recvspace=131072 net.inet.tcp.tcbhashsize=16384 net.inet.tcp.rfc1323=1 net.isr.enable=1 net.inet.tcp.path_mtu_discovery=0 net.inet.tcp.delayed_ack=0 net.inet.tcp.newreno=0 net.inet6.ip6.accept_rtadv=1 Also from loader.conf: kern.ipc.nmbclusters=64512 kern.ipc.nmbufs=129024 net.inet.tcp.tcbhashsize=16384 kern.maxdsiz=1342177280 # deprecated in 5.2.1 if_em_load="YES" # made this driver as a module to play with tunables in if_em.h kern.hz="1000" My friend from OpenBSD core team advised to try to increase IFQ_MAXLEN from 50 to 500, but it didn't help too. If anybody has any ideas or suggestions, all are welcome and appreciated. -- Thanks and regards, Igor. If TCP/IP handshaking was less formal, perhaps SYN/ACK would be YO!/SUP! instead...