From owner-freebsd-questions@FreeBSD.ORG Tue Jun 14 20:40:54 2005 Return-Path: X-Original-To: freebsd-questions@freebsd.org Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D521416A41C for ; Tue, 14 Jun 2005 20:40:54 +0000 (GMT) (envelope-from ken@idealinter.net) Received: from mail.idealinter.net (mail.idealinter.net [72.242.8.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 983E443D48 for ; Tue, 14 Jun 2005 20:40:54 +0000 (GMT) (envelope-from ken@idealinter.net) Received: from [192.168.0.2] (unknown [72.242.8.254]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mail.idealinter.net (Postfix) with ESMTP id ADA3610E42C for ; Tue, 14 Jun 2005 16:40:53 -0400 (EDT) Mime-Version: 1.0 (Apple Message framework v730) Content-Transfer-Encoding: 7bit Message-Id: <92276D37-99E1-4909-8283-C2A60E280B3C@idealinter.net> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed To: freebsd-questions@freebsd.org From: Ken Ebling Date: Tue, 14 Jun 2005 16:40:52 -0400 X-Mailer: Apple Mail (2.730) Subject: Stability issues with FreeBSD + MySQL X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jun 2005 20:40:54 -0000 Hello everyone, I've been experiencing some issues with FreeBSD + MySQL and I was hoping someone on this list might have some advice. I have two servers that run a web site that receives about 650,000 hits per day (about 15,000 unique visits/day). Both the machines have dual Xeon 3.06 GHz chips with 2GB ECC REG RAM, and RAID Arrays (The web server has an Adaptec 2100S U160 controller, and the database server has an Adaptec 2120S U320 controller.) Both machines have a 2-disk raid-1 array for the OS, and a raid-5 array for the data. The web server is running FreeBSD 5.3 with Apache+mod_perl (1.3.33_2, installed from ports), mysql-client (4.0.24_1, installed from ports, compiled with LinuxThreads), and uses Postfix as an MTA. The database server is running FreeBSD 4.11 (had all sorts of problems with MySQL after upgrading to 5.x) with mysql-server (4.0.24_1, installed from ports, compiled with LinuxThreads) The database server isn't connected to the internet, the web server connects to it over a gigabit cross-over link. Both the machines have custom kernels, that are basically GENERIC with all the unnecessary hardware (SCSI controllers, RAID controllers, NICs, USB & Firewire) stripped out, and with SMP enabled. The web server uses an Intel Pro/100 (fxp) for it's external interface and an Intel Pro/1000 (em) for it's connection to the database server, which also has an Intel Pro/1000. The problem I'm having is that a few times a day, a perl script on the web server will fail to connect to the mysql server, and send out an email letting me know that the connection and whatever query it was going to execute have failed. Whenever this happens I check the machines out and everything looks okay to me, but I am obviously overlooking something. Here are some relevant settings on the machines: /etc/sysctl.conf on web server: kern.ipc.somaxconn=2048 /etc/sysctl.conf on database server: kern.ipc.somaxconn=1024 /boot/loader.conf on web server: kern.ipc.nmbclusters="65536" /boot/loader.conf on database server: kern.ipc.nmbclusters="32768" netstat -m on web server: 4294928101 mbufs in use 2516/65536 mbuf clusters in use (current/max) 0/3/6656 sfbufs in use (current/peak/max) 4189537 KBytes allocated to network 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 137 calls to protocol drain routines netstat -m on database server: 450/592/131072 mbufs in use (current/peak/max): 450 mbufs allocated to data 448/570/32768 mbuf clusters in use (current/peak/max) 1288 Kbytes allocated to network (1% of mb_map in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines netstat -na | grep tcp | wc -l on web server: 402 netstat -na | grep udp | wc -l on web server: 11 netstat -na | grep tcp | wc -l on database server: 34 netstat -na | grep udp | wc -l on database server: 6 vmstat on webserver: procs memory page disks faults cpu r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy id 2 2 0 408776 1172772 2447 0 0 0 1958 21 0 0 3257 11668 5447 15 5 79 vmstat on database server: procs memory page disks faults cpu r b w avm fre flt re pi po fr sr aa0 md0 in sy cs us sy id 1 0 0 358932 163800 40 0 0 0 309 192 0 0 1660 27525 992 7 8 85 vmstat -i on web server: interrupt total rate irq1: atkbd0 558 0 irq6: fdc0 9 0 irq8: rtc 11062204 127 irq13: npx0 1 0 irq16: fxp0 11129226 128 irq48: asr0 891086 10 irq54: em0 37098604 429 irq0: clk 86418736 999 Total 146600424 1696 vmstat -i on database server: interrupt total rate em0 irq11 50726686 501 aac0 irq16 3114675 30 fxp0 irq10 6 0 fdc0 irq6 1 0 atkbd0 irq1 1 0 clk irq0 101253713 1000 rtc irq8 12961771 128 Total 168056853 1660 The database server is using the default /usr/local/share/mysql/my- large.cnf config file (my-huge.cnf always makes mysql act buggy) with binary logging disabled. Is there any special tuning or something else I am missing that would be appropriate for me to try? Thanks in advance, Ken Ebling