From owner-freebsd-smp@FreeBSD.ORG Tue Feb 7 12:39:11 2006 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2E2F416A420 for ; Tue, 7 Feb 2006 12:39:11 +0000 (GMT) (envelope-from astatech@mail.ru) Received: from f36.mail.ru (f36.mail.ru [194.67.57.74]) by mx1.FreeBSD.org (Postfix) with ESMTP id C7F9543D48 for ; Tue, 7 Feb 2006 12:39:10 +0000 (GMT) (envelope-from astatech@mail.ru) Received: from mail by f36.mail.ru with local id 1F6S7h-000HkR-00 for freebsd-smp@freebsd.org; Tue, 07 Feb 2006 15:39:09 +0300 Received: from [217.107.221.51] by win.mail.ru with HTTP; Tue, 07 Feb 2006 15:39:09 +0300 From: astatech@mail.ru To: freebsd-smp@freebsd.org Mime-Version: 1.0 X-Mailer: mPOP Web-Mail 2.19 X-Originating-IP: [217.107.221.51] Date: Tue, 07 Feb 2006 15:39:09 +0300 Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: 8bit Message-Id: Subject: 6.0 Stable on 8 Core AMD Opteron scalability. X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: astatech@mail.ru List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Feb 2006 12:39:11 -0000 Hi all! Please help solve problem. I have following server: - 8 cores at 2.2GHz/16GB RAM - FreeBSD 6.0 kernel compiled with SMP support for amd64 (4BSD scheduler, maxusers=2048, disabled unused hardware) - Apache, PHP, Mysql - 1GBit NIC (bge) I perform some benchmark and have very strange results with scalability. I start getting timeouts on client side after at 256 client threads, but CPU idle on server 30-40%. The top utility display that almost all httpd processes running on 7s core, and other 3-4 cores are idle. I am sure that bottleneck not on client side or server physical IO/memory. With 256 threads I get 30 Requests\sec on FreeBSD and 280 Requests\sec on Solaris (100% load). What can I do to utilize all CPU power on FreeBSD? WBR, Alex. From owner-freebsd-smp@FreeBSD.ORG Tue Feb 7 16:31:03 2006 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5917016A420 for ; Tue, 7 Feb 2006 16:31:03 +0000 (GMT) (envelope-from joseph.koshy@gmail.com) Received: from xproxy.gmail.com (xproxy.gmail.com [66.249.82.201]) by mx1.FreeBSD.org (Postfix) with ESMTP id E241343D49 for ; Tue, 7 Feb 2006 16:31:02 +0000 (GMT) (envelope-from joseph.koshy@gmail.com) Received: by xproxy.gmail.com with SMTP id s9so973168wxc for ; Tue, 07 Feb 2006 08:31:02 -0800 (PST) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=GGI6wqHuucf5nZknoZ3n5U8ws2EvsSRnKRZlfczSYNe29xmCXHNahtdqnjgBnPZ3d4sl8RMuhmHFtrHxgf0fjDv+2ZeR/tPUkYyWbn2mdldnuRDw9FmP8sL9WNOdgPgKgbxWnr7n4/eu/h7k+A5fzSfeXcIU0RF4iZGnTUie67Q= Received: by 10.70.31.8 with SMTP id e8mr8085857wxe; Tue, 07 Feb 2006 08:31:02 -0800 (PST) Received: by 10.70.105.2 with HTTP; Tue, 7 Feb 2006 08:31:02 -0800 (PST) Message-ID: <84dead720602070831g32e14257x16841f31e5b8527d@mail.gmail.com> Date: Tue, 7 Feb 2006 22:01:02 +0530 From: Joseph Koshy To: astatech@mail.ru In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline References: Cc: freebsd-smp@freebsd.org Subject: Re: 6.0 Stable on 8 Core AMD Opteron scalability. X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Feb 2006 16:31:03 -0000 a> - 8 cores at 2.2GHz/16GB RAM ... a> What can I do to utilize all CPU power on FreeBSD? 1) Which benchmark are you using? 2) Please post the output of dmesg(8). 3) Please post outputs from vmstat or any other high-level performance statistics that you've collected while the benchmark is running. -- FreeBSD Volunteer, http://people.freebsd.org/~jkoshy From owner-freebsd-smp@FreeBSD.ORG Wed Feb 8 12:46:10 2006 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2A67816A425 for ; Wed, 8 Feb 2006 12:46:10 +0000 (GMT) (envelope-from astatech@mail.ru) Received: from f64.mail.ru (f64.mail.ru [194.67.57.98]) by mx1.FreeBSD.org (Postfix) with ESMTP id 705DF43D45 for ; Wed, 8 Feb 2006 12:46:09 +0000 (GMT) (envelope-from astatech@mail.ru) Received: from mail by f64.mail.ru with local id 1F6ohz-0001cf-00; Wed, 08 Feb 2006 15:46:07 +0300 Received: from [217.107.221.51] by win.mail.ru with HTTP; Wed, 08 Feb 2006 15:46:07 +0300 From: astatech@mail.ru To: Joseph Koshy Mime-Version: 1.0 X-Mailer: mPOP Web-Mail 2.19 X-Originating-IP: [217.107.221.51] Date: Wed, 08 Feb 2006 15:46:07 +0300 In-Reply-To: <84dead720602070831g32e14257x16841f31e5b8527d@mail.gmail.com> Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: 8bit Message-Id: Cc: freebsd-smp@freebsd.org Subject: 6.0 Stable on 8 Core AMD Opteron scalability. (perf stat) X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: astatech@mail.ru List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2006 12:46:10 -0000 > a> - 8 cores at 2.2GHz/16GB RAM ... > a> What can I do to utilize all CPU power on FreeBSD? > 1) Which benchmark are you using? jakarta jmeter with customly builded test plan for PHPBB site. It`s my standart benchmark and already tested on others platforms. > 2) Please post the output of dmesg(8). See below... > 3) Please post outputs from vmstat or any other high-level > performance statistics that you've collected while > the benchmark is running. See below... PS: maybe httpd wait for log write? $ dmesg -a Copyright (c) 1992-2005 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 6.0-RELEASE #2: Mon Feb 6 20:50:32 MSK 2006 root@XXXX:/usr/src/sys/amd64/compile/SMP Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Dual Core AMD Opteron(tm) Processor 875 (2190.07-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x20f10 Stepping = 0 Features=0x178bfbff Features2=0x1 AMD Features=0xe2500800,LM,3DNow+,3DNow> Hyperthreading: 2 logical CPUs real memory = 17179869184 (16384 MB) avail memory = 16188342272 (15438 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 6 cpu7 (AP): APIC ID: 7 MADT: Forcing active-low polarity and level trigger for SCI ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-27 on motherboard ioapic2 irqs 28-31 on motherboard ioapic3 irqs 32-35 on motherboard ioapic4 irqs 36-39 on motherboard ioapic5 irqs 40-43 on motherboard ioapic6 irqs 44-47 on motherboard acpi0: on motherboard acpi0: Power Button (fixed) pci_link0: irq 10 on acpi0 pci_link1: irq 5 on acpi0 pci_link2: irq 7 on acpi0 pci_link3: irq 11 on acpi0 unknown: I/O range not supported unknown: I/O range not supported Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0xf008-0xf00b on acpi0 cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cpu4: on acpi0 cpu5: on acpi0 cpu6: on acpi0 cpu7: on acpi0 acpi_button0: on acpi0 pcib0: port 0xcf8-0xcff,0xf000-0xf07f,0xf080-0xf0ff iomem 0xd8000-0xdbfff on acpi0 pci0: on pcib0 pcib1: at device 6.0 on pci0 pci1: on pcib1 pci1: at device 0.0 (no driver attached) pci1: at device 0.1 (no driver attached) pci1: at device 5.0 (no driver attached) isab0: at device 7.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1000-0x100f at device 7.1 on pci0 ata0: on atapci0 ata1: on atapci0 pci0: at device 7.3 (no driver attached) pcib2: at device 10.0 on pci0 pci2: on pcib2 bge0: mem 0xe5810000-0xe581ffff,0xe5800000-0xe580ffff irq 25 at device 2.0 on pci2 miibus0: on bge0 brgphy0: on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX-FDX, auto bge0: Ethernet address: 00:09:3d:12:e8:b5 bge1: mem 0xe5830000-0xe583ffff,0xe5820000-0xe582ffff irq 26 at device 2.1 on pci2 miibus1: on bge1 brgphy1: on miibus1 brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX-FDX, auto bge1: Ethernet address: 00:09:3d:12:e8:b6 mpt0: port 0x2000-0x20ff mem 0xe5850000-0xe585ffff,0xe5840000-0xe584ffff irq 27 at device 4.0 on pci2 mpt0: [GIANT-LOCKED] mpt0: MPI Version=1.2.15.0 mpt0: Unhandled Event Notify Frame. Event 0xa. mpt0: Capabilities: ( RAID-1E RAID-1 SAFTE ) mpt0: 1 Active Volume (1 Max) mpt0: 2 Hidden Drive Members (6 Max) pci0: at device 10.1 (no driver attached) pcib3: at device 11.0 on pci0 pci3: on pcib3 pci0: at device 11.1 (no driver attached) pcib4: iomem 0xe5b01000-0xe5b01fff,0xe5b03000-0xe5b03fff,0xe5b05000-0xe5b05fff,0xe5b07000-0xe5b07fff on acpi0 pci32: on pcib4 pcib5: mem 0xe5b00000-0xe5b00fff irq 32 at device 1.0 on pci32 pci33: on pcib5 pci32: at device 1.1 (no driver attached) pcib6: mem 0xe5b02000-0xe5b02fff irq 36 at device 2.0 on pci32 pci37: on pcib6 pci32: at device 2.1 (no driver attached) pcib7: mem 0xe5b04000-0xe5b04fff irq 40 at device 3.0 on pci32 pci41: on pcib7 pci32: at device 3.1 (no driver attached) pcib8: mem 0xe5b06000-0xe5b06fff irq 44 at device 4.0 on pci32 pci45: on pcib8 pci32: at device 4.1 (no driver attached) atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: flags 0x1 irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: model IntelliMouse Explorer, device ID 4 sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FAST] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 orm0: at iomem 0xc0000-0xc7fff,0xc8000-0xc97ff,0xc9800-0xcafff,0xcb000-0xcefff on isa0 ppc0: cannot reserve I/O port range sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled sio1 at port 0x2f8-0x2ff irq 3 on isa0 sio1: type 16550A vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 1.000 msec Waiting 5 seconds for SCSI devices to settle acd0: DVDROM at ata1-master UDMA33 mpt0:vol0(mpt0:0:0): Settings ( Hot-Plug-Spares ) mpt0:vol0(mpt0:0:0): Using Spare Pool: 0 mpt0:vol0(mpt0:0:0): 2 Members: (mpt0:0:0): Primary (mpt0:0:1): Secondary mpt0:vol0(mpt0:0:0): RAID-1 - Optimal mpt0:vol0(mpt0:0:0): Status ( Enabled ) (mpt0:vol0:0): Physical (mpt0:0:0), Pass-thru (mpt0:1:0) (mpt0:vol0:0): Online (mpt0:vol0:1): Physical (mpt0:0:1), Pass-thru (mpt0:1:1) (mpt0:vol0:1): Online ses0 at mpt0 bus 0 target 6 lun 0 ses0: Fixed Processor SCSI-2 device ses0: 3.300MB/s transfers ses0: SAF-TE Compliant Device pass2 at mpt0 bus 1 target 0 lun 0 pass2: Fixed unknown SCSI-3 device pass2: 320.000MB/s transfers (160.000MHz, offset 63, 16bit), Tagged Queueing Enabled pass3 at mpt0 bus 1 target 1 lun 0 pass3: Fixed unknown SCSI-3 device pass3: 320.000MB/s transfers (160.000MHz, offset 63, 16bit), Tagged Queueing Enabled da0 at mpt0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-2 device da0: 320.000MB/s transfers (160.000MHz, offset 63, 16bit), Tagged Queueing Enabled da0: 69878MB (143110144 512 byte sectors: 255H 63S/T 8908C) SMP: AP CPU #1 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #7 Launched! Trying to mount root from ufs:/dev/da0s1a Loading configuration files. /dev/da0s1b is smaller than physical memory No suitable dump device was found. Entropy harvesting: interrupts ethernet point_to_point kickstart . swapon: adding /dev/da0s1b as swap device Starting file system checks: /dev/da0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/da0s1a: clean, 220365 free (493 frags, 27484 blocks, 0.2% fragmentation) /dev/da0s1e: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/da0s1e: clean, 243870 free (38 frags, 30479 blocks, 0.0% fragmentation) /dev/da0s1f: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/da0s1f: clean, 3364076 free (43252 frags, 415103 blocks, 0.9% fragmentation) /dev/da0s1d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/da0s1d: clean, 460696 free (256 frags, 57555 blocks, 0.1% fragmentation) Setting hostname: XXXXX lo0: flags=8049 mtu 16384 inet 127.0.0.1 netmask 0xff000000 bge0: flags=8843 mtu 1500 options=1a inet 192.168.10.197 netmask 0xffffff00 broadcast 192.168.10.255 ether 00:09:3d:12:e8:b5 media: Ethernet autoselect (none) status: no carrier add net default: gateway 192.168.10.25 Additional routing options: . Starting devd. Mounting NFS file systems: . Creating and/or trimming log files: . Starting syslogd. /etc/rc: WARNING: Dump device does not exist. Savecore not run. ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib ldconfig: /usr/local/lib: ignoring directory not owned by root Starting usbd. ohci0: mem 0xe4120000-0xe4120fff irq 19 at device 0.0 on pci1 ohci0: [GIANT-LOCKED] usb0: OHCI version 1.0, legacy support usb0: SMM does not respond, resetting usb0: on ohci0 usb0: USB revision 1.0 bge0: link state changed to UP uhub0: AMD OHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 3 ports with 3 removable, self powered ohci1: mem 0xe4121000-0xe4121fff irq 19 at device 0.1 on pci1 ohci1: [GIANT-LOCKED] usb1: OHCI version 1.0, legacy support usb1: SMM does not respond, resetting usb1: on ohci1 usb1: USB revision 1.0 uhub1: AMD OHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub1: 3 ports with 3 removable, self powered Starting local daemons: . Updating motd . Configuring syscons: blanktime . Starting sshd. Initial amd64 initialization: . Additional ABI support: . Starting cron. Local package initialization: . Additional TCP options: . Starting background file system checks in 60 seconds. ------------------------------------------------------------------------------------------------------ $ vmstat 2 99999 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr da0 pa0 in sy cs us sy id ..... 10 65 3 1321932 14450328 29310 0 0 0 30897 0 15 0 3009 311093 135143 48 25 26 92 13 4 1363148 14426332 32961 0 0 0 30981 0 14 0 3556 340291 93069 54 21 25 95 48 8 1340540 14440024 26367 0 0 0 27873 0 26 0 3967 294134 68154 49 21 29 80 1 1 1104284 14485872 44924 0 0 0 50532 0 28 0 3447 165200 68333 49 23 28 121 0 0 1365440 14432084 43587 0 0 0 37317 0 23 0 1754 211795 83845 46 30 23 54 11 3 1338724 14446888 19043 0 0 0 21133 0 9 0 2800 291969 13703539 25 36 66 0 0 1375484 14423592 34484 0 0 0 31638 0 15 0 3503 259224 109744 53 21 26 44 40 0 1350336 14437612 27347 0 0 0 29507 0 87 0 2966 281139 131237 44 27 29 71 1 2 1400156 14404564 30590 0 0 0 26273 0 23 0 3256 321923 143233 45 30 26 28 89 0 1381720 14417128 27507 0 0 0 28523 0 13 0 2797 261042 128871 46 21 34 64 0 19 1383200 14414360 34167 0 0 0 34986 0 25 0 4321 269089 106260 52 18 30 40 42 3 1383896 14409896 36772 0 0 0 36896 0 114 0 3747 278703 96025 53 22 25 107 11 3 1332280 14441928 27602 0 0 0 30553 0 10 0 3643 304230 84296 53 23 24 47 59 0 1345140 14431752 29048 0 0 0 29329 0 16 0 3862 346969 120789 49 25 27 44 58 3 1342888 14431316 22663 0 0 0 22538 0 15 0 3323 309007 137080 44 25 30 28 74 2 1410220 14396556 29990 0 0 0 25843 0 9 0 2946 270054 124079 46 26 28 151 0 9 1413572 14392208 41542 0 0 0 41407 0 17 0 3293 242969 77110 55 18 27 ..... $ top 8403 nobody 1 80 0 32548K 8704K RUN 7 0:22 2.25% httpd 8503 nobody 1 77 0 32668K 8796K CPU3 3 0:19 1.46% httpd 8520 nobody 1 79 0 33004K 9136K CPU0 0 0:19 1.46% httpd 8478 nobody 1 4 0 35452K 10340K sbwait 2 0:19 1.46% httpd 8490 nobody 1 -4 0 32600K 8756K ufs 7 0:18 1.46% httpd 8413 nobody 1 77 0 32880K 8848K RUN 6 0:17 1.46% httpd 8505 nobody 1 -4 0 32300K 8456K ufs 7 0:19 0.73% httpd 8416 nobody 1 -4 0 32396K 8552K ufs 7 0:19 0.73% httpd 8443 nobody 1 -4 0 32396K 8568K ufs 7 0:19 0.73% httpd 8414 nobody 1 -4 0 32564K 8720K ufs 7 0:19 0.73% httpd 8539 nobody 1 -4 0 35588K 10476K ufs 7 0:18 0.73% httpd 8485 nobody 1 4 0 32552K 8708K sbwait 2 0:18 0.73% httpd 8473 nobody 1 -4 0 31916K 8072K ufs 7 0:18 0.73% httpd 8444 nobody 1 77 0 32552K 8708K RUN 2 0:18 0.73% httpd 8452 nobody 1 -4 0 32424K 8580K ufs 7 0:18 0.73% httpd 8479 nobody 1 78 0 35372K 10260K CPU2 2 0:18 0.73% httpd 8535 nobody 1 -4 0 35544K 10432K ufs 7 0:17 0.73% httpd 8453 nobody 1 -4 0 35380K 10268K ufs 7 0:17 0.73% httpd 8634 mysql 111 76 0 367M 122M RUN 7 0:52 0.00% mysqld 8423 nobody 1 4 0 35424K 10312K sbwait 2 0:20 0.00% httpd 8436 nobody 1 -4 0 32200K 8356K ufs 7 0:20 0.00% httpd 8427 nobody 1 4 0 32300K 8456K sbwait 6 0:20 0.00% httpd 8402 nobody 1 77 0 35128K 10040K RUN 2 0:20 0.00% httpd 8486 nobody 1 -4 0 35332K 10244K ufs 6 0:20 0.00% httpd 8511 nobody 1 -4 0 32252K 8408K ufs 6 0:20 0.00% httpd 8410 nobody 1 4 0 35400K 10312K sbwait 6 0:20 0.00% httpd 8447 nobody 1 4 0 35428K 10316K sbwait 7 0:19 0.00% httpd 8482 nobody 1 -4 0 32296K 8452K ufs 6 0:19 0.00% httpd 8472 nobody 1 -4 0 32852K 9012K ufs 7 0:19 0.00% httpd 8458 nobody 1 -4 0 32552K 8720K ufs 7 0:19 0.00% httpd 8441 nobody 1 -4 0 35460K 10352K ufs 7 0:19 0.00% httpd 8442 nobody 1 4 0 32608K 8788K sbwait 0 0:19 0.00% httpd 8450 nobody 1 -4 0 32516K 8672K ufs 7 0:19 0.00% httpd 8542 nobody 1 4 0 35400K 10288K sbwait 3 0:19 0.00% httpd 8438 nobody 1 -4 0 35388K 10276K ufs 7 0:19 0.00% httpd $ mount /dev/da0s1a on / (ufs, local) devfs on /dev (devfs, local) /dev/da0s1e on /tmp (ufs, local, soft-updates) /dev/da0s1f on /usr (ufs, local, soft-updates) /dev/da0s1d on /var (ufs, local, soft-updates) From owner-freebsd-smp@FreeBSD.ORG Wed Feb 8 19:05:04 2006 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 77FD916A420 for ; Wed, 8 Feb 2006 19:05:04 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id F1B3343D45 for ; Wed, 8 Feb 2006 19:05:03 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id D4DB61A3C30; Wed, 8 Feb 2006 11:05:03 -0800 (PST) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 0A72C515AE; Wed, 8 Feb 2006 14:05:03 -0500 (EST) Date: Wed, 8 Feb 2006 14:05:02 -0500 From: Kris Kennaway To: astatech@mail.ru Message-ID: <20060208190502.GA55490@xor.obsecurity.org> References: <84dead720602070831g32e14257x16841f31e5b8527d@mail.gmail.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="zYM0uCDKw75PZbzx" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.1i Cc: freebsd-smp@freebsd.org, Joseph Koshy Subject: Re: 6.0 Stable on 8 Core AMD Opteron scalability. (perf stat) X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2006 19:05:04 -0000 --zYM0uCDKw75PZbzx Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Wed, Feb 08, 2006 at 03:46:07PM +0300, astatech@mail.ru wrote: > See below... > PS: maybe httpd wait for log write? Yes, could be (most of them are stuck in 'ufs'). Try disabling logging. Kris --zYM0uCDKw75PZbzx Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (FreeBSD) iD8DBQFD6kDeWry0BWjoQKURAsd7AKDTZNpsd2emG/CZjoYzFZzyFlT4ZACfYfWR Y6U0UJXQg9cvO4yLJCKRKYY= =N4gS -----END PGP SIGNATURE----- --zYM0uCDKw75PZbzx-- From owner-freebsd-smp@FreeBSD.ORG Wed Feb 8 19:24:53 2006 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 12C6F16A420 for ; Wed, 8 Feb 2006 19:24:53 +0000 (GMT) (envelope-from joseph.koshy@gmail.com) Received: from xproxy.gmail.com (xproxy.gmail.com [66.249.82.206]) by mx1.FreeBSD.org (Postfix) with ESMTP id 966B943D45 for ; Wed, 8 Feb 2006 19:24:52 +0000 (GMT) (envelope-from joseph.koshy@gmail.com) Received: by xproxy.gmail.com with SMTP id s9so1158141wxc for ; Wed, 08 Feb 2006 11:24:52 -0800 (PST) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=FUPT0IwOswwlrh+7m2Jaibw142oSdUT2nGdJSv5N8p96lyLu0eze3KlBplNedTU+ClEL75GT96VTsB6eMBwvg/4Embsd6yE+l8Hhh0Ej+Cp5Aknzv51joGFPfoVDuUiqde7dXqRK8YHEY041KrnsHIgd8LE2h53/cVogQ5rCCVE= Received: by 10.70.92.1 with SMTP id p1mr9498245wxb; Wed, 08 Feb 2006 11:24:51 -0800 (PST) Received: by 10.70.105.2 with HTTP; Wed, 8 Feb 2006 11:24:51 -0800 (PST) Message-ID: <84dead720602081124o651693ccy7920658a54d97e61@mail.gmail.com> Date: Thu, 9 Feb 2006 00:54:51 +0530 From: Joseph Koshy To: astatech@mail.ru In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline References: <84dead720602070831g32e14257x16841f31e5b8527d@mail.gmail.com> Cc: freebsd-smp@freebsd.org Subject: Re: 6.0 Stable on 8 Core AMD Opteron scalability. (perf stat) X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2006 19:24:53 -0000 jk> which benchmark? a> jakarta jmeter with customly builded test plan for PHPBB No Java on my system :(. a> bge0: flags=3D8843 mtu 1500 ... a> media: Ethernet autoselect (none) a> status: no carrier It wasn't clear if you were running over a localhost-only benchmark. If not, did your network card manage to negotiate full-duplex mode? a> See below... a> PS: maybe httpd wait for log write? a> $ vmstat 2 99999 ... [vmstat output] a> 66 0 0 1375484 14423592 34484 0 0 0 31638 0 15 0 3503 259224 109744 53 21 26 Ok, no memory pressure, a large system-call rate and a large number of context switches per second. a> 8634 mysql 111 76 0 367M 122M RUN 7 0:52 0.00% mysqld This process looks suspicious. Please check if you are running MySQL with an appropriate configuration (my-huge.cnf?). http://wikitest.freebsd.org/moin.cgi/MySQL has more hints about tuning MySQL. Does changing sysctl kern.timecounter help? I would appreciate a hwpmc profile of your system under this load. Step 1 # kldload hwpmc 2 # pmcstat -S k8-bu-cpu-clk-unhalted -O /tmp/logfile [for less than a minute] 3 ^C [kill pmcstat] 4 # pmcstat -R /tmp/logfile -g 5 # [for each foo.gmon file generated, run gprof -l /path/to/unstripped/foo foo.gmon to get a flat profile] -- FreeBSD Volunteer, http://people.freebsd.org/~jkoshy From owner-freebsd-smp@FreeBSD.ORG Wed Feb 8 20:06:39 2006 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D26DE16A420 for ; Wed, 8 Feb 2006 20:06:39 +0000 (GMT) (envelope-from astatech@mail.ru) Received: from f58.mail.ru (f58.mail.ru [194.67.57.92]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6CD3E43D49 for ; Wed, 8 Feb 2006 20:06:39 +0000 (GMT) (envelope-from astatech@mail.ru) Received: from mail by f58.mail.ru with local id 1F6vaH-000J6l-00; Wed, 08 Feb 2006 23:06:37 +0300 Received: from [213.247.201.20] by win.mail.ru with HTTP; Wed, 08 Feb 2006 23:06:37 +0300 From: astatech@mail.ru To: Joseph Koshy Mime-Version: 1.0 X-Mailer: mPOP Web-Mail 2.19 X-Originating-IP: 192.168.1.2 via proxy [213.247.201.20] Date: Wed, 08 Feb 2006 23:06:37 +0300 In-Reply-To: <84dead720602081124o651693ccy7920658a54d97e61@mail.gmail.com> Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: 8bit Message-Id: Cc: freebsd-smp@freebsd.org Subject: Re[2]: 6.0 Stable on 8 Core AMD Opteron scalability. (perf stat) X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: astatech@mail.ru List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2006 20:06:39 -0000 > a> bge0: flags=8843 mtu 1500 > .... > a> media: Ethernet autoselect (none) > a> status: no carrier Latter in log: bge0: link state changed to UP > It wasn't clear if you were running over a localhost-only > benchmark. If not, did your network card manage > to negotiate full-duplex mode? Cross link via bge0 to client machine with identical NIC. Same configuration used for Solaris testing. > a> 8634 mysql 111 76 0 367M 122M RUN 7 0:52 0.00% mysqld > > This process looks suspicious. Please check if you are > running MySQL with an appropriate configuration (my-huge.cnf?). MySQL database only 350MB, and so I use my-large.cnf > http://wikitest.freebsd.org/moin.cgi/MySQL has more > hints about tuning MySQL. Does changing sysctl > kern.timecounter help? I try rebuild mysql and post results here. > I would appreciate a hwpmc profile of your system under > this load. Tomorrow. Thank for your help. I post results as soon as posible. WBR, Alexander. From owner-freebsd-smp@FreeBSD.ORG Wed Feb 8 20:14:19 2006 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2963316A420 for ; Wed, 8 Feb 2006 20:14:19 +0000 (GMT) (envelope-from astatech@mail.ru) Received: from f29.mail.ru (f29.mail.ru [194.67.57.22]) by mx1.FreeBSD.org (Postfix) with ESMTP id B7CA543D48 for ; Wed, 8 Feb 2006 20:14:17 +0000 (GMT) (envelope-from astatech@mail.ru) Received: from mail by f29.mail.ru with local id 1F6vhf-000HOg-00; Wed, 08 Feb 2006 23:14:15 +0300 Received: from [213.247.201.20] by win.mail.ru with HTTP; Wed, 08 Feb 2006 23:14:15 +0300 From: astatech@mail.ru To: Kris Kennaway Mime-Version: 1.0 X-Mailer: mPOP Web-Mail 2.19 X-Originating-IP: 192.168.1.2 via proxy [213.247.201.20] Date: Wed, 08 Feb 2006 23:14:15 +0300 In-Reply-To: <20060208190502.GA55490@xor.obsecurity.org> Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: 8bit Message-Id: Cc: freebsd-smp@freebsd.org, Joseph Koshy Subject: Re[2]: 6.0 Stable on 8 Core AMD Opteron scalability. (perf stat) X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: astatech@mail.ru List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2006 20:14:19 -0000 > > See below... > > PS: maybe httpd wait for log write? > > Yes, could be (most of them are stuck in 'ufs'). Try disabling logging. Do you mean apache or ufs logging? For apache I try it for testing, not for production. I need logs for my server :) May be mount -o noatime can help. I try it with Joseph`s recomendations. WBR, Alexander From owner-freebsd-smp@FreeBSD.ORG Wed Feb 8 20:22:31 2006 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 18D3B16A422 for ; Wed, 8 Feb 2006 20:22:31 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id A20CF43D55 for ; Wed, 8 Feb 2006 20:22:30 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 7B30B1A3C20; Wed, 8 Feb 2006 12:22:30 -0800 (PST) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id A03865161D; Wed, 8 Feb 2006 15:22:29 -0500 (EST) Date: Wed, 8 Feb 2006 15:22:29 -0500 From: Kris Kennaway To: astatech@mail.ru Message-ID: <20060208202229.GA57534@xor.obsecurity.org> References: <20060208190502.GA55490@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="wac7ysb48OaltWcw" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.1i Cc: freebsd-smp@freebsd.org, Joseph Koshy , Kris Kennaway Subject: Re: 6.0 Stable on 8 Core AMD Opteron scalability. (perf stat) X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2006 20:22:31 -0000 --wac7ysb48OaltWcw Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Feb 08, 2006 at 11:14:15PM +0300, astatech@mail.ru wrote: > > > See below... > > > PS: maybe httpd wait for log write? > >=20 > > Yes, could be (most of them are stuck in 'ufs'). Try disabling logging. >=20 > Do you mean apache or ufs logging?=20 apache (no such thing as ufs logging). > For apache I try it for testing, not for production. I need logs for my s= erver :) If your processes are spamming at the logfile faster than the filesystem can keep up (which seems to be happening), that's going to serialize all of your httpd operations. > May be mount -o noatime can help. I try it with Joseph`s recomendations. You could also try mount -o async, but this may destroy your /var in the event of a crash. Kris --wac7ysb48OaltWcw Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (FreeBSD) iD8DBQFD6lMFWry0BWjoQKURAg0GAJ4gRLcIHVckHSpHaVUi2KD3En6xWwCgtbTB m2zAjkEVH+5R1upsyMlCcJY= =71NZ -----END PGP SIGNATURE----- --wac7ysb48OaltWcw-- From owner-freebsd-smp@FreeBSD.ORG Wed Feb 8 20:37:31 2006 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E695216A422 for ; Wed, 8 Feb 2006 20:37:31 +0000 (GMT) (envelope-from astatech@mail.ru) Received: from f25.mail.ru (f25.mail.ru [194.67.57.151]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3029A43D8A for ; Wed, 8 Feb 2006 20:37:22 +0000 (GMT) (envelope-from astatech@mail.ru) Received: from mail by f25.mail.ru with local id 1F6w40-0001Am-00; Wed, 08 Feb 2006 23:37:20 +0300 Received: from [213.247.201.20] by win.mail.ru with HTTP; Wed, 08 Feb 2006 23:37:20 +0300 From: astatech@mail.ru To: Kris Kennaway Mime-Version: 1.0 X-Mailer: mPOP Web-Mail 2.19 X-Originating-IP: 192.168.1.2 via proxy [213.247.201.20] Date: Wed, 08 Feb 2006 23:37:20 +0300 In-Reply-To: <20060208202229.GA57534@xor.obsecurity.org> Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: 8bit Message-Id: Cc: freebsd-smp@freebsd.org, Joseph Koshy Subject: UFS bottleneck X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: astatech@mail.ru List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2006 20:37:32 -0000 > > > > PS: maybe httpd wait for log write? > > > Yes, could be (most of them are stuck in 'ufs'). Try disabling logging. > > Do you mean apache or ufs logging? > apache (no such thing as ufs logging). Sorry, ufs logging it's a solaris feature. > > For apache I try it for testing, not for production. I need logs for my server :) > > May be mount -o noatime can help. I try it with Joseph`s recomendations. > You could also try mount -o async, but this may destroy your /var in > the event of a crash. I put logs to other slice and try acync since it's acceptable to lost logs of apache. WBR, Alexander.