From owner-freebsd-stable@FreeBSD.ORG Sat Dec 1 21:58:00 2007 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 271B616A41A for ; Sat, 1 Dec 2007 21:58:00 +0000 (UTC) (envelope-from renton@df.ru) Received: from cannabis.dataforce.net (cannabis.dataforce.net [195.42.160.18]) by mx1.freebsd.org (Postfix) with ESMTP id 4C97E13C461 for ; Sat, 1 Dec 2007 21:57:59 +0000 (UTC) (envelope-from renton@df.ru) Received: by cannabis.dataforce.net (Postfix, from userid 46126) id 9B16117A07; Sun, 2 Dec 2007 00:37:32 +0300 (MSK) Date: Sun, 2 Dec 2007 00:37:32 +0300 From: Alexey Vlasov To: freebsd-stable@freebsd.org Message-ID: <20071201213732.GA16638@cannabis.dataforce.net> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="SUOF0GtieIMvvwua" Content-Disposition: inline User-Agent: Mutt/1.5.16 (2007-06-09) Subject: Re: 2 x quad-core system is slower that 2 x dual core on FreeBSD X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Dec 2007 21:58:00 -0000 --SUOF0GtieIMvvwua Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Hi, It seems that I'm not the only one who faced the problem that FreeBSD is non productive on multiprocessors platforms. I use OS Linux on my hosting for web-servers, base for all servers is the same m/b S5000PAL ( SR1500), 2 quad kernel cpu Xeon E5320 or E5345, 8Gb RAM. I decided to install FreeBSD 6.2 i386 on one of the servers, and the result was totally non productive. Used software, Apache 2.2.6 (worker) as frontend Proxy, backend Apache 2.2.3 (prefork) By the time we understood,that something was wrong with FreeBSD, there had already been placed about 10 sites with high-capacity and about hundred of usual ones. And this was the limit for FreeBSD. It came along with a great amount of Context Switches, about hundred thousands. I attached the log what was then happening with FreeBSD. After playing with ab (ApacheBenchmark) options, it turned out that even with the following options you can totally down the server: ./ab -n 100 -c 20 -t 30 http://somesite-freebsd.com I copied at the same time somesite.com (php scripts) to Linux server, launched ab with the same options, and saw that it has no influence on work of the server. (And by the way there work about 1.5 virtual hosts on that server) All options for Apache on Linux and FreeBSD are the same: FreeBSD: This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0 Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Copyright 2006 The Apache Software Foundation, http://www.apache.org/ Benchmarking somesite-freebsd.com (be patient) Finished 29 requests Server Software: Apache/2.2.3 Server Hostname: somesite-freebsd.com Server Port: 80 Document Path: /ab/ Document Length: 41450 bytes Concurrency Level: 20 Time taken for tests: 30.44765 seconds Complete requests: 29 Failed requests: 22 (Connect: 0, Length: 22, Exceptions: 0) Write errors: 0 Total transferred: 1529557 bytes HTML transferred: 1513497 bytes Requests per second: 0.97 [#/sec] (mean) Time per request: 20720.527 [ms] (mean) Time per request: 1036.026 [ms] (mean, across all concurrent requests) Transfer rate: 49.69 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 1760 1503.4 3002 3002 Processing: 866 13328 9460.5 13853 26246 Waiting: 139 2286 2319.0 1129 6764 Total: 871 15089 10642.6 16855 29248 Percentage of the requests served within a certain time (ms) 50% 16705 66% 22670 75% 25439 80% 26342 90% 29160 95% 29188 98% 29248 99% 29248 100% 29248 (longest request) Linux: (the same site) This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0 Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Copyright 2006 The Apache Software Foundation, http://www.apache.org/ Benchmarking linux.1gb.ru (be patient) Finished 814 requests Server Software: Apache/2.2.3 Server Hostname: somesite-linux.com Server Port: 80 Document Path: /ab/ Document Length: 41451 bytes Concurrency Level: 20 Time taken for tests: 30.3216 seconds Complete requests: 814 Failed requests: 759 (Connect: 0, Length: 759, Exceptions: 0) Write errors: 0 Non-2xx responses: 1 Total transferred: 34430291 bytes HTML transferred: 34126461 bytes Requests per second: 27.13 [#/sec] (mean) Time per request: 737.180 [ms] (mean) Time per request: 36.859 [ms] (mean, across all concurrent requests) Transfer rate: 1120.65 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.0 0 0 Processing: 214 725 575.2 557 5001 Waiting: 41 265 376.0 131 3280 Total: 214 725 575.2 557 5001 Percentage of the requests served within a certain time (ms) 50% 557 66% 716 75% 863 80% 967 90% 1398 95% 1749 98% 2529 99% 3064 100% 5001 (longest request) # cat /etc/sysctl.conf security.bsd.see_other_uids=0 kern.maxfiles=204800 kern.maxfilesperproc=202400 kernel: machine i386 cpu I686_CPU ident F1RNT1 options PAE options SMP options SCHED_4BSD options PREEMPTION options INET options FFS options SOFTUPDATES options UFS_ACL options UFS_DIRHASH options NULLFS options MD_ROOT options CD9660 options PROCFS options PSEUDOFS options GEOM_GPT options GEOM_LABEL options GEOM_MIRROR options COMPAT_43 options COMPAT_FREEBSD4 options COMPAT_FREEBSD5 options SCSI_DELAY=5000 options KTRACE options SYSVSHM options SYSVMSG options SYSVSEM options _KPOSIX_PRIORITY_SCHEDULING options KBD_INSTALL_CDEV options ADAPTIVE_GIANT options LIBICONV options NETSMB options NETSMBCRYPTO options LIBMCHAIN options SMBFS options IPFIREWALL options IPFIREWALL_VERBOSE options IPFIREWALL_VERBOSE_LIMIT=1000 options IPFIREWALL_DEFAULT_TO_ACCEPT options IPFIREWALL_FORWARD options IPDIVERT options IPSTEALTH options TCPDEBUG options DUMMYNET options IPSEC options IPSEC_DEBUG options ACCEPT_FILTER_DATA options ACCEPT_FILTER_HTTP # cat /etc/make.conf CPUTYPE?=nocona CFLAGS=-O2 -pipe CXXFLAGS+=-fconserve-space COPTFLAGS=-O2 -pipe NO_CPU_CFLAGS=false NO_CPU_COPTFLAGS=false NO_MODULES= -- BRGDS. Alexey Vlasov. --SUOF0GtieIMvvwua Content-Type: text/plain; charset=koi8-r Content-Disposition: attachment; filename="logs.txt" last pid: 31039; load averages: 51.33, 26.31, 18.09 up 15+22:58:54 16:47:30 692 processes: 80 running, 608 sleeping, 2 stopped, 1 zombie, 1 lock CPU states: 2.4% user, 0.0% nice, 89.1% system, 1.6% interrupt, 6.9% idle Mem: 1587M Active, 5614M Inact, 416M Wired, 369M Cache, 112M Buf, 14M Free Swap: 16G Total, 46M Used, 16G Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 31027 w_istok2_d424 1 -4 0 76240K 36532K ufs 4 0:00 9.74% httpd 30826 w_nova-i_7638 1 -4 0 79340K 40116K ufs 4 0:02 7.78% httpd 31025 w_shop2y_b290 1 -4 0 76212K 36512K RUN 7 0:00 7.70% httpd 28349 w_urtest_7dc3 1 -4 0 79616K 39496K RUN 4 1:27 7.32% httpd 30801 w_new-vw_5e17 1 -4 0 76092K 37184K ufs 5 0:02 7.25% httpd 30914 w_avtolo_3761 1 -4 0 76316K 37304K RUN 7 0:01 7.18% httpd 29952 w_svekon_9de8 1 -4 0 77976K 42916K RUN 6 0:11 7.04% httpd 29875 w_lancer_12cf 1 -4 0 79584K 41408K RUN 6 0:12 6.94% httpd 29818 w_iprit-_be41 1 -4 0 77340K 39880K RUN 6 0:13 6.92% httpd 30811 w_dfase__508a 1 -4 0 76456K 37928K RUN 4 0:02 6.91% httpd 30200 w_iprit-_be41 1 -4 0 76292K 37260K ufs 1 0:07 6.86% httpd 30445 w_new-vw_5e17 1 119 0 77048K 40196K CPU3 0 0:04 6.74% httpd 30025 w_forum-_89b3 1 -4 0 76824K 41240K RUN 4 0:10 6.71% httpd 30687 w_new-vw_5e17 1 -4 0 76232K 38616K RUN 1 0:02 6.69% httpd 30561 w_avtolo_3761 1 105 0 76512K 39500K select 2 0:03 6.67% httpd 30519 w_new-vw_5e17 1 -4 0 76784K 39396K ufs 4 0:04 6.62% httpd 31018 w_tucson_bd87 1 -4 0 76000K 36464K RUN 6 0:00 6.62% httpd 30867 w_as-vid_ac7e 1 -4 0 76356K 37452K RUN 6 0:01 6.60% httpd 30997 w_rfscla_06fc 1 -4 0 76820K 38652K RUN 2 0:01 6.60% httpd 31015 w_tucson_bd87 1 -4 0 76024K 36908K RUN 1 0:00 6.59% httpd 30532 w_shop2y_b290 1 -4 0 76904K 38232K ufs 4 0:04 6.55% httpd 30936 w_tucson_bd87 1 -4 0 77836K 39740K RUN 6 0:01 6.53% httpd 30376 w_vannay_4606 1 -4 0 78252K 41596K RUN 1 0:05 6.53% httpd 30452 w_new-vw_5e17 1 -4 0 77356K 40124K RUN 6 0:04 6.50% httpd 30201 w_berega_3091 1 -4 0 81636K 42992K RUN 4 0:07 6.46% httpd 30840 w_forum-_89b3 1 -4 0 76412K 38752K RUN 4 0:02 6.44% httpd 30809 w_rm-for_636d 1 -4 0 76148K 38028K RUN 6 0:02 6.43% httpd 30700 w_ott-gu_2def 1 -4 0 76888K 39272K RUN 4 0:02 6.40% httpd 29971 w_mobi-m_2e46 1 -4 0 76872K 41484K RUN 4 0:05 6.40% httpd 31016 w_wplane_ebed 1 -4 0 76320K 37056K RUN 7 0:00 6.39% httpd 30839 w_lyubar_f1e2 1 -4 0 77332K 39788K RUN 4 0:02 6.36% httpd 31004 w_digitr_77d4 1 -4 0 76016K 36656K CPU5 5 0:01 6.36% httpd # vmstat procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad4 ad6 in sy cs us sy id 110 9 0 10884836 384132 1167 2 2 1 1340 273 0 0 19 809 1282 13 9 78 110 10 0 10950600 382964 4530 0 0 0 3968 0 1 13 3925 35129 162137 9 87 4 131 9 0 10855252 387356 3552 0 2 0 4782 0 1 9 2505 27529 166233 5 92 3 129 11 0 10924792 379056 6723 0 4 0 4658 0 1 26 4015 29908 162047 3 94 3 122 10 1 10936140 367864 5289 0 2 0 2145 0 0 17 3788 27790 162641 3 95 3 66 55 0 10807504 378600 4236 0 4 0 6818 0 1 6 3164 26302 167760 2 95 2 128 11 0 10815488 374308 4092 0 3 0 2910 0 1 13 2962 32777 167559 2 93 5 80 42 1 10784792 377988 4592 0 3 0 5821 0 5 13 3296 28117 163950 3 94 2 64 62 0 10785528 377076 5640 0 3 0 5238 0 1 6 2890 31425 163924 3 94 3 52 77 0 10656100 386120 2994 0 1 0 4768 0 1 8 2427 23684 159985 2 95 3 70 52 0 10591528 393632 3381 0 0 0 5335 0 1 4 3010 27593 158225 3 94 3 105 9 0 10549508 402488 5426 0 0 0 7436 0 2 7 2318 31753 168856 3 94 4 77 40 0 10554208 397512 5848 0 0 0 4448 0 1 9 2275 31423 156746 3 92 5 114 9 4 10483152 403624 3369 0 0 0 4974 0 0 5 2243 27000 160225 2 93 5 109 9 0 10550332 394796 4972 0 1 0 2708 0 1 11 2515 26056 158628 3 92 5 125 9 0 10518288 423240 4256 0 3 0 11099 0 1 9 2779 28773 158967 3 93 4 118 11 0 10529144 415176 5887 4 3 0 3670 0 1 11 2595 30282 159381 2 95 3 116 9 0 10497144 418012 3407 0 1 0 3929 0 1 11 2815 24890 159056 2 93 5 119 9 0 10463764 419260 3391 0 0 0 3797 0 1 9 2558 28508 159086 2 94 3 5 116 0 10436304 420156 4263 0 0 0 4368 0 24 8 3348 27563 159025 3 94 4 84 43 0 10563104 412824 5012 0 0 0 3630 0 2 32 2658 32205 158341 1 94 5 82 59 0 10620056 415188 6158 4 2 0 4858 0 11 11 2743 30929 159186 3 94 4 87 39 0 10453904 431328 3867 0 0 0 8894 0 1 5 2560 28903 168316 2 95 3 102 37 2 10642596 417724 6146 0 1 0 3285 0 5 11 2396 35490 158978 2 95 4 145 10 0 10470216 461180 5202 4 0 0 15462 0 77 10 2522 30525 163784 4 93 3 122 12 0 10507604 452820 4561 0 0 50 2341 0 138 6 2368 31650 158428 3 94 3 108 29 0 10669764 439152 4808 0 0 0 1294 0 4 8 3456 27089 166675 2 95 3 124 10 0 10631948 444132 6171 0 0 0 7239 0 6 6 2875 31822 167193 4 94 3 7 users Load 54.35 28.49 23.64 Nov 27 17:04 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 927804 34556 8233412 53740 489892 count All 8071840 267008983194432 3143860 pages Interrupts Proc:r p d s w Csw Trp Sys Int Sof Flt 1189 cow 16584 total 89 9454 153k 533325086 2022 309 5066 425580 wire 1: atkb 1499680 act 14: ata 90.0%Sys 0.6%Intr 2.9%User 0.0%Nice 6.4%Idl 5776020 inact 546 18: em0 | | | | | | | | | | 425236 cache 31 20: ata =============================================>> 64968 free 2001 cpu0: time daefr 2002 cpu1: time Namei Name-cache Dir-cache 2333 prcfr 2001 cpu2: time Calls hits % hits % react 2001 cpu3: time 207699 207669 100 5 0 pdwak 2000 cpu7: time 3514 zfod pdpgs 2001 cpu4: time Disks ad4 ad6 ad8 3463 ozfod intrn 2000 cpu6: time KB/t 16.00 4.20 6.90 98 %slo-z 114784 buf 2001 cpu5: time tps 13 1 1 3077 tfree 229 dirtybuf MB/s 0.21 0.00 0.01 100000 desiredvnodes % busy 0 1 1 89091 numvnodes 24971 freevnodes --SUOF0GtieIMvvwua--