From owner-freebsd-performance@FreeBSD.ORG Mon May 14 10:10:18 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 76C8D16A404 for ; Mon, 14 May 2007 10:10:18 +0000 (UTC) (envelope-from cheffo@FreeBSD-BG.org) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id D552C13C4B0 for ; Mon, 14 May 2007 10:10:15 +0000 (UTC) (envelope-from cheffo@FreeBSD-BG.org) Received: from blah.sun-fish.com (localhost [127.0.0.1]) by blah.sun-fish.com (Postfix) with ESMTP id 616D21B10EA4; Mon, 14 May 2007 12:10:14 +0200 (CEST) Received: from hater.cmotd.com (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id 4064E1B10C26; Mon, 14 May 2007 12:10:12 +0200 (CEST) Message-ID: <46483584.70608@FreeBSD-BG.org> Date: Mon, 14 May 2007 13:10:12 +0300 From: Cheffo User-Agent: Thunderbird 2.0.0.0 (X11/20070425) MIME-Version: 1.0 To: "Michael K. Smith - Adhost" References: <46443103.9010008@otel.net> <17838240D9A5544AAA5FF95F8D52031602018EFA@ad-exh01.adhost.lan> In-Reply-To: <17838240D9A5544AAA5FF95F8D52031602018EFA@ad-exh01.adhost.lan> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV using ClamSMTP on BLAH Cc: freebsd-performance@freebsd.org, ivo tasev Subject: Re: 6.2 Stable + Mysql 5.0 poor performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 May 2007 10:10:18 -0000 Hello, Michael K. Smith - Adhost wrote: > Hello Taseff: > >> -----Original Message----- >> From: owner-freebsd-performance@freebsd.org [mailto:owner-freebsd- >> performance@freebsd.org] On Behalf Of ivo tasev >> Sent: Friday, May 11, 2007 2:02 AM >> To: freebsd-performance@freebsd.org >> Subject: 6.2 Stable + Mysql 5.0 poor performance >> >> Hi all, >> i have machine with 2 x Quad Core Xeon cpu`s 2G ram and 4 3ware disks >> in >> raid10. There is 6.2 Stable on it and mysql 5.0 from the ports. >> >> The mysql is installed with the following build options: >> WITH_XCHARSET=all WITH_OPENSSL=yes BUILD_OPTIMIZED=yes > WITH_ARCHIVE=yes >> WITH_FEDERATED=yes WITH_NDB=yes >> >> I`m using libmap.conf with this in it: >> >> [mysqld] >> libpthread.so.2 libthr.so.2 >> >> This is what I have in my.cnf : >> >> set-variable = key_buffer=1024M >> set-variable = max_allowed_packet=64M >> set-variable = thread_stack=1024K >> set-variable = read_buffer_size=2M >> set-variable = read_buffer_size=2M >> >> set-variable = max_connections=350 >> set-variable = interactive_timeout=100 >> set-variable = wait_timeout=120 >> set-variable = max_user_connections=340 >> >> set-variable = query_cache_limit=1M >> set-variable = query_cache_size=32M >> set-variable = query_cache_type=1 >> >> set-variable = table_cache=1024 >> set-variable = thread_cache=128 >> set-variable = thread_cache_size=40 >> set-variable = thread_concurrency=16 >> >> I`m performing the following test with super-smack: >> :~# time super-smack -d mysql >> /usr/local/share/super-smack/select-key.smack 10 10000 >> >> the results are:Query Barrel Report for client smacker1 >> connect: max=1ms min=0ms avg= 0ms from 10 clients >> Query_type num_queries max_time min_time > q_per_s >> select_index 200000 0 0 18599.71 >> >> I can achieve better performance even on my colleague`s notebook with >> 6.2 stable !? >> >> I`ll be very thankful if someone can give me any ideas:) >> > Here are some system tweaks that may help. > > /etc/sysctl.conf > > kern.threads.max_groups_per_proc=40000 > kern.threads.max_threads_per_proc=40000 > kern.maxfiles=65535 > kern.maxfilesperproc=65535 > > /etc/libmap.conf > > [mysqld] > libpthread.so.2 libthr.so.2 > libpthread.so libthr.so > > /boot/loader.conf > > kern.maxdsiz="1073741824" # 1GB > kern.dfldsiz="1073741824" # 1GB > kern.maxssiz="134217728" # 128MB > > Mike This doesn't change a thing for me. What I can't understand is how this is possible: On my laptop - CPU: Intel(R) Core(TM)2 CPU T7200 @ 2.00GHz (1995.02-MHz K8-class CPU) avail memory = 1024991232 (977 MB) FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs ad4: 76319MB at ata2-master SATA150 FreeBSD hater.cmotd.com 6.2-STABLE FreeBSD 6.2-STABLE #6: Wed Apr 25 14:27:07 EEST 2007 root@hater.cmotd.com:/usr/obj/usr/src/sys/CORE64-SMP amd64 I have started KDE and thunderbird mail client during tests: Test : for i in 1 2 3 4 5; do super-smack -d mysql /usr/local/share/super-smack/select-key.smack 10 1000 | grep select_index; done 1)Default settings: select_index 20000 0 0 12895.14 select_index 20000 0 0 12782.88 select_index 20000 0 0 12769.19 select_index 20000 1 0 12645.70 select_index 20000 0 0 12772.67 2)kern.timecounter.hardware: ACPI-fast -> TSC select_index 20000 0 0 14629.49 select_index 20000 1 0 14826.03 select_index 20000 1 0 14789.27 select_index 20000 0 0 15157.29 select_index 20000 0 0 14494.76 3)Changing /etc/libmap.conf to: [mysqld] libpthread.so.2 libthr.so.2 libpthread.so libthr.so select_index 20000 0 0 19290.25 select_index 20000 0 0 18872.48 select_index 20000 0 0 19399.30 select_index 20000 0 0 19038.62 select_index 20000 0 0 19480.00 4)cp /usr/local/share/mysql/my-large.cnf /var/db/mysql/my.cnf select_index 20000 0 0 35112.42 select_index 20000 0 0 33295.93 select_index 20000 0 0 33616.33 select_index 20000 1 0 33415.43 select_index 20000 0 0 33200.31 So the results seems ok for me and quite impressive :) but look at this: CPU: Intel(R) Xeon(R) CPU 5130 @ 2.00GHz (2002.99-MHz K8-class CPU) avail memory = 8265285632 (7882 MB) FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs da0 at arcmsr0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 166.666MB/s transfers (83.333MHz, offset 32, 16bit), Tagged Queueing Enabled da0: 610351MB (1249999872 512 byte sectors: 255H 63S/T 77808C) (RAID 10 4x160GB SATA) FreeBSD tiger.cmotd.com 6.2-STABLE FreeBSD 6.2-STABLE #5: Mon May 14 17:12:39 EEST 2007 root@tiger.cmotd.com:/usr/obj/usr/src/sys/CORE64-SMP amd64 1)Default settings: select_index 20000 1 0 8075.95 select_index 20000 1 0 8111.38 select_index 20000 1 0 8329.87 select_index 20000 1 0 8105.15 select_index 20000 1 0 8108.48 2)kern.timecounter.hardware: HPET -> TSC select_index 20000 1 0 8927.00 select_index 20000 1 0 8978.22 select_index 20000 1 0 9076.20 select_index 20000 1 0 9043.15 select_index 20000 1 0 9158.74 3)Changing /etc/libmap.conf select_index 20000 0 0 35888.48 select_index 20000 0 0 38044.08 select_index 20000 0 0 37984.83 select_index 20000 0 0 37763.64 select_index 20000 0 0 37799.25 4)cp /usr/local/share/mysql/my-large.cnf /var/db/mysql/my.cnf select_index 20000 0 0 40816.66 select_index 20000 0 0 39808.36 select_index 20000 0 0 39508.28 select_index 20000 0 0 38147.48 select_index 20000 0 0 38002.08 As you can see in 1) & 2) my laptop outperform server that should be at least twice more powerful. and the final results 4) - ~33K tps vs ~39 tps ... is little disappointing ( have in mind on my laptop during benchmarks I have working Xorg, KDE, Thunderbird, skype and few others not so greedy apps) When I have little more time I'll rerun the bench without X started. And here is the diff -urN between 2 kernels: --- CORE64-SMP Mon May 14 12:52:41 2007 (laptop 2 cores) +++ CORE64-SMP-TIGER Mon May 14 12:52:43 2007 (4 cores) -options ACCEPT_FILTER_HTTP -options ACCEPT_FILTER_DATA -options COMPAT_FREEBSD4 # Compatible with FreeBSD4 -options COMPAT_FREEBSD5 # Compatible with FreeBSD5 -device ataraid # ATA RAID drives -device scbus # SCSI bus (required for SCSI) -device da # Direct Access (disks) -device cd # CD -device pass # Passthrough device (direct SCSI access) -device ses # SCSI Environmental Services (and SAF-TE) +device scbus # SCSI bus (required for SCSI) +device da # Direct Access (disks) +device pass # Passthrough device (direct SCSI access) +device arcmsr # Areca SATA II RAID -device cbb # cardbus (yenta) bridge -device pccard # PC Card (16-bit) bus -device cardbus # CardBus (32-bit) bus -device miibus # MII bus support -device bge # Broadcom BCM570xx Gigabit Ethernet -device wlan # 802.11 support -device wlan_wep # 802.11 WEP support -device wlan_ccmp # 802.11 CCMP support -device wlan_tkip # 802.11 TKIP support -device ath # Atheros pci/cardbus NIC's -device ath_hal # Atheros HAL (Hardware Access Layer) -device ath_rate_sample # SampleRate tx rate control for ath -device firmware +device em # Intel PRO/1000 adapter Gigabit -device -device sbp # SCSI over FireWire (Requires scbus and da) -device fwe # Ethernet over FireWire (non-standard!) #addons -device pf -device pflog -device pfsync -device carp -options IPSTEALTH -options NETSMB -options NETSMBCRYPTO -options SMBFS -options LIBMCHAIN -options LIBICONV +device drm +device radeondrm Do you need more info? -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-performance@FreeBSD.ORG Mon May 14 11:59:24 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9A5F216A400 for ; Mon, 14 May 2007 11:59:24 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from multiplay.co.uk (core6.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 2705C13C48C for ; Mon, 14 May 2007 11:59:23 +0000 (UTC) (envelope-from killing@multiplay.co.uk) X-Spam-Checker-Version: SpamAssassin 3.1.5 (2006-08-29) on core6.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-24.7 required=6.0 tests=BAYES_00, USER_IN_WHITELIST, USER_IN_WHITELIST_TO autolearn=ham version=3.1.5 Received: from r2d2 ([212.135.219.182]) by multiplay.co.uk (multiplay.co.uk [85.236.96.23]) (MDaemon PRO v9.5.4) with ESMTP id md50003818757.msg for ; Mon, 14 May 2007 12:55:03 +0100 Message-ID: <018401c7961e$b0732570$b6db87d4@multiplay.co.uk> From: "Steven Hartland" To: "Cheffo" , "Michael K. Smith - Adhost" References: <46443103.9010008@otel.net><17838240D9A5544AAA5FF95F8D52031602018EFA@ad-exh01.adhost.lan> <46483584.70608@FreeBSD-BG.org> Date: Mon, 14 May 2007 12:54:55 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="windows-1251"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3028 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3028 X-MDRemoteIP: 212.135.219.182 X-Return-Path: killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-performance@freebsd.org X-Spam-Processed: multiplay.co.uk, Mon, 14 May 2007 12:55:04 +0100 X-MDAV-Processed: multiplay.co.uk, Mon, 14 May 2007 12:55:07 +0100 Cc: freebsd-performance@freebsd.org, ivo tasev Subject: Re: 6.2 Stable + Mysql 5.0 poor performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 May 2007 11:59:24 -0000 ----- Original Message ----- From: "Cheffo" > CPU: Intel(R) Xeon(R) CPU 5130 @ 2.00GHz (2002.99-MHz K8-class CPU) > avail memory = 8265285632 (7882 MB) > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs > da0 at arcmsr0 bus 0 target 0 lun 0 > da0: Fixed Direct Access SCSI-5 device > da0: 166.666MB/s transfers (83.333MHz, offset 32, 16bit), Tagged Queueing Enabled > da0: 610351MB (1249999872 512 byte sectors: 255H 63S/T 77808C) > (RAID 10 4x160GB SATA) > FreeBSD tiger.cmotd.com 6.2-STABLE FreeBSD 6.2-STABLE #5: Mon May 14 17:12:39 EEST 2007 > root@tiger.cmotd.com:/usr/obj/usr/src/sys/CORE64-SMP amd64 > > 1)Default settings: > > select_index 20000 1 0 8075.95 > select_index 20000 1 0 8111.38 > select_index 20000 1 0 8329.87 > select_index 20000 1 0 8105.15 > select_index 20000 1 0 8108.48 > > 2)kern.timecounter.hardware: HPET -> TSC > > select_index 20000 1 0 8927.00 > select_index 20000 1 0 8978.22 > select_index 20000 1 0 9076.20 > select_index 20000 1 0 9043.15 > select_index 20000 1 0 9158.74 > > 3)Changing /etc/libmap.conf > > select_index 20000 0 0 35888.48 > select_index 20000 0 0 38044.08 > select_index 20000 0 0 37984.83 > select_index 20000 0 0 37763.64 > select_index 20000 0 0 37799.25 > > 4)cp /usr/local/share/mysql/my-large.cnf /var/db/mysql/my.cnf > > select_index 20000 0 0 40816.66 > select_index 20000 0 0 39808.36 > select_index 20000 0 0 39508.28 > select_index 20000 0 0 38147.48 > select_index 20000 0 0 38002.08 This sparked me to test our machine here: CPU: Intel(R) Xeon(R) CPU X5355 @ 2.66GHz (2666.68-MHz K8-class CPU) real memory = 9395240960 (8960 MB) avail memory = 8295161856 (7910 MB) FreeBSD/SMP: Multiprocessor System Detected: 8 CPU da0 at arcmsr0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 166.666MB/s transfers (83.333MHz, offset 32, 16bit), Tagged Queueing Enabled da0: 333785MB (683591680 512 byte sectors: 255H 63S/T 42551C) FreeBSD test 6.2-RELEASE-p3 FreeBSD 6.2-RELEASE-p3 #2: Thu Apr 26 15:20:20 BST 2007 root@test1:/usr/src/sys/amd64/compile/MPUK_SMP amd64 Using all the enhanced settings mentioned except TSC as it doesnt seem to apply even when set. select_index 200000 0 0 23990.95 select_index 200000 0 0 21578.29 select_index 200000 0 0 21342.61 select_index 200000 0 0 20808.31 select_index 200000 0 0 22401.80 So something very bad is happening here as with 21.28Ghz of CPU its scoring half that of your machine which only has 8Ghz. Is STABLE that much quicker than RELEASE? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Mon May 14 12:01:23 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D1A2516A405 for ; Mon, 14 May 2007 12:01:23 +0000 (UTC) (envelope-from cheffo@FreeBSD-BG.org) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id 8D20913C44B for ; Mon, 14 May 2007 12:01:23 +0000 (UTC) (envelope-from cheffo@FreeBSD-BG.org) Received: from blah.sun-fish.com (localhost [127.0.0.1]) by blah.sun-fish.com (Postfix) with ESMTP id 8A4DD1B10EE9; Mon, 14 May 2007 14:01:22 +0200 (CEST) Received: from hater.cmotd.com (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id 876531B10EE0; Mon, 14 May 2007 14:01:22 +0200 (CEST) Message-ID: <46484F91.4020803@FreeBSD-BG.org> Date: Mon, 14 May 2007 15:01:21 +0300 From: Cheffo User-Agent: Thunderbird 2.0.0.0 (X11/20070425) MIME-Version: 1.0 To: Alexandr References: <46443103.9010008@otel.net><17838240D9A5544AAA5FF95F8D52031602018EFA@ad-exh01.adhost.lan> <46483584.70608@FreeBSD-BG.org> <017f01c79614$dbae6420$0200a8c0@ssoniccp3pvd> In-Reply-To: <017f01c79614$dbae6420$0200a8c0@ssoniccp3pvd> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV using ClamSMTP on BLAH Cc: freebsd-performance@freebsd.org, ivo tasev Subject: Re: 6.2 Stable + Mysql 5.0 poor performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 May 2007 12:01:23 -0000 Alexandr wrote: >> Do you need more info? > > I think you should look at this > http://people.freebsd.org/~kris/scaling/mysql.html > > What exactly you suggest ? :) Increasing thread_concurrency from 8 to 16 brings me worst results (I have only 4 cores not 8, and this is supposed to be CPUs*2 ) Or I miss something else from this post ? BTW I'm using 4BSD scheduler not ULE (big improvements in ULE are in 7.x right?) -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-performance@FreeBSD.ORG Mon May 14 14:10:12 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A276D16A403 for ; Mon, 14 May 2007 14:10:12 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 3202B13C4AD for ; Mon, 14 May 2007 14:10:11 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from root by ciao.gmane.org with local (Exim 4.43) id 1HnbFS-0004kH-Qk for freebsd-performance@freebsd.org; Mon, 14 May 2007 16:10:02 +0200 Received: from synergetica.dn.ua ([82.207.115.117]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 14 May 2007 16:10:02 +0200 Received: from c.kworr by synergetica.dn.ua with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 14 May 2007 16:10:02 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: Volodymyr Kostyrko Date: Mon, 14 May 2007 16:19:29 +0300 Lines: 54 Message-ID: References: <46443103.9010008@otel.net> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: synergetica.dn.ua User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; ru-RU; rv:1.8.1.2) Gecko/20070302 SeaMonkey/1.1.1 In-Reply-To: <46443103.9010008@otel.net> Sender: news X-Mailman-Approved-At: Mon, 14 May 2007 16:05:40 +0000 Subject: Re: 6.2 Stable + Mysql 5.0 poor performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 May 2007 14:10:12 -0000 ivo tasev wrote: > Hi all, > i have machine with 2 x Quad Core Xeon cpu`s 2G ram and 4 3ware disks in > raid10. There is 6.2 Stable on it and mysql 5.0 from the ports. > > The mysql is installed with the following build options: > WITH_XCHARSET=all WITH_OPENSSL=yes BUILD_OPTIMIZED=yes WITH_ARCHIVE=yes > WITH_FEDERATED=yes WITH_NDB=yes > > I`m using libmap.conf with this in it: > > [mysqld] > libpthread.so.2 libthr.so.2 > > This is what I have in my.cnf : > > set-variable = key_buffer=1024M > set-variable = max_allowed_packet=64M > set-variable = thread_stack=1024K > set-variable = read_buffer_size=2M > set-variable = read_buffer_size=2M > > set-variable = max_connections=350 > set-variable = interactive_timeout=100 > set-variable = wait_timeout=120 > set-variable = max_user_connections=340 > > set-variable = query_cache_limit=1M > set-variable = query_cache_size=32M > set-variable = query_cache_type=1 > > set-variable = table_cache=1024 > set-variable = thread_cache=128 > set-variable = thread_cache_size=40 > set-variable = thread_concurrency=16 > > I`m performing the following test with super-smack: > :~# time super-smack -d mysql > /usr/local/share/super-smack/select-key.smack 10 10000 > > the results are:Query Barrel Report for client smacker1 > connect: max=1ms min=0ms avg= 0ms from 10 clients > Query_type num_queries max_time min_time q_per_s > select_index 200000 0 0 18599.71 > > I can achieve better performance even on my colleague`s notebook with > 6.2 stable !? > > I`ll be very thankful if someone can give me any ideas:) Try looking at hard drive usage. -- Sphinx of black quartz judge my vow! From owner-freebsd-performance@FreeBSD.ORG Mon May 14 20:39:39 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D249116A402 for ; Mon, 14 May 2007 20:39:39 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from multiplay.co.uk (core6.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 6F0F013C43E for ; Mon, 14 May 2007 20:39:39 +0000 (UTC) (envelope-from killing@multiplay.co.uk) X-Spam-Checker-Version: SpamAssassin 3.1.5 (2006-08-29) on core6.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-24.7 required=6.0 tests=BAYES_00, USER_IN_WHITELIST, USER_IN_WHITELIST_TO autolearn=ham version=3.1.5 Received: from r2d2 ([212.135.219.182]) by multiplay.co.uk (multiplay.co.uk [85.236.96.23]) (MDaemon PRO v9.5.4) with ESMTP id md50003820566.msg for ; Mon, 14 May 2007 21:38:18 +0100 Message-ID: <01bd01c79667$cb0133c0$b6db87d4@multiplay.co.uk> From: "Steven Hartland" To: , "Volodymyr Kostyrko" References: <46443103.9010008@otel.net> Date: Mon, 14 May 2007 21:38:12 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3028 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3028 X-MDRemoteIP: 212.135.219.182 X-Return-Path: killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-performance@freebsd.org X-Spam-Processed: multiplay.co.uk, Mon, 14 May 2007 21:38:19 +0100 X-MDAV-Processed: multiplay.co.uk, Mon, 14 May 2007 21:38:19 +0100 Cc: Subject: Re: 6.2 Stable + Mysql 5.0 poor performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 May 2007 20:39:39 -0000 ----- Original Message ----- From: "Volodymyr Kostyrko" >> I`ll be very thankful if someone can give me any ideas:) > > Try looking at hard drive usage. Totally idle during the test ( all in cache ) ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Tue May 15 03:40:02 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 488F116A406 for ; Tue, 15 May 2007 03:40:02 +0000 (UTC) (envelope-from schulra@earlham.edu) Received: from sipala.earlham.edu (sipala.earlham.edu [159.28.1.75]) by mx1.freebsd.org (Postfix) with ESMTP id 2495A13C459 for ; Tue, 15 May 2007 03:40:01 +0000 (UTC) (envelope-from schulra@earlham.edu) Received: from tdream.lly.earlham.edu (tdream.lly.earlham.edu [159.28.7.241]) by sipala.earlham.edu (8.13.6/8.13.6) with ESMTP id l4F3e1JO003061; Mon, 14 May 2007 23:40:01 -0400 (EDT) Date: Mon, 14 May 2007 23:41:20 -0400 (EDT) From: Randy Schultz X-X-Sender: schulra@tdream.lly.earlham.edu To: Matthew Jacob In-Reply-To: <7579f7fb0705110735h1a65ef7atcab00bdbc25224d6@mail.gmail.com> Message-ID: References: <464474F0.3040306@tomjudge.com> <7579f7fb0705110735h1a65ef7atcab00bdbc25224d6@mail.gmail.com> MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="0-388326016-1179200480=:58912" Cc: freebsd-performance@freebsd.org Subject: Dell SAS 5i/mpt driver[was: possible issues with Dell/Perc5 raid] X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 May 2007 03:40:02 -0000 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --0-388326016-1179200480=:58912 Content-Type: TEXT/PLAIN; charset=US-ASCII On Fri, 11 May 2007, Matthew Jacob spaketh thusly: -}Time frame to resolution involves getting a machine into my lab that -}evidences the poor performance and the time to sort out what the -}problem is and fix it. Remote access doesn't work for me on this one -}(contact me offline as to why). Hey Matthew, FWIW, I installed fedora core 5 on the box. The difference was significant. Writes, though bursty, increased significantly. Reads increased by 1-2 orders of magnitude. I've attached the output of blogbench for fc5 FWIW. Being new to all this, what would be the next step? Should I file a bug report? Or perhaps I should be asking what your timeline is? All I have to offer for help is remote access, which in this case doesn't work for you, or perhaps testing as my C skills are fairly pathetic. ;> -- Randy (schulra@earlham.edu) 765.983.1283 <*> Rain puts a hole in stone because of its constancy, not its force. - H. Joseph Gerber --0-388326016-1179200480=:58912 Content-Type: TEXT/PLAIN; charset=US-ASCII; name=dell.fc5 Content-Transfer-Encoding: BASE64 Content-ID: Content-Description: Content-Disposition: attachment; filename=dell.fc5 W3Jvb3RAa3V0aHVsYSBibG9nYmVuY2gtMS4wXSMgYmxvZ2JlbmNoIC1kIC92 YXIvdG1wDQoNCkZyZXF1ZW5jeSA9IDEwIHNlY3MNClNjcmF0Y2ggZGlyID0g Wy92YXIvdG1wXQ0KU3Bhd25pbmcgMyB3cml0ZXJzLi4uDQpTcGF3bmluZyAx IHJld3JpdGVycy4uLg0KU3Bhd25pbmcgNSBjb21tZW50ZXJzLi4uDQpTcGF3 bmluZyAxMDAgcmVhZGVycy4uLg0KQmVuY2htYXJraW5nIGZvciAzMCBpdGVy YXRpb25zLg0KVGhlIHRlc3Qgd2lsbCBydW4gZHVyaW5nIDUgbWludXRlcy4N Cg0KICBOYiBibG9ncyAgIFIgYXJ0aWNsZXMgICAgVyBhcnRpY2xlcyAgICBS IHBpY3R1cmVzICAgIFcgcGljdHVyZXMgICAgUiBjb21tZW50cyAgICBXIGNv bW1lbnRzDQogICAgICAgIDM3ICAgICAgIDIzNzM5OSAgICAgICAgICAyMTA3 ICAgICAgICAxNzMzNDUgICAgICAgICAgMjAwMCAgICAgICAgIDY3OTY1ICAg ICAgICAgIDI0OTENCiAgICAgICAgMzcgICAgICAgMjQ0MjUzICAgICAgICAg ICAgIDAgICAgICAgIDE3Mjc0NiAgICAgICAgICAgICAwICAgICAgICAgNjk1 NzQgICAgICAgICAgICAgMA0KICAgICAgICA0MSAgICAgICAyNDE2MzYgICAg ICAgICAgIDQ2MyAgICAgICAgMTY0ODc5ICAgICAgICAgICAyMzYgICAgICAg ICA4MjE3MyAgICAgICAgICAxODQ0DQogICAgICAgIDQxICAgICAgIDIzNTY4 OSAgICAgICAgICAgNDAxICAgICAgICAxNjI2MjAgICAgICAgICAgIDE3MyAg ICAgICAgMTEwMjkyICAgICAgICAgICAgIDANCiAgICAgICAgNDcgICAgICAg MjI5ODM5ICAgICAgICAgICA0MzQgICAgICAgIDE2MzA2NiAgICAgICAgICAg NDEwICAgICAgICAxMzE5OTMgICAgICAgICAgNDg2NQ0KICAgICAgICA0NyAg ICAgICAyMzAxMjcgICAgICAgICAgICAxMSAgICAgICAgMTYzOTgyICAgICAg ICAgICAgIDkgICAgICAgIDEzODIxMyAgICAgICAgICAgIDY0DQogICAgICAg IDQ3ICAgICAgIDIyNTUzNSAgICAgICAgICAgIDc1ICAgICAgICAxNjA0NzEg ICAgICAgICAgICA0MCAgICAgICAgMTQ4NDkxICAgICAgICAgIDE2MDgNCiAg ICAgICAgNDcgICAgICAgMjI5MDgwICAgICAgICAgICAgIDAgICAgICAgIDE2 MTc0MSAgICAgICAgICAgICAwICAgICAgICAxNTU5NzkgICAgICAgICAgICAg MA0KICAgICAgICA0NyAgICAgICAyMjc1NzQgICAgICAgICAgIDEwNyAgICAg ICAgMTYzMjAyICAgICAgICAgICAgNTYgICAgICAgIDE3MDE2OSAgICAgICAg ICAxNTMwDQogICAgICAgIDQ3ICAgICAgIDIyNTA0MyAgICAgICAgICAgICAw ICAgICAgICAxNjAxNzAgICAgICAgICAgICAgMCAgICAgICAgMTczNDE4ICAg ICAgICAgICAgIDENCiAgICAgICAgNDcgICAgICAgMjE4MjUzICAgICAgICAg ICA1MDEgICAgICAgIDE1NTgwNCAgICAgICAgICAgMjYyICAgICAgICAxNzQy MzQgICAgICAgICAgMTc2OQ0KICAgICAgICA0NyAgICAgICAyMjA1MjEgICAg ICAgICAgICAyMCAgICAgICAgMTYyNzMyICAgICAgICAgICAgIDggICAgICAg IDE3NTk4NyAgICAgICAgICAgMjYxDQogICAgICAgIDQ3ICAgICAgIDIxOTIz MiAgICAgICAgICAgICAwICAgICAgICAxNjExMTggICAgICAgICAgICAgMCAg ICAgICAgMTgxMDAzICAgICAgICAgICAgIDANCiAgICAgICAgNDcgICAgICAg MjE0MTY4ICAgICAgICAgICAgIDAgICAgICAgIDE1NTc1MyAgICAgICAgICAg ICAwICAgICAgICAxNzM5MzggICAgICAgICAgICAgMg0KICAgICAgICA0OCAg ICAgICAyMTczODAgICAgICAgICAgICA3MSAgICAgICAgMTU4ODg1ICAgICAg ICAgICAgOTUgICAgICAgIDE3MDEzMiAgICAgICAgICAxMjY5DQogICAgICAg IDQ4ICAgICAgIDIxODYxNyAgICAgICAgICAgICAwICAgICAgICAxNTg3MzYg ICAgICAgICAgICAgMCAgICAgICAgMTcxMjc3ICAgICAgICAgIDExMzcNCiAg ICAgICAgNDggICAgICAgMjIzMTYyICAgICAgICAgICAgIDAgICAgICAgIDE2 MzEzNCAgICAgICAgICAgICAwICAgICAgICAxNzU2NTggICAgICAgICAgIDEz MA0KICAgICAgICA1NiAgICAgICAyMjI2MzMgICAgICAgICAgIDM2MyAgICAg ICAgMTY1NzQwICAgICAgICAgICA0MjQgICAgICAgIDE2MzQ3MyAgICAgICAg ICAyMTU1DQogICAgICAgIDU2ICAgICAgIDIxNzI0MSAgICAgICAgICAgICAw ICAgICAgICAxNjA1NTQgICAgICAgICAgICAgMCAgICAgICAgMTY0NzY4ICAg ICAgICAgIDE4MDMNCiAgICAgICAgNTYgICAgICAgMjI1NTc1ICAgICAgICAg ICAgMTQgICAgICAgIDE2ODgyNiAgICAgICAgICAgICA2ICAgICAgICAxNzc3 MDggICAgICAgICAgMjg4OQ0KICAgICAgICA1NiAgICAgICAyMTE3NzQgICAg ICAgICAgICAgMCAgICAgICAgMTU2MzI0ICAgICAgICAgICAgIDAgICAgICAg IDE3MjMzNyAgICAgICAgICAgMTY5DQogICAgICAgIDU2ICAgICAgIDIxNDM1 MSAgICAgICAgICAgIDU2ICAgICAgICAxNTkyNzUgICAgICAgICAgICAyNSAg ICAgICAgMTcyNTAzICAgICAgICAgIDMyOTgNCiAgICAgICAgNTcgICAgICAg MjE1MjQxICAgICAgICAgICAxMTggICAgICAgIDE2MTQ4NyAgICAgICAgICAg IDYyICAgICAgICAxNzg4MTcgICAgICAgICAgIDEzNA0KICAgICAgICA1NyAg ICAgICAyMTYyNTMgICAgICAgICAgIDQyMCAgICAgICAgMTYyODkxICAgICAg ICAgICAyMDAgICAgICAgIDE3NTUwMSAgICAgICAgICAgMjAwDQogICAgICAg IDYzICAgICAgIDIxODAzNiAgICAgICAgICAgNDU4ICAgICAgICAxNjI2OTUg ICAgICAgICAgIDQyOCAgICAgICAgMTcyMjU1ICAgICAgICAgIDQwNjENCiAg ICAgICAgNjMgICAgICAgMjEyMzUyICAgICAgICAgICAgNTAgICAgICAgIDE2 MDgwMyAgICAgICAgICAgIDI0ICAgICAgICAxNzM2MjQgICAgICAgICAgNjMx Ng0KICAgICAgICA2NSAgICAgICAyMTk1ODAgICAgICAgICAgIDI4OSAgICAg ICAgMTY3ODQ3ICAgICAgICAgICAyMDIgICAgICAgIDE5NjkzMSAgICAgICAg ICA1MjU2DQogICAgICAgIDc1ICAgICAgIDIyMDg5MCAgICAgICAgICAgNDgx ICAgICAgICAxNjUzNjQgICAgICAgICAgIDU3OCAgICAgICAgMTgwNDM5ICAg ICAgICAgIDQxOTINCiAgICAgICAgODUgICAgICAgMjI3NDAxICAgICAgICAg ICA2NzIgICAgICAgIDE3NDEzNSAgICAgICAgICAgNjk2ICAgICAgICAxODQ2 NzkgICAgICAgICAgNDM0OA0KICAgICAgIDEwMyAgICAgICAyMzIyMjIgICAg ICAgICAgIDgwNSAgICAgICAgMTY4ODI0ICAgICAgICAgICA4MjEgICAgICAg IDE3MzY4MiAgICAgICAgICA0ODY5DQoNCkZpbmFsIHNjb3JlIGZvciB3cml0 ZXM6ICAgICAgICAgICAxMDMNCkZpbmFsIHNjb3JlIGZvciByZWFkcyA6ICAg ICAgICAxMjg2ODQNCg0KDQo= --0-388326016-1179200480=:58912-- From owner-freebsd-performance@FreeBSD.ORG Tue May 15 12:40:46 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E5C3416A400 for ; Tue, 15 May 2007 12:40:45 +0000 (UTC) (envelope-from kkobb@skylinecorp.com) Received: from mail.skylinecorp.com (mail.skylinecorp.com [216.117.21.137]) by mx1.freebsd.org (Postfix) with ESMTP id 9BD1213C455 for ; Tue, 15 May 2007 12:40:40 +0000 (UTC) (envelope-from kkobb@skylinecorp.com) Received: from [192.168.1.149] (in-elk-216-117-2-1.skylinecorp.com [216.117.2.1]) by mail.skylinecorp.com (8.12.9/8.12.9) with ESMTP id l4FCRo6A011098; Tue, 15 May 2007 08:27:50 -0400 Message-ID: <4649A71C.4030302@skylinecorp.com> Date: Tue, 15 May 2007 08:27:08 -0400 From: Kevin Kobb User-Agent: Thunderbird 2.0.0.0 (Windows/20070326) MIME-Version: 1.0 To: Matthew Jacob References: <464474F0.3040306@tomjudge.com> <7579f7fb0705110735h1a65ef7atcab00bdbc25224d6@mail.gmail.com> In-Reply-To: <7579f7fb0705110735h1a65ef7atcab00bdbc25224d6@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-SkylineCorp-MailScanner-Information: Please contact the ISP for more information X-SkylineCorp-MailScanner: Found to be clean X-SkylineCorp-MailScanner-SpamCheck: not spam, SpamAssassin (not cached, score=-1.796, required 4.25, autolearn=not spam, AWL -0.60, BAYES_00 -1.20) X-MailScanner-From: kkobb@skylinecorp.com X-Mailman-Approved-At: Tue, 15 May 2007 13:06:42 +0000 Cc: freebsd-performance@freebsd.org Subject: Re: possible issues with Dell/Perc5 raid X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 May 2007 12:40:46 -0000 Matthew Jacob wrote: > Time frame to resolution involves getting a machine into my lab that > evidences the poor performance and the time to sort out what the > problem is and fix it. Remote access doesn't work for me on this one > (contact me offline as to why). > > On 5/11/07, Tom Judge wrote: >> Randy Schultz wrote: >> > Hi there, >> > >> > We just purchased a Dell 860 with these specifics: >> > - dual core pentium 915, 2.8 GHz, 800MHz FSB >> > - 2x512 MB 533 MHz DDR2 RAM >> > - Dell's SAS/SATA drive controller(which is actually an LSILogic) >> > - their 5IR PCI-Express internal RAID controller >> > - a pair of 250 GB SATA II's set up RAID 1 >> > >> > After installing 6.2-RELEASE on the system I noticed, while pulling the >> > ports >> > collection off the CDROM, that the speed to the drives started out just >> > fine >> > but long before the ports collection was installed the transfer rate >> was >> > down >> > to ~35kB/s. Huh. So I started paying closer attention. I installed a >> > couple >> > of ports and every time there was anything but miniscule data being >> > transferred to/from disk the transfer rate bottomed out. I >> installed the >> > blogbench port and it confirmed the I/O as tremendously sluggish. I've >> > attached the output of the blogbench runs. The first is from an old >> 700 >> > MHz >> > pc, Acer M25C mobo, with 1 IDE drive just for comparison. The second >> > data set >> > is the new Dell. I've also attached the output of dmesg. >> Everything looks >> > fine except for the mpt lines. Would those be telling of something >> > amiss or >> > are they innocuous barks? >> > >> > I understand that RAID 1 is not the fastest but I certainly expected it >> > to be >> > faster than it is, esp. with hw RAID. Are my expectations too high and >> > this >> > is normal? >> > >> > -- >> >> From the boot messages you have included below it seems that you have a >> Dell SAS-5i (Internal PCIe) controller and not a Perc/5[ie]. >> >> The SAS-5 known to have very poor performance (1Mb/s 100%busy) under >> certain loads (cvsup). >> >> I have CC'd the author of the mpt driver (Matthew Jacob) as he may be >> able to shed more light on the problem and time frame to resolution. >> >> Tom >> >> > >> > Copyright (c) 1992-2007 The FreeBSD Project. >> > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, >> 1994 >> > The Regents of the University of California. All rights >> reserved. >> > FreeBSD is a registered trademark of The FreeBSD Foundation. >> > FreeBSD 6.2-STABLE #0: Mon May 7 20:16:35 EDT 2007 >> > root@maxu.earlham.edu:/usr/obj/usr/src/sys/maxu.070507 >> > Timecounter "i8254" frequency 1193182 Hz quality 0 >> > CPU: Intel(R) Pentium(R) D CPU 2.80GHz (2800.11-MHz 686-class CPU) >> > Origin = "GenuineIntel" Id = 0xf64 Stepping = 4 >> > >> Features=0xbfebfbff> >> > ,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> >> > Features2=0xe49d,> >> > AMD Features=0x20100000 >> > AMD Features2=0x1 >> > Cores per package: 2 >> > real memory = 1073479680 (1023 MB) >> > avail memory = 1041326080 (993 MB) >> > ACPI APIC Table: >> > FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs >> > cpu0 (BSP): APIC ID: 0 >> > cpu1 (AP): APIC ID: 1 >> >> > mpt0: port 0xec00-0xecff mem >> 0xfeafc000-0xfeafffff,0xfeae0000-0xf >> > eaeffff irq 16 at device 8.0 on pci2 >> > mpt0: [GIANT-LOCKED] >> > mpt0: MPI Version=1.5.13.0 >> > mpt0: mpt_cam_event: 0x16 >> > mpt0: Unhandled Event Notify Frame. Event 0x16 (ACK not required). >> > mpt0: mpt_cam_event: 0x12 >> > mpt0: Unhandled Event Notify Frame. Event 0x12 (ACK not required). >> > mpt0: mpt_cam_event: 0x12 >> > mpt0: Unhandled Event Notify Frame. Event 0x12 (ACK not required). >> > mpt0: mpt_cam_event: 0x16 >> > mpt0: Unhandled Event Notify Frame. Event 0x16 (ACK not required). >> >> > da0 at mpt0 bus 0 target 0 lun 0 >> > da0: Fixed Direct Access SCSI-5 device >> > da0: 300.000MB/s transfers, Tagged Queueing Enabled >> > da0: 237464MB (486326272 512 byte sectors: 255H 63S/T 30272C) >> > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to > "freebsd-performance-unsubscribe@freebsd.org" > These reports on poor performance using mpt seem to be on SATA drives. Has anybody been seeing this using SAS drives? We are testing Dell PE840s with hot swap SAS drives, and seem to get decent performance, though I haven't run any benchmarks yet. Any opinions if the PERC5i/mfi is a better choice than the SAS5iR/mpt combination? From owner-freebsd-performance@FreeBSD.ORG Tue May 15 13:55:54 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E722116A40B for ; Tue, 15 May 2007 13:55:54 +0000 (UTC) (envelope-from schulra@earlham.edu) Received: from sipala.earlham.edu (sipala.earlham.edu [159.28.1.75]) by mx1.freebsd.org (Postfix) with ESMTP id B920F13C447 for ; Tue, 15 May 2007 13:55:54 +0000 (UTC) (envelope-from schulra@earlham.edu) Received: from tdream.lly.earlham.edu (tdream.lly.earlham.edu [159.28.7.241]) by sipala.earlham.edu (8.13.6/8.13.6) with ESMTP id l4FDtsEM025240; Tue, 15 May 2007 09:55:54 -0400 (EDT) Date: Tue, 15 May 2007 09:57:13 -0400 (EDT) From: Randy Schultz X-X-Sender: schulra@tdream.lly.earlham.edu To: Kevin Kobb In-Reply-To: <4649A71C.4030302@skylinecorp.com> Message-ID: References: <464474F0.3040306@tomjudge.com> <7579f7fb0705110735h1a65ef7atcab00bdbc25224d6@mail.gmail.com> <4649A71C.4030302@skylinecorp.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: freebsd-performance@freebsd.org Subject: PERC5i throughput [was: possible issues with Dell/Perc5 raid] X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 May 2007 13:55:55 -0000 On Tue, 15 May 2007, Kevin Kobb spaketh thusly: -}These reports on poor performance using mpt seem to be on SATA -}drives. Has anybody been seeing this using SAS drives? -} -}We are testing Dell PE840s with hot swap SAS drives, and seem to get decent -}performance, though I haven't run any benchmarks yet. Any opinions if the -}PERC5i/mfi is a better choice than the SAS5iR/mpt combination? Would it be possible for you to install blogbench and give it a quick run while your system is idle? We will be ordering a new box in 1-2 months that we need high disk I/O capabilities. We've been looking at the PERC5i set up RAID 5 with 4-6 SAS drives. I'ld be interested in what you're seeing for throughput on the PERC5i. -- Randy (schulra@earlham.edu) 765.983.1283 <*> Rain puts a hole in stone because of its constancy, not its force. - H. Joseph Gerber From owner-freebsd-performance@FreeBSD.ORG Tue May 15 15:04:57 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8F4F316A400 for ; Tue, 15 May 2007 15:04:57 +0000 (UTC) (envelope-from tom@tomjudge.com) Received: from s200aog11.obsmtp.com (s200aog11.obsmtp.com [207.126.144.125]) by mx1.freebsd.org (Postfix) with SMTP id 2538213C45D for ; Tue, 15 May 2007 15:04:51 +0000 (UTC) (envelope-from tom@tomjudge.com) Received: from source ([217.206.187.80]) by eu1sys200aob011.postini.com ([207.126.147.11]) with SMTP; Tue, 15 May 2007 15:04:46 UTC Received: from [10.0.0.89] (bill.mintel.co.uk [10.0.0.89]) by rodney.mintel.co.uk (Postfix) with ESMTP id 447B018142B; Tue, 15 May 2007 16:04:46 +0100 (BST) Message-ID: <4649CC6B.1050203@tomjudge.com> Date: Tue, 15 May 2007 16:06:19 +0100 From: Tom Judge User-Agent: Thunderbird 1.5.0.10 (X11/20070306) MIME-Version: 1.0 To: Randy Schultz References: <464474F0.3040306@tomjudge.com> <7579f7fb0705110735h1a65ef7atcab00bdbc25224d6@mail.gmail.com> <4649A71C.4030302@skylinecorp.com> In-Reply-To: Content-Type: multipart/mixed; boundary="------------030905000100030409010208" Cc: freebsd-performance@freebsd.org, Kevin Kobb Subject: Re: PERC5i throughput [was: possible issues with Dell/Perc5 raid] X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 May 2007 15:04:57 -0000 This is a multi-part message in MIME format. --------------030905000100030409010208 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Randy Schultz wrote: > On Tue, 15 May 2007, Kevin Kobb spaketh thusly: > > -}These reports on poor performance using mpt seem to be on SATA > -}drives. Has anybody been seeing this using SAS drives? > -} > -}We are testing Dell PE840s with hot swap SAS drives, and seem to get decent > -}performance, though I haven't run any benchmarks yet. Any opinions if the > -}PERC5i/mfi is a better choice than the SAS5iR/mpt combination? > > Would it be possible for you to install blogbench and give it a quick run > while your system is idle? We will be ordering a new box in 1-2 months that > we need high disk I/O capabilities. We've been looking at the PERC5i set up > RAID 5 with 4-6 SAS drives. I'ld be interested in what you're seeing for > throughput on the PERC5i. > I have attached some blogbench tests from the following configs: Perc5i 4 * SAS 15K RPM 146 Gig disks in raid 5. Perc5e 15 * SATA 7.2K RPM 500 Gig disks in raid 50 (3 raid 5 volumes 5 disks each) Other configurations that I may be able to test are 2 SATA in raid 1 and 2 SAS in raid 1 (both on perc5i). --------------030905000100030409010208 Content-Type: text/plain; name="benchmarks.txt" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="benchmarks.txt" Perc5i 4 * SAS 15K RPM 146 Gig disks in raid 5. CPU: Intel(R) Xeon(R) CPU 5160 @ 3.00GHz (2995.54-MHz K8-class CPU) Frequency = 10 secs Scratch dir = [/data] Spawning 3 writers... Spawning 1 rewriters... Spawning 5 commenters... Spawning 100 readers... Benchmarking for 30 iterations. The test will run during 5 minutes. Nb blogs R articles W articles R pictures W pictures R comments W comments 15 13958 890 11193 986 10196 3139 29 22525 902 16806 636 14457 2668 41 22776 730 17004 718 14427 2473 50 20498 528 14791 664 10622 2184 57 16051 494 11838 514 8818 1630 67 25084 655 30781 748 15000 2578 74 15590 447 11598 345 8768 1270 78 10908 327 8137 211 6505 915 92 26503 1106 18745 613 15369 2827 96 16371 339 10909 373 7937 1103 103 15408 402 10736 287 9274 1140 114 22229 666 15143 566 10028 2079 122 17964 471 12721 466 9633 1551 127 17699 376 11947 289 9238 1061 134 12314 325 9021 431 6934 1293 139 20510 529 14899 471 12919 1684 143 19382 455 13374 253 10189 1080 147 10657 272 7126 214 5208 821 159 22676 657 14956 586 13469 2007 163 17458 147 11940 283 10322 759 168 14347 221 9881 274 7397 777 177 20031 616 13832 541 12240 1796 178 13840 135 9766 168 9133 488 184 19061 320 12801 246 10504 849 189 18462 501 12587 469 10500 1753 192 16019 228 10975 258 10133 769 194 14234 128 10232 220 6950 520 197 11275 82 8126 180 6392 501 204 19404 482 14123 424 12712 1495 204 11758 105 7814 160 8072 444 Final score for writes: 204 Final score for reads : 10414 Perc5e 15 * SATA 7.2K RPM 500 Gig disks in raid 50 (3 raid 5 volumes 5 disks each) CPU: Intel(R) Xeon(R) CPU 5110 @ 1.60GHz (1597.64-MHz K8-class CPU) Frequency = 10 secs Scratch dir = [/compere/test] Spawning 3 writers... Spawning 1 rewriters... Spawning 5 commenters... Spawning 100 readers... Benchmarking for 30 iterations. The test will run during 5 minutes. Nb blogs R articles W articles R pictures W pictures R comments W comments 1 4314 189 1999 109 341 439 5 14061 329 10016 250 9840 873 14 28924 416 23146 464 11733 1317 23 31039 470 23999 531 14171 1444 31 34713 523 27226 527 17347 1470 41 35581 416 27988 626 21633 1390 51 38418 491 29663 517 21576 1307 56 41466 506 32501 488 24634 1370 61 53966 561 41339 386 31210 1237 68 55243 504 42738 423 30776 1279 76 50607 508 38520 420 26856 1278 83 46003 471 35468 481 26124 1222 88 51993 483 40542 369 31031 1143 96 54179 505 42080 362 31857 1118 102 54421 349 40906 344 29561 940 106 60051 501 45166 354 33161 1113 113 62562 432 46139 347 37840 1041 120 67518 403 49800 463 40662 1148 128 62442 459 46504 357 35074 1062 134 62545 350 47605 391 38809 967 139 86653 402 64632 342 51244 1009 145 69267 377 52526 429 40214 1065 152 59945 390 45232 338 35558 929 158 81802 386 61518 412 48679 1078 164 71721 431 53173 317 40869 992 170 81977 432 61320 264 46273 886 174 77979 405 58017 323 45199 962 183 73185 367 54622 343 38939 956 189 74043 402 54737 299 42417 942 195 74745 325 57095 408 41014 962 Final score for writes: 195 Final score for reads : 35412 --------------030905000100030409010208-- From owner-freebsd-performance@FreeBSD.ORG Tue May 15 15:40:00 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E53AD16A400 for ; Tue, 15 May 2007 15:40:00 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from multiplay.co.uk (core6.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 6BB8213C43E for ; Tue, 15 May 2007 15:39:59 +0000 (UTC) (envelope-from killing@multiplay.co.uk) X-Spam-Checker-Version: SpamAssassin 3.1.5 (2006-08-29) on core6.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-24.7 required=6.0 tests=BAYES_00, USER_IN_WHITELIST, USER_IN_WHITELIST_TO autolearn=ham version=3.1.5 Received: from r2d2 ([212.135.219.182]) by multiplay.co.uk (multiplay.co.uk [85.236.96.23]) (MDaemon PRO v9.5.4) with ESMTP id md50003823047.msg for ; Tue, 15 May 2007 16:35:36 +0100 Message-ID: <156801c79706$aa776da0$b6db87d4@multiplay.co.uk> From: "Steven Hartland" To: "Randy Schultz" , "Kevin Kobb" References: <464474F0.3040306@tomjudge.com><7579f7fb0705110735h1a65ef7atcab00bdbc25224d6@mail.gmail.com><4649A71C.4030302@skylinecorp.com> Date: Tue, 15 May 2007 16:35:28 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3028 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3028 X-MDRemoteIP: 212.135.219.182 X-Return-Path: killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-performance@freebsd.org X-Spam-Processed: multiplay.co.uk, Tue, 15 May 2007 16:35:36 +0100 X-MDAV-Processed: multiplay.co.uk, Tue, 15 May 2007 16:35:38 +0100 Cc: freebsd-performance@freebsd.org Subject: Re: PERC5i throughput [was: possible issues with Dell/Perc5 raid] X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 May 2007 15:40:01 -0000 ----- Original Message ----- From: "Randy Schultz" > On Tue, 15 May 2007, Kevin Kobb spaketh thusly: > > -}These reports on poor performance using mpt seem to be on SATA > -}drives. Has anybody been seeing this using SAS drives? > -} > -}We are testing Dell PE840s with hot swap SAS drives, and seem to get decent > -}performance, though I haven't run any benchmarks yet. Any opinions if the > -}PERC5i/mfi is a better choice than the SAS5iR/mpt combination? > > Would it be possible for you to install blogbench and give it a quick run > while your system is idle? We will be ordering a new box in 1-2 months that > we need high disk I/O capabilities. We've been looking at the PERC5i set up > RAID 5 with 4-6 SAS drives. I'ld be interested in what you're seeing for > throughput on the PERC5i. Some more results for you: == Areca ARC-1220 == Areca ARC-1220 + Battery Backup with 8 * 150Gb WD Raptors (10K RPM) configured in RAID6 + Hot Spare so 7 working spindles results in: Frequency = 10 secs Scratch dir = [/data/tmp] Spawning 3 writers... Spawning 1 rewriters... Spawning 5 commenters... Spawning 100 readers... Benchmarking for 30 iterations. The test will run during 5 minutes. Nb blogs R articles W articles R pictures W pictures R comments W comments 41 37719 2228 22258 2013 23190 7008 57 39954 1043 24603 779 26945 3550 71 38746 1014 25218 666 23354 3414 86 36110 941 23533 703 22249 3282 97 32346 874 21055 740 20869 2981 112 36707 1056 23720 750 22823 3386 124 35534 888 23323 821 21940 3356 137 25196 660 16208 676 16977 2358 149 35040 840 23653 807 21080 3232 158 24329 718 16193 572 15066 2317 167 24366 532 16708 640 15233 2337 185 35928 997 24651 882 21375 3420 193 22099 715 15436 522 12676 2117 202 27299 809 18668 637 17300 2592 214 25118 754 16876 680 14600 2582 222 20329 667 13849 410 12075 1964 231 21969 600 15775 690 13738 2379 240 22660 750 16020 509 14186 2276 249 19831 745 13579 481 14324 2235 256 23765 668 16641 559 14750 2333 264 22115 627 15617 654 15476 2333 270 20561 532 14158 558 11860 2043 280 25558 723 17943 626 15589 2434 291 16869 598 11561 435 10055 1965 301 19401 546 14082 508 10716 1869 311 20494 453 15141 705 13066 2126 321 16084 554 11046 430 9324 2065 331 25701 752 17375 626 15792 2620 338 14169 409 10341 438 8506 1611 347 24724 680 18128 625 15903 2465 Final score for writes: 347 Final score for reads : 15832 Copyright (c) 1992-2007 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 6.2-RELEASE-p3 #2: Thu Apr 26 15:20:20 BST 2007 root@xxx:/usr/src/sys/amd64/compile/MPUK_SMP Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU X5355 @ 2.66GHz (2666.68-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x6f7 Stepping = 7 Features=0xbfebfbff Features2=0x4e3bd,CX16,,,> AMD Features=0x20100800 AMD Features2=0x1 Cores per package: 4 real memory = 9395240960 (8960 MB) avail memory = 8295161856 (7910 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs == Highpoint 1820a == Highpoint 1820a with 6 * 400Gb WD's (7.2K RPM) in RAID5 Frequency = 10 secs Scratch dir = [/usr/tmp] Spawning 3 writers... Spawning 1 rewriters... Spawning 5 commenters... Spawning 100 readers... Benchmarking for 30 iterations. The test will run during 5 minutes. Nb blogs R articles W articles R pictures W pictures R comments W comments 5 9334 443 5561 363 4166 1477 13 11977 166 8193 331 5334 773 16 10382 211 7732 202 5528 789 22 14594 351 11012 263 7480 1118 23 5804 39 4138 59 1516 378 26 9506 214 6429 323 4683 706 33 17630 356 12042 276 10262 1113 36 17793 339 12650 225 10952 1197 40 8724 218 6474 271 6319 710 45 10215 309 7221 259 6435 872 50 17840 453 12532 367 11503 1155 51 12039 132 9008 125 6669 734 52 8224 96 5970 79 4691 466 57 12451 386 8609 383 7810 1026 59 16391 216 11850 250 9473 871 61 5343 114 4069 98 3982 401 65 13373 269 10051 327 8619 956 70 10978 251 7848 279 8041 1114 70 20616 121 15436 76 11681 558 75 12599 313 9119 250 8175 1063 76 15073 135 11249 165 9889 602 77 12477 29 9612 45 8797 253 82 13614 302 10454 274 7634 1069 83 12168 117 9496 50 8031 373 87 11113 253 7938 220 7348 856 90 14509 334 10864 260 10642 989 91 9866 56 7755 62 6415 277 92 2930 37 2052 41 1413 207 96 19459 385 13961 275 12529 1231 98 10386 86 7774 124 6582 344 Final score for writes: 98 Final score for reads : 7875 Copyright (c) 1992-2007 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 6.2-RELEASE-p1 #2: Mon Mar 19 19:25:53 GMT 2007 root@xxx:/usr/src/sys/i386/compile/MPUK_SMP_200HZ ACPI APIC Table: Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Opteron(tm) Processor 244 (1794.42-MHz 686-class CPU) Origin = "AuthenticAMD" Id = 0xf5a Stepping = 10 Features=0x78bfbff AMD Features=0xe0500800 real memory = 2146893824 (2047 MB) avail memory = 2095816704 (1998 MB) FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Tue May 15 15:51:22 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4A3AC16A404 for ; Tue, 15 May 2007 15:51:22 +0000 (UTC) (envelope-from schulra@earlham.edu) Received: from sipala.earlham.edu (sipala.earlham.edu [159.28.1.75]) by mx1.freebsd.org (Postfix) with ESMTP id 23F9213C45D for ; Tue, 15 May 2007 15:51:21 +0000 (UTC) (envelope-from schulra@earlham.edu) Received: from tdream.lly.earlham.edu (tdream.lly.earlham.edu [159.28.7.241]) by sipala.earlham.edu (8.13.6/8.13.6) with ESMTP id l4FFpLvA010444 for ; Tue, 15 May 2007 11:51:21 -0400 (EDT) Date: Tue, 15 May 2007 11:52:41 -0400 (EDT) From: Randy Schultz X-X-Sender: schulra@tdream.lly.earlham.edu To: freebsd-performance@freebsd.org In-Reply-To: <4649CC6B.1050203@tomjudge.com> Message-ID: References: <464474F0.3040306@tomjudge.com> <7579f7fb0705110735h1a65ef7atcab00bdbc25224d6@mail.gmail.com> <4649A71C.4030302@skylinecorp.com> <4649CC6B.1050203@tomjudge.com> MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY=------------030905000100030409010208 Content-ID: Subject: Re: PERC5i throughput [was: possible issues with Dell/Perc5 raid] X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 May 2007 15:51:22 -0000 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --------------030905000100030409010208 Content-Type: TEXT/PLAIN; CHARSET=US-ASCII; FORMAT=flowed Content-ID: On Tue, 15 May 2007, Tom Judge spaketh thusly: -}I have attached some blogbench tests from the following configs: Woo-hoo! Tnx heaps and loads Tom. -} -}Perc5i 4 * SAS 15K RPM 146 Gig disks in raid 5. Hmm. This one was a bit troubling. It was only about twice as fast as the SATA/RAID 1 I tested on our 860. I would have expected it to be much faster than that, with the stripe across 4 15k RPM drives. Are there any known issues with the PERC5i driver as well? Has anybody compared to any other OS? My apologies if it seems like I'm fixated, I'm just new to FreeBSD on Dells. I'm used to FreeBSD on several types of hardware, especially (now) old SGI 1100's where FreeBSD blew away anything I put up against it(well, anything defined as Redhat, Debian and SuSE). ;> -- Randy (schulra@earlham.edu) 765.983.1283 <*> Rain puts a hole in stone because of its constancy, not its force. - H. Joseph Gerber --------------030905000100030409010208 Content-Type: TEXT/PLAIN; NAME=benchmarks.txt Content-ID: Content-Description: Content-Disposition: INLINE; FILENAME=benchmarks.txt Perc5i 4 * SAS 15K RPM 146 Gig disks in raid 5. CPU: Intel(R) Xeon(R) CPU 5160 @ 3.00GHz (2995.54-MHz K8-class CPU) Frequency = 10 secs Scratch dir = [/data] Spawning 3 writers... Spawning 1 rewriters... Spawning 5 commenters... Spawning 100 readers... Benchmarking for 30 iterations. The test will run during 5 minutes. Nb blogs R articles W articles R pictures W pictures R comments W comments 15 13958 890 11193 986 10196 3139 29 22525 902 16806 636 14457 2668 41 22776 730 17004 718 14427 2473 50 20498 528 14791 664 10622 2184 57 16051 494 11838 514 8818 1630 67 25084 655 30781 748 15000 2578 74 15590 447 11598 345 8768 1270 78 10908 327 8137 211 6505 915 92 26503 1106 18745 613 15369 2827 96 16371 339 10909 373 7937 1103 103 15408 402 10736 287 9274 1140 114 22229 666 15143 566 10028 2079 122 17964 471 12721 466 9633 1551 127 17699 376 11947 289 9238 1061 134 12314 325 9021 431 6934 1293 139 20510 529 14899 471 12919 1684 143 19382 455 13374 253 10189 1080 147 10657 272 7126 214 5208 821 159 22676 657 14956 586 13469 2007 163 17458 147 11940 283 10322 759 168 14347 221 9881 274 7397 777 177 20031 616 13832 541 12240 1796 178 13840 135 9766 168 9133 488 184 19061 320 12801 246 10504 849 189 18462 501 12587 469 10500 1753 192 16019 228 10975 258 10133 769 194 14234 128 10232 220 6950 520 197 11275 82 8126 180 6392 501 204 19404 482 14123 424 12712 1495 204 11758 105 7814 160 8072 444 Final score for writes: 204 Final score for reads : 10414 Perc5e 15 * SATA 7.2K RPM 500 Gig disks in raid 50 (3 raid 5 volumes 5 disks each) CPU: Intel(R) Xeon(R) CPU 5110 @ 1.60GHz (1597.64-MHz K8-class CPU) Frequency = 10 secs Scratch dir = [/compere/test] Spawning 3 writers... Spawning 1 rewriters... Spawning 5 commenters... Spawning 100 readers... Benchmarking for 30 iterations. The test will run during 5 minutes. Nb blogs R articles W articles R pictures W pictures R comments W comments 1 4314 189 1999 109 341 439 5 14061 329 10016 250 9840 873 14 28924 416 23146 464 11733 1317 23 31039 470 23999 531 14171 1444 31 34713 523 27226 527 17347 1470 41 35581 416 27988 626 21633 1390 51 38418 491 29663 517 21576 1307 56 41466 506 32501 488 24634 1370 61 53966 561 41339 386 31210 1237 68 55243 504 42738 423 30776 1279 76 50607 508 38520 420 26856 1278 83 46003 471 35468 481 26124 1222 88 51993 483 40542 369 31031 1143 96 54179 505 42080 362 31857 1118 102 54421 349 40906 344 29561 940 106 60051 501 45166 354 33161 1113 113 62562 432 46139 347 37840 1041 120 67518 403 49800 463 40662 1148 128 62442 459 46504 357 35074 1062 134 62545 350 47605 391 38809 967 139 86653 402 64632 342 51244 1009 145 69267 377 52526 429 40214 1065 152 59945 390 45232 338 35558 929 158 81802 386 61518 412 48679 1078 164 71721 431 53173 317 40869 992 170 81977 432 61320 264 46273 886 174 77979 405 58017 323 45199 962 183 73185 367 54622 343 38939 956 189 74043 402 54737 299 42417 942 195 74745 325 57095 408 41014 962 Final score for writes: 195 Final score for reads : 35412 --------------030905000100030409010208-- From owner-freebsd-performance@FreeBSD.ORG Tue May 15 16:31:56 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3D10A16A404 for ; Tue, 15 May 2007 16:31:56 +0000 (UTC) (envelope-from kkobb@skylinecorp.com) Received: from mail.skylinecorp.com (mail.skylinecorp.com [216.117.21.137]) by mx1.freebsd.org (Postfix) with ESMTP id F237113C459 for ; Tue, 15 May 2007 16:31:55 +0000 (UTC) (envelope-from kkobb@skylinecorp.com) Received: from kkobb (in-elk-216-117-2-1.skylinecorp.com [216.117.2.1]) by mail.skylinecorp.com (8.12.9/8.12.9) with ESMTP id l4FGVd6A020083; Tue, 15 May 2007 12:31:39 -0400 From: "Kevin Kobb" To: "'Tom Judge'" , "'Randy Schultz'" References: <464474F0.3040306@tomjudge.com> <7579f7fb0705110735h1a65ef7atcab00bdbc25224d6@mail.gmail.com> <4649A71C.4030302@skylinecorp.com> <4649CC6B.1050203@tomjudge.com> Date: Tue, 15 May 2007 12:30:57 -0400 Message-ID: <000301c7970e$6dbdcc80$9501a8c0@skylinecorp.net> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_NextPart_000_0004_01C796EC.E6AE9D80" X-Mailer: Microsoft Office Outlook 11 In-Reply-To: <4649CC6B.1050203@tomjudge.com> X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3028 Thread-Index: AceXAmTJZGkByMrNQOiL4wJ820vcPAACouvA X-SkylineCorp-MailScanner-Information: Please contact the ISP for more information X-SkylineCorp-MailScanner: Found to be clean X-SkylineCorp-MailScanner-SpamCheck: not spam, SpamAssassin (not cached, score=-1.796, required 4.25, autolearn=not spam, AWL -0.60, BAYES_00 -1.20) X-MailScanner-From: kkobb@skylinecorp.com X-Mailman-Approved-At: Tue, 15 May 2007 17:07:26 +0000 Cc: freebsd-performance@freebsd.org Subject: RE: PERC5i throughput [was: possible issues with Dell/Perc5 raid] X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 May 2007 16:31:56 -0000 This is a multi-part message in MIME format. ------=_NextPart_000_0004_01C796EC.E6AE9D80 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Tom Judge wrote: > Randy Schultz wrote: >> On Tue, 15 May 2007, Kevin Kobb spaketh thusly: >> >> -}These reports on poor performance using mpt seem to be on SATA >> -}drives. Has anybody been seeing this using SAS drives? -} >> -}We are testing Dell PE840s with hot swap SAS drives, and seem to >> get >> decent -}performance, though I haven't run any benchmarks yet. Any >> opinions if the -}PERC5i/mfi is a better choice than the SAS5iR/mpt >> combination? >> >> Would it be possible for you to install blogbench and give it a quick >> run while your system is idle? We will be ordering a new box in 1-2 >> months that we need high disk I/O capabilities. We've been looking >> at >> the PERC5i set up RAID 5 with 4-6 SAS drives. I'ld be interested in >> what you're seeing for throughput on the PERC5i. >> > > I have attached some blogbench tests from the following configs: > > Perc5i 4 * SAS 15K RPM 146 Gig disks in raid 5. > > > Perc5e 15 * SATA 7.2K RPM 500 Gig disks in raid 50 (3 raid 5 volumes > 5 disks each) > > > Other configurations that I may be able to test are 2 SATA in raid 1 > and 2 SAS in raid 1 (both on perc5i). Ok, I installed blogbench (which I have not used before) and ran a couple of quick tests. I have a SAS 5iR controller, (not a PERC5 though) on a PE840, with 2 GB RAM, and 2 146 GB 10K RPM hot plug drives in a RAID 1. I ran the test and noticed: "mpt0: QUEUE FULL EVENT: Bus 0x00 Target 0x20 Depth 121" messages in my logs. Then I ran, camcontrol tags 0:0:0 -N 119 -v, and ran it again. I didn't get any messages this time and got the results indicated in test 2. ------=_NextPart_000_0004_01C796EC.E6AE9D80 Content-Type: text/plain; name="test1.txt" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="test1.txt" Frequency =3D 10 secs=0A= Scratch dir =3D [/var/blogbench/]=0A= Spawning 3 writers...=0A= Spawning 1 rewriters...=0A= Spawning 5 commenters...=0A= Spawning 100 readers...=0A= Benchmarking for 30 iterations.=0A= The test will run during 5 minutes.=0A= =0A= Nb blogs R articles W articles R pictures W pictures R = comments W comments=0A= 17 55260 973 39370 880 = 32248 3650=0A= 28 134486 532 98010 549 = 86733 3106=0A= 30 134573 257 101060 146 = 107526 4947=0A= 34 98176 280 68817 192 = 76737 2610=0A= 42 146159 569 101205 526 = 108409 4214=0A= 44 113053 103 77234 95 = 80420 4117=0A= 47 123475 58 82482 60 = 101734 4138=0A= 53 116878 333 79349 328 = 100729 4242=0A= 62 129703 491 88760 511 = 102082 2740=0A= 75 56569 1017 39268 646 = 43163 3330=0A= 78 78428 125 53036 107 = 60883 5010=0A= 83 41037 452 28260 301 = 31707 4043=0A= 88 41144 492 28448 407 = 29760 4011=0A= 93 104037 347 71189 473 = 81383 3972=0A= 100 138044 287 95126 316 = 106019 5474=0A= 103 136079 187 94750 203 = 107826 5754=0A= 106 80718 230 55433 146 = 67429 3845=0A= 111 29693 326 21125 284 = 23386 3978=0A= 120 47330 400 31754 381 = 36253 4219=0A= 123 104226 342 70924 341 = 85267 4291=0A= 128 120925 808 87777 533 = 97476 2673=0A= 129 158526 163 114288 116 = 135061 3884=0A= 134 33258 389 23219 328 = 31020 4083=0A= 140 56495 362 39954 323 = 49051 3525=0A= 142 130233 223 93422 220 = 110732 2766=0A= 150 63513 603 46071 326 = 57740 4760=0A= 155 19686 305 13779 288 = 14446 3257=0A= 160 56576 511 40326 390 = 46761 4171=0A= 162 15760 188 11627 193 = 15065 3569=0A= 166 33365 176 23615 251 = 27148 4420=0A= =0A= Final score for writes: 166=0A= Final score for reads : 53465=0A= ------=_NextPart_000_0004_01C796EC.E6AE9D80 Content-Type: text/plain; name="test2.txt" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="test2.txt" Frequency =3D 10 secs=0A= Scratch dir =3D [/var/blogbench/]=0A= Spawning 3 writers...=0A= Spawning 1 rewriters...=0A= Spawning 5 commenters...=0A= Spawning 100 readers...=0A= Benchmarking for 30 iterations.=0A= The test will run during 5 minutes.=0A= =0A= Nb blogs R articles W articles R pictures W pictures R = comments W comments=0A= 39 104065 1902 92397 1908 = 81915 1920=0A= 52 89671 812 75477 796 = 64955 2786=0A= 63 40679 526 33195 646 = 31461 2682=0A= 78 51740 1173 42408 778 = 35679 3384=0A= 88 113424 624 91326 546 = 78975 2915=0A= 91 46138 275 37605 254 = 33564 5530=0A= 97 36149 370 28729 231 = 26711 2891=0A= 101 85994 274 68781 101 = 63879 4942=0A= 116 48741 646 39656 758 = 33324 3320=0A= 124 26635 436 20939 492 = 19201 3246=0A= 129 62322 387 50055 356 = 42611 3542=0A= 133 34041 176 27625 103 = 23844 4214=0A= 136 70655 187 57334 92 = 47747 3775=0A= 140 91423 268 73591 322 = 64109 3731=0A= 145 40426 310 32623 369 = 30511 4642=0A= 149 51952 350 43269 207 = 36259 3263=0A= 155 45495 313 37330 294 = 32495 4303=0A= 161 71781 381 59076 337 = 53068 5047=0A= 166 82942 245 67613 221 = 58052 4236=0A= 167 107542 138 87544 114 = 71804 2991=0A= 176 114451 480 93543 204 = 80621 5079=0A= 178 149129 107 122331 141 = 102999 2691=0A= 180 125073 168 102207 106 = 87452 5430=0A= 181 152969 77 123829 74 = 106024 5851=0A= 187 161513 395 130680 131 = 112562 4197=0A= 189 175944 186 138771 32 = 122199 5258=0A= 192 154242 196 121718 214 = 102413 2791=0A= 197 215053 416 168652 274 = 144365 2056=0A= 201 199915 342 155300 317 = 131471 2108=0A= 205 189504 141 146564 64 = 124165 1588=0A= =0A= Final score for writes: 205=0A= Final score for reads : 45783=0A= ------=_NextPart_000_0004_01C796EC.E6AE9D80-- From owner-freebsd-performance@FreeBSD.ORG Tue May 15 21:02:17 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id F065916A400 for ; Tue, 15 May 2007 21:02:16 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from mh2.centtech.com (moat3.centtech.com [64.129.166.50]) by mx1.freebsd.org (Postfix) with ESMTP id BD44813C465 for ; Tue, 15 May 2007 21:02:16 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from neutrino.centtech.com (neutrino.centtech.com [10.177.171.220]) by mh2.centtech.com (8.13.8/8.13.8) with ESMTP id l4FL2E0t029894 for ; Tue, 15 May 2007 16:02:14 -0500 (CDT) (envelope-from anderson@freebsd.org) Message-ID: <464A1FD6.8070104@freebsd.org> Date: Tue, 15 May 2007 16:02:14 -0500 From: Eric Anderson User-Agent: Thunderbird 2.0.0.0 (X11/20070420) MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <464474F0.3040306@tomjudge.com> <7579f7fb0705110735h1a65ef7atcab00bdbc25224d6@mail.gmail.com> <4649A71C.4030302@skylinecorp.com> <4649CC6B.1050203@tomjudge.com> <000301c7970e$6dbdcc80$9501a8c0@skylinecorp.net> In-Reply-To: <000301c7970e$6dbdcc80$9501a8c0@skylinecorp.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.88.4/3247/Tue May 15 06:31:00 2007 on mh2.centtech.com X-Virus-Status: Clean X-Spam-Status: No, score=-2.5 required=8.0 tests=AWL,BAYES_00 autolearn=ham version=3.1.6 X-Spam-Checker-Version: SpamAssassin 3.1.6 (2006-10-03) on mh2.centtech.com Subject: Re: PERC5i throughput [was: possible issues with Dell/Perc5 raid] X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 May 2007 21:02:17 -0000 On 05/15/07 11:30, Kevin Kobb wrote: > Tom Judge wrote: >> Randy Schultz wrote: >>> On Tue, 15 May 2007, Kevin Kobb spaketh thusly: >>> >>> -}These reports on poor performance using mpt seem to be on SATA >>> -}drives. Has anybody been seeing this using SAS drives? -} >>> -}We are testing Dell PE840s with hot swap SAS drives, and seem to >>> get >>> decent -}performance, though I haven't run any benchmarks yet. Any >>> opinions if the -}PERC5i/mfi is a better choice than the SAS5iR/mpt >>> combination? >>> >>> Would it be possible for you to install blogbench and give it a quick >>> run while your system is idle? We will be ordering a new box in 1-2 >>> months that we need high disk I/O capabilities. We've been looking >>> at >>> the PERC5i set up RAID 5 with 4-6 SAS drives. I'ld be interested in >>> what you're seeing for throughput on the PERC5i. >>> >> I have attached some blogbench tests from the following configs: >> >> Perc5i 4 * SAS 15K RPM 146 Gig disks in raid 5. >> >> >> Perc5e 15 * SATA 7.2K RPM 500 Gig disks in raid 50 (3 raid 5 volumes >> 5 disks each) >> >> >> Other configurations that I may be able to test are 2 SATA in raid 1 >> and 2 SAS in raid 1 (both on perc5i). > > Ok, I installed blogbench (which I have not used before) and ran a > couple of quick tests. > I have a SAS 5iR controller, (not a PERC5 though) on a PE840, with 2 GB > RAM, and 2 146 GB 10K RPM hot plug drives in a RAID 1. > > I ran the test and noticed: > "mpt0: QUEUE FULL EVENT: Bus 0x00 Target 0x20 Depth 121" messages in my > logs. > Then I ran, camcontrol tags 0:0:0 -N 119 -v, and ran it again. > > I didn't get any messages this time and got the results indicated in > test 2. Just for an additional data point, here's what a 2Gb Fiber channel connected array with 16 750GB SATA disks looks like: Frequency = 10 secs Scratch dir = [/vol3/test/] Spawning 3 writers... Spawning 1 rewriters... Spawning 5 commenters... Spawning 100 readers... Benchmarking for 30 iterations. The test will run during 5 minutes. Nb blogs R articles W articles R pictures W pictures R comments W comments 48 85428 2630 60484 2865 46462 8899 78 77246 1670 56118 1547 48798 5714 101 68730 1634 47639 1103 41230 5108 127 64230 1663 43522 1422 35531 4517 150 64072 1326 42968 1330 35485 4165 168 39332 1163 26511 993 20236 2697 194 53474 1527 35969 1137 32142 4251 215 55310 1362 37140 1274 30882 4401 232 49766 1203 32995 1046 29133 3979 251 38767 1061 27122 909 23652 3130 272 40820 1344 29009 920 23557 3728 285 23580 771 15778 746 14036 2406 300 26545 979 18758 853 14721 3001 323 34491 1319 23422 1222 20155 4197 333 15418 732 10068 738 7867 2324 361 21929 1340 15030 1505 12914 5008 373 13122 524 9329 743 7842 2145 396 24737 871 17423 1177 14929 3182 404 8858 323 6651 345 5864 1165 433 25465 1450 18236 1472 15915 3982 438 9460 379 6477 259 4627 1284 454 15580 869 10450 1087 9702 3297 470 11886 874 8590 603 7578 2541 487 16931 1088 11651 803 9846 3231 496 11122 559 7517 580 5341 2065 517 13609 883 9288 1150 8291 3611 545 8043 1610 5581 1022 4973 3771 557 5049 686 3784 542 2195 2383 578 9534 1451 6703 1334 6541 5202 593 4383 834 3230 1011 3432 1381 Final score for writes: 593 Final score for reads : 18671 My system also is pretty busy doing an rsync to/from some other arrays, so the numbers are lower that what I'd get on an idle system. I'll try this again on a faster system if I get the chance. Eric From owner-freebsd-performance@FreeBSD.ORG Wed May 16 00:36:52 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1ED0216A410 for ; Wed, 16 May 2007 00:36:52 +0000 (UTC) (envelope-from lydianconcepts@gmail.com) Received: from nz-out-0506.google.com (nz-out-0506.google.com [64.233.162.230]) by mx1.freebsd.org (Postfix) with ESMTP id BEAED13C455 for ; Wed, 16 May 2007 00:36:51 +0000 (UTC) (envelope-from lydianconcepts@gmail.com) Received: by nz-out-0506.google.com with SMTP id s1so400411nze for ; Tue, 15 May 2007 17:36:51 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=mXrA8SVoYoUbG31m1KegP2z2E2aodgCHdSAHR9t6iCUYYg23dqMYXvtl1FghpMLjKeh82vF/oKPyvb7RXha2kMKUcCB4rjG4SFxevg7E2/Zj7GszYhtEYFx7mLrqHYBl7/k4PTfMXRv5yCDxbwXw1qGPsOUiZ34HUDT/eQiqVkE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=Q1r0eH3hGw2QThcVYs2M5PWQYLp5SNrdIJpeYg1MTPednAF7nLFb7c9tzGc6i4zk4CiA1pJIvacPIEa2XffndJC6Uzbz2QW1rXwD5e33Rl9f2UrbawuXoWryqL/YJXNPOBO5WEEfZ7zXAxy5OOsWODa162u3tNUcYkXU6feyZWs= Received: by 10.115.22.1 with SMTP id z1mr1865831wai.1179275810875; Tue, 15 May 2007 17:36:50 -0700 (PDT) Received: by 10.114.24.2 with HTTP; Tue, 15 May 2007 17:36:45 -0700 (PDT) Message-ID: <7579f7fb0705151736hf1118cck71898d572f817475@mail.gmail.com> Date: Tue, 15 May 2007 17:36:45 -0700 From: "Matthew Jacob" To: "Randy Schultz" In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <464474F0.3040306@tomjudge.com> <7579f7fb0705110735h1a65ef7atcab00bdbc25224d6@mail.gmail.com> X-Mailman-Approved-At: Wed, 16 May 2007 00:41:32 +0000 Cc: freebsd-performance@freebsd.org Subject: Re: Dell SAS 5i/mpt driver[was: possible issues with Dell/Perc5 raid] X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 May 2007 00:36:52 -0000 filesystem? ext3? I want one of these failing machines *in my lab*. On 5/14/07, Randy Schultz wrote: > On Fri, 11 May 2007, Matthew Jacob spaketh thusly: > > -}Time frame to resolution involves getting a machine into my lab that > -}evidences the poor performance and the time to sort out what the > -}problem is and fix it. Remote access doesn't work for me on this one > -}(contact me offline as to why). > > Hey Matthew, > > FWIW, I installed fedora core 5 on the box. The difference was significant. > Writes, though bursty, increased significantly. Reads increased by 1-2 orders > of magnitude. I've attached the output of blogbench for fc5 FWIW. > > Being new to all this, what would be the next step? Should I file a bug > report? Or perhaps I should be asking what your timeline is? All I have to > offer for help is remote access, which in this case doesn't work for you, or > perhaps testing as my C skills are fairly pathetic. ;> > > -- > Randy (schulra@earlham.edu) 765.983.1283 <*> > > Rain puts a hole in stone because of its constancy, not its force. > - H. Joseph Gerber > > From owner-freebsd-performance@FreeBSD.ORG Wed May 16 12:13:52 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7738616A400 for ; Wed, 16 May 2007 12:13:52 +0000 (UTC) (envelope-from schulra@earlham.edu) Received: from sipala.earlham.edu (sipala.earlham.edu [159.28.1.75]) by mx1.freebsd.org (Postfix) with ESMTP id 48CCC13C455 for ; Wed, 16 May 2007 12:13:52 +0000 (UTC) (envelope-from schulra@earlham.edu) Received: from tdream.lly.earlham.edu (tdream.lly.earlham.edu [159.28.7.241]) by sipala.earlham.edu (8.13.6/8.13.6) with ESMTP id l4GCDpcj001238; Wed, 16 May 2007 08:13:51 -0400 (EDT) Date: Wed, 16 May 2007 08:15:13 -0400 (EDT) From: Randy Schultz X-X-Sender: schulra@tdream.lly.earlham.edu To: Matthew Jacob In-Reply-To: <7579f7fb0705151736hf1118cck71898d572f817475@mail.gmail.com> Message-ID: References: <464474F0.3040306@tomjudge.com> <7579f7fb0705110735h1a65ef7atcab00bdbc25224d6@mail.gmail.com> <7579f7fb0705151736hf1118cck71898d572f817475@mail.gmail.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: freebsd-performance@freebsd.org Subject: Re: Dell SAS 5i/mpt driver[was: possible issues with Dell/Perc5 raid] X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 May 2007 12:13:52 -0000 On Tue, 15 May 2007, Matthew Jacob spaketh thusly: -}filesystem? ext3? Correct. -} -}I want one of these failing machines *in my lab*. -} -}On 5/14/07, Randy Schultz wrote: -}> On Fri, 11 May 2007, Matthew Jacob spaketh thusly: -}> -}> -}Time frame to resolution involves getting a machine into my lab that -}> -}evidences the poor performance and the time to sort out what the -}> -}problem is and fix it. Remote access doesn't work for me on this one -}> -}(contact me offline as to why). -}> -}> Hey Matthew, -}> -}> FWIW, I installed fedora core 5 on the box. The difference was significant. -}> Writes, though bursty, increased significantly. Reads increased by 1-2 -}> orders -}> of magnitude. I've attached the output of blogbench for fc5 FWIW. -}> -}> Being new to all this, what would be the next step? Should I file a bug -}> report? Or perhaps I should be asking what your timeline is? All I have to -}> offer for help is remote access, which in this case doesn't work for you, or -}> perhaps testing as my C skills are fairly pathetic. ;> -- Randy (schulra@earlham.edu) 765.983.1283 <*> Rain puts a hole in stone because of its constancy, not its force. - H. Joseph Gerber From owner-freebsd-performance@FreeBSD.ORG Wed May 16 16:20:29 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7405916A400 for ; Wed, 16 May 2007 16:20:29 +0000 (UTC) (envelope-from aedwards@sandvine.com) Received: from gw.sandvine.com (gw.sandvine.com [199.243.201.138]) by mx1.freebsd.org (Postfix) with ESMTP id D4D7213C45D for ; Wed, 16 May 2007 16:20:28 +0000 (UTC) (envelope-from aedwards@sandvine.com) Received: from exchange-2.sandvine.com ([192.168.16.12]) by gw.sandvine.com with Microsoft SMTPSVC(6.0.3790.1830); Wed, 16 May 2007 12:08:25 -0400 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable Date: Wed, 16 May 2007 12:08:24 -0400 Message-ID: <5230D3C40B842D4F9FB3CD368021BEF72F08F1@exchange-2.sandvine.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Ufs dead-locks on freebsd 6.2 Thread-Index: AceX1G4ru5sFcTfBQyOVPtiqpV6KBw== From: "Andrew Edwards" To: X-OriginalArrivalTime: 16 May 2007 16:08:25.0272 (UTC) FILETIME=[6EE52380:01C797D4] Subject: Ufs dead-locks on freebsd 6.2 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 May 2007 16:20:29 -0000 I have a system running a dual intel zeon 2.8Ghz with 4G of ram and using an intel raid controller model SRCU42X which uses the amr driver. I have had this server running 5.4 upgraded to 6.2 and was running fine for several months and then after a normal reboot I've started having all sorts of problems with what appears to be dead-locks in the filesystem. This server is my backup server and I rsync files from various servers onto this one fairly non-stop. If I stop the rsync's the system appears to be stable although I did have a kernel core just last night. When I have been able to observe the problem I ususally see one filesystem become inaccessible, perhaps var but I'm not sure, and then in a short period of time the whole system is inaccessible. Usually if I startup just one of the rsync's within a couple of hours the system will be un-usable. I did find this thread which seems to describe similar issues but this is a different driver. http://lists.freebsd.org/pipermail/freebsd-questions/2006-August/127835. html Currently I'm running with debug.mpsafevfs=3D0, debug.mpsafenet=3D1and debug.mpsafevm=3D0 but this doesn't seem to help. On perahps a related issue I have two other nearly identical systems which were going to be upgrading to 6.2 as on 5.4 I am experiencing deadlocks and when I hit ctrl-t I see the system is either stuck in ufs or zoneinfo and I have not found very much information about zoneinfo. Does anyone have any suggestions on what I can look for or other tuning options or had similar experiences? From owner-freebsd-performance@FreeBSD.ORG Wed May 16 16:33:07 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id F40EF16A405 for ; Wed, 16 May 2007 16:33:06 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id E3BAE13C483 for ; Wed, 16 May 2007 16:33:06 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 985EC1A3C19; Wed, 16 May 2007 09:33:58 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id A27EF52C45; Wed, 16 May 2007 12:33:05 -0400 (EDT) Date: Wed, 16 May 2007 12:33:05 -0400 From: Kris Kennaway To: Andrew Edwards Message-ID: <20070516163305.GA73495@xor.obsecurity.org> References: <5230D3C40B842D4F9FB3CD368021BEF72F08F1@exchange-2.sandvine.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5230D3C40B842D4F9FB3CD368021BEF72F08F1@exchange-2.sandvine.com> User-Agent: Mutt/1.4.2.2i Cc: freebsd-performance@freebsd.org Subject: Re: Ufs dead-locks on freebsd 6.2 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 May 2007 16:33:07 -0000 On Wed, May 16, 2007 at 12:08:24PM -0400, Andrew Edwards wrote: > I have a system running a dual intel zeon 2.8Ghz with 4G of ram and > using an intel raid controller model SRCU42X which uses the amr driver. > I have had this server running 5.4 upgraded to 6.2 and was running fine > for several months and then after a normal reboot I've started having > all sorts of problems with what appears to be dead-locks in the > filesystem. This server is my backup server and I rsync files from > various servers onto this one fairly non-stop. If I stop the rsync's > the system appears to be stable although I did have a kernel core just > last night. > > When I have been able to observe the problem I ususally see one > filesystem become inaccessible, perhaps var but I'm not sure, and then > in a short period of time the whole system is inaccessible. Usually if > I startup just one of the rsync's within a couple of hours the system > will be un-usable. > > I did find this thread which seems to describe similar issues but this > is a different driver. > http://lists.freebsd.org/pipermail/freebsd-questions/2006-August/127835. > html Probably not relevant then. Deadlocks come in many varieties, all different. > Currently I'm running with debug.mpsafevfs=0, debug.mpsafenet=1and > debug.mpsafevm=0 but this doesn't seem to help. > > On perahps a related issue I have two other nearly identical systems > which were going to be upgrading to 6.2 as on 5.4 I am experiencing > deadlocks and when I hit ctrl-t I see the system is either stuck in ufs > or zoneinfo and I have not found very much information about zoneinfo. > > Does anyone have any suggestions on what I can look for or other tuning > options or had similar experiences? See the chapter on kernel debugging in the developers handbook for the information you need to provide before we can begin to debug your problem. Kris From owner-freebsd-performance@FreeBSD.ORG Wed May 16 16:32:09 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4176516A405; Wed, 16 May 2007 16:32:09 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from mh1.centtech.com (moat3.centtech.com [64.129.166.50]) by mx1.freebsd.org (Postfix) with ESMTP id 12C2713C46C; Wed, 16 May 2007 16:32:08 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from neutrino.centtech.com (neutrino.centtech.com [10.177.171.220]) by mh1.centtech.com (8.13.8/8.13.8) with ESMTP id l4GGW6VU042903; Wed, 16 May 2007 11:32:06 -0500 (CDT) (envelope-from anderson@freebsd.org) Message-ID: <464B3206.7000007@freebsd.org> Date: Wed, 16 May 2007 11:32:06 -0500 From: Eric Anderson User-Agent: Thunderbird 2.0.0.0 (X11/20070420) MIME-Version: 1.0 To: Andrew Edwards References: <5230D3C40B842D4F9FB3CD368021BEF72F08F1@exchange-2.sandvine.com> In-Reply-To: <5230D3C40B842D4F9FB3CD368021BEF72F08F1@exchange-2.sandvine.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.88.4/3260/Wed May 16 08:17:18 2007 on mh1.centtech.com X-Virus-Status: Clean X-Spam-Status: No, score=-2.6 required=8.0 tests=AWL,BAYES_00 autolearn=ham version=3.1.6 X-Spam-Checker-Version: SpamAssassin 3.1.6 (2006-10-03) on mh1.centtech.com X-Mailman-Approved-At: Wed, 16 May 2007 17:01:19 +0000 Cc: "freebsd-fs@freebsd.org" Subject: Re: Ufs dead-locks on freebsd 6.2 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 May 2007 16:32:09 -0000 On 05/16/07 11:08, Andrew Edwards wrote: > I have a system running a dual intel zeon 2.8Ghz with 4G of ram and > using an intel raid controller model SRCU42X which uses the amr driver. > I have had this server running 5.4 upgraded to 6.2 and was running fine > for several months and then after a normal reboot I've started having > all sorts of problems with what appears to be dead-locks in the > filesystem. This server is my backup server and I rsync files from > various servers onto this one fairly non-stop. If I stop the rsync's > the system appears to be stable although I did have a kernel core just > last night. > > When I have been able to observe the problem I ususally see one > filesystem become inaccessible, perhaps var but I'm not sure, and then > in a short period of time the whole system is inaccessible. Usually if > I startup just one of the rsync's within a couple of hours the system > will be un-usable. > > I did find this thread which seems to describe similar issues but this > is a different driver. > http://lists.freebsd.org/pipermail/freebsd-questions/2006-August/127835. > html > > Currently I'm running with debug.mpsafevfs=0, debug.mpsafenet=1and > debug.mpsafevm=0 but this doesn't seem to help. > > On perahps a related issue I have two other nearly identical systems > which were going to be upgrading to 6.2 as on 5.4 I am experiencing > deadlocks and when I hit ctrl-t I see the system is either stuck in ufs > or zoneinfo and I have not found very much information about zoneinfo. What's ctrl-t? > Does anyone have any suggestions on what I can look for or other tuning > options or had similar experiences? When you say deadlock, you mean no access at all, but pingable? Does it panic? Can you still run things like 'ps'? You should (if you haven't already) add debugging to your kernel, and make sure you are running 6-STABLE. A ps -auxl of a system would be great if you can get it. Otherwise, you'll need to get a crash dump. You said you have a kernel core, can you send the output of a full back trace to the list? Also - I've cc'ed freebsd-fs@, since it's more appropriate there. Eric From owner-freebsd-performance@FreeBSD.ORG Wed May 16 17:07:29 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 257D716A403 for ; Wed, 16 May 2007 17:07:29 +0000 (UTC) (envelope-from lydianconcepts@gmail.com) Received: from py-out-1112.google.com (py-out-1112.google.com [64.233.166.181]) by mx1.freebsd.org (Postfix) with ESMTP id D36F413C484 for ; Wed, 16 May 2007 17:07:28 +0000 (UTC) (envelope-from lydianconcepts@gmail.com) Received: by py-out-1112.google.com with SMTP id f31so550238pyh for ; Wed, 16 May 2007 10:07:28 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=cftdBQP/MPVZA6rdSVVXLlU4oJ0F0kyjpxvVMlxe9gVvCM28BGL1TbruhPs8WsXt8Qd70oFkOJcwFOwwDBFIBXmAS4Ec0jxrRpHKSrbCQAIFKEibhHiNoWqsrl7RDlcim1untU4R/FX7PGlD3/6gJxyManUzqzB1rFIgVgXVnK4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=m1Eg/RU8eFY6ic0NthsFikpf/5QXBd53aYYO7+8z2ifbcvUvPgd1M6cPa+FoU7zkl+byQk/XExd6LlibCwid8CdKPHO+ldE68WSjS5M/IqJd5WlMZkMINDut0PRzGZjiAmOZ6GnRnvopt13iqvicJZc0yTSi+/TP207b39pD5HM= Received: by 10.114.168.1 with SMTP id q1mr2337093wae.1179335247914; Wed, 16 May 2007 10:07:27 -0700 (PDT) Received: by 10.114.24.2 with HTTP; Wed, 16 May 2007 10:07:27 -0700 (PDT) Message-ID: <7579f7fb0705161007i75468c80l2baf2ea462ecafc5@mail.gmail.com> Date: Wed, 16 May 2007 10:07:27 -0700 From: "Matthew Jacob" To: "Randy Schultz" In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <464474F0.3040306@tomjudge.com> <7579f7fb0705110735h1a65ef7atcab00bdbc25224d6@mail.gmail.com> <7579f7fb0705151736hf1118cck71898d572f817475@mail.gmail.com> X-Mailman-Approved-At: Wed, 16 May 2007 17:12:13 +0000 Cc: freebsd-performance@freebsd.org Subject: Re: Dell SAS 5i/mpt driver[was: possible issues with Dell/Perc5 raid] X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 May 2007 17:07:29 -0000 It wouldn't surprise me in the slightest that ext3 is better albeit bursty for writes. That's the Linux aggressive write-behind and ext2/ext3 for you in a nutshell. It's also quite likely true that in many cases that reads and I/O scheduling is a lot better in Linux. If you have time to run some benchmarks, using something like dt on a FreeBSD raw partition with lots of small sequential writes and using the (now deprecated) 'raw' bound devices in fc5 might get more of an apple-apple. On 5/16/07, Randy Schultz wrote: > On Tue, 15 May 2007, Matthew Jacob spaketh thusly: > > -}filesystem? ext3? > > Correct. > > -} > -}I want one of these failing machines *in my lab*. > -} > -}On 5/14/07, Randy Schultz wrote: > -}> On Fri, 11 May 2007, Matthew Jacob spaketh thusly: > -}> > -}> -}Time frame to resolution involves getting a machine into my lab that > -}> -}evidences the poor performance and the time to sort out what the > -}> -}problem is and fix it. Remote access doesn't work for me on this one > -}> -}(contact me offline as to why). > -}> > -}> Hey Matthew, > -}> > -}> FWIW, I installed fedora core 5 on the box. The difference was significant. > -}> Writes, though bursty, increased significantly. Reads increased by 1-2 > -}> orders > -}> of magnitude. I've attached the output of blogbench for fc5 FWIW. > -}> > -}> Being new to all this, what would be the next step? Should I file a bug > -}> report? Or perhaps I should be asking what your timeline is? All I have to > -}> offer for help is remote access, which in this case doesn't work for you, or > -}> perhaps testing as my C skills are fairly pathetic. ;> > > -- > Randy (schulra@earlham.edu) 765.983.1283 <*> > > Rain puts a hole in stone because of its constancy, not its force. > - H. Joseph Gerber > > From owner-freebsd-performance@FreeBSD.ORG Wed May 16 17:17:34 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8DAB616A404 for ; Wed, 16 May 2007 17:17:34 +0000 (UTC) (envelope-from aedwards@sandvine.com) Received: from gw.sandvine.com (gw.sandvine.com [199.243.201.138]) by mx1.freebsd.org (Postfix) with ESMTP id 1BB0913C45A for ; Wed, 16 May 2007 17:17:32 +0000 (UTC) (envelope-from aedwards@sandvine.com) Received: from exchange-2.sandvine.com ([192.168.16.12]) by gw.sandvine.com with Microsoft SMTPSVC(6.0.3790.1830); Wed, 16 May 2007 13:16:29 -0400 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable Date: Wed, 16 May 2007 13:16:27 -0400 Message-ID: <5230D3C40B842D4F9FB3CD368021BEF72F08F3@exchange-2.sandvine.com> In-Reply-To: <20070516163305.GA73495@xor.obsecurity.org> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Ufs dead-locks on freebsd 6.2 Thread-Index: AceX1+P2HSPmcMyKSLK15m7jv2eBpAAAVXqQ From: "Andrew Edwards" To: X-OriginalArrivalTime: 16 May 2007 17:16:29.0850 (UTC) FILETIME=[F17E47A0:01C797DD] Cc: freebsd-fs@freebsd.org Subject: RE: Ufs dead-locks on freebsd 6.2 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 May 2007 17:17:34 -0000 Here's the backtrace from the last crash along with the output from show alllocks when the system was deadlocked. I have been running 6.2-release and compliled with makeoptions debug=3D-g, invariants, invariant_support and witness. I will update to 6-STABLE add diagnositc, debug_locks and debug_vfs_locks as per the handbook recommendation and retry. Yes, when the system was un-usable I was still able to ping it. I have the serial console setup as the default console so I can remotely access the box and break into the debugger etc. (kgdb) bt #0 doadump () at pcpu.h:165 #1 0xc059b480 in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0xc059b795 in panic (fmt=3D0xc0787b04 "Most recently used by %s\n") at /usr/src/sys/kern/kern_shutdown.c:565 #3 0xc06c4a4d in mtrash_ctor (mem=3D0xce74fa00, size=3D0, arg=3D0x0, flags=3D258) at /usr/src/sys/vm/uma_dbg.c:137 #4 0xc06c2c07 in uma_zalloc_arg (zone=3D0xc10615a0, udata=3D0x0, = flags=3D258) at /usr/src/sys/vm/uma_core.c:1850 #5 0xc0591416 in malloc (size=3D272, mtp=3D0xc07c32c0, flags=3D258) at uma.h:275 #6 0xc05edfab in __mnt_vnode_first (mvp=3D0xf3741c48, mp=3D0xcaa14cf8) at /usr/src/sys/kern/vfs_mount.c:1813 #7 0xc05f2467 in vfs_msync (mp=3D0xcaa14cf8, flags=3D2) at /usr/src/sys/kern/vfs_subr.c:2874 #8 0xc05f2bbd in sync_fsync (ap=3D0x0) at /usr/src/sys/kern/vfs_subr.c:3119 #9 0xc072f4ee in VOP_FSYNC_APV (vop=3D0x0, a=3D0xf3741cbc) at vnode_if.c:1020 #10 0xc05f097c in sync_vnode (bo=3D0xca854e90, td=3D0xca435000) at vnode_if.h:537 #11 0xc05f0bf1 in sched_sync () at /usr/src/sys/kern/vfs_subr.c:1698 #12 0xc0587248 in fork_exit (callout=3D0xc05f0a04 , = arg=3D0x0, frame=3D0xf3741d38) at /usr/src/sys/kern/kern_fork.c:821 #13 0xc070712c in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:208 db> show alllocks Process 36596 (sshd) thread 0xd1238c00 (102406) exclusive sleep mutex vm object (standard object) r =3D 0 (0xce2c87bc) locked @ /usr/src/sys/vm/vm_object.c:446 exclusive sx user map r =3D 0 (0xd128060c) locked @ /usr/src/sys/vm/vm_map.c:307 Process 887 (sshd) thread 0xca7d2000 (100056) exclusive sleep mutex vm object (standard object) r =3D 0 (0xcb713ad4) locked @ /usr/src/sys/vm/vm_fault.c:297 exclusive sx user map r =3D 0 (0xcaae4734) locked @ /usr/src/sys/vm/vm_map.c:3074 db> show lockedvnods Locked vnodes 0xcaa78660: tag ufs, type VREG usecount 2, writecount 1, refcount 3 mountedhere 0 flags () v_object 0xc1046738 ref 0 pages 1596 lock type ufs: EXCL (count 1) by thread 0xca689c00 (pid 536) with 1 pending ino 494620, on dev amrd0s1d 0xcaa86110: tag ufs, type VREG usecount 1, writecount 1, refcount 3 mountedhere 0 flags () v_object 0xca85f738 ref 0 pages 44 lock type ufs: EXCL (count 1) by thread 0xca7d2780 (pid 715) ino 494633, on dev amrd0s1d 0xcabe4110: tag ufs, type VDIR usecount 12, writecount 0, refcount 14 mountedhere 0 flags () v_object 0xcab28840 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xca9bbd80 (pid 14253) with 3 pending ino 423947, on dev amrd0s1d 0xcb437990: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcb3c3dec ref 0 pages 4100 lock type ufs: EXCL (count 1) by thread 0xcaffac00 (pid 20868) ino 282640, on dev amrd0s1d 0xcb99e550: tag ufs, type VDIR usecount 2, writecount 0, refcount 4 mountedhere 0 flags () v_object 0xcef979cc ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xca9bb600 (pid 881) with 1 pending ino 423987, on dev amrd0s1d 0xcfc97dd0: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcd4975ac ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcb748900 (pid 2518) ino 424275, on dev amrd0s1d 0xccad9aa0: tag ufs, type VREG usecount 1, writecount 1, refcount 3 mountedhere 0 flags () v_object 0xcf0c539c ref 0 pages 5 lock type ufs: EXCL (count 1) by thread 0xca7d1c00 (pid 600) ino 188446, on dev amrd0s1d 0xccb0f110: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcb4609cc ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcafbf480 (pid 11054) ino 424100, on dev amrd0s1d 0xcc501bb0: tag ufs, type VREG usecount 1, writecount 1, refcount 3 mountedhere 0 flags () v_object 0xcc7d19cc ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcb76d600 (pid 13743) with 1 pending ino 424279, on dev amrd0s1d 0xcf96b220: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcf135c60 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcafbf900 (pid 29458) ino 424374, on dev amrd0s1d 0xcc5bbbb0: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcdd5b318 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcaad9900 (pid 50782) ino 424276, on dev amrd0s1d 0xcec1d000: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcd3d7108 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcb76dc00 (pid 59514) ino 424500, on dev amrd0s1d 0xcebe5110: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xccee95ac ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcb650780 (pid 59975) ino 424509, on dev amrd0s1d 0xce0c1880: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xca8b64a4 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcb768a80 (pid 69466) ino 424555, on dev amrd0s1d 0xcf652110: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcf4a318c ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcaff8600 (pid 75577) ino 424579, on dev amrd0s1d 0xce282550: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcf261318 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xd0235a80 (pid 81734) ino 424927, on dev amrd0s1d 0xcc1d4dd0: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcd4a6630 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcaccb900 (pid 81772) ino 424928, on dev amrd0s1d 0xcb820bb0: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcb251ad4 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcadaf480 (pid 84037) ino 424935, on dev amrd0s1d 0xced5aaa0: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcb784210 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xd0236000 (pid 202) ino 425039, on dev amrd0s1d 0xcbe45220: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xce55de70 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcaad9a80 (pid 230) ino 425043, on dev amrd0s1d 0xcc098220: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xce4a9dec ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcafbfd80 (pid 9902) ino 425093, on dev amrd0s1d 0xcd585110: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcf8c1e70 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcaccba80 (pid 24017) ino 425144, on dev amrd0s1d 0xceeac000: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcb1225ac ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcaff5a80 (pid 24775) ino 425149, on dev amrd0s1d 0xcc549aa0: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcab2d318 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcaff8a80 (pid 42358) ino 425227, on dev amrd0s1d 0xcc6f7000: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcfd4139c ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcafbf600 (pid 43117) ino 425230, on dev amrd0s1d 0xccc44bb0: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcfd18d68 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcadafd80 (pid 42859) ino 425234, on dev amrd0s1d 0xcc7a7220: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcfedf420 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcb76c600 (pid 48968) ino 425264, on dev amrd0s1d 0xcc693aa0: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcf92f738 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcb655300 (pid 55381) ino 425286, on dev amrd0s1d 0xcbabf220: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcfe297bc ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xd0234a80 (pid 63802) ino 425322, on dev amrd0s1d 0xcd760220: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcb11ac60 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcadb0480 (pid 69938) ino 425348, on dev amrd0s1d 0xcc044990: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcaaff084 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcadeda80 (pid 70418) ino 425360, on dev amrd0s1d 0xcc190660: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcfed9108 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xca7d2900 (pid 76803) ino 425378, on dev amrd0s1d 0xcc676330: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcf8c14a4 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcaff8900 (pid 76841) ino 425384, on dev amrd0s1d 0xcf0ad110: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xce53b5ac ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcadb0900 (pid 79849) ino 425394, on dev amrd0s1d 0xce4f6aa0: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xce3cc4a4 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcb767300 (pid 79620) ino 425402, on dev amrd0s1d 0xce80d110: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcf502108 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcaff5d80 (pid 98225) ino 425478, on dev amrd0s1d 0xcd218990: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcf08e18c ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcaff5000 (pid 98241) ino 425482, on dev amrd0s1d 0xcbcb3440: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcf8a8738 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcaff8180 (pid 1341) ino 425505, on dev amrd0s1d 0xcf6fe440: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcf88f39c ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcaad7900 (pid 4512) ino 425512, on dev amrd0s1d 0xcdd07aa0: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcf7e9a50 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcadaf600 (pid 4464) ino 425513, on dev amrd0s1d 0xcc18eaa0: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xce2fa108 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcafbf300 (pid 13669) ino 425549, on dev amrd0s1d 0xcb9e1440: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcfbea39c ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcb64e480 (pid 13656) ino 425555, on dev amrd0s1d 0xcb8a8bb0: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcf6565ac ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcadb0180 (pid 22845) ino 425596, on dev amrd0s1d 0xccc47660: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcf551b58 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcb64e180 (pid 22870) ino 425597, on dev amrd0s1d 0xcec8ecc0: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcab61948 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xd0234600 (pid 32036) ino 425633, on dev amrd0s1d 0xcdca1bb0: tag ufs, type VREG usecount 2, writecount 2, refcount 888 mountedhere 0 flags () v_object 0xcaf23294 ref 0 pages 41308 lock type ufs: EXCL (count 1) by thread 0xcadb0a80 (pid 5541) ino 130855246, on dev amrd1s1d=20 > -----Original Message----- > From: Kris Kennaway [mailto:kris@obsecurity.org]=20 > Sent: Wednesday, May 16, 2007 12:33 PM > To: Andrew Edwards > Cc: freebsd-performance@freebsd.org > Subject: Re: Ufs dead-locks on freebsd 6.2 >=20 > On Wed, May 16, 2007 at 12:08:24PM -0400, Andrew Edwards wrote: > > I have a system running a dual intel zeon 2.8Ghz with 4G of ram and=20 > > using an intel raid controller model SRCU42X which uses the=20 > amr driver. > > I have had this server running 5.4 upgraded to 6.2 and was running=20 > > fine for several months and then after a normal reboot I've started=20 > > having all sorts of problems with what appears to be=20 > dead-locks in the=20 > > filesystem. This server is my backup server and I rsync files from=20 > > various servers onto this one fairly non-stop. If I stop=20 > the rsync's=20 > > the system appears to be stable although I did have a=20 > kernel core just=20 > > last night. > >=20 > > When I have been able to observe the problem I ususally see one=20 > > filesystem become inaccessible, perhaps var but I'm not=20 > sure, and then=20 > > in a short period of time the whole system is inaccessible.=20 > Usually=20 > > if I startup just one of the rsync's within a couple of hours the=20 > > system will be un-usable. > >=20 > > I did find this thread which seems to describe similar=20 > issues but this=20 > > is a different driver. > >=20 > http://lists.freebsd.org/pipermail/freebsd-questions/2006-Augu st/127835. > > html >=20 > Probably not relevant then. Deadlocks come in many=20 > varieties, all different. >=20 > > Currently I'm running with debug.mpsafevfs=3D0, = debug.mpsafenet=3D1and=20 > > debug.mpsafevm=3D0 but this doesn't seem to help. > >=20 > > On perahps a related issue I have two other nearly=20 > identical systems=20 > > which were going to be upgrading to 6.2 as on 5.4 I am experiencing=20 > > deadlocks and when I hit ctrl-t I see the system is either stuck in=20 > > ufs or zoneinfo and I have not found very much information=20 > about zoneinfo. > >=20 > > Does anyone have any suggestions on what I can look for or other=20 > > tuning options or had similar experiences? >=20 > See the chapter on kernel debugging in the developers=20 > handbook for the information you need to provide before we=20 > can begin to debug your problem. >=20 > Kris >=20 >=20 From owner-freebsd-performance@FreeBSD.ORG Wed May 16 17:19:37 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C7A7716A400; Wed, 16 May 2007 17:19:37 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id B6E9513C45D; Wed, 16 May 2007 17:19:37 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 84D401A3C19; Wed, 16 May 2007 10:20:29 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id CDE4351406; Wed, 16 May 2007 13:19:36 -0400 (EDT) Date: Wed, 16 May 2007 13:19:36 -0400 From: Kris Kennaway To: Andrew Edwards Message-ID: <20070516171936.GA74455@xor.obsecurity.org> References: <20070516163305.GA73495@xor.obsecurity.org> <5230D3C40B842D4F9FB3CD368021BEF72F08F3@exchange-2.sandvine.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5230D3C40B842D4F9FB3CD368021BEF72F08F3@exchange-2.sandvine.com> User-Agent: Mutt/1.4.2.2i Cc: freebsd-fs@freebsd.org, freebsd-performance@freebsd.org Subject: Re: Ufs dead-locks on freebsd 6.2 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 May 2007 17:19:37 -0000 On Wed, May 16, 2007 at 01:16:27PM -0400, Andrew Edwards wrote: > Here's the backtrace from the last crash along with the output from show > alllocks when the system was deadlocked. I have been running > 6.2-release and compliled with makeoptions debug=-g, invariants, > invariant_support and witness. I will update to 6-STABLE add > diagnositc, debug_locks and debug_vfs_locks as per the handbook > recommendation and retry. > > Yes, when the system was un-usable I was still able to ping it. I have > the serial console setup as the default console so I can remotely access > the box and break into the debugger etc. > > (kgdb) bt > #0 doadump () at pcpu.h:165 > #1 0xc059b480 in boot (howto=260) at > /usr/src/sys/kern/kern_shutdown.c:409 > #2 0xc059b795 in panic (fmt=0xc0787b04 "Most recently used by %s\n") > at /usr/src/sys/kern/kern_shutdown.c:565 That's a memory use-after-free. Check the DEBUG_MEMGUARD infrastructure for debugging of this. Kris From owner-freebsd-performance@FreeBSD.ORG Thu May 17 09:37:42 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E900916A400 for ; Thu, 17 May 2007 09:37:42 +0000 (UTC) (envelope-from david@solexine.fr) Received: from zimbra-ne01.network-studio.com (zimbra-ne01.network-studio.com [62.93.225.49]) by mx1.freebsd.org (Postfix) with ESMTP id A9B2413C459 for ; Thu, 17 May 2007 09:37:42 +0000 (UTC) (envelope-from david@solexine.fr) Received: from localhost (localhost.localdomain [127.0.0.1]) by zimbra-ne01.network-studio.com (Postfix) with ESMTP id 864552086F0; Thu, 17 May 2007 11:23:20 +0200 (CEST) X-Virus-Scanned: amavisd-new at X-Spam-Score: 1.393 X-Spam-Level: * X-Spam-Status: No, score=1.393 tagged_above=-10 required=4 tests=[BAYES_00=-2.599, RCVD_IN_NJABL_DUL=1.946, RCVD_IN_SORBS_DUL=2.046] Received: from zimbra-ne01.network-studio.com ([127.0.0.1]) by localhost (zimbra-ne01.network-studio.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 96cSLi7Y2Tyk; Thu, 17 May 2007 11:23:19 +0200 (CEST) Received: from [192.168.254.104] (AMontpellier-152-1-10-117.w81-251.abo.wanadoo.fr [81.251.36.117]) by zimbra-ne01.network-studio.com (Postfix) with ESMTP id 717A72086ED; Thu, 17 May 2007 11:23:19 +0200 (CEST) Message-ID: <464C1E07.1000403@solexine.fr> Date: Thu, 17 May 2007 11:19:03 +0200 From: David Touitou Organization: Solexine User-Agent: Thunderbird 1.5.0.10 (Windows/20070221) MIME-Version: 1.0 To: Tom Judge References: <464474F0.3040306@tomjudge.com> In-Reply-To: <464474F0.3040306@tomjudge.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit Cc: lydianconcepts@gmail.com, freebsd-performance@freebsd.org, Randy Schultz Subject: Re: possible issues with Dell/Perc5 raid X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 May 2007 09:37:43 -0000 Hello all, Tom Judge a écrit : > From the boot messages you have included below it seems that you have a > Dell SAS-5i (Internal PCIe) controller and not a Perc/5[ie]. > > The SAS-5 known to have very poor performance (1Mb/s 100%busy) under > certain loads (cvsup). I have something that looks like the same kind of problem, on a PE860 with SAS-5i and two SATA 160GB HD in RAID 1. The server is a webserver and "top -S" shows very high CPU usage (idle is 0%, user CPU is 40%, system CPU is 60%, interupt is 0%) when running 60 concurents httpd threads... I've browsed quite a bit arround this issue and ended in this thread (plus several other threads arround the same problem). I've just realised Dell has released a new firmware on 17 of april (beware of long URL) : http://support.euro.dell.com/support/downloads/download.aspx?c=fr&cs=RC1077983&l=fr&s=pad&releaseid=R149732&formatcnt=6&libid=0&fileid=199166 Has anyone tried it ? Best, David. From owner-freebsd-performance@FreeBSD.ORG Thu May 17 15:44:20 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1C54B16A401; Thu, 17 May 2007 15:44:20 +0000 (UTC) (envelope-from aedwards@sandvine.com) Received: from gw.sandvine.com (gw.sandvine.com [199.243.201.138]) by mx1.freebsd.org (Postfix) with ESMTP id 7B2C213C489; Thu, 17 May 2007 15:44:17 +0000 (UTC) (envelope-from aedwards@sandvine.com) Received: from exchange-2.sandvine.com ([192.168.16.12]) by gw.sandvine.com with Microsoft SMTPSVC(6.0.3790.1830); Thu, 17 May 2007 11:44:16 -0400 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable Date: Thu, 17 May 2007 11:44:15 -0400 Message-ID: <5230D3C40B842D4F9FB3CD368021BEF72F0908@exchange-2.sandvine.com> In-Reply-To: <5230D3C40B842D4F9FB3CD368021BEF72F08F3@exchange-2.sandvine.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Ufs dead-locks on freebsd 6.2 Thread-Index: AceX1+P2HSPmcMyKSLK15m7jv2eBpAAAVXqQAC/+rgA= From: "Andrew Edwards" To: X-OriginalArrivalTime: 17 May 2007 15:44:16.0473 (UTC) FILETIME=[39C1B490:01C7989A] Cc: freebsd-fs@freebsd.org Subject: RE: Ufs dead-locks on freebsd 6.2 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 May 2007 15:44:20 -0000 I've upgraded to 6-stable, added the kernel options as per the kernel handbook. After about 5 hours of they system in a deadlock it panic'd. Here's the backtrace, and show pcpu, show allpcpu, show locks, show alllocks, show lockedvnods and alltrace. I will have the system down for approx another 15-20mins if there's anything else someone would like while I'm in the debugger. db> bt Tracing pid 46784 tid 105112 td 0xd44a8000 kdb_enter(c0785f13) at kdb_enter+0x2b vfs_badlock(c0785f2c,c0786051,ccd47984) at vfs_badlock+0x47 assert_vop_locked(ccd47984,c0786051) at assert_vop_locked+0x4a vop_lock_post(f9f709dc,0,1002,ccd47984,f9f709f8,...) at vop_lock_post+0x2a VOP_LOCK_APV(c07dc2e0,f9f709dc) at VOP_LOCK_APV+0xa0 vn_lock(ccd47984,1002,d44a8000) at vn_lock+0xac lookup(f9f70c08) at lookup+0xde namei(f9f70c08) at namei+0x39a unp_connect(d44b2de8,d44dc380,d44a8000,d44b2de8,25,...) at unp_connect+0xf0 uipc_connect(d44b2de8,d44dc380,d44a8000) at uipc_connect+0x66 soconnect(d44b2de8,d44dc380,d44a8000) at soconnect+0x4e kern_connect(d44a8000,7,d44dc380,d44dc380,0,...) at kern_connect+0x74 connect(d44a8000,f9f70d04) at connect+0x2f syscall(3b,805003b,bfbf003b,bfbfd920,bfbfd922,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (98, FreeBSD ELF32, connect), eip =3D 0x6812d553, esp =3D 0xbfbfd90c, ebp =3D 0xbfbfd9a8 --- db> show pcpu cpuid =3D 1 curthread =3D 0xd44a8000: pid 46784 "cron" curpcb =3D 0xf9f70d90 fpcurthread =3D none idlethread =3D 0xccb5c600: pid 10 "idle: cpu1" APIC ID =3D 6 currentldt =3D 0x50 spin locks held: db> show allpcpu Current CPU: 1 cpuid =3D 0 curthread =3D 0xd4490d80: pid 46779 "cron" curpcb =3D 0xf9f13d90 fpcurthread =3D none idlethread =3D 0xccb5c780: pid 11 "idle: cpu0" APIC ID =3D 0 currentldt =3D 0x50 spin locks held: cpuid =3D 1 curthread =3D 0xd44a8000: pid 46784 "cron" curpcb =3D 0xf9f70d90 fpcurthread =3D none idlethread =3D 0xccb5c600: pid 10 "idle: cpu1" APIC ID =3D 6 currentldt =3D 0x50 spin locks held: db> show locks db> show alllocks db> show lockedvnods Locked vnodes 0xd12d16cc: tag ufs, type VREG usecount 1, writecount 1, refcount 4 mountedhere 0 flags () v_object 0xcd64ac60 ref 0 pages 48 lock type ufs: EXCL (count 1) by thread 0xcd6a6180 (pid 22025)#0 0xc0593f0d at lockmgr+0x4ed #1 0xc06b8e0e at ffs_lock+0x76 #2 0xc0739787 at VOP_LOCK_APV+0x87 #3 0xc0601c28 at vn_lock+0xac #4 0xc06015c4 at vn_write+0x138 #5 0xc05c4544 at dofilewrite+0x7c #6 0xc05c43e3 at kern_writev+0x3b #7 0xc05c4309 at write+0x45 #8 0xc0723e2b at syscall+0x25b #9 0xc070ee0f at Xint0x80_syscall+0x1f ino 494646, on dev amrd0s1d 0xccf9f2b8: tag ufs, type VDIR usecount 1, writecount 0, refcount 2495 mountedhere 0 flags (VV_ROOT) v_object 0xd23c39cc ref 0 pages 0 lock type ufs: EXCL (count 1) by thread 0xcd23fd80 (pid 40153) with 2493 pending#0 0xc0593f0d at lockmgr+0x4ed #1 0xc06b8e0e at ffs_lock+0x76 #2 0xc0739787 at VOP_LOCK_APV+0x87 #3 0xc0601c28 at vn_lock+0xac #4 0xc05f5d12 at vget+0xbe #5 0xc05ed9f9 at vfs_hash_get+0x8d #6 0xc06b7b8f at ffs_vget+0x27 #7 0xc06c1435 at ufs_root+0x19 #8 0xc05eef1c at lookup+0x7c8 #9 0xc05ee4b2 at namei+0x39a #10 0xc0600a13 at vn_open_cred+0x5b #11 0xc06009b6 at vn_open+0x1e #12 0xc05fa126 at kern_open+0xb6 #13 0xc05fa03a at open+0x1a #14 0xc0723e2b at syscall+0x25b #15 0xc070ee0f at Xint0x80_syscall+0x1f ino 2, on dev amrd0s1d 0xcd030984: tag ufs, type VDIR usecount 7, writecount 0, refcount 9 mountedhere 0 flags () v_object 0xcffb2108 ref 0 pages 0 lock type ufs: EXCL (count 1) by thread 0xcd9bc780 (pid 29680) with 1 pending#0 0xc0593f0d at lockmgr+0x4ed #1 0xc06b8e0e at ffs_lock+0x76 #2 0xc0739787 at VOP_LOCK_APV+0x87 #3 0xc0601c28 at vn_lock+0xac #4 0xc05f5d12 at vget+0xbe #5 0xc05ea48e at cache_lookup+0x34a #6 0xc05ea9c2 at vfs_cache_lookup+0x92 #7 0xc0737847 at VOP_LOOKUP_APV+0x87 #8 0xc05eebf8 at lookup+0x4a4 #9 0xc05ee4b2 at namei+0x39a #10 0xc0600a13 at vn_open_cred+0x5b #11 0xc06009b6 at vn_open+0x1e #12 0xc05fa126 at kern_open+0xb6 #13 0xc05fa03a at open+0x1a #14 0xc0723e2b at syscall+0x25b #15 0xc070ee0f at Xint0x80_syscall+0x1f ino 494592, on dev amrd0s1d 0xd1bad828: tag ufs, type VREG usecount 2, writecount 2, refcount 6 mountedhere 0 flags () v_object 0xd08b28c4 ref 0 pages 82 lock type ufs: EXCL (count 1) by thread 0xcd5b0780 (pid 36375) with 2 pending#0 0xc0593f0d at lockmgr+0x4ed #1 0xc06b8e0e at ffs_lock+0x76 #2 0xc0739787 at VOP_LOCK_APV+0x87 #3 0xc0601c28 at vn_lock+0xac #4 0xc06015c4 at vn_write+0x138 #5 0xc05c4544 at dofilewrite+0x7c #6 0xc05c43e3 at kern_writev+0x3b #7 0xc05c4309 at write+0x45 #8 0xc0723e2b at syscall+0x25b #9 0xc070ee0f at Xint0x80_syscall+0x1f ino 494775, on dev amrd0s1d 0xcfed0984: tag ufs, type VREG usecount 1, writecount 1, refcount 4 mountedhere 0 flags () v_object 0xd0dcf39c ref 0 pages 7 lock type ufs: EXCL (count 1) by thread 0xccfa9000 (pid 653)#0 0xc0593f0d at lockmgr+0x4ed #1 0xc06b8e0e at ffs_lock+0x76 #2 0xc0739787 at VOP_LOCK_APV+0x87 #3 0xc0601c28 at vn_lock+0xac #4 0xc05fd950 at fsync+0x9c #5 0xc0723e2b at syscall+0x25b #6 0xc070ee0f at Xint0x80_syscall+0x1f ino 188447, on dev amrd0s1d 0xd2e3b828: tag ufs, type VREG usecount 2, writecount 2, refcount 1612 mountedhere 0 flags () v_object 0xd2dee630 ref 0 pages 6436 lock type ufs: SHARED (count 1) with 1 pending#0 0xc0593b80 at lockmgr+0x160 #1 0xc06b8e0e at ffs_lock+0x76 #2 0xc0739787 at VOP_LOCK_APV+0x87 #3 0xc0601c28 at vn_lock+0xac #4 0xc06012fe at vn_read+0x132 #5 0xc05c4269 at dofileread+0x85 #6 0xc05c4102 at kern_readv+0x36 #7 0xc05c402d at read+0x45 #8 0xc0723e2b at syscall+0x25b #9 0xc070ee0f at Xint0x80_syscall+0x1f ino 130807917, on dev amrd1s1d 0xd157f570: tag ufs, type VREG usecount 1, writecount 1, refcount 2713 mountedhere 0 flags () v_object 0xd2d5118c ref 0 pages 245644 lock type ufs: EXCL (count 1) by thread 0xcd6a6300 (pid 26383)#0 0xc0593f0d at lockmgr+0x4ed #1 0xc06b8e0e at ffs_lock+0x76 #2 0xc0739787 at VOP_LOCK_APV+0x87 #3 0xc0601c28 at vn_lock+0xac #4 0xc06015c4 at vn_write+0x138 #5 0xc05c4544 at dofilewrite+0x7c #6 0xc05c43e3 at kern_writev+0x3b #7 0xc05c4309 at write+0x45 #8 0xc0723e2b at syscall+0x25b #9 0xc070ee0f at Xint0x80_syscall+0x1f ino 116230408, on dev amrd1s1d db> alltrace Tracing command pid 46787 tid 105110 td 0xd44a8300 *** error reading from address 72d76109 *** db> > -----Original Message----- > From: owner-freebsd-performance@freebsd.org=20 > [mailto:owner-freebsd-performance@freebsd.org] On Behalf Of=20 > Andrew Edwards > Sent: Wednesday, May 16, 2007 1:16 PM > To: freebsd-performance@freebsd.org > Cc: freebsd-fs@freebsd.org > Subject: RE: Ufs dead-locks on freebsd 6.2 >=20 > Here's the backtrace from the last crash along with the=20 > output from show alllocks when the system was deadlocked. I=20 > have been running 6.2-release and compliled with makeoptions=20 > debug=3D-g, invariants, invariant_support and witness. I will=20 > update to 6-STABLE add diagnositc, debug_locks and=20 > debug_vfs_locks as per the handbook recommendation and retry. >=20 > Yes, when the system was un-usable I was still able to ping=20 > it. I have the serial console setup as the default console=20 > so I can remotely access the box and break into the debugger etc. >=20 > (kgdb) bt > #0 doadump () at pcpu.h:165 > #1 0xc059b480 in boot (howto=3D260) at > /usr/src/sys/kern/kern_shutdown.c:409 > #2 0xc059b795 in panic (fmt=3D0xc0787b04 "Most recently used by = %s\n") > at /usr/src/sys/kern/kern_shutdown.c:565 > #3 0xc06c4a4d in mtrash_ctor (mem=3D0xce74fa00, size=3D0, arg=3D0x0, > flags=3D258) > at /usr/src/sys/vm/uma_dbg.c:137 > #4 0xc06c2c07 in uma_zalloc_arg (zone=3D0xc10615a0, udata=3D0x0,=20 > flags=3D258) > at /usr/src/sys/vm/uma_core.c:1850 > #5 0xc0591416 in malloc (size=3D272, mtp=3D0xc07c32c0, flags=3D258) = at > uma.h:275 > #6 0xc05edfab in __mnt_vnode_first (mvp=3D0xf3741c48, = mp=3D0xcaa14cf8) > at /usr/src/sys/kern/vfs_mount.c:1813 > #7 0xc05f2467 in vfs_msync (mp=3D0xcaa14cf8, flags=3D2) > at /usr/src/sys/kern/vfs_subr.c:2874 > #8 0xc05f2bbd in sync_fsync (ap=3D0x0) at > /usr/src/sys/kern/vfs_subr.c:3119 > #9 0xc072f4ee in VOP_FSYNC_APV (vop=3D0x0, a=3D0xf3741cbc) at=20 > vnode_if.c:1020 #10 0xc05f097c in sync_vnode (bo=3D0xca854e90,=20 > td=3D0xca435000) at > vnode_if.h:537 > #11 0xc05f0bf1 in sched_sync () at /usr/src/sys/kern/vfs_subr.c:1698 > #12 0xc0587248 in fork_exit (callout=3D0xc05f0a04 , = arg=3D0x0, > frame=3D0xf3741d38) at /usr/src/sys/kern/kern_fork.c:821 > #13 0xc070712c in fork_trampoline () at > /usr/src/sys/i386/i386/exception.s:208 >=20 >=20 > db> show alllocks > Process 36596 (sshd) thread 0xd1238c00 (102406) exclusive=20 > sleep mutex vm object (standard object) r =3D 0 (0xce2c87bc)=20 > locked @ /usr/src/sys/vm/vm_object.c:446 exclusive sx user=20 > map r =3D 0 (0xd128060c) locked @ > /usr/src/sys/vm/vm_map.c:307 > Process 887 (sshd) thread 0xca7d2000 (100056) exclusive sleep=20 > mutex vm object (standard object) r =3D 0 (0xcb713ad4) locked @=20 > /usr/src/sys/vm/vm_fault.c:297 exclusive sx user map r =3D 0=20 > (0xcaae4734) locked @ > /usr/src/sys/vm/vm_map.c:3074 > db> show lockedvnods > Locked vnodes >=20 > 0xcaa78660: tag ufs, type VREG > usecount 2, writecount 1, refcount 3 mountedhere 0 > flags () > v_object 0xc1046738 ref 0 pages 1596 > lock type ufs: EXCL (count 1) by thread 0xca689c00 (pid=20 > 536) with 1 pending > ino 494620, on dev amrd0s1d >=20 > 0xcaa86110: tag ufs, type VREG > usecount 1, writecount 1, refcount 3 mountedhere 0 > flags () > v_object 0xca85f738 ref 0 pages 44 > lock type ufs: EXCL (count 1) by thread 0xca7d2780 (pid 715) > ino 494633, on dev amrd0s1d >=20 > 0xcabe4110: tag ufs, type VDIR > usecount 12, writecount 0, refcount 14 mountedhere 0 > flags () > v_object 0xcab28840 ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xca9bbd80 (pid=20 > 14253) with > 3 pending > ino 423947, on dev amrd0s1d >=20 > 0xcb437990: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcb3c3dec ref 0 pages 4100 > lock type ufs: EXCL (count 1) by thread 0xcaffac00 (pid 20868) > ino 282640, on dev amrd0s1d >=20 > 0xcb99e550: tag ufs, type VDIR > usecount 2, writecount 0, refcount 4 mountedhere 0 > flags () > v_object 0xcef979cc ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xca9bb600 (pid=20 > 881) with 1 pending > ino 423987, on dev amrd0s1d >=20 > 0xcfc97dd0: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcd4975ac ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcb748900 (pid 2518) > ino 424275, on dev amrd0s1d >=20 > 0xccad9aa0: tag ufs, type VREG > usecount 1, writecount 1, refcount 3 mountedhere 0 > flags () > v_object 0xcf0c539c ref 0 pages 5 > lock type ufs: EXCL (count 1) by thread 0xca7d1c00 (pid 600) > ino 188446, on dev amrd0s1d >=20 > 0xccb0f110: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcb4609cc ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcafbf480 (pid 11054) > ino 424100, on dev amrd0s1d >=20 > 0xcc501bb0: tag ufs, type VREG > usecount 1, writecount 1, refcount 3 mountedhere 0 > flags () > v_object 0xcc7d19cc ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcb76d600 (pid=20 > 13743) with > 1 pending > ino 424279, on dev amrd0s1d >=20 > 0xcf96b220: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcf135c60 ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcafbf900 (pid 29458) > ino 424374, on dev amrd0s1d >=20 > 0xcc5bbbb0: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcdd5b318 ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcaad9900 (pid 50782) > ino 424276, on dev amrd0s1d >=20 > 0xcec1d000: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcd3d7108 ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcb76dc00 (pid 59514) > ino 424500, on dev amrd0s1d >=20 > 0xcebe5110: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xccee95ac ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcb650780 (pid 59975) > ino 424509, on dev amrd0s1d >=20 > 0xce0c1880: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xca8b64a4 ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcb768a80 (pid 69466) > ino 424555, on dev amrd0s1d >=20 > 0xcf652110: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcf4a318c ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcaff8600 (pid 75577) > ino 424579, on dev amrd0s1d >=20 > 0xce282550: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcf261318 ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xd0235a80 (pid 81734) > ino 424927, on dev amrd0s1d >=20 > 0xcc1d4dd0: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcd4a6630 ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcaccb900 (pid 81772) > ino 424928, on dev amrd0s1d >=20 > 0xcb820bb0: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcb251ad4 ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcadaf480 (pid 84037) > ino 424935, on dev amrd0s1d >=20 > 0xced5aaa0: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcb784210 ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xd0236000 (pid 202) > ino 425039, on dev amrd0s1d >=20 > 0xcbe45220: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xce55de70 ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcaad9a80 (pid 230) > ino 425043, on dev amrd0s1d >=20 > 0xcc098220: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xce4a9dec ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcafbfd80 (pid 9902) > ino 425093, on dev amrd0s1d >=20 > 0xcd585110: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcf8c1e70 ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcaccba80 (pid 24017) > ino 425144, on dev amrd0s1d >=20 > 0xceeac000: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcb1225ac ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcaff5a80 (pid 24775) > ino 425149, on dev amrd0s1d >=20 > 0xcc549aa0: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcab2d318 ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcaff8a80 (pid 42358) > ino 425227, on dev amrd0s1d >=20 > 0xcc6f7000: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcfd4139c ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcafbf600 (pid 43117) > ino 425230, on dev amrd0s1d >=20 > 0xccc44bb0: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcfd18d68 ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcadafd80 (pid 42859) > ino 425234, on dev amrd0s1d >=20 > 0xcc7a7220: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcfedf420 ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcb76c600 (pid 48968) > ino 425264, on dev amrd0s1d >=20 > 0xcc693aa0: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcf92f738 ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcb655300 (pid 55381) > ino 425286, on dev amrd0s1d >=20 > 0xcbabf220: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcfe297bc ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xd0234a80 (pid 63802) > ino 425322, on dev amrd0s1d >=20 > 0xcd760220: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcb11ac60 ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcadb0480 (pid 69938) > ino 425348, on dev amrd0s1d >=20 > 0xcc044990: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcaaff084 ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcadeda80 (pid 70418) > ino 425360, on dev amrd0s1d >=20 > 0xcc190660: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcfed9108 ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xca7d2900 (pid 76803) > ino 425378, on dev amrd0s1d >=20 > 0xcc676330: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcf8c14a4 ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcaff8900 (pid 76841) > ino 425384, on dev amrd0s1d >=20 > 0xcf0ad110: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xce53b5ac ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcadb0900 (pid 79849) > ino 425394, on dev amrd0s1d >=20 > 0xce4f6aa0: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xce3cc4a4 ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcb767300 (pid 79620) > ino 425402, on dev amrd0s1d >=20 > 0xce80d110: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcf502108 ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcaff5d80 (pid 98225) > ino 425478, on dev amrd0s1d >=20 > 0xcd218990: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcf08e18c ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcaff5000 (pid 98241) > ino 425482, on dev amrd0s1d >=20 > 0xcbcb3440: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcf8a8738 ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcaff8180 (pid 1341) > ino 425505, on dev amrd0s1d >=20 > 0xcf6fe440: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcf88f39c ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcaad7900 (pid 4512) > ino 425512, on dev amrd0s1d >=20 > 0xcdd07aa0: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcf7e9a50 ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcadaf600 (pid 4464) > ino 425513, on dev amrd0s1d >=20 > 0xcc18eaa0: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xce2fa108 ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcafbf300 (pid 13669) > ino 425549, on dev amrd0s1d >=20 > 0xcb9e1440: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcfbea39c ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcb64e480 (pid 13656) > ino 425555, on dev amrd0s1d >=20 > 0xcb8a8bb0: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcf6565ac ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcadb0180 (pid 22845) > ino 425596, on dev amrd0s1d >=20 > 0xccc47660: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcf551b58 ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xcb64e180 (pid 22870) > ino 425597, on dev amrd0s1d >=20 > 0xcec8ecc0: tag ufs, type VREG > usecount 1, writecount 1, refcount 2 mountedhere 0 > flags () > v_object 0xcab61948 ref 0 pages 1 > lock type ufs: EXCL (count 1) by thread 0xd0234600 (pid 32036) > ino 425633, on dev amrd0s1d >=20 > 0xcdca1bb0: tag ufs, type VREG > usecount 2, writecount 2, refcount 888 mountedhere 0 > flags () > v_object 0xcaf23294 ref 0 pages 41308 > lock type ufs: EXCL (count 1) by thread 0xcadb0a80 (pid 5541) > ino 130855246, on dev amrd1s1d=20 >=20 > > -----Original Message----- > > From: Kris Kennaway [mailto:kris@obsecurity.org] > > Sent: Wednesday, May 16, 2007 12:33 PM > > To: Andrew Edwards > > Cc: freebsd-performance@freebsd.org > > Subject: Re: Ufs dead-locks on freebsd 6.2 > >=20 > > On Wed, May 16, 2007 at 12:08:24PM -0400, Andrew Edwards wrote: > > > I have a system running a dual intel zeon 2.8Ghz with 4G=20 > of ram and=20 > > > using an intel raid controller model SRCU42X which uses the > > amr driver. > > > I have had this server running 5.4 upgraded to 6.2 and=20 > was running=20 > > > fine for several months and then after a normal reboot=20 > I've started=20 > > > having all sorts of problems with what appears to be > > dead-locks in the > > > filesystem. This server is my backup server and I rsync=20 > files from=20 > > > various servers onto this one fairly non-stop. If I stop > > the rsync's > > > the system appears to be stable although I did have a > > kernel core just > > > last night. > > >=20 > > > When I have been able to observe the problem I ususally see one=20 > > > filesystem become inaccessible, perhaps var but I'm not > > sure, and then > > > in a short period of time the whole system is inaccessible.=20 > > Usually > > > if I startup just one of the rsync's within a couple of hours the=20 > > > system will be un-usable. > > >=20 > > > I did find this thread which seems to describe similar > > issues but this > > > is a different driver. > > >=20 > > http://lists.freebsd.org/pipermail/freebsd-questions/2006-Augu > st/127835. > > > html > >=20 > > Probably not relevant then. Deadlocks come in many varieties, all=20 > > different. > >=20 > > > Currently I'm running with debug.mpsafevfs=3D0,=20 > debug.mpsafenet=3D1and=20 > > > debug.mpsafevm=3D0 but this doesn't seem to help. > > >=20 > > > On perahps a related issue I have two other nearly > > identical systems > > > which were going to be upgrading to 6.2 as on 5.4 I am=20 > experiencing=20 > > > deadlocks and when I hit ctrl-t I see the system is=20 > either stuck in=20 > > > ufs or zoneinfo and I have not found very much information > > about zoneinfo. > > >=20 > > > Does anyone have any suggestions on what I can look for or other=20 > > > tuning options or had similar experiences? > >=20 > > See the chapter on kernel debugging in the developers=20 > handbook for the=20 > > information you need to provide before we can begin to debug your=20 > > problem. > >=20 > > Kris > >=20 > >=20 > _______________________________________________ > freebsd-performance@freebsd.org mailing list=20 > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to=20 > "freebsd-performance-unsubscribe@freebsd.org" >=20 From owner-freebsd-performance@FreeBSD.ORG Thu May 17 17:04:40 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id CABFE16A411; Thu, 17 May 2007 17:04:40 +0000 (UTC) (envelope-from aedwards@sandvine.com) Received: from gw.sandvine.com (gw.sandvine.com [199.243.201.138]) by mx1.freebsd.org (Postfix) with ESMTP id 3E23413C501; Thu, 17 May 2007 17:04:39 +0000 (UTC) (envelope-from aedwards@sandvine.com) Received: from exchange-2.sandvine.com ([192.168.16.12]) by gw.sandvine.com with Microsoft SMTPSVC(6.0.3790.1830); Thu, 17 May 2007 13:03:38 -0400 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable Date: Thu, 17 May 2007 13:03:37 -0400 Message-ID: <5230D3C40B842D4F9FB3CD368021BEF72F090C@exchange-2.sandvine.com> In-Reply-To: <20070517170100.GA41395@deviant.kiev.zoral.com.ua> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Ufs dead-locks on freebsd 6.2 Thread-Index: AceYpPfEkW01y9yNRIWuy5UIKL+SuwAADblQ From: "Andrew Edwards" To: , X-OriginalArrivalTime: 17 May 2007 17:03:38.0127 (UTC) FILETIME=[4FEC71F0:01C798A5] Cc: Subject: RE: Ufs dead-locks on freebsd 6.2 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 May 2007 17:04:40 -0000 Here it is. db> show vnode 0xccd47984 vnode 0xccd47984: tag ufs, type VDIR usecount 5135, writecount 0, refcount 5137 mountedhere 0 flags (VV_ROOT) v_object 0xcd02518c ref 0 pages 1 #0 0xc0593f0d at lockmgr+0x4ed #1 0xc06b8e0e at ffs_lock+0x76 #2 0xc0739787 at VOP_LOCK_APV+0x87 #3 0xc0601c28 at vn_lock+0xac #4 0xc05ee832 at lookup+0xde #5 0xc05ee4b2 at namei+0x39a #6 0xc05e2ab0 at unp_connect+0xf0 #7 0xc05e1a6a at uipc_connect+0x66 #8 0xc05d9992 at soconnect+0x4e #9 0xc05dec60 at kern_connect+0x74 #10 0xc05debdf at connect+0x2f #11 0xc0723e2b at syscall+0x25b #12 0xc070ee0f at Xint0x80_syscall+0x1f ino 2, on dev amrd0s1a=20 > -----Original Message----- > From: Kostik Belousov [mailto:kostikbel@gmail.com]=20 > Sent: Thursday, May 17, 2007 1:01 PM > To: Andrew Edwards > Cc: freebsd-performance@freebsd.org; freebsd-fs@freebsd.org > Subject: Re: Ufs dead-locks on freebsd 6.2 >=20 > On Thu, May 17, 2007 at 11:44:15AM -0400, Andrew Edwards wrote: > > I've upgraded to 6-stable, added the kernel options as per=20 > the kernel=20 > > handbook. After about 5 hours of they system in a deadlock=20 > it panic'd. > > Here's the backtrace, and show pcpu, show allpcpu, show locks, show=20 > > alllocks, show lockedvnods and alltrace. > >=20 > > I will have the system down for approx another 15-20mins if there's=20 > > anything else someone would like while I'm in the debugger. > >=20 > >=20 > > db> bt > > Tracing pid 46784 tid 105112 td 0xd44a8000 > > kdb_enter(c0785f13) at kdb_enter+0x2b > > vfs_badlock(c0785f2c,c0786051,ccd47984) at vfs_badlock+0x47 > > assert_vop_locked(ccd47984,c0786051) at assert_vop_locked+0x4a > > vop_lock_post(f9f709dc,0,1002,ccd47984,f9f709f8,...) at=20 > > vop_lock_post+0x2a > > VOP_LOCK_APV(c07dc2e0,f9f709dc) at VOP_LOCK_APV+0xa0 > > vn_lock(ccd47984,1002,d44a8000) at vn_lock+0xac > > lookup(f9f70c08) at lookup+0xde > > namei(f9f70c08) at namei+0x39a > > unp_connect(d44b2de8,d44dc380,d44a8000,d44b2de8,25,...) at=20 > > unp_connect+0xf0 > > uipc_connect(d44b2de8,d44dc380,d44a8000) at uipc_connect+0x66 > > soconnect(d44b2de8,d44dc380,d44a8000) at soconnect+0x4e > > kern_connect(d44a8000,7,d44dc380,d44dc380,0,...) at=20 > kern_connect+0x74 > > connect(d44a8000,f9f70d04) at connect+0x2f > > syscall(3b,805003b,bfbf003b,bfbfd920,bfbfd922,...) at syscall+0x25b > > Xint0x80_syscall() at Xint0x80_syscall+0x1f >=20 > Could you, please, do the "show vnode 0xccd47984" from ddb=20 > prompt or "p/x *(struct vnode *)0xccd47984" from kgdb using=20 > dump for this panic ? >=20 > Side note: it seems that on HEAD, vnode_if.awk does not=20 > generate call to vop_lock_{pre,post} due to mismatch in the=20 > name of vop and pre/post names. >=20 >=20 >=20 From owner-freebsd-performance@FreeBSD.ORG Thu May 17 18:09:09 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E36D416A406; Thu, 17 May 2007 18:09:09 +0000 (UTC) (envelope-from aedwards@sandvine.com) Received: from gw.sandvine.com (gw.sandvine.com [199.243.201.138]) by mx1.freebsd.org (Postfix) with ESMTP id 962FF13C468; Thu, 17 May 2007 18:09:09 +0000 (UTC) (envelope-from aedwards@sandvine.com) Received: from exchange-2.sandvine.com ([192.168.16.12]) by gw.sandvine.com with Microsoft SMTPSVC(6.0.3790.1830); Thu, 17 May 2007 14:08:08 -0400 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable Date: Thu, 17 May 2007 14:08:07 -0400 Message-ID: <5230D3C40B842D4F9FB3CD368021BEF72F090E@exchange-2.sandvine.com> In-Reply-To: <20070517174735.GB41395@deviant.kiev.zoral.com.ua> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Ufs dead-locks on freebsd 6.2 Thread-Index: AceYq3iW9WrZe9tvSra5wo/FsSspEgAAXReA From: "Andrew Edwards" To: , X-OriginalArrivalTime: 17 May 2007 18:08:08.0919 (UTC) FILETIME=[53186E70:01C798AE] Cc: Subject: RE: Ufs dead-locks on freebsd 6.2 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 May 2007 18:09:10 -0000 When I initially built my kernel it was from a fresh copy of 6.2-release which I then sync'd to 6-stable and I used cvsup to do it so I believe it should be clean. I will look into running memtest86 on it to see if anything had happened to the memory. > -----Original Message----- > From: Kostik Belousov [mailto:kostikbel@gmail.com]=20 > Sent: Thursday, May 17, 2007 1:48 PM > To: Andrew Edwards > Cc: freebsd-performance@freebsd.org; freebsd-fs@freebsd.org > Subject: Re: Ufs dead-locks on freebsd 6.2 >=20 > On Thu, May 17, 2007 at 01:03:37PM -0400, Andrew Edwards wrote: > > Here it is. > >=20 > > db> show vnode 0xccd47984 > > vnode 0xccd47984: tag ufs, type VDIR > > usecount 5135, writecount 0, refcount 5137 mountedhere 0 > > flags (VV_ROOT) > > v_object 0xcd02518c ref 0 pages 1 > > #0 0xc0593f0d at lockmgr+0x4ed > > #1 0xc06b8e0e at ffs_lock+0x76 > > #2 0xc0739787 at VOP_LOCK_APV+0x87 > > #3 0xc0601c28 at vn_lock+0xac > > #4 0xc05ee832 at lookup+0xde > > #5 0xc05ee4b2 at namei+0x39a > > #6 0xc05e2ab0 at unp_connect+0xf0 > > #7 0xc05e1a6a at uipc_connect+0x66 > > #8 0xc05d9992 at soconnect+0x4e > > #9 0xc05dec60 at kern_connect+0x74 > > #10 0xc05debdf at connect+0x2f > > #11 0xc0723e2b at syscall+0x25b > > #12 0xc070ee0f at Xint0x80_syscall+0x1f > >=20 > > ino 2, on dev amrd0s1a > It seems to be the sort of things that cannot happen.=20 > VOP_LOCK() returned 0, but vnode was not really locked. >=20 > Although claiming that kernel code cannot have such bug is=20 > too optimistic, I would first make sure that: > 1. You checked the memory of the machine. > 2. Your kernel is built from pristine sources. >=20 > >=20 > > > -----Original Message----- > > > From: Kostik Belousov [mailto:kostikbel@gmail.com] > > > Sent: Thursday, May 17, 2007 1:01 PM > > > To: Andrew Edwards > > > Cc: freebsd-performance@freebsd.org; freebsd-fs@freebsd.org > > > Subject: Re: Ufs dead-locks on freebsd 6.2 > > >=20 > > > On Thu, May 17, 2007 at 11:44:15AM -0400, Andrew Edwards wrote: > > > > I've upgraded to 6-stable, added the kernel options as per > > > the kernel > > > > handbook. After about 5 hours of they system in a deadlock > > > it panic'd. > > > > Here's the backtrace, and show pcpu, show allpcpu, show locks,=20 > > > > show alllocks, show lockedvnods and alltrace. > > > >=20 > > > > I will have the system down for approx another 15-20mins if=20 > > > > there's anything else someone would like while I'm in=20 > the debugger. > > > >=20 > > > >=20 > > > > db> bt > > > > Tracing pid 46784 tid 105112 td 0xd44a8000 > > > > kdb_enter(c0785f13) at kdb_enter+0x2b > > > > vfs_badlock(c0785f2c,c0786051,ccd47984) at vfs_badlock+0x47 > > > > assert_vop_locked(ccd47984,c0786051) at assert_vop_locked+0x4a > > > > vop_lock_post(f9f709dc,0,1002,ccd47984,f9f709f8,...) at=20 > > > > vop_lock_post+0x2a > > > > VOP_LOCK_APV(c07dc2e0,f9f709dc) at VOP_LOCK_APV+0xa0 > > > > vn_lock(ccd47984,1002,d44a8000) at vn_lock+0xac > > > > lookup(f9f70c08) at lookup+0xde > > > > namei(f9f70c08) at namei+0x39a > > > > unp_connect(d44b2de8,d44dc380,d44a8000,d44b2de8,25,...) at=20 > > > > unp_connect+0xf0 > > > > uipc_connect(d44b2de8,d44dc380,d44a8000) at uipc_connect+0x66 > > > > soconnect(d44b2de8,d44dc380,d44a8000) at soconnect+0x4e > > > > kern_connect(d44a8000,7,d44dc380,d44dc380,0,...) at > > > kern_connect+0x74 > > > > connect(d44a8000,f9f70d04) at connect+0x2f > > > > syscall(3b,805003b,bfbf003b,bfbfd920,bfbfd922,...) at=20 > > > > syscall+0x25b > > > > Xint0x80_syscall() at Xint0x80_syscall+0x1f > > >=20 > > > Could you, please, do the "show vnode 0xccd47984" from=20 > ddb prompt or=20 > > > "p/x *(struct vnode *)0xccd47984" from kgdb using dump for this=20 > > > panic ? > > >=20 > > > Side note: it seems that on HEAD, vnode_if.awk does not generate=20 > > > call to vop_lock_{pre,post} due to mismatch in the name=20 > of vop and=20 > > > pre/post names. > > >=20 > > >=20 > > >=20 > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to=20 > "freebsd-fs-unsubscribe@freebsd.org" >=20 From owner-freebsd-performance@FreeBSD.ORG Thu May 17 17:34:13 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BB50216A403 for ; Thu, 17 May 2007 17:34:13 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from relay01.kiev.sovam.com (relay01.kiev.sovam.com [62.64.120.200]) by mx1.freebsd.org (Postfix) with ESMTP id 4F78B13C465 for ; Thu, 17 May 2007 17:34:13 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from [212.82.216.227] (helo=fw.zoral.com.ua) by relay01.kiev.sovam.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.60) (envelope-from ) id 1HojLh-0009bW-1o; Thu, 17 May 2007 20:01:09 +0300 Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by fw.zoral.com.ua (8.13.4/8.13.4) with ESMTP id l4HH119P086194 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 17 May 2007 20:01:01 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.1/8.14.1) with ESMTP id l4HH11tj043395; Thu, 17 May 2007 20:01:01 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.1/8.14.1/Submit) id l4HH10BM043394; Thu, 17 May 2007 20:01:00 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 17 May 2007 20:01:00 +0300 From: Kostik Belousov To: Andrew Edwards Message-ID: <20070517170100.GA41395@deviant.kiev.zoral.com.ua> References: <5230D3C40B842D4F9FB3CD368021BEF72F08F3@exchange-2.sandvine.com> <5230D3C40B842D4F9FB3CD368021BEF72F0908@exchange-2.sandvine.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="oyUTqETQ0mS9luUI" Content-Disposition: inline In-Reply-To: <5230D3C40B842D4F9FB3CD368021BEF72F0908@exchange-2.sandvine.com> User-Agent: Mutt/1.4.2.2i X-Virus-Scanned: ClamAV version 0.88.7, clamav-milter version 0.88.7 on fw.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-0.1 required=5.0 tests=ALL_TRUSTED,SPF_NEUTRAL autolearn=failed version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on fw.zoral.com.ua X-Scanner-Signature: a96565856d7d777f03ae631b61e05b17 X-DrWeb-checked: yes X-SpamTest-Envelope-From: kostikbel@gmail.com X-SpamTest-Group-ID: 00000000 X-SpamTest-Info: Profiles 1053 [May 16 2007] X-SpamTest-Info: helo_type=3 X-SpamTest-Info: {received from trusted relay: not dialup} X-SpamTest-Method: none X-SpamTest-Method: Local Lists X-SpamTest-Rate: 0 X-SpamTest-Status: Not detected X-SpamTest-Status-Extended: not_detected X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0255], KAS30/Release X-Mailman-Approved-At: Thu, 17 May 2007 23:02:57 +0000 Cc: freebsd-fs@freebsd.org, freebsd-performance@freebsd.org Subject: Re: Ufs dead-locks on freebsd 6.2 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 May 2007 17:34:13 -0000 --oyUTqETQ0mS9luUI Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, May 17, 2007 at 11:44:15AM -0400, Andrew Edwards wrote: > I've upgraded to 6-stable, added the kernel options as per the kernel > handbook. After about 5 hours of they system in a deadlock it panic'd. > Here's the backtrace, and show pcpu, show allpcpu, show locks, show > alllocks, show lockedvnods and alltrace. >=20 > I will have the system down for approx another 15-20mins if there's > anything else someone would like while I'm in the debugger. >=20 >=20 > db> bt > Tracing pid 46784 tid 105112 td 0xd44a8000 > kdb_enter(c0785f13) at kdb_enter+0x2b > vfs_badlock(c0785f2c,c0786051,ccd47984) at vfs_badlock+0x47 > assert_vop_locked(ccd47984,c0786051) at assert_vop_locked+0x4a > vop_lock_post(f9f709dc,0,1002,ccd47984,f9f709f8,...) at > vop_lock_post+0x2a > VOP_LOCK_APV(c07dc2e0,f9f709dc) at VOP_LOCK_APV+0xa0 > vn_lock(ccd47984,1002,d44a8000) at vn_lock+0xac > lookup(f9f70c08) at lookup+0xde > namei(f9f70c08) at namei+0x39a > unp_connect(d44b2de8,d44dc380,d44a8000,d44b2de8,25,...) at > unp_connect+0xf0 > uipc_connect(d44b2de8,d44dc380,d44a8000) at uipc_connect+0x66 > soconnect(d44b2de8,d44dc380,d44a8000) at soconnect+0x4e > kern_connect(d44a8000,7,d44dc380,d44dc380,0,...) at kern_connect+0x74 > connect(d44a8000,f9f70d04) at connect+0x2f > syscall(3b,805003b,bfbf003b,bfbfd920,bfbfd922,...) at syscall+0x25b > Xint0x80_syscall() at Xint0x80_syscall+0x1f Could you, please, do the "show vnode 0xccd47984" from ddb prompt or "p/x *(struct vnode *)0xccd47984" from kgdb using dump for this panic ? Side note: it seems that on HEAD, vnode_if.awk does not generate call to vop_lock_{pre,post} due to mismatch in the name of vop and pre/post names. --oyUTqETQ0mS9luUI Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFGTIpLC3+MBN1Mb4gRAtdjAJ4pBRLR0W6+tKEzFLZBQaO/8TiSlgCZAa1B npATY/aNbd2+iRhJy8Mdjn8= =WWY8 -----END PGP SIGNATURE----- --oyUTqETQ0mS9luUI-- From owner-freebsd-performance@FreeBSD.ORG Thu May 17 17:47:44 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9178616A403; Thu, 17 May 2007 17:47:44 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from relay01.kiev.sovam.com (relay01.kiev.sovam.com [62.64.120.200]) by mx1.freebsd.org (Postfix) with ESMTP id 22D3313C455; Thu, 17 May 2007 17:47:44 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from [212.82.216.227] (helo=fw.zoral.com.ua) by relay01.kiev.sovam.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.60) (envelope-from ) id 1Hok4k-0000Sr-F3; Thu, 17 May 2007 20:47:43 +0300 Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by fw.zoral.com.ua (8.13.4/8.13.4) with ESMTP id l4HHlanA087265 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 17 May 2007 20:47:36 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.1/8.14.1) with ESMTP id l4HHlZpq044397; Thu, 17 May 2007 20:47:35 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.1/8.14.1/Submit) id l4HHlZSt044396; Thu, 17 May 2007 20:47:35 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 17 May 2007 20:47:35 +0300 From: Kostik Belousov To: Andrew Edwards Message-ID: <20070517174735.GB41395@deviant.kiev.zoral.com.ua> References: <20070517170100.GA41395@deviant.kiev.zoral.com.ua> <5230D3C40B842D4F9FB3CD368021BEF72F090C@exchange-2.sandvine.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="9zSXsLTf0vkW971A" Content-Disposition: inline In-Reply-To: <5230D3C40B842D4F9FB3CD368021BEF72F090C@exchange-2.sandvine.com> User-Agent: Mutt/1.4.2.2i X-Virus-Scanned: ClamAV version 0.88.7, clamav-milter version 0.88.7 on fw.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-0.1 required=5.0 tests=ALL_TRUSTED,SPF_NEUTRAL autolearn=failed version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on fw.zoral.com.ua X-Scanner-Signature: 811d4bb63238b2e39a625e79cff8aaca X-DrWeb-checked: yes X-SpamTest-Envelope-From: kostikbel@gmail.com X-SpamTest-Group-ID: 00000000 X-SpamTest-Info: Profiles 1053 [May 16 2007] X-SpamTest-Info: helo_type=3 X-SpamTest-Info: {received from trusted relay: not dialup} X-SpamTest-Method: none X-SpamTest-Method: Local Lists X-SpamTest-Rate: 0 X-SpamTest-Status: Not detected X-SpamTest-Status-Extended: not_detected X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0255], KAS30/Release X-Mailman-Approved-At: Thu, 17 May 2007 23:02:57 +0000 Cc: freebsd-fs@freebsd.org, freebsd-performance@freebsd.org Subject: Re: Ufs dead-locks on freebsd 6.2 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 May 2007 17:47:44 -0000 --9zSXsLTf0vkW971A Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, May 17, 2007 at 01:03:37PM -0400, Andrew Edwards wrote: > Here it is. >=20 > db> show vnode 0xccd47984 > vnode 0xccd47984: tag ufs, type VDIR > usecount 5135, writecount 0, refcount 5137 mountedhere 0 > flags (VV_ROOT) > v_object 0xcd02518c ref 0 pages 1 > #0 0xc0593f0d at lockmgr+0x4ed > #1 0xc06b8e0e at ffs_lock+0x76 > #2 0xc0739787 at VOP_LOCK_APV+0x87 > #3 0xc0601c28 at vn_lock+0xac > #4 0xc05ee832 at lookup+0xde > #5 0xc05ee4b2 at namei+0x39a > #6 0xc05e2ab0 at unp_connect+0xf0 > #7 0xc05e1a6a at uipc_connect+0x66 > #8 0xc05d9992 at soconnect+0x4e > #9 0xc05dec60 at kern_connect+0x74 > #10 0xc05debdf at connect+0x2f > #11 0xc0723e2b at syscall+0x25b > #12 0xc070ee0f at Xint0x80_syscall+0x1f >=20 > ino 2, on dev amrd0s1a=20 It seems to be the sort of things that cannot happen. VOP_LOCK() returned 0, but vnode was not really locked. Although claiming that kernel code cannot have such bug is too optimistic, I would first make sure that: 1. You checked the memory of the machine. 2. Your kernel is built from pristine sources. >=20 > > -----Original Message----- > > From: Kostik Belousov [mailto:kostikbel@gmail.com]=20 > > Sent: Thursday, May 17, 2007 1:01 PM > > To: Andrew Edwards > > Cc: freebsd-performance@freebsd.org; freebsd-fs@freebsd.org > > Subject: Re: Ufs dead-locks on freebsd 6.2 > >=20 > > On Thu, May 17, 2007 at 11:44:15AM -0400, Andrew Edwards wrote: > > > I've upgraded to 6-stable, added the kernel options as per=20 > > the kernel=20 > > > handbook. After about 5 hours of they system in a deadlock=20 > > it panic'd. > > > Here's the backtrace, and show pcpu, show allpcpu, show locks, show= =20 > > > alllocks, show lockedvnods and alltrace. > > >=20 > > > I will have the system down for approx another 15-20mins if there's= =20 > > > anything else someone would like while I'm in the debugger. > > >=20 > > >=20 > > > db> bt > > > Tracing pid 46784 tid 105112 td 0xd44a8000 > > > kdb_enter(c0785f13) at kdb_enter+0x2b > > > vfs_badlock(c0785f2c,c0786051,ccd47984) at vfs_badlock+0x47 > > > assert_vop_locked(ccd47984,c0786051) at assert_vop_locked+0x4a > > > vop_lock_post(f9f709dc,0,1002,ccd47984,f9f709f8,...) at=20 > > > vop_lock_post+0x2a > > > VOP_LOCK_APV(c07dc2e0,f9f709dc) at VOP_LOCK_APV+0xa0 > > > vn_lock(ccd47984,1002,d44a8000) at vn_lock+0xac > > > lookup(f9f70c08) at lookup+0xde > > > namei(f9f70c08) at namei+0x39a > > > unp_connect(d44b2de8,d44dc380,d44a8000,d44b2de8,25,...) at=20 > > > unp_connect+0xf0 > > > uipc_connect(d44b2de8,d44dc380,d44a8000) at uipc_connect+0x66 > > > soconnect(d44b2de8,d44dc380,d44a8000) at soconnect+0x4e > > > kern_connect(d44a8000,7,d44dc380,d44dc380,0,...) at=20 > > kern_connect+0x74 > > > connect(d44a8000,f9f70d04) at connect+0x2f > > > syscall(3b,805003b,bfbf003b,bfbfd920,bfbfd922,...) at syscall+0x25b > > > Xint0x80_syscall() at Xint0x80_syscall+0x1f > >=20 > > Could you, please, do the "show vnode 0xccd47984" from ddb=20 > > prompt or "p/x *(struct vnode *)0xccd47984" from kgdb using=20 > > dump for this panic ? > >=20 > > Side note: it seems that on HEAD, vnode_if.awk does not=20 > > generate call to vop_lock_{pre,post} due to mismatch in the=20 > > name of vop and pre/post names. > >=20 > >=20 > >=20 > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" --9zSXsLTf0vkW971A Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFGTJU2C3+MBN1Mb4gRAi/sAKC7GVHcLs8Lq1rUKs/VL2JIk01atACfemRU vELK27Wc9MkcDj6Mn2mJRHQ= =h+0C -----END PGP SIGNATURE----- --9zSXsLTf0vkW971A-- From owner-freebsd-performance@FreeBSD.ORG Fri May 18 10:38:57 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E9EF216A401 for ; Fri, 18 May 2007 10:38:57 +0000 (UTC) (envelope-from redcrash@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.245]) by mx1.freebsd.org (Postfix) with ESMTP id 5AD8F13C46E for ; Fri, 18 May 2007 10:38:54 +0000 (UTC) (envelope-from redcrash@gmail.com) Received: by an-out-0708.google.com with SMTP id d23so208339and for ; Fri, 18 May 2007 03:38:53 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type; b=of33Fn4tEQbX5lT1DwCOcYEIn9TavYioGL6ptSuSJ2ODz7DD3bSa0W9xxYKzHk8bVAmmce8GdxD5eSXWI1p2wmS8KrW6YAQNuMRkedo+Z4nfRbSM2p16Kz0I4d8R+K1+No+AAewvloQ78LAzuOs46IL5hc7IfG2dzkjJpQX6EL0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:mime-version:content-type; b=VzgjTK6yuGPgoPlEFWgMJThYph3tV8LLJFOWaWpxokxY2bHTtYB3D2abtrL7CHTuPxnGyN74KfP3016Vy6CB44IEViybbuAVJ4xazrlItg7KanRaiiC4AvhMkORnn9MAwU08k+Oc5pitVzJM4YH3boAJxaLE3FarOhzEGLUWMjA= Received: by 10.100.171.16 with SMTP id t16mr1015080ane.1179484733345; Fri, 18 May 2007 03:38:53 -0700 (PDT) Received: by 10.100.109.2 with HTTP; Fri, 18 May 2007 03:38:53 -0700 (PDT) Message-ID: Date: Fri, 18 May 2007 12:38:53 +0200 From: "Harald Servat" To: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org, freebsd-hpc@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: PAPI for FreeBSD X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 May 2007 10:38:58 -0000 Hello, I'm working on the port of PAPI (Performance API) library to FreeBSD using Joseph Koshy's hwpmc / libpmc (see hwpmc(4) / pmc(3)). From the PAPI homepage: PAPI aims to provide the tool designer and application engineer with a consistent interface and methodology for use of the performance counter hardware found in most major microprocessors. PAPI enables software engineers to see, in near real time, the relation between software performance and processor events. I'm searching some testers for my first version of the port because I'm only able to test it on my laptop (FreeBSD 6.2 / Pentium M) and it would be great to test it in other kind of processors (now it's only supported on Pentium 2/3/4/Celeron AMD K7/8) before releasing it (and providing my patches to PAPI developers). Anyone interested on doing this test, please, send me an email and I'll reply you with some instructions to follow. Thank you very much. -- _________________________________________________________________ Empty your memory, with a free()... like a pointer! If you cast a pointer to an integer, it becomes an integer, if you cast a pointer to a struct, it becomes a struct. The pointer can crash..., and can overflow. Be a pointer my friend... From owner-freebsd-performance@FreeBSD.ORG Fri May 18 07:57:41 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 98B7C16A405 for ; Fri, 18 May 2007 07:57:41 +0000 (UTC) (envelope-from redcrash@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.245]) by mx1.freebsd.org (Postfix) with ESMTP id 5945913C459 for ; Fri, 18 May 2007 07:57:41 +0000 (UTC) (envelope-from redcrash@gmail.com) Received: by an-out-0708.google.com with SMTP id d23so199750and for ; Fri, 18 May 2007 00:57:40 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type; b=chRJ/W5kiuILpZ0rnyGa5xRJBs3OUNwbsZa3wrYbFRCGsxYGmmSVbSHV7GA6JZz4ae2yi7B1tPDchwzRbDVL2qB5THIqFbinMPUnc4VmJZ8D/g58C/EEaDTSOC9LTk9M1tz9ITSq8UV5UyGWYkLrahpdIQHqQgC2JSLvHRnSv2Y= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:mime-version:content-type; b=kblutsHhXJxBHXXwcFJbeyw5/+Wp/hid4GI7SweHgyBrYk1+vQ6l1TXqJZVAyQ0UJeWjZXdE04RdpN0FpwZ+T7l+I4z53jerIUuMCK1rpRTNY9jyUU63Jnp3nMgC2aogw0Hg2tQtFMxdfNPsJD6D56yYgs1ONKoDk6LTrJ9OBY8= Received: by 10.100.191.5 with SMTP id o5mr899895anf.1179473492682; Fri, 18 May 2007 00:31:32 -0700 (PDT) Received: by 10.100.109.2 with HTTP; Fri, 18 May 2007 00:31:32 -0700 (PDT) Message-ID: Date: Fri, 18 May 2007 09:31:32 +0200 From: "Harald Servat" To: freebsd-performance@freebsd.org, freebsd-hpc@freebsd.org MIME-Version: 1.0 X-Mailman-Approved-At: Fri, 18 May 2007 11:59:45 +0000 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: PAPI for FreeBSD X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 May 2007 07:57:41 -0000 Hello, I'm working on the port of PAPI (Performance API) library to FreeBSD using Joseph Koshy's hwpmc / libpmc (see hwpmc(4) / pmc(3)). From the PAPI homepage: PAPI aims to provide the tool designer and application engineer with a consistent interface and methodology for use of the performance counter hardware found in most major microprocessors. PAPI enables software engineers to see, in near real time, the relation between software performance and processor events. I'm searching testers for my first version of the port because I'm only able to test it on my laptop (FreeBSD 6.2 / Pentium M) and it would be great to test it in other kind of processors (now it's only supported on Pentium 2/3/4/Celeron AMD K7/8) before releasing it (and providing my patches to PAPI developers). Anyone interested on doing this test, please, send me an email and I'll reply you with some instructions to follow. Thank you very much. -- _________________________________________________________________ Empty your memory, with a free()... like a pointer! If you cast a pointer to an integer, it becomes an integer, if you cast a pointer to a struct, it becomes a struct. The pointer can crash..., and can overflow. Be a pointer my friend... From owner-freebsd-performance@FreeBSD.ORG Fri May 18 13:24:49 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BB95F16A404 for ; Fri, 18 May 2007 13:24:49 +0000 (UTC) (envelope-from redcrash@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.243]) by mx1.freebsd.org (Postfix) with ESMTP id 71C6A13C44B for ; Fri, 18 May 2007 13:24:49 +0000 (UTC) (envelope-from redcrash@gmail.com) Received: by an-out-0708.google.com with SMTP id d23so219244and for ; Fri, 18 May 2007 06:24:48 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=pe0fBzuNHhmZ7V2XDCgMFJJeywLyZPOn1OwHxXb6vNJOWNemQ94RQT1Oxlk3HZADbunCk7C1kwZMUm+A3r2eAZ8PmuiJTsRgfNgMGJMuuVVlRQWgCwFGRZ5hk8dudNIflt9mBT6o0bxrGbR9iTR4MSQuXrz4V5QBuRsmek9e/jc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=nLoWa9GwdZHPxs3JuAzUkoci4g7Kxwhm/f4RLu+oCHOv7Z59pNGPeeg+A7Ib8ungad3T5wMyKsR5rKli1sCOiYUlLwrqOt/pwdSh5oGLh0g8k2mjLIDLcizfnZ6st3wCEKLyAvEK9FU4MXbPlC2nvHBf/wsxJpNvnF/NoA9pe+Q= Received: by 10.100.35.17 with SMTP id i17mr1116528ani.1179494688673; Fri, 18 May 2007 06:24:48 -0700 (PDT) Received: by 10.100.109.2 with HTTP; Fri, 18 May 2007 06:24:48 -0700 (PDT) Message-ID: Date: Fri, 18 May 2007 15:24:48 +0200 From: "Harald Servat" To: "Benjamin Lutz" In-Reply-To: <200705181359.59883.mail@maxlor.com> MIME-Version: 1.0 References: <200705181359.59883.mail@maxlor.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org, freebsd-hpc@freebsd.org Subject: Re: PAPI for FreeBSD X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 May 2007 13:24:49 -0000 I'm sorry Benjamin, but libpmc/hwpmc (and my version of PAPI by extension) does not support Core Duo processors nowadays. Maybe Joseph Koshy (the person who is responsible for libpmc/hwpmc) could give you some info on what he needs to support them. Thank you, 2007/5/18, Benjamin Lutz : > > On Friday 18 May 2007 12:38, Harald Servat wrote: > > I'm searching some testers for my first version of the port because > > I'm only able to test it on my laptop (FreeBSD 6.2 / Pentium M) and > > it would be great to test it in other kind of processors (now it's > > only supported on Pentium 2/3/4/Celeron AMD K7/8) before releasing it > > (and providing my patches to PAPI developers). > > Anyone interested on doing this test, please, send me an email and > > I'll reply you with some instructions to follow. > > I've got a Core 2 Duo/Asus P5B Deluxe system here running FreeBSD 6.2 > here. If that's useful to you I'm willing to do some tests. > > Cheers > Benjamin > > -- _________________________________________________________________ Empty your memory, with a free()... like a pointer! If you cast a pointer to an integer, it becomes an integer, if you cast a pointer to a struct, it becomes a struct. The pointer can crash..., and can overflow. Be a pointer my friend... From owner-freebsd-performance@FreeBSD.ORG Fri May 18 22:43:36 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AC5AB16A400; Fri, 18 May 2007 22:43:36 +0000 (UTC) (envelope-from aedwards@sandvine.com) Received: from gw.sandvine.com (gw.sandvine.com [199.243.201.138]) by mx1.freebsd.org (Postfix) with ESMTP id 6746713C447; Fri, 18 May 2007 22:43:36 +0000 (UTC) (envelope-from aedwards@sandvine.com) Received: from exchange-2.sandvine.com ([192.168.16.12]) by gw.sandvine.com with Microsoft SMTPSVC(6.0.3790.1830); Fri, 18 May 2007 18:43:35 -0400 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable Date: Fri, 18 May 2007 18:43:35 -0400 Message-ID: <5230D3C40B842D4F9FB3CD368021BEF72F0926@exchange-2.sandvine.com> In-Reply-To: <464DF9CA.9090900@freebsd.org> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Ufs dead-locks on freebsd 6.2 Thread-Index: AceZgA8XJQM9a6noQX+h86ioyzHh9wAHOkwg From: "Andrew Edwards" To: , X-OriginalArrivalTime: 18 May 2007 22:43:35.0785 (UTC) FILETIME=[F849CD90:01C7999D] Cc: Subject: RE: Ufs dead-locks on freebsd 6.2 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 May 2007 22:43:36 -0000 Okay, I let memtest run for a full day and there has been no memory errors. What do I do next? Just to be on the safe side I'll fsck all of my fs's and try to reproduce the problem again. I also don't know what zonelimit is, I see this on similarily configured machines but running 5.4. I know it's related to network as I periodically get network connections to work i.e. ssh, ftp (both server and client side) but eventually the box will deadlock. Should I start a different thread on this? Happens about once every 30 days on two server although I havn't checked the exact timing. -----Original Message----- From: owner-freebsd-fs@freebsd.org [mailto:owner-freebsd-fs@freebsd.org] On Behalf Of Eric Anderson Sent: Friday, May 18, 2007 3:09 PM To: Kris Kennaway Cc: freebsd-fs@freebsd.org Subject: Re: Ufs dead-locks on freebsd 6.2 On 05/18/07 14:00, Kris Kennaway wrote: > On Thu, May 17, 2007 at 11:38:20PM -0500, Eric Anderson wrote: >> On 05/17/07 12:47, Kostik Belousov wrote: >>> On Thu, May 17, 2007 at 01:03:37PM -0400, Andrew Edwards wrote: >>>> Here it is. >>>> >>>> db> show vnode 0xccd47984 >>>> vnode 0xccd47984: tag ufs, type VDIR >>>> usecount 5135, writecount 0, refcount 5137 mountedhere 0 >>>> flags (VV_ROOT) >>>> v_object 0xcd02518c ref 0 pages 1 >>>> #0 0xc0593f0d at lockmgr+0x4ed >>>> #1 0xc06b8e0e at ffs_lock+0x76 >>>> #2 0xc0739787 at VOP_LOCK_APV+0x87 >>>> #3 0xc0601c28 at vn_lock+0xac >>>> #4 0xc05ee832 at lookup+0xde >>>> #5 0xc05ee4b2 at namei+0x39a >>>> #6 0xc05e2ab0 at unp_connect+0xf0 >>>> #7 0xc05e1a6a at uipc_connect+0x66 >>>> #8 0xc05d9992 at soconnect+0x4e >>>> #9 0xc05dec60 at kern_connect+0x74 >>>> #10 0xc05debdf at connect+0x2f >>>> #11 0xc0723e2b at syscall+0x25b >>>> #12 0xc070ee0f at Xint0x80_syscall+0x1f >>>> >>>> ino 2, on dev amrd0s1a >>> It seems to be the sort of things that cannot happen. VOP_LOCK()=20 >>> returned 0, but vnode was not really locked. >>> >>> Although claiming that kernel code cannot have such bug is too=20 >>> optimistic, I would first make sure that: >>> 1. You checked the memory of the machine. >>> 2. Your kernel is built from pristine sources. >> >> This looks precisely like a lock I was seeing on one of my NFS servers.=20 >> Only one of the filesystems would cause it, but it was the same one=20 >> each time, not necessarily under any kind of load. Things like=20 >> mountd would get wedged in state 'ufs', and other things would get=20 >> stuck in one of the lock states (I can't recall). >=20 > ...so you cannot conclude that it looks "precisely like" this case. >=20 > Please, don't confuse bug reports by this kind of claim unless you=20 > have made a detailed comparison of the debugging traces to yours. Understood - my mistake. Eric _______________________________________________ freebsd-fs@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-performance@FreeBSD.ORG Sat May 19 04:35:27 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3068116A400; Sat, 19 May 2007 04:35:27 +0000 (UTC) (envelope-from aedwards@sandvine.com) Received: from gw.sandvine.com (gw.sandvine.com [199.243.201.138]) by mx1.freebsd.org (Postfix) with ESMTP id D196813C459; Sat, 19 May 2007 04:35:26 +0000 (UTC) (envelope-from aedwards@sandvine.com) Received: from exchange-2.sandvine.com ([192.168.16.12]) by gw.sandvine.com with Microsoft SMTPSVC(6.0.3790.1830); Sat, 19 May 2007 00:34:25 -0400 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable Date: Sat, 19 May 2007 00:34:25 -0400 Message-ID: <5230D3C40B842D4F9FB3CD368021BEF72F092A@exchange-2.sandvine.com> In-Reply-To: <5230D3C40B842D4F9FB3CD368021BEF72F0926@exchange-2.sandvine.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Ufs dead-locks on freebsd 6.2 Thread-Index: AceZgA8XJQM9a6noQX+h86ioyzHh9wAHOkwgAAtyb/A= From: "Andrew Edwards" To: , X-OriginalArrivalTime: 19 May 2007 04:34:25.0729 (UTC) FILETIME=[FB085B10:01C799CE] Cc: Subject: RE: Ufs dead-locks on freebsd 6.2 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 May 2007 04:35:27 -0000 Fsck didn't help but below is a list of processes that were stuck in disk. Also, one potential problem I've hit is I have mrtg scripts that get launched from cron every min. MRTG is supposed to have a locking mechanism to prevent the same script from running at the same time but I suspect since the filesystem was unaccessible the cron jobs just kept piling up and piling up until the system would eventually crash. I caught it when the load avg. was at 620 and killed all the cron's I could. That brought the load avg. down to under 1 however system is still taking up 30% of the processor time and the disks are basically idle. I can still do an ls -l on the root of all my mounted ufs and nfs filesystems but on one it's taking a considerable amount longer than the rest. This particular rsync that I was running is copying into the /d2 fs. The system is still running and I can make tpc connections and somethings I have running from inetd work but ssh stops responding right away and I can't logon via the console. So, I've captured a core dump of the system and rebooted so that I could use it again. Are there any suggestion as to what to do next? I'm debaiting installing an adaptec raid and rebuilding the system to see if I get the same problem, my worry is that it's the intel raid drivers that are causing this problem and I have 4 other systems with the same card. PID TT STAT TIME COMMAND 2 ?? DL 0:04.86 [g_event] 3 ?? DL 2:05.90 [g_up] 4 ?? DL 1:07.95 [g_down] 5 ?? DL 0:00.00 [xpt_thrd] 6 ?? DL 0:00.00 [kqueue taskq] 7 ?? DL 0:00.00 [thread taskq] 8 ?? DL 0:06.96 [pagedaemon] 9 ?? DL 0:00.00 [vmdaemon] 15 ?? DL 0:22.28 [yarrow] 24 ?? DL 0:00.01 [usb0] 25 ?? DL 0:00.00 [usbtask] 27 ?? DL 0:00.01 [usb1] 29 ?? DL 0:00.01 [usb2] 36 ?? DL 1:28.73 [pagezero] 37 ?? DL 0:08.76 [bufdaemon] 38 ?? DL 0:00.54 [vnlru] 39 ?? DL 1:08.12 [syncer] 40 ?? DL 0:04.00 [softdepflush] 41 ?? DL 0:11.05 [schedcpu] 27182 ?? Ds 0:05.75 /usr/sbin/syslogd -l /var/run/log -l /var/named/var/run/log -b 127.0.0.1 -a 10.128.0.0/10 27471 ?? Is 0:01.10 /usr/local/bin/postmaster -D /usr/local/pgsql/data (postgres) 27594 ?? Is 0:00.04 /usr/libexec/ftpd -m -D -l -l 27602 ?? DL 0:00.28 [smbiod1] 96581 ?? D 0:00.00 cron: running job (cron) 96582 ?? D 0:00.00 cron: running job (cron) 96583 ?? D 0:00.00 cron: running job (cron) 96585 ?? D 0:00.00 cron: running job (cron) 96586 ?? D 0:00.00 cron: running job (cron) 96587 ?? D 0:00.00 cron: running job (cron) 96588 ?? D 0:00.00 cron: running job (cron) 96589 ?? D 0:00.00 cron: running job (cron) 96590 ?? D 0:00.00 cron: running job (cron) 96591 ?? D 0:00.00 cron: running job (cron) 96592 ?? D 0:00.00 cron: running job (cron) 96593 ?? D 0:00.00 cron: running job (cron) 96594 ?? D 0:00.00 cron: running job (cron) 96607 ?? D 0:00.00 cron: running job (cron) 96608 ?? D 0:00.00 cron: running job (cron) 96609 ?? D 0:00.00 cron: running job (cron) 96610 ?? D 0:00.00 cron: running job (cron) 96611 ?? D 0:00.00 cron: running job (cron) 96612 ?? D 0:00.00 cron: running job (cron) 96613 ?? D 0:00.00 cron: running job (cron) 96614 ?? D 0:00.00 cron: running job (cron) 96615 ?? D 0:00.00 cron: running job (cron) 96616 ?? D 0:00.00 cron: running job (cron) 96617 ?? D 0:00.00 cron: running job (cron) 96631 ?? D 0:00.00 cron: running job (cron) 96632 ?? D 0:00.00 cron: running job (cron) 96633 ?? D 0:00.00 cron: running job (cron) 96634 ?? D 0:00.00 cron: running job (cron) 96635 ?? D 0:00.00 cron: running job (cron) 96636 ?? D 0:00.00 cron: running job (cron) 96637 ?? D 0:00.00 cron: running job (cron) 96638 ?? D 0:00.00 cron: running job (cron) 96639 ?? D 0:00.00 cron: running job (cron) 96642 ?? D 0:00.00 cron: running job (cron) 96650 ?? D 0:00.00 cron: running job (cron) 29393 p0 D+ 22:04.58 /usr/local/bin/rsync real 0m0.012s user 0m0.000s sys 0m0.010s / real 0m0.019s user 0m0.000s sys 0m0.016s /var real 0m0.028s user 0m0.008s sys 0m0.018s /diskless real 0m0.017s user 0m0.008s sys 0m0.007s /usr real 0m0.016s user 0m0.000s sys 0m0.015s /d2 real 0m0.024s user 0m0.000s sys 0m0.023s /exports/home real 0m2.559s user 0m0.216s sys 0m2.307s -----Original Message----- From: owner-freebsd-fs@freebsd.org [mailto:owner-freebsd-fs@freebsd.org] On Behalf Of Andrew Edwards Sent: Friday, May 18, 2007 6:44 PM To: freebsd-fs@freebsd.org; freebsd-performance@freebsd.org Subject: RE: Ufs dead-locks on freebsd 6.2 Okay, I let memtest run for a full day and there has been no memory errors. What do I do next? Just to be on the safe side I'll fsck all of my fs's and try to reproduce the problem again. I also don't know what zonelimit is, I see this on similarily configured machines but running 5.4. I know it's related to network as I periodically get network connections to work i.e. ssh, ftp (both server and client side) but eventually the box will deadlock. Should I start a different thread on this? Happens about once every 30 days on two server although I havn't checked the exact timing. -----Original Message----- From: owner-freebsd-fs@freebsd.org [mailto:owner-freebsd-fs@freebsd.org] On Behalf Of Eric Anderson Sent: Friday, May 18, 2007 3:09 PM To: Kris Kennaway Cc: freebsd-fs@freebsd.org Subject: Re: Ufs dead-locks on freebsd 6.2 On 05/18/07 14:00, Kris Kennaway wrote: > On Thu, May 17, 2007 at 11:38:20PM -0500, Eric Anderson wrote: >> On 05/17/07 12:47, Kostik Belousov wrote: >>> On Thu, May 17, 2007 at 01:03:37PM -0400, Andrew Edwards wrote: >>>> Here it is. >>>> >>>> db> show vnode 0xccd47984 >>>> vnode 0xccd47984: tag ufs, type VDIR >>>> usecount 5135, writecount 0, refcount 5137 mountedhere 0 >>>> flags (VV_ROOT) >>>> v_object 0xcd02518c ref 0 pages 1 >>>> #0 0xc0593f0d at lockmgr+0x4ed >>>> #1 0xc06b8e0e at ffs_lock+0x76 >>>> #2 0xc0739787 at VOP_LOCK_APV+0x87 >>>> #3 0xc0601c28 at vn_lock+0xac >>>> #4 0xc05ee832 at lookup+0xde >>>> #5 0xc05ee4b2 at namei+0x39a >>>> #6 0xc05e2ab0 at unp_connect+0xf0 >>>> #7 0xc05e1a6a at uipc_connect+0x66 >>>> #8 0xc05d9992 at soconnect+0x4e >>>> #9 0xc05dec60 at kern_connect+0x74 >>>> #10 0xc05debdf at connect+0x2f >>>> #11 0xc0723e2b at syscall+0x25b >>>> #12 0xc070ee0f at Xint0x80_syscall+0x1f >>>> >>>> ino 2, on dev amrd0s1a >>> It seems to be the sort of things that cannot happen. VOP_LOCK()=20 >>> returned 0, but vnode was not really locked. >>> >>> Although claiming that kernel code cannot have such bug is too=20 >>> optimistic, I would first make sure that: >>> 1. You checked the memory of the machine. >>> 2. Your kernel is built from pristine sources. >> >> This looks precisely like a lock I was seeing on one of my NFS servers.=20 >> Only one of the filesystems would cause it, but it was the same one=20 >> each time, not necessarily under any kind of load. Things like=20 >> mountd would get wedged in state 'ufs', and other things would get=20 >> stuck in one of the lock states (I can't recall). >=20 > ...so you cannot conclude that it looks "precisely like" this case. >=20 > Please, don't confuse bug reports by this kind of claim unless you=20 > have made a detailed comparison of the debugging traces to yours. Understood - my mistake. Eric _______________________________________________ freebsd-fs@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" _______________________________________________ freebsd-fs@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-performance@FreeBSD.ORG Fri May 18 23:31:04 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 31AD816A402 for ; Fri, 18 May 2007 23:31:04 +0000 (UTC) (envelope-from mjacob@freebsd.org) Received: from ns1.feral.com (ns1.feral.com [192.67.166.1]) by mx1.freebsd.org (Postfix) with ESMTP id E9B8913C487 for ; Fri, 18 May 2007 23:31:03 +0000 (UTC) (envelope-from mjacob@freebsd.org) Received: from ns1.feral.com (localhost [127.0.0.1]) by ns1.feral.com (8.14.1/8.14.1) with ESMTP id l4INAjv9081246; Fri, 18 May 2007 16:10:53 -0700 (PDT) (envelope-from mjacob@freebsd.org) Received: from localhost (mjacob@localhost) by ns1.feral.com (8.14.1/8.14.1/Submit) with ESMTP id l4INAjAp081243; Fri, 18 May 2007 16:10:45 -0700 (PDT) (envelope-from mjacob@freebsd.org) X-Authentication-Warning: ns1.feral.com: mjacob owned process doing -bs Date: Fri, 18 May 2007 16:10:45 -0700 (PDT) From: mjacob@freebsd.org To: freebsd-scsi@freebsd.org Message-ID: <20070518153412.C75640@ns1.feral.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Mailman-Approved-At: Sat, 19 May 2007 11:41:34 +0000 Cc: freebsd-performance@freebsd.org Subject: Dell/Perc5 raid/MPT SAS Integrated Raid Write Performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: mjacob@freebsd.org List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 May 2007 23:31:04 -0000 A lot of users have seen some very poor write performance on the MPT (LSI-Logic) driven SAS/SATA controllers, particularly those that function in Integrated Raid (mirroring) mode. Some of the reported performance issues are pretty clearly single spindle small transfer IOPS issues. For example, directory intensive and small file operations like mail server applications can do very poorly on single spindle SATA drives that are connected via a SAS channel that doesn't enable write cacheing on the SATA drive (i.e., does not flow through WCE for SCSI emulation, as the LSI-Logic *apparently* does not). Benchmarks like Postmark show pretty amazing differences when run on a PATA or native SATA based drive (1000s of ops/second) and on single Fibre Channel or SCSI drives (100s of ops/second) and can be even worse for SATA drives on a SAS controller. In these cases, there isn't much to be done- the h/w being picked doesn't match the application. However, other users have reported things which are *clearly* bad performance issues. In these cases users have reported sequential write speed to be a small fraction of read speed. That is, a single threaded read of a 10GB file will get spindle rotational speed magnitude for the disk in question (~40MB/s) but will only write at around ~6MB/s. This is clearly broken and wrong. Since I don't actually have a *lot* of MPT h/w and none that shows this write performance problem could folks do me a favor and at the next reboot get into the LSI-Logic BIOS utilities and find me all the firmware revision numbers? This might help me nail down some differences to go talk to LSI-Logic about. The overriding LSI-Logic BIOS revision is of interest, but also any of the firmware revision numbers. For example, the loaned Sunfire 4100 I have has 6.0.2.0 for the BIOS revision, but 1.04.00.00-IR for the firmware- and this system, which has two integral SAS 2.5" drives, writes at 50MB/s with them set up as an integrated mirror. -matt From owner-freebsd-performance@FreeBSD.ORG Sat May 19 01:49:00 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4EE4D16A402; Sat, 19 May 2007 01:49:00 +0000 (UTC) (envelope-from Johannes.Kruger@nokia.com) Received: from mgw-ext14.nokia.com (smtp.nokia.com [131.228.20.173]) by mx1.freebsd.org (Postfix) with ESMTP id C90A513C448; Sat, 19 May 2007 01:48:59 +0000 (UTC) (envelope-from Johannes.Kruger@nokia.com) Received: from esebh105.NOE.Nokia.com (esebh105.ntc.nokia.com [172.21.138.211]) by mgw-ext14.nokia.com (Switch-3.2.5/Switch-3.2.5) with ESMTP id l4J1PEgo023489; Sat, 19 May 2007 04:25:37 +0300 Received: from daebh101.NOE.Nokia.com ([10.241.35.111]) by esebh105.NOE.Nokia.com with Microsoft SMTPSVC(6.0.3790.1830); Sat, 19 May 2007 04:25:20 +0300 Received: from daebe102.NOE.Nokia.com ([10.241.35.115]) by daebh101.NOE.Nokia.com with Microsoft SMTPSVC(6.0.3790.1830); Fri, 18 May 2007 20:25:17 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Fri, 18 May 2007 20:25:16 -0500 Message-ID: In-Reply-To: <20070518153412.C75640@ns1.feral.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Dell/Perc5 raid/MPT SAS Integrated Raid Write Performance Thread-Index: AceZoeGVA9x/aURGQ/SNXOxe/7u4+wAECAIQ References: <20070518153412.C75640@ns1.feral.com> From: To: , X-OriginalArrivalTime: 19 May 2007 01:25:17.0736 (UTC) FILETIME=[8F1A0A80:01C799B4] X-Nokia-AV: Clean X-Mailman-Approved-At: Sat, 19 May 2007 11:41:34 +0000 Cc: freebsd-performance@freebsd.org Subject: RE: Dell/Perc5 raid/MPT SAS Integrated Raid Write Performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 May 2007 01:49:00 -0000 I am using a LSI-1064E 2 boards, each with firmware images I compiled from a base firmware image concatenated with an NVDATA file that you configure yourself. I set up the ports as narrow and attach 2 SATA drives in RAID-1 configuration and I get 28-30 MBytes/sec. I compare this to the same drives running on an SATA capable IDE controller and the performance is the same. So RAID-1 does not seem to impact it ... if the volume is in sync. I have the sync rate set to about 2% default in the firmware config (NVDATA) file, and when the volume is out of sync, the performance drop to about 15 MBytes/second and jump back t full speed once in sync. Attaching SAS 2.5 inch drives I get 55-60 MBytes/sec. Using firmware revision 01.18.00.00 with MPI Version 1.5.13.0 using NVDATA file version 25. and on the other board I am experimenting with a beta phase9 (whatever that means) firmware, compiled with MPI version 1.5.14.0 and NVDATA file version 28. On both setups I removed the LSI BIOS part (mptsas.rom) after a while, since I do not need it. I configure the controller from the OS itself. This all with Matthew's driver ported to an FreeBSD 2.1.0 variant, including the NEW CAM layer. The firmware for 01.18.00.00 is freely available from their site. =20 http://www.lsi.com/storage_home/products_home/standard_product_ics/sas_i cs/lsisas1064e/index.html Use the host bus adapter firmware if you want to try it out: http://www.lsi.com/cm/License.do?url=3D/files/support/ssp/fusionmpt/sas/b= i os_firmware/SAS3041E-R_IR_10-30-06.zip&prodName=3DLSISAS3041E-R&subType=3D= Fi rmware&locale=3DEN Johan -----Original Message----- From: owner-freebsd-scsi@freebsd.org [mailto:owner-freebsd-scsi@freebsd.org]=20 Sent: Friday, May 18, 2007 7:11 PM To: freebsd-scsi@freebsd.org Cc: freebsd-performance@freebsd.org Subject: Dell/Perc5 raid/MPT SAS Integrated Raid Write Performance A lot of users have seen some very poor write performance on the MPT=20 (LSI-Logic) driven SAS/SATA controllers, particularly those that=20 function in Integrated Raid (mirroring) mode. Some of the reported performance issues are pretty clearly single=20 spindle small transfer IOPS issues. For example, directory intensive and small file operations like mail server applications can do very poorly=20 on single spindle SATA drives that are connected via a SAS channel that=20 doesn't enable write cacheing on the SATA drive (i.e., does not flow=20 through WCE for SCSI emulation, as the LSI-Logic *apparently* does not). Benchmarks like Postmark show pretty amazing differences when run on a=20 PATA or native SATA based drive (1000s of ops/second) and on single=20 Fibre Channel or SCSI drives (100s of ops/second) and can be even worse=20 for SATA drives on a SAS controller. In these cases, there isn't much to be done- the h/w being picked=20 doesn't match the application. However, other users have reported things which are *clearly* bad=20 performance issues. In these cases users have reported sequential write=20 speed to be a small fraction of read speed. That is, a single threaded=20 read of a 10GB file will get spindle rotational speed magnitude for the=20 disk in question (~40MB/s) but will only write at around ~6MB/s. This is clearly broken and wrong. Since I don't actually have a *lot* of MPT h/w and none that shows this=20 write performance problem could folks do me a favor and at the next=20 reboot get into the LSI-Logic BIOS utilities and find me all the=20 firmware revision numbers? This might help me nail down some differences to go talk to LSI-Logic about. The overriding LSI-Logic BIOS revision is of interest, but also any of=20 the firmware revision numbers. For example, the loaned Sunfire 4100 I=20 have has 6.0.2.0 for the BIOS revision, but 1.04.00.00-IR for the=20 firmware- and this system, which has two integral SAS 2.5" drives,=20 writes at 50MB/s with them set up as an integrated mirror. -matt _______________________________________________ freebsd-scsi@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-scsi To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org"