From owner-freebsd-scsi@FreeBSD.ORG Wed Jun 26 11:20:01 2013 Return-Path: Delivered-To: freebsd-scsi@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id AC354475 for ; Wed, 26 Jun 2013 11:20:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 9D6091E5E for ; Wed, 26 Jun 2013 11:20:01 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r5QBK1rK098466 for ; Wed, 26 Jun 2013 11:20:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r5QBK0nr098465; Wed, 26 Jun 2013 11:20:00 GMT (envelope-from gnats) Date: Wed, 26 Jun 2013 11:20:00 GMT Message-Id: <201306261120.r5QBK0nr098465@freefall.freebsd.org> To: freebsd-scsi@FreeBSD.org Cc: From: Philipp Maechler Subject: Re: kern/179932: [ciss] ciss i/o stall problem with HP Bl Gen8 (and HP Bl Gen7 Storage Blade) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Philipp Maechler List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Jun 2013 11:20:01 -0000 The following reply was made to PR kern/179932; it has been noted by GNATS. From: Philipp Maechler To: sbruno@freebsd.org Cc: bug-followup@FreeBSD.org, Sean Bruno Subject: Re: kern/179932: [ciss] ciss i/o stall problem with HP Bl Gen8 (and HP Bl Gen7 Storage Blade) Date: Wed, 26 Jun 2013 13:12:40 +0200 This is a multi-part message in MIME format. --------------070005030706090006050108 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit hi Sean At the moment we are running the serial console at 9600 baud; we have to tune this to 115200 baud and also then an alltrace will take "forever" because of hundreds of apache threads; because this machines are productive and non-redundant we have to figure out a procedure how we can handle to do this long-running alltrace without interrupting the services for such a time. As an attachement, I can provide you an old debugger session with a (part) of a ps and some backtraces. It is from one of our HP Gen7-StorageBlade-Systems running on 9.1-RELEASE or higher, hanging earlier this year with the same problem. If you find there enough information to say "alltrace is not needed", we'd be happy. But until then we will search for possibilities to do such an alltrace... The break possibility we already enabled by setting the sysctl debug.kdb.break_to_debugger=1 As soon as you send us the mentioned patch we will begin to test it and introduce it to the affected systems. Philipp p.s. please note; even there are a lot of processes in some zfs states, we also tried to do dd to some raw partitions on disk and were not successful, so we think it's not a zfs problem. In gstat we saw the wait queue's stall at specific values and nothing going on anymore. --------------070005030706090006050108 Content-Type: text/plain; charset=UTF-8; name="20130626_olddebugsession.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="20130626_olddebugsession.txt" Notes: * the ps is not complete, we think there is a part missing; but it gives you an overview * are you looking for specific processes or threads? please tell us! hpiLO-> vsp Virtual Serial Port Active: COM1 Starting virtual serial port. Press 'ESC (' to return to the CLI Session. Tracing pid 12 tid 100043 td 0xfffffe0019ce0000 kdb_enter() at kdb_enter+0x3b kdb_break() at kdb_break+0x27 scgetc() at scgetc+0x361 sckbdevent() at sckbdevent+0xed kbdmux_intr() at kbdmux_intr+0x43 kbdmux_kbd_intr() at kbdmux_kbd_intr+0x20 taskqueue_run_locked() at taskqueue_run_locked+0x85 taskqueue_run() at taskqueue_run+0x3a intr_event_execute_handlers() at intr_event_execute_handlers+0x104 ithread_loop() at ithread_loop+0xa6 fork_exit() at fork_exit+0x11f fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80003ffcf0, rbp = 0 --- db> ps pid ppid pgrp uid state wmesg wchan cmd 87125 87123 87125 0 Ds+ zfs 0xfffffe05c0c80db8 bash 87123 1683 87123 0 Ss select 0xfffffe0a5d29a8c0 sshd 87122 87120 87122 0 Ds+ zfs 0xfffffe05c0c80db8 bash 87120 1683 87120 0 Ss select 0xfffffe0a5d4603c0 sshd 86929 86927 86929 0 Ds zio->io_ 0xfffffe02a13b4d70 atrun 86928 86926 86928 2 Ds zio->io_ 0xfffffe08a73d5a00 sh 86927 1687 1687 0 S piperd 0xfffffe02f036a5b0 cron 86926 1687 1687 0 S piperd 0xfffffe050c0cb5b0 cron 86738 86736 86738 0 Ds+ zio->io_ 0xfffffe03f1d81690 bash 86736 1683 86736 0 Ss select 0xfffffe0b7f10b7c0 sshd 86729 86728 86729 181 Ds zfs 0xfffffe026a16d458 python 86728 86726 86726 181 S select 0xfffffe02996c7a40 sshd 86726 1683 86726 0 Ss sbwait 0xfffffe03a1d6ee8c sshd 86725 86724 86725 181 Ds zfs 0xfffffe026a16d458 python 86724 86722 86722 181 S select 0xfffffe0aa6b7e840 sshd 86722 1683 86722 0 Ss sbwait 0xfffffe0c4f90ae8c sshd 86721 86719 86718 181 S piperd 0xfffffe064c9d8b60 wc 86720 86719 86718 181 D spa_name 0xffffffff817f7d00 zpool 86719 86718 86718 181 S wait 0xfffffe0e9fe53000 bash 86718 86717 86718 181 Ss piperd 0xfffffe0604ecd2d8 bash 86717 86715 86715 181 S select 0xfffffe03761605c0 sshd 86715 1683 86715 0 Ss sbwait 0xfffffe0d3d8a83ec sshd 86535 86534 86535 181 Ds zfs 0xfffffe026a16d458 python 86534 86528 86528 181 S select 0xfffffe081b6a1040 sshd 86533 86532 86533 181 Ds zfs 0xfffffe026a16d458 python 86532 86526 86526 181 S select 0xfffffe02815b85c0 sshd 86531 86530 86531 181 Ds zfs 0xfffffe026a16d458 python 86530 86524 86524 181 S select 0xfffffe048360e7c0 sshd 86528 1683 86528 0 Ss sbwait 0xfffffe035cbb0e8c sshd 86526 1683 86526 0 Ss sbwait 0xfffffe006acb03ec sshd 86524 1683 86524 0 Ss sbwait 0xfffffe0030653694 sshd 86328 86327 86328 181 Ds zfs 0xfffffe026a16d458 python 86327 86323 86323 181 S select 0xfffffe0a8855a4c0 sshd 86325 86324 86325 181 Ds zfs 0xfffffe026a16d458 python 86324 86315 86315 181 S select 0xfffffe09ed0b7cc0 sshd 86323 1683 86323 0 Ss sbwait 0xfffffe07afe3d144 sshd 86322 86321 86322 181 Ds zfs 0xfffffe026a16d458 python 86321 86311 86311 181 S select 0xfffffe0e3265b840 sshd 86320 86319 86320 181 Ds zfs 0xfffffe026a16d458 python 86319 86304 86304 181 S select 0xfffffe088678d2c0 sshd 86318 86317 86318 181 Ds zfs 0xfffffe026a16d458 python 86317 86303 86303 181 S select 0xfffffe02418f3640 sshd 86315 1683 86315 0 Ss sbwait 0xfffffe02a72e23ec sshd 86313 86312 86313 181 Ds zfs 0xfffffe026a16d458 python 86312 86297 86297 181 S select 0xfffffe009dcc10c0 sshd 86311 1683 86311 0 Ss sbwait 0xfffffe013d385144 sshd 86310 86309 86310 181 Ds zfs 0xfffffe026a16d458 python 86309 86301 86301 181 S select 0xfffffe01fd72d3c0 sshd 86308 86307 86308 181 Ds zfs 0xfffffe026a16d458 python 86307 86298 86298 181 S select 0xfffffe0483fa1a40 sshd 86304 1683 86304 0 Ss sbwait 0xfffffe02544f5144 sshd 86303 1683 86303 0 Ss sbwait 0xfffffe0ddf800694 sshd 86301 1683 86301 0 Ss sbwait 0xfffffe0b4ef6b144 sshd 86298 1683 86298 0 Ss sbwait 0xfffffe0beda78e8c sshd 86297 1683 86297 0 Ss sbwait 0xfffffe0a36823144 sshd 86292 86291 86292 181 Ds zfs 0xfffffe026a16d458 python 86291 86285 86285 181 S select 0xfffffe0d0524b640 sshd 86290 86289 86290 181 Ds zfs 0xfffffe026a16d458 python 86289 86283 86283 181 S select 0xfffffe0f480e1cc0 sshd 86288 86287 86288 181 Ds zio->io_ 0xfffffe039861f690 python 86287 86281 86281 181 S select 0xfffffe04cf9c80c0 sshd 86285 1683 86285 0 Ss sbwait 0xfffffe03fe528694 sshd 86283 1683 86283 0 Ss sbwait 0xfffffe0311e9593c sshd 86281 1683 86281 0 Ss sbwait 0xfffffe0d3d8a8be4 sshd 86279 1668 1668 0 D zio->io_ 0xfffffe028063ad70 exim-4.80.1-0 86091 86089 86088 181 S piperd 0xfffffe06da459b60 wc 86090 86089 86088 181 D spa_name 0xffffffff817f7d00 zpool 86089 86088 86088 181 S wait 0xfffffe0a88c16000 bash 86088 86087 86088 181 Ss piperd 0xfffffe050c0cb888 bash 86087 86085 86085 181 S select 0xfffffe0bb51f8cc0 sshd 86085 1683 86085 0 Ss sbwait 0xfffffe02ab86e3ec sshd 86080 86078 83614 0 S piperd 0xfffffe0e58ef85b0 grep 86079 86078 83614 0 D spa_name 0xffffffff817f7d00 zfs 86078 83628 83614 0 S wait 0xfffffe0a99074000 bash 85728 83566 49671 0 D tx->tx_s 0xfffffe00209eee20 zfs 85332 85321 85332 0 Ss+ ttyin 0xfffffe0524a144a8 bash 85321 1683 85321 0 Ss select 0xfffffe0b122beb40 sshd 83628 83621 83614 0 S piperd 0xfffffe013ff055b0 bash 83621 83614 83614 0 S wait 0xfffffe0ef81f74a0 bash 83614 83610 83614 0 Ss piperd 0xfffffe0ad77c7000 sh 83610 1687 1687 0 S piperd 0xfffffe03d14ffb60 cron 83566 49854 49671 0 S wait 0xfffffe0eae6ab000 bash 49854 49685 49671 0 S wait 0xfffffe0d80e8b4a0 bash 49685 49671 49671 0 S wait 0xfffffe0caf48d940 bash 49671 49665 49671 0 Ss piperd 0xfffffe0cd7726b60 sh 49665 1687 1687 0 S piperd 0xfffffe0b5d9482d8 cron 34311 34306 34311 0 Ss+ ttyin 0xfffffe002c9388a8 bash 34306 1683 34306 0 Ss select 0xfffffe079c3c0d40 sshd 24912 24564 24912 0 Ss+ ttyin 0xfffffe03e79d2ca8 bash 24564 1683 24564 0 Ss select 0xfffffe0f6a802540 sshd 64399 63540 34618 88 S (threaded) mysqld 105150 S uwait 0xfffffe022c939900 mysqld 105149 S uwait 0xfffffe0603a95280 mysqld 105148 S uwait 0xfffffe0e6d323300 mysqld 105147 S uwait 0xfffffe0f11b16680 mysqld 105146 S sigwait 0xfffffe086602d000 mysqld 105145 S uwait 0xfffffe03c8eff880 mysqld 105144 S uwait 0xfffffe07370e5100 mysqld 105143 S uwait 0xfffffe0ef7fa9b00 mysqld 105142 S uwait 0xfffffe02d415ba00 mysqld 105140 S uwait 0xfffffe0e58e9a100 mysqld 105139 S uwait 0xfffffe06c9cfd000 mysqld 105138 S uwait 0xfffffe01f5eb7e00 mysqld 105137 S uwait 0xfffffe0de3773100 mysqld 105136 S uwait 0xfffffe077c241900 mysqld 105135 S uwait 0xfffffe03dae38800 mysqld 105134 S uwait 0xfffffe09069d7400 mysqld 105133 S uwait 0xfffffe045cc16980 mysqld 105132 S uwait 0xfffffe083cacc000 mysqld 105131 S uwait 0xfffffe02ece6fa80 mysqld 102539 S select 0xfffffe099d9a6040 mysqld 63540 1 34618 0 S wait 0xfffffe004fa1d940 sh 47863 1 47863 0 Ss+ ttyin 0xfffffe001e47e8a8 getty 11143 1 11143 88 DsJ (threaded) mysqld 105622 S uwait 0xfffffe028377db00 mysqld 100617 S uwait 0xfffffe0d3a2d7080 mysqld 100557 S sbwait 0xfffffe00ae8d5144 mysqld 105432 S sbwait 0xfffffe0466736694 mysqld 105431 S sbwait 0xfffffe05950073ec mysqld 105401 S sbwait 0xfffffe069c2dbe8c mysqld 105365 D zio->io_ 0xfffffe0cbafe1a00 mysqld 104814 S uwait 0xfffffe08b0699180 mysqld 104813 S uwait 0xfffffe0b4ee56900 mysqld 104812 S sbwait 0xfffffe017e6fd144 mysqld 104811 S sbwait 0xfffffe094050e694 mysqld 104810 S sbwait 0xfffffe004e27abe4 mysqld 104773 S sbwait 0xfffffe06b343e144 mysqld 104772 S sbwait 0xfffffe059500be8c mysqld 104771 S sbwait 0xfffffe06a97ba93c mysqld 104766 S sbwait 0xfffffe059500ae8c mysqld 104765 S uwait 0xfffffe052bc88c00 mysqld 104764 S sbwait 0xfffffe02ad61a694 mysqld 104763 S sbwait 0xfffffe01fab54694 mysqld 104762 S sbwait 0xfffffe0912318694 mysqld 104759 S sbwait 0xfffffe06b343ebe4 mysqld 104756 S uwait 0xfffffe084b1e4400 mysqld 104738 S umtxn 0xfffffe07bbb1b700 mysqld 103653 S uwait 0xfffffe0393be2280 mysqld 103643 S uwait 0xfffffe0b4e03a600 mysqld 103633 S sbwait 0xfffffe04b04b693c mysqld 103534 D zio->io_ 0xfffffe0d5d939690 mysqld 103217 S sbwait 0xfffffe0e588afe8c mysqld 103216 S sbwait 0xfffffe077e407be4 mysqld 103128 S uwait 0xfffffe043b164c00 mysqld 103126 S uwait 0xfffffe0426a71b00 mysqld 103125 S sbwait 0xfffffe006a0f63ec mysqld 103105 S sbwait 0xfffffe02ad61a93c mysqld 102497 D zio->io_ 0xfffffe0b5de39320 mysqld 102496 S sbwait 0xfffffe006acb0e8c mysqld 102487 S sbwait 0xfffffe077e4063ec mysqld 102475 S sbwait 0xfffffe040f987e8c mysqld 102427 S sbwait 0xfffffe00a35643ec mysqld 102394 S uwait 0xfffffe09ed0c0e00 mysqld 102360 S sbwait 0xfffffe0beda7b694 mysqld 102357 S sbwait 0xfffffe0e6673ee8c mysqld 102356 S sigwait 0xfffffe0b7e213000 mysqld 102355 D zio->io_ 0xfffffe03c317cd70 mysqld 102354 S uwait 0xfffffe0bad460880 mysqld 102353 D zio->io_ 0xfffffe0d09658690 mysqld 102352 S uwait 0xfffffe0bad379b00 mysqld 102350 S uwait 0xfffffe079c3b5680 mysqld 102349 S uwait 0xfffffe09ed530400 mysqld 102348 S uwait 0xfffffe09edae6100 mysqld 102347 S uwait 0xfffffe0c4f119580 mysqld 102346 S uwait 0xfffffe0de1a35780 mysqld 102345 S uwait 0xfffffe0c4f11a900 mysqld 102344 S uwait 0xfffffe0c4f116c00 mysqld 102343 S uwait 0xfffffe079c2f1480 mysqld 102342 S uwait 0xfffffe09ed935d80 mysqld 102341 S uwait 0xfffffe079c1e6880 mysqld 102299 S select 0xfffffe0b1cec4140 mysqld 4970 4968 4970 0 Ss+ ttyin 0xfffffe02b83830a8 bash 4968 1683 4968 0 Ss select 0xfffffe0bad48b840 sshd 2777 2183 2777 0 S+ kqread 0xfffffe027cd92900 tail 2183 2181 2183 0 Ss+ wait 0xfffffe002c5a1940 bash 2181 1683 2181 0 Ss select 0xfffffe01e42f4140 sshd 2180 1 1 0 S ttydcd 0xfffffe001e4758e8 getty 2178 1 2178 0 Ss+ ttyin 0xfffffe001e47bca8 getty 2177 1 2177 0 Ss+ ttyin 0xfffffe001e47b4a8 getty 2176 1 2176 0 Ss+ ttyin 0xfffffe001e47b0a8 getty 2175 1 2175 0 Ss+ ttyin 0xfffffe001e47a8a8 getty 2174 1 2174 0 Ss+ ttyin 0xfffffe001e47a4a8 getty 2173 1 2173 0 Ss+ ttyin 0xfffffe001e4768a8 getty 2172 1 2172 0 Ss+ ttyin 0xfffffe001e476ca8 getty 2171 1 2171 0 DLs+ zio->io_ 0xfffffe0325905d70 login 2129 1 2129 88 DsJ (threaded) mysqld 108446 S umtxn 0xfffffe0d99f8c380 mysqld 108436 S umtxn 0xfffffe05f7e9cb00 mysqld 108431 S umtxn 0xfffffe0a23934200 mysqld 108413 S umtxn 0xfffffe09e6cedb00 mysqld 108397 S umtxn 0xfffffe02d4458080 mysqld 108389 S umtxn 0xfffffe0ab413ea00 mysqld 108352 S umtxn 0xfffffe07612efe00 mysqld 108350 S umtxn 0xfffffe0ddf748180 mysqld 108332 S umtxn 0xfffffe0d89b09100 mysqld 108326 S umtxn 0xfffffe0254472880 mysqld 108322 S umtxn 0xfffffe09dd542300 mysqld 108318 S umtxn 0xfffffe09371be200 mysqld 108316 S umtxn 0xfffffe09371be380 mysqld 108311 S umtxn 0xfffffe0e95eff700 mysqld 108294 S umtxn 0xfffffe0a886e0680 mysqld 108263 S umtxn 0xfffffe0ee0175780 mysqld 108253 S umtxn 0xfffffe0ee0175a80 mysqld 108239 S umtxn 0xfffffe0573530a80 mysqld 108236 S umtxn 0xfffffe0573530d80 mysqld 108234 S umtxn 0xfffffe096063cc80 mysqld 108214 S umtxn 0xfffffe07afb6ac80 mysqld 108213 S umtxn 0xfffffe018c8dbc00 mysqld 108204 S umtxn 0xfffffe0ec8cffc80 mysqld 108192 S umtxn 0xfffffe0aec5fd500 mysqld 108167 S umtxn 0xfffffe020ce9a600 mysqld 108164 S umtxn 0xfffffe094f908e00 mysqld 108145 S umtxn 0xfffffe01361ccc80 mysqld 108113 S umtxn 0xfffffe03b7288200 mysqld 108104 S umtxn 0xfffffe0543900500 mysqld 108086 S umtxn 0xfffffe0575f95a80 mysqld 108077 S umtxn 0xfffffe02f9862c80 mysqld 108069 S umtxn 0xfffffe0f6b7d3480 mysqld 108055 S umtxn 0xfffffe0967e77300 mysqld 108054 S umtxn 0xfffffe0d397a8080 mysqld 108050 S umtxn 0xfffffe013f554980 mysqld 108038 S umtxn 0xfffffe01248d0780 mysqld 108035 S umtxn 0xfffffe06e288f100 mysqld 108025 S umtxn 0xfffffe0299997880 mysqld 108022 S umtxn 0xfffffe0444f7ed80 mysqld 107991 S umtxn 0xfffffe0b0769bb00 mysqld 107969 S umtxn 0xfffffe01bf108480 mysqld 107944 S umtxn 0xfffffe0a230e2b80 mysqld 107916 S umtxn 0xfffffe017e2b4c80 mysqld 107884 S umtxn 0xfffffe017e2b4a80 mysqld 107883 S umtxn 0xfffffe0418fc7480 mysqld 107871 S umtxn 0xfffffe03acb51b00 mysqld 107855 S umtxn 0xfffffe08ed201c80 mysqld 107824 S umtxn 0xfffffe0aece27d00 mysqld 107820 S umtxn 0xfffffe0bb1988c00 mysqld 107809 S umtxn 0xfffffe0de2182a00 mysqld 107794 S umtxn 0xfffffe06ad810380 mysqld 107787 S umtxn 0xfffffe0f56112100 mysqld 107781 S umtxn 0xfffffe0f56112380 mysqld 107736 S umtxn 0xfffffe05a29c6100 mysqld 107717 S umtxn 0xfffffe0c179dd980 mysqld 107716 S umtxn 0xfffffe0650e87a00 mysqld 107709 S umtxn 0xfffffe0650e87c80 mysqld 107690 S umtxn 0xfffffe0ed40b6880 mysqld 107674 S umtxn 0xfffffe02c3e7bc80 mysqld 107664 S umtxn 0xfffffe02c3e7b080 mysqld 107645 S umtxn 0xfffffe04ff252e00 mysqld 107641 S umtxn 0xfffffe01d77a3800 mysqld 107640 S umtxn 0xfffffe04bd459400 mysqld 107629 S umtxn 0xfffffe02cfeba700 mysqld 107600 S umtxn 0xfffffe03406e8980 mysqld 107598 S umtxn 0xfffffe0d6cdd5c00 mysqld 107581 S umtxn 0xfffffe03406e8880 mysqld 107580 S umtxn 0xfffffe0b1cee4d80 mysqld 107569 S umtxn 0xfffffe09e6c14580 mysqld 107567 S umtxn 0xfffffe08b2a70500 mysqld 107563 S umtxn 0xfffffe03df90e100 mysqld 107562 S umtxn 0xfffffe0cdbee6580 mysqld 107556 S umtxn 0xfffffe0bbab0d200 mysqld 107527 S umtxn 0xfffffe0fb315cd00 mysqld 107526 S umtxn 0xfffffe057338e580 mysqld 107525 S umtxn 0xfffffe0bbab0db00 mysqld 107517 S umtxn 0xfffffe01db5b7700 mysqld 107516 S umtxn 0xfffffe01db5b7900 mysqld 107507 S umtxn 0xfffffe0e7bc38c80 mysqld 107472 S umtxn 0xfffffe0e7bc38a00 mysqld 107460 S umtxn 0xfffffe02f05c9000 mysqld 107449 S umtxn 0xfffffe02cbde6d00 mysqld 107443 S umtxn 0xfffffe0e64653d80 mysqld 107422 S umtxn 0xfffffe0dc6963700 mysqld 107420 S umtxn 0xfffffe06d8f20680 mysqld 107418 S umtxn 0xfffffe02a812e480 mysqld 107417 S umtxn 0xfffffe035e4cdb00 mysqld 107403 S umtxn 0xfffffe016ca31300 mysqld 107399 S umtxn 0xfffffe0d73b0bb00 mysqld 107382 S umtxn 0xfffffe03d4208980 mysqld 107352 S umtxn 0xfffffe0eb2423d00 mysqld 107340 S umtxn 0xfffffe05c0dc6280 mysqld 107337 S umtxn 0xfffffe0b0f27c600 mysqld 107333 S umtxn 0xfffffe0a519d8700 mysqld 107312 S umtxn 0xfffffe0c5451da80 mysqld 107308 S umtxn 0xfffffe0f0df60b00 mysqld 107305 S umtxn 0xfffffe06487eb980 mysqld 107303 S umtxn 0xfffffe0a3df35980 mysqld 107299 S umtxn 0xfffffe0f58fea700 mysqld 107291 S umtxn 0xfffffe0a5f6b0300 mysqld 107278 S umtxn 0xfffffe020095be00 mysqld 107277 S umtxn 0xfffffe03783a2200 mysqld 107264 S umtxn 0xfffffe0595e24d00 mysqld 107260 S umtxn 0xfffffe0d989d6400 mysqld 107255 S umtxn 0xfffffe0c25a91380 mysqld 107254 S umtxn 0xfffffe0dfc812080 mysqld 107253 S umtxn 0xfffffe08ed8a6a80 mysqld 107251 S umtxn 0xfffffe039ee19700 mysqld 107249 S umtxn 0xfffffe039ee19980 mysqld 107221 S umtxn 0xfffffe0d807c2e00 mysqld 107215 S umtxn 0xfffffe0a4d784380 mysqld 107207 S umtxn 0xfffffe06e90da980 mysqld 107204 S umtxn 0xfffffe0cc3992780 mysqld 107200 S umtxn 0xfffffe0512754d80 mysqld 107190 S umtxn 0xfffffe0d7a47b100 mysqld 107173 S umtxn 0xfffffe03a16c5c80 mysqld 107171 S umtxn 0xfffffe0404607c00 mysqld 107170 S umtxn 0xfffffe0cc8c81980 mysqld 107162 S umtxn 0xfffffe0ad8a51580 mysqld 107157 S umtxn 0xfffffe0631cc2380 mysqld 107152 S umtxn 0xfffffe0de3bd9500 mysqld 107129 S umtxn 0xfffffe0e32d8d600 mysqld 107121 S umtxn 0xfffffe0e32d8dc00 mysqld 107120 S umtxn 0xfffffe012738b780 mysqld 107110 S umtxn 0xfffffe0e589c6880 mysqld 107107 S umtxn 0xfffffe088617f880 mysqld 107101 S umtxn 0xfffffe0937ba2c80 mysqld 107098 S umtxn 0xfffffe022e48bd80 mysqld 107085 S umtxn 0xfffffe0886039980 mysqld 107084 S umtxn 0xfffffe094f042080 mysqld 107069 S umtxn 0xfffffe03df31b180 mysqld 107045 S umtxn 0xfffffe023a19a780 mysqld 107036 S umtxn 0xfffffe07a002ed80 mysqld 107035 S umtxn 0xfffffe021bcee280 mysqld 107022 S umtxn 0xfffffe0a51abcc80 mysqld 107007 S umtxn 0xfffffe0da93f9400 mysqld 107006 S umtxn 0xfffffe0687561300 mysqld 106997 S umtxn 0xfffffe0b76c23280 mysqld 106995 S umtxn 0xfffffe0faf70b800 mysqld 106985 S umtxn 0xfffffe04ff405680 mysqld 106980 S umtxn 0xfffffe0b7f5ba680 mysqld 106973 S umtxn 0xfffffe0393be2c00 mysqld 106965 S umtxn 0xfffffe03f50fcc00 mysqld 106962 S umtxn 0xfffffe0e8034bb80 mysqld 106933 S umtxn 0xfffffe03b4bb0680 mysqld 106932 S umtxn 0xfffffe0dc1ac0680 mysqld 106916 S umtxn 0xfffffe0abcaab980 mysqld 106883 S umtxn 0xfffffe043db24300 mysqld 106876 S umtxn 0xfffffe072773be00 mysqld 106871 S umtxn 0xfffffe0e1d447100 mysqld 106863 S umtxn 0xfffffe07a052b780 mysqld 106847 S umtxn 0xfffffe0ad78da800 mysqld 106844 S umtxn 0xfffffe0445acfc00 mysqld 106842 S umtxn 0xfffffe025bba2e00 mysqld 106841 S umtxn 0xfffffe036539c280 mysqld 106810 S umtxn 0xfffffe058719b800 mysqld 106786 S umtxn 0xfffffe0e32703880 mysqld 106781 S umtxn 0xfffffe03dfed7c00 mysqld 106779 S umtxn 0xfffffe0c686f7a00 mysqld 106769 S umtxn 0xfffffe01bf889c00 mysqld 106760 S umtxn 0xfffffe07da3bb800 mysqld 106754 S umtxn 0xfffffe044e35a800 mysqld 106744 S umtxn 0xfffffe03d6e88900 mysqld 106714 S umtxn 0xfffffe075f052180 mysqld 106710 S umtxn 0xfffffe0d80cd7a80 mysqld 106692 S umtxn 0xfffffe03df96a500 mysqld 106690 S umtxn 0xfffffe04bd2fcc80 mysqld 106673 S umtxn 0xfffffe0cc75d1300 mysqld 106656 S umtxn 0xfffffe03b58d0180 mysqld 106650 S umtxn 0xfffffe0516d16480 mysqld 106647 S umtxn 0xfffffe056ea70d80 mysqld 106643 S umtxn 0xfffffe0f8a322b00 mysqld 106641 S umtxn 0xfffffe0d9834e480 mysqld 106633 S umtxn 0xfffffe02306f7900 mysqld 106604 S umtxn 0xfffffe034dc04000 mysqld 106583 S umtxn 0xfffffe0d7aad5880 mysqld 106579 S umtxn 0xfffffe0abc16a600 mysqld 106561 S umtxn 0xfffffe03d6188680 mysqld 106537 S umtxn 0xfffffe01afebe000 mysqld 106521 S umtxn 0xfffffe0360772700 mysqld 106516 S umtxn 0xfffffe025a7a4900 mysqld 106506 S umtxn 0xfffffe03423fb480 mysqld 106495 S umtxn 0xfffffe06ea2d2280 mysqld 106488 S umtxn 0xfffffe0a2f142d00 mysqld 106487 S umtxn 0xfffffe02f0a14c80 mysqld 106479 S umtxn 0xfffffe0ddf63c780 mysqld 106438 S umtxn 0xfffffe07be277100 mysqld 106433 S umtxn 0xfffffe037a37cc00 mysqld 106428 S umtxn 0xfffffe0d0eb3eb80 mysqld 106397 S umtxn 0xfffffe0d7a07c200 mysqld 106390 S umtxn 0xfffffe094f76d600 mysqld 106379 S umtxn 0xfffffe014423ab00 mysqld 106377 S umtxn 0xfffffe0721e68000 mysqld 106376 S umtxn 0xfffffe0d47d1a400 mysqld 106364 S umtxn 0xfffffe07a07d2980 mysqld 106348 S umtxn 0xfffffe0293963b00 mysqld 106311 S umtxn 0xfffffe03264fa080 mysqld 106302 S umtxn 0xfffffe02306f6780 mysqld 106274 S umtxn 0xfffffe0f60f58200 mysqld 106272 S umtxn 0xfffffe084f6f9100 mysqld 106250 S umtxn 0xfffffe09304f3300 mysqld 106242 S umtxn 0xfffffe005fa65180 mysqld 106241 S umtxn 0xfffffe009d3aea00 mysqld 106236 S umtxn 0xfffffe0fb3ffbc80 mysqld 106235 S umtxn 0xfffffe058d8a7480 mysqld 106218 S umtxn 0xfffffe0aefd36180 mysqld 106191 S umtxn 0xfffffe009d683880 mysqld 106187 S umtxn 0xfffffe024d76dc80 mysqld 106186 S umtxn 0xfffffe09bda65400 mysqld 106185 S umtxn 0xfffffe0ae627d780 mysqld 106175 S umtxn 0xfffffe0432633600 mysqld 106144 S umtxn 0xfffffe048381ae00 mysqld 106143 S umtxn 0xfffffe09bfb5c980 mysqld db> bt 86738 Tracing pid 86738 tid 102823 td 0xfffffe0cc7ecc470 sched_switch() at sched_switch+0x13d mi_switch() at mi_switch+0x196 sleepq_wait() at sleepq_wait+0x42 _cv_wait() at _cv_wait+0x121 zio_wait() at zio_wait+0x61 dmu_buf_hold_array_by_dnode() at dmu_buf_hold_array_by_dnode+0x22b dmu_buf_hold_array() at dmu_buf_hold_array+0x67 dmu_read_uio() at dmu_read_uio+0x3f zfs_freebsd_read() at zfs_freebsd_read+0x3e3 VOP_READ_APV() at VOP_READ_APV+0x44 vn_read() at vn_read+0x316 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x6c sys_read() at sys_read+0x64 amd64_syscall() at amd64_syscall+0x5ea Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x4eeafc, rsp = 0x7fffffffd888, rbp = 0x3 --- db> bt 86279 Tracing pid 86279 tid 104231 td 0xfffffe0360fd3470 sched_switch() at sched_switch+0x13d mi_switch() at mi_switch+0x196 sleepq_wait() at sleepq_wait+0x42 _cv_wait() at _cv_wait+0x121 zio_wait() at zio_wait+0x61 dmu_tx_count_write() at dmu_tx_count_write+0x34b dmu_tx_hold_write() at dmu_tx_hold_write+0x4a zfs_freebsd_write() at zfs_freebsd_write+0x43b VOP_WRITE_APV() at VOP_WRITE_APV+0xc6 vn_write() at vn_write+0x38c dofilewrite() at dofilewrite+0x8b kern_writev() at kern_writev+0x6c sys_write() at sys_write+0x64 amd64_syscall() at amd64_syscall+0x5ea Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x802aa0b2c, rsp = 0x7fffffffc488, rbp = 0x7fffffffc4b0 --- db> bt 102355 Tracing pid 11143 tid 102355 td 0xfffffe0c8214a8e0 sched_switch() at sched_switch+0x13d mi_switch() at mi_switch+0x196 sleepq_wait() at sleepq_wait+0x42 _cv_wait() at _cv_wait+0x121 zio_wait() at zio_wait+0x61 zil_commit() at zil_commit+0x3e5 zfs_freebsd_fsync() at zfs_freebsd_fsync+0xdc VOP_FSYNC_APV() at VOP_FSYNC_APV+0x4a sys_fsync() at sys_fsync+0x187 amd64_syscall() at amd64_syscall+0x5ea Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (95, FreeBSD ELF64, sys_fsync), rip = 0x80237848c, rsp = 0x7ffffdfefd98, rbp = 0x7ffffdfefdb0 --- db> bt 86292 Tracing pid 86292 tid 101376 td 0xfffffe006152c000 sched_switch() at sched_switch+0x13d mi_switch() at mi_switch+0x196 sleepq_wait() at sleepq_wait+0x42 __lockmgr_args() at __lockmgr_args+0x610 vop_stdlock() at vop_stdlock+0x39 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x52 _vn_lock() at _vn_lock+0x47 vget() at vget+0x70 cache_lookup_times() at cache_lookup_times+0x5f5 vfs_cache_lookup() at vfs_cache_lookup+0xc8 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x4c lookup() at lookup+0x424 namei() at namei+0x535 vn_open_cred() at vn_open_cred+0x3cb kern_openat() at kern_openat+0x1f9 amd64_syscall() at amd64_syscall+0x5ea Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (5, FreeBSD ELF64, sys_open), rip = 0x8010a00ac, rsp = 0x7fffffffbd28, rbp = 0x7fffffffbdc0 --- db> bt 87125 Tracing pid 87125 tid 103485 td 0xfffffe0cb6330000 sched_switch() at sched_switch+0x13d mi_switch() at mi_switch+0x196 sleepq_wait() at sleepq_wait+0x42 __lockmgr_args() at __lockmgr_args+0x610 vop_stdlock() at vop_stdlock+0x39 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x52 _vn_lock() at _vn_lock+0x47 vget() at vget+0x70 cache_lookup_times() at cache_lookup_times+0x5f5 vfs_cache_lookup() at vfs_cache_lookup+0xc8 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x4c lookup() at lookup+0x424 namei() at namei+0x535 vn_open_cred() at vn_open_cred+0x3cb kern_openat() at kern_openat+0x1f9 amd64_syscall() at amd64_syscall+0x5ea Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (5, FreeBSD ELF64, sys_open), rip = 0x4e1a1c, rsp = 0x7fffffffd8b8, rbp = 0 --- db> bt 86928 Tracing pid 86928 tid 104507 td 0xfffffe0e2ffdb000 sched_switch() at sched_switch+0x13d mi_switch() at mi_switch+0x196 sleepq_wait() at sleepq_wait+0x42 _cv_wait() at _cv_wait+0x121 zio_wait() at zio_wait+0x61 dbuf_read() at dbuf_read+0x5e5 dmu_buf_hold() at dmu_buf_hold+0xe0 zap_lockdir() at zap_lockdir+0x58 zap_lookup_norm() at zap_lookup_norm+0x45 zap_lookup() at zap_lookup+0x2e zfs_dirent_lock() at zfs_dirent_lock+0x4ff zfs_dirlook() at zfs_dirlook+0x69 zfs_lookup() at zfs_lookup+0x26b zfs_freebsd_lookup() at zfs_freebsd_lookup+0x81 VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0x44 vfs_cache_lookup() at vfs_cache_lookup+0xf8 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x4c lookup() at lookup+0x424 namei() at namei+0x535 kern_statat_vnhook() at kern_statat_vnhook+0xb6 kern_statat() at kern_statat+0x15 sys_stat() at sys_stat+0x2a amd64_syscall() at amd64_syscall+0x5ea Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (188, FreeBSD ELF64, sys_stat), rip = 0x800da971c, rsp = 0x7fffffffd6c8, rbp = 0x7fffffffd7f0 --- db> bt 86720 Tracing pid 86720 tid 104539 td 0xfffffe0ad6cce8e0 sched_switch() at sched_switch+0x13d mi_switch() at mi_switch+0x196 sleepq_wait() at sleepq_wait+0x42 _sx_xlock_hard() at _sx_xlock_hard+0x4f1 _sx_xlock() at _sx_xlock+0x75 spa_all_configs() at spa_all_configs+0x5e zfs_ioc_pool_configs() at zfs_ioc_pool_configs+0x29 zfsdev_ioctl() at zfsdev_ioctl+0xe6 devfs_ioctl_f() at devfs_ioctl_f+0x7b kern_ioctl() at kern_ioctl+0x115 sys_ioctl() at sys_ioctl+0xfd amd64_syscall() at amd64_syscall+0x5ea Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x801df0aec, rsp = 0x7fffffff83b8, rbp = 0x7fffffff8400 --- db> bt 86079 Tracing pid 86079 tid 104370 td 0xfffffe06f80cd8e0 sched_switch() at sched_switch+0x13d mi_switch() at mi_switch+0x196 sleepq_wait() at sleepq_wait+0x42 _sx_xlock_hard() at _sx_xlock_hard+0x4f1 _sx_xlock() at _sx_xlock+0x75 spa_open_common() at spa_open_common+0x7a pool_status_check() at pool_status_check+0x19 zfsdev_ioctl() at zfsdev_ioctl+0x223 devfs_ioctl_f() at devfs_ioctl_f+0x7b kern_ioctl() at kern_ioctl+0x115 sys_ioctl() at sys_ioctl+0xfd amd64_syscall() at amd64_syscall+0x5ea Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x801befaec, rsp = 0x7fffffff8ed8, rbp = 0x7fffffff8f20 --- db> bt 85728 Tracing pid 85728 tid 105053 td 0xfffffe0294bc5470 sched_switch() at sched_switch+0x13d mi_switch() at mi_switch+0x196 sleepq_wait() at sleepq_wait+0x42 _cv_wait() at _cv_wait+0x121 txg_wait_synced() at txg_wait_synced+0x85 dsl_sync_task_group_wait() at dsl_sync_task_group_wait+0x128 dsl_dataset_user_release() at dsl_dataset_user_release+0x1c9 dsl_dataset_user_release_tmp() at dsl_dataset_user_release_tmp+0xee dsl_dataset_user_release_onexit() at dsl_dataset_user_release_onexit+0x21 zfs_onexit_destroy() at zfs_onexit_destroy+0x56 zfsdev_close() at zfsdev_close+0x4f devfs_destroy_cdevpriv() at devfs_destroy_cdevpriv+0x69 devfs_close_f() at devfs_close_f+0x54 _fdrop() at _fdrop+0x23 closef() at closef+0x52 kern_close() at kern_close+0x172 amd64_syscall() at amd64_syscall+0x5ea Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (6, FreeBSD ELF64, sys_close), rip = 0x801befb0c, rsp = 0x7fffffffc5a8, rbp = 0x7fffffffcff0 --- --------------070005030706090006050108--