From owner-freebsd-stable@FreeBSD.ORG Thu Feb 23 17:53:09 2006 Return-Path: X-Original-To: stable@freebsd.org Delivered-To: freebsd-stable@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C9B7616A420 for ; Thu, 23 Feb 2006 17:53:09 +0000 (GMT) (envelope-from wayne@manor.msen.com) Received: from manor.msen.com (manor.msen.com [148.59.4.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6361B43D46 for ; Thu, 23 Feb 2006 17:53:09 +0000 (GMT) (envelope-from wayne@manor.msen.com) Received: from manor.msen.com (localhost [127.0.0.1]) by manor.msen.com (8.12.9p2/8.12.9) with ESMTP id k1NHr8c1079056 for ; Thu, 23 Feb 2006 12:53:08 -0500 (EST) (envelope-from wayne@manor.msen.com) Message-Id: <200602231753.k1NHr8c1079056@manor.msen.com> To: stable@freebsd.org From: "Michael R. Wayne" Date: Thu, 23 Feb 2006 12:53:08 -0500 Sender: wayne@manor.msen.com Cc: Subject: Disk I/O system hang on 5.4-RELEASE-p8 i386 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Feb 2006 17:53:10 -0000 Been fighting this for a while. We have an older server, running 5.4-RELEASE-p8 i386 and used primarily for email, which hangs every couple of weeks. The hang seems to be in the disk I/O system; pings succeed, and I can continue get a login: prompt on the console until I enter a login at which the response stops. I'm suspecting this is a 5.8 issue as we have the same problem on a another box running 5.4-RELEASE-p8 amd64 with a 3ware controller. I do not have a dump on that one. Based on the times of the hangs, the triggering event seems to be running dump. We have a serial console set up, I broke to the debugger and got the following info. Since the hang is in the disk I/O system, a dump is not possible. The many versions of inetd are likely due to users attempting to POP their email and hanging on disk I/O. Any suggestions or tips on how to track this down would be appreciated. db> ps pid proc uid ppid pgrp flag stat wmesg wchan cmd 67487 c5ea98d4 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail 67486 c3b8a1c4 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail 67485 c634c710 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67484 c62931c4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67483 c58a9388 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67482 c6293710 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67481 c58ab8d4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67480 c6292c5c 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67479 c62938d4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67478 c634f000 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67477 c62941c4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67476 c5e55c5c 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67475 c5f1fe20 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67474 c5da854c 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail 67473 c5f9ee20 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail 67472 c58a9e20 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail 67471 c602d8d4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67470 c61191c4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67469 c58ab1c4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67468 c5f19a98 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67467 c58c3388 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67466 c5f1f388 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67465 c5f19000 0 442 67465 0000100 [SLPQ ufs 0xc3851c04][SLP] sshd 67464 c5fa68d4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67463 c5eab54c 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67462 c6294000 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67461 c5fa6710 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67460 c6119000 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67459 c634c388 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67458 c5da8710 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67457 c3b8e388 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail 67456 c62948d4 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail 67455 c62921c4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67454 c5f1954c 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67453 c5eab388 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67452 c5f171c4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67451 c6294388 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67450 c60291c4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67449 c5ea9710 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67448 c3b8ac5c 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail 67447 c6293388 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail 67446 c5f1fa98 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67445 c5e55e20 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67444 c6117710 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67443 c5fa7a98 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67442 c6026388 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67441 c5e5a1c4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67440 c6119710 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67439 c5e5a000 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67438 c3b8e8d4 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail 67437 c6293e20 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67436 c5dfee20 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67435 c5ea8e20 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67434 c5fa7388 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67433 c39dde20 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67432 c6118000 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67431 c58c31c4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67430 c634fe20 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67429 c5f191c4 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail 67428 c58a9c5c 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail 67427 c63508d4 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail 67426 c3b8ee20 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67425 c3b8ec5c 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67424 c5eabc5c 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67423 c5e5a54c 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67422 c39de710 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67421 c6117e20 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67420 c5da454c 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67419 c5e551c4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67418 c5da88d4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67417 c611ce20 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67416 c5e55000 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67415 c5f19c5c 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67414 c5f198d4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd 67413 c60261c4 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail 67412 c6293c5c 0 574 67412 0004000 [SLPQ ufs 0xc3851c04][SLP] qpopper 67411 c58a9a98 0 574 67411 0004000 [SLPQ ufs 0xc3851c04][SLP] qpopper 67410 c38898d4 0 574 67410 0004000 [SLPQ suspfs 0xc37cac6c][SLP] qpopper 67409 c611c54c 0 574 67409 0004000 [SLPQ ufs 0xc38515d4][SLP] qpopper 67408 c634c1c4 0 574 67408 0004000 [SLPQ ufs 0xc3851c04][SLP] qpopper 67407 c5ea9000 0 574 67407 0004000 [SLPQ ufs 0xc3851c04][SLP] qpopper 67406 c58c3000 0 574 67406 0004000 [SLPQ ufs 0xc388d9f4][SLP] qpopper 67405 c5da41c4 2 67404 67401 0004100 [SLPQ ufs 0xc3851c04][SLP] mksnap_ffs 67404 c58c5a98 2 67403 67401 0004000 [SLPQ wait 0xc58c5a98][SLP] sh 67403 c5fa7e20 2 67402 67401 0004000 [SLPQ wait 0xc5fa7e20][SLP] dump 67402 c5fa71c4 2 67401 67401 0004000 [SLPQ piperd 0xc60fb600][SLP] gzip 67401 c5f19710 2 67400 67401 0004000 [SLPQ pause 0xc5f19748][SLP] tcsh 67400 c5e5554c 2 67398 67398 0000100 [SLPQ select 0xc08f1f44][SLP] sshd 67398 c611854c 0 442 67398 0000100 [SLPQ sbwait 0xc6273974][SLP] sshd 67322 c39dea98 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail 62457 c39dee20 0 514 514 0000100 [SLPQ accept 0xc3997916][SLP] perl5.8.6 62370 c5f1f710 0 514 514 0000100 [SLPQ accept 0xc3997916][SLP] perl5.8.6 61704 c58c5c5c 0 514 514 0000100 [SLPQ accept 0xc3997916][SLP] perl5.8.6 61031 c58a91c4 0 514 514 0000100 [SLPQ accept 0xc3997916][SLP] perl5.8.6 59856 c5da48d4 0 514 514 0000100 [SLPQ accept 0xc3997916][SLP] perl5.8.6 43502 c58c3710 5147 589 43502 0004002 [SLPQ ufs 0xc3851c04][SLP] tcsh 589 c38848d4 0 1 589 0004102 [SLPQ wait 0xc38848d4][SLP] login 588 c3884c5c 0 1 1 0004000 [SLPQ ufs 0xc3851c04][SLP] getty 587 c39dd000 0 1 1 0004000 [SLPQ ufs 0xc3851c04][SLP] getty 586 c3889e20 0 1 1 0004000 [SLPQ ufs 0xc3851c04][SLP] getty 574 c3884e20 0 1 574 0000000 [SLPQ select 0xc08f1f44][SLP] inetd 551 c39dd388 0 1 551 0000100 [SLPQ select 0xc08f1f44][SLP] sendmail 542 c39dda98 0 1 542 0008080 (threaded) spamass-milter thread 0xc628e000 ksegrp 0xc353aaf0 [SLPQ kserel 0xc353ab30][SLP] thread 0xc5fa1900 ksegrp 0xc353aaf0 [SLPQ select 0xc08f1f44][SLP] thread 0xc39e0900 ksegrp 0xc353a5b0 [SLPQ ksesigwait 0xc39ddb98][SLP] 532 c39de000 25 1 532 0000100 [SLPQ pause 0xc39de038][SLP] sendmail 527 c39ddc5c 0 521 521 0000001 [SLPQ lockf 0xc38193c0][SLP] saslauthd 526 c39dd710 0 521 521 0000001 [SLPQ lockf 0xc35e4c40][SLP] saslauthd 525 c39dd54c 0 521 521 0000001 [SLPQ lockf 0xc37e2580][SLP] saslauthd 524 c39dd1c4 0 521 521 0000001 [SLPQ lockf 0xc3819140][SLP] saslauthd 521 c3889c5c 0 1 521 0000001 [SLPQ accept 0xc3997b9e][SLP] saslauthd 514 c3889a98 0 1 514 0000000 [SLPQ pause 0xc3889ad0][SLP] perl5.8.6 499 c37eda98 65534 1 499 0000100 [SLPQ select 0xc08f1f44][SLP] spamd 493 c3889000 0 1 493 0000000 [SLPQ select 0xc08f1f44][SLP] rpc.dracd 484 c3884388 106 1 484 0008180 (threaded) clamav-milter thread 0xc5f1e900 ksegrp 0xc3448380 [SLPQ kserel 0xc34483c0][SLP] thread 0xc6032780 ksegrp 0xc3448380 [SLPQ select 0xc08f1f44][SLP] thread 0xc5df8780 ksegrp 0xc3448380 [SLPQ ufs 0xc3851c04][SLP] thread 0xc39d6900 ksegrp 0xc353ae00 [SLPQ ksesigwait 0xc3884488][SLP] 477 c3884710 106 1 477 0000100 [SLPQ pause 0xc3884748][SLP] freshclam 470 c388454c 106 1 470 0008180 (threaded) clamd thread 0xc5e34900 ksegrp 0xc3448310 [SLPQ kserel 0xc3448350][SLP] thread 0xc5e56000 ksegrp 0xc3448310 [SLPQ accept 0xc3924e26][SLP] thread 0xc3aa2000 ksegrp 0xc388abd0 [SLPQ ksesigwait 0xc388464c][SLP] 455 c3889388 0 1 455 0000000 [SLPQ ufs 0xc3851c04][SLP] cron 442 c38841c4 0 1 442 0000100 [SLPQ ufs 0xc3851c04][SLP] sshd 429 c3884000 0 1 429 0000000 [SLPQ select 0xc08f1f44][SLP] ntpd 338 c37ede20 0 1 338 0000000 [SLPQ select 0xc08f1f44][SLP] rpcbind 325 c3884a98 0 1 325 0000000 [SLPQ select 0xc08f1f44][SLP] syslogd 307 c38891c4 0 1 307 0000000 [SLPQ select 0xc08f1f44][SLP] devd 58 c353c54c 0 0 0 0000204 [SLPQ - 0xe67d4d18][SLP] schedcpu 57 c353c710 0 0 0 0000204 [SLPQ - 0xc08f996c][SLP] nfsiod 3 56 c353c8d4 0 0 0 0000204 [SLPQ - 0xc08f9968][SLP] nfsiod 2 55 c353ca98 0 0 0 0000204 [SLPQ - 0xc08f9964][SLP] nfsiod 1 54 c353cc5c 0 0 0 0000204 [SLPQ - 0xc08f9960][SLP] nfsiod 0 53 c353ce20 0 0 0 0000204 [SLPQ vlruwt 0xc353ce20][SLP] vnlru 52 c37ed000 0 0 0 0000204 [SLPQ syncer 0xc08ee6cc][SLP] syncer 51 c37ed1c4 0 0 0 0000204 [SLPQ psleep 0xc08f250c][SLP] bufdaemon 50 c37ed388 0 0 0 000020c [SLPQ pgzero 0xc09002d4][SLP] pagezero 49 c37ed54c 0 0 0 0000204 [SLPQ psleep 0xc0900328][SLP] vmdaemon 48 c37ed710 0 0 0 0000204 [SLPQ psleep 0xc09002e4][SLP] pagedaemon 47 c37ed8d4 0 0 0 0000204 [SLPQ m:w2 0xc37ea000][SLP] g_mirror gm0s2 46 c349ba98 0 0 0 0000204 [SLPQ m:w2 0xc37ea500][SLP] g_mirror gm0s1 45 c349bc5c 0 0 0 0000204 [IWAIT] swi0: sio 44 c349be20 0 0 0 0000204 [SLPQ - 0xc354223c][SLP] fdc0 43 c3538000 0 0 0 0000204 [SLPQ idle 0xc3540e00][SLP] aic_recovery1 9 c35381c4 0 0 0 0000204 [SLPQ idle 0xc3540e00][SLP] aic_recovery1 8 c3538388 0 0 0 0000204 [SLPQ idle 0xc3540400][SLP] aic_recovery0 7 c353854c 0 0 0 0000204 [SLPQ idle 0xc3540400][SLP] aic_recovery0 42 c3538710 0 0 0 0000204 [IWAIT] swi6: task queue 6 c35388d4 0 0 0 0000204 [SLPQ - 0xc352e3c0][SLP] kqueue taskq 41 c3538a98 0 0 0 0000204 [IWAIT] swi3: cambio 40 c3538c5c 0 0 0 0000204 [IWAIT] swi2: camnet 39 c3538e20 0 0 0 0000204 [IWAIT] swi6:+ 5 c353c000 0 0 0 0000204 [SLPQ - 0xc352ed80][SLP] thread taskq 38 c348a54c 0 0 0 0000204 [IWAIT] swi6:+ 37 c348a710 0 0 0 0000204 [SLPQ - 0xc08e4660][SLP] yarrow 4 c348a8d4 0 0 0 0000204 [SLPQ - 0xc08e8fa8][SLP] g_down 3 c348aa98 0 0 0 0000204 [SLPQ - 0xc08e8fa4][SLP] g_up 2 c348ac5c 0 0 0 0000204 [SLPQ - 0xc08e8f9c][SLP] g_event 36 c348ae20 0 0 0 0000204 [IWAIT] swi1: net 35 c349b000 0 0 0 0000204 [IWAIT] swi4: vm 34 c349b1c4 0 0 0 000020c [IWAIT] swi5: clock sio 33 c349b388 0 0 0 0000204 [IWAIT] irq0: clk 32 c349b54c 0 0 0 0000204 [IWAIT] irq22: 31 c349b710 0 0 0 0000204 [IWAIT] irq21: 30 c349b8d4 0 0 0 0000204 [IWAIT] irq20: 29 c34491c4 0 0 0 0000204 [IWAIT] irq19: fxp0 28 c3449388 0 0 0 0000204 [IWAIT] irq18: 27 c344954c 0 0 0 0000204 [IWAIT] irq17: 26 c3449710 0 0 0 0000204 [IWAIT] irq16: fxp1 ahc0+ 25 c34498d4 0 0 0 0000204 [IWAIT] irq15: ata1 24 c3449a98 0 0 0 0000204 [IWAIT] irq14: ata0 23 c3449c5c 0 0 0 0000204 [IWAIT] irq13: 22 c3449e20 0 0 0 0000204 [IWAIT] irq12: 21 c348a000 0 0 0 0000204 [IWAIT] irq11: 20 c348a1c4 0 0 0 0000204 [IWAIT] irq10: 19 c348a388 0 0 0 0000204 [IWAIT] irq9: 18 c3442000 0 0 0 0000204 [IWAIT] irq8: rtc 17 c34421c4 0 0 0 0000204 [IWAIT] irq7: ppc0 16 c3442388 0 0 0 0000204 [IWAIT] irq6: fdc0 15 c344254c 0 0 0 0000204 [IWAIT] irq5: 14 c3442710 0 0 0 0000204 [IWAIT] irq4: sio0 13 c34428d4 0 0 0 0000204 [IWAIT] irq3: sio1 12 c3442a98 0 0 0 0000204 [IWAIT] irq1: atkbd0 11 c3442c5c 0 0 0 000020c [CPU 0] idle 1 c3442e20 0 0 1 0004200 [SLPQ wait 0xc3442e20][SLP] init 10 c3449000 0 0 0 0000204 [SLPQ ktrace 0xc08ec8f8][SLP] ktrace 0 c08e90a0 0 0 0 0000200 [SLPQ sched 0xc08e90a0][SLP] swapper db> where Tracing pid 11 tid 100003 td 0xc3443480 kdb_enter(c08479e3) at kdb_enter+0x2b siointr1(c35d6000) at siointr1+0xd5 siointr(c35d6000) at siointr+0x38 intr_execute_handlers(c343dc90,e4d53cc8,4,e4d53d0c,c07b9483) at intr_execute_handlers+0x7d lapic_handle_intr(34) at lapic_handle_intr+0x2e Xapic_isr1() at Xapic_isr1+0x33 --- interrupt, eip = 0xc07c057d, esp = 0xe4d53d0c, ebp = 0xe4d53d0c --- cpu_idle_default(e4d53d20,c0604971,c3442c5c,e4d53d34,c0604720) at cpu_idle_default+0x5 cpu_idle(c3442c5c,e4d53d34,c0604720,0,e4d53d48) at cpu_idle+0x1f idle_proc(0,e4d53d48) at idle_proc+0x11 fork_exit(c0604960,0,e4d53d48) at fork_exit+0x74 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe4d53d7c, ebp = 0 --- Relevant dmesg info: ahc0: port 0xe400-0xe4ff mem 0xffafe000-0xffafefff irq 16 at device 11.0 on pci0 aic7896/97: Ultra2 Wide Channel A, SCSI Id=7, 32/253 SCBs ahc1: port 0xe800-0xe8ff mem 0xffaff000-0xffafffff irq 16 at device 11.1 on pci0 aic7896/97: Ultra2 Wide Channel B, SCSI Id=7, 32/253 SCBs da0 at ahc0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-3 device da0: 80.000MB/s transfers (40.000MHz, offset 31, 16bit), Tagged Queueing Enabled da0: 17522MB (35885168 512 byte sectors: 255H 63S/T 2233C) da1 at ahc1 bus 0 target 0 lun 0 da1: Fixed Direct Access SCSI-3 device da1: 80.000MB/s transfers (40.000MHz, offset 63, 16bit), Tagged Queueing Enabled da1: 35242MB (72176567 512 byte sectors: 255H 63S/T 4492C) da2 at ahc1 bus 0 target 1 lun 0 da2: Fixed Direct Access SCSI-3 device da2: 80.000MB/s transfers (40.000MHz, offset 31, 16bit), Tagged Queueing Enabled da2: 17522MB (35885168 512 byte sectors: 255H 63S/T 2233C) GEOM_MIRROR: Device gm0s1 created (id=520792649). GEOM_MIRROR: Device gm0s1: provider da0s1 detected. GEOM_MIRROR: Device gm0s2 created (id=3744871543). GEOM_MIRROR: Device gm0s2: provider da0s2 detected. GEOM_MIRROR: Device gm0s1: provider da2s1 detected. GEOM_MIRROR: Device gm0s1: provider da2s1 activated. GEOM_MIRROR: Device gm0s1: provider da0s1 activated. GEOM_MIRROR: Device gm0s1: provider mirror/gm0s1 launched. GEOM_MIRROR: Device gm0s2: provider da2s2 detected. GEOM_MIRROR: Device gm0s2: provider da2s2 activated. GEOM_MIRROR: Device gm0s2: provider da0s2 activated. GEOM_MIRROR: Device gm0s2: provider mirror/gm0s2 launched.