Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 23 Feb 2006 12:53:08 -0500
From:      "Michael R. Wayne" <freebsd@wayne47.com>
To:        stable@freebsd.org
Subject:   Disk I/O system hang on 5.4-RELEASE-p8 i386
Message-ID:  <200602231753.k1NHr8c1079056@manor.msen.com>

next in thread | raw e-mail | index | archive | help

Been fighting this for a while.  We have an older server, running
5.4-RELEASE-p8 i386 and used primarily for email, which hangs every
couple of weeks.  The hang seems to be in the disk I/O system; pings
succeed, and I can continue get a login: prompt on the console until
I enter a login at which the response stops.

I'm suspecting this is a 5.8 issue as we have the same problem on
a another box running 5.4-RELEASE-p8 amd64 with a 3ware controller.
I do not have a dump on that one.

Based on the times of the hangs, the triggering event seems to be
running dump.

We have a serial console set up, I broke to the debugger and got
the following info.  Since the hang is in the disk I/O system, a
dump is not possible.  The many versions of inetd are likely due
to users attempting to POP their email and hanging on disk I/O.

Any suggestions or tips on how to track this down would be appreciated.


db> ps
  pid   proc     uid  ppid  pgrp  flag   stat  wmesg    wchan  cmd
67487 c5ea98d4    0   551   551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67486 c3b8a1c4    0   551   551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67485 c634c710    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67484 c62931c4    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67483 c58a9388    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67482 c6293710    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67481 c58ab8d4    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67480 c6292c5c    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67479 c62938d4    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67478 c634f000    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67477 c62941c4    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67476 c5e55c5c    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67475 c5f1fe20    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67474 c5da854c    0   551   551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67473 c5f9ee20    0   551   551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67472 c58a9e20    0   551   551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67471 c602d8d4    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67470 c61191c4    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67469 c58ab1c4    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67468 c5f19a98    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67467 c58c3388    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67466 c5f1f388    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67465 c5f19000    0   442 67465 0000100 [SLPQ ufs 0xc3851c04][SLP] sshd
67464 c5fa68d4    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67463 c5eab54c    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67462 c6294000    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67461 c5fa6710    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67460 c6119000    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67459 c634c388    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67458 c5da8710    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67457 c3b8e388    0   551   551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67456 c62948d4    0   551   551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67455 c62921c4    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67454 c5f1954c    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67453 c5eab388    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67452 c5f171c4    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67451 c6294388    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67450 c60291c4    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67449 c5ea9710    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67448 c3b8ac5c    0   551   551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67447 c6293388    0   551   551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67446 c5f1fa98    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67445 c5e55e20    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67444 c6117710    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67443 c5fa7a98    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67442 c6026388    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67441 c5e5a1c4    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67440 c6119710    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67439 c5e5a000    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67438 c3b8e8d4    0   551   551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67437 c6293e20    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67436 c5dfee20    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67435 c5ea8e20    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67434 c5fa7388    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67433 c39dde20    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67432 c6118000    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67431 c58c31c4    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67430 c634fe20    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67429 c5f191c4    0   551   551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67428 c58a9c5c    0   551   551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67427 c63508d4    0   551   551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67426 c3b8ee20    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67425 c3b8ec5c    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67424 c5eabc5c    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67423 c5e5a54c    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67422 c39de710    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67421 c6117e20    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67420 c5da454c    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67419 c5e551c4    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67418 c5da88d4    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67417 c611ce20    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67416 c5e55000    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67415 c5f19c5c    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67414 c5f198d4    0   574   574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67413 c60261c4    0   551   551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67412 c6293c5c    0   574 67412 0004000 [SLPQ ufs 0xc3851c04][SLP] qpopper
67411 c58a9a98    0   574 67411 0004000 [SLPQ ufs 0xc3851c04][SLP] qpopper
67410 c38898d4    0   574 67410 0004000 [SLPQ suspfs 0xc37cac6c][SLP] qpopper
67409 c611c54c    0   574 67409 0004000 [SLPQ ufs 0xc38515d4][SLP] qpopper
67408 c634c1c4    0   574 67408 0004000 [SLPQ ufs 0xc3851c04][SLP] qpopper
67407 c5ea9000    0   574 67407 0004000 [SLPQ ufs 0xc3851c04][SLP] qpopper
67406 c58c3000    0   574 67406 0004000 [SLPQ ufs 0xc388d9f4][SLP] qpopper
67405 c5da41c4    2 67404 67401 0004100 [SLPQ ufs 0xc3851c04][SLP] mksnap_ffs
67404 c58c5a98    2 67403 67401 0004000 [SLPQ wait 0xc58c5a98][SLP] sh
67403 c5fa7e20    2 67402 67401 0004000 [SLPQ wait 0xc5fa7e20][SLP] dump
67402 c5fa71c4    2 67401 67401 0004000 [SLPQ piperd 0xc60fb600][SLP] gzip
67401 c5f19710    2 67400 67401 0004000 [SLPQ pause 0xc5f19748][SLP] tcsh
67400 c5e5554c    2 67398 67398 0000100 [SLPQ select 0xc08f1f44][SLP] sshd
67398 c611854c    0   442 67398 0000100 [SLPQ sbwait 0xc6273974][SLP] sshd
67322 c39dea98    0   551   551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
62457 c39dee20    0   514   514 0000100 [SLPQ accept 0xc3997916][SLP] perl5.8.6
62370 c5f1f710    0   514   514 0000100 [SLPQ accept 0xc3997916][SLP] perl5.8.6
61704 c58c5c5c    0   514   514 0000100 [SLPQ accept 0xc3997916][SLP] perl5.8.6
61031 c58a91c4    0   514   514 0000100 [SLPQ accept 0xc3997916][SLP] perl5.8.6
59856 c5da48d4    0   514   514 0000100 [SLPQ accept 0xc3997916][SLP] perl5.8.6
43502 c58c3710 5147   589 43502 0004002 [SLPQ ufs 0xc3851c04][SLP] tcsh
  589 c38848d4    0     1   589 0004102 [SLPQ wait 0xc38848d4][SLP] login
  588 c3884c5c    0     1     1 0004000 [SLPQ ufs 0xc3851c04][SLP] getty
  587 c39dd000    0     1     1 0004000 [SLPQ ufs 0xc3851c04][SLP] getty
  586 c3889e20    0     1     1 0004000 [SLPQ ufs 0xc3851c04][SLP] getty
  574 c3884e20    0     1   574 0000000 [SLPQ select 0xc08f1f44][SLP] inetd
  551 c39dd388    0     1   551 0000100 [SLPQ select 0xc08f1f44][SLP] sendmail
  542 c39dda98    0     1   542 0008080 (threaded)  spamass-milter
   thread 0xc628e000 ksegrp 0xc353aaf0 [SLPQ kserel 0xc353ab30][SLP]
   thread 0xc5fa1900 ksegrp 0xc353aaf0 [SLPQ select 0xc08f1f44][SLP]
   thread 0xc39e0900 ksegrp 0xc353a5b0 [SLPQ ksesigwait 0xc39ddb98][SLP]
  532 c39de000   25     1   532 0000100 [SLPQ pause 0xc39de038][SLP] sendmail
  527 c39ddc5c    0   521   521 0000001 [SLPQ lockf 0xc38193c0][SLP] saslauthd
  526 c39dd710    0   521   521 0000001 [SLPQ lockf 0xc35e4c40][SLP] saslauthd
  525 c39dd54c    0   521   521 0000001 [SLPQ lockf 0xc37e2580][SLP] saslauthd
  524 c39dd1c4    0   521   521 0000001 [SLPQ lockf 0xc3819140][SLP] saslauthd
  521 c3889c5c    0     1   521 0000001 [SLPQ accept 0xc3997b9e][SLP] saslauthd
  514 c3889a98    0     1   514 0000000 [SLPQ pause 0xc3889ad0][SLP] perl5.8.6
  499 c37eda98 65534     1   499 0000100 [SLPQ select 0xc08f1f44][SLP] spamd
  493 c3889000    0     1   493 0000000 [SLPQ select 0xc08f1f44][SLP] rpc.dracd
  484 c3884388  106     1   484 0008180 (threaded)  clamav-milter
   thread 0xc5f1e900 ksegrp 0xc3448380 [SLPQ kserel 0xc34483c0][SLP]
   thread 0xc6032780 ksegrp 0xc3448380 [SLPQ select 0xc08f1f44][SLP]
   thread 0xc5df8780 ksegrp 0xc3448380 [SLPQ ufs 0xc3851c04][SLP]
   thread 0xc39d6900 ksegrp 0xc353ae00 [SLPQ ksesigwait 0xc3884488][SLP]
  477 c3884710  106     1   477 0000100 [SLPQ pause 0xc3884748][SLP] freshclam
  470 c388454c  106     1   470 0008180 (threaded)  clamd
   thread 0xc5e34900 ksegrp 0xc3448310 [SLPQ kserel 0xc3448350][SLP]
   thread 0xc5e56000 ksegrp 0xc3448310 [SLPQ accept 0xc3924e26][SLP]
   thread 0xc3aa2000 ksegrp 0xc388abd0 [SLPQ ksesigwait 0xc388464c][SLP]
  455 c3889388    0     1   455 0000000 [SLPQ ufs 0xc3851c04][SLP] cron
  442 c38841c4    0     1   442 0000100 [SLPQ ufs 0xc3851c04][SLP] sshd
  429 c3884000    0     1   429 0000000 [SLPQ select 0xc08f1f44][SLP] ntpd
  338 c37ede20    0     1   338 0000000 [SLPQ select 0xc08f1f44][SLP] rpcbind
  325 c3884a98    0     1   325 0000000 [SLPQ select 0xc08f1f44][SLP] syslogd
  307 c38891c4    0     1   307 0000000 [SLPQ select 0xc08f1f44][SLP] devd
   58 c353c54c    0     0     0 0000204 [SLPQ - 0xe67d4d18][SLP] schedcpu
   57 c353c710    0     0     0 0000204 [SLPQ - 0xc08f996c][SLP] nfsiod 3
   56 c353c8d4    0     0     0 0000204 [SLPQ - 0xc08f9968][SLP] nfsiod 2
   55 c353ca98    0     0     0 0000204 [SLPQ - 0xc08f9964][SLP] nfsiod 1
   54 c353cc5c    0     0     0 0000204 [SLPQ - 0xc08f9960][SLP] nfsiod 0
   53 c353ce20    0     0     0 0000204 [SLPQ vlruwt 0xc353ce20][SLP] vnlru
   52 c37ed000    0     0     0 0000204 [SLPQ syncer 0xc08ee6cc][SLP] syncer
   51 c37ed1c4    0     0     0 0000204 [SLPQ psleep 0xc08f250c][SLP] bufdaemon
   50 c37ed388    0     0     0 000020c [SLPQ pgzero 0xc09002d4][SLP] pagezero
   49 c37ed54c    0     0     0 0000204 [SLPQ psleep 0xc0900328][SLP] vmdaemon
   48 c37ed710    0     0     0 0000204 [SLPQ psleep 0xc09002e4][SLP] pagedaemon
   47 c37ed8d4    0     0     0 0000204 [SLPQ m:w2 0xc37ea000][SLP] g_mirror gm0s2
   46 c349ba98    0     0     0 0000204 [SLPQ m:w2 0xc37ea500][SLP] g_mirror gm0s1
   45 c349bc5c    0     0     0 0000204 [IWAIT] swi0: sio
   44 c349be20    0     0     0 0000204 [SLPQ - 0xc354223c][SLP] fdc0
   43 c3538000    0     0     0 0000204 [SLPQ idle 0xc3540e00][SLP] aic_recovery1
    9 c35381c4    0     0     0 0000204 [SLPQ idle 0xc3540e00][SLP] aic_recovery1
    8 c3538388    0     0     0 0000204 [SLPQ idle 0xc3540400][SLP] aic_recovery0
    7 c353854c    0     0     0 0000204 [SLPQ idle 0xc3540400][SLP] aic_recovery0
   42 c3538710    0     0     0 0000204 [IWAIT] swi6: task queue
    6 c35388d4    0     0     0 0000204 [SLPQ - 0xc352e3c0][SLP] kqueue taskq
   41 c3538a98    0     0     0 0000204 [IWAIT] swi3: cambio
   40 c3538c5c    0     0     0 0000204 [IWAIT] swi2: camnet
   39 c3538e20    0     0     0 0000204 [IWAIT] swi6:+
    5 c353c000    0     0     0 0000204 [SLPQ - 0xc352ed80][SLP] thread taskq
   38 c348a54c    0     0     0 0000204 [IWAIT] swi6:+
   37 c348a710    0     0     0 0000204 [SLPQ - 0xc08e4660][SLP] yarrow
    4 c348a8d4    0     0     0 0000204 [SLPQ - 0xc08e8fa8][SLP] g_down
    3 c348aa98    0     0     0 0000204 [SLPQ - 0xc08e8fa4][SLP] g_up
    2 c348ac5c    0     0     0 0000204 [SLPQ - 0xc08e8f9c][SLP] g_event
   36 c348ae20    0     0     0 0000204 [IWAIT] swi1: net
   35 c349b000    0     0     0 0000204 [IWAIT] swi4: vm
   34 c349b1c4    0     0     0 000020c [IWAIT] swi5: clock sio
   33 c349b388    0     0     0 0000204 [IWAIT] irq0: clk
   32 c349b54c    0     0     0 0000204 [IWAIT] irq22:
   31 c349b710    0     0     0 0000204 [IWAIT] irq21:
   30 c349b8d4    0     0     0 0000204 [IWAIT] irq20:
   29 c34491c4    0     0     0 0000204 [IWAIT] irq19: fxp0
   28 c3449388    0     0     0 0000204 [IWAIT] irq18:
   27 c344954c    0     0     0 0000204 [IWAIT] irq17:
   26 c3449710    0     0     0 0000204 [IWAIT] irq16: fxp1 ahc0+
   25 c34498d4    0     0     0 0000204 [IWAIT] irq15: ata1
   24 c3449a98    0     0     0 0000204 [IWAIT] irq14: ata0
   23 c3449c5c    0     0     0 0000204 [IWAIT] irq13:
   22 c3449e20    0     0     0 0000204 [IWAIT] irq12:
   21 c348a000    0     0     0 0000204 [IWAIT] irq11:
   20 c348a1c4    0     0     0 0000204 [IWAIT] irq10:
   19 c348a388    0     0     0 0000204 [IWAIT] irq9:
   18 c3442000    0     0     0 0000204 [IWAIT] irq8: rtc
   17 c34421c4    0     0     0 0000204 [IWAIT] irq7: ppc0
   16 c3442388    0     0     0 0000204 [IWAIT] irq6: fdc0
   15 c344254c    0     0     0 0000204 [IWAIT] irq5:
   14 c3442710    0     0     0 0000204 [IWAIT] irq4: sio0
   13 c34428d4    0     0     0 0000204 [IWAIT] irq3: sio1
   12 c3442a98    0     0     0 0000204 [IWAIT] irq1: atkbd0
   11 c3442c5c    0     0     0 000020c [CPU 0] idle
    1 c3442e20    0     0     1 0004200 [SLPQ wait 0xc3442e20][SLP] init
   10 c3449000    0     0     0 0000204 [SLPQ ktrace 0xc08ec8f8][SLP] ktrace
    0 c08e90a0    0     0     0 0000200 [SLPQ sched 0xc08e90a0][SLP] swapper
db> where
Tracing pid 11 tid 100003 td 0xc3443480
kdb_enter(c08479e3) at kdb_enter+0x2b
siointr1(c35d6000) at siointr1+0xd5
siointr(c35d6000) at siointr+0x38
intr_execute_handlers(c343dc90,e4d53cc8,4,e4d53d0c,c07b9483) at intr_execute_handlers+0x7d
lapic_handle_intr(34) at lapic_handle_intr+0x2e
Xapic_isr1() at Xapic_isr1+0x33
--- interrupt, eip = 0xc07c057d, esp = 0xe4d53d0c, ebp = 0xe4d53d0c ---
cpu_idle_default(e4d53d20,c0604971,c3442c5c,e4d53d34,c0604720) at cpu_idle_default+0x5
cpu_idle(c3442c5c,e4d53d34,c0604720,0,e4d53d48) at cpu_idle+0x1f
idle_proc(0,e4d53d48) at idle_proc+0x11
fork_exit(c0604960,0,e4d53d48) at fork_exit+0x74
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4d53d7c, ebp = 0 ---


Relevant dmesg info:
   ahc0: <Adaptec aic7896/97 Ultra2 SCSI adapter> port 0xe400-0xe4ff mem 0xffafe000-0xffafefff irq 16 at device 11.0 on pci0
   aic7896/97: Ultra2 Wide Channel A, SCSI Id=7, 32/253 SCBs
   ahc1: <Adaptec aic7896/97 Ultra2 SCSI adapter> port 0xe800-0xe8ff mem 0xffaff000-0xffafffff irq 16 at device 11.1 on pci0
   aic7896/97: Ultra2 Wide Channel B, SCSI Id=7, 32/253 SCBs

   da0 at ahc0 bus 0 target 0 lun 0
   da0: <SEAGATE ST318436LW 0010> Fixed Direct Access SCSI-3 device 
   da0: 80.000MB/s transfers (40.000MHz, offset 31, 16bit), Tagged Queueing Enabled
   da0: 17522MB (35885168 512 byte sectors: 255H 63S/T 2233C)
   da1 at ahc1 bus 0 target 0 lun 0
   da1: <SEAGATE ST336938LW 0003> Fixed Direct Access SCSI-3 device 
   da1: 80.000MB/s transfers (40.000MHz, offset 63, 16bit), Tagged Queueing Enabled
   da1: 35242MB (72176567 512 byte sectors: 255H 63S/T 4492C)
   da2 at ahc1 bus 0 target 1 lun 0
   da2: <SEAGATE ST318436LW 0010> Fixed Direct Access SCSI-3 device 
   da2: 80.000MB/s transfers (40.000MHz, offset 31, 16bit), Tagged Queueing Enabled
   da2: 17522MB (35885168 512 byte sectors: 255H 63S/T 2233C)
   GEOM_MIRROR: Device gm0s1 created (id=520792649).
   GEOM_MIRROR: Device gm0s1: provider da0s1 detected.
   GEOM_MIRROR: Device gm0s2 created (id=3744871543).
   GEOM_MIRROR: Device gm0s2: provider da0s2 detected.
   GEOM_MIRROR: Device gm0s1: provider da2s1 detected.
   GEOM_MIRROR: Device gm0s1: provider da2s1 activated.
   GEOM_MIRROR: Device gm0s1: provider da0s1 activated.
   GEOM_MIRROR: Device gm0s1: provider mirror/gm0s1 launched.
   GEOM_MIRROR: Device gm0s2: provider da2s2 detected.
   GEOM_MIRROR: Device gm0s2: provider da2s2 activated.
   GEOM_MIRROR: Device gm0s2: provider da0s2 activated.
   GEOM_MIRROR: Device gm0s2: provider mirror/gm0s2 launched.




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200602231753.k1NHr8c1079056>