From owner-freebsd-stable@FreeBSD.ORG Wed May 15 12:40:20 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id D0CB0299 for ; Wed, 15 May 2013 12:40:20 +0000 (UTC) (envelope-from db@nipsi.de) Received: from fop.bsdsystems.de (mx.bsdsystems.de [88.198.57.43]) by mx1.freebsd.org (Postfix) with ESMTP id A21D09BB for ; Wed, 15 May 2013 12:40:19 +0000 (UTC) Received: from [172.16.6.25] (p579A60A2.dip0.t-ipconnect.de [87.154.96.162]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by fop.bsdsystems.de (Postfix) with ESMTP id 2F3BC54BF7 for ; Wed, 15 May 2013 14:40:18 +0200 (CEST) Subject: still mbuf leak in 9.0 / 9.1? Mime-Version: 1.0 (Apple Message framework v1085) Content-Type: multipart/mixed; boundary=Apple-Mail-36--527606426 From: dennis berger Resent-From: dennis berger Date: Wed, 15 May 2013 11:00:23 +0200 Resent-Date: Wed, 15 May 2013 14:40:17 +0200 Resent-To: freebsd-stable@freebsd.org Message-Id: To: freebsd-stable@freebsd.org X-Mailer: Apple Mail (2.1085) Resent-Message-Id: <20130515124018.2F3BC54BF7@fop.bsdsystems.de> X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 May 2013 12:40:20 -0000 --Apple-Mail-36--527606426 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 Hi list, since we activated 10gbe on ixgbe cards + jumbo frames(9k) on 9.0 and = now on 9.1 we recognize that after a random period of time, sometimes a = week, sometimes only a day, the system doesn't send any packets out. The phenomenon is that you can't = login via ssh, nfs and istgt is not operative. Yet you can login on the = console and execute commands. A clean shutdown isn't possible though. It hangs after vnode cleaning, = normally you would see detaching of usb devices here, or other devices = maybe?=20 I've read the other post on this ML about mbuf leak in the arp handling = code in if_ether.c line 558. We don't see any of those notices in dmesg = so I don't think that glebius fix would apply for us. I'm collecting system and memory information every hour.=20 Script looks like this. less /etc/periodic/hourly/100.report-memory.sh=20 #!/bin/sh reporttimestamp=3D`date +%d-%m-%Y-%H-%M` reportname=3D${reporttimestamp}.txt cd /root/memory-mon top -b > $reportname echo "" >> $reportname vmstat -m >> $reportname echo "" >> $reportname vmstat -z >> $reportname echo "" >> $reportname netstat -Q >> $reportname echo "" >> $reportname netstat -n -x >> $reportname echo "" >> $reportname netstat -m >> $reportname /usr/bin/perl /usr/local/bin/zfs-stats -a >> $reportname When you grep for mbuf or mbuf usage you will see this for example: root@freenas:/root/memory-mon # grep mbuf_packet: * 14-05-2013-14-09.txt:mbuf_packet: 256, 0, 9246, = 2786,201700429, 0, 0 14-05-2013-15-09.txt:mbuf_packet: 256, 0, 9256, = 2776,201773122, 0, 0 14-05-2013-16-09.txt:mbuf_packet: 256, 0, 9266, = 2766,201871553, 0, 0 14-05-2013-17-09.txt:mbuf_packet: 256, 0, 9276, = 2756,201915405, 0, 0 14-05-2013-18-09.txt:mbuf_packet: 256, 0, 9286, = 2746,201927956, 0, 0 14-05-2013-19-09.txt:mbuf_packet: 256, 0, 9296, = 2352,201935681, 0, 0 14-05-2013-20-09.txt:mbuf_packet: 256, 0, 9306, = 2342,201943754, 0, 0 14-05-2013-21-09.txt:mbuf_packet: 256, 0, 9316, = 2332,201950961, 0, 0 14-05-2013-22-09.txt:mbuf_packet: 256, 0, 9326, = 2450,201958150, 0, 0 14-05-2013-23-09.txt:mbuf_packet: 256, 0, 9336, = 2440,201967178, 0, 0 15-05-2013-00-09.txt:mbuf_packet: 256, 0, 9346, = 2430,201974561, 0, 0 15-05-2013-01-09.txt:mbuf_packet: 256, 0, 9356, = 2420,201982105, 0, 0 15-05-2013-02-09.txt:mbuf_packet: 256, 0, 9366, = 2410,201989463, 0, 0 15-05-2013-03-09.txt:mbuf_packet: 256, 0, 9378, = 1502,203019168, 0, 0 15-05-2013-04-09.txt:mbuf_packet: 256, 0, 9384, = 1624,205953601, 0, 0 15-05-2013-05-09.txt:mbuf_packet: 256, 0, 9394, = 1870,205959258, 0, 0 15-05-2013-06-09.txt:mbuf_packet: 256, 0, 9404, = 2500,205969396, 0, 0 15-05-2013-07-09.txt:mbuf_packet: 256, 0, 9414, = 3386,207945161, 0, 0 15-05-2013-08-09.txt:mbuf_packet: 256, 0, 9424, = 3376,208094689, 0, 0 15-05-2013-09-09.txt:mbuf_packet: 256, 0, 9434, = 2982,208172465, 0, 0 15-05-2013-10-09.txt:mbuf_packet: 256, 0, 9444, = 3100,208270369, 0, 0 and root@freenas:/root/memory-mon # grep "mbufs in use" * 14-05-2013-14-09.txt:58444/5816/64260 mbufs in use (current/cache/total) 14-05-2013-15-09.txt:58455/5805/64260 mbufs in use (current/cache/total) 14-05-2013-16-09.txt:58464/5796/64260 mbufs in use (current/cache/total) 14-05-2013-17-09.txt:58475/5785/64260 mbufs in use (current/cache/total) 14-05-2013-18-09.txt:58484/5776/64260 mbufs in use (current/cache/total) 14-05-2013-19-09.txt:58493/5767/64260 mbufs in use (current/cache/total) 14-05-2013-20-09.txt:58503/5757/64260 mbufs in use (current/cache/total) 14-05-2013-21-09.txt:58513/5747/64260 mbufs in use (current/cache/total) 14-05-2013-22-09.txt:58523/5737/64260 mbufs in use (current/cache/total) 14-05-2013-23-09.txt:58533/5727/64260 mbufs in use (current/cache/total) 15-05-2013-00-09.txt:58543/5717/64260 mbufs in use (current/cache/total) 15-05-2013-01-09.txt:58554/5706/64260 mbufs in use (current/cache/total) 15-05-2013-02-09.txt:58563/5697/64260 mbufs in use (current/cache/total) 15-05-2013-03-09.txt:58639/5621/64260 mbufs in use (current/cache/total) 15-05-2013-04-09.txt:58581/5679/64260 mbufs in use (current/cache/total) 15-05-2013-05-09.txt:58591/5669/64260 mbufs in use (current/cache/total) 15-05-2013-06-09.txt:58602/5658/64260 mbufs in use (current/cache/total) 15-05-2013-07-09.txt:58613/5647/64260 mbufs in use (current/cache/total) 15-05-2013-08-09.txt:58623/6027/64650 mbufs in use (current/cache/total) 15-05-2013-09-09.txt:58634/6016/64650 mbufs in use (current/cache/total) 15-05-2013-10-09.txt:58645/6005/64650 mbufs in use (current/cache/total) This increasing number of used mbuf_packets and mbufs in use makes me = nervous. See the complete reports http://knownhosts.org:/reports-14-15.tgz=20 Thanks for help, -dennis --------------BEGIN System information--------------- It's a stock FreeBSD 9.1, yet the hostname is called freenas. Don't be = confused. igb0: flags=3D8c02 metric 0 mtu = 1500 = options=3D401bb ether 00:25:90:34:c1:12 nd6 options=3D21 media: Ethernet autoselect (1000baseT ) status: active igb1: flags=3D8843 metric 0 mtu = 1500 = options=3D401bb ether 00:25:90:34:c1:13 inet 172.16.1.6 netmask 0xfffff000 broadcast 172.16.15.255 inet6 fe80::225:90ff:fe34:c113%igb1 prefixlen 64 scopeid 0x2=20 nd6 options=3D21 media: Ethernet autoselect (1000baseT ) status: active ix0: flags=3D8843 metric 0 mtu = 9000 = options=3D401bb ether 00:1b:21:cc:12:8b inet 10.254.254.242 netmask 0xfffffffc broadcast 10.254.254.243 inet6 fe80::21b:21ff:fecc:128b%ix0 prefixlen 64 scopeid 0xb=20 nd6 options=3D21 media: Ethernet autoselect (10Gbase-Twinax ) status: active ix1: flags=3D8843 metric 0 mtu = 9000 = options=3D401bb ether 00:1b:21:cc:12:8a inet 10.254.254.254 netmask 0xfffffffc broadcast 10.254.254.255 inet6 fe80::21b:21ff:fecc:128a%ix1 prefixlen 64 scopeid 0xc=20 nd6 options=3D21 media: Ethernet autoselect (10Gbase-Twinax ) status: active ix2: flags=3D8843 metric 0 mtu = 9000 = options=3D401bb ether 00:1b:21:cc:12:b3 inet 10.254.254.246 netmask 0xfffffffc broadcast 10.254.254.247 inet6 fe80::21b:21ff:fecc:12b3%ix2 prefixlen 64 scopeid 0xd=20 nd6 options=3D21 media: Ethernet autoselect status: no carrier ix3: flags=3D8802 metric 0 mtu 1500 = options=3D401bb ether 00:1b:21:cc:12:b2 nd6 options=3D21 media: Ethernet autoselect status: no carrier lo0: flags=3D8049 metric 0 mtu 16384 options=3D600003 inet6 ::1 prefixlen 128=20 inet6 fe80::1%lo0 prefixlen 64 scopeid 0xf=20 inet 127.0.0.1 netmask 0xff000000=20 nd6 options=3D21 #dmesg =85.. mfi0: 21294 (421879975s/0x0008/info) - Battery started charging ix1: link state changed to DOWN ix1: link state changed to UP ix1: link state changed to DOWN ix1: link state changed to UP ix1: link state changed to DOWN ix1: link state changed to UP ix1: link state changed to DOWN ix1: link state changed to UP ix1: link state changed to DOWN ix1: link state changed to UP ix1: link state changed to DOWN ix1: link state changed to UP ix1: link state changed to DOWN ix1: link state changed to UP ix1: link state changed to DOWN ix1: link state changed to UP ix1: link state changed to DOWN ix1: link state changed to UP ix1: link state changed to DOWN ix1: link state changed to UP ix1: link state changed to DOWN ix1: link state changed to UP ix1: link state changed to DOWN ix1: link state changed to UP ix1: link state changed to DOWN ix1: link state changed to UP ix1: link state changed to DOWN ix1: link state changed to UP ix1: link state changed to DOWN ix1: link state changed to UP ix1: link state changed to DOWN ix1: link state changed to UP ix1: link state changed to DOWN ix1: link state changed to UP ix1: link state changed to DOWN ix1: link state changed to UP ix1: link state changed to DOWN ix1: link state changed to UP ix1: link state changed to DOWN ix1: link state changed to UP ix1: link state changed to DOWN ix1: link state changed to UP I should add that the servers that are directly connected to this = freebsd server reboot every night. This is why you see ix0 UP/DOWN messages in dmesg. --Apple-Mail-36--527606426 Content-Disposition: attachment; filename=dmesg.boot Content-Type: application/octet-stream; name="dmesg.boot" Content-Transfer-Encoding: 7bit Copyright (c) 1992-2012 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.1-RELEASE #0 r243825: Tue Dec 4 09:23:10 UTC 2012 root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 can't re-use a leaf (geom_label)! module_register: module g_label already exists! Module g_label failed to register: 17 CPU: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz (2400.14-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x206c2 Family = 6 Model = 2c Stepping = 2 Features=0xbfebfbff Features2=0x29ee3ff AMD Features=0x2c100800 AMD Features2=0x1 TSC: P-state invariant, performance statistics real memory = 51543801856 (49156 MB) avail memory = 49657782272 (47357 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 18 cpu5 (AP): APIC ID: 19 cpu6 (AP): APIC ID: 20 cpu7 (AP): APIC ID: 21 cpu8 (AP): APIC ID: 32 cpu9 (AP): APIC ID: 33 cpu10 (AP): APIC ID: 34 cpu11 (AP): APIC ID: 35 cpu12 (AP): APIC ID: 50 cpu13 (AP): APIC ID: 51 cpu14 (AP): APIC ID: 52 cpu15 (AP): APIC ID: 53 ioapic0: Changing APIC ID to 6 ioapic1: Changing APIC ID to 7 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard ioapic2 irqs 48-71 on motherboard kbd1 at kbdmux0 ctl: CAM Target Layer loaded acpi0: on motherboard acpi0: Power Button (fixed) acpi0: reservation of 400, 100 (3) failed cpu0: on acpi0 ACPI Warning: Incorrect checksum in table [OEMB] - 0x98, should be 0x95 (20110527/tbutils-282) cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cpu4: on acpi0 cpu5: on acpi0 cpu6: on acpi0 cpu7: on acpi0 cpu8: on acpi0 cpu9: on acpi0 cpu10: on acpi0 cpu11: on acpi0 cpu12: on acpi0 cpu13: on acpi0 cpu14: on acpi0 cpu15: on acpi0 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: port 0x70-0x71 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 450 Event timer "HPET1" frequency 14318180 Hz quality 440 Event timer "HPET2" frequency 14318180 Hz quality 440 Event timer "HPET3" frequency 14318180 Hz quality 440 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 igb0: port 0xcc00-0xcc1f mem 0xfafe0000-0xfaffffff,0xfafc0000-0xfafdffff,0xfaf9c000-0xfaf9ffff irq 28 at device 0.0 on pci1 igb0: Using MSIX interrupts with 9 vectors igb0: Ethernet address: 00:25:90:34:c1:12 igb0: Bound queue 0 to cpu 0 igb0: Bound queue 1 to cpu 1 igb0: Bound queue 2 to cpu 2 igb0: Bound queue 3 to cpu 3 igb0: Bound queue 4 to cpu 4 igb0: Bound queue 5 to cpu 5 igb0: Bound queue 6 to cpu 6 igb0: Bound queue 7 to cpu 7 igb1: port 0xc800-0xc81f mem 0xfaf60000-0xfaf7ffff,0xfaf40000-0xfaf5ffff,0xfaf98000-0xfaf9bfff irq 40 at device 0.1 on pci1 igb1: Using MSIX interrupts with 9 vectors igb1: Ethernet address: 00:25:90:34:c1:13 igb1: Bound queue 0 to cpu 8 igb1: Bound queue 1 to cpu 9 igb1: Bound queue 2 to cpu 10 igb1: Bound queue 3 to cpu 11 igb1: Bound queue 4 to cpu 12 igb1: Bound queue 5 to cpu 13 igb1: Bound queue 6 to cpu 14 igb1: Bound queue 7 to cpu 15 pcib2: at device 3.0 on pci0 pci2: on pcib2 pcib3: at device 5.0 on pci0 pci3: on pcib3 pcib4: at device 7.0 on pci0 pci4: on pcib4 pcib5: at device 9.0 on pci0 pci5: on pcib5 pci0: at device 20.0 (no driver attached) pci0: at device 20.1 (no driver attached) pci0: at device 20.2 (no driver attached) pci0: at device 20.3 (no driver attached) pci0: at device 22.0 (no driver attached) pci0: at device 22.1 (no driver attached) pci0: at device 22.2 (no driver attached) pci0: at device 22.3 (no driver attached) pci0: at device 22.4 (no driver attached) pci0: at device 22.5 (no driver attached) pci0: at device 22.6 (no driver attached) pci0: at device 22.7 (no driver attached) uhci0: port 0xaf80-0xaf9f irq 16 at device 26.0 on pci0 uhci0: LegSup = 0x2f00 usbus0 on uhci0 uhci1: port 0xaf40-0xaf5f irq 21 at device 26.1 on pci0 uhci1: LegSup = 0x2f00 usbus1 on uhci1 uhci2: port 0xaf20-0xaf3f irq 19 at device 26.2 on pci0 uhci2: LegSup = 0x2f00 usbus2 on uhci2 ehci0: mem 0xfaefc000-0xfaefc3ff irq 18 at device 26.7 on pci0 usbus3: EHCI version 1.0 usbus3 on ehci0 uhci3: port 0xaf00-0xaf1f irq 23 at device 29.0 on pci0 uhci3: LegSup = 0x2f00 usbus4 on uhci3 uhci4: port 0xaec0-0xaedf irq 19 at device 29.1 on pci0 uhci4: LegSup = 0x2f00 usbus5 on uhci4 uhci5: port 0xaea0-0xaebf irq 18 at device 29.2 on pci0 uhci5: LegSup = 0x2f00 usbus6 on uhci5 ehci1: mem 0xfaeda000-0xfaeda3ff irq 23 at device 29.7 on pci0 usbus7: EHCI version 1.0 usbus7 on ehci1 pcib6: at device 30.0 on pci0 pci6: on pcib6 vgapci0: mem 0xf9000000-0xf9ffffff,0xfbefc000-0xfbefffff,0xfb000000-0xfb7fffff irq 16 at device 4.0 on pci6 isab0: at device 31.0 on pci0 isa0: on isab0 ahci0: port 0xaff0-0xaff7,0xafac-0xafaf,0xafe0-0xafe7,0xafa8-0xafab,0xae80-0xae9f mem 0xfaed8000-0xfaed87ff irq 19 at device 31.2 on pci0 ahci0: AHCI v1.20 with 6 3Gbps ports, Port Multiplier not supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 ahcich4: at channel 4 on ahci0 ahcich5: at channel 5 on ahci0 pci0: at device 31.3 (no driver attached) pcib7: on acpi0 pci128: on pcib7 pcib8: at device 0.0 on pci128 pci129: on pcib8 pcib9: at device 1.0 on pci128 pci130: on pcib9 pcib10: at device 3.0 on pci128 pci131: on pcib10 ix0: port 0xdc00-0xdc1f mem 0xf8da0000-0xf8dbffff,0xf8dc0000-0xf8dfffff,0xf8d9c000-0xf8d9ffff irq 48 at device 0.0 on pci131 ix0: Using MSIX interrupts with 9 vectors ix0: Ethernet address: 00:1b:21:cc:12:8b ix0: PCI Express Bus: Speed 2.5Gb/s Width x8 ix1: port 0xd800-0xd81f mem 0xf8d20000-0xf8d3ffff,0xf8d40000-0xf8d7ffff,0xf8d98000-0xf8d9bfff irq 58 at device 0.1 on pci131 ix1: Using MSIX interrupts with 9 vectors ix1: Ethernet address: 00:1b:21:cc:12:8a ix1: PCI Express Bus: Speed 2.5Gb/s Width x8 pcib11: at device 5.0 on pci128 pci132: on pcib11 ix2: port 0xec00-0xec1f mem 0xf8ea0000-0xf8ebffff,0xf8ec0000-0xf8efffff,0xf8e9c000-0xf8e9ffff irq 50 at device 0.0 on pci132 ix2: Using MSIX interrupts with 9 vectors ix2: Ethernet address: 00:1b:21:cc:12:b3 ix2: PCI Express Bus: Speed 2.5Gb/s Width x8 ix3: port 0xe800-0xe81f mem 0xf8e20000-0xf8e3ffff,0xf8e40000-0xf8e7ffff,0xf8e98000-0xf8e9bfff irq 49 at device 0.1 on pci132 ix3: Using MSIX interrupts with 9 vectors ix3: No SFP+ Module found ix3: Ethernet address: 00:1b:21:cc:12:b2 ix3: PCI Express Bus: Speed 2.5Gb/s Width x8 pcib12: at device 7.0 on pci128 pci133: on pcib12 mfi0: port 0xf800-0xf8ff mem 0xf8f9c000-0xf8f9ffff,0xf8fc0000-0xf8ffffff irq 54 at device 0.0 on pci133 mfi0: Using MSI mfi0: Megaraid SAS driver Ver 4.23 mfip0: on mfi0 pcib13: at device 9.0 on pci128 pci134: on pcib13 pci128: at device 20.0 (no driver attached) pci128: at device 20.1 (no driver attached) pci128: at device 20.2 (no driver attached) pci128: at device 20.3 (no driver attached) pci128: at device 22.0 (no driver attached) pci128: at device 22.1 (no driver attached) pci128: at device 22.2 (no driver attached) pci128: at device 22.3 (no driver attached) pci128: at device 22.4 (no driver attached) pci128: at device 22.5 (no driver attached) pci128: at device 22.6 (no driver attached) pci128: at device 22.7 (no driver attached) acpi_button0: on acpi0 ipmi0: port 0xca2-0xca3 on acpi0 ipmi0: KCS mode found at io 0xca2 on acpi atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: model IntelliMouse Explorer, device ID 4 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 qpi0: on motherboard pcib14: pcibus 255 on qpi0 pci255: on pcib14 pcib15: pcibus 254 on qpi0 pci254: on pcib15 ipmi1: on isa0 device_attach: ipmi1 attach returned 16 ipmi1: on isa0 device_attach: ipmi1 attach returned 16 orm0: at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ppc0: cannot reserve I/O port range est0: on cpu0 p4tcc0: on cpu0 est1: on cpu1 p4tcc1: on cpu1 est2: on cpu2 p4tcc2: on cpu2 est3: on cpu3 p4tcc3: on cpu3 est4: on cpu4 p4tcc4: on cpu4 est5: on cpu5 p4tcc5: on cpu5 est6: on cpu6 p4tcc6: on cpu6 est7: on cpu7 p4tcc7: on cpu7 est8: on cpu8 p4tcc8: on cpu8 est9: on cpu9 p4tcc9: on cpu9 est10: on cpu10 p4tcc10: on cpu10 est11: on cpu11 p4tcc11: on cpu11 est12: on cpu12 p4tcc12: on cpu12 est13: on cpu13 p4tcc13: on cpu13 est14: on cpu14 p4tcc14: on cpu14 est15: on cpu15 p4tcc15: on cpu15 Timecounters tick every 1.000 msec mfi0: 19571 (417203816s/0x0020/info) - Shutdown command received from host mfi0: 19572 (boot + 3s/0x0020/info) - Firmware initialization started (PCI ID 0079/1000/9260/1000) mfi0: 19573 (boot + 3s/0x0020/info) - Firmware version 2.130.363-1846 mfi0: 19574 (boot + 9s/0x0008/info) - Battery Present mfi0: 19575 (boot + 9s/0x0020/info) - Package version 12.12.0-0124 mfi0: 19576 (boot + 9s/0x0020/info) - Board Revision 61A mfi0: 19577 (boot + 27s/0x0004/info) - Enclosure (SES) discovered on PD 16(c Port 0 - 3/p1) mfi0: 19578 (boot + 27s/0x0004/info) - Enclosure PD 16(c Port 0 - 3/p1) communication restored mfi0: 19579 (boot + 27s/0x0002/info) - Inserted: Encl PD 16 mfi0: 19580 (boot + 27s/0x0002/info) - Inserted: PD 16(c Port 0 - 3/p1) Info: enclPd=16, scsiType=d, portMap=00, sasAddr=500304800000007d,0000000000000000 mfi0: 19581 (boot + 27s/0x0002/info) - Inserted: PD 04(e0x16/s0) mfi0: 19582 (boot + 27s/0x0002/info) - Inserted: PD 04(e0x16/s0) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500304800000004c,0000000000000000 mfi0: 19583 (boot + 27s/0x0002/info) - Inserted: PD 05(e0x16/s1) mfi0: 19584 (boot + 27s/0x0002/info) - Inserted: PD 05(e0x16/s1) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500304800000004d,0000000000000000 mfi0: 19585 (boot + 27s/0x0002/info) - Inserted: PD 06(e0x16/s2) mfi0: 19586 (boot + 27s/0x0002/info) - Inserted: PD 06(e0x16/s2) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=5000c5003a5439ed,0000000000000000 mfi0: 19587 (boot + 27s/0x0002/info) - Inserted: PD 07(e0x16/s3) mfi0: 19588 (boot + 27s/0x0002/info) - Inserted: PD 07(e0x16/s3) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=5000c5003a4dcafd,0000000000000000 mfi0: 19589 (boot + 27s/0x0002/info) - Inserted: PD 08(e0x16/s4) mfi0: 19590 (boot + 27s/0x0002/info) - Inserted: PD 08(e0x16/s4) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=5000c50039d51041,0000000000000000 mfi0: 19591 (boot + 27s/0x0002/info) - Inserted: PD 09(e0x16/s5) mfi0: 19592 (boot + 27s/0x0002/info) - Inserted: PD 09(e0x16/s5) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d41a,0000000000000000 mfi0: 19593 (boot + 27s/0x0002/info) - Inserted: PD 0a(e0x16/s6) mfi0: 19594 (boot + 27s/0x0002/info) - Inserted: PD 0a(e0x16/s6) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d3be,0000000000000000 mfi0: 19595 (boot + 27s/0x0002/info) - Inserted: PD 0b(e0x16/s7) mfi0: 19596 (boot + 27s/0x0002/info) - Inserted: PD 0b(e0x16/s7) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d372,0000000000000000 mfi0: 19597 (boot + 27s/0x0002/info) - Inserted: PD 0c(e0x16/s8) mfi0: 19598 (boot + 27s/0x0002/info) - Inserted: PD 0c(e0x16/s8) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d3ea,0000000000000000 mfi0: 19599 (boot + 27s/0x0002/info) - Inserted: PD 0d(e0x16/s9) mfi0: 19600 (boot + 27s/0x0002/info) - Inserted: PD 0d(e0x16/s9) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d37a,0000000000000000 mfi0: 19601 (boot + 27s/0x0002/info) - Inserted: PD 0e(e0x16/s10) mfi0: 19602 (boot + 27s/0x0002/info) - Inserted: PD 0e(e0x16/s10) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d352,0000000000000000 mfi0: 19603 (boot + 27s/0x0002/info) - Inserted: PD 0f(e0x16/s11) mfi0: 19604 (boot + 27s/0x0002/info) - Inserted: PD 0f(e0x16/s11) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d406,0000000000000000 mfi0: 19605 (boot + 27s/0x0002/info) - Inserted: PD 10(e0x16/s12) mfi0: 19606 (boot + 27s/0x0002/info) - Inserted: PD 10(e0x16/s12) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d33e,0000000000000000 mfi0: 19607 (boot + 27s/0x0002/info) - Inserted: PD 11(e0x16/s13) mfi0: 19608 (boot + 27s/0x0002/info) - Inserted: PD 11(e0x16/s13) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d3de,0000000000000000 mfi0: 19609 (boot + 27s/0x0002/info) - Inserted: PD 12(e0x16/s14) mfi0: 19610 (boot + 27s/0x0002/info) - Inserted: PD 12(e0x16/s14) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d3da,0000000000000000 mfi0: 19611 (boot + 27s/0x0002/info) - Inserted: PD 13(e0x16/s15) mfi0: 19612 (boot + 27s/0x0002/info) - Inserted: PD 13(e0x16/s15) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d37e,0000000000000000 mfi0: 19613 (boot + 27s/0x0002/info) - Inserted: PD 14(e0x16/s16) mfi0: 19614 (boot + 27s/0x0002/info) - Inserted: PD 14(e0x16/s16) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d3e6,0000000000000000 mfi0: 19615 (boot + 27s/0x0002/info) - Inserted: PD 15(e0x16/s17) mfi0: 19616 (boot + 27s/0x0002/info) - Inserted: PD 15(e0x16/s17) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d33a,0000000000000000 mfi0: 19617 (boot + 27s/0x0002/info) - Inserted: PD 17(e0x16/s18) mfi0: 19618 (boot + 27s/0x0002/info) - Inserted: PD 17(e0x16/s18) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500304800000005e,0000000000000000 mfi0: 19619 (boot + 27s/0x0002/info) - Inserted: PD 18(e0x16/s19) mfi0: 19620 (boot + 27s/0x0002/info) - Inserted: PD 18(e0x16/s19) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500304800000005f,0000000000000000 mfi0: 19621 (417203877s/0x0020/info) - Time established as 03/21/13 17:57:57; (46 seconds since power on) mfi0: 19622 (417203895s/0x0008/info) - Battery started charging mfi0: 19623 (417203895s/0x0008/info) - Battery temperature is normal mfi0: 19624 (417203926s/0x0020/info) - Host driver is loaded and operational mfi0: 19625 (417203936s/0x0002/info) - Unexpected sense: Encl PD 16 Path 500304800000007d, CDB: 1a 00 0a 00 14 00, Sense: 5/24/00 mfi0: 19626 (417259210s/0x0008/info) - Battery relearn will start in 2 day mfi0: 19627 (417322797s/0x0020/info) - Patrol Read started mfi0: 19808 (417344758s/0x0020/info) - Patrol Read complete mfi0: 19809 (417345595s/0x0008/info) - Battery relearn will start in 1 day mfi0: 19810 (417413975s/0x0008/info) - Battery relearn will start in 5 hours mfi0: 19811 (417427560s/0x0008/info) - Battery charge complete mfi0: 19812 (417927597s/0x0020/info) - Patrol Read started mfi0: 20011 (417951400s/0x0020/info) - Patrol Read complete mfi0: 20012 (418050975s/0x0008/info) - Battery started charging mfi0: 20013 (418164335s/0x0008/info) - Battery charge complete mfi0: 20014 (418282570s/0x0008/info) - Battery started charging mfi0: 20015 (418324234s/0x0002/WARN) - Removed: PD 17(e0x16/s18) mfi0: 20016 (418324234s/0x0002/info) - Removed: PD 17(e0x16/s18) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500304800000005e,0000000000000000 mfi0: 20017 (418324234s/0x0002/info) - State change on PD 17(e0x16/s18) from ONLINE(18) to FAILED(11) mfi0: 20018 (418324234s/0x0001/info) - State change on VD 00/10 from OPTIMAL(3) to OFFLINE(0) mfi0: 20019 (418324234s/0x0001/FATAL) - VD 00/10 is now OFFLINE mfi0: 20020 (418324235s/0x0002/info) - State change on PD 17(e0x16/s18) from FAILED(11) to UNCONFIGURED_BAD(1) mfi0: 20021 (418324235s/0x0041/info) - Deleted VD 00/10 mfi0: 20022 (418324244s/0x0002/info) - Inserted: PD 17(e0x16/s18) mfi0: 20023 (418324244s/0x0002/info) - Inserted: PD 17(e0x16/s18) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500304800000005e,0000000000000000 mfi0: 20024 (418532397s/0x0020/info) - Patrol Read started mfi0: 20212 (418553611s/0x0020/info) - Patrol Read complete mfi0: 20213 (418901370s/0x0008/info) - Battery charge complete mfi0: 20214 (419019670s/0x0008/info) - Battery started charging mfi0: 20215 (419137197s/0x0020/info) - Patrol Read started mfi0: 20403 (419159066s/0x0020/info) - Patrol Read complete mfi0: 20404 (419638470s/0x0008/info) - Battery charge complete mfi0: 20405 (boot + 3s/0x0020/info) - Firmware initialization started (PCI ID 0079/1000/9260/1000) mfi0: 20406 (boot + 3s/0x0020/info) - Firmware version 2.130.363-1846 mfi0: 20407 (boot + 9s/0x0008/info) - Battery Present mfi0: 20408 (boot + 9s/0x0020/info) - Package version 12.12.0-0124 mfi0: 20409 (boot + 9s/0x0020/info) - Board Revision 61A mfi0: 20410 (boot + 28s/0x0004/info) - Enclosure (SES) discovered on PD 16(c Port 0 - 3/p1) mfi0: 20411 (boot + 28s/0x0004/info) - Enclosure PD 16(c Port 0 - 3/p1) communication restored mfi0: 20412 (boot + 28s/0x0002/info) - Inserted: Encl PD 16 mfi0: 20413 (boot + 28s/0x0002/info) - Inserted: PD 16(c Port 0 - 3/p1) Info: enclPd=16, scsiType=d, portMap=00, sasAddr=500304800000007d,0000000000000000 mfi0: 20414 (boot + 28s/0x0002/info) - Inserted: PD 04(e0x16/s0) mfi0: 20415 (boot + 28s/0x0002/info) - Inserted: PD 04(e0x16/s0) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500304800000004c,0000000000000000 mfi0: 20416 (boot + 28s/0x0002/info) - Inserted: PD 05(e0x16/s1) mfi0: 20417 (boot + 28s/0x0002/info) - Inserted: PD 05(e0x16/s1) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500304800000004d,0000000000000000 mfi0: 20418 (boot + 28s/0x0002/info) - Inserted: PD 06(e0x16/s2) mfi0: 20419 (boot + 28s/0x0002/info) - Inserted: PD 06(e0x16/s2) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=5000c5003a5439ed,0000000000000000 mfi0: 20420 (boot + 28s/0x0002/info) - Inserted: PD 07(e0x16/s3) mfi0: 20421 (boot + 28s/0x0002/info) - Inserted: PD 07(e0x16/s3) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=5000c5003a4dcafd,0000000000000000 mfi0: 20422 (boot + 28s/0x0002/info) - Inserted: PD 08(e0x16/s4) mfi0: 20423 (boot + 28s/0x0002/info) - Inserted: PD 08(e0x16/s4) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=5000c50039d51041,0000000000000000 mfi0: 20424 (boot + 28s/0x0002/info) - Inserted: PD 09(e0x16/s5) mfi0: 20425 (boot + 28s/0x0002/info) - Inserted: PD 09(e0x16/s5) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d41a,0000000000000000 mfi0: 20426 (boot + 28s/0x0002/info) - Inserted: PD 0a(e0x16/s6) mfi0: 20427 (boot + 28s/0x0002/info) - Inserted: PD 0a(e0x16/s6) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d3be,0000000000000000 mfi0: 20428 (boot + 28s/0x0002/info) - Inserted: PD 0b(e0x16/s7) mfi0: 20429 (boot + 28s/0x0002/info) - Inserted: PD 0b(e0x16/s7) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d372,0000000000000000 mfi0: 20430 (boot + 28s/0x0002/info) - Inserted: PD 0c(e0x16/s8) mfi0: 20431 (boot + 28s/0x0002/info) - Inserted: PD 0c(e0x16/s8) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d3ea,0000000000000000 mfi0: 20432 (boot + 28s/0x0002/info) - Inserted: PD 0d(e0x16/s9) mfi0: 20433 (boot + 28s/0x0002/info) - Inserted: PD 0d(e0x16/s9) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d37a,0000000000000000 mfi0: 20434 (boot + 28s/0x0002/info) - Inserted: PD 0e(e0x16/s10) mfi0: 20435 (boot + 28s/0x0002/info) - Inserted: PD 0e(e0x16/s10) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d352,0000000000000000 mfi0: 20436 (boot + 28s/0x0002/info) - Inserted: PD 0f(e0x16/s11) mfi0: 20437 (boot + 28s/0x0002/info) - Inserted: PD 0f(e0x16/s11) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d406,0000000000000000 mfi0: 20438 (boot + 28s/0x0002/info) - Inserted: PD 10(e0x16/s12) mfi0: 20439 (boot + 28s/0x0002/info) - Inserted: PD 10(e0x16/s12) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d33e,0000000000000000 mfi0: 20440 (boot + 28s/0x0002/info) - Inserted: PD 11(e0x16/s13) mfi0: 20441 (boot + 28s/0x0002/info) - Inserted: PD 11(e0x16/s13) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d3de,0000000000000000 mfi0: 20442 (boot + 28s/0x0002/info) - Inserted: PD 12(e0x16/s14) mfi0: 20443 (boot + 28s/0x0002/info) - Inserted: PD 12(e0x16/s14) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d3da,0000000000000000 mfi0: 20444 (boot + 28s/0x0002/info) - Inserted: PD 13(e0x16/s15) mfi0: 20445 (boot + 28s/0x0002/info) - Inserted: PD 13(e0x16/s15) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d37e,0000000000000000 mfi0: 20446 (boot + 28s/0x0002/info) - Inserted: PD 14(e0x16/s16) mfi0: 20447 (boot + 28s/0x0002/info) - Inserted: PD 14(e0x16/s16) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d3e6,0000000000000000 mfi0: 20448 (boot + 28s/0x0002/info) - Inserted: PD 15(e0x16/s17) mfi0: 20449 (boot + 28s/0x0002/info) - Inserted: PD 15(e0x16/s17) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d33a,0000000000000000 mfi0: 20450 (boot + 28s/0x0002/info) - Inserted: PD 17(e0x16/s18) mfi0: 20451 (boot + 28s/0x0002/info) - Inserted: PD 17(e0x16/s18) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500304800000005e,0000000000000000 mfi0: 20452 (boot + 28s/0x0002/info) - Inserted: PD 18(e0x16/s19) mfi0: 20453 (boot + 28s/0x0002/info) - Inserted: PD 18(e0x16/s19) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500304800000005f,0000000000000000 mfi0: 20454 (boot + 64s/0x0008/info) - Battery temperature is normal mfi0: 20455 (419670570s/0x0020/info) - Time established as 04/19/13 7:09:30; (74 seconds since power on) mfi0: 20456 (419670625s/0x0008/info) - Periodic Battery Relearn was missed, and rescheduled to 04/21/13 9:19:30. mfi0: 20457 (419670627s/0x0020/info) - Host driver is loaded and operational mfi0: 20458 (419670637s/0x0002/info) - Unexpected sense: Encl PD 16 Path 500304800000007d, CDB: 1a 00 0a 00 14 00, Sense: 5/24/00 mfi0: 20459 (419678425s/0x0008/info) - Battery relearn will start in 2 day mfi0: 20460 (419741997s/0x0020/info) - Patrol Read started mfi0: 20569 (419760845s/0x0008/info) - Battery started charging mfi0: 20632 (419764409s/0x0020/info) - Patrol Read complete mfi0: 20633 (419764810s/0x0008/info) - Battery relearn will start in 1 day mfi0: 20634 (419833190s/0x0008/info) - Battery relearn will start in 5 hours mfi0: 20635 (420346797s/0x0020/info) - Patrol Read started mfi0: 20823 (420372124s/0x0020/info) - Patrol Read complete mfi0: 20824 (420379775s/0x0008/info) - Battery charge complete mfi0: 20825 (boot + 3s/0x0020/info) - Firmware initialization started (PCI ID 0079/1000/9260/1000) mfi0: 20826 (boot + 3s/0x0020/info) - Firmware version 2.130.363-1846 mfi0: 20827 (boot + 9s/0x0008/info) - Battery Present mfi0: 20828 (boot + 9s/0x0020/info) - Package version 12.12.0-0124 mfi0: 20829 (boot + 9s/0x0020/info) - Board Revision 61A mfi0: 20830 (boot + 28s/0x0004/info) - Enclosure (SES) discovered on PD 16(c Port 0 - 3/p1) mfi0: 20831 (boot + 28s/0x0004/info) - Enclosure PD 16(c Port 0 - 3/p1) communication restored mfi0: 20832 (boot + 28s/0x0002/info) - Inserted: Encl PD 16 mfi0: 20833 (boot + 28s/0x0002/info) - Inserted: PD 16(c Port 0 - 3/p1) Info: enclPd=16, scsiType=d, portMap=00, sasAddr=500304800000007d,0000000000000000 mfi0: 20834 (boot + 28s/0x0002/info) - Inserted: PD 04(e0x16/s0) mfi0: 20835 (boot + 28s/0x0002/info) - Inserted: PD 04(e0x16/s0) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500304800000004c,0000000000000000 mfi0: 20836 (boot + 28s/0x0002/info) - Inserted: PD 05(e0x16/s1) mfi0: 20837 (boot + 28s/0x0002/info) - Inserted: PD 05(e0x16/s1) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500304800000004d,0000000000000000 mfi0: 20838 (boot + 28s/0x0002/info) - Inserted: PD 06(e0x16/s2) mfi0: 20839 (boot + 28s/0x0002/info) - Inserted: PD 06(e0x16/s2) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=5000c5003a5439ed,0000000000000000 mfi0: 20840 (boot + 28s/0x0002/info) - Inserted: PD 07(e0x16/s3) mfi0: 20841 (boot + 28s/0x0002/info) - Inserted: PD 07(e0x16/s3) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=5000c5003a4dcafd,0000000000000000 mfi0: 20842 (boot + 28s/0x0002/info) - Inserted: PD 08(e0x16/s4) mfi0: 20843 (boot + 28s/0x0002/info) - Inserted: PD 08(e0x16/s4) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=5000c50039d51041,0000000000000000 mfi0: 20844 (boot + 28s/0x0002/info) - Inserted: PD 09(e0x16/s5) mfi0: 20845 (boot + 28s/0x0002/info) - Inserted: PD 09(e0x16/s5) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d41a,0000000000000000 mfi0: 20846 (boot + 28s/0x0002/info) - Inserted: PD 0a(e0x16/s6) mfi0: 20847 (boot + 28s/0x0002/info) - Inserted: PD 0a(e0x16/s6) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d3be,0000000000000000 mfi0: 20848 (boot + 28s/0x0002/info) - Inserted: PD 0b(e0x16/s7) mfi0: 20849 (boot + 28s/0x0002/info) - Inserted: PD 0b(e0x16/s7) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d372,0000000000000000 mfi0: 20850 (boot + 28s/0x0002/info) - Inserted: PD 0c(e0x16/s8) mfi0: 20851 (boot + 28s/0x0002/info) - Inserted: PD 0c(e0x16/s8) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d3ea,0000000000000000 mfi0: 20852 (boot + 28s/0x0002/info) - Inserted: PD 0d(e0x16/s9) mfi0: 20853 (boot + 28s/0x0002/info) - Inserted: PD 0d(e0x16/s9) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d37a,0000000000000000 mfi0: 20854 (boot + 28s/0x0002/info) - Inserted: PD 0e(e0x16/s10) mfi0: 20855 (boot + 28s/0x0002/info) - Inserted: PD 0e(e0x16/s10) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d352,0000000000000000 mfi0: 20856 (boot + 28s/0x0002/info) - Inserted: PD 0f(e0x16/s11) mfi0: 20857 (boot + 28s/0x0002/info) - Inserted: PD 0f(e0x16/s11) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d406,0000000000000000 mfi0: 20858 (boot + 28s/0x0002/info) - Inserted: PD 10(e0x16/s12) mfi0: 20859 (boot + 28s/0x0002/info) - Inserted: PD 10(e0x16/s12) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d33e,0000000000000000 mfi0: 20860 (boot + 28s/0x0002/info) - Inserted: PD 11(e0x16/s13) mfi0: 20861 (boot + 28s/0x0002/info) - Inserted: PD 11(e0x16/s13) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d3de,0000000000000000 mfi0: 20862 (boot + 28s/0x0002/info) - Inserted: PD 12(e0x16/s14) mfi0: 20863 (boot + 28s/0x0002/info) - Inserted: PD 12(e0x16/s14) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d3da,0000000000000000 mfi0: 20864 (boot + 28s/0x0002/info) - Inserted: PD 13(e0x16/s15) mfi0: 20865 (boot + 28s/0x0002/info) - Inserted: PD 13(e0x16/s15) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d37e,0000000000000000 mfi0: 20866 (boot + 28s/0x0002/info) - Inserted: PD 14(e0x16/s16) mfi0: 20867 (boot + 28s/0x0002/info) - Inserted: PD 14(e0x16/s16) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d3e6,0000000000000000 mfi0: 20868 (boot + 28s/0x0002/info) - Inserted: PD 15(e0x16/s17) mfi0: 20869 (boot + 28s/0x0002/info) - Inserted: PD 15(e0x16/s17) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d33a,0000000000000000 mfi0: 20870 (boot + 28s/0x0002/info) - Inserted: PD 17(e0x16/s18) mfi0: 20871 (boot + 28s/0x0002/info) - Inserted: PD 17(e0x16/s18) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500304800000005e,0000000000000000 mfi0: 20872 (boot + 28s/0x0002/info) - Inserted: PD 18(e0x16/s19) mfi0: 20873 (boot + 28s/0x0002/info) - Inserted: PD 18(e0x16/s19) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500304800000005f,0000000000000000 mfi0: 20874 (420788325s/0x0020/info) - Time established as 05/02/13 5:38:45; (62 seconds since power on) mfi0: 20875 (420788327s/0x0008/info) - Battery started charging mfi0: 20876 (420788327s/0x0008/info) - Battery temperature is normal mfi0: 20877 (420788327s/0x0008/info) - Periodic Battery Relearn was missed, and rescheduled to 05/05/13 9:19:30. mfi0: 20878 (420788374s/0x0020/info) - Host driver is loaded and operational mfi0: 20879 (420788384s/0x0002/info) - Unexpected sense: Encl PD 16 Path 500304800000007d, CDB: 1a 00 0a 00 14 00, Sense: 5/24/00 mfi0: 20880 (420887972s/0x0008/info) - Battery relearn will start in 2 day mfi0: 20881 (420951597s/0x0020/info) - Patrol Read started mfi0: 21052 (420965623s/0x0020/info) - Patrol Read complete mfi0: 21053 (420974422s/0x0008/info) - Battery relearn will start in 1 day mfi0: 21054 (421015567s/0x0008/info) - Battery charge complete mfi0: 21055 (421042802s/0x0008/info) - Battery relearn will start in 5 hours mfi0: 21056 (421133867s/0x0008/info) - Battery started charging mfi0: 21057 (boot + 3s/0x0020/info) - Firmware initialization started (PCI ID 0079/1000/9260/1000) mfi0: 21058 (boot + 3s/0x0020/info) - Firmware version 2.130.363-1846 mfi0: 21059 (boot + 9s/0x0008/info) - Battery Present mfi0: 21060 (boot + 9s/0x0020/info) - Package version 12.12.0-0124 mfi0: 21061 (boot + 9s/0x0020/info) - Board Revision 61A mfi0: 21062 (boot + 28s/0x0004/info) - Enclosure (SES) discovered on PD 16(c Port 0 - 3/p1) mfi0: 21063 (boot + 28s/0x0004/info) - Enclosure PD 16(c Port 0 - 3/p1) communication restored mfi0: 21064 (boot + 28s/0x0002/info) - Inserted: Encl PD 16 The GEOM class LABEL is already loaded. usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 480Mbps High Speed USB v2.0 usbus4: 12Mbps Full Speed USB v1.0 usbus5: 12Mbps Full Speed USB v1.0 usbus6: 12Mbps Full Speed USB v1.0 usbus7: 480Mbps High Speed USB v2.0 ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 ugen4.1: at usbus4 uhub4: on usbus4 ugen5.1: at usbus5 uhub5: on usbus5 ugen6.1: at usbus6 uhub6: on usbus6 ugen7.1: at usbus7 uhub7: on usbus7 mfid0 on mfi0 mfid0: 285568MB (584843264 sectors) RAID volume (no label) is optimal mfid1 on mfi0 mfid1: 285568MB (584843264 sectors) RAID volume (no label) is optimal mfid2 on mfi0 mfid2: 285568MB (584843264 sectors) RAID volume (no label) is optimal mfid3 on mfi0 mfid3: 285568MB (584843264 sectors) RAID volume (no label) is optimal mfid4 on mfi0 mfid4: 285568MB (584843264 sectors) RAID volume (no label) is optimal mfid5 on mfi0 mfid5: 285568MB (584843264 sectors) RAID volume (no label) is optimal mfid6 on mfi0 mfid6: 285568MB (584843264 sectors) RAID volume (no label) is optimal mfid7 on mfi0 mfid7: 285568MB (584843264 sectors) RAID volume (no label) is optimal mfid8 on mfi0 mfid8: 285568MB (584843264 sectors) RAID volume (no label) is optimal mfid9 on mfi0 mfid9: 285568MB (584843264 sectors) RAID volume (no label) is optimal mfid10 on mfi0 mfid10: 285568MB (584843264 sectors) RAID volume (no label) is optimal mfid11 on mfi0 mfid11: 285568MB (584843264 sectors) RAID volume (no label) is optimal mfid12 on mfi0 mfid12: 285568MB (584843264 sectors) RAID volume (no label) is optimal mfid13 on mfi0 mfid13: 285568MB (584843264 sectors) RAID volume (no label) is optimal mfid14 on mfi0 mfid14: 285568MB (584843264 sectors) RAID volume (no label) is optimal mfid15 on mfi0 mfid15: 285568MB (584843264 sectors) RAID volume (no label) is optimal mfid16 on mfi0 mfid16: 953344MB (1952448512 sectors) RAID volume (no label) is optimal mfid17 on mfi0 mfid17: 113952MB (233373696 sectors) RAID volume (no label) is optimal mfid18 on mfi0 mfid18: 113952MB (233373696 sectors) RAID volume (no label) is optimal mfi0: 21065 (boot + 28s/0x0002/info) - Inserted: PD 16(c Port 0 - 3/p1) Info: enclPd=16, scsiType=d, portMap=00, sasAddr=500304800000007d,0000000000000000 mfi0: 21066 (boot + 28s/0x0002/info) - Inserted: PD 04(e0x16/s0) mfi0: 21067 (boot + 28s/0x0002/info) - Inserted: PD 04(e0x16/s0) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500304800000004c,0000000000000000 mfi0: 21068 (boot + 28s/0x0002/info) - Inserted: PD 05(e0x16/s1) mfi0: 21069 (boot + 28s/0x0002/info) - Inserted: PD 05(e0x16/s1) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500304800000004d,0000000000000000 mfi0: 21070 (boot + 28s/0x0002/info) - Inserted: PD 06(e0x16/s2) mfi0: 21071 (boot + 28s/0x0002/info) - Inserted: PD 06(e0x16/s2) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=5000c5003a5439ed,0000000000000000 mfi0: 21072 (boot + 28s/0x0002/info) - Inserted: PD 07(e0x16/s3) mfi0: 21073 (boot + 28s/0x0002/info) - Inserted: PD 07(e0x16/s3) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=5000c5003a4dcafd,0000000000000000 mfi0: 21074 (boot + 28s/0x0002/info) - Inserted: PD 08(e0x16/s4) mfi0: 21075 (boot + 28s/0x0002/info) - Inserted: PD 08(e0x16/s4) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=5000c50039d51041,0000000000000000 mfi0: 21076 (boot + 28s/0x0002/info) - Inserted: PD 09(e0x16/s5) mfi0: 21077 (boot + 28s/0x0002/info) - Inserted: PD 09(e0x16/s5) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d41a,0000000000000000 mfi0: 21078 (boot + 28s/0x0002/info) - Inserted: PD 0a(e0x16/s6) mfi0: 21079 (boot + 28s/0x0002/info) - Inserted: PD 0a(e0x16/s6) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d3be,0000000000000000 mfi0: 21080 (boot + 28s/0x0002/info) - Inserted: PD 0b(e0x16/s7) mfi0: 21081 (boot + 28s/0x0002/info) - Inserted: PD 0b(e0x16/s7) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d372,0000000000000000 mfi0: 21082 (boot + 28s/0x0002/info) - Inserted: PD 0c(e0x16/s8) mfi0: 21083 (boot + 28s/0x0002/info) - Inserted: PD 0c(e0x16/s8) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d3ea,0000000000000000 mfi0: 21084 (boot + 28s/0x0002/info) - Inserted: PD 0d(e0x16/s9) mfi0: 21085 (boot + 28s/0x0002/info) - Inserted: PD 0d(e0x16/s9) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d37a,0000000000000000 mfi0: 21086 (boot + 28s/0x0002/info) - Inserted: PD 0e(e0x16/s10) mfi0: 21087 (boot + 28s/0x0002/info) - Inserted: PD 0e(e0x16/s10) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d352,0000000000000000 mfi0: 21088 (boot + 28s/0x0002/info) - Inserted: PD 0f(e0x16/s11) mfi0: 21089 (boot + 28s/0x0002/info) - Inserted: PD 0f(e0x16/s11) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d406,0000000000000000 mfi0: 21090 (boot + 28s/0x0002/info) - Inserted: PD 10(e0x16/s12) mfi0: 21091 (boot + 28s/0x0002/info) - Inserted: PD 10(e0x16/s12) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d33e,0000000000000000 mfi0: 21092 (boot + 28s/0x0002/info) - Inserted: PD 11(e0x16/s13) mfi0: 21093 (boot + 28s/0x0002/info) - Inserted: PD 11(e0x16/s13) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d3de,0000000000000000 mfi0: 21094 (boot + 28s/0x0002/info) - Inserted: PD 12(e0x16/s14) mfi0: 21095 (boot + 28s/0x0002/info) - Inserted: PD 12(e0x16/s14) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d3da,0000000000000000 mfi0: 21096 (boot + 28s/0x0002/info) - Inserted: PD 13(e0x16/s15) mfi0: 21097 (boot + 28s/0x0002/info) - Inserted: PD 13(e0x16/s15) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d37e,0000000000000000 mfi0: 21098 (boot + 28s/0x0002/info) - Inserted: PD 14(e0x16/s16) mfi0: 21099 (boot + 28s/0x0002/info) - Inserted: PD 14(e0x16/s16) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d3e6,0000000000000000 mfi0: 21100 (boot + 28s/0x0002/info) - Inserted: PD 15(e0x16/s17) mfi0: 21101 (boot + 28s/0x0002/info) - Inserted: PD 15(e0x16/s17) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500003937813d33a,0000000000000000 mfi0: 21102 (boot + 28s/0x0002/info) - Inserted: PD 17(e0x16/s18) mfi0: 21103 (boot + 28s/0x0002/info) - Inserted: PD 17(e0x16/s18) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500304800000005e,0000000000000000 mfi0: 21104 (boot + 28s/0x0002/info) - Inserted: PD 18(e0x16/s19) mfi0: 21105 (boot + 28s/0x0002/info) - Inserted: PD 18(e0x16/s19) Info: enclPd=16, scsiType=0, portMap=00, sasAddr=500304800000005f,0000000000000000 mfi0: 21106 (421479053s/0x0020/info) - Time established as 05/10/13 5:30:53; (62 seconds since power on) mfi0: 21107 (421479055s/0x0008/info) - Battery started charging mfi0: 21108 (421479055s/0x0008/info) - Battery temperature is normal mfi0: 21109 (421479055s/0x0008/info) - Periodic Battery Relearn was missed, and rescheduled to 05/12/13 9:19:30. mfi0: 21110 (421479109s/0x0020/info) - Host driver is loaded and operational ipmi0: IPMI device rev. 1, firmware rev. 2.04, version 2.0 ipmi0: Number of channels 2 ipmi0: Attached watchdog GEOM: mfid2: corrupt or invalid GPT detected. GEOM: mfid2: GPT rejected -- may not be recoverable. GEOM: mfid3: corrupt or invalid GPT detected. GEOM: mfid3: GPT rejected -- may not be recoverable. uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered uhub5: 2 ports with 2 removable, self powered GEOM: mfid4: corrupt or invalid GPT detected. GEOM: mfid4: GPT rejected -- may not be recoverable. uhub6: 2 ports with 2 removable, self powered GEOM: mfid5: corrupt or invalid GPT detected. GEOM: mfid5: GPT rejected -- may not be recoverable. GEOM: mfid6: corrupt or invalid GPT detected. GEOM: mfid6: GPT rejected -- may not be recoverable. GEOM: mfid7: corrupt or invalid GPT detected. GEOM: mfid7: GPT rejected -- may not be recoverable. GEOM: mfid8: corrupt or invalid GPT detected. GEOM: mfid8: GPT rejected -- may not be recoverable. GEOM: mfid9: corrupt or invalid GPT detected. GEOM: mfid9: GPT rejected -- may not be recoverable. GEOM: mfid10: corrupt or invalid GPT detected. GEOM: mfid10: GPT rejected -- may not be recoverable. GEOM: mfid11: corrupt or invalid GPT detected. GEOM: mfid11: GPT rejected -- may not be recoverable. GEOM: mfid12: corrupt or invalid GPT detected. GEOM: mfid12: GPT rejected -- may not be recoverable. GEOM: mfid13: corrupt or invalid GPT detected. GEOM: mfid13: GPT rejected -- may not be recoverable. GEOM: mfid14: corrupt or invalid GPT detected. GEOM: mfid14: GPT rejected -- may not be recoverable. GEOM: mfid15: corrupt or invalid GPT detected. GEOM: mfid15: GPT rejected -- may not be recoverable. uhub3: 6 ports with 6 removable, self powered uhub7: 6 ports with 6 removable, self powered mfi0: 21111 (421479119s/0x0002/info) - Unexpected sense: Encl PD 16 Path 500304800000007d, CDB: 1a 00 0a 00 14 00, Sense: 5/24/00 ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: ATA-7 SATA 2.x device ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 512bytes) ada0: 3775MB (7732368 512 byte sectors: 16H 63S/T 7671C) ada0: Previously was known as ad4 ses0 at mfi0 bus 0 scbus6 target 22 lun 0 ses0: Fixed Enclosure Services SCSI-5 device ses0: 150.000MB/s transfers ses0: SCSI-3 SES Device pass1 at mfi0 bus 0 scbus6 target 4 lun 0 pass1: Fixed Uninstalled SCSI-5 device pass1: 150.000MB/s transfers pass2 at mfi0 bus 0 scbus6 target 5 lun 0 pass2: Fixed Uninstalled SCSI-5 device pass2: 150.000MB/s transfers pass3 at mfi0 bus 0 scbus6 target 6 lun 0 pass3: Fixed Uninstalled SCSI-5 device pass3: 150.000MB/s transfers pass4 at mfi0 bus 0 scbus6 target 7 lun 0 pass4: Fixed Uninstalled SCSI-5 device pass4: 150.000MB/s transfers pass5 at mfi0 bus 0 scbus6 target 8 lun 0 pass5: Fixed Uninstalled SCSI-5 device pass5: 150.000MB/s transfers pass6 at mfi0 bus 0 scbus6 target 9 lun 0 pass6: Fixed Uninstalled SCSI-5 device pass6: 150.000MB/s transfers pass7 at mfi0 bus 0 scbus6 target 10 lun 0 pass7: Fixed Uninstalled SCSI-5 device pass7: 150.000MB/s transfers pass8 at mfi0 bus 0 scbus6 target 11 lun 0 pass8: Fixed Uninstalled SCSI-5 device pass8: 150.000MB/s transfers pass9 at mfi0 bus 0 scbus6 target 12 lun 0 pass9: Fixed Uninstalled SCSI-5 device pass9: 150.000MB/s transfers pass10 at mfi0 bus 0 scbus6 target 13 lun 0 pass10: Fixed Uninstalled SCSI-5 device pass10: 150.000MB/s transfers pass11 at mfi0 bus 0 scbus6 target 14 lun 0 pass11: Fixed Uninstalled SCSI-5 device pass11: 150.000MB/s transfers pass12 at mfi0 bus 0 scbus6 target 15 lun 0 pass12: Fixed Uninstalled SCSI-5 device pass12: 150.000MB/s transfers pass13 at mfi0 bus 0 scbus6 target 16 lun 0 pass13: Fixed Uninstalled SCSI-5 device pass13: 150.000MB/s transfers pass14 at mfi0 bus 0 scbus6 target 17 lun 0 pass14: Fixed Uninstalled SCSI-5 device pass14: 150.000MB/s transfers pass15 at mfi0 bus 0 scbus6 target 18 lun 0 pass15: Fixed Uninstalled SCSI-5 device pass15: 150.000MB/s transfers pass16 at mfi0 bus 0 scbus6 target 19 lun 0 pass16: Fixed Uninstalled SCSI-5 device pass16: 150.000MB/s transfers pass17 at mfi0 bus 0 scbus6 target 20 lun 0 pass17: Fixed Uninstalled SCSI-5 device pass17: 150.000MB/s transfers pass18 at mfi0 bus 0 scbus6 target 21 lun 0 pass18: Fixed Uninstalled SCSI-5 device pass18: 150.000MB/s transfers pass20 at mfi0 bus 0 scbus6 target 23 lun 0 pass20: Fixed Uninstalled SCSI-5 device pass20: 150.000MB/s transfers pass21 at mfi0 bus 0 scbus6 target 24 lun 0 pass21: Fixed Uninstalled SCSI-5 device pass21: 150.000MB/s transfers SMP: AP CPU #1 Launched! SMP: AP CPU #15 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #11 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #12 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #8 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #13 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #9 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #14 Launched! SMP: AP CPU #10 Launched! Timecounter "TSC-low" frequency 9375545 Hz quality 1000 Trying to mount root from ufs:/dev/ufs/rootfs []... WARNING: / was not properly dismounted ugen1.2: at usbus1 ukbd0: on usbus1 kbd2 at ukbd0 ums0: on usbus1 ZFS filesystem version 5 ZFS storage pool version 28 --Apple-Mail-36--527606426 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii ------------- END System information------------ --Apple-Mail-36--527606426--