From owner-freebsd-stable@FreeBSD.ORG Fri Jun 15 05:18:55 2012 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 583F0106564A for ; Fri, 15 Jun 2012 05:18:55 +0000 (UTC) (envelope-from cowens@greatbaysoftware.com) Received: from ecbiz102.inmotionhosting.com (ecbiz102.inmotionhosting.com [70.39.235.94]) by mx1.freebsd.org (Postfix) with ESMTP id D5EF68FC0C for ; Fri, 15 Jun 2012 05:18:54 +0000 (UTC) Received: from c-50-136-23-27.hsd1.nh.comcast.net ([50.136.23.27]:64638 helo=jack.bspruce.com) by ecbiz102.inmotionhosting.com with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.69) (envelope-from ) id 1SfO9X-0005ix-5F for stable@freebsd.org; Fri, 15 Jun 2012 00:28:55 -0400 Message-ID: <4FDABA0B.5030702@greatbaysoftware.com> Date: Fri, 15 Jun 2012 00:28:59 -0400 From: Charles Owens Organization: Great Bay Software User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 To: stable@freebsd.org X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - ecbiz102.inmotionhosting.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - greatbaysoftware.com Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: mfi(4) IO performance regression, post 8.1 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Jun 2012 05:18:55 -0000 Hello FreeBSD folk, We're seeing what appears to be a storage performance regression as we try to move from 8.1 (i386) to 8.3. We looked at 8.2 also and it appears that the regression happened between 8.1 and 8.2. Our system is an Intel S5520UR Server with 12 GB RAM, dual 4-core CPUs. Storage is a LSI MegaSAS 1078 controller (mfi) in a RAID-10 configuration, using UFS + geom_journal for filesystem. Postgresql performance, as seen via pgbench, dropped by approx 20%. This testing was done with our usual PAE-enabled kernels. We then went back to GENERIC kernels and did comparisons using "bonnie", results below. Following that is a kernel boot log. Notably, we're seeing this regression only with our RAID mfi(4) based systems. Notably, from looking at FreeBSD source changelogs it appears that the mfi(4) code has seen some changes since 8.1. How can I investigate further? Assistance with sorting this out would be greatly appreciated. Thanks much, Charles Bonnie comparison 8.3 GENERIC -------Sequential Output-------- ---Sequential Input-- --Random-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks--- Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU 100 93580 98.3 82136 26.6 113709 78.8 152081 98.8 3223876 100.0 233590.3 240.4 8.2 GENERIC -------Sequential Output-------- ---Sequential Input-- --Random-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks--- Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU 100 95923 99.1 84042 56.7 110568 69.8 152088 100.6 4290802 82.8 239779.4 234.5 8.1 GENERIC -------Sequential Output-------- ---Sequential Input-- --Random-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks--- Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU 100 140708 100.0 164261 44.4 208553 48.5 153472 100.0 3298756 100.0 270325.1 238.4 Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.1-RELEASE-p8 #1: Fri Jan 6 12:13:34 EST 2012 cowens@newcastle.greatbaysoftware.com:/usr/obj/usr/relbuild/os/RELENG_8_1/sys/GENERIC i386 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz (2394.28-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x106a5 Family = 6 Model = 1a Stepping = 5 Features=0xbfebfbff Features2=0x9ce3bd AMD Features=0x28100000 AMD Features2=0x1 TSC: P-state invariant real memory = 12884901888 (12288 MB) avail memory = 2289147904 (2183 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 6 cpu7 (AP): APIC ID: 7 cpu8 (AP): APIC ID: 16 cpu9 (AP): APIC ID: 17 cpu10 (AP): APIC ID: 18 cpu11 (AP): APIC ID: 19 cpu12 (AP): APIC ID: 20 cpu13 (AP): APIC ID: 21 cpu14 (AP): APIC ID: 22 cpu15 (AP): APIC ID: 23 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard lapic0: Forcing LINT1 to edge trigger kbd0 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0:<24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cpu4: on acpi0 cpu5: on acpi0 cpu6: on acpi0 cpu7: on acpi0 cpu8: on acpi0 cpu9: on acpi0 cpu10: on acpi0 cpu11: on acpi0 cpu12: on acpi0 cpu13: on acpi0 cpu14: on acpi0 cpu15: on acpi0 acpi_hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 900 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: irq 28 at device 1.0 on pci0 pci1: on pcib1 igb0: port 0x4020-0x403f mem 0xb1f20000-0xb1f3ffff,0xb1f44000-0xb1f47fff irq 40 at device 0.0 on pci1 igb0: Using MSIX interrupts with 5 vectors igb0: [ITHREAD] igb0: [ITHREAD] igb0: [ITHREAD] igb0: [ITHREAD] igb0: [ITHREAD] igb0: Ethernet address: 00:15:17:f2:1b:a0 igb1: port 0x4000-0x401f mem 0xb1f00000-0xb1f1ffff,0xb1f40000-0xb1f43fff irq 28 at device 0.1 on pci1 igb1: Using MSIX interrupts with 5 vectors igb1: [ITHREAD] igb1: [ITHREAD] igb1: [ITHREAD] igb1: [ITHREAD] igb1: [ITHREAD] igb1: Ethernet address: 00:15:17:f2:1b:a1 pcib2: irq 24 at device 3.0 on pci0 pci2: on pcib2 pcib3: mem 0xb1e00000-0xb1e1ffff irq 24 at device 0.0 on pci2 pci3: on pcib3 pcib4: irq 24 at device 0.0 on pci3 pci4: on pcib4 igb2: port 0x3020-0x303f mem 0xb1c60000-0xb1c7ffff,0xb1c40000-0xb1c5ffff,0xb1d04000-0xb1d07fff irq 34 at device 0.0 on pci4 igb2: Using MSIX interrupts with 10 vectors igb2: [ITHREAD] igb2: [ITHREAD] igb2: [ITHREAD] igb2: [ITHREAD] igb2: [ITHREAD] igb2: [ITHREAD] igb2: [ITHREAD] igb2: [ITHREAD] igb2: [ITHREAD] igb2: [ITHREAD] igb2: Ethernet address: 00:0c:bd:00:9d:eb igb3: port 0x3000-0x301f mem 0xb1c20000-0xb1c3ffff,0xb1c00000-0xb1c1ffff,0xb1d00000-0xb1d03fff irq 24 at device 0.1 on pci4 igb3: Using MSIX interrupts with 10 vectors igb3: [ITHREAD] igb3: [ITHREAD] igb3: [ITHREAD] igb3: [ITHREAD] igb3: [ITHREAD] igb3: [ITHREAD] igb3: [ITHREAD] igb3: [ITHREAD] igb3: [ITHREAD] igb3: [ITHREAD] igb3: Ethernet address: 00:0c:bd:00:9d:ea pcib5: irq 34 at device 1.0 on pci3 pci7: on pcib5 igb4: port 0x2020-0x203f mem 0xb1a60000-0xb1a7ffff,0xb1a40000-0xb1a5ffff,0xb1b04000-0xb1b07fff irq 35 at device 0.0 on pci7 igb4: Using MSIX interrupts with 10 vectors igb4: [ITHREAD] igb4: [ITHREAD] igb4: [ITHREAD] igb4: [ITHREAD] igb4: [ITHREAD] igb4: [ITHREAD] igb4: [ITHREAD] igb4: [ITHREAD] igb4: [ITHREAD] igb4: [ITHREAD] igb4: Ethernet address: 00:0c:bd:00:9d:ed igb5: port 0x2000-0x201f mem 0xb1a20000-0xb1a3ffff,0xb1a00000-0xb1a1ffff,0xb1b00000-0xb1b03fff irq 34 at device 0.1 on pci7 igb5: Using MSIX interrupts with 10 vectors igb5: [ITHREAD] igb5: [ITHREAD] igb5: [ITHREAD] igb5: [ITHREAD] igb5: [ITHREAD] igb5: [ITHREAD] igb5: [ITHREAD] igb5: [ITHREAD] igb5: [ITHREAD] igb5: [ITHREAD] igb5: Ethernet address: 00:0c:bd:00:9d:ec pcib6: irq 24 at device 4.0 on pci3 pci10: on pcib6 pcib7: irq 34 at device 5.0 on pci3 pci11: on pcib7 pcib8: irq 30 at device 7.0 on pci0 pci12: on pcib8 pcib9: irq 32 at device 9.0 on pci0 pci13: on pcib9 pcib10: irq 33 at device 10.0 on pci0 pci14: on pcib10 pci0: at device 16.0 (no driver attached) pci0: at device 16.1 (no driver attached) pci0: at device 17.0 (no driver attached) pci0: at device 17.1 (no driver attached) pci0: at device 20.0 (no driver attached) pci0: at device 20.1 (no driver attached) pci0: at device 20.2 (no driver attached) pci0: at device 20.3 (no driver attached) pci0: at device 22.0 (no driver attached) pci0: at device 22.1 (no driver attached) pci0: at device 22.2 (no driver attached) pci0: at device 22.3 (no driver attached) pci0: at device 22.4 (no driver attached) pci0: at device 22.5 (no driver attached) pci0: at device 22.6 (no driver attached) pci0: at device 22.7 (no driver attached) uhci0: port 0x50c0-0x50df irq 19 at device 26.0 on pci0 uhci0: [ITHREAD] uhci0: LegSup = 0x3000 usbus0: on uhci0 uhci1: port 0x50a0-0x50bf irq 19 at device 26.1 on pci0 uhci1: [ITHREAD] usbus1: on uhci1 uhci2: port 0x5080-0x509f irq 19 at device 26.2 on pci0 uhci2: [ITHREAD] usbus2: on uhci2 ehci0: mem 0xb2021000-0xb20213ff irq 19 at device 26.7 on pci0 ehci0: [ITHREAD] usbus3: EHCI version 1.0 usbus3: on ehci0 pcib11: irq 16 at device 28.0 on pci0 pci15: on pcib11 mfi0: port 0x1000-0x10ff mem 0xb1900000-0xb193ffff,0xb1940000-0xb197ffff irq 16 at device 0.0 on pci15 mfi0: Megaraid SAS driver Ver 3.00 mfi0: 36759 (393046626s/0x0020/info) - Shutdown command received from host mfi0: 36760 (boot + 3s/0x0020/info) - Firmware initialization started (PCI ID 0060/1000/3505/8086) mfi0: 36761 (boot + 3s/0x0020/info) - Firmware version 1.40.232-1007 mfi0: 36762 (boot + 4s/0x0008/info) - Battery Present mfi0: 36763 (boot + 5s/0x0020/info) - Board Revision mfi0: 36764 (boot + 49s/0x0002/WARN) - PD 110(e0x00/s16) Path 50001017e03bdc00 reset (Type 03) mfi0: 36765 (boot + 65s/0x0002/WARN) - PD 111(e0x00/s17) Path 50001017e13bdc00 reset (Type 03) mfi0: 36766 (boot + 66s/0x0002/WARN) - PD 110(e0x00/s16) Path 50001017e03bdc00 reset (Type 03) mfi0: 36767 (boot + 82s/0x0002/WARN) - PD 110(e0x00/s16) Path 50001017e03bdc00 reset (Type 03) mfi0: 36768 (boot + 83s/0x0002/WARN) - PD 111(e0x00/s17) Path 50001017e13bdc00 reset (Type 03) mfi0: 36769 (boot + 99s/0x0002/WARN) - PD 111(e0x00/s17) Path 50001017e13bdc00 reset (Type 03) mfi0: 36770 (boot + 100s/0x0002/WARN) - PD 110(e0x00/s16) Path 50001017e03bdc00 reset (Type 03) mfi0: 36771 (boot + 104s/0x0002/info) - Inserted: PD 01(e0xff/s1) mfi0: 36772 (boot + 104s/0x0002/info) - Inserted: PD 01(e0xff/s1) Info: enclPd=ffff, scsiType=0, portMap=00, sasAddr=5000c5001ab0d441,0000000000000000 mfi0: 36773 (boot + 104s/0x0002/info) - Inserted: PD 02(e0xff/s2) mfi0: 36774 (boot + 104s/0x0002/info) - Inserted: PD 02(e0xff/s2) Info: enclPd=ffff, scsiType=0, portMap=01, sasAddr=5000c5001ab0d4ed,0000000000000000 mfi0: 36775 (boot + 104s/0x0002/info) - Inserted: PD 03(e0xff/s3) mfi0: 36776 (boot + 104s/0x0002/info) - Inserted: PD 03(e0xff/s3) Info: enclPd=ffff, scsiType=0, portMap=04, sasAddr=5000c5001ab0d585,0000000000000000 mfi0: 36777 (boot + 104s/0x0002/info) - Inserted: PD 04(e0xff/s4) mfi0: 36778 (boot + 104s/0x0002/info) - Inserted: PD 04(e0xff/s4) Info: enclPd=ffff, scsiType=0, portMap=02, sasAddr=5000c5001ab0d869,0000000000000000 mfi0: 36779 (boot + 104s/0x0002/info) - Inserted: PD 05(e0xff/s5) mfi0: 36780 (boot + 104s/0x0002/info) - Inserted: PD 05(e0xff/s5) Info: enclPd=ffff, scsiType=0, portMap=03, sasAddr=5000c5001ab0d655,0000000000000000 mfi0: 36781 (boot + 104s/0x0042/info) - Global Hot Spare created on PD 05(e0xff/s5) (global,rev) mfi0: 36782 (393046751s/0x0020/info) - Time established as 06/15/12 3:39:11; (113 seconds since power on) mfi0: [ITHREAD] pcib12: irq 16 at device 28.4 on pci0 pci16: on pcib12 vgapci0: mem 0xb0000000-0xb0ffffff,0xb1800000-0xb1803fff,0xb1000000-0xb17fffff irq 16 at device 0.0 on pci16 pcib13: irq 17 at device 28.5 on pci0 pci17: on pcib13 uhci3: port 0x5060-0x507f irq 16 at device 29.0 on pci0 uhci3: [ITHREAD] usbus4: on uhci3 uhci4: port 0x5040-0x505f irq 16 at device 29.1 on pci0 uhci4: [ITHREAD] usbus5: on uhci4 uhci5: port 0x5020-0x503f irq 16 at device 29.2 on pci0 uhci5: [ITHREAD] usbus6: on uhci5 ehci1: mem 0xb2020000-0xb20203ff irq 16 at device 29.7 on pci0 ehci1: [ITHREAD] usbus7: EHCI version 1.0 usbus7: on ehci1 pcib14: at device 30.0 on pci0 pci18: on pcib14 isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x5138-0x513f,0x514c-0x514f,0x5130-0x5137,0x5148-0x514b,0x5110-0x511f,0x5100-0x510f irq 18 at device 31.2 on pci0 atapci0: [ITHREAD] ata2: on atapci0 ata2: [ITHREAD] ata3: on atapci0 ata3: [ITHREAD] pci0: at device 31.3 (no driver attached) atapci1: port 0x5128-0x512f,0x5144-0x5147,0x5120-0x5127,0x5140-0x5143,0x50f0-0x50ff,0x50e0-0x50ef irq 21 at device 31.5 on pci0 atapci1: [ITHREAD] ata4: on atapci1 ata4: [ITHREAD] ata5: on atapci1 ata5: [ITHREAD] acpi_button0: on acpi0 atrtc0: port 0x70-0x71,0x74-0x77 irq 8 on acpi0 uart0:<16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] uart0: console (9600,n,8,1) uart1:<16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart1: [FILTER] pmtimer0 on isa0 orm0: at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff,0xc9000-0xc9fff,0xca000-0xcffff pnpid ORM0000 on isa0 sc0: at flags 0x100 on isa0 sc0: VGA<16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ata0 at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0 ata0: [ITHREAD] ata1 at port 0x170-0x177,0x376 irq 15 on isa0 ata1: [ITHREAD] ppc0: parallel port not found. est0: on cpu0 p4tcc0: on cpu0 est1: on cpu1 p4tcc1: on cpu1 est2: on cpu2 p4tcc2: on cpu2 est3: on cpu3 p4tcc3: on cpu3 est4: on cpu4 p4tcc4: on cpu4 est5: on cpu5 p4tcc5: on cpu5 est6: on cpu6 p4tcc6: on cpu6 est7: on cpu7 p4tcc7: on cpu7 est8: on cpu8 p4tcc8: on cpu8 est9: on cpu9 p4tcc9: on cpu9 est10: on cpu10 p4tcc10: on cpu10 est11: on cpu11 p4tcc11: on cpu11 est12: on cpu12 p4tcc12: on cpu12 est13: on cpu13 p4tcc13: on cpu13 est14: on cpu14 p4tcc14: on cpu14 est15: on cpu15 p4tcc15: on cpu15 Timecounters tick every 1.000 msec mfi0: 36783 (393046780s/0x0008/info) - Battery temperature is normal usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 480Mbps High Speed USB v2.0 usbus4: 12Mbps Full Speed USB v1.0 usbus5: 12Mbps Full Speed USB v1.0 usbus6: 12Mbps Full Speed USB v1.0 usbus7: 480Mbps High Speed USB v2.0 mfid0: on mfi0 mfid0: 278472MB (570310656 sectors) RAID volume '' is optimal lapic18: Forcing LINT1 to edge trigger SMP: AP CPU #10 Launched! lapic5: Forcing LINT1 to edge trigger SMP: AP CPU #5 Launched! lapic17: Forcing LINT1 to edge trigger SMP: AP CPU #9 Launched! lapic3: Forcing LINT1 to edge trigger SMP: AP CPU #3 Launched! lapic6: Forcing LINT1 to edge trigger SMP: AP CPU #6 Launched! lapic23: Forcing LINT1 to edge trigger SMP: AP CPU #15 Launched! lapic22: Forcing LINT1 to edge trigger SMP: AP CPU #14 Launched! lapic1: Forcing LINT1 to edge trigger SMP: AP CPU #1 Launched! lapic4: Forcing LINT1 to edge trigger SMP: AP CPU #4 Launched! lapic2: Forcing LINT1 to edge trigger SMP: AP CPU #2 Launched! lapic16: Forcing LINT1 to edge trigger SMP: AP CPU #8 Launched! lapic21: Forcing LINT1 to edge trigger SMP: AP CPU #13 Launched! lapic19: Forcing LINT1 to edge trigger SMP: AP CPU #11 Launched! lapic20: Forcing LINT1 to edge trigger SMP: AP CPU #12 Launched! lapic7: Forcing LINT1 to edge trigger SMP: AP CPU #7 Launched! ugen2.1: at usbus2ugen5.1: at usbus5ugen7.1: at usbus7ugen1.1: at usbus1ugen4.1: at usbus4ugen6.1: at usbus6ugen0.1: at usbus0ugen3.1: at usbus3 uhub0: on usbus2 uhub1: on usbus5 uhub2: on usbus0 uhub3: on usbus7 uhub4: on usbus6 uhub5: on usbus3 uhub6: on usbus1 uhub7: on usbus4 GEOM_JOURNAL: Journal 3746672352: mfid0s1a contains data. GEOM_JOURNAL: Journal 3746672352: mfid0s1a contains journal. GEOM_JOURNAL: Journal mfid0s1a clean. GEOM_JOURNALuhub0: : 2 ports with 2 removable, self poweredBIO_FLUSH not supported by mfid0s1a. uhub2: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self poweredGEOM_JOURNAL : Journal 137861468: mfid0s1d contains data.uhub7: 2 ports with 2 removable, self powered GEOM_JOURNAL uhub6: : 2 ports with 2 removable, self poweredJournal 137861468: mfid0s1d contains journal. GEOM_JOURNAL: Journal mfid0s1d clean. GEOM_JOURNAL: BIO_FLUSH not supported by mfid0s1d. Root mount waiting for: usbus7 usbus3 Root mount waiting for: usbus7 usbus3 uhub5: 6 ports with 6 removable, self powered uhub3: 6 ports with 6 removable, self powered Root mount waiting for: usbus7 usbus3 ugen3.2: at usbus3 uhub8: on usbus3 ugen7.2: at usbus7 umass0: on usbus7 umass0: SCSI over Bulk-Only; quirks = 0x0000 uhub8: 4 ports with 4 removable, self powered umass0:0:0:-1: Attached to scbus0 Root mount waiting for: usbus7 (probe0:umass-sim0:0:0:0): TEST UNIT READY. CDB: 0 0 0 0 0 0 (probe0:umass-sim0:0:0:0): CAM status: SCSI Status ErrorTrying to mount root from ufs:/dev/ufs/root (probe0:umass-sim0:0:0:0): SCSI status: Check Condition (probe0:umass-sim0:0:0:0): SCSI sense: NOT READY asc:3a,2 (Medium not present - tray open) cd0 at umass-sim0 bus 0 scbus0 target 0 lun 0 cd0: Removable CD-ROM SCSI-0 device cd0: 40.000MB/s transfers cd0: Attempt to query device size failed: NOT READY, Medium not present - tray open ugen5.2: at usbus5ugen2.2: at usbus2 uhub9: on usbus5 ukbd0: on usbus2 kbd1 at ukbd0 ums0: on usbus2 ums0: 3 buttons and [Z] coordinates ID=0 uhub9: 4 ports with 4 removable, self powered ugen5.3: at usbus5 ukbd1: on usbus5 kbd2 at ukbd1 uhid0: on usbus5 ums1: on usbus5 ums1: 5 buttons and [XYZ] coordinates ID=0 ums2: on usbus5 ums2: 3 buttons and [Z] coordinates ID=0 eth0: link state changed to UP -- Charles Owens Great Bay Software, Inc.