Date: Sat, 2 Oct 2010 15:08:45 +0200 From: Patrick Proniewski <patpro@patpro.net> To: freebsd-testing@freebsd.org Subject: 8.1 / strange bug during benchmark Message-ID: <8D4C13FF-AD25-4097-B8C4-E10CE56099CD@patpro.net>
next in thread | raw e-mail | index | archive | help
Hi all, I'm currently benchmarking my new server, testing UFS and ZFS, and different settings of my RAID card. I've used super-smack (from ports), to test MySQL performance depending on FS and RAID card settings. After many runs, something in the system becomes corrupted, and super-smack won't run unless I reboot the server. # uname -a FreeBSD patbsd.univ-lyon2.fr 8.1-RELEASE FreeBSD 8.1-RELEASE #0: Mon Jul 19 02:36:49 UTC 2010 root@mason.cse.buffalo.edu:/usr/obj/usr/ src/sys/GENERIC amd64 # super-smack update-select.smack 2 10000 super-smack: Error creating semaphore errno = 28, error is No space left on device # df -h Filesystem Size Used Avail Capacity Mounted on /dev/da0s1a 4.8G 266M 4.2G 6% / devfs 1.0K 1.0K 0B 100% /dev /dev/da0s1e 1.9G 20K 1.8G 0% /tmp /dev/da0s1f 19G 2.3G 16G 13% /usr /dev/da0s1d 16G 108M 15G 1% /var home 866G 846M 865G 0% /usr/home (<- zfs) Here is a partial truss output for super-smack, showing an ioctl error: # truss super-smack update-select.smack 2 10000 2>&1| grep -C2 ERR#25 munmap(0x801079000,1601536) = 0 (0x0) open("update-select.smack",O_RDONLY,0666) = 2 (0x2) ioctl(2,TIOCGETA,0xffffdef0) ERR#25 'Inappropriate ioctl for device' fstat(2,{ mode=-rw-r--r-- ,inode=120591,size=3527,blksize=16384 }) = 0 (0x0) read(2,"// this is will be used in the t"...,16384) = 3527 (0xdc7) -- stat("/etc/nsswitch.conf",{ mode=-rw-r-- r-- ,inode=588848,size=327,blksize=16384 }) = 0 (0x0) open("/etc/nsswitch.conf",O_RDONLY,0666) = 3 (0x3) ioctl(3,TIOCGETA,0xffffda20) ERR#25 'Inappropriate ioctl for device' fstat(3,{ mode=-rw-r--r-- ,inode=588848,size=327,blksize=16384 }) = 0 (0x0) read(3,"#\n# nsswitch.conf(5) - name ser"...,16384) = 327 (0x147) -- access("/usr/lib/nss_dns.so.1",0) ERR#2 'No such file or directory' sigprocmask(SIG_SETMASK,0x0,0x0) = 0 (0x0) ioctl(3,TIOCGETA,0xffffda30) ERR#25 'Inappropriate ioctl for device' close(3) = 0 (0x0) sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM| SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO| SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1| SIGUSR2,0x0) = 0 (0x0) -- close(3) = 0 (0x0) fork() = 18889 (0x49c9) ioctl(2,TIOCGETA,0xffffdf00) ERR#25 'Inappropriate ioctl for device' wait4(0x49c9,0x0,0x0,0x0,0x4,0x1) = 18889 (0x49c9) sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM| SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO| SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1| SIGUSR2,0x0) = 0 (0x0) After a reboot, super-smack runs ok again, until it fails after a large number of runs (I would say between 320 and 640 runs). The system seems to behave normally apart from that. Any idea what's going wrong? regards, patpro ------------------------------------------------------------------------- dmesg output: Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.1-RELEASE #0: Mon Jul 19 02:36:49 UTC 2010 root@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU E5450 @ 3.00GHz (3000.01-MHz K8- class CPU) Origin = "GenuineIntel" Id = 0x1067a Family = 6 Model = 17 Stepping = 10 Features = 0xbfebfbff < FPU ,VME ,DE ,PSE ,TSC ,MSR ,PAE ,MCE ,CX8 ,APIC ,SEP ,MTRR ,PGE ,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2 = 0x40ce3bd < SSE3 ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE> AMD Features=0x20100800<SYSCALL,NX,LM> AMD Features2=0x1<LAHF> TSC: P-state invariant real memory = 17179869184 (16384 MB) avail memory = 16562409472 (15795 MB) ACPI APIC Table: <PTLTD APIC > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 ioapic0 <Version 2.0> irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: <PTLTD XSDT> on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) unknown: I/O range not supported Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 cpu0: <ACPI CPU> on acpi0 cpu1: <ACPI CPU> on acpi0 cpu2: <ACPI CPU> on acpi0 cpu3: <ACPI CPU> on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0 pci1: <ACPI PCI bus> on pcib1 pcib2: <ACPI PCI-PCI bridge> at device 4.0 on pci0 pci2: <ACPI PCI bus> on pcib2 3ware device driver for 9000 series storage controllers, version: 3.70.05.001 twa0: <3ware 9000 series Storage Controller> port 0x2000-0x20ff mem 0xf8000000-0xf9ffffff,0xfc100000-0xfc100fff irq 16 at device 0.0 on pci2 twa0: [ITHREAD] twa0: INFO: (0x15: 0x1300): Controller details:: Model 9650SE-4LPML, 4 ports, Firmware FE9X 3.08.00.016, BIOS BE9X 3.08.00.004 pcib3: <ACPI PCI-PCI bridge> at device 6.0 on pci0 pci3: <ACPI PCI bus> on pcib3 pci0: <base peripheral> at device 8.0 (no driver attached) uhci0: <Intel 82801I (ICH9) USB controller> port 0x1820-0x183f irq 16 at device 26.0 on pci0 uhci0: [ITHREAD] usbus0: <Intel 82801I (ICH9) USB controller> on uhci0 ehci0: <Intel 82801I (ICH9) USB 2.0 controller> mem 0xfc600000-0xfc6003ff irq 18 at device 26.7 on pci0 ehci0: [ITHREAD] usbus1: EHCI version 1.0 usbus1: <Intel 82801I (ICH9) USB 2.0 controller> on ehci0 pcib4: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0 pci4: <ACPI PCI bus> on pcib4 pcib5: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0 pci5: <ACPI PCI bus> on pcib5 em0: <Intel(R) PRO/1000 Network Connection 7.0.5> port 0x3000-0x301f mem 0xfc200000-0xfc21ffff irq 16 at device 0.0 on pci5 em0: Using MSI interrupt em0: [FILTER] em0: Ethernet address: 00:30:48:d2:e3:fe pcib6: <ACPI PCI-PCI bridge> irq 17 at device 28.5 on pci0 pci6: <ACPI PCI bus> on pcib6 em1: <Intel(R) PRO/1000 Network Connection 7.0.5> port 0x4000-0x401f mem 0xfc300000-0xfc31ffff irq 17 at device 0.0 on pci6 em1: Using MSI interrupt em1: [FILTER] em1: Ethernet address: 00:30:48:d2:e3:ff uhci1: <Intel 82801I (ICH9) USB controller> port 0x1840-0x185f irq 16 at device 29.0 on pci0 uhci1: [ITHREAD] usbus2: <Intel 82801I (ICH9) USB controller> on uhci1 uhci2: <Intel 82801I (ICH9) USB controller> port 0x1860-0x187f irq 17 at device 29.1 on pci0 uhci2: [ITHREAD] usbus3: <Intel 82801I (ICH9) USB controller> on uhci2 uhci3: <Intel 82801I (ICH9) USB controller> port 0x1880-0x189f irq 18 at device 29.2 on pci0 uhci3: [ITHREAD] usbus4: <Intel 82801I (ICH9) USB controller> on uhci3 ehci1: <Intel 82801I (ICH9) USB 2.0 controller> mem 0xfc600400-0xfc6007ff irq 16 at device 29.7 on pci0 ehci1: [ITHREAD] usbus5: EHCI version 1.0 usbus5: <Intel 82801I (ICH9) USB 2.0 controller> on ehci1 pcib7: <ACPI PCI-PCI bridge> at device 30.0 on pci0 pci7: <ACPI PCI bus> on pcib7 atapci0: <ITE IT8213F UDMA133 controller> port 0x5098-0x509f, 0x5090-0x5093,0x5080-0x508f irq 17 at device 0.0 on pci7 atapci0: [ITHREAD] ata2: <ATA channel 0> on atapci0 ata2: [ITHREAD] vgapci0: <VGA-compatible display> port 0x5000-0x507f mem 0xfa000000-0xfbffffff,0xfc000000-0xfc03ffff at device 1.0 on pci7 isab0: <PCI-ISA bridge> at device 31.0 on pci0 isa0: <ISA bus> on isab0 atapci1: <Intel ICH9 SATA300 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x18d0-0x18df,0x18c0-0x18cf at device 31.2 on pci0 ata0: <ATA channel 0> on atapci1 ata0: [ITHREAD] ata1: <ATA channel 1> on atapci1 ata1: [ITHREAD] pci0: <serial bus, SMBus> at device 31.3 (no driver attached) atapci2: <Intel ICH9 SATA300 controller> port 0x1c28-0x1c2f, 0x1c1c-0x1c1f,0x1c20-0x1c27,0x1c18-0x1c1b,0x18f0-0x18ff,0x18e0-0x18ef irq 18 at device 31.5 on pci0 atapci2: [ITHREAD] ata3: <ATA channel 0> on atapci2 ata3: [ITHREAD] ata4: <ATA channel 1> on atapci2 ata4: [ITHREAD] acpi_button0: <Power Button> on acpi0 atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0 atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart1: [FILTER] fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc9fff on isa0 sc0: <System console> at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ppc0: cannot reserve I/O port range est0: <Enhanced SpeedStep Frequency Control> on cpu0 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 91e091e0600091e device_attach: est0 attach returned 6 p4tcc0: <CPU Frequency Thermal Control> on cpu0 est1: <Enhanced SpeedStep Frequency Control> on cpu1 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 91e091e0600091e device_attach: est1 attach returned 6 p4tcc1: <CPU Frequency Thermal Control> on cpu1 est2: <Enhanced SpeedStep Frequency Control> on cpu2 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 91e091e0600091e device_attach: est2 attach returned 6 p4tcc2: <CPU Frequency Thermal Control> on cpu2 est3: <Enhanced SpeedStep Frequency Control> on cpu3 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 91e091e0600091e device_attach: est3 attach returned 6 p4tcc3: <CPU Frequency Thermal Control> on cpu3 Timecounters tick every 1.000 msec usbus0: 12Mbps Full Speed USB v1.0 usbus1: 480Mbps High Speed USB v2.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 12Mbps Full Speed USB v1.0 usbus4: 12Mbps Full Speed USB v1.0 usbus5: 480Mbps High Speed USB v2.0 ugen0.1: <Intel> at usbus0 uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0 ugen1.1: <Intel> at usbus1 uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1 ugen2.1: <Intel> at usbus2 uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2 ugen3.1: <Intel> at usbus3 uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3 ugen4.1: <Intel> at usbus4 uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4 ugen5.1: <Intel> at usbus5 uhub5: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5 uhub0: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub5: 6 ports with 6 removable, self powered da0 at twa0 bus 0 scbus0 target 0 lun 0SMP: AP CPU #2 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! da0: <AMCC 9650SE-4LP DISK 3.08> Fixed Direct Access SCSI-5 device da0: 100.000MB/s transfers da0: 953664MB (1953103872 512 byte sectors: 255H 63S/T 121575C) ugen2.2: <ATEN> at usbus2 ukbd0: <HID Keyboard> on usbus2 kbd2 at ukbd0 ums0: <HID Mouse> on usbus2 ums0: 5 buttons and [XYZ] coordinates ID=0 Trying to mount root from ufs:/dev/da0s1a ZFS filesystem version 3 ZFS storage pool version 14 em0: link state changed to UP
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?8D4C13FF-AD25-4097-B8C4-E10CE56099CD>