From owner-freebsd-testing@FreeBSD.ORG Sat Oct 2 13:26:17 2010 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2A094106564A for ; Sat, 2 Oct 2010 13:26:17 +0000 (UTC) (envelope-from patpro@patpro.net) Received: from rack.patpro.net (rack.patpro.net [193.30.227.216]) by mx1.freebsd.org (Postfix) with ESMTP id 6B19F8FC12 for ; Sat, 2 Oct 2010 13:26:16 +0000 (UTC) Received: from rack.patpro.net (localhost [127.0.0.1]) by rack.patpro.net (Postfix) with ESMTP id A2EA36A for ; Sat, 2 Oct 2010 15:08:53 +0200 (CEST) X-Virus-Scanned: amavisd-new at patpro.net Received: from amavis-at-patpro.net ([127.0.0.1]) by rack.patpro.net (rack.patpro.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id h-e7ki5JQG6p for ; Sat, 2 Oct 2010 15:08:46 +0200 (CEST) Received: from [IPv6:::1] (localhost [127.0.0.1]) by rack.patpro.net (Postfix) with ESMTP for ; Sat, 2 Oct 2010 15:08:46 +0200 (CEST) Message-Id: <8D4C13FF-AD25-4097-B8C4-E10CE56099CD@patpro.net> From: Patrick Proniewski To: freebsd-testing@freebsd.org Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v936) Date: Sat, 2 Oct 2010 15:08:45 +0200 X-Mailer: Apple Mail (2.936) Subject: 8.1 / strange bug during benchmark X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance and Stability Tests List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Oct 2010 13:26:17 -0000 Hi all, I'm currently benchmarking my new server, testing UFS and ZFS, and different settings of my RAID card. I've used super-smack (from ports), to test MySQL performance depending on FS and RAID card settings. After many runs, something in the system becomes corrupted, and super-smack won't run unless I reboot the server. # uname -a FreeBSD patbsd.univ-lyon2.fr 8.1-RELEASE FreeBSD 8.1-RELEASE #0: Mon Jul 19 02:36:49 UTC 2010 root@mason.cse.buffalo.edu:/usr/obj/usr/ src/sys/GENERIC amd64 # super-smack update-select.smack 2 10000 super-smack: Error creating semaphore errno = 28, error is No space left on device # df -h Filesystem Size Used Avail Capacity Mounted on /dev/da0s1a 4.8G 266M 4.2G 6% / devfs 1.0K 1.0K 0B 100% /dev /dev/da0s1e 1.9G 20K 1.8G 0% /tmp /dev/da0s1f 19G 2.3G 16G 13% /usr /dev/da0s1d 16G 108M 15G 1% /var home 866G 846M 865G 0% /usr/home (<- zfs) Here is a partial truss output for super-smack, showing an ioctl error: # truss super-smack update-select.smack 2 10000 2>&1| grep -C2 ERR#25 munmap(0x801079000,1601536) = 0 (0x0) open("update-select.smack",O_RDONLY,0666) = 2 (0x2) ioctl(2,TIOCGETA,0xffffdef0) ERR#25 'Inappropriate ioctl for device' fstat(2,{ mode=-rw-r--r-- ,inode=120591,size=3527,blksize=16384 }) = 0 (0x0) read(2,"// this is will be used in the t"...,16384) = 3527 (0xdc7) -- stat("/etc/nsswitch.conf",{ mode=-rw-r-- r-- ,inode=588848,size=327,blksize=16384 }) = 0 (0x0) open("/etc/nsswitch.conf",O_RDONLY,0666) = 3 (0x3) ioctl(3,TIOCGETA,0xffffda20) ERR#25 'Inappropriate ioctl for device' fstat(3,{ mode=-rw-r--r-- ,inode=588848,size=327,blksize=16384 }) = 0 (0x0) read(3,"#\n# nsswitch.conf(5) - name ser"...,16384) = 327 (0x147) -- access("/usr/lib/nss_dns.so.1",0) ERR#2 'No such file or directory' sigprocmask(SIG_SETMASK,0x0,0x0) = 0 (0x0) ioctl(3,TIOCGETA,0xffffda30) ERR#25 'Inappropriate ioctl for device' close(3) = 0 (0x0) sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM| SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO| SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1| SIGUSR2,0x0) = 0 (0x0) -- close(3) = 0 (0x0) fork() = 18889 (0x49c9) ioctl(2,TIOCGETA,0xffffdf00) ERR#25 'Inappropriate ioctl for device' wait4(0x49c9,0x0,0x0,0x0,0x4,0x1) = 18889 (0x49c9) sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM| SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO| SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1| SIGUSR2,0x0) = 0 (0x0) After a reboot, super-smack runs ok again, until it fails after a large number of runs (I would say between 320 and 640 runs). The system seems to behave normally apart from that. Any idea what's going wrong? regards, patpro ------------------------------------------------------------------------- dmesg output: Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.1-RELEASE #0: Mon Jul 19 02:36:49 UTC 2010 root@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU E5450 @ 3.00GHz (3000.01-MHz K8- class CPU) Origin = "GenuineIntel" Id = 0x1067a Family = 6 Model = 17 Stepping = 10 Features = 0xbfebfbff < FPU ,VME ,DE ,PSE ,TSC ,MSR ,PAE ,MCE ,CX8 ,APIC ,SEP ,MTRR ,PGE ,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2 = 0x40ce3bd < SSE3 ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE> AMD Features=0x20100800 AMD Features2=0x1 TSC: P-state invariant real memory = 17179869184 (16384 MB) avail memory = 16562409472 (15795 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) unknown: I/O range not supported Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 2.0 on pci0 pci1: on pcib1 pcib2: at device 4.0 on pci0 pci2: on pcib2 3ware device driver for 9000 series storage controllers, version: 3.70.05.001 twa0: <3ware 9000 series Storage Controller> port 0x2000-0x20ff mem 0xf8000000-0xf9ffffff,0xfc100000-0xfc100fff irq 16 at device 0.0 on pci2 twa0: [ITHREAD] twa0: INFO: (0x15: 0x1300): Controller details:: Model 9650SE-4LPML, 4 ports, Firmware FE9X 3.08.00.016, BIOS BE9X 3.08.00.004 pcib3: at device 6.0 on pci0 pci3: on pcib3 pci0: at device 8.0 (no driver attached) uhci0: port 0x1820-0x183f irq 16 at device 26.0 on pci0 uhci0: [ITHREAD] usbus0: on uhci0 ehci0: mem 0xfc600000-0xfc6003ff irq 18 at device 26.7 on pci0 ehci0: [ITHREAD] usbus1: EHCI version 1.0 usbus1: on ehci0 pcib4: irq 16 at device 28.0 on pci0 pci4: on pcib4 pcib5: irq 16 at device 28.4 on pci0 pci5: on pcib5 em0: port 0x3000-0x301f mem 0xfc200000-0xfc21ffff irq 16 at device 0.0 on pci5 em0: Using MSI interrupt em0: [FILTER] em0: Ethernet address: 00:30:48:d2:e3:fe pcib6: irq 17 at device 28.5 on pci0 pci6: on pcib6 em1: port 0x4000-0x401f mem 0xfc300000-0xfc31ffff irq 17 at device 0.0 on pci6 em1: Using MSI interrupt em1: [FILTER] em1: Ethernet address: 00:30:48:d2:e3:ff uhci1: port 0x1840-0x185f irq 16 at device 29.0 on pci0 uhci1: [ITHREAD] usbus2: on uhci1 uhci2: port 0x1860-0x187f irq 17 at device 29.1 on pci0 uhci2: [ITHREAD] usbus3: on uhci2 uhci3: port 0x1880-0x189f irq 18 at device 29.2 on pci0 uhci3: [ITHREAD] usbus4: on uhci3 ehci1: mem 0xfc600400-0xfc6007ff irq 16 at device 29.7 on pci0 ehci1: [ITHREAD] usbus5: EHCI version 1.0 usbus5: on ehci1 pcib7: at device 30.0 on pci0 pci7: on pcib7 atapci0: port 0x5098-0x509f, 0x5090-0x5093,0x5080-0x508f irq 17 at device 0.0 on pci7 atapci0: [ITHREAD] ata2: on atapci0 ata2: [ITHREAD] vgapci0: port 0x5000-0x507f mem 0xfa000000-0xfbffffff,0xfc000000-0xfc03ffff at device 1.0 on pci7 isab0: at device 31.0 on pci0 isa0: on isab0 atapci1: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x18d0-0x18df,0x18c0-0x18cf at device 31.2 on pci0 ata0: on atapci1 ata0: [ITHREAD] ata1: on atapci1 ata1: [ITHREAD] pci0: at device 31.3 (no driver attached) atapci2: port 0x1c28-0x1c2f, 0x1c1c-0x1c1f,0x1c20-0x1c27,0x1c18-0x1c1b,0x18f0-0x18ff,0x18e0-0x18ef irq 18 at device 31.5 on pci0 atapci2: [ITHREAD] ata3: on atapci2 ata3: [ITHREAD] ata4: on atapci2 ata4: [ITHREAD] acpi_button0: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart1: [FILTER] fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 orm0: at iomem 0xc0000-0xc7fff,0xc8000-0xc9fff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ppc0: cannot reserve I/O port range est0: on cpu0 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 91e091e0600091e device_attach: est0 attach returned 6 p4tcc0: on cpu0 est1: on cpu1 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 91e091e0600091e device_attach: est1 attach returned 6 p4tcc1: on cpu1 est2: on cpu2 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 91e091e0600091e device_attach: est2 attach returned 6 p4tcc2: on cpu2 est3: on cpu3 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 91e091e0600091e device_attach: est3 attach returned 6 p4tcc3: on cpu3 Timecounters tick every 1.000 msec usbus0: 12Mbps Full Speed USB v1.0 usbus1: 480Mbps High Speed USB v2.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 12Mbps Full Speed USB v1.0 usbus4: 12Mbps Full Speed USB v1.0 usbus5: 480Mbps High Speed USB v2.0 ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 ugen4.1: at usbus4 uhub4: on usbus4 ugen5.1: at usbus5 uhub5: on usbus5 uhub0: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub5: 6 ports with 6 removable, self powered da0 at twa0 bus 0 scbus0 target 0 lun 0SMP: AP CPU #2 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! da0: Fixed Direct Access SCSI-5 device da0: 100.000MB/s transfers da0: 953664MB (1953103872 512 byte sectors: 255H 63S/T 121575C) ugen2.2: at usbus2 ukbd0: on usbus2 kbd2 at ukbd0 ums0: on usbus2 ums0: 5 buttons and [XYZ] coordinates ID=0 Trying to mount root from ufs:/dev/da0s1a ZFS filesystem version 3 ZFS storage pool version 14 em0: link state changed to UP