Skip site navigation (1)Skip section navigation (2)
Date:      Sat, 2 Oct 2010 15:08:45 +0200
From:      Patrick Proniewski <patpro@patpro.net>
To:        freebsd-testing@freebsd.org
Subject:   8.1 / strange bug during benchmark
Message-ID:  <8D4C13FF-AD25-4097-B8C4-E10CE56099CD@patpro.net>

next in thread | raw e-mail | index | archive | help
Hi all,

I'm currently benchmarking my new server, testing UFS and ZFS, and  
different settings of my RAID card.
I've used super-smack (from ports), to test MySQL performance  
depending on FS and RAID card settings. After many runs, something in  
the system becomes corrupted, and super-smack won't run unless I  
reboot the server.

	# uname -a
	FreeBSD patbsd.univ-lyon2.fr 8.1-RELEASE FreeBSD 8.1-RELEASE #0: Mon  
Jul 19 02:36:49 UTC 2010     root@mason.cse.buffalo.edu:/usr/obj/usr/ 
src/sys/GENERIC  amd64

	# super-smack update-select.smack 2 10000
	super-smack: Error creating semaphore errno = 28, error is No space  
left on device

	# df -h
	Filesystem     Size    Used   Avail Capacity  Mounted on
	/dev/da0s1a    4.8G    266M    4.2G     6%    /
	devfs          1.0K    1.0K      0B   100%    /dev
	/dev/da0s1e    1.9G     20K    1.8G     0%    /tmp
	/dev/da0s1f     19G    2.3G     16G    13%    /usr
	/dev/da0s1d     16G    108M     15G     1%    /var
	home           866G    846M    865G     0%    /usr/home  (<- zfs)

Here is a partial truss output for super-smack, showing an ioctl error:

	# truss super-smack update-select.smack 2 10000 2>&1| grep -C2 ERR#25
	munmap(0x801079000,1601536)			 = 0 (0x0)
	open("update-select.smack",O_RDONLY,0666)	 = 2 (0x2)
	ioctl(2,TIOCGETA,0xffffdef0)			 ERR#25 'Inappropriate ioctl for device'
	fstat(2,{ mode=-rw-r--r-- ,inode=120591,size=3527,blksize=16384 }) =  
0 (0x0)
	read(2,"// this is will be used in the t"...,16384) = 3527 (0xdc7)
	--
	stat("/etc/nsswitch.conf",{ mode=-rw-r-- 
r-- ,inode=588848,size=327,blksize=16384 }) = 0 (0x0)
	open("/etc/nsswitch.conf",O_RDONLY,0666)	 = 3 (0x3)
	ioctl(3,TIOCGETA,0xffffda20)			 ERR#25 'Inappropriate ioctl for device'
	fstat(3,{ mode=-rw-r--r-- ,inode=588848,size=327,blksize=16384 }) = 0  
(0x0)
	read(3,"#\n# nsswitch.conf(5) - name ser"...,16384) = 327 (0x147)
	--
	access("/usr/lib/nss_dns.so.1",0)		 ERR#2 'No such file or directory'
	sigprocmask(SIG_SETMASK,0x0,0x0)		 = 0 (0x0)
	ioctl(3,TIOCGETA,0xffffda30)			 ERR#25 'Inappropriate ioctl for device'
	close(3)					 = 0 (0x0)
	sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM| 
SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO| 
SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1| 
SIGUSR2,0x0) = 0 (0x0)
	--
	close(3)					 = 0 (0x0)
	fork()						 = 18889 (0x49c9)
	ioctl(2,TIOCGETA,0xffffdf00)			 ERR#25 'Inappropriate ioctl for device'
	wait4(0x49c9,0x0,0x0,0x0,0x4,0x1)		 = 18889 (0x49c9)
	sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM| 
SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO| 
SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1| 
SIGUSR2,0x0) = 0 (0x0)


After a reboot, super-smack runs ok again, until it fails after a  
large number of runs (I would say between 320 and 640 runs). The  
system seems to behave normally apart from that.
Any idea what's going wrong?

regards,
patpro

-------------------------------------------------------------------------
dmesg output:

Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.1-RELEASE #0: Mon Jul 19 02:36:49 UTC 2010
     root@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU           E5450  @ 3.00GHz (3000.01-MHz K8- 
class CPU)
   Origin = "GenuineIntel"  Id = 0x1067a  Family = 6  Model = 17   
Stepping = 10
    
Features 
= 
0xbfebfbff 
< 
FPU 
,VME 
,DE 
,PSE 
,TSC 
,MSR 
,PAE 
,MCE 
,CX8 
,APIC 
,SEP 
,MTRR 
,PGE 
,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
    
Features2 
= 
0x40ce3bd 
< 
SSE3 
,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE>
   AMD Features=0x20100800<SYSCALL,NX,LM>
   AMD Features2=0x1<LAHF>
   TSC: P-state invariant
real memory  = 17179869184 (16384 MB)
avail memory = 16562409472 (15795 MB)
ACPI APIC Table: <PTLTD  	 APIC  >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
  cpu0 (BSP): APIC ID:  0
  cpu1 (AP): APIC ID:  1
  cpu2 (AP): APIC ID:  2
  cpu3 (AP): APIC ID:  3
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <PTLTD 	 XSDT> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
unknown: I/O range not supported
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci2: <ACPI PCI bus> on pcib2
3ware device driver for 9000 series storage controllers, version:  
3.70.05.001
twa0: <3ware 9000 series Storage Controller> port 0x2000-0x20ff mem  
0xf8000000-0xf9ffffff,0xfc100000-0xfc100fff irq 16 at device 0.0 on pci2
twa0: [ITHREAD]
twa0: INFO: (0x15: 0x1300): Controller details:: Model 9650SE-4LPML, 4  
ports, Firmware FE9X 3.08.00.016, BIOS BE9X 3.08.00.004
pcib3: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci3: <ACPI PCI bus> on pcib3
pci0: <base peripheral> at device 8.0 (no driver attached)
uhci0: <Intel 82801I (ICH9) USB controller> port 0x1820-0x183f irq 16  
at device 26.0 on pci0
uhci0: [ITHREAD]
usbus0: <Intel 82801I (ICH9) USB controller> on uhci0
ehci0: <Intel 82801I (ICH9) USB 2.0 controller> mem  
0xfc600000-0xfc6003ff irq 18 at device 26.7 on pci0
ehci0: [ITHREAD]
usbus1: EHCI version 1.0
usbus1: <Intel 82801I (ICH9) USB 2.0 controller> on ehci0
pcib4: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0
pci5: <ACPI PCI bus> on pcib5
em0: <Intel(R) PRO/1000 Network Connection 7.0.5> port 0x3000-0x301f  
mem 0xfc200000-0xfc21ffff irq 16 at device 0.0 on pci5
em0: Using MSI interrupt
em0: [FILTER]
em0: Ethernet address: 00:30:48:d2:e3:fe
pcib6: <ACPI PCI-PCI bridge> irq 17 at device 28.5 on pci0
pci6: <ACPI PCI bus> on pcib6
em1: <Intel(R) PRO/1000 Network Connection 7.0.5> port 0x4000-0x401f  
mem 0xfc300000-0xfc31ffff irq 17 at device 0.0 on pci6
em1: Using MSI interrupt
em1: [FILTER]
em1: Ethernet address: 00:30:48:d2:e3:ff
uhci1: <Intel 82801I (ICH9) USB controller> port 0x1840-0x185f irq 16  
at device 29.0 on pci0
uhci1: [ITHREAD]
usbus2: <Intel 82801I (ICH9) USB controller> on uhci1
uhci2: <Intel 82801I (ICH9) USB controller> port 0x1860-0x187f irq 17  
at device 29.1 on pci0
uhci2: [ITHREAD]
usbus3: <Intel 82801I (ICH9) USB controller> on uhci2
uhci3: <Intel 82801I (ICH9) USB controller> port 0x1880-0x189f irq 18  
at device 29.2 on pci0
uhci3: [ITHREAD]
usbus4: <Intel 82801I (ICH9) USB controller> on uhci3
ehci1: <Intel 82801I (ICH9) USB 2.0 controller> mem  
0xfc600400-0xfc6007ff irq 16 at device 29.7 on pci0
ehci1: [ITHREAD]
usbus5: EHCI version 1.0
usbus5: <Intel 82801I (ICH9) USB 2.0 controller> on ehci1
pcib7: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci7: <ACPI PCI bus> on pcib7
atapci0: <ITE IT8213F UDMA133 controller> port 0x5098-0x509f, 
0x5090-0x5093,0x5080-0x508f irq 17 at device 0.0 on pci7
atapci0: [ITHREAD]
ata2: <ATA channel 0> on atapci0
ata2: [ITHREAD]
vgapci0: <VGA-compatible display> port 0x5000-0x507f mem  
0xfa000000-0xfbffffff,0xfc000000-0xfc03ffff at device 1.0 on pci7
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci1: <Intel ICH9 SATA300 controller> port  
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x18d0-0x18df,0x18c0-0x18cf at  
device 31.2 on pci0
ata0: <ATA channel 0> on atapci1
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci1
ata1: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
atapci2: <Intel ICH9 SATA300 controller> port 0x1c28-0x1c2f, 
0x1c1c-0x1c1f,0x1c20-0x1c27,0x1c18-0x1c1b,0x18f0-0x18ff,0x18e0-0x18ef  
irq 18 at device 31.5 on pci0
atapci2: [ITHREAD]
ata3: <ATA channel 0> on atapci2
ata3: [ITHREAD]
ata4: <ATA channel 1> on atapci2
ata4: [ITHREAD]
acpi_button0: <Power Button> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart1: [FILTER]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on  
acpi0
fdc0: [FILTER]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc9fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on  
isa0
ppc0: cannot reserve I/O port range
est0: <Enhanced SpeedStep Frequency Control> on cpu0
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 91e091e0600091e
device_attach: est0 attach returned 6
p4tcc0: <CPU Frequency Thermal Control> on cpu0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 91e091e0600091e
device_attach: est1 attach returned 6
p4tcc1: <CPU Frequency Thermal Control> on cpu1
est2: <Enhanced SpeedStep Frequency Control> on cpu2
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 91e091e0600091e
device_attach: est2 attach returned 6
p4tcc2: <CPU Frequency Thermal Control> on cpu2
est3: <Enhanced SpeedStep Frequency Control> on cpu3
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 91e091e0600091e
device_attach: est3 attach returned 6
p4tcc3: <CPU Frequency Thermal Control> on cpu3
Timecounters tick every 1.000 msec
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 480Mbps High Speed USB v2.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 12Mbps Full Speed USB v1.0
usbus4: 12Mbps Full Speed USB v1.0
usbus5: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
ugen4.1: <Intel> at usbus4
uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
ugen5.1: <Intel> at usbus5
uhub5: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5
uhub0: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
uhub4: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub5: 6 ports with 6 removable, self powered
da0 at twa0 bus 0 scbus0 target 0 lun 0SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!

da0: <AMCC 9650SE-4LP DISK 3.08> Fixed Direct Access SCSI-5 device
da0: 100.000MB/s transfers
da0: 953664MB (1953103872 512 byte sectors: 255H 63S/T 121575C)
ugen2.2: <ATEN> at usbus2
ukbd0: <HID Keyboard> on usbus2
kbd2 at ukbd0
ums0: <HID Mouse> on usbus2
ums0: 5 buttons and [XYZ] coordinates ID=0
Trying to mount root from ufs:/dev/da0s1a
ZFS filesystem version 3
ZFS storage pool version 14
em0: link state changed to UP





Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?8D4C13FF-AD25-4097-B8C4-E10CE56099CD>