Date: Fri, 22 Mar 2013 17:34:43 -0700 (PDT) From: Paul Pathiakis <pathiaki2@yahoo.com> To: Davide D'Amico <davide.damico@contactlab.com>, =?utf-8?B?0JXQstCz0LXQvdC40Lkg0KXQvtGA0LrQuNC9?= <khorkin.eugeniy@gmail.com> Cc: "freebsd-performance@freebsd.org" <freebsd-performance@freebsd.org> Subject: Re: FreeBSD 9.1 vs CentOS 6.3 Message-ID: <1363998883.22604.YahooMailNeo@web141401.mail.bf1.yahoo.com> In-Reply-To: <ebfc033f-d8db-4162-86be-db013e3d7eb2@email.android.com> References: <CAF3xD3nDDUYB94TS9AUUQ=CPztB1S8mU4fRwNB5GupmK8MgwXg@mail.gmail.com> <ebfc033f-d8db-4162-86be-db013e3d7eb2@email.android.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi,=0A=0AThere are several things about this that are highly suspect.=C2=A0= =0A=0A=0AFirst, wipe out the hardware RAID.=C2=A0 The processor doing RAID= computation is, probably, MUCH slower than a core on the CPU.=C2=A0 Even i= f it's RAID-1 (Simple Mirror) this RAID card is performing tasks that is do= es not need to do including replicating writes to two targets from the cont= roller or checking it's cache, battery, etc.=C2=A0 If it's possible to disa= ble the onboard cache, do it.=0A=0A=0A(This controller seems to use the mfi= driver so that's probably a re-badged LSI with an LSI200x chipset.=C2=A0 (= Lately, I have not been too thrilled with the mfi driver performance.=C2=A0= The mpt seems much more stable and robust - my opinion not bashing to whoe= ver wrote it.)=0A=0A=0ATake the card to full JBOD mode before testing your = "throughput".=0A=0AAlso, make sure that all on-disk caches are "off".=C2=A0= You do not want to get into a double-buffer flush corruption issue.=0A=0AI= see you are using two SSDs as ARC2 caches....=C2=A0 Quite frankly, junk th= em as well, for now.=C2=A0 You may want to deal with increasing throughput = once you find out what the issue is with the base throughput.=0A=0A=0ACreat= e the appropriate pools/mirrors as necessary, all on simple JBOD.=0A=0AFrom= there, run a simple test to see what your throughput is on single spindle.= =C2=A0 (I'm talking SIMPLE.=C2=A0 dd if=3D/dev/zero of=3D/<path>/zeroes.out= =C2=A0 bs=3D128k count=3D<some ridiculous number like 100000 or more> - Thi= s should be over 100 MB/sec throughput.=C2=A0 (with those drives it may be = 150 MB/sec or more)=0A=0AAfter that, check the throughput of the card/chann= el.=C2=A0 Write a very simple script script that performs the same throughp= ut test on half the available spindles (something as ludicrous as the same = dd command to half your spindles throwing it in the background as each kick= s off - Then, do it for all of your drives)=C2=A0 =0A=0A=0AThis is an imper= fect test but it's down and dirty.=C2=A0 You can use the performance/benchm= ark tools later.=0A=0ALet's just remember that the throughput of 6 Gb/sec i= s gigaBITS not gigaBYTES.=C2=A0 There's a factor of 8 difference.... or 750= MB/sec... theoretical.=0A=0AYou should exceed channel speed with just 5 of= these drives/spindles... use 6 to be sure (I would think you won't see agg= regate throughput beyond 725 MB/sec....=C2=A0 If any tool is showing you th= at on any OS with this configuration, they aren't doing proper calculations= and/or they are doing MB vs MiB.=C2=A0 =0A=0A=0AIf you can't get up to tha= t speed, you may have a driver issue or channel issue.=0A=0A(I was getting = as high as 700 MB/sec with 6 SATA drives doing it this way.... No SSD, ASUS= MB 990FX with 1100T 6 core proc, 16 GB RAM... )=0A=0AP.=0A=0A=0A=0A_______= _________________________=0A From: Davide D'Amico <davide.damico@contactlab= .com>=0ATo: =D0=95=D0=B2=D0=B3=D0=B5=D0=BD=D0=B8=D0=B9 =D0=A5=D0=BE=D1=80= =D0=BA=D0=B8=D0=BD <khorkin.eugeniy@gmail.com> =0ACc: freebsd-performance@f= reebsd.org =0ASent: Friday, March 22, 2013 12:15 PM=0ASubject: Re: FreeBSD = 9.1 vs CentOS 6.3=0A =0AWell, the I/O isn't the bottleneck (if you follow t= he link to freebsd-fs, you'll see iostats values) but it seems something re= lated to cpu/scheduler or something else.=0ANow I am trying vmware 5 on the= same server and a vm with centos6: the vm outperforms freebsd with every c= oncurrency from 1 to 48 threads in sysbench oltp tests.=0A=0AThanks,=0Ad.= =0A=0A"=D0=95=D0=B2=D0=B3=D0=B5=D0=BD=D0=B8=D0=B9 =D0=A5=D0=BE=D1=80=D0=BA= =D0=B8=D0=BD" <khorkin.eugeniy@gmail.com> ha scritto:=0A=0A>Hi Davide!=0A>= =0A>Are you sure that disk is the bottleneck in your test?=0A>Does systat -= vm 1 show 100% busy for disk ?=0A>=0A>Evgeny.=0A>=0A>=0A>2013/3/22 Davide D= 'Amico <davide.damico@contactlab.com>=0A>=0A>> Hi, I'm doing performance te= sts on a DELL R720, follows dmesg:=0A>>=0A>> Copyright (c) 1992-2012 The Fr= eeBSD Project.=0A>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991,= 1992, 1993,=0A>1994=0A>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 The Regents of the Un= iversity of California. All rights=0A>reserved.=0A>> FreeBSD is a registere= d trademark of The FreeBSD Foundation.=0A>> FreeBSD 9.1-RELEASE #0: Tue Mar= 19 10:24:21 CET 2013=0A>>=C2=A0 =C2=A0 root:/usr/obj/usr/src/sys/R720 amd= 64=0A>> CPU: Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz (2500.05-MHz K8-class= =0A>CPU)=0A>>=C2=A0 Origin =3D "GenuineIntel"=C2=A0 Id =3D 0x206d7=C2=A0 F= amily =3D 6=C2=A0 Model =3D 2d=0A>Stepping =3D=0A>> 7=0A>> Features=3D0xbfe= bfbff<FPU,VME,**DE,PSE,TSC,MSR,PAE,MCE,CX8,**=0A>> APIC,SEP,MTRR,PGE,MCA,CM= OV,**PAT,PSE36,CLFLUSH,DTS,ACPI,**=0A>> MMX,FXSR,SSE,SSE2,SS,HTT,TM,**PBE>= =0A>> Features2=3D0x1fbee3ff<SSE3,**PCLMULQDQ,DTES64,MON,DS_CPL,**=0A>> VMX= ,SMX,EST,TM2,SSSE3,CX16,**xTPR,PDCM,PCID,DCA,SSE4.1,**=0A>> SSE4.2,x2APIC,P= OPCNT,TSCDLT,**AESNI,XSAVE,OSXSAVE,AVX>=0A>>=C2=A0 AMD Features=3D0x2c1008= 00<SYSCALL,**NX,Page1GB,RDTSCP,LM>=0A>>=C2=A0 AMD Features2=3D0x1<LAHF>=0A= >>=C2=A0 TSC: P-state invariant, performance statistics=0A>> real memory= =C2=A0 =3D 34359738368 (32768 MB)=0A>> avail memory =3D 33027436544 (31497 = MB)=0A>> Event timer "LAPIC" quality 600=0A>> ACPI APIC Table: <DELL=C2=A0 = PE_SC3=C2=A0 >=0A>> FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs=0A= >> FreeBSD/SMP: 2 package(s) x 2 core(s) x 2 SMT threads=0A>>=C2=A0 cpu0 (B= SP): APIC ID:=C2=A0 0=0A>>=C2=A0 cpu1 (AP): APIC ID:=C2=A0 1=0A>>=C2=A0 cpu= 2 (AP): APIC ID: 10=0A>>=C2=A0 cpu3 (AP): APIC ID: 11=0A>>=C2=A0 cpu4 (AP):= APIC ID: 32=0A>>=C2=A0 cpu5 (AP): APIC ID: 33=0A>>=C2=A0 cpu6 (AP): APIC I= D: 42=0A>>=C2=A0 cpu7 (AP): APIC ID: 43=0A>> ioapic1: Changing APIC ID to 1= =0A>> ioapic2: Changing APIC ID to 2=0A>> ioapic0 <Version 2.0> irqs 0-23 o= n motherboard=0A>> ioapic1 <Version 2.0> irqs 32-55 on motherboard=0A>> ioa= pic2 <Version 2.0> irqs 64-87 on motherboard=0A>> kbd1 at kbdmux0=0A>> ctl:= CAM Target Layer loaded=0A>> acpi0: <DELL PE_SC3> on motherboard=0A>> acpi= 0: Power Button (fixed)=0A>> acpi0: reservation of 0, 2000 (3) failed=0A>> = cpu0: <ACPI CPU> on acpi0=0A>> cpu1: <ACPI CPU> on acpi0=0A>> cpu2: <ACPI C= PU> on acpi0=0A>> cpu3: <ACPI CPU> on acpi0=0A>> cpu4: <ACPI CPU> on acpi0= =0A>> cpu5: <ACPI CPU> on acpi0=0A>> cpu6: <ACPI CPU> on acpi0=0A>> cpu7: <= ACPI CPU> on acpi0=0A>> atrtc0: <AT realtime clock> port 0x70-0x7f irq 8 on= acpi0=0A>> Event timer "RTC" frequency 32768 Hz quality 0=0A>> attimer0: <= AT timer> port 0x40-0x5f irq 0 on acpi0=0A>> Timecounter "i8254" frequency = 1193182 Hz quality 0=0A>> Event timer "i8254" frequency 1193182 Hz quality = 100=0A>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900=0A>> acpi= _timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0=0A>> hpet0= : <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on=0A>acpi0=0A>>= Timecounter "HPET" frequency 14318180 Hz quality 950=0A>> Event timer "HPE= T" frequency 14318180 Hz quality 550=0A>> pcib0: <ACPI Host-PCI bridge> por= t 0xcf8-0xcff on acpi0=0A>> pci0: <ACPI PCI bus> on pcib0=0A>> pcib1: <ACPI= PCI-PCI bridge> irq 53 at device 1.0 on pci0=0A>> pci1: <ACPI PCI bus> on = pcib1=0A>> igb0: <Intel(R) PRO/1000 Network Connection version - 2.3.4> por= t=0A>> 0xfc80-0xfc9f mem 0xdcb00000-0xdcbfffff,**0xdcff0000-0xdcff3fff irq= =0A>38 at=0A>> device 0.0 on pci1=0A>> igb0: Using MSIX interrupts with 9 v= ectors=0A>> igb0: Ethernet address: bc:30:5b:ef:27:b8=0A>> [...]=0A>> igb1:= <Intel(R) PRO/1000 Network Connection version - 2.3.4> port=0A>> 0xfca0-0x= fcbf mem 0xdcc00000-0xdccfffff,**0xdcff4000-0xdcff7fff irq=0A>37 at=0A>> de= vice 0.1 on pci1=0A>> igb1: Using MSIX interrupts with 9 vectors=0A>> igb1:= Ethernet address: bc:30:5b:ef:27:b9=0A>> [...]=0A>> igb2: <Intel(R) PRO/10= 00 Network Connection version - 2.3.4> port=0A>> 0xfcc0-0xfcdf mem 0xdcd000= 00-0xdcdfffff,**0xdcff8000-0xdcffbfff irq=0A>36 at=0A>> device 0.2 on pci1= =0A>> igb2: Using MSIX interrupts with 9 vectors=0A>> igb2: Ethernet addres= s: bc:30:5b:ef:27:ba=0A>> [...]=0A>> igb3: <Intel(R) PRO/1000 Network Conne= ction version - 2.3.4> port=0A>> 0xfce0-0xfcff mem 0xdce00000-0xdcefffff,**= 0xdcffc000-0xdcffffff irq=0A>34 at=0A>> device 0.3 on pci1=0A>> igb3: Using= MSIX interrupts with 9 vectors=0A>> igb3: Ethernet address: bc:30:5b:ef:27= :bb=0A>> [...]=0A>> pcib2: <ACPI PCI-PCI bridge> irq 53 at device 2.0 on pc= i0=0A>> pci3: <ACPI PCI bus> on pcib2=0A>> pcib3: <ACPI PCI-PCI bridge> irq= 53 at device 2.2 on pci0=0A>> pci2: <ACPI PCI bus> on pcib3=0A>> mfi0: <De= ll PERC H710P Mini (monolithics)> port 0xec00-0xecff mem=0A>> 0xddffc000-0x= ddffffff,**0xddf80000-0xddfbffff irq 42 at device 0.0 on=0A>pci2=0A>> mfi0:= Using MSI=0A>> mfi0: Megaraid SAS driver Ver 4.23=0A>> mfi0: MaxCmd =3D 3f= 0 MaxSgl =3D 46 state =3D b73c03f0=0A>> pcib4: <ACPI PCI-PCI bridge> irq 53= at device 3.0 on pci0=0A>> pci4: <ACPI PCI bus> on pcib4=0A>> pcib5: <ACPI= PCI-PCI bridge> irq 53 at device 3.2 on pci0=0A>> pci5: <ACPI PCI bus> on = pcib5=0A>> pci0: <base peripheral> at device 5.0 (no driver attached)=0A>> = pci0: <base peripheral> at device 5.2 (no driver attached)=0A>> pcib6: <PCI= -PCI bridge> irq 16 at device 17.0 on pci0=0A>> pci6: <PCI bus> on pcib6=0A= >> pci0: <simple comms> at device 22.0 (no driver attached)=0A>> pci0: <sim= ple comms> at device 22.1 (no driver attached)=0A>> ehci0: <EHCI (generic) = USB 2.0 controller> mem 0xdf8fe000-0xdf8fe3ff=0A>irq=0A>> 23 at device 26.0= on pci0=0A>> usbus0: EHCI version 1.0=0A>> usbus0 on ehci0=0A>> pcib7: <AC= PI PCI-PCI bridge> at device 28.0 on pci0=0A>> device_attach: pcib7 attach = returned 6=0A>> pcib7: <ACPI PCI-PCI bridge> irq 19 at device 28.7 on pci0= =0A>> pci8: <ACPI PCI bus> on pcib7=0A>> pcib8: <PCI-PCI bridge> at device = 0.0 on pci8=0A>> pci9: <PCI bus> on pcib8=0A>> pcib9: <PCI-PCI bridge> at d= evice 0.0 on pci9=0A>> pci10: <PCI bus> on pcib9=0A>> pcib10: <PCI-PCI brid= ge> at device 0.0 on pci10=0A>> pci11: <PCI bus> on pcib10=0A>> vgapci0: <V= GA-compatible display> mem 0xd8000000-0xd8ffffff,**=0A>> 0xdeffc000-0xdefff= fff,**0xde000000-0xde7fffff irq 19 at device 0.0 on=0A>> pci11=0A>> pcib11:= <PCI-PCI bridge> at device 1.0 on pci9=0A>> pci12: <PCI bus> on pcib11=0A>= > ehci1: <EHCI (generic) USB 2.0 controller> mem 0xdf8ff000-0xdf8ff3ff=0A>i= rq=0A>> 22 at device 29.0 on pci0=0A>> usbus1: EHCI version 1.0=0A>> usbus1= on ehci1=0A>> pcib12: <PCI-PCI bridge> at device 30.0 on pci0=0A>> pci13: = <PCI bus> on pcib12=0A>> isab0: <PCI-ISA bridge> at device 31.0 on pci0=0A>= > isa0: <ISA bus> on isab0=0A>> pcib13: <ACPI Host-PCI bridge> on acpi0=0A>= > pci64: <ACPI PCI bus> on pcib13=0A>> pcib14: <ACPI PCI-PCI bridge> irq 85= at device 1.0 on pci64=0A>> pci65: <ACPI PCI bus> on pcib14=0A>> pcib15: <= ACPI PCI-PCI bridge> irq 85 at device 2.0 on pci64=0A>> pci66: <ACPI PCI bu= s> on pcib15=0A>> pcib16: <ACPI PCI-PCI bridge> irq 85 at device 3.0 on pci= 64=0A>> pci67: <ACPI PCI bus> on pcib16=0A>> pcib17: <ACPI PCI-PCI bridge> = irq 85 at device 3.2 on pci64=0A>> pci68: <ACPI PCI bus> on pcib17=0A>> pci= 64: <base peripheral> at device 5.0 (no driver attached)=0A>> pci64: <base = peripheral> at device 5.2 (no driver attached)=0A>> pcib18: <ACPI Host-PCI = bridge> on acpi0=0A>> pci63: <ACPI PCI bus> on pcib18=0A>> pci63: <base per= ipheral> at device 8.0 (no driver attached)=0A>> [...]=0A>> pci63: <dasp> a= t device 14.1 (no driver attached)=0A>> [...]=0A>> pci63: <dasp> at device = 19.1 (no driver attached)=0A>> pci63: <dasp> at device 19.4 (no driver atta= ched)=0A>> pci63: <dasp> at device 19.5 (no driver attached)=0A>> pci63: <b= ase peripheral> at device 19.6 (no driver attached)=0A>> pcib19: <ACPI Host= -PCI bridge> on acpi0=0A>> pci127: <ACPI PCI bus> on pcib19=0A>> [...]=0A>>= pci127: <dasp> at device 14.1 (no driver attached)=0A>> [...]=0A>> pci127:= <dasp> at device 19.1 (no driver attached)=0A>> pci127: <dasp> at device 1= 9.4 (no driver attached)=0A>> pci127: <dasp> at device 19.5 (no driver atta= ched)=0A>> pci127: <base peripheral> at device 19.6 (no driver attached)=0A= >> uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0=0A>> uart0:= <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on=0A>acpi0=0A>> o= rm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xec000-**0xeffff on=0A>isa= 0=0A>> sc0: <System console> at flags 0x100 on isa0=0A>> sc0: VGA <16 virtu= al consoles, flags=3D0x300>=0A>> vga0: <Generic ISA VGA> at port 0x3c0-0x3d= f iomem 0xa0000-0xbffff on=0A>isa0=0A>> est0: <Enhanced SpeedStep Frequency= Control> on cpu0=0A>> est: CPU supports Enhanced Speedstep, but is not rec= ognized.=0A>> est: cpu_vendor GenuineIntel, msr 226d00001e00=0A>> device_at= tach: est0 attach returned 6=0A>> p4tcc0: <CPU Frequency Thermal Control> o= n cpu0=0A>> est1: <Enhanced SpeedStep Frequency Control> on cpu1=0A>> est: = CPU supports Enhanced Speedstep, but is not recognized.=0A>> est: cpu_vendo= r GenuineIntel, msr 226d00001e00=0A>> device_attach: est1 attach returned 6= =0A>> p4tcc1: <CPU Frequency Thermal Control> on cpu1=0A>> est2: <Enhanced = SpeedStep Frequency Control> on cpu2=0A>> est: CPU supports Enhanced Speeds= tep, but is not recognized.=0A>> est: cpu_vendor GenuineIntel, msr 226d0000= 1e00=0A>> device_attach: est2 attach returned 6=0A>> p4tcc2: <CPU Frequency= Thermal Control> on cpu2=0A>> est3: <Enhanced SpeedStep Frequency Control>= on cpu3=0A>> est: CPU supports Enhanced Speedstep, but is not recognized.= =0A>> est: cpu_vendor GenuineIntel, msr 224400001e00=0A>> device_attach: es= t3 attach returned 6=0A>> p4tcc3: <CPU Frequency Thermal Control> on cpu3= =0A>> est4: <Enhanced SpeedStep Frequency Control> on cpu4=0A>> est: CPU su= pports Enhanced Speedstep, but is not recognized.=0A>> est: cpu_vendor Genu= ineIntel, msr 226d00001e00=0A>> device_attach: est4 attach returned 6=0A>> = p4tcc4: <CPU Frequency Thermal Control> on cpu4=0A>> est5: <Enhanced SpeedS= tep Frequency Control> on cpu5=0A>> est: CPU supports Enhanced Speedstep, b= ut is not recognized.=0A>> est: cpu_vendor GenuineIntel, msr 224400001e00= =0A>> device_attach: est5 attach returned 6=0A>> p4tcc5: <CPU Frequency The= rmal Control> on cpu5=0A>> est6: <Enhanced SpeedStep Frequency Control> on = cpu6=0A>> est: CPU supports Enhanced Speedstep, but is not recognized.=0A>>= est: cpu_vendor GenuineIntel, msr 224400001e00=0A>> device_attach: est6 at= tach returned 6=0A>> p4tcc6: <CPU Frequency Thermal Control> on cpu6=0A>> e= st7: <Enhanced SpeedStep Frequency Control> on cpu7=0A>> est: CPU supports = Enhanced Speedstep, but is not recognized.=0A>> est: cpu_vendor GenuineInte= l, msr 224400001e00=0A>> device_attach: est7 attach returned 6=0A>> p4tcc7:= <CPU Frequency Thermal Control> on cpu7=0A>> mfi0: 6040 (417120269s/0x0020= /info) - Shutdown command received from=0A>host=0A>> mfi0: 6041 (boot + 4s/= 0x0020/info) - Firmware initialization started=0A>(PCI=0A>> ID 005b/1000/1f= 34/1028)=0A>> mfi0: 6042 (boot + 4s/0x0020/info) - Firmware version 3.130.0= 5-1796=0A>> mfi0: 6043 (boot + 5s/0x0008/info) - Battery Present=0A>> mfi0:= 6044 (boot + 5s/0x0020/info) - Package version 21.1.0-0007=0A>> mfi0: 6045= (boot + 5s/0x0020/info) - Board Revision A01=0A>> mfi0: 6046 (boot + 6s/0x= 0008/info) - Battery temperature is normal=0A>> mfi0: 6047 (boot + 6s/0x000= 8/info) - Current capacity of the battery=0A>is=0A>> above threshold=0A>> m= fi0: 6048 (boot + 19s/0x0004/info) - Enclosure PD 20(c None/p1)=0A>> commun= ication restored=0A>> mfi0: 6049 (boot + 19s/0x0002/info) - Inserted: Encl = PD 20=0A>> mfi0: 6050 (boot + 19s/0x0002/info) - Inserted: PD 20(c None/p1)= =0A>Info:=0A>> enclPd=3D20, scsiType=3Dd, portMap=3D00, sasAddr=3D5e4ae020a= 26cb500,**=0A>> 0000000000000000=0A>> mfi0: 6051 (boot + 19s/0x0002/info) -= Inserted: PD 00(e0x20/s0)=0A>> mfi0: 6052 (boot + 19s/0x0002/info) - Inser= ted: PD 00(e0x20/s0) Info:=0A>> enclPd=3D20, scsiType=3D0, portMap=3D04, sa= sAddr=3D5000c50054982339,**=0A>> 0000000000000000=0A>> mfi0: 6053 (boot + 1= 9s/0x0002/info) - Inserted: PD 01(e0x20/s1)=0A>> mfi0: 6054 (boot + 19s/0x0= 002/info) - Inserted: PD 01(e0x20/s1) Info:=0A>> enclPd=3D20, scsiType=3D0,= portMap=3D05, sasAddr=3D5000c50054986c91,**=0A>> 0000000000000000=0A>> mfi= 0: 6055 (boot + 19s/0x0002/info) - Inserted: PD 02(e0x20/s2)=0A>> mfi0: 605= 6 (boot + 19s/0x0002/info) - Inserted: PD 02(e0x20/s2) Info:=0A>> enclPd=3D= 20, scsiType=3D0, portMap=3D06, sasAddr=3D5000c50053da8f75,**=0A>> 00000000= 00000000=0A>> mfi0: 6057 (boot + 19s/0x0002/info) - Inserted: PD 03(e0x20/s= 3)=0A>> mfi0: 6058 (boot + 19s/0x0002/info) - Inserted: PD 03(e0x20/s3) Inf= o:=0A>> enclPd=3D20, scsiType=3D0, portMap=3D07, sasAddr=3D5000c50053c57eb5= ,**=0A>> 0000000000000000=0A>> mfi0: 6059 (boot + 19s/0x0002/info) - Insert= ed: PD 04(e0x20/s4)=0A>> mfi0: 6060 (boot + 19s/0x0002/info) - Inserted: PD= 04(e0x20/s4) Info:=0A>> enclPd=3D20, scsiType=3D0, portMap=3D00, sasAddr= =3D5000c50053d364c5,**=0A>> 0000000000000000=0A>> mfi0: 6061 (boot + 19s/0x= 0002/info) - Inserted: PD 05(e0x20/s5)=0A>> mfi0: 6062 (boot + 19s/0x0002/i= nfo) - Inserted: PD 05(e0x20/s5) Info:=0A>> enclPd=3D20, scsiType=3D0, port= Map=3D03, sasAddr=3D5000c50053da2b41,**=0A>> 0000000000000000=0A>> mfi0: 60= 63 (boot + 19s/0x0002/info) - Inserted: PD 06(e0x20/s6)=0A>> mfi0: 6064 (bo= ot + 19s/0x0002/info) - Inserted: PD 06(e0x20/s6) Info:=0A>> enclPd=3D20, s= csiType=3D0, portMap=3D02, sasAddr=3D4433221102000000,**=0A>> 0000000000000= 000=0A>> mfi0: 6065 (boot + 19s/0x0002/WARN) - PD 06(e0x20/s6) is not a=0A>= certified=0A>> drive=0A>> mfi0: 6066 (boot + 19s/0x0002/info) - Inserted: P= D 07(e0x20/s7)=0A>> mfi0: 6067 (boot + 19s/0x0002/info) - Inserted: PD 07(e= 0x20/s7) Info:=0A>> enclPd=3D20, scsiType=3D0, portMap=3D01, sasAddr=3D4433= 221103000000,**=0A>> 0000000000000000=0A>> mfi0: 6068 (boot + 19s/0x0002/WA= RN) - PD 07(e0x20/s7) is not a=0A>certified=0A>> drive=0A>> mfi0: 6069 (417= 120312s/0x0020/info) - Time established as 03/20/13=0A>> 18:45:12; (32 seco= nds since power on)=0A>> mfi0: 6070 (417120445s/0x0020/info) - Host driver = is loaded and=0A>operational=0A>> Timecounters tick every 1.000 msec=0A>> u= sbus0: 480Mbps High Speed USB v2.0=0A>> usbus1: 480Mbps High Speed USB v2.0= =0A>> mfid0 on mfi0=0A>> mfid0: 285568MB (584843264 sectors) RAID volume 'B= ASE' is optimal=0A>> mfid1 on mfi0=0A>> mfid1: 121472MB (248774656 sectors)= RAID volume 'L2ARC0' is optimal=0A>> mfid2 on mfi0=0A>> mfid2: 121472MB (2= 48774656 sectors) RAID volume 'ZIL0' is optimal=0A>> mfid3 on mfi0=0A>> mfi= d3: 1143552MB (2341994496 sectors) RAID volume 'DATASTRIPE1' is=0A>optimal= =0A>> mfid4 on mfi0=0A>> mfid4: 1143552MB (2341994496 sectors) RAID volume = 'DATASTRIPE2' is=0A>optimal=0A>> ugen0.1: <Intel> at usbus0=0A>> uhub0: <In= tel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on=0A>usbus0=0A>> ugen= 1.1: <Intel> at usbus1=0A>> uhub1: <Intel EHCI root HUB, class 9/0, rev 2.0= 0/1.00, addr 1> on=0A>usbus1=0A>> uhub0: 2 ports with 2 removable, self pow= ered=0A>> uhub1: 2 ports with 2 removable, self powered=0A>> ugen0.2: <vend= or 0x8087> at usbus0=0A>> uhub2: <vendor 0x8087 product 0x0024, class 9/0, = rev 2.00/0.00, addr=0A>2> on=0A>> usbus0=0A>> ugen1.2: <vendor 0x8087> at u= sbus1=0A>> uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, = addr=0A>2> on=0A>> usbus1=0A>> uhub2: 6 ports with 6 removable, self powere= d=0A>> uhub3: 8 ports with 8 removable, self powered=0A>> ugen0.3: <no manu= facturer> at usbus0=0A>> uhub4: <no manufacturer Gadget USB HUB, class 9/0,= rev 2.00/0.00,=0A>addr 3>=0A>> on usbus0=0A>> SMP: AP CPU #1 Launched!=0A>= > SMP: AP CPU #7 Launched!=0A>> SMP: AP CPU #3 Launched!=0A>> SMP: AP CPU #= 6 Launched!=0A>> SMP: AP CPU #2 Launched!=0A>> SMP: AP CPU #5 Launched!=0A>= > SMP: AP CPU #4 Launched!=0A>> Timecounter "TSC-low" frequency 9765827 Hz = quality 1000=0A>> Root mount waiting for: usbus0=0A>> uhub4: 6 ports with 6= removable, self powered=0A>> ugen0.4: <Avocent> at usbus0=0A>> ukbd0: <Key= board> on usbus0=0A>> kbd0 at ukbd0=0A>> Root mount waiting for: usbus0=0A>= > Root mount waiting for: usbus0=0A>> ugen0.5: <Avocent> at usbus0=0A>> uma= ss0: <SCSI Transparent Interface 0> on usbus0=0A>> Trying to mount root fro= m ufs:/dev/mfid0p3 [rw]...=0A>> cd0 at umass-sim0 bus 0 scbus1 target 0 lun= 0=0A>> cd0: <iDRAC Virtual CD 0329> Removable CD-ROM SCSI-0 device=0A>> cd= 0: 40.000MB/s transfers=0A>> cd0: Attempt to query device size failed: NOT = READY, Medium not=0A>present=0A>> da0 at umass-sim0 bus 0 scbus1 target 0 l= un 1=0A>> da0: <iDRAC Virtual Floppy 0329> Removable Direct Access SCSI-0= =0A>device=0A>> da0: 40.000MB/s transfers=0A>> da0: Attempt to query device= size failed: NOT READY, Medium not=0A>present=0A>> ZFS filesystem version = 5=0A>> ZFS storage pool version 28=0A>> ums0: <Mouse> on usbus0=0A>> ums0: = 3 buttons and [Z] coordinates ID=3D0=0A>> ums1: <Mouse REL> on usbus0=0A>> = ums1: 3 buttons and [XYZ] coordinates ID=3D0=0A>> igb0: link state changed = to UP=0A>>=0A>> With this loader.conf:=0A>> vm.kmem_size_max=3D"64G"=0A>> v= m.kmem_size=3D"48G"=0A>> vfs.zfs.arc_min=3D"4096M"=0A>> vfs.zfs.arc_max=3D"= 15872M"=0A>> vfs.zfs.txg.synctime=3D"1"=0A>> vfs.zfs.txg.timeout=3D"5"=0A>>= vfs.zfs.write_limit_override=3D"**0"=0A>> vfs.zfs.txg.write_limit_**overri= de=3D"0"=0A>> vfs.zfs.zil_disable=3D"0"=0A>>=0A>> And with this sysctl.conf= :=0A>> security.bsd.see_other_uids=3D0=0A>> net.inet.tcp.blackhole=3D2=0A>>= net.inet.udp.blackhole=3D1=0A>> net.inet.tcp.sendspace=3D65536=0A>> net.in= et.tcp.recvspace=3D65536=0A>> net.link.ether.inet.log_arp_**wrong_iface=3D0= =0A>> kern.ipc.somaxconn=3D32768=0A>> kern.threads.max_threads_per_**proc= =3D16384=0A>> kern.maxfiles=3D262144=0A>> kern.maxfilesperproc=3D131072=0A>= > kern.ipc.nmbclusters=3D65536=0A>> kern.corefile=3D"/var/coredumps/**%U.%N= .%P.core"=0A>>=0A>> vfs.zfs.txg.timeout=3D5=0A>> vfs.zfs.l2arc_headroom=3D1= 6=0A>> vfs.zfs.l2arc_write_max=3D**26843546=0A>> vfs.zfs.l2arc_write_boost= =3D**26843546=0A>> vfs.zfs.vdev.min_pending=3D4=0A>> vfs.zfs.vdev.max_pendi= ng=3D64=0A>> vfs.zfs.l2arc_noprefetch=3D1=0A>> vfs.zfs.dedup.prefetch=3D0= =0A>> vfs.zfs.prefetch_disable=3D1=0A>>=0A>> I will use this server as a my= sql-5.6 dbserver so I have a root=0A>partition=0A>> using a hw raid1 and a = /DATAZFS partition, follows configuration:=0A>>=0A>> root# mfiutil show dri= ves=0A>> mfi0 Physical Drives:=0A>>=C2=A0 0 (=C2=A0 279G) ONLINE <SEAGATE S= T3300657SS ES65> SAS E1:S0=0A>>=C2=A0 1 (=C2=A0 279G) ONLINE <SEAGATE ST330= 0657SS ES65> SAS E1:S1=0A>>=C2=A0 2 (=C2=A0 558G) ONLINE <SEAGATE ST3600057= SS ES65> SAS E1:S2=0A>>=C2=A0 3 (=C2=A0 558G) ONLINE <SEAGATE ST3600057SS E= S65> SAS E1:S3=0A>>=C2=A0 4 (=C2=A0 558G) ONLINE <SEAGATE ST3600057SS ES65>= SAS E1:S4=0A>>=C2=A0 5 (=C2=A0 558G) ONLINE <SEAGATE ST3600057SS ES65> SAS= E1:S5=0A>>=C2=A0 6 (=C2=A0 119G) ONLINE <OCZ-VERTEX4 1.4> SATA E1:S6=0A>>= =C2=A0 7 (=C2=A0 119G) ONLINE <OCZ-VERTEX4 1.4> SATA E1:S7=0A>> root# mfiut= il show volumes=0A>> mfi0 Volumes:=0A>>=C2=A0 Id=C2=A0 =C2=A0 Size=C2=A0 = =C2=A0 Level=C2=A0 Stripe=C2=A0 State=C2=A0 Cache=C2=A0 Name=0A>>=C2=A0 = mfid0 (=C2=A0 278G) RAID-1=C2=A0 =C2=A0 =C2=A0 64k OPTIMAL Disabled <BASE>= =0A>>=C2=A0 mfid1 (=C2=A0 118G) RAID-0=C2=A0 =C2=A0 =C2=A0 64k OPTIMAL Disa= bled <L2ARC0>=0A>>=C2=A0 mfid2 (=C2=A0 118G) RAID-0=C2=A0 =C2=A0 =C2=A0 64k= OPTIMAL Disabled <ZIL0>=0A>>=C2=A0 mfid3 ( 1116G) RAID-1=C2=A0 =C2=A0 =C2= =A0 64k OPTIMAL Disabled <DATA>=0A>>=0A>> I have a /DATAZFS partition, too:= =0A>> root# zfs get all DATAZFS=0A>> NAME=C2=A0 =C2=A0 PROPERTY=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 VALUE=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 SOURCE=0A>> DATAZFS=C2=A0 type=C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 filesystem=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 -=0A>> DATAZFS=C2=A0 creation=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Thu Mar 21 10:45 2013=C2=A0 -=0A>> DATAZFS= =C2=A0 used=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 5= 4.2G=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -=0A>> D= ATAZFS=C2=A0 available=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 1.02T=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -=0A>> DATAZFS= =C2=A0 referenced=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 54.2G=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -=0A>> DATAZFS=C2=A0 c= ompressratio=C2=A0 =C2=A0 =C2=A0 =C2=A0 1.00x=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -=0A>> DATAZFS=C2=A0 mounted=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 yes=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -=0A>> DATAZFS=C2=A0 quota=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 none=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS=C2=A0 reservat= ion=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 none=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS=C2=A0 recordsize=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 16K=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 local=0A>> DATAZFS=C2=A0 mountpoint= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /DATAZFS=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS=C2=A0 sharenfs=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 off=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS=C2=A0 checksum=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 on=C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS=C2=A0 comp= ression=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 off=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS=C2=A0 atime= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 off=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 local=0A>> DATAZFS= =C2=A0 devices=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 on=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>= > DATAZFS=C2=A0 exec=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 on=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 default=0A>> DATAZFS=C2=A0 setuid=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 on=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS=C2=A0 readonly=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 off=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS=C2=A0 jailed=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 off=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS=C2=A0 sn= apdir=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 hidden=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS=C2=A0 aclmo= de=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 discard=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS=C2=A0 aclinh= erit=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 restricted=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS=C2=A0 canmount=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 on=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS=C2=A0 xattr=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 off=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 temporary=0A>> DATAZFS=C2=A0 = copies=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 1=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default= =0A>> DATAZFS=C2=A0 version=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= 5=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 -=0A>> DATAZFS=C2=A0 utf8only=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 off=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 -=0A>> DATAZFS=C2=A0 normalization=C2=A0 =C2=A0 =C2=A0 =C2=A0 n= one=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -=0A>> D= ATAZFS=C2=A0 casesensitivity=C2=A0 =C2=A0 =C2=A0 sensitive=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -=0A>> DATAZFS=C2=A0 vscan=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 off=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS=C2=A0 nbmand= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 off=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZF= S=C2=A0 sharesmb=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 off=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>>= DATAZFS=C2=A0 refquota=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 non= e=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A= >> DATAZFS=C2=A0 refreservation=C2=A0 =C2=A0 =C2=A0 =C2=A0 none=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS= =C2=A0 primarycache=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 all=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS= =C2=A0 secondarycache=C2=A0 =C2=A0 =C2=A0 =C2=A0 all=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS=C2=A0= usedbysnapshots=C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -=0A>> DATAZFS=C2=A0 usedbydatase= t=C2=A0 =C2=A0 =C2=A0 =C2=A0 54.2G=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 -=0A>> DATAZFS=C2=A0 usedbychildren=C2=A0 =C2=A0 = =C2=A0 =C2=A0 1.41M=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 -=0A>> DATAZFS=C2=A0 usedbyrefreservation=C2=A0 0=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -=0A>> DATAZFS= =C2=A0 logbias=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 latency=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS= =C2=A0 dedup=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 off=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 local=0A= >> DATAZFS=C2=A0 mlslabel=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 -=0A>> DATAZFS=C2=A0 sync=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 standard=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = default=0A>> DATAZFS=C2=A0 refcompressratio=C2=A0 =C2=A0 =C2=A0 1.00x=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -=0A>> DATAZFS= =C2=A0 written=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 54.2G=C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -=0A>>=0A>> root# = zpool status -v=0A>>=C2=A0 pool: DATAZFS=0A>>=C2=A0 state: ONLINE=0A>>=C2= =A0 scan: none requested=0A>> config:=0A>>=0A>> NAME=C2=A0 =C2=A0 =C2=A0 = =C2=A0 STATE=C2=A0 =C2=A0 READ WRITE CKSUM=0A>> DATAZFS=C2=A0 =C2=A0 ONLI= NE=C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 0=C2=A0 =C2=A0 0=0A>>=C2=A0 mfid3= =C2=A0 =C2=A0 ONLINE=C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 0=C2=A0 =C2=A0 = 0=0A>> logs=0A>>=C2=A0 mfid2=C2=A0 =C2=A0 ONLINE=C2=A0 =C2=A0 =C2=A0 0= =C2=A0 =C2=A0 0=C2=A0 =C2=A0 0=0A>> cache=0A>>=C2=A0 mfid1=C2=A0 =C2=A0 = ONLINE=C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 0=C2=A0 =C2=A0 0=0A>>=0A>> er= rors: No known data errors=0A>>=0A>> root# mfiutil show firmware=0A>> mfi0 = Firmware Package Version: 21.1.0-0007=0A>> mfi0 Firmware Images:=0A>> Name= =C2=A0 Version=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 Date=C2=A0 =C2=A0 =C2=A0 =C2=A0 Time=C2=A0 =C2=A0= =C2=A0 =C2=A0 Status=0A>> BIOS=C2=A0 5.31.01_4.12.05.00_0x05110000=C2=A0 = 6/20/2012=0A>>=C2=A0 =C2=A0 6/20/2012=0A>>=C2=A0 active=0A>> CTLR=C2=A0 4= .00-0014=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 Aug 04 2011=C2=A0 12:49:17=C2=A0 =C2=A0 active=0A>> PCLI=C2=A0 = 05.00-03:#%00008=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Feb 17 20= 11=C2=A0 14:03:12=C2=A0 =C2=A0 active=0A>> APP=C2=A0 3.130.05-1796=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Jul 27 2012=C2=A0 1= 5:58:31=C2=A0 =C2=A0 active=0A>> NVDT=C2=A0 2.1108.03-0094=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Jun 22 2012=C2=A0 12:40:55=C2=A0= =C2=A0 active=0A>> BTBL=C2=A0 2.03.00.00-0004=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 Jul 27 2012=C2=A0 16:39:29=C2=A0 =C2=A0 active= =0A>> BOOT=C2=A0 06.253.57.219=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 9/9/2010=C2=A0 =C2=A0 15:32:25=C2=A0 =C2=A0 active=0A>>= root#=0A>>=0A>> Then, I installed CentOS on the same hardware, same versio= n of mysql=0A>> (5.6.10-enterprise and 5.6.10-GA), and I tried an OLTP test= (included=0A>in=0A>> sysbench-0.5) executing 2000000 queries on 10000000 r= ows, with these=0A>> results (transactions per seconds):=0A>>=0A>> BSD 9.1 = amd64 ZFS (without tweaks):=0A>> 1 thread: 2780=0A>> 2 threads: 4371=0A>> 4= threads: 6876=0A>> 8 threads: 8202=0A>> 12 threads: 8077=0A>> 16 threads: = 7780=0A>> 24 threads: 7563=0A>> 32 threads: 7632=0A>> 48 threads: 7960=0A>>= 64 threads: 8062=0A>>=0A>> BSD 9.1 amd64 ZFS (with tweaks):=0A>> 1 thread:= 2589=0A>> 2 threads: 4679=0A>> 4 threads: 6438=0A>> 8 threads: 7073=0A>> 1= 2 threads: 7121=0A>> 16 threads: 7227=0A>> 24 threads: 7132=0A>> 32 threads= : 7273=0A>> 48 threads: 7623=0A>> 64 threads: 7672=0A>>=0A>> CentOS 6.3 (ex= t4, without tweaks):=0A>> 1 thread: 4163=0A>> 2 threads: 7653=0A>> 4 thread= s: 10905=0A>> 8 threads: 12511=0A>> 12 threads: 13556=0A>> 16 threads: 1483= 2=0A>> 24 threads: 16270=0A>> 32 threads: 16733=0A>> 48 threads: 16925=0A>>= 64 threads: 16895=0A>>=0A>>=0A>> Well, any of you could ask "why zfs, plea= se try ufs that is better":=0A>you=0A>> are right, but changing ZFS with UF= S doesn't change so much:=0A>>=0A>http://lists.freebsd.org/**pipermail/free= bsd-fs/2013-**March/016747.html<http://lists.freebsd.org/pipermail/freebsd-= fs/2013-March/016747.html>=0A>>=0A>> So I hope I'm missing something becaus= e the performances of a=0A>standard=0A>> CentOS installation on the same ha= rdware performs much better than a=0A>9.1=0A>> amd64 installation (I'm usin= g the standard ULE scheduler included in=0A>the=0A>> 9.1 release).=0A>>=0A>= > Could anyone help me?=0A>>=0A>> Thanks,=0A>> d.=0A>> ____________________= __________**_________________=0A>> freebsd-performance@freebsd.**org=0A><fr= eebsd-performance@freebsd.org>mailing list=0A>>=0A>http://lists.freebsd.org= /**mailman/listinfo/freebsd-**performance<http://lists.freebsd.org/mailman/= listinfo/freebsd-performance>=0A>> To unsubscribe, send any mail to "freebs= d-performance-**=0A>> unsubscribe@freebsd.org=0A><freebsd-performance-unsub= scribe@freebsd.org>"=0A>>=0A=0A--=0Ad.=0A__________________________________= _____________=0Afreebsd-performance@freebsd.org mailing list=0Ahttp://lists= .freebsd.org/mailman/listinfo/freebsd-performance=0ATo unsubscribe, send an= y mail to "freebsd-performance-unsubscribe@freebsd.org" From owner-freebsd-performance@FreeBSD.ORG Sat Mar 23 07:44:34 2013 Return-Path: <owner-freebsd-performance@FreeBSD.ORG> Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 8632261A for <freebsd-performance@freebsd.org>; Sat, 23 Mar 2013 07:44:34 +0000 (UTC) (envelope-from davide.damico@contactlab.com) Received: from mail2.shared.smtp.contactlab.it (mail2.shared.smtp.contactlab.it [93.94.37.7]) by mx1.freebsd.org (Postfix) with ESMTP id 14A42979 for <freebsd-performance@freebsd.org>; Sat, 23 Mar 2013 07:44:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; d=contactlab.it; s=clab1; c=relaxed/relaxed; q=dns/txt; i=@contactlab.it; t=1364024671; h=From:Subject:Date:To:MIME-Version:Content-Type; bh=sVP3v1AW07tk+cpo0iIlPaC3P7T3DOUwyUlqA589++k=; b=VHxXWhW9/BZ619E3u+b6/OD4A5E8O2kPCaU1KKKVQT3jY5LxuovJ3eH4cFQ6kAbO Ina/P4HYUs7OmJlTl0tBWAgaTU6TqFO6+aUOZ3arVqlGrqGNv8pA5ZsX/SDPEGnc OsGhAzw078HrPOFG+i7ptFGYWzEzQ4vWwt5yWoFyj+E=; Received: from [213.92.90.12] ([213.92.90.12:26194] helo=mail3.tomato.it) by t.contactlab.it (envelope-from <davide.damico@contactlab.com>) (ecelerity 3.5.1.37854 r(Momo-dev:3.5.1.0)) with ESMTP id FB/91-24145-F5D5D415; Sat, 23 Mar 2013 08:44:31 +0100 Received: from mx3-master.housing.tomato.lan ([172.16.7.55]) by mail3.tomato.it with smtp (Exim 4.80.1 (FreeBSD)) (envelope-from <davide.damico@contactlab.com>) id 1UJJ7v-000Pn0-CO for freebsd-performance@freebsd.org; Sat, 23 Mar 2013 08:44:31 +0100 Received: (qmail 99132 invoked by uid 80); 23 Mar 2013 07:44:31 -0000 To: Paul Pathiakis <pathiaki2@yahoo.com> Subject: Re: FreeBSD 9.1 vs CentOS 6.3 X-PHP-Script: uebmeil.sys.tomatointeractive.it/index.php for 172.16.16.227 X-PHP-Originating-Script: 0:main.inc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Sat, 23 Mar 2013 08:44:31 +0100 From: Davide D'Amico <davide.damico@contactlab.com> Organization: ContactLab Mail-Reply-To: <davide.damico@contactlab.com> In-Reply-To: <1363998883.22604.YahooMailNeo@web141401.mail.bf1.yahoo.com> References: <CAF3xD3nDDUYB94TS9AUUQ=CPztB1S8mU4fRwNB5GupmK8MgwXg@mail.gmail.com> <ebfc033f-d8db-4162-86be-db013e3d7eb2@email.android.com> <1363998883.22604.YahooMailNeo@web141401.mail.bf1.yahoo.com> Message-ID: <5d10e2a0b0f6477293459a26df1fc272@sys.tomatointeractive.it> X-Sender: davide.damico@contactlab.com User-Agent: Roundcube Webmail/0.8.5 Cc: freebsd-performance@freebsd.org, =?UTF-8?Q?=D0=95=D0=B2=D0=B3=D0=B5=D0=BD=D0=B8=D0=B9_=D0=A5=D0=BE?= =?UTF-8?Q?=D1=80=D0=BA=D0=B8=D0=BD?= <khorkin.eugeniy@gmail.com> X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: davide.damico@contactlab.com List-Id: Performance/tuning <freebsd-performance.freebsd.org> List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-performance>, <mailto:freebsd-performance-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-performance>; List-Post: <mailto:freebsd-performance@freebsd.org> List-Help: <mailto:freebsd-performance-request@freebsd.org?subject=help> List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-performance>, <mailto:freebsd-performance-request@freebsd.org?subject=subscribe> X-List-Received-Date: Sat, 23 Mar 2013 07:44:34 -0000 Il 23.03.2013 01:34 Paul Pathiakis ha scritto: > Hi, > > There are several things about this that are highly suspect. > > First, wipe out the hardware RAID. The processor doing RAID > computation is, probably, MUCH slower than a core on the CPU. Even if > it's RAID-1 (Simple Mirror) this RAID card is performing tasks that is > does not need to do including replicating writes to two targets from > the controller or checking it's cache, battery, etc. If it's possible > to disable the onboard cache, do it. Hi Paul, thanks for your suggestions (some of them I've applied before starting any consideration, like disabling all on-disk caches or controller buffers) I'll try next monday. Anyway, the fact is that using the same hardware configuration (raid1+raid10) I saw that a centos 6.x outperformed freebsd 9.1. Another test I made yesterday was: on the same hardware I installed vmware esx 5.x and created a vm with centos inside it. The result was really impressive: the centos vm outperformed the 'real' freebsd 9.1 too and checking vmware performances graphs I didn't see any huge need for a massive throughput (I saw values from KBps to 10MBps), instead I saw a big use of CPU (using OLTP tests with a concurrency of 32 threads it's performaces began to slow down). I don't know is using some magic value for HZ or setting some trick with scheduler, I could gain something: I hope so, because I don't want to "pinguinate" my farm :) Thanks, d.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1363998883.22604.YahooMailNeo>