Date: Fri, 4 Feb 2005 17:44:42 -0700 From: Nick Pavlica <linicks@gmail.com> To: Jeff Roberson <jroberson@chesapeake.net> Cc: freebsd-performance@freebsd.org Subject: Re: My disk I/O testing methods for FreeBSD 5.3 ... Message-ID: <dc9ba04405020416443d9fb168@mail.gmail.com> In-Reply-To: <20050203230343.M18864@mail.chesapeake.net> References: <dc9ba044050203143647cee0c2@mail.gmail.com> <20050203175519.K18864@mail.chesapeake.net> <dc9ba0440502031548544723a1@mail.gmail.com> <20050203230343.M18864@mail.chesapeake.net>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --]
Jeff,
Here is the data from the same test on the exact same server with
4.11R installed.
minime# time dd bs=1024 if=/dev/zero of=tstfile count=1m
1048576+0 records in
1048576+0 records out
1073741824 bytes transferred in 25.278113 secs (42477135 bytes/sec)
------------------------------------------------------------------------------------------------------------------------
real 0m25.387s
user 0m0.719s
sys 0m6.794s
------------------------------------------------------------------------------------------------------------------------
minime# sysctl hw.ata
hw.ata.ata_dma: 1
hw.ata.wc: 1
hw.ata.tags: 0
hw.ata.suspend: 0
hw.ata.atapi_dma: 0
------------------------------------------------------------------------------------------------------------------------
minime# atacontrol list
ATA channel 0:
Master: ad0 <ST340014A/3.16> ATA/ATAPI rev 6
Slave: no device present
ATA channel 1:
Master: acd0 <Lite-On LTN486S 48x Max/YDS6> ATA/ATAPI rev 0
Slave: no device present
ATA channel 2:
Master: no device present
Slave: no device present
ATA channel 3:
Master: no device present
Slave: no device present
------------------------------------------------------------------------------------------------------------------------
minime# atacontrol mode 0
Master = UDMA100
Slave = ???
------------------------------------------------------------------------------------------------------------------------
minime# atacontrol mode 1
Master = PIO4
Slave = ???
------------------------------------------------------------------------------------------------------------------------
--Nick
On Thu, 3 Feb 2005 23:03:55 -0500 (EST), Jeff Roberson
<jroberson@chesapeake.net> wrote:
>
> On Thu, 3 Feb 2005, Nick Pavlica wrote:
>
> > Jeff,
> > One of the tests where I saw a large difference was in DD. I did a
> > quick test on a server that was brought up to RELENG_5 via cvsup on
> > 2/2/05.
>
> Can you give me the same from RELENG_4?
>
> >
> > The Test:
> > -bash-2.05b$ time dd bs=1024 if=/dev/zero of=tstfile count=1M
> > 1048576+0 records in
> > 1048576+0 records out
> > 1073741824 bytes transferred in 74.402757 secs (14431479 bytes/sec)
> >
> > real 1m14.498s
> > user 0m0.550s
> > sys 0m8.838s
> >
> >
> > The vmstat -1 info is attached.
> >
> >
> > Thanks!
> > --Nick
> >
> >
> >
> > On Thu, 3 Feb 2005 17:59:30 -0500 (EST), Jeff Roberson
> > <jroberson@chesapeake.net> wrote:
> > >
> > > On Thu, 3 Feb 2005, Nick Pavlica wrote:
> > >
> > > > All,
> > > > I would like to share the methods that I have been using in my disk
> > > > I/O testing. The detailed results of these tests have been posted to
> > > > the performance and questions mailing lists under the title " FreeBSD
> > > > 5.3 I/O Performance / Linux 2.6.10 | Continued Discussion". I
> > > > originally started this testing as due diligence in an up coming
> > > > project. As a result of this testing I discovered an elegant
> > > > operating system that I enjoy working with.
> > >
> > > Nick, first, I'd like to thank you for your efforts so far. I think your
> > > tests have been very informative. I'd like to see what we can do to get
> > > to the bottom of the differences. Can you perform one test which varied
> > > greatly between 5.x and 4.x and collect some data for us? To start with,
> > > the output of vmstat 1 piped to a file would be informative. Do you have
> > > any indication that 5.x is actually cpu bound in a case where 4.x is not?
> > > I'm wondering if this is a latency issue or a cpu utilization issue.
> > >
> > > I intend to backport some code that lets me graph system activity into
> > > RELENG_5. Are you setup to cvsup to this tag? Would it be convenient for
> > > you to do so?
> > >
> > > Thanks,
> > > Jeff
> > >
> > > >
> > > > Intent Of This Testing:
> > > > 1)To measure the disk I/O performance of various operating systems for
> > > > use as a production database server.
> > > > 2)Help improve the disk I/O performance of FreeBSD 5.x and greater by
> > > > assisting the FreeBSD development team in identifying possible
> > > > performance issues, and provide them with data to measure the success
> > > > of various changes to the operating system.
> > > >
> > > > Operating Systems tested:
> > > > Fedora Core 3 with EXT3, and XFS. I tested with and with out patches.
> > > > SUSE Enterprise Server 9 with Riser FS.
> > > > FreeBSD 4.11R
> > > > FreeBSD 5.3R, RELENG_5_3, RELENG_5
> > > > NetBSD 2.0R
> > > > OpenBSD 3.6R
> > > >
> > > > Test Hardware:
> > > > Compaq DeskPro, PIII 800, 384Mb Ram, 10Gb IDE HD.
> > > > Dell PE 2400, Dual PIII 550, 512Mb Ram, (2)10K,LVD SCSI, RAID 1, PERC
> > > > 2SI controller with 64Mb ram.
> > > > Dell PE SC400, 2.4Ghz P4, 256MB Ram, 40Gb IDE HD.
> > > > Dell 4600, 2.8 Ghz P4 with HT, 512MB Ram, 80GB IDE HD.
> > > >
> > > > Installation Notes:
> > > > It's my intention to test these Operating Systems using as many of
> > > > the default installation options as possible with no special tuning.
> > > > The only deviations in my previous testing were as follows: The #linux
> > > > xfs option was used when installing Fedora so that I could use XFS,
> > > > and a special test where I installed 5.3R with UFS instead of UFS2 (I
> > > > didn't see any improvement when using UFS). I installed FreeBSD using
> > > > the standard install option, and used the auto allocate features for
> > > > partitioning and slicing. I installed Fedora with the stock server
> > > > packages and created a 100Mb /boot, 512Mb swap, and allocated the
> > > > remaining space to /. I tested FreeBSD5.3R and FC3R with and without
> > > > updates. I used cvsup to update FreeBSD and yum update to update
> > > > Fedora. I didn't do any updating to FreeBSD4.11R, NetBSD2.0, and
> > > > OpenBSD3.6.
> > > >
> > > > I used the following utilities/tools in my testing:
> > > > DD
> > > > CP
> > > > IOSTAT (iostat -d 2)
> > > > Bonnie++
> > > > TOP
> > > > SQL,PL, PSQL
> > > > Postgresql 8.0
> > > >
> > > > DD Example Tests:
> > > > - #time dd bs=1024 if=/dev/zero of=tstfile count=1M
> > > > - #time dd bs=1024 if=/dev/zero of=tstfile count=2M
> > > > - #time dd bs=1024 if=/dev/zero of=tstfile count=3M
> > > >
> > > > Bonnie++ Example Tests:
> > > > #bonnie++ -u root -s 1024 -r 512 -n 5
> > > > #bonnie++ -u root -s 2048 -r 512 -n 5
> > > > #bonnie++ -u root -s 3072 -r 512 -n 5
> > > >
> > > > CP Example Tests:
> > > > #time cp tstfile tstfile2
> > > >
> > > > SQL, PL, PSQL Example Tests:
> > > >
> > > > CREATE TABLE test1 (
> > > > thedate TIMESTAMP,
> > > > astring VARCHAR(200),
> > > > anumber INTEGER
> > > > );
> > > >
> > > > CREATE FUNCTION build_data() RETURNS integer AS '
> > > > DECLARE
> > > > i INTEGER DEFAULT 0;
> > > > curtime TIMESTAMP;
> > > > BEGIN
> > > > FOR i IN 1..1000000 LOOP
> > > > curtime := ''now'';
> > > > INSERT INTO test1 VALUES (curtime, ''test string'', i);
> > > > END LOOP;
> > > > RETURN 1;
> > > > END;
> > > > ' LANGUAGE 'plpgsql';
> > > >
> > > > SELECT build_data();
> > > > Then the following script is run under the time program to ascertain
> > > > how long it takes to run:
> > > > CREATE TABLE test2 (
> > > > thedate TIMESTAMP,
> > > > astring VARCHAR(200),
> > > > anumber INTEGER
> > > > );
> > > > CREATE TABLE test3 AS SELECT * FROM test1;
> > > > INSERT INTO test2 SELECT * FROM test1 WHERE ((anumber % 2) = 0);
> > > > DELETE FROM test3 WHERE ((anumber % 2) = 0);
> > > > DELETE FROM test3 WHERE ((anumber % 13) = 0);
> > > > CREATE TABLE test4 AS
> > > > SELECT test1.thedate AS t1date,
> > > > test2.thedate AS t2date,
> > > > test1.astring AS t1string,
> > > > test2.astring AS t2string,
> > > > test1.anumber AS t1number,
> > > > test2.anumber AS t2number
> > > > FROM test1 JOIN test2 ON test1.anumber=test2.anumber;
> > > > UPDATE test3 SET thedate='now' WHERE ((anumber % 5) = 0);
> > > > DROP TABLE test4;
> > > > CREATE TABLE test4 AS SELECT * FROM test1;
> > > > DELETE FROM test4 WHERE ((anumber % 27) = 0);
> > > > VACUUM ANALYZE;
> > > > VACUUM FULL;
> > > > DROP TABLE test4;
> > > > DROP TABLE test3;
> > > > DROP TABLE test2;
> > > > VACUUM FULL;
> > > >
> > > > Example FS TAB:
> > > >
> > > > minime# cat /etc/fstab
> > > > # Device Mountpoint FStype Options Dump Pass#
> > > > /dev/ad0s1b none swap sw 0 0
> > > > /dev/ad0s1a / ufs rw 1 1
> > > > /dev/ad0s1e /tmp ufs rw 2 2
> > > > /dev/ad0s1f /usr ufs rw 2 2
> > > > /dev/ad0s1d /var ufs rw 2 2
> > > > /dev/acd0 /cdrom cd9660 ro,noauto 0 0
> > > >
> > > > Verification Of Test:
> > > > I have been able to get consistent results in all of my testing.
> > > > However, I think the best verification would be to have as many people
> > > > as possible test the disk I/O performance on a range of hardware,
> > > > testing methods, and configurations.
> > > >
> > > > Summary Of Results:
> > > > The results of my testing have consistently demonstrated that
> > > > FreeBSD5.3+ has dramatically slower disk I/O performance than all of
> > > > the other operating systems that were tested. FreeBSD 4.11R was the
> > > > performance leader followed by Fedora C3 with XFS. All of the BSD
> > > > distributions, with the exception of 5.3+, were able to consistently
> > > > demonstrate a throughput of 56-58Mb/s sustained throughput, while 5.3+
> > > > consistently demonstrated a throughput of 12-15Mb/s (58 -15 = 43 ?).
> > > >
> > > > Please let me know if you need any additional details.
> > > >
> > > > Thanks!
> > > > --Nick Pavlica
> > > > _______________________________________________
> > > > freebsd-performance@freebsd.org mailing list
> > > > http://lists.freebsd.org/mailman/listinfo/freebsd-performance
> > > > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"
> > > >
> > >
> >
>
[-- Attachment #2 --]
Copyright (c) 1992-2005 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD 4.11-RELEASE #0: Fri Jan 21 17:21:22 GMT 2005
root@perseus.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC
Timecounter "i8254" frequency 1193182 Hz
CPU: Intel(R) Pentium(R) 4 CPU 2.26GHz (2261.01-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0xf29 Stepping = 9
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
real memory = 267862016 (261584K bytes)
avail memory = 255152128 (249172K bytes)
Preloaded elf kernel "kernel" at 0xc055c000.
Warning: Pentium 4 CPU: PSE disabled
Pentium Pro MTRR support enabled
md0: Malloc disk
Using $PIR table, 8 entries at 0xc00feae0
npx0: <math processor> on motherboard
npx0: INT 16 interface
pcib0: <Host to PCI bridge> on motherboard
pci0: <PCI bus> on pcib0
agp0: <Intel 82875P host to AGP bridge> mem 0xf0000000-0xf7ffffff at device 0.0 on pci0
pcib1: <PCI to PCI bridge (vendor=8086 device=2579)> at device 1.0 on pci0
pci1: <PCI bus> on pcib1
uhci0: <Intel 82801EB (ICH5) USB controller USB-A> port 0xff80-0xff9f irq 11 at device 29.0 on pci0
usb0: <Intel 82801EB (ICH5) USB controller USB-A> on uhci0
usb0: USB revision 1.0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1: <Intel 82801EB (ICH5) USB controller USB-B> port 0xff60-0xff7f irq 10 at device 29.1 on pci0
usb1: <Intel 82801EB (ICH5) USB controller USB-B> on uhci1
usb1: USB revision 1.0
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2: <Intel 82801EB (ICH5) USB controller USB-C> port 0xff40-0xff5f irq 9 at device 29.2 on pci0
usb2: <Intel 82801EB (ICH5) USB controller USB-C> on uhci2
usb2: USB revision 1.0
uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
uhci3: <Intel 82801EB (ICH5) USB controller USB-D> port 0xff20-0xff3f irq 11 at device 29.3 on pci0
usb3: <Intel 82801EB (ICH5) USB controller USB-D> on uhci3
usb3: USB revision 1.0
uhub3: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub3: 2 ports with 2 removable, self powered
pci0: <USB controller> at 29.7 irq 5
pcib2: <Intel 82801BA/BAM (ICH2) Hub to PCI bridge> at device 30.0 on pci0
pci2: <PCI bus> on pcib2
pci2: <ATI Mach64-GR graphics accelerator> at 0.0 irq 5
em0: <Intel(R) PRO/1000 Network Connection, Version - 1.7.42> port 0xddc0-0xddff mem 0xfe9e0000-0xfe9fffff irq 9 at device 12.0 on pci2
em0: Speed:N/A Duplex:N/A
isab0: <PCI to ISA bridge (vendor=8086 device=24d0)> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH5 ATA100 controller> port 0xffa0-0xffaf,0x374-0x377,0x170-0x177,0x3f4-0x3f7,0x1f0-0x1f7 mem 0xfebffc00-0xfebfffff irq 9 at device 31.1 on pci0
ata0: at 0x1f0 irq 14 on atapci0
ata1: at 0x170 irq 15 on atapci0
atapci1: <Intel ICH5 SATA150 controller> port 0xfea0-0xfeaf,0xfe30-0xfe33,0xfe20-0xfe27,0xfe10-0xfe13,0xfe00-0xfe07 irq 9 at device 31.2 on pci0
ata2: at 0xfe00 on atapci1
ata3: at 0xfe20 on atapci1
pci0: <unknown card> (vendor=0x8086, dev=0x24d3) at 31.3 irq 10
pci0: <unknown card> (vendor=0x8086, dev=0x24d5) at 31.5 irq 10
orm0: <Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc97ff,0xc9800-0xcbfff on isa0
pmtimer0 on isa0
fdc0: <NEC 72065B or clone> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> flags 0x1 irq 1 on atkbdc0
kbd0 at atkbd0
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: model Generic PS/2 mouse, device ID 0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A
sio1 at port 0x2f8-0x2ff irq 3 on isa0
sio1: type 16550A
ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/8 bytes threshold
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
ad0: 38146MB <ST340014A> [77504/16/63] at ata0-master UDMA100
acd0: CDROM <Lite-On LTN486S 48x Max> at ata1-master PIO4
Mounting root from ufs:/dev/ad0s1a
em0: Link is up 100 Mbps Full Duplex
[-- Attachment #3 --]
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr ad0 md0 in sy cs us sy id
0 0 0 13708 219060 9 0 0 0 6 0 0 0 231 31 6 0 0 100
0 0 0 13272 219052 8 0 0 0 0 0 0 0 232 23 5 0 0 100
0 0 0 12920 219052 3 0 0 0 0 0 0 0 232 10 4 0 0 100
1 0 0 13156 194000 36 0 3 0 23 0 187 0 426 49623 43 1 17 82
0 1 0 13156 151476 3 0 0 0 18 0 331 0 564 84744 61 2 25 73
0 1 0 13156 106888 6 0 0 0 0 0 341 0 575 88854 62 3 29 67
0 1 0 13156 63792 3 0 0 0 0 0 334 0 567 85770 60 4 26 71
0 1 0 13156 21240 3 0 0 0 0 0 336 0 570 84748 59 2 30 68
0 1 0 13592 14920 3 0 0 0 6009 10363 338 0 571 88845 70 4 33 64
0 1 0 13592 10392 3 0 0 0 10710 9578 333 0 566 85258 69 2 31 67
0 11 0 17628 12092 24 0 1 0 11084 11493 342 0 587 88191 85 4 28 68
1 1 0 18096 9476 9 0 0 0 10229 9574 316 0 549 81400 70 3 29 68
0 5 0 18452 14488 7 0 0 0 10237 11490 316 0 549 81460 75 2 29 69
0 5 0 18452 10004 3 0 0 0 10698 9577 329 0 562 85258 69 2 31 67
0 5 0 18488 14648 4 0 2 0 10333 11494 320 0 557 82188 74 4 27 69
0 13 0 18492 11884 7 0 1 0 10266 9575 321 0 556 81694 72 1 31 68
0 13 0 18492 15068 3 0 0 0 10697 11493 329 0 563 85258 72 3 29 68
1 12 0 18512 12368 4 0 1 0 10250 9576 319 0 554 81580 69 2 29 68
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr ad0 md0 in sy cs us sy id
0 5 0 18596 9564 13 0 2 0 10282 9577 321 0 557 81791 74 1 29 70
0 5 0 18596 12608 3 0 0 0 10730 11491 329 0 562 85514 74 2 33 65
0 4 0 14936 10216 7 0 1 0 10173 9575 316 0 551 80935 71 1 29 71
0 4 0 14936 13268 3 0 0 0 10730 11496 329 0 563 85514 72 4 31 65
1 0 0 10140 10296 5 0 0 0 10348 9577 320 0 554 82366 72 2 27 71
0 1 0 10140 14680 3 0 0 0 10397 11493 324 0 559 82807 75 4 27 69
0 1 0 10140 10068 3 0 0 0 10730 9577 329 0 569 85565 76 4 28 68
0 5 0 10028 15884 37 0 2 0 10068 11493 313 0 555 79993 87 3 33 64
1 7 0 10844 13924 25 0 3 0 10067 9576 315 0 551 79845 73 5 24 71
0 8 0 10408 11612 3 0 0 0 10155 9577 314 0 547 80968 69 3 29 68
0 1 0 12880 15592 196 9 6 0 4968 5747 179 0 437 38204 115 5 12 83
0 0 0 13332 15436 50 0 1 0 179 0 22 0 259 109 29 0 1 99
0 0 0 13332 15652 6 0 0 0 60 0 21 0 253 22 6 0 0 100
0 0 0 12892 15652 3 0 0 0 0 0 0 0 232 10 4 0 0 100
0 0 0 12540 15652 3 0 0 0 0 0 0 0 231 10 4 0 0 100
0 0 0 12540 15668 3 0 0 0 8 0 2 0 234 10 5 0 0 100
0 0 0 12980 15644 60 4 2 0 56 0 5 0 242 149 51 0 0 100
0 0 0 12980 15644 6 0 0 0 0 0 0 0 231 22 5 0 0 100
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr ad0 md0 in sy cs us sy id
0 0 0 12980 15644 3 0 0 0 0 0 0 0 230 10 4 0 0 100
0 0 0 14344 15056 286 0 8 0 230 0 31 0 286 1010 139 1 0 99
0 0 0 14344 15056 3 0 0 0 0 0 0 0 231 10 5 0 0 100
0 0 0 14344 15056 6 0 0 0 0 0 0 0 234 50 10 0 0 100
0 0 0 14344 15056 6 0 0 0 0 0 0 0 234 62 15 0 1 99
0 0 0 14344 15056 3 0 0 0 0 0 5 0 239 34 8 0 0 100
0 0 0 14696 15056 3 0 0 0 0 0 0 0 240 93 17 0 0 100
0 0 0 14696 15056 3 0 0 0 0 0 0 0 242 90 17 0 0 100
0 0 0 14696 15056 3 0 0 0 0 0 0 0 236 50 10 0 0 100
0 0 0 14696 15056 6 0 0 0 0 0 0 0 233 23 5 0 0 100
0 0 0 14696 15052 3 0 0 0 1 0 0 0 232 10 4 0 0 100
0 0 0 14696 15052 3 0 0 0 0 0 0 0 235 50 10 0 0 100
0 0 0 11028 15052 3 0 0 0 0 0 0 0 235 50 11 0 2 98
0 0 0 11372 14820 113 0 2 0 94 0 8 0 245 205 24 0 1 99
0 0 0 11372 14820 6 0 0 0 0 0 0 0 230 22 5 0 0 100
0 0 0 11372 14820 5 0 0 0 0 0 0 0 231 29 7 0 0 100
0 0 0 11372 14820 3 0 0 0 0 0 0 0 231 10 4 0 0 100
help
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?dc9ba04405020416443d9fb168>