Date: Thu, 3 Feb 2005 16:48:20 -0700 From: Nick Pavlica <linicks@gmail.com> To: Jeff Roberson <jroberson@chesapeake.net> Cc: freebsd-performance@freebsd.org Subject: Re: My disk I/O testing methods for FreeBSD 5.3 ... Message-ID: <dc9ba0440502031548544723a1@mail.gmail.com> In-Reply-To: <20050203175519.K18864@mail.chesapeake.net> References: <dc9ba044050203143647cee0c2@mail.gmail.com> <20050203175519.K18864@mail.chesapeake.net>
next in thread | previous in thread | raw e-mail | index | archive | help
------=_Part_2051_5270362.1107474500093 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Disposition: inline Jeff, One of the tests where I saw a large difference was in DD. I did a quick test on a server that was brought up to RELENG_5 via cvsup on 2/2/05. The Test: -bash-2.05b$ time dd bs=1024 if=/dev/zero of=tstfile count=1M 1048576+0 records in 1048576+0 records out 1073741824 bytes transferred in 74.402757 secs (14431479 bytes/sec) real 1m14.498s user 0m0.550s sys 0m8.838s The vmstat -1 info is attached. Thanks! --Nick On Thu, 3 Feb 2005 17:59:30 -0500 (EST), Jeff Roberson <jroberson@chesapeake.net> wrote: > > On Thu, 3 Feb 2005, Nick Pavlica wrote: > > > All, > > I would like to share the methods that I have been using in my disk > > I/O testing. The detailed results of these tests have been posted to > > the performance and questions mailing lists under the title " FreeBSD > > 5.3 I/O Performance / Linux 2.6.10 | Continued Discussion". I > > originally started this testing as due diligence in an up coming > > project. As a result of this testing I discovered an elegant > > operating system that I enjoy working with. > > Nick, first, I'd like to thank you for your efforts so far. I think your > tests have been very informative. I'd like to see what we can do to get > to the bottom of the differences. Can you perform one test which varied > greatly between 5.x and 4.x and collect some data for us? To start with, > the output of vmstat 1 piped to a file would be informative. Do you have > any indication that 5.x is actually cpu bound in a case where 4.x is not? > I'm wondering if this is a latency issue or a cpu utilization issue. > > I intend to backport some code that lets me graph system activity into > RELENG_5. Are you setup to cvsup to this tag? Would it be convenient for > you to do so? > > Thanks, > Jeff > > > > > Intent Of This Testing: > > 1)To measure the disk I/O performance of various operating systems for > > use as a production database server. > > 2)Help improve the disk I/O performance of FreeBSD 5.x and greater by > > assisting the FreeBSD development team in identifying possible > > performance issues, and provide them with data to measure the success > > of various changes to the operating system. > > > > Operating Systems tested: > > Fedora Core 3 with EXT3, and XFS. I tested with and with out patches. > > SUSE Enterprise Server 9 with Riser FS. > > FreeBSD 4.11R > > FreeBSD 5.3R, RELENG_5_3, RELENG_5 > > NetBSD 2.0R > > OpenBSD 3.6R > > > > Test Hardware: > > Compaq DeskPro, PIII 800, 384Mb Ram, 10Gb IDE HD. > > Dell PE 2400, Dual PIII 550, 512Mb Ram, (2)10K,LVD SCSI, RAID 1, PERC > > 2SI controller with 64Mb ram. > > Dell PE SC400, 2.4Ghz P4, 256MB Ram, 40Gb IDE HD. > > Dell 4600, 2.8 Ghz P4 with HT, 512MB Ram, 80GB IDE HD. > > > > Installation Notes: > > It's my intention to test these Operating Systems using as many of > > the default installation options as possible with no special tuning. > > The only deviations in my previous testing were as follows: The #linux > > xfs option was used when installing Fedora so that I could use XFS, > > and a special test where I installed 5.3R with UFS instead of UFS2 (I > > didn't see any improvement when using UFS). I installed FreeBSD using > > the standard install option, and used the auto allocate features for > > partitioning and slicing. I installed Fedora with the stock server > > packages and created a 100Mb /boot, 512Mb swap, and allocated the > > remaining space to /. I tested FreeBSD5.3R and FC3R with and without > > updates. I used cvsup to update FreeBSD and yum update to update > > Fedora. I didn't do any updating to FreeBSD4.11R, NetBSD2.0, and > > OpenBSD3.6. > > > > I used the following utilities/tools in my testing: > > DD > > CP > > IOSTAT (iostat -d 2) > > Bonnie++ > > TOP > > SQL,PL, PSQL > > Postgresql 8.0 > > > > DD Example Tests: > > - #time dd bs=1024 if=/dev/zero of=tstfile count=1M > > - #time dd bs=1024 if=/dev/zero of=tstfile count=2M > > - #time dd bs=1024 if=/dev/zero of=tstfile count=3M > > > > Bonnie++ Example Tests: > > #bonnie++ -u root -s 1024 -r 512 -n 5 > > #bonnie++ -u root -s 2048 -r 512 -n 5 > > #bonnie++ -u root -s 3072 -r 512 -n 5 > > > > CP Example Tests: > > #time cp tstfile tstfile2 > > > > SQL, PL, PSQL Example Tests: > > > > CREATE TABLE test1 ( > > thedate TIMESTAMP, > > astring VARCHAR(200), > > anumber INTEGER > > ); > > > > CREATE FUNCTION build_data() RETURNS integer AS ' > > DECLARE > > i INTEGER DEFAULT 0; > > curtime TIMESTAMP; > > BEGIN > > FOR i IN 1..1000000 LOOP > > curtime := ''now''; > > INSERT INTO test1 VALUES (curtime, ''test string'', i); > > END LOOP; > > RETURN 1; > > END; > > ' LANGUAGE 'plpgsql'; > > > > SELECT build_data(); > > Then the following script is run under the time program to ascertain > > how long it takes to run: > > CREATE TABLE test2 ( > > thedate TIMESTAMP, > > astring VARCHAR(200), > > anumber INTEGER > > ); > > CREATE TABLE test3 AS SELECT * FROM test1; > > INSERT INTO test2 SELECT * FROM test1 WHERE ((anumber % 2) = 0); > > DELETE FROM test3 WHERE ((anumber % 2) = 0); > > DELETE FROM test3 WHERE ((anumber % 13) = 0); > > CREATE TABLE test4 AS > > SELECT test1.thedate AS t1date, > > test2.thedate AS t2date, > > test1.astring AS t1string, > > test2.astring AS t2string, > > test1.anumber AS t1number, > > test2.anumber AS t2number > > FROM test1 JOIN test2 ON test1.anumber=test2.anumber; > > UPDATE test3 SET thedate='now' WHERE ((anumber % 5) = 0); > > DROP TABLE test4; > > CREATE TABLE test4 AS SELECT * FROM test1; > > DELETE FROM test4 WHERE ((anumber % 27) = 0); > > VACUUM ANALYZE; > > VACUUM FULL; > > DROP TABLE test4; > > DROP TABLE test3; > > DROP TABLE test2; > > VACUUM FULL; > > > > Example FS TAB: > > > > minime# cat /etc/fstab > > # Device Mountpoint FStype Options Dump Pass# > > /dev/ad0s1b none swap sw 0 0 > > /dev/ad0s1a / ufs rw 1 1 > > /dev/ad0s1e /tmp ufs rw 2 2 > > /dev/ad0s1f /usr ufs rw 2 2 > > /dev/ad0s1d /var ufs rw 2 2 > > /dev/acd0 /cdrom cd9660 ro,noauto 0 0 > > > > Verification Of Test: > > I have been able to get consistent results in all of my testing. > > However, I think the best verification would be to have as many people > > as possible test the disk I/O performance on a range of hardware, > > testing methods, and configurations. > > > > Summary Of Results: > > The results of my testing have consistently demonstrated that > > FreeBSD5.3+ has dramatically slower disk I/O performance than all of > > the other operating systems that were tested. FreeBSD 4.11R was the > > performance leader followed by Fedora C3 with XFS. All of the BSD > > distributions, with the exception of 5.3+, were able to consistently > > demonstrate a throughput of 56-58Mb/s sustained throughput, while 5.3+ > > consistently demonstrated a throughput of 12-15Mb/s (58 -15 = 43 ?). > > > > Please let me know if you need any additional details. > > > > Thanks! > > --Nick Pavlica > > _______________________________________________ > > freebsd-performance@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > > > ------=_Part_2051_5270362.1107474500093 Content-Type: application/octet-stream; name="vmstat_1_info" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="vmstat_1_info" IHByb2NzICAgICAgbWVtb3J5ICAgICAgcGFnZSAgICAgICAgICAgICAgICAgICBkaXNrICAgZmF1 bHRzICAgICAgY3B1CiByIGIgdyAgICAgYXZtICAgIGZyZSAgZmx0ICByZSAgcGkgIHBvICBmciAg c3IgYWQwICAgaW4gICBzeSAgY3MgdXMgc3kgaWQKIDAgMCAwICAgNjY0ODAgIDg1NDQwICAgMTMg ICAwICAgMCAgIDAgIDEyICAgOCAgIDAgIDMzMiAgMTA1IDI1OCAgMCAgMCAxMDAKIDAgMCAwICAg NjY0ODAgIDg1NDM2ICAgIDMgICAwICAgMCAgIDAgICAzICAgMCAgIDAgIDMzNiAgMTE5IDI1NyAg MCAgMSA5OQogMCAwIDAgICA2NjQ4MCAgODU0MzYgICAgMCAgIDAgICAwICAgMCAgIDAgICAwICAg MCAgMzMzICAxMTkgMjU1ICAwICAwIDEwMAogMCAxIDAgICA2NzU3NiAgODE1NDggICA4MSAgIDAg ICAwICAgMCAgNDcgICAwICAyMCAgMzc2IDc2NDkgNDM4ICAxICAyIDk3CiAwIDEgMCAgIDY3NTc2 ICA2NTUyNCAgICAwICAgMCAgIDAgICAwICAgMCAgIDAgMTI1ICA1ODYgMzIxMzAgMTIyOCAgMCAx NCA4NgogMCAxIDAgICA2NzU3NiAgNDk0NjQgICAgMCAgIDAgICAwICAgMCAgIDIgICAwIDEyNCAg NTgwIDMxOTgwIDEyMTYgIDIgMTIgODYKIDAgMSAwICAgNjc1NzYgIDMzNDA0ICAgIDAgICAwICAg MCAgIDAgICAwICAgMCAxMjQgIDU4MyAzMjExOSAxMjMxICAyIDEzIDg0CiAxIDAgMCAgIDY3NTc2 ICAxNzg5MiAgICAwICAgMCAgIDAgICAwICAgMCAgIDAgMTIzICA1NzkgMzA4OTIgMTIxNCAgMSAx NCA4NQogMCAxIDAgICA2NzU3NiAgIDkyNzYgICAgMCAgIDAgICAwICAgMCAgMjcgMjAwNiAxMjQg IDU4NiAzMjMxNyAxMjQzICAyIDE0IDg1CiAwIDEgMCAgIDY3NTc2ICAgODg4OCAgICAwICAgMCAg IDAgICAwIDM1NTggMzc2MiAxMjQgIDU4MyAzMDgzNSAxMjI2ICAwIDE2IDg0CiAwIDEgMCAgIDY3 NTc2ICAxNDcyMCAgICAwICAgMCAgIDAgICAwIDQxODYgNTY0NCAxMjMgIDU4MCAzMzM5NyAxMjM5 ICAyIDE2IDgyCiAwIDEgMCAgIDY3NTc2ICAxMzcwNCAgICAwICAgMCAgIDAgICAwIDQwMTcgMzc2 MyAxMjQgIDU4NCAzMjEwNiAxMjM1ICAwIDE2IDg0CiAwIDEgMCAgIDY3NTc2ICAxMzE2OCAgICAw ICAgMCAgIDAgICAwIDM5MDEgMzc2NyAxMjIgIDU3OSAzMTEwMyAxMjE4ICAxIDE2IDg0CiAwIDEg MCAgIDY3NTc2ICAxMjE1NiAgICAwICAgMCAgIDAgICAwIDQwMTggMzc2NSAxMjQgIDU4MiAzMjEw OCAxMjM0ICAwIDE2IDg0CiAwIDEgMCAgIDY3NTc2ICAxMTYxMiAgICAwICAgMCAgIDAgICAwIDM5 MDEgMzk2NCAxMjIgIDU3OCAzMTEwMiAxMjE0ICAyIDE2IDgzCiAwIDEgMCAgIDY3NTc2ICAxMDYw MCAgICAwICAgMCAgIDAgICAwIDQwMTcgMzc5NCAxMjQgIDU4MyAzMjExOCAxMjM4ICAxIDE1IDg0 CiAxIDAgMCAgIDY3NTc2ICAgOTkyOCAgICAwICAgMCAgIDAgICAwIDM5MzMgMzc2NSAxMjEgIDU3 NSAzMTMyMSAxMjA0ICAyIDE0IDg0CiAwIDEgMCAgIDY3NTc2ICAgOTI4NCAgICAwICAgMCAgIDAg ICAwIDM5MjQgMzc2MyAxMjQgIDU4MCAzMTM3NyAxMjI2ICAwIDE5IDgxCiBwcm9jcyAgICAgIG1l bW9yeSAgICAgIHBhZ2UgICAgICAgICAgICAgICAgICAgZGlzayAgIGZhdWx0cyAgICAgIGNwdQog ciBiIHcgICAgIGF2bSAgICBmcmUgIGZsdCAgcmUgIHBpICBwbyAgZnIgIHNyIGFkMCAgIGluICAg c3kgIGNzIHVzIHN5IGlkCiAwIDQgMCAgIDY3OTYwICAxNTk1MiAgIDU0ICAgMCAgIDAgICAyIDM5 OTcgNTY1MiAxMjEgIDU3NyAzMTM5NiAxMjIzICAyIDEzIDg1CiAwIDQgMCAgIDY3OTYwICAxNDk0 NCAgICAwICAgMCAgIDAgICAwIDQwMTcgMzc2OSAxMjQgIDU4MCAzMjExMyAxMjI4ICAyIDE1IDg0 CiAwIDMgMCAgIDY3OTYwICAxNTAwOCAgICAzICAgMCAgIDAgICAwIDM3NTIgMzc2OCAxMjQgIDU3 NyAyOTgyMCAxMjA3ICAxIDE0IDg1CiAwIDIgMCAgIDY3OTY0ICAxNTg3MiAgICA1ICAgMCAgIDAg ICAwIDM4MzcgMzUwNCAxMDggIDg2MyAyODI3NCAxNTE3ICAwIDEwIDkwCiAwIDIgMCAgIDY3OTY0 ICAxMTY4MCAgICA1ICAgMCAgIDAgICAwIDI5MzIgMTg4NCAgOTQgIDUyMSAyMzQyOCA5ODggIDAg MTIgODgKIDAgMiAwICAgNjc5NjQgIDExMzQ0ICAgIDAgICAwICAgMCAgIDAgMzg1MCAzNzY2IDEx OCAgNTY5IDMwODMzIDExODYgIDAgMTUgODUKIDAgMiAwICAgNjc5NjQgIDExMzQ0ICAgIDAgICAw ICAgMCAgIDAgMzc2MyAzNzYzIDExOSAgNTcyIDMwMDc1IDExODcgIDEgMTUgODQKIDEgMiAwICAg Njc5NjggIDE2MzM2ICAgIDUgICAwICAgMCAgMTcgMzU5MiAxNTgzOSAxMjkgIDU3OSAyODI0NCAx MjIwICAwIDE4IDgyCiAwIDIgMCAgIDY3OTY4ICAxNTU2OCAgIDMyICAgMCAgIDAgICAwIDM5MzUg NjY1MCAxMjIgIDU3MyAzMTY2NiAxMjA2ICAxIDE1IDg0CiAwIDMgMCAgIDY4MjY4ICAxNTg5NiAg IDk1ICAgMSAgIDAgICAwIDM3MDggMzc2NiAxMTYgIDU2NCAyOTExNCAxMTYzICAwIDE1IDg1CiAw IDMgMCAgIDY4MjY4ICAxNDg5NiAgICAwICAgMCAgIDAgICAwIDQwMTUgMzc2NSAxMjMgIDU4MiAz MjEyNiAxMjMxICAxIDE2IDgzCiAwIDQgMCAgIDY4MzU2ICAxNTA5NiAgICAyICAgMCAgIDIgICAw IDM3MTQgMzc3NCAxMTkgIDU2OCAyOTU1MCAxMTc4ICAyIDE2IDgzCiAwIDQgMCAgIDY4MzU2ICAx NDg1NiAgICAwICAgMCAgIDAgICAwIDM4MjcgMzc2NyAxMjEgIDU3OCAzMDU4MyAxMjA4ICAyIDEy IDg2CiAwIDEyIDAgICA2ODM2MCAgMTAyNTYgICAgMSAgIDAgICAwICAgMCAzNjIxIDM3NjggMTIx ICA1NjcgMjg3OTcgMTE1NSAgMSAxNSA4NQogMCAxMiAwICAgNjgzNjAgIDExMTgwICAgIDAgICAw ICAgMSAgIDAgMTA0ODQgOTQxNiAxMTUgMTU4NyA4MzU4MyAzMjg0ICAxIDEzIDg2CiAwIDEyIDAg ICA2ODM2MCAgMTA4MDQgICAgMCAgIDAgICAwICAgMCAzODU3IDM3NjQgMTE3ICA1NzAgMzA4Mzkg MTE4MiAgMiAxNSA4MwogMCAxMiAwICAgNjgzOTIgIDExNjU2ICAgIDEgICAwICAgMSAgIDAgMzU0 OSAzODQ5IDExMyAgNTU3IDI4MjY4IDExMzMgIDEgMTMgODYKIDAgNCAwICAgNjkyODAgIDEyMjA0 ICAgIDIgICAwICAgMCAgIDAgMzIyMCAzNzY5IDEwOSAgNTQzIDI1NzMxIDEwNjggIDAgMTIgODgK IHByb2NzICAgICAgbWVtb3J5ICAgICAgcGFnZSAgICAgICAgICAgICAgICAgICBkaXNrICAgZmF1 bHRzICAgICAgY3B1CiByIGIgdyAgICAgYXZtICAgIGZyZSAgZmx0ICByZSAgcGkgIHBvICBmciAg c3IgYWQwICAgaW4gICBzeSAgY3MgdXMgc3kgaWQKIDAgNCAwICAgNjkyODAgIDEzMTEyICAgIDAg ICAwICAgMSAgIDAgNTg2MCA1NjQ0ICA5OSAgOTY4IDQ2NzE0IDE4NzUgIDAgMTQgODYKIDAgNCAw ICAgNjkyODAgIDEzMzU2ICAgIDAgICAwICAgMCAgIDQgMzY5OCAzNzY1IDExNyAgNTY5IDI5NTU5 IDExNzYgIDAgMTYgODQKIDAgNCAwICAgNjkyODggIDEzNTMyICAgMjAgICAwICAgMSAgIDAgMzcy NSAzNzY2IDEyMCAgNTY5IDI5NTU3IDExNzUgIDAgMTQgODYKIDAgNCAwICAgNjkyODggIDEyNTEy ICAgIDAgICAwICAgMCAgIDAgNDAxNyAzNzY1IDEyMCAgNTc3IDMyMTE5IDEyMTggIDIgMTQgODUK IDAgNCAwICAgNjkyNjQgIDEzMjA4ICAgMjcgICAwICAgMiAgIDAgMzYwNSAzNzYzIDExNiAgNTYy IDI4NTMzIDExNTEgIDEgMTMgODYKIDAgNCAwICAgNjkyNjQgIDEyODI4ICAgMTQgICAwICAgMCAg IDEgMzg2NiAzNzY2IDExOSAgNTcyIDMwODYwIDEyMDAgIDAgMTYgODQKIDEgMyAwICAgNjkyNzYg IDEzNDE2ICAgMjAgICAwICAgMSAgIDAgMzYyNCAzNzY3IDExNSAgNTYwIDI4NzQ5IDExNDQgIDAg MTcgODMKIDAgNCAwICAgNjkyNzYgIDEzMDI4ICAgIDAgICAwICAgMCAgIDAgMzg2MiAzNzY1IDEx OCAgNTczIDMwODgxIDExOTUgIDEgMTggODIKIDAgNCAwICAgNjkyNzYgICA3OTQ0ICAgIDMgICAw ICAgMSAgIDQgMzg3MiAyNjA1IDEyMiAgNTczIDMwODQzIDExOTcgIDEgMTcgODIKIDAgMyAwICAg Njk0ODggIDE2MTAwICAgNjAgIDEwICAgMyAgNDkgMzYzMCAxMDk3NSAxNjkgIDYxNSAyODU2MCAx NDIyICAxIDE0IDg1CiAwIDMgMCAgIDY5NDg4ICAxNTU2OCAgIDUwICAgMCAgIDAgICAwIDM5MTEg Mzc2NSAxMTkgIDU2OSAzMDU3OCAxMTg2ICAxIDE4IDgyCiAxIDMgMCAgIDcwMzYwICAxMzYxNiAg IDI3ICAgMCAgIDEgICAwIDI0MzIgMTQ4MyAgOTMgIDUwNiAxOTM5OSA5MDggIDEgMTAgODkKIDAg MyAwICAgNzAzNzYgIDEwNzEyICAgMTQgICAwICAgMCAgIDAgODI2NCA3NTM4ICA4MSAxNTY4IDY1 OTA4IDI4MDggIDAgIDkgOTEKIDEgMiAwICAgNzAzNzYgIDEyMzIwICAgMTAgICAwICAgMCAgIDAg OTAxNiA5NDE2ICA4NyAxNjI1IDcxNzQ4IDI5ODkgIDAgMTAgODkKIDAgMyAwICAgNzAzNzYgIDEz NjA4ICAgIDAgICAwICAgMCAgIDAgMzQ0MyAzNzY2IDEwOCAgNTUwIDI3NTcxIDExMDMgIDAgMTQg ODYKIDAgMSAxICAgNjc1NzYgIDE0ODcyICAgIDMgICAwICAgMSAgIDAgMzU4MSAzNzY0IDExMCAg NTU1IDI4NTQyIDExMjggIDEgMTQgODUKIDEgMCAxICAgNjc1NzYgIDE1MDYwICAgIDAgICAwICAg MCAgIDAgMzcxNiAzNzYzIDExOCAgNTczIDI5NzE3IDExODMgIDAgMTMgODcKIDAgMSAxICAgNjc1 NzYgIDE1MjMyICAgIDAgICAwICAgMSAgIDAgMzcyMCAzNzcwIDExNSAgNTYxIDI5NjQ0IDExNTUg IDAgMTQgODYKIHByb2NzICAgICAgbWVtb3J5ICAgICAgcGFnZSAgICAgICAgICAgICAgICAgICBk aXNrICAgZmF1bHRzICAgICAgY3B1CiByIGIgdyAgICAgYXZtICAgIGZyZSAgZmx0ICByZSAgcGkg IHBvICBmciAgc3IgYWQwICAgaW4gICBzeSAgY3MgdXMgc3kgaWQKIDAgMSAxICAgNjc1NzYgIDE0 ODU2ICAgIDAgICAwICAgMCAgIDAgMzg1NyAzNzYzIDExNyAgNTcyIDMwODM5IDExODkgIDIgMTMg ODUKIDAgMSAwICAgNjc1NzYgIDE0OTM2ICAgMTMgICAwICAgMSAgIDAgMzc0OSAzNzY3IDExOCAg NTY2IDI5ODA3IDExNzUgIDAgMTQgODYKIDAgMSAxICAgNjc1NzYgIDE0NTY0ICAgIDAgICAwICAg MSAgIDAgMzg1OSAzNzY2IDEyMiAgNTc5IDMwODU4IDEyMjEgIDAgMTYgODQKIDAgMSAxICAgNjc1 NzYgIDE0MDIwICAgIDAgICAwICAgMSAgIDAgMzkwMCAzNzY4IDEyMiAgNTczIDMxMDgyIDEyMDMg IDEgMTYgODMKIDEgMCAxICAgNjc1NzYgIDE0MTIwICAgIDAgICAwICAgMCAgIDAgMzczOSAzNzY0 IDExOCAgNTcxIDI5ODk1IDExNzggIDMgMTIgODUKIDAgMSAxICAgNjc1NzYgIDEzODAwICAgIDAg ICAwICAgMCAgIDAgMzg0NSAzNzY1IDExNyAgNTY3IDMwNzQ2IDExNzkgIDAgMTUgODUKIDAgMSAw ICAgNjc1NzYgIDE0MDIwICAgMTIgICAwICAgMSAgIDAgMzcxNyAzNzY3IDExOCAgNTY5IDI5NTYz IDExNzggIDIgMTIgODUKIDEgMCAwICAgNjc1NzYgIDE0MDU2ICAgIDAgICAwICAgMCAgIDAgMzc1 NSAzNzY0IDExNyAgNTY3IDMwMDIyIDExNzEgIDEgMTYgODMKIDAgMSAwICAgNjc1NzYgIDEzNzUy ICAgIDAgICAwICAgMCAgIDAgMzg0MSAzNzY1IDExNyAgNTcwIDMwNjM0IDExODggIDEgMTggODEK IDAgMSAwICAgNjc1NzYgIDE0MDI0ICAgIDAgICAwICAgMCAgIDAgMzY5NSAzNzYzIDExNyAgNTY5 IDI5NTQ2IDExNzAgIDEgMTQgODUKIDAgMSAwICAgNjc1NzYgIDE0NzQ0ICAgIDAgICAwICAgMCAg IDQgMzU3OSAzNzY1IDExMiAgNTU0IDI4NTM1IDExMjMgIDIgMTYgODIKIDAgMSAwICAgNjc1NzYg ICA5NDE2ICAgIDAgICAwICAgMCAgIDAgMzIxMyAxODgxIDEwMCAgNTM0IDI1NzA2IDEwMzcgIDAg MTIgODgKIDAgMSAwICAgNjc1NzYgIDExNjEyICAgIDAgICAwICAgMCAgIDAgMzIxNCAzNzYzIDEw MSAgNTM3IDI1NzM4IDEwNTIgIDIgMTEgODgKIDAgMSAxICAgNjc1NzYgIDEzNjEyICAgMTQgICAw ICAgNCAgIDAgMzI3MiAzNzY2IDEwMSAgNTMyIDI1OTY2IDEwNDkgIDEgMTIgODcKIDAgMSAxICAg Njc1NzYgICA5MTY0ICAgIDAgICAwICAgMCAgIDAgMzA4NSAxODgxIDEwMCAgNTM2IDI0Mjg5IDEw MzEgIDEgMTEgODgKIDAgMCAwICAgNjY0ODAgICA4OTUyICAxNDAgICAwICAgOCAgIDAgMTk5OCAx ODgyICA4NCAgNDgxIDE1ODEyIDgxOSAgMSAgNyA5MgogMCAwIDAgICA2NjQ4MCAgIDg5NTIgICAg NyAgIDAgICAwICAgMCAgIDIgICAwICAgMCAgMzMzICAxMjYgMjU2ICAwICAxIDk5CiAwIDAgMCAg IDY2NDgwICAgODk1MiAgICAyICAgMCAgIDAgICAwICAgMCAgIDAgICAwICAzMzEgIDEyMiAyNTEg IDAgIDEgOTkKIHByb2NzICAgICAgbWVtb3J5ICAgICAgcGFnZSAgICAgICAgICAgICAgICAgICBk aXNrICAgZmF1bHRzICAgICAgY3B1CiByIGIgdyAgICAgYXZtICAgIGZyZSAgZmx0ICByZSAgcGkg IHBvICBmciAgc3IgYWQwICAgaW4gICBzeSAgY3MgdXMgc3kgaWQKIDAgMCAwICAgNjY0ODAgIDE2 NTI4ICAgIDAgICAwICAgMCAgIDAgIDc3IDE4OTYgIDE1ICAzNDkgIDExOSAzMDEgIDAgIDIgOTgK ------=_Part_2051_5270362.1107474500093--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?dc9ba0440502031548544723a1>