Date: Sat, 1 Dec 2007 17:33:34 +0100 From: Gergely CZUCZY <phoemix@harmless.hu> To: Jeff Roberson <jroberson@chesapeake.net> Cc: freebsd-performance@freebsd.org Subject: Re: mysql scaling questions Message-ID: <20071201163334.GA21709@harmless.hu> In-Reply-To: <20071130143023.I884@192.168.1.107> References: <20071129101729.GA57985@harmless.hu> <20071130143023.I884@192.168.1.107>
next in thread | previous in thread | raw e-mail | index | archive | help
--J/dobhs11T7y2rNN Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Nov 30, 2007 at 02:31:50PM -1000, Jeff Roberson wrote: > I've forwarded this mail to the freebsd performance list so more people c= an take a look at it. Thanks for all of the details. What=20 > was the test that you're doing? sysbench? With writes or without? Or s= ome other benchmark? Answering the questions: I am using sysbench, with read-write tests. The test-table has 10M tuples, and i'm querying it for two minutes. On Sat, Dec 01, 2007 at 04:41:58PM +0100, Miroslav Lachman wrote: > > As Mike Tancsa suggested - rebuild MySQL dynamic and use these values in = /etc/libmap.conf > > # /etc/libmap.conf > # > # candidate mapping > # > [/usr/local/libexec/mysqld] > libpthread.so.2 libthr.so.2 > libpthread.so libthr.so > > Miroslav Lachman no need for this, since on 7 libthr is the default threading implementation. Now, some test results. The numbers show Queries/Second and in order they correspond to the following number of threads: 1 2 4 8 16 32 64 128 192 256 384 512 fbsd7_ufs_TSC_rw_plot: 1891.74 1935.85 1630.94 1966.75 1816.05 1700.58 1590.84 1536.18 1533.99 152= 6.05 1520.00 1464.44 fbsd7_ufs_rw_plot: 1740.98 1605.12 1520.87 1548.42 1566.22 1465.80 1464.17 1420.33 1384.02 143= 3.44 1489.23 1445.75 (ZM stands for zmirror with 2 drives) fbsd7_zfs_zm_rw_plot: 333.62 297.55 265.34 312.15 315.67 241.05 200.08 177.46 162.11 147.02 146.5= 2 125.72 (cdb2 means a setup we have. It's mysql 5.0.22) linux26_cdb2_rw_plot: 2013.59 2591.36 3515.02 3629.28 3129.89 3139.59 2747.31 linux26_mysql5041_rw_plot: 3415.20 4193.24 4670.92 4619.93 4288.13 3942.70 3591.34 3192.13 2966.72 285= 5.82 2580.51 2405.44 (This one is still in progress) linux26_mysql5041_tcmalloc_rw_plot: 2916.28 3551.13 3728.86 3856.79 3866.78 3707.94 3440.99 3251.88 3180.07 307= 0.65 2937.17 The testbox has a dual opteron 246, 12G of memory and a 3ware-9550 with two disks in a mirror for the rootfs (also this is where the mysql datafiles were, except for the ZFS setup). The freebsd has mysql-5.0.45_1. As you see for the best case linux outperforms fbsd7 around twice, this means that i couldn't reproduce jeff@'s results. >=20 > Thanks, > Jeff >=20 > On Thu, 29 Nov 2007, Gergely CZUCZY wrote: >=20 > >Hello > > > >I've been testing mysql on 7-BETA2 for a few days, compared to a > >linux -2.6.19.2-grsec, and i've found that the linux setup does > >around two times more queries then the FreeBSD ones. This seems > >to be a bit different according to your test results. > > > >Could you help me to check out what am I missing? > > > >I've installed mysql from ports, 5.0.45, and i'm using a production > >config file from a linux system, slightly modified (i've updated the > >paths, and the bind-address). > > > >For 1 2 4 8 16 32 64 128 192 256 384 512 threads it does the > >following results (queries/sec) on freebsd with UFS: > >1740.98 > >1605.12 > >1520.87 > >1548.42 > >1566.22 > >1465.80 > >1464.17 > >1420.33 > >1384.02 > >1433.44 > >1489.23 > >1445.75 > > > >And for linux (it stops at 128 threads, i'm debugging this atm): > >2791.05 > >3328.08 > >4459.14 > >3668.25 > >3489.43 > >3954.07 > >3055.25 > > > >As you see, it's a bit more the two times more. Something isn't right > >it seems to me. > > > >About the FreeBSD system: > >FreeBSD sqltest.in.publishing.hu 7.0-BETA2 FreeBSD 7.0-BETA2 #3: Mon Nov= 5 10:49:48 CET 2007 =20 > >toor@sqltest.in.publishing.hu:/usr/obj/usr/src/sys/SQLTEST amd64 > > > >I have the following kernel config: > >cpu HAMMER > >ident GENERIC > >options SCHED_ULE > >options PREEMPTION # Enable kernel thread preemption > >options INET # InterNETworking > >options FFS # Berkeley Fast Filesystem > >options SOFTUPDATES # Enable FFS soft updates support > >options UFS_ACL # Support for access control lis= ts > >options UFS_DIRHASH # Improve performance on big dir= ectories > >options UFS_GJOURNAL # Enable gjournal-based UFS jour= naling > >options MD_ROOT # MD is a potential root device > >options NFSCLIENT # Network Filesystem Client > >options NFSSERVER # Network Filesystem Server > >options NFS_ROOT # NFS usable as /, requires NFSC= LIENT > >options NTFS # NT File System > >options MSDOSFS # MSDOS Filesystem > >options CD9660 # ISO 9660 Filesystem > >options PROCFS # Process filesystem (requires P= SEUDOFS) > >options PSEUDOFS # Pseudo-filesystem framework > >options GEOM_PART_GPT # GUID Partition Tables. > >options GEOM_LABEL # Provides labelization > >options COMPAT_43TTY # BSD 4.3 TTY compat [KEEP THIS!] > >options COMPAT_IA32 # Compatible with i386 binaries > >options COMPAT_FREEBSD6 # Compatible with FreeBSD6 > >options SCSI_DELAY=3D5000 # Delay (in ms) before probing= SCSI > >options KTRACE # ktrace(1) support > >options SYSVSHM # SYSV-style shared memory > >options SYSVMSG # SYSV-style message queues > >options SYSVSEM # SYSV-style semaphores > >options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time e= xtensions > >options KBD_INSTALL_CDEV # install a CDEV entry in /dev > >options ADAPTIVE_GIANT # Giant mutex is adaptive. > >options STOP_NMI # Stop CPUS using NMI instead of= IPI > >options AUDIT # Security event auditing > >options SMP # Symmetric MultiProcessor Kernel > >device cpufreq > >device acpi > >device pci > >device fdc > >device ata > >device atadisk # ATA disk drives > >device ataraid # ATA RAID drives > >device atapicd # ATAPI CDROM drives > >device atapifd # ATAPI floppy drives > >device atapist # ATAPI tape drives > >options ATA_STATIC_ID # Static device numbering > >device scbus # SCSI bus (required for SCSI) > >device ch # SCSI media changers > >device da # Direct Access (disks) > >device sa # Sequential Access (tape etc) > >device cd # CD > >device pass # Passthrough device (direct SCSI access) > >device ses # SCSI Environmental Services (and SAF-T= E) > >device twa # 3ware 9000 series PATA/SATA RAID > >device atkbdc # AT keyboard controller > >device atkbd # AT keyboard > >device psm # PS/2 mouse > >device kbdmux # keyboard multiplexer > >device vga # VGA video card driver > >device sc > >device sio # 8250, 16[45]50 based serial ports > >device uart # Generic UART driver > >device de # DEC/Intel DC21x4x (``Tulip'') > >device em # Intel PRO/1000 adapter Gigabit Etherne= t Card > >device le # AMD Am7900 LANCE and Am79C9xx PCnet > >device txp # 3Com 3cR990 (``Typhoon'') > >device vx # 3Com 3c590, 3c595 (``Vortex'') > >device miibus # MII bus support > >device bge # Broadcom BCM570xx Gigabit Ethernet > >device fxp # Intel EtherExpress PRO/100B (82557, 82= 558) > >device msk # Marvell/SysKonnect Yukon II Gigabit Et= hernet > >device loop # Network loopback > >device random # Entropy device > >device ether # Ethernet support > >device ppp # Kernel PPP > >device tun # Packet tunnel. > >device pty # Pseudo-ttys (telnet etc) > >device md # Memory "disks" > >device firmware # firmware assist module > >device bpf # Berkeley packet filter > >device uhci # UHCI PCI->USB interface > >device ehci # EHCI PCI->USB interface (USB 2.0) > >device usb # USB Bus (required) > >device ugen # Generic > >device uhid # "Human Interface Devices" > >device ukbd # Keyboard > >device ulpt # Printer > >device umass # Disks/Mass storage - Requires scbus an= d da > >device ums # Mouse > > > >It's a slightly modified GENERIC. > > > >Loaded modules: > >Id Refs Address Size Name > >1 2 0xffffffff80100000 65cb90 kernel > >2 1 0xffffffff8075d000 f4da0 zfs.ko > > > >The my.cnf: > ># grep '^[\[a-z]' /var/db/mysql/my.cnf > >[client] > >port =3D 3307 > >socket =3D /tmp/mysql.sock > > > >[mysqld_safe] > >socket =3D /tmp/mysql.sock > >nice =3D 0 > > > >[mysqld] > >default-character-set =3D latin2 > >default-collation =3D latin2_hungarian_ci > >user =3D mysql > >pid-file =3D /var/run/mysqld/mysqld.pid > >socket =3D /var/run/mysqld/mysqld.sock > >port =3D 3306 > >basedir =3D /usr/local > >datadir =3D /var/db/mysql/ > >tmpdir =3D /tmp > >old_passwords =3D 1 > >bind-address =3D 10.0.0.1 > >key_buffer =3D 1024M > >max_allowed_packet =3D 16M > >thread_stack =3D 512K > >query_cache_limit =3D 20M > >query_cache_size =3D 1024M > >query_cache_type =3D 1 > >max_connections =3D 3600 > >thread_cache_size =3D 200 > >wait_timeout =3D 60 > >table_cache =3D 2048 > >sort_buffer =3D 12M > >join_buffer_size =3D 12M > >record_buffer =3D 12M > >read_rnd_buffer_size =3D 8M > >max_heap_table_size =3D 128M > >tmp_table_size =3D 128M > >myisam_sort_buffer_size =3D 64M > >max_connect_errors =3D 64 > >server-id =3D 2 > >skip-bdb > >skip-innodb > > > >[mysqldump] > >quick > >quote-names > >max_allowed_packet =3D 16M > > > >[mysql] > > > >[isamchk] > >key_buffer =3D 16M > > > >The box is a dual opteron 246 with 12GB of memory with 10K RPM > >SATA disks on a 9550 3ware. > > > >So, what can cause this big difference? > > > >the MySQL binary is statically linked, i've built it > >that way. > > > >ldd: /usr/local/libexec/mysqld: not a dynamic executable > > > >Though, maybe I should rebuild it dynamically to ensure it's > >linked against libthr (and not pthread or c_r)... > > > >So, any tips, guesses, anything what can cause this? > > > >Thanks in advance. > > > >Sincerely, > > > >Gergely Czuczy > >mailto: gergely.czuczy@harmless.hu > > > >--=20 > >Weenies test. Geniuses solve problems that arise. > > >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 Sincerely, Gergely Czuczy mailto: gergely.czuczy@harmless.hu --=20 Weenies test. Geniuses solve problems that arise. --J/dobhs11T7y2rNN Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) owGVWb1vI0l2X+/BCYHD4UIbMFzeCUbCis1mNz9lcGcpktLwRpS4IjXr2bk5XrG7 SPaqv6Y/JHGiiwwb2MC4wIHPNi65xICBCy504MCBo8scOLg/wID/AduJf6+qSZFU U7vH+WCz6r1X76ve+1X133z/ex99/MPf/vo3bz/95ue/+IN/Lv7n9FMvTRJ/XvR4 dOv4xbKul4tGua43i81i3bYsvVw3jUpFr9pW5eyLb37XCfxE+ElxvAzFMUvEfVIK Xe74f86sBY9ikbTSZFZsFFZ0XScOg9hJnMA/Zo7vOr5Yz40j7sczERV7vhXYjj8/ Zu/TIBF2MYwcP+FTVxQKlz47jZwjdhHcMlM/Yoau1xlPmG4cm+Xjqj4cMNIaMz8S sxm7CqYiigOf3UWQdFz4jPWf3wo2C6I7HtnCZsnCiZnHHZclAX5gKhJiGtssFBGo PO5bgrlOnLA4YF4QCUwEoSuY1SpwnyX8RjDO3CC4IS2cRGNsvOD+TUxrMO66LJhJ ubZIsEqM+S8XPGkZOnS547GcSwTkJxhmyyB9jjXsAOa/YCxexlPhWws8fukkC1jh gJZB8h1+BmmCicuIxa1C4AkWQFbEJAPid/Oi0PbjOwHfzeUq71MsA8fHx4VCn3GP pTFNrdY4kjJZJLhdlOtItWKtMM40LMoQsAWULusDlqRwQ3zEuG8z57lH4qMlCXQS aXtyB4c5fgpWTcZtxJMj1hUW08sbcascVxC3BuL2qV6muA2cKIhdfsvOOdnxEDo4 rB1jGh4fIywxZ3E6n0MxhLEIvaep49pssBx9cc7spc89x5LKpbEg+/H/LXfhA+Qd axVKIrFKrjP1eKhZgT+T8p+xx8PsmZywIMqxOdyy/oAmhMEZydtSGkclN7C4SwLE vbBK3jJ+79rvMIuRMFmQc7U40AwlAIMYkwO7JKs11iRSv13fFPyA+QL2S4cjlY8Y YoqMRcbXM17mxFkCznjqUp7REqS34yGCHrYep7RAkC6COwigVJIZGYkYDEhZygA/ 9WgrsXgR3LEvEGtHxKWRgI9slQJ+IcCOIjXEkllBBO6QJlf7KnDd4I7yQ0lS+4JU QT6WmcEqrMHKNWYarFZhZQM/mgYzqhhpVFi1bBQKM+zL+iSdxZPxqDOJ7iahGyRg bjTLWh08TbOqNaoQYupak37Xalodvxvlmqbju67rWhVyq01dg8xy1axpZfptmlqz iW+jVVCUVUPXdJ2VK7WKVqlsrvywar2CVUhlvaqVDcXTqOO70tAq9BurGwbJgFaZ rDLmK6AzTVaGWZpO82arYGIVPDWamoGZSqUKvQuFg68GLE7gXFVMPnhOFGVbH/6y I+dWxIeZbh+g2wfvQT0TNtXgv2Zdq1aZAR3MCjPLhlau4quq1erMqJTJWOxETYcd 9bpWqcEckJShQ10pV9OqrQIeDKiEEBxY9tRgnkClRtlDfU9Ddkc14VZorJ88Ry2l lGdVTYfxhwUU+PTeqE2I7UE5Qy+bWrWJ6CJyJiJchUZYzqwZ8ECD9GxqjSa+zaak q0Mds1xYi5OLVPVKecPgCmQYOqsgDTSjwiq1OuKDtKqVm1rTZBWj0dDKJjObFUOr 61iT1iafNA0aN2S2wGMNGIyIUe41kC9ITbQ7mQUHY2oWgS9oS6GUorijlIRRMEeu x4c52iWWhxYQWBumN5GMZGK1Wpbq1I2G1qAkr2J5mNwgNTBf1+uUxGaF0gzjBugb 5BoohdJp6vVWQddqiF/TrCOxCus6PQ3uZZHmzE45+k+YiAj1wKjUjhDHM9p4nkAr W8p9y5mJViiKzWpVl6lVoMJtO/GNLJScZVmnKoxgURAks5gdcDcOVPfE3zs0Hllk s/CjTvKZg/ZQQP8RR0zcWyJM1jK+Oh2p3DlUtWXVc0lrKaBI+VOpTsooSqj56Iqg F2v+KVUni8fUmeFyhkaYtWvsFNoOjEdBSrXnzrHEUUE1eZm0sss6qE+pa/vPqcYh fnaKkvk18MLnSOBV1SsUPlNdWvXzIzwRpFiNoqGNFykaWVNiEepnR+xMRHPhLlnn q+vOV2/WnUtxfPZSIBfogf5JHDIVwpcxo7qoXEeVu3jSG7cNhSDYTNzBn0uUdivw QoRKllROMpT1RUNDHcPGKc6BuKxVR5YwRzqBTCa3KXK1ae0AwYGItaMQTMdDc5QY 572q78TlS9ZTROhk1KXkl/2ANoAQnhQBbaYEgqZo/LYzA4JDS2HcQg+wJfYIKIDR dk9ZuaFDgZABXgg3ZNR6AmBHYd1QVJFY0B1QpY80jAmuvNjyn+OjProuXKKcN4sC j4VBlMBbKoUe8IlCO5ypeFPHIzHU4505o2RV3HzlpiWAhYd26DrzRYKYesCkMwdL HUjfpiGBAfKuIDkhTxYZGJIZ6vh2kdu2LAxrY08R0O/S6laNkaAUBWq1yEMPzdzI DrJIlRD4Q8qd1VaSTeL6dESQ97OsV8lH1a7ko+pY6lE2LfUo+5Z8VK0re6TupR5l A5OPqoepUZOamHqUfUw9qlaW2d/2FVJRHj6AdXEShDHhQPJCZvWRDJcNPDefK+zq EIl3KG0x6k1qW/RomqidujQL6zS1slzerNUamqEISJOK1MRsVqGqNMDUq9WMQCq1 ri9YmDqYymS5DyQ439oaGhsBH0ElArqxrCCUHySIzKE9QRnsiXXQ21NK5M1NpFJL WrMeeu/S9tAcXwvTKc4btIC2SBnacFYNVqQPI8/MYzZAzFF/WgXGgFv040rzuNJg nd5YIWx8suKTBEH0+b5ljiV2DaZfy+84skrQsQQgPe6Nxgwb0K5V1htPdvsdTHcj Il+4TO0maZkVpmzz87I9GPSupJ9sqg8Pn7PeRe+q36Ep9Co6oKynRp2Xve7k+ryX Nzm86vUGw3H/8mJrIWD1ni9PKplSKq2w7xGccLXvd4X1L+CynM8z1sfpNMLsXRDd KLT/iPkU/Syf+UREN8IFID7lKH6n1BJl8HONvTwdXw+7bfg83x5aJg5mSVZ7UILT kIpdnjBs/Um7c56j0kgxqe5iWShQFLYkClw657YKSbxPXrd/9bI9ernrHw8V9VZs HZiRlFNUVduJWgVhJQGVqH1Sz350eX110T7flpqZPP8ancPnbnGKbm8TPaORVgFj e4Ix6E6uLi8fB/MZZgiroAMEdN/gABsRmEGluQVKyBN1cTrqnPd7F7vCnrELkVA+ bESUdVwHUveIGfWuXveuvouYkYhuRbRHTK5lz2gG7U06DAiqdIT28D510COkBa2C tCFX5Dg/cyFyLJVio73pOhh1L0c57M/UzLcke6cLvK3nrdwfXTI597SA4dVlJ0/5 Z2wYBTKrZw9OPVg7ZNgqjHrX3cvT0WGu1GwyR2osUjsobgidRdwTFL48QWe9y8Fk 2L4aT86G4y1BZ9f9LhvyKJG3YGxMYVN4KFfGefukt7ONpYm3KKMxc/lUuM4Hvq+s dS4Hw/Z4UjHH4zfbIqiVVDST0YQElgl7+6rXG7Lxy/7oz949IazfNo0dfTpSgEMJ KIGHg2MMYSC+b+Nnok5RwqFI7QlRWdur5beHUX/S7Z2337TMblXX9Q05XeHyJTCG z7z4EAB1Ju/uogBazdEtiTVP5KvxVbvT2w0/CbxJIm6Jg/LhU2V39Gb0evRykMNO M8U4WcKueCFRvDqE7ZMyGJ09LQWIJOZzidXTfCdLZXrfpozweLiAc3JFTF4NL0f9 v5gMr/qXV/3xm4nsyNfn/YszSkOaY8OyrpuT8gldHbpFgkpMtAr3qLExycr18kl3 0r8Yjdvn55NOt/f6QbMM0aNKy3FUVJxTEcQSqnSepHa3DQjwujc567c36zT2mcMB M7w0Efey7NscrLcid6eNxpfDycWgn+MpAFTWGV6PsvMDEZGOhCiCGRKpP8zNo/Z1 t58HKCBRWGnkJEsmbuU5KbWdZE8rGw2GOSKkkKUHFBo5FhvgHOBkJQ8d/ZXEPCRM tbUHJgAyHA7e501xK3TyxkMrd3hmW7lSEr5nmG4TNpRvj9vygiG7wtrDFHHH3mG6 aqN0PskUOtYO07CPTLq6HHwb3+wx38wNwnD5bYxx8ogRw2KD7VFujNsT5P6435nA HpVlKHlWhkSyC9IsJ3bXjK1p+iBKMqOUMRpcdTl1yKLhw9xEWLCtTybBE7bD6bWN PxdRrrk232XsYjUrYW0FIw/krVHumvEj1hF0zVDYil26TSRWvtb2roRONzdneRxv 0w0xgjNAkM4XKw9DU6m4NFyB4Hy1xZaslat6/q0TBb68OnclYgMX3YnhfDtqnxbH rUIvV15yx3fkycs31qTeFas7F/TFdmm0Svf8tLuZ2tamlPYYh53lNOCRvQLzrgKR ubxsD2+uP2NvR+XhqGTgLJzGIo8e4j0c7jfo15p5VKpCV9znq3Y733XO67M2I5wT MIv45Y7KZY1zCxLaz47AhlHVj1i59rZSfVfVmTpUkN8RRnltlCcmBVbbFnMmfEG1 9xr47gmtbMHYNmO31ynRedJl3Y5Rvq/cs4Of/nScuk74/HluwghvV4RiB/ot0WtO 1dZEhG4353Rt0aN3gD76b8I6eyLqPlKrjZNR26sjC9l5+wLwhxKZBjrN+3s27Pgi yc3m+3BHkAn0xkzrqtnUpWVLIIvA32Pb7T3LZ682EST6qpKQ14iLuN8jw3OcrXKI w0dfFcMNkLbLNJ1vewBgOAq4DRTMTjqDal2H0Tv+zJUze2S/Co7k6d2HdPm3itQJ O0DyVetHyMFWoVpt5Juz0SczcziOg65bwjnsVeD7VLTepDc4N8DMBx1bhf1aukEQ bstcnTppZsqtmzyuCCkQeJtcPSor1A7Xx+VHySrfP2/xZN57KhphuOtFhWHYcDjM zbrU36EfwgasgQmwablrAG7t8KjjXJIsqfEIl5Tc13q8R61noN6ffCIb3ie5ueFE nizta571CNoRgQYvsFM314/TcLaz3voaKVSm4hSa5NecdGE527zXLzt97OF+8bPr 0QmwKxhnfE8AHzH38pnZAf00ND3XX2k83dGfqE82EUo+31z423xZoc031LG3iT95 mXrcV/d1UsmuZMmPT7rdBmXW7W+CqRvutIBhJJ2RS+xtQxCCSUiT0oCG4ySI6NRW ZFerKwmF6LjfKqDF5ULo1NuFIYNV+5U3surW+vGbiuxmdX0VfY4qRwdPmXmxvKbt 21BkFrO2elexucrI+aB0uOCeXKtMPwym38+yT0OnHoSuUata0yadvm/Wxw95RVDe Iq5XbSKeVWwuT+ofZrF2s34hNpZvDzXLn0nFnrF5JEL2/Cdvf/yWFz+8e85Ktzwq 2dOSfNVTUqRE+daSN2/yykLebG58WmaXmaa6948DuXe25kqJFyqBGk2vdHkrh+xJ zGfi3e/B62+FTZHpO0LfqQjPONBQEXA74hYSqRhDPJG7OAn4xhYNwJy84NmYnyxS f86BW/yJOqMhHaLdleV60iuOLS+utrQnb0apryywsy8NpHvtzedY2b7H9fLiRgIt J9oWR68a3MDi6rQqj4nR4wXXAZcvMbxwk2gdBnnKQrwI/KO12XE2V5aLb7yL21Ww rGv0R9IBp06mKb2/ZE9+FJ9RGRCTx+8nnN6CCFp922lr4pokVe8hJnECsn1yq2Xj FdHSa73lxOLWQkxcx3OSDRpDH+ySxKutuqveJlGyDLeJVvpbClhsnVG3oljT9Q0D Hi+5pZwkveNOMqGrIHr3tcfWmhJK959K5rd63dArDZWdUfJdQiWtNKQjvg4cP2PJ U32TFCdDpNB++Vuk8Efk27uSiaSxzo+F4OFE2floaSVM0SKR95E9ovWWTsy9yYYn FJN07ENqZqGdiCgKojiblS6UrxmKjp2zhqw+8Y0TFqf2dP3s+H6gfm4Us9QL36k8 c1QVeJ8GiSj66BjxU9tjY1s8SHu3/km2WYubd09tyx0R1D6mgbrqY3aKA11AByNU TaNSU1fJZePshAWz7OI1G9NfsauhlDJa3UvF9P6Ks2YVR0R5Pl830FFwxO4WPMF5 1Mc/FF31dlq97CIdhW+JFytyekc6WI6+OFd34UtSLpZXPfDJkrmOfyNset19C91T YDrmJIoPS9zx5Xpd17aPNwpmyXWmOEVbWQk+Zn6QkNVL+B3HUppKZSI9OIduP46Y x5dTwfosxm/XZpGgVW2suuKVeiUBE36cAqo6QBZyfakp43NOd59QfIpaoO47aOkw e8UaRMyaRIeatuUw7kOkE8ZHbJ6iBItYDiX06jnPmS8elOY+YuEgFvYtvVZ8kOrg VyTc5dFq5ExEc/xmnQ+p9WGpMg/+DI7ZXM1olpz5HP3WA/aJtUW64i0WW4YsRV8K 4dMNTCLiRCPs6aR0ARQH7q16deAKADEZHLTeeKWP5P6O/xU2VC/sKP0dNC4oXX9/ Rf/qxff+8KOzL7753W9//Zu3n37z81/88OPv//tH//Cn//rL4X//15cXP/uTX/31 v40//slf/t/f/cdHv/zbl60P//TmZ/97+cc/+KNf/eM3f5/+S/g//w8= =CRF9 -----END PGP SIGNATURE----- --J/dobhs11T7y2rNN--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20071201163334.GA21709>