Date: Thu, 13 Nov 2008 18:49:03 -0800 From: Xin LI <delphij@delphij.net> To: FreeBSD Stable <freebsd-stable@freebsd.org>, Pawel Jakub Dawidek <pjd@FreeBSD.org> Cc: Quake Lee <quakelee@geekcn.org> Subject: ZFS crashes on heavy threaded environment Message-ID: <491CE71F.2020208@delphij.net>
next in thread | raw e-mail | index | archive | help
This is a multi-part message in MIME format. --------------060902050107000202090406 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, Pawel, We can still reproduce the ZFS crash (threading+heavy I/O load) on a fresh 7.1-STABLE build, in a few minutes: /usr/local/bin/iozone -M -e -+u -T -t 128 -S 4096 -L 64 -r 4k -s 30g -i 0 -i 1 -i 2 -i 8 -+p 70 -C I have included a backtrace output from my colleague who has his hands on the test environment. Should there is more information necessary please let us know and we wish to provide help on this. It looks like that the problem has been fixed with the new ZFS version in the last patchset against -CURRENT though. Cheers, - -- Xin LI <delphij@delphij.net> http://www.delphij.net/ FreeBSD - The Power to Serve! -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAkkc5x8ACgkQi+vbBBjt66B4igCfQTz0FM2yFYFwyJY26dVmdXCq ZeIAoJWGeWaBPNH31ZOoAnbbnottGzKQ =tcMs -----END PGP SIGNATURE----- --------------060902050107000202090406 Content-Type: text/plain; name="transcript.txt" Content-Transfer-Encoding: base64 Content-Disposition: inline; filename="transcript.txt" U2NyaXB0IHN0YXJ0ZWQgb24gRnJpIE5vdiAxNCAxMDozNjozMyAyMDA4CkdOVSBnZGIgNi4x LjEgW0ZyZWVCU0RdDQpDb3B5cmlnaHQgMjAwNCBGcmVlIFNvZnR3YXJlIEZvdW5kYXRpb24s IEluYy4NCkdEQiBpcyBmcmVlIHNvZnR3YXJlLCBjb3ZlcmVkIGJ5IHRoZSBHTlUgR2VuZXJh bCBQdWJsaWMgTGljZW5zZSwgYW5kIHlvdSBhcmUNCndlbGNvbWUgdG8gY2hhbmdlIGl0IGFu ZC9vciBkaXN0cmlidXRlIGNvcGllcyBvZiBpdCB1bmRlciBjZXJ0YWluIGNvbmRpdGlvbnMu DQpUeXBlICJzaG93IGNvcHlpbmciIHRvIHNlZSB0aGUgY29uZGl0aW9ucy4NClRoZXJlIGlz IGFic29sdXRlbHkgbm8gd2FycmFudHkgZm9yIEdEQi4gIFR5cGUgInNob3cgd2FycmFudHki IGZvciBkZXRhaWxzLg0KVGhpcyBHREIgd2FzIGNvbmZpZ3VyZWQgYXMgImFtZDY0LW1hcmNl bC1mcmVlYnNkIi4uLg0KDQpVbnJlYWQgcG9ydGlvbiBvZiB0aGUga2VybmVsIG1lc3NhZ2Ug YnVmZmVyOg0KDQoNCkZhdGFsIHRyYXAgMTI6IHBhZ2UgZmF1bHQgd2hpbGUgaW4ga2VybmVs IG1vZGUNCmNwdWlkID0gMDsgYXBpYyBpZCA9IDAwDQpmYXVsdCB2aXJ0dWFsIGFkZHJlc3MJ PSAweDE4DQpmYXVsdCBjb2RlCQk9IHN1cGVydmlzb3IgcmVhZCBkYXRhLCBwYWdlIG5vdCBw cmVzZW50DQppbnN0cnVjdGlvbiBwb2ludGVyCT0gMHg4OjB4ZmZmZmZmZmZiNDgzOWZiNg0K c3RhY2sgcG9pbnRlcgkgICAgICAgID0gMHgxMDoweGZmZmZmZmZmYjRiZWI4ZjANCmZyYW1l IHBvaW50ZXIJICAgICAgICA9IDB4MTA6MHhmZmZmZmZmZmI0YmViOTIwDQpjb2RlIHNlZ21l bnQJCT0gYmFzZSAweDAsIGxpbWl0IDB4ZmZmZmYsIHR5cGUgMHgxYg0KCQkJPSBEUEwgMCwg cHJlcyAxLCBsb25nIDEsIGRlZjMyIDAsIGdyYW4gMQ0KcHJvY2Vzc29yIGVmbGFncwk9IGlu dGVycnVwdCBlbmFibGVkLCByZXN1bWUsIElPUEwgPSAwDQpjdXJyZW50IHByb2Nlc3MJCT0g MzI2ICh0eGdfdGhyZWFkX2VudGVyKQ0KdHJhcCBudW1iZXIJCT0gMTINCnBhbmljOiBwYWdl IGZhdWx0DQpjcHVpZCA9IDANClVwdGltZTogOG0xNXMNClBoeXNpY2FsIG1lbW9yeTogODE3 NSBNQg0KRHVtcGluZyA2MjkgTUI6IDYxNCA1OTggNTgyIDU2NiA1NTAgNTM0IDUxOCA1MDIg NDg2IDQ3MCA0NTQgNDM4IDQyMiA0MDYgMzkwIDM3NCAzNTggMzQyIDMyNiAzMTAgMjk0IDI3 OCAyNjIgMjQ2IDIzMCAyMTQgMTk4IDE4MiAxNjYgMTUwIDEzNCAxMTggMTAyIDg2IDcwIDU0 IDM4IDIyIDYNCg0KUmVhZGluZyBzeW1ib2xzIGZyb20gL2Jvb3Qva2VybmVsL3pmcy5rby4u LlJlYWRpbmcgc3ltYm9scyBmcm9tIC9ib290L2tlcm5lbC96ZnMua28uc3ltYm9scy4uLmRv bmUuDQpkb25lLg0KTG9hZGVkIHN5bWJvbHMgZm9yIC9ib290L2tlcm5lbC96ZnMua28NClJl YWRpbmcgc3ltYm9scyBmcm9tIC9ib290L2tlcm5lbC9vcGVuc29sYXJpcy5rby4uLlJlYWRp bmcgc3ltYm9scyBmcm9tIC9ib290L2tlcm5lbC9vcGVuc29sYXJpcy5rby5zeW1ib2xzLi4u ZG9uZS4NCmRvbmUuDQpMb2FkZWQgc3ltYm9scyBmb3IgL2Jvb3Qva2VybmVsL29wZW5zb2xh cmlzLmtvDQojMCAgZG9hZHVtcCAoKSBhdCBwY3B1Lmg6MTk1DQoxOTUJcGNwdS5oOiBObyBz dWNoIGZpbGUgb3IgZGlyZWN0b3J5Lg0KCWluIHBjcHUuaA0KKGtnZGIpIGJ0DQojMCAgZG9h ZHVtcCAoKSBhdCBwY3B1Lmg6MTk1DQojMSAgMHhmZmZmZmZmZmI0YmViNTkwIGluID8/ICgp DQojMiAgMHhmZmZmZmZmZjgwNDNjYzU5IGluIGJvb3QgKGhvd3RvPTI2MCkgYXQgL3Vzci9z cmMvc3lzL2tlcm4va2Vybl9zaHV0ZG93bi5jOjQxOA0KIzMgIDB4ZmZmZmZmZmY4MDQzZDA2 MiBpbiBwYW5pYyAoZm10PTB4MTA0IDxBZGRyZXNzIDB4MTA0IG91dCBvZiBib3VuZHM+KSBh dCAvdXNyL3NyYy9zeXMva2Vybi9rZXJuX3NodXRkb3duLmM6NTc0DQojNCAgMHhmZmZmZmZm ZjgwNmQ0YzQzIGluIHRyYXBfZmF0YWwgKGZyYW1lPTB4ZmZmZmZmMDAwNDgxYjM3MCwgZXZh PVZhcmlhYmxlICJldmEiIGlzIG5vdCBhdmFpbGFibGUuDQopIGF0IC91c3Ivc3JjL3N5cy9h bWQ2NC9hbWQ2NC90cmFwLmM6NzY0DQojNSAgMHhmZmZmZmZmZjgwNmQ1MDE1IGluIHRyYXBf cGZhdWx0IChmcmFtZT0weGZmZmZmZmZmYjRiZWI4NDAsIHVzZXJtb2RlPTApIGF0IC91c3Iv c3JjL3N5cy9hbWQ2NC9hbWQ2NC90cmFwLmM6NjgwDQojNiAgMHhmZmZmZmZmZjgwNmQ1OTU4 IGluIHRyYXAgKGZyYW1lPTB4ZmZmZmZmZmZiNGJlYjg0MCkgYXQgL3Vzci9zcmMvc3lzL2Ft ZDY0L2FtZDY0L3RyYXAuYzo0NDkNCiM3ICAweGZmZmZmZmZmODA2YmIxNGUgaW4gY2FsbHRy YXAgKCkgYXQgL3Vzci9zcmMvc3lzL2FtZDY0L2FtZDY0L2V4Y2VwdGlvbi5TOjIwOQ0KIzgg IDB4ZmZmZmZmZmZiNDgzOWZiNiBpbiBkbXVfb2Jqc2V0X3N5bmNfZG5vZGVzIChsaXN0PTB4 ZmZmZmZmMDAwNDdmNDZjMCwgdHg9MHhmZmZmZmYwMDA0ZjMzMjgwKQ0KICAgIGF0IC91c3Iv c3JjL3N5cy9tb2R1bGVzL3pmcy8uLi8uLi9jZGRsL2NvbnRyaWIvb3BlbnNvbGFyaXMvdXRz L2NvbW1vbi9mcy96ZnMvZG11X29ianNldC5jOjcwNw0KIzkgIDB4ZmZmZmZmZmZiNDgzYTEx ZCBpbiBkbXVfb2Jqc2V0X3N5bmMgKG9zPTB4ZmZmZmZmMDAwNDdmNDYwMCwgcGlvPTB4ZmZm ZmZmMDAwNDM2N2FjMCwgdHg9MHhmZmZmZmYwMDA0ZjMzMjgwKQ0KICAgIGF0IC91c3Ivc3Jj L3N5cy9tb2R1bGVzL3pmcy8uLi8uLi9jZGRsL2NvbnRyaWIvb3BlbnNvbGFyaXMvdXRzL2Nv bW1vbi9mcy96ZnMvZG11X29ianNldC5jOjgwOQ0KIzEwIDB4ZmZmZmZmZmZiNDg0NzZmMiBp biBkc2xfcG9vbF9zeW5jIChkcD0weGZmZmZmZjAwMDQ3OWU0MDAsIHR4Zz00ODY0KSBhdCAv dXNyL3NyYy9zeXMvbW9kdWxlcy96ZnMvLi4vLi4vY2RkbC9jb250cmliL29wZW5zb2xhcmlz L3V0cy9jb21tb24vZnMvemZzL2RzbF9wb29sLmM6MTg4DQojMTEgMHhmZmZmZmZmZmI0ODUy MDIwIGluIHNwYV9zeW5jIChzcGE9MHhmZmZmZmYwMDA0MDI4MDAwLCB0eGc9NDg2NCkgYXQg L3Vzci9zcmMvc3lzL21vZHVsZXMvemZzLy4uLy4uL2NkZGwvY29udHJpYi9vcGVuc29sYXJp cy91dHMvY29tbW9uL2ZzL3pmcy9zcGEuYzoyOTg5DQojMTIgMHhmZmZmZmZmZmI0ODU3ZWVm IGluIHR4Z19zeW5jX3RocmVhZCAoYXJnPVZhcmlhYmxlICJhcmciIGlzIG5vdCBhdmFpbGFi bGUuDQopIGF0IC91c3Ivc3JjL3N5cy9tb2R1bGVzL3pmcy8uLi8uLi9jZGRsL2NvbnRyaWIv b3BlbnNvbGFyaXMvdXRzL2NvbW1vbi9mcy96ZnMvdHhnLmM6MzMxDQojMTMgMHhmZmZmZmZm ZjgwNDFhZTUzIGluIGZvcmtfZXhpdCAoY2FsbG91dD0weGZmZmZmZmZmYjQ4NTdkYzAgPHR4 Z19zeW5jX3RocmVhZD4sIGFyZz0weGZmZmZmZjAwMDQ3OWU0MDAsIGZyYW1lPTB4ZmZmZmZm ZmZiNGJlYmM4MCkgYXQgL3Vzci9zcmMvc3lzL2tlcm4va2Vybl9mb3JrLmM6ODA0DQojMTQg MHhmZmZmZmZmZjgwNmJiNTFlIGluIGZvcmtfdHJhbXBvbGluZSAoKSBhdCAvdXNyL3NyYy9z eXMvYW1kNjQvYW1kNjQvZXhjZXB0aW9uLlM6NDU1DQojMTUgMHgwMDAwMDAwMDAwMDAwMDAw IGluID8/ICgpDQojMTYgMHgwMDAwMDAwMDAwMDAwMDAwIGluID8/ICgpDQojMTcgMHgwMDAw MDAwMDAwMDAwMDAxIGluID8/ICgpDQojMTggMHgwMDAwMDAwMDAwMDAwMDAwIGluID8/ICgp DQojMTkgMHgwMDAwMDAwMDAwMDAwMDAwIGluID8/ICgpDQojMjAgMHgwMDAwMDAwMDAwMDAw MDAwIGluID8/ICgpDQojMjEgMHgwMDAwMDAwMDAwMDAwMDAwIGluID8/ICgpDQojMjIgMHgw MDAwMDAwMDAwMDAwMDAwIGluID8/ICgpDQojMjMgMHgwMDAwMDAwMDAwMDAwMDAwIGluID8/ ICgpDQojMjQgMHgwMDAwMDAwMDAwMDAwMDAwIGluID8/ICgpDQojMjUgMHgwMDAwMDAwMDAw MDAwMDAwIGluID8/ICgpDQojMjYgMHgwMDAwMDAwMDAwMDAwMDAwIGluID8/ICgpDQojMjcg MHgwMDAwMDAwMDAwMDAwMDAwIGluID8/ICgpDQojMjggMHgwMDAwMDAwMDAwMDAwMDAwIGlu ID8/ICgpDQojMjkgMHgwMDAwMDAwMDAwMDAwMDAwIGluID8/ICgpDQojMzAgMHgwMDAwMDAw MDAwMDAwMDAwIGluID8/ICgpDQojMzEgMHgwMDAwMDAwMDAwMDAwMDAwIGluID8/ICgpDQoj MzIgMHgwMDAwMDAwMDAwMDAwMDAwIGluID8/ICgpDQojMzMgMHgwMDAwMDAwMDAwMDAwMDAw IGluID8/ICgpDQojMzQgMHgwMDAwMDAwMDAwMDAwMDAwIGluID8/ICgpDQojMzUgMHgwMDAw MDAwMDAwMDAwMDAwIGluID8/ICgpDQojMzYgMHgwMDAwMDAwMDAwMDAwMDAwIGluID8/ICgp DQojMzcgMHgwMDAwMDAwMDAwMDAwMDAwIGluID8/ICgpDQojMzggMHgwMDAwMDAwMDAwMDAw MDAwIGluID8/ICgpDQojMzkgMHgwMDAwMDAwMDAwYzQxMDAwIGluID8/ICgpDQojNDAgMHhm ZmZmZmZmZjgwOWRhODgwIGluIHRkZ19tYXhpZCAoKQ0KLS0tVHlwZSA8cmV0dXJuPiB0byBj b250aW51ZSwgb3IgcSA8cmV0dXJuPiB0byBxdWl0LS0tDQojNDEgMHhmZmZmZmZmZjgwOWU3 MDgwIGluIHRkcV9jcHUgKCkNCiM0MiAweGZmZmZmZmZmODA5ZTcwODAgaW4gdGRxX2NwdSAo KQ0KIzQzIDB4ZmZmZmZmMDAwNDgxYjM3MCBpbiA/PyAoKQ0KIzQ0IDB4ZmZmZmZmMDAwNDgx YjZhMCBpbiA/PyAoKQ0KIzQ1IDB4ZmZmZmZmZmZiNGJlYjFjOCBpbiA/PyAoKQ0KIzQ2IDB4 MDAwMDAwMDAwMDAwMDAwMCBpbiA/PyAoKQ0KIzQ3IDB4ZmZmZmZmZmY4MDQ1ZDhiOCBpbiBz Y2hlZF9zd2l0Y2ggKHRkPTB4ZmZmZmZmZmZiNDg1N2RjMCwgbmV3dGQ9MHg4MDA1NWE1ZDAs IGZsYWdzPVZhcmlhYmxlICJmbGFncyIgaXMgbm90IGF2YWlsYWJsZS4NCikgYXQgL3Vzci9z cmMvc3lzL2tlcm4vc2NoZWRfdWxlLmM6MTkzOA0KIzQ4IDB4MDAwMDAwMDAwMDAwMDAwMCBp biA/PyAoKQ0KIzQ5IDB4MDAwMDAwMDAwMDAwMDAwMCBpbiA/PyAoKQ0KIzUwIDB4MDAwMDAw MDAwMDAwMDAwMCBpbiA/PyAoKQ0KIzUxIDB4MDAwMDAwMDAwMDAwMDAwMCBpbiA/PyAoKQ0K IzUyIDB4MDAwMDAwMDAwMDAwMDAwMCBpbiA/PyAoKQ0KIzUzIDB4MDAwMDAwMDAwMDAwMDAw MCBpbiA/PyAoKQ0KIzU0IDB4MDAwMDAwMDAwMDAwMDAwMCBpbiA/PyAoKQ0KIzU1IDB4MDAw MDAwMDAwMDAwMDAwMCBpbiA/PyAoKQ0KIzU2IDB4MDAwMDAwMDAwMDAwMDAwMCBpbiA/PyAo KQ0KIzU3IDB4MDAwMDAwMDAwMDAwMDAwMCBpbiA/PyAoKQ0KIzU4IDB4MDAwMDAwMDAwMDAw MDAwMCBpbiA/PyAoKQ0KIzU5IDB4MDAwMDAwMDAwMDAwMDAwMCBpbiA/PyAoKQ0KIzYwIDB4 MDAwMDAwMDAwMDAwMDAwMCBpbiA/PyAoKQ0KIzYxIDB4MDAwMDAwMDAwMDAwMDAwMCBpbiA/ PyAoKQ0KIzYyIDB4MDAwMDAwMDAwMDAwMDAwMCBpbiA/PyAoKQ0KIzYzIDB4MDAwMDAwMDAw MDAwMDAwMCBpbiA/PyAoKQ0KIzY0IDB4MDAwMDAwMDAwMDAwMDAwMCBpbiA/PyAoKQ0KIzY1 IDB4MDAwMDAwMDAwMDAwMDAwMCBpbiA/PyAoKQ0KIzY2IDB4MDAwMDAwMDAwMDAwMDAwMCBp biA/PyAoKQ0KIzY3IDB4MDAwMDAwMDAwMDAwMDAwMCBpbiA/PyAoKQ0KIzY4IDB4MDAwMDAw MDAwMDAwMDAwMCBpbiA/PyAoKQ0KIzY5IDB4MDAwMDAwMDAwMDAwMDAwMCBpbiA/PyAoKQ0K IzcwIDB4MDAwMDAwMDAwMDAwMDAwMCBpbiA/PyAoKQ0KIzcxIDB4MDAwMDAwMDAwMDAwMDAw MCBpbiA/PyAoKQ0KIzcyIDB4MDAwMDAwMDAwMDAwMDAwMCBpbiA/PyAoKQ0KIzczIDB4MDAw MDAwMDAwMDAwMDAwMCBpbiA/PyAoKQ0KIzc0IDB4MDAwMDAwMDAwMDAwMDAwMCBpbiA/PyAo KQ0KIzc1IDB4MDAwMDAwMDAwMDAwMDAwMCBpbiA/PyAoKQ0KIzc2IDB4MDAwMDAwMDAwMDAw MDAwMCBpbiA/PyAoKQ0KIzc3IDB4MDAwMDAwMDAwMDAwMDAwMCBpbiA/PyAoKQ0KIzc4IDB4 MDAwMDAwMDAwMDAwMDAwMCBpbiA/PyAoKQ0KIzc5IDB4MDAwMDAwMDAwMDAwMDAwMCBpbiA/ PyAoKQ0KIzgwIDB4MDAwMDAwMDAwMDAwMDAwMCBpbiA/PyAoKQ0KIzgxIDB4MDAwMDAwMDAw MDAwMDAwMCBpbiA/PyAoKQ0KIzgyIDB4MDAwMDAwMDAwMDAwMDAwMCBpbiA/PyAoKQ0KIzgz IDB4MDAwMDAwMDAwMDAwMDAwMCBpbiA/PyAoKQ0KLS0tVHlwZSA8cmV0dXJuPiB0byBjb250 aW51ZSwgb3IgcSA8cmV0dXJuPiB0byBxdWl0LS0tDQojODQgMHgwMDAwMDAwMDAwMDAwMDAw IGluID8/ICgpDQojODUgMHgwMDAwMDAwMDAwMDAwMDAwIGluID8/ICgpDQojODYgMHgwMDAw MDAwMDAwMDAwMDAwIGluID8/ICgpDQojODcgMHgwMDAwMDAwMDAwMDAwMDAwIGluID8/ICgp DQojODggMHgwMDAwMDAwMDAwMDAwMDAwIGluID8/ICgpDQojODkgMHgwMDAwMDAwMDAwMDAw MDAwIGluID8/ICgpDQojOTAgMHgwMDAwMDAwMDAwMDAwMDAwIGluID8/ICgpDQojOTEgMHgw MDAwMDAwMDAwMDAwMDAwIGluID8/ICgpDQojOTIgMHgwMDAwMDAwMDAwMDAwMDAwIGluID8/ ICgpDQojOTMgMHgwMDAwMDAwMDAwMDAwMDAwIGluID8/ICgpDQojOTQgMHgwMDAwMDAwMDAw MDAwMDAwIGluID8/ICgpDQojOTUgMHgwMDAwMDAwMDAwMDAwMDAwIGluID8/ICgpDQojOTYg MHgwMDAwMDAwMDAwMDAwMDAwIGluID8/ICgpDQojOTcgMHgwMDAwMDAwMDAwMDAwMDAwIGlu ID8/ICgpDQojOTggMHgwMDAwMDAwMDAwMDAwMDAwIGluID8/ICgpDQojOTkgMHgwMDAwMDAw MDAwMDAwMDAwIGluID8/ICgpDQojMTAwIDB4MDAwMDAwMDAwMDAwMDAwMCBpbiA/PyAoKQ0K IzEwMSAweDAwMDAwMDAwMDAwMDAwMDAgaW4gPz8gKCkNCiMxMDIgMHgwMDAwMDAwMDAwMDAw MDAwIGluID8/ICgpDQojMTAzIDB4MDAwMDAwMDAwMDAwMDAwMCBpbiA/PyAoKQ0KIzEwNCAw eDAwMDAwMDAwMDAwMDAwMDAgaW4gPz8gKCkNCiMxMDUgMHgwMDAwMDAwMDAwMDAwMDAwIGlu ID8/ICgpDQojMTA2IDB4MDAwMDAwMDAwMDAwMDAwMCBpbiA/PyAoKQ0KIzEwNyAweDAwMDAw MDAwMDAwMDAwMDAgaW4gPz8gKCkNCiMxMDggMHgwMDAwMDAwMDAwMDAwMDAwIGluID8/ICgp DQojMTA5IDB4MDAwMDAwMDAwMDAwMDAwMCBpbiA/PyAoKQ0KIzExMCAweDAwMDAwMDAwMDAw MDAwMDAgaW4gPz8gKCkNCiMxMTEgMHgwMDAwMDAwMDAwMDAwMDAwIGluID8/ICgpDQojMTEy IDB4MDAwMDAwMDAwMDAwMDAwMCBpbiA/PyAoKQ0KIzExMyAweDAwMDAwMDAwMDAwMDAwMDAg aW4gPz8gKCkNCiMxMTQgMHgwMDAwMDAwMDAwMDAwMDAwIGluID8/ICgpDQojMTE1IDB4MDAw MDAwMDAwMDAwMDAwMCBpbiA/PyAoKQ0KQ2Fubm90IGFjY2VzcyBtZW1vcnkgYXQgYWRkcmVz cyAweGZmZmZmZmZmYjRiZWMwMDANCihrZ2RiKSBxdWl0DQoKU2NyaXB0IGRvbmUgb24gRnJp IE5vdiAxNCAxMDozNjo0MyAyMDA4Cg== --------------060902050107000202090406--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?491CE71F.2020208>