Date: Wed, 06 Apr 2022 13:18:49 +0200 From: egoitz@ramattack.net To: Freebsd hackers <freebsd-hackers@freebsd.org> Subject: Desperate with 870 QVO and ZFS Message-ID: <6cf6c03c5a4aa8128575ec4e2f70b168@ramattack.net>
next in thread | raw e-mail | index | archive | help
--=_94ea2695193cf369660b9253165fcef5 Content-Type: multipart/alternative; boundary="=_fa7f63b00c51f44a9a0b358901477e45" --=_fa7f63b00c51f44a9a0b358901477e45 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII Good morning, I write this post with the expectation that perhaps someone could help me I am running some mail servers with FreeBSD and ZFS. They use 870 QVO (not EVO or other Samsung SSD disks) disks as storage. They can easily have from 1500 to 2000 concurrent connections. The machines have 128GB of ram and the CPU is almost absolutely idle. The disk IO is normally at 30 or 40% percent at most. The problem I'm facing is that they could be running just fine and suddenly at some peak hour, the IO goes to 60 or 70% and the machine becomes extremely slow. ZFS is all by default, except the sync parameter which is set disabled. Apart from that the ARC is limited to 64GB. But even this is extremely odd. The used ARC is near 20GB. I have seen, that meta cache in arc is very near to the limit that FreeBSD automatically sets depending on the size of the ARC you set. It seems that almost all ARC is used by meta cache. I have seen this effect in all my mail servers with this hardware and software config. I do attach a zfs-stats output, but from now that the servers are not so loaded as described. I do explain. I run a couple of Cyrus instances in these servers. One as master, one as slave on each server. The commented situation from above, happens when both Cyrus instances become master, so when we are using two Cyrus instances giving service in the same machine. For avoiding issues, know we have balanced and we have a master and a slave in each server. You know, a slave instance has almost no io and only a single connection for replication. So the zfs-stats output is from now we have let's say half of load in each server, because they have one master and one slave instance. As said before, when I place two masters in same server, perhaps all day works, but just at 11:00 am (for example) the IO goes to 60% (it doesn't increase) but it seems like if the IO where not being able to be served, let's say more than a limit. More than a concrete io limit (I'd say 60%). I don't really know if, perhaps the QVO technology could be the guilty here.... because... they say are desktop computers disks... but later... I have get a nice performance when copying for instance mailboxes from five to five.... I can flood a gigabit interface when copying mailboxes between servers from five to five.... they seem to perform.... Could anyone please shed us some light in this issue?. I don't really know what to think. Best regards, --=_fa7f63b00c51f44a9a0b358901477e45 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=UTF-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; charset= =3DUTF-8" /></head><body style=3D'font-size: 10pt; font-family: Verdana,Gen= eva,sans-serif'> <div class=3D"bbWrapper">Good morning,<br /> <br /> I write this post with = the expectation that perhaps someone could help me <img class=3D"smilie smi= lie--sprite smilie--sprite1" title=3D"Smile :)" src=3D"./?_task=3Dmail&a= mp;_id=3D110741749624d770402d5b&_action=3Ddisplay-attachment&_file= =3Drcmfile107211649243908048562900" alt=3D":)" /><br /> <br /> I am running= some mail servers with FreeBSD and ZFS. They use 870 QVO (not EVO or other= Samsung SSD disks) disks as storage. They can easily have from 1500 to 200= 0 concurrent connections. The machines have 128GB of ram and the CPU is alm= ost absolutely idle. The disk IO is normally at 30 or 40% percent at most= =2E<br /> <br /> The problem I'm facing is that they could be running just = fine and suddenly at some peak hour, the IO goes to 60 or 70% and the machi= ne becomes extremely slow. ZFS is all by default, except the sync parameter= which is set disabled. Apart from that the ARC is limited to 64GB. But eve= n this is extremely odd. The used ARC is near 20GB. I have seen, that meta = cache in arc is very near to the limit that FreeBSD automatically sets depe= nding on the size of the ARC you set. It seems that almost all ARC is used = by meta cache. I have seen this effect in all my mail servers with this har= dware and software config.<br /> <br /> I do attach a zfs-stats output, but= from now that the servers are not so loaded as described. I do explain. I = run a couple of Cyrus instances in these servers. One as master, one as sla= ve on each server. The commented situation from above, happens when both Cy= rus instances become master, so when we are using two Cyrus instances givin= g service in the same machine. For avoiding issues, know we have balanced a= nd we have a master and a slave in each server. You know, a slave instance = has almost no io and only a single connection for replication. So the zfs-s= tats output is from now we have let's say half of load in each server, beca= use they have one master and one slave instance.<br /> <br /> As said befor= e, when I place two masters in same server, perhaps all day works, but just= at 11:00 am (for example) the IO goes to 60% (it doesn't increase) but it = seems like if the IO where not being able to be served, let's say more than= a limit. More than a concrete io limit (I'd say 60%).<br /> <br /> I don't= really know if, perhaps the QVO technology could be the guilty here.... be= cause... they say are desktop computers disks... but later... I have get a = nice performance when copying for instance mailboxes from five to five...= =2E I can flood a gigabit interface when copying mailboxes between servers = from five to five.... they seem to perform....<br /> <br /> Could anyone pl= ease shed us some light in this issue?. I don't really know what to think= =2E<br /> <br /> Best regards,</div> <div class=3D"js-selectToQuoteEnd"> </div> <p><br /></p> </body></html> --=_fa7f63b00c51f44a9a0b358901477e45-- --=_94ea2695193cf369660b9253165fcef5 Content-Transfer-Encoding: base64 Content-Type: text/plain; name=zfs-stats.txt Content-Disposition: attachment; filename=zfs-stats.txt; size=12369 L3RtcC96ZnMtc3RhdHMgLWEKCi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQpaRlMgU3Vic3lzdGVtIFJlcG9ydAkJ CQlXZWQgQXByICA2IDExOjU4OjE4IDIwMjIKLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCgpTeXN0ZW0gSW5mb3Jt YXRpb246CgoJS2VybmVsIFZlcnNpb246CQkJCTEyMDIwMDAgKG9zcmVsZGF0ZSkKCUhhcmR3YXJl IFBsYXRmb3JtOgkJCWFtZDY0CglQcm9jZXNzb3IgQXJjaGl0ZWN0dXJlOgkJCWFtZDY0CgoJWkZT IFN0b3JhZ2UgcG9vbCBWZXJzaW9uOgkJNTAwMAoJWkZTIEZpbGVzeXN0ZW0gVmVyc2lvbjoJCQk1 CgpGcmVlQlNEIDEyLjItUkVMRUFTRS1wNiByMzY5ODU1IEdFTkVSSUMgMTE6NThBTSAgdXAgMTQ4 IGRheXMsIDIwOjI5LCAxIHVzZXIsIGxvYWQgYXZlcmFnZXM6IDIuMTEsIDIuNDcsIDIuMjYKCi0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLQoKU3lzdGVtIE1lbW9yeToKCgkzLjI0JQk0LjAzCUdpQiBBY3RpdmUsCTcx LjUwJQk4OC45MwlHaUIgSW5hY3QKCTIyLjc4JQkyOC4zMwlHaUIgV2lyZWQsCTAuMDAlCTAJQnl0 ZXMgQ2FjaGUKCTIuMjUlCTIuODAJR2lCIEZyZWUsCTAuMjMlCTI5Mi42MQlNaUIgR2FwCgoJUmVh bCBJbnN0YWxsZWQ6CQkJCTEyOC4wMAlHaUIKCVJlYWwgQXZhaWxhYmxlOgkJCTk5LjcwJQkxMjcu NjIJR2lCCglSZWFsIE1hbmFnZWQ6CQkJOTcuNDclCTEyNC4zOQlHaUIKCglMb2dpY2FsIFRvdGFs OgkJCQkxMjguMDAJR2lCCglMb2dpY2FsIFVzZWQ6CQkJMjguMzMlCTM2LjI2CUdpQgoJTG9naWNh bCBGcmVlOgkJCTcxLjY3JQk5MS43NAlHaUIKCktlcm5lbCBNZW1vcnk6CQkJCQkzLjIxCUdpQgoJ RGF0YToJCQkJOTguODIlCTMuMTcJR2lCCglUZXh0OgkJCQkxLjE4JQkzOC42NQlNaUIKCktlcm5l bCBNZW1vcnkgTWFwOgkJCQkxMjQuMzkJR2lCCglTaXplOgkJCQkyMi4zNCUJMjcuNzkJR2lCCglG cmVlOgkJCQk3Ny42NiUJOTYuNjAJR2lCCgotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KCkFSQyBTdW1tYXJ5OiAo SEVBTFRIWSkKCU1lbW9yeSBUaHJvdHRsZSBDb3VudDoJCQkwCgpBUkMgTWlzYzoKCURlbGV0ZWQ6 CQkJCTEuNzgJYgoJTXV0ZXggTWlzc2VzOgkJCQkxODcuMjUJbQoJRXZpY3QgU2tpcHM6CQkJCTg3 LjM1CWIKCkFSQyBTaXplOgkJCQkyOS4yNSUJMTguNzIJR2lCCglUYXJnZXQgU2l6ZTogKEFkYXB0 aXZlKQkJMTIuNTAlCTguMDAJR2lCCglNaW4gU2l6ZSAoSGFyZCBMaW1pdCk6CQkxMi41MCUJOC4w MAlHaUIKCU1heCBTaXplIChIaWdoIFdhdGVyKToJCTg6MQk2NC4wMAlHaUIKCURlY29tcHJlc3Nl ZCBEYXRhIFNpemU6CQkJOS41MglHaUIKCUNvbXByZXNzaW9uIEZhY3RvcjoJCQkwLjUxCgpBUkMg U2l6ZSBCcmVha2Rvd246CglSZWNlbnRseSBVc2VkIENhY2hlIFNpemU6CTIuNjclCTUxMi4wMAlN aUIKCUZyZXF1ZW50bHkgVXNlZCBDYWNoZSBTaXplOgk5Ny4zMyUJMTguMjIJR2lCCgpBUkMgSGFz aCBCcmVha2Rvd246CglFbGVtZW50cyBNYXg6CQkJCTMuMTEJbQoJRWxlbWVudHMgQ3VycmVudDoJ CTE5LjIxJQk1OTYuNjIJawoJQ29sbGlzaW9uczoJCQkJMjAzLjY0CW0KCUNoYWluIE1heDoJCQkJ NQoJQ2hhaW5zOgkJCQkJMTAuMjMJawoKLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCgpBUkMgRWZmaWNpZW5jeToJ CQkJCTE4OC4wNgliCglDYWNoZSBIaXQgUmF0aW86CQk5OC42MyUJMTg1LjQ4CWIKCUNhY2hlIE1p c3MgUmF0aW86CQkxLjM3JQkyLjU4CWIKCUFjdHVhbCBIaXQgUmF0aW86CQk5OC42MiUJMTg1LjQ2 CWIKCglEYXRhIERlbWFuZCBFZmZpY2llbmN5OgkJOTguNDQlCTU3LjM2CWIKCURhdGEgUHJlZmV0 Y2ggRWZmaWNpZW5jeToJMi40NSUJODU5LjAzCW0KCglDQUNIRSBISVRTIEJZIENBQ0hFIExJU1Q6 CgkgIE1vc3QgUmVjZW50bHkgVXNlZDoJCTEwLjg5JQkyMC4yMAliCgkgIE1vc3QgRnJlcXVlbnRs eSBVc2VkOgkJODkuMTAlCTE2NS4yNgliCgkgIE1vc3QgUmVjZW50bHkgVXNlZCBHaG9zdDoJMC4w NyUJMTM4LjcwCW0KCSAgTW9zdCBGcmVxdWVudGx5IFVzZWQgR2hvc3Q6CTAuMTYlCTMwMC44NAlt CgoJQ0FDSEUgSElUUyBCWSBEQVRBIFRZUEU6CgkgIERlbWFuZCBEYXRhOgkJCTMwLjQ0JQk1Ni40 NgliCgkgIFByZWZldGNoIERhdGE6CQkwLjAxJQkyMS4wMQltCgkgIERlbWFuZCBNZXRhZGF0YToJ CTY5LjU0JQkxMjguOTgJYgoJICBQcmVmZXRjaCBNZXRhZGF0YToJCTAuMDElCTE2LjA5CW0KCglD QUNIRSBNSVNTRVMgQlkgREFUQSBUWVBFOgoJICBEZW1hbmQgRGF0YToJCQkzNC43NSUJODk2LjY3 CW0KCSAgUHJlZmV0Y2ggRGF0YToJCTMyLjQ3JQk4MzguMDIJbQoJICBEZW1hbmQgTWV0YWRhdGE6 CQkzMS4xOCUJODA0LjU3CW0KCSAgUHJlZmV0Y2ggTWV0YWRhdGE6CQkxLjYwJQk0MS4yNwltCgot LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0KCkwyQVJDIGlzIGRpc2FibGVkCgotLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KClBlciBk YXRhc2V0IHN0YXRpc3RpY3MgYXJlIG5vdCBhdmFpbGFibGUKCi0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQoKRmls ZS1MZXZlbCBQcmVmZXRjaDoKCkRNVSBFZmZpY2llbmN5OgkJCQkJODMuNTIJYgoJSGl0IFJhdGlv OgkJCTIuNzQlCTIuMjkJYgoJTWlzcyBSYXRpbzoJCQk5Ny4yNiUJODEuMjMJYgoKLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tCgpWREVWIGNhY2hlIGlzIGRpc2FibGVkCgotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KClpGUyBUdW5h YmxlcyAoc3lzY3RsKToKCWtlcm4ubWF4dXNlcnMgICAgICAgICAgICAgICAgICAgICAgICAgICA4 NTAzCgl2bS5rbWVtX3NpemUgICAgICAgICAgICAgICAgICAgICAgICAgICAgMTMzNTYwMzA3NzEy Cgl2bS5rbWVtX3NpemVfc2NhbGUgICAgICAgICAgICAgICAgICAgICAgMQoJdm0ua21lbV9zaXpl X21pbiAgICAgICAgICAgICAgICAgICAgICAgIDAKCXZtLmttZW1fc2l6ZV9tYXggICAgICAgICAg ICAgICAgICAgICAgICAxMzE5NDEzOTUwODc0Cgl2ZnMuemZzLnRyaW0ubWF4X2ludGVydmFsICAg ICAgICAgICAgICAgMQoJdmZzLnpmcy50cmltLnRpbWVvdXQgICAgICAgICAgICAgICAgICAgIDMw Cgl2ZnMuemZzLnRyaW0udHhnX2RlbGF5ICAgICAgICAgICAgICAgICAgMzIKCXZmcy56ZnMudHJp bS5lbmFibGVkICAgICAgICAgICAgICAgICAgICAxCgl2ZnMuemZzLnZvbC5pbW1lZGlhdGVfd3Jp dGVfc3ogICAgICAgICAgMzI3NjgKCXZmcy56ZnMudm9sLnVubWFwX3N5bmNfZW5hYmxlZCAgICAg ICAgICAwCgl2ZnMuemZzLnZvbC51bm1hcF9lbmFibGVkICAgICAgICAgICAgICAgMQoJdmZzLnpm cy52b2wucmVjdXJzaXZlICAgICAgICAgICAgICAgICAgIDAKCXZmcy56ZnMudm9sLm1vZGUgICAg ICAgICAgICAgICAgICAgICAgICAxCgl2ZnMuemZzLnZlcnNpb24uenBsICAgICAgICAgICAgICAg ICAgICAgNQoJdmZzLnpmcy52ZXJzaW9uLnNwYSAgICAgICAgICAgICAgICAgICAgIDUwMDAKCXZm cy56ZnMudmVyc2lvbi5hY2wgICAgICAgICAgICAgICAgICAgICAxCgl2ZnMuemZzLnZlcnNpb24u aW9jdGwgICAgICAgICAgICAgICAgICAgNwoJdmZzLnpmcy5kZWJ1ZyAgICAgICAgICAgICAgICAg ICAgICAgICAgIDAKCXZmcy56ZnMuc3VwZXJfb3duZXIgICAgICAgICAgICAgICAgICAgICAwCgl2 ZnMuemZzLmltbWVkaWF0ZV93cml0ZV9zeiAgICAgICAgICAgICAgMzI3NjgKCXZmcy56ZnMuc3lu Y19wYXNzX3Jld3JpdGUgICAgICAgICAgICAgICAyCgl2ZnMuemZzLnN5bmNfcGFzc19kb250X2Nv bXByZXNzICAgICAgICAgNQoJdmZzLnpmcy5zeW5jX3Bhc3NfZGVmZXJyZWRfZnJlZSAgICAgICAg IDIKCXZmcy56ZnMuemlvLmR2YV90aHJvdHRsZV9lbmFibGVkICAgICAgICAxCgl2ZnMuemZzLnpp by5leGNsdWRlX21ldGFkYXRhICAgICAgICAgICAgMAoJdmZzLnpmcy56aW8udXNlX3VtYSAgICAg ICAgICAgICAgICAgICAgIDEKCXZmcy56ZnMuemlvLnRhc2txX2JhdGNoX3BjdCAgICAgICAgICAg ICA3NQoJdmZzLnpmcy56aWxfbWF4YmxvY2tzaXplICAgICAgICAgICAgICAgIDEzMTA3MgoJdmZz Lnpmcy56aWxfc2xvZ19idWxrICAgICAgICAgICAgICAgICAgIDc4NjQzMgoJdmZzLnpmcy56aWxf bm9jYWNoZWZsdXNoICAgICAgICAgICAgICAgIDAKCXZmcy56ZnMuemlsX3JlcGxheV9kaXNhYmxl ICAgICAgICAgICAgICAwCgl2ZnMuemZzLmNhY2hlX2ZsdXNoX2Rpc2FibGUgICAgICAgICAgICAg MAoJdmZzLnpmcy5zdGFuZGFyZF9zbV9ibGtzeiAgICAgICAgICAgICAgIDEzMTA3MgoJdmZzLnpm cy5kdGxfc21fYmxrc3ogICAgICAgICAgICAgICAgICAgIDQwOTYKCXZmcy56ZnMubWluX2F1dG9f YXNoaWZ0ICAgICAgICAgICAgICAgICAxMgoJdmZzLnpmcy5tYXhfYXV0b19hc2hpZnQgICAgICAg ICAgICAgICAgIDEzCgl2ZnMuemZzLnZkZXYudHJpbV9tYXhfcGVuZGluZyAgICAgICAgICAgMTAw MDAKCXZmcy56ZnMudmRldi5iaW9fZGVsZXRlX2Rpc2FibGUgICAgICAgICAwCgl2ZnMuemZzLnZk ZXYuYmlvX2ZsdXNoX2Rpc2FibGUgICAgICAgICAgMAoJdmZzLnpmcy52ZGV2LmRlZl9xdWV1ZV9k ZXB0aCAgICAgICAgICAgIDMyCgl2ZnMuemZzLnZkZXYucXVldWVfZGVwdGhfcGN0ICAgICAgICAg ICAgMTAwMAoJdmZzLnpmcy52ZGV2LndyaXRlX2dhcF9saW1pdCAgICAgICAgICAgIDQwOTYKCXZm cy56ZnMudmRldi5yZWFkX2dhcF9saW1pdCAgICAgICAgICAgICAzMjc2OAoJdmZzLnpmcy52ZGV2 LmFnZ3JlZ2F0aW9uX2xpbWl0X25vbl9yb3RhdGluZzEzMTA3MgoJdmZzLnpmcy52ZGV2LmFnZ3Jl Z2F0aW9uX2xpbWl0ICAgICAgICAgIDEwNDg1NzYKCXZmcy56ZnMudmRldi5pbml0aWFsaXppbmdf bWF4X2FjdGl2ZSAgICAxCgl2ZnMuemZzLnZkZXYuaW5pdGlhbGl6aW5nX21pbl9hY3RpdmUgICAg MQoJdmZzLnpmcy52ZGV2LnJlbW92YWxfbWF4X2FjdGl2ZSAgICAgICAgIDIKCXZmcy56ZnMudmRl di5yZW1vdmFsX21pbl9hY3RpdmUgICAgICAgICAxCgl2ZnMuemZzLnZkZXYudHJpbV9tYXhfYWN0 aXZlICAgICAgICAgICAgNjQKCXZmcy56ZnMudmRldi50cmltX21pbl9hY3RpdmUgICAgICAgICAg ICAxCgl2ZnMuemZzLnZkZXYuc2NydWJfbWF4X2FjdGl2ZSAgICAgICAgICAgMgoJdmZzLnpmcy52 ZGV2LnNjcnViX21pbl9hY3RpdmUgICAgICAgICAgIDEKCXZmcy56ZnMudmRldi5hc3luY193cml0 ZV9tYXhfYWN0aXZlICAgICAxMAoJdmZzLnpmcy52ZGV2LmFzeW5jX3dyaXRlX21pbl9hY3RpdmUg ICAgIDEKCXZmcy56ZnMudmRldi5hc3luY19yZWFkX21heF9hY3RpdmUgICAgICAzCgl2ZnMuemZz LnZkZXYuYXN5bmNfcmVhZF9taW5fYWN0aXZlICAgICAgMQoJdmZzLnpmcy52ZGV2LnN5bmNfd3Jp dGVfbWF4X2FjdGl2ZSAgICAgIDEwCgl2ZnMuemZzLnZkZXYuc3luY193cml0ZV9taW5fYWN0aXZl ICAgICAgMTAKCXZmcy56ZnMudmRldi5zeW5jX3JlYWRfbWF4X2FjdGl2ZSAgICAgICAxMAoJdmZz Lnpmcy52ZGV2LnN5bmNfcmVhZF9taW5fYWN0aXZlICAgICAgIDEwCgl2ZnMuemZzLnZkZXYubWF4 X2FjdGl2ZSAgICAgICAgICAgICAgICAgMTAwMAoJdmZzLnpmcy52ZGV2LmFzeW5jX3dyaXRlX2Fj dGl2ZV9tYXhfZGlydHlfcGVyY2VudDYwCgl2ZnMuemZzLnZkZXYuYXN5bmNfd3JpdGVfYWN0aXZl X21pbl9kaXJ0eV9wZXJjZW50MzAKCXZmcy56ZnMudmRldi5taXJyb3Iubm9uX3JvdGF0aW5nX3Nl ZWtfaW5jMQoJdmZzLnpmcy52ZGV2Lm1pcnJvci5ub25fcm90YXRpbmdfaW5jICAgIDAKCXZmcy56 ZnMudmRldi5taXJyb3Iucm90YXRpbmdfc2Vla19vZmZzZXQxMDQ4NTc2Cgl2ZnMuemZzLnZkZXYu bWlycm9yLnJvdGF0aW5nX3NlZWtfaW5jICAgNQoJdmZzLnpmcy52ZGV2Lm1pcnJvci5yb3RhdGlu Z19pbmMgICAgICAgIDAKCXZmcy56ZnMudmRldi50cmltX29uX2luaXQgICAgICAgICAgICAgICAx Cgl2ZnMuemZzLnZkZXYuY2FjaGUuYnNoaWZ0ICAgICAgICAgICAgICAgMTYKCXZmcy56ZnMudmRl di5jYWNoZS5zaXplICAgICAgICAgICAgICAgICAwCgl2ZnMuemZzLnZkZXYuY2FjaGUubWF4ICAg ICAgICAgICAgICAgICAgMTYzODQKCXZmcy56ZnMudmRldi52YWxpZGF0ZV9za2lwICAgICAgICAg ICAgICAwCgl2ZnMuemZzLnZkZXYubWF4X21zX3NoaWZ0ICAgICAgICAgICAgICAgMzQKCXZmcy56 ZnMudmRldi5kZWZhdWx0X21zX3NoaWZ0ICAgICAgICAgICAyOQoJdmZzLnpmcy52ZGV2Lm1heF9t c19jb3VudF9saW1pdCAgICAgICAgIDEzMTA3MgoJdmZzLnpmcy52ZGV2Lm1pbl9tc19jb3VudCAg ICAgICAgICAgICAgIDE2Cgl2ZnMuemZzLnZkZXYuZGVmYXVsdF9tc19jb3VudCAgICAgICAgICAg MjAwCgl2ZnMuemZzLnR4Zy50aW1lb3V0ICAgICAgICAgICAgICAgICAgICAgNQoJdmZzLnpmcy5z cGFjZV9tYXBfaWJzICAgICAgICAgICAgICAgICAgIDE0Cgl2ZnMuemZzLnNwZWNpYWxfY2xhc3Nf bWV0YWRhdGFfcmVzZXJ2ZV9wY3QyNQoJdmZzLnpmcy51c2VyX2luZGlyZWN0X2lzX3NwZWNpYWwg ICAgICAgIDEKCXZmcy56ZnMuZGR0X2RhdGFfaXNfc3BlY2lhbCAgICAgICAgICAgICAxCgl2ZnMu emZzLnNwYV9hbGxvY2F0b3JzICAgICAgICAgICAgICAgICAgNAoJdmZzLnpmcy5zcGFfbWluX3Ns b3AgICAgICAgICAgICAgICAgICAgIDEzNDIxNzcyOAoJdmZzLnpmcy5zcGFfc2xvcF9zaGlmdCAg ICAgICAgICAgICAgICAgIDUKCXZmcy56ZnMuc3BhX2FzaXplX2luZmxhdGlvbiAgICAgICAgICAg ICAyNAoJdmZzLnpmcy5kZWFkbWFuX2VuYWJsZWQgICAgICAgICAgICAgICAgIDEKCXZmcy56ZnMu ZGVhZG1hbl9jaGVja3RpbWVfbXMgICAgICAgICAgICA1MDAwCgl2ZnMuemZzLmRlYWRtYW5fc3lu Y3RpbWVfbXMgICAgICAgICAgICAgMTAwMDAwMAoJdmZzLnpmcy5kZWJ1Z2ZsYWdzICAgICAgICAg ICAgICAgICAgICAgIDAKCXZmcy56ZnMucmVjb3ZlciAgICAgICAgICAgICAgICAgICAgICAgICAw Cgl2ZnMuemZzLnNwYV9sb2FkX3ZlcmlmeV9kYXRhICAgICAgICAgICAgMQoJdmZzLnpmcy5zcGFf bG9hZF92ZXJpZnlfbWV0YWRhdGEgICAgICAgIDEKCXZmcy56ZnMuc3BhX2xvYWRfdmVyaWZ5X21h eGluZmxpZ2h0ICAgICAxMDAwMAoJdmZzLnpmcy5tYXhfbWlzc2luZ190dmRzX3NjYW4gICAgICAg ICAgIDAKCXZmcy56ZnMubWF4X21pc3NpbmdfdHZkc19jYWNoZWZpbGUgICAgICAyCgl2ZnMuemZz Lm1heF9taXNzaW5nX3R2ZHMgICAgICAgICAgICAgICAgMAoJdmZzLnpmcy5zcGFfbG9hZF9wcmlu dF92ZGV2X3RyZWUgICAgICAgIDAKCXZmcy56ZnMuY2N3X3JldHJ5X2ludGVydmFsICAgICAgICAg ICAgICAzMDAKCXZmcy56ZnMuY2hlY2tfaG9zdGlkICAgICAgICAgICAgICAgICAgICAxCgl2ZnMu emZzLm11bHRpaG9zdF9mYWlsX2ludGVydmFscyAgICAgICAgMTAKCXZmcy56ZnMubXVsdGlob3N0 X2ltcG9ydF9pbnRlcnZhbHMgICAgICAyMAoJdmZzLnpmcy5tdWx0aWhvc3RfaW50ZXJ2YWwgICAg ICAgICAgICAgIDEwMDAKCXZmcy56ZnMubWdfZnJhZ21lbnRhdGlvbl90aHJlc2hvbGQgICAgICA4 NQoJdmZzLnpmcy5tZ19ub2FsbG9jX3RocmVzaG9sZCAgICAgICAgICAgIDAKCXZmcy56ZnMuY29u ZGVuc2VfcGN0ICAgICAgICAgICAgICAgICAgICAyMDAKCXZmcy56ZnMubWV0YXNsYWJfc21fYmxr c3ogICAgICAgICAgICAgICA0MDk2Cgl2ZnMuemZzLm1ldGFzbGFiLmJpYXNfZW5hYmxlZCAgICAg ICAgICAgMQoJdmZzLnpmcy5tZXRhc2xhYi5sYmFfd2VpZ2h0aW5nX2VuYWJsZWQgIDEKCXZmcy56 ZnMubWV0YXNsYWIuZnJhZ21lbnRhdGlvbl9mYWN0b3JfZW5hYmxlZDEKCXZmcy56ZnMubWV0YXNs YWIucHJlbG9hZF9lbmFibGVkICAgICAgICAxCgl2ZnMuemZzLm1ldGFzbGFiLnByZWxvYWRfbGlt aXQgICAgICAgICAgMwoJdmZzLnpmcy5tZXRhc2xhYi51bmxvYWRfZGVsYXkgICAgICAgICAgIDgK CXZmcy56ZnMubWV0YXNsYWIubG9hZF9wY3QgICAgICAgICAgICAgICA1MAoJdmZzLnpmcy5tZXRh c2xhYi5taW5fYWxsb2Nfc2l6ZSAgICAgICAgIDMzNTU0NDMyCgl2ZnMuemZzLm1ldGFzbGFiLmRm X2ZyZWVfcGN0ICAgICAgICAgICAgNAoJdmZzLnpmcy5tZXRhc2xhYi5kZl9hbGxvY190aHJlc2hv bGQgICAgIDEzMTA3MgoJdmZzLnpmcy5tZXRhc2xhYi5kZWJ1Z191bmxvYWQgICAgICAgICAgIDAK CXZmcy56ZnMubWV0YXNsYWIuZGVidWdfbG9hZCAgICAgICAgICAgICAwCgl2ZnMuemZzLm1ldGFz bGFiLmZyYWdtZW50YXRpb25fdGhyZXNob2xkNzAKCXZmcy56ZnMubWV0YXNsYWIuZm9yY2VfZ2Fu Z2luZyAgICAgICAgICAxNjc3NzIxNwoJdmZzLnpmcy5mcmVlX2Jwb2JqX2VuYWJsZWQgICAgICAg ICAgICAgIDEKCXZmcy56ZnMuZnJlZV9tYXhfYmxvY2tzICAgICAgICAgICAgICAgICAtMQoJdmZz Lnpmcy56ZnNfc2Nhbl9jaGVja3BvaW50X2ludGVydmFsICAgIDcyMDAKCXZmcy56ZnMuemZzX3Nj YW5fbGVnYWN5ICAgICAgICAgICAgICAgICAwCgl2ZnMuemZzLm5vX3NjcnViX3ByZWZldGNoICAg ICAgICAgICAgICAgMAoJdmZzLnpmcy5ub19zY3J1Yl9pbyAgICAgICAgICAgICAgICAgICAgIDAK CXZmcy56ZnMucmVzaWx2ZXJfbWluX3RpbWVfbXMgICAgICAgICAgICAzMDAwCgl2ZnMuemZzLmZy ZWVfbWluX3RpbWVfbXMgICAgICAgICAgICAgICAgMTAwMAoJdmZzLnpmcy5zY2FuX21pbl90aW1l X21zICAgICAgICAgICAgICAgIDEwMDAKCXZmcy56ZnMuc2Nhbl9pZGxlICAgICAgICAgICAgICAg ICAgICAgICA1MAoJdmZzLnpmcy5zY3J1Yl9kZWxheSAgICAgICAgICAgICAgICAgICAgIDQKCXZm cy56ZnMucmVzaWx2ZXJfZGVsYXkgICAgICAgICAgICAgICAgICAyCgl2ZnMuemZzLnpmZXRjaC5h cnJheV9yZF9zeiAgICAgICAgICAgICAgMTA0ODU3NgoJdmZzLnpmcy56ZmV0Y2gubWF4X2lkaXN0 YW5jZSAgICAgICAgICAgIDY3MTA4ODY0Cgl2ZnMuemZzLnpmZXRjaC5tYXhfZGlzdGFuY2UgICAg ICAgICAgICAgODM4ODYwOAoJdmZzLnpmcy56ZmV0Y2gubWluX3NlY19yZWFwICAgICAgICAgICAg IDIKCXZmcy56ZnMuemZldGNoLm1heF9zdHJlYW1zICAgICAgICAgICAgICA4Cgl2ZnMuemZzLnBy ZWZldGNoX2Rpc2FibGUgICAgICAgICAgICAgICAgMAoJdmZzLnpmcy5kZWxheV9zY2FsZSAgICAg ICAgICAgICAgICAgICAgIDUwMDAwMAoJdmZzLnpmcy5kZWxheV9taW5fZGlydHlfcGVyY2VudCAg ICAgICAgIDYwCgl2ZnMuemZzLmRpcnR5X2RhdGFfc3luY19wY3QgICAgICAgICAgICAgMjAKCXZm cy56ZnMuZGlydHlfZGF0YV9tYXhfcGVyY2VudCAgICAgICAgICAxMAoJdmZzLnpmcy5kaXJ0eV9k YXRhX21heF9tYXggICAgICAgICAgICAgIDQyOTQ5NjcyOTYKCXZmcy56ZnMuZGlydHlfZGF0YV9t YXggICAgICAgICAgICAgICAgICA0Mjk0OTY3Mjk2Cgl2ZnMuemZzLm1heF9yZWNvcmRzaXplICAg ICAgICAgICAgICAgICAgMTA0ODU3NgoJdmZzLnpmcy5kZWZhdWx0X2licyAgICAgICAgICAgICAg ICAgICAgIDE3Cgl2ZnMuemZzLmRlZmF1bHRfYnMgICAgICAgICAgICAgICAgICAgICAgOQoJdmZz Lnpmcy5zZW5kX2hvbGVzX3dpdGhvdXRfYmlydGhfdGltZSAgIDEKCXZmcy56ZnMubWRjb21wX2Rp c2FibGUgICAgICAgICAgICAgICAgICAwCgl2ZnMuemZzLnBlcl90eGdfZGlydHlfZnJlZXNfcGVy Y2VudCAgICAgNQoJdmZzLnpmcy5ub3B3cml0ZV9lbmFibGVkICAgICAgICAgICAgICAgIDEKCXZm cy56ZnMuZGVkdXAucHJlZmV0Y2ggICAgICAgICAgICAgICAgICAxCgl2ZnMuemZzLmRidWZfY2Fj aGVfbG93YXRlcl9wY3QgICAgICAgICAgMTAKCXZmcy56ZnMuZGJ1Zl9jYWNoZV9oaXdhdGVyX3Bj dCAgICAgICAgICAxMAoJdmZzLnpmcy5kYnVmX21ldGFkYXRhX2NhY2hlX292ZXJmbG93ICAgIDAK CXZmcy56ZnMuZGJ1Zl9tZXRhZGF0YV9jYWNoZV9zaGlmdCAgICAgICA2Cgl2ZnMuemZzLmRidWZf Y2FjaGVfc2hpZnQgICAgICAgICAgICAgICAgNQoJdmZzLnpmcy5kYnVmX21ldGFkYXRhX2NhY2hl X21heF9ieXRlcyAgIDEwNzM3NDE4MjQKCXZmcy56ZnMuZGJ1Zl9jYWNoZV9tYXhfYnl0ZXMgICAg ICAgICAgICAyMTQ3NDgzNjQ4Cgl2ZnMuemZzLmFyY19taW5fcHJlc2NpZW50X3ByZWZldGNoX21z ICAgNgoJdmZzLnpmcy5hcmNfbWluX3ByZWZldGNoX21zICAgICAgICAgICAgIDEKCXZmcy56ZnMu bDJjX29ubHlfc2l6ZSAgICAgICAgICAgICAgICAgICAwCgl2ZnMuemZzLm1mdV9naG9zdF9kYXRh X2VzaXplICAgICAgICAgICAgMTQ2NjE1MzQ3MgoJdmZzLnpmcy5tZnVfZ2hvc3RfbWV0YWRhdGFf ZXNpemUgICAgICAgIDcwNzIzODA0MTYKCXZmcy56ZnMubWZ1X2dob3N0X3NpemUgICAgICAgICAg ICAgICAgICA4NTM4NTMzODg4Cgl2ZnMuemZzLm1mdV9kYXRhX2VzaXplICAgICAgICAgICAgICAg ICAgMAoJdmZzLnpmcy5tZnVfbWV0YWRhdGFfZXNpemUgICAgICAgICAgICAgIDAKCXZmcy56ZnMu bWZ1X3NpemUgICAgICAgICAgICAgICAgICAgICAgICA5ODMxMjcwNDAKCXZmcy56ZnMubXJ1X2do b3N0X2RhdGFfZXNpemUgICAgICAgICAgICAwCgl2ZnMuemZzLm1ydV9naG9zdF9tZXRhZGF0YV9l c2l6ZSAgICAgICAgMAoJdmZzLnpmcy5tcnVfZ2hvc3Rfc2l6ZSAgICAgICAgICAgICAgICAgIDAK CXZmcy56ZnMubXJ1X2RhdGFfZXNpemUgICAgICAgICAgICAgICAgICAwCgl2ZnMuemZzLm1ydV9t ZXRhZGF0YV9lc2l6ZSAgICAgICAgICAgICAgMAoJdmZzLnpmcy5tcnVfc2l6ZSAgICAgICAgICAg ICAgICAgICAgICAgIDEyOTAzMzUxMjk2Cgl2ZnMuemZzLmFub25fZGF0YV9lc2l6ZSAgICAgICAg ICAgICAgICAgMAoJdmZzLnpmcy5hbm9uX21ldGFkYXRhX2VzaXplICAgICAgICAgICAgIDAKCXZm cy56ZnMuYW5vbl9zaXplICAgICAgICAgICAgICAgICAgICAgICAxMjUxMjI1NjAKCXZmcy56ZnMu bDJhcmNfbm9ydyAgICAgICAgICAgICAgICAgICAgICAxCgl2ZnMuemZzLmwyYXJjX2ZlZWRfYWdh aW4gICAgICAgICAgICAgICAgMQoJdmZzLnpmcy5sMmFyY19ub3ByZWZldGNoICAgICAgICAgICAg ICAgIDEKCXZmcy56ZnMubDJhcmNfZmVlZF9taW5fbXMgICAgICAgICAgICAgICAyMDAKCXZmcy56 ZnMubDJhcmNfZmVlZF9zZWNzICAgICAgICAgICAgICAgICAxCgl2ZnMuemZzLmwyYXJjX2hlYWRy b29tICAgICAgICAgICAgICAgICAgMgoJdmZzLnpmcy5sMmFyY193cml0ZV9ib29zdCAgICAgICAg ICAgICAgIDgzODg2MDgKCXZmcy56ZnMubDJhcmNfd3JpdGVfbWF4ICAgICAgICAgICAgICAgICA4 Mzg4NjA4Cgl2ZnMuemZzLmFyY19tZXRhX3N0cmF0ZWd5ICAgICAgICAgICAgICAgMAoJdmZzLnpm cy5hcmNfbWV0YV9saW1pdCAgICAgICAgICAgICAgICAgIDE3MTc5ODY5MTg0Cgl2ZnMuemZzLmFy Y19mcmVlX3RhcmdldCAgICAgICAgICAgICAgICAgNjk0NzI0Cgl2ZnMuemZzLmFyY19rbWVtX2Nh Y2hlX3JlYXBfcmV0cnlfbXMgICAgMTAwMAoJdmZzLnpmcy5jb21wcmVzc2VkX2FyY19lbmFibGVk ICAgICAgICAgIDEKCXZmcy56ZnMuYXJjX2dyb3dfcmV0cnkgICAgICAgICAgICAgICAgICA2MAoJ dmZzLnpmcy5hcmNfc2hyaW5rX3NoaWZ0ICAgICAgICAgICAgICAgIDcKCXZmcy56ZnMuYXJjX2F2 ZXJhZ2VfYmxvY2tzaXplICAgICAgICAgICA4MTkyCgl2ZnMuemZzLmFyY19ub19ncm93X3NoaWZ0 ICAgICAgICAgICAgICAgNQoJdmZzLnpmcy5hcmNfbWluICAgICAgICAgICAgICAgICAgICAgICAg IDg1ODk5MzQ1OTIKCXZmcy56ZnMuYXJjX21heCAgICAgICAgICAgICAgICAgICAgICAgICA2ODcx OTQ3NjczNgoJdmZzLnpmcy5hYmRfY2h1bmtfc2l6ZSAgICAgICAgICAgICAgICAgIDQwOTYKCXZm cy56ZnMuYWJkX3NjYXR0ZXJfZW5hYmxlZCAgICAgICAgICAgICAxCgotLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0K --=_94ea2695193cf369660b9253165fcef5--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?6cf6c03c5a4aa8128575ec4e2f70b168>