Date: Wed, 4 Apr 2018 14:27:30 -0400 From: Mike Tancsa <mike@sentex.net> To: "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org> Subject: Linux NFS client and FreeBSD server strangeness Message-ID: <369fab06-6213-ba87-cc66-c9829e8a76a0@sentex.net>
next in thread | raw e-mail | index | archive | help
This is a multi-part message in MIME format.
--------------0092DAC1F1203FC0CDCBEF55
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
Not sure where the tweaking needs to happen, but I am getting strange
behaviour between a Linux nfs client and FreeBSD RELENG_11 NFS server.
The FreeBSD server starts with
nfs_client_enable="YES"
nfs_server_enable="YES"
rpcbind_enable="YES"
rpc_lockd_enable="YES"
rpc_statd_enable="YES"
nfs_server_flags="-u -t -n 16"
and on the Linux client I have been trying various options to no avail.
The mount works, but on a straight up write to the FreeBSD server,
everything is very bursty. I noticed this (I think) a few months ago
where Linux dumps across an nfs mount seemed to take a lot longer and
were getting very bursty.
It seems if there are a mixture of reads and writes, everything is
pretty fast. But if a client is just writing to the server, something,
somewhere is blocking. Doing something simple like
ls -l /nfsmount
from the client "wakes" up the server/client so that write stream can
keep going. Otherwise, it will do a big blast of writes and then several
seconds of pausing on the dump.
Linux Dump is a simple
/sbin/dump u -0 -f - / | /bin/bzip2 >/backup/dump-root-0.bz2
Mount is
mount.nfs -o tcp,intr,noatime,vers=3 192.168.yy.xx:/path
If I run ifstat on the FreeBSD nfs server, the traffic pattern looks like
# ifstat -b -i cxl0
cxl0
Kbps in Kbps out
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00
8.12e+06 45127.03
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00
6.04e+06 33525.76
901122.1 4983.72
0.00 0.00
if I do a bunch of ls -l /nfsmount on the client
eg
while true
do
ls -l /backup/ > /dev/null
done
traffic pattern is
cxl0
Kbps in Kbps out
0.00 0.00
3.31e+06 18520.03
5.89e+06 32571.52
4.84e+06 28325.71
2.12e+06 19466.56
614727.0 12246.10
874927.6 13557.18
1.06e+06 14386.78
917865.4 13696.87
1.09e+06 14608.64
1.06e+06 14376.12
164077.3 5286.64
Leading up to the stall, pcap snippet attached.
Note, doing something like
dd if=/dev/zero of=/backup/test.bin bs=4096 count=5000000
I can saturate the 10G link and max out the disk on the server
# dd if=/dev/zero of=/backup/test.bin bs=4096 count=5000000
5000000+0 records in
5000000+0 records out
20480000000 bytes (20 GB, 19 GiB) copied, 36.6238 s, 559 MB/s
and its a pretty steady stream unlike the dump. Any ideas whats going
on and how I might be able to work around this ?
192.168.xx.yy:/zbackup1/virtbox4b/backup on /backup type nfs
(rw,noatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.242.254,mountvers=3,mountport=774,mountproto=tcp,local_lock=none,addr=192.168.yy.xx)
---Mike
-------------------
Mike Tancsa, tel +1 519 651 3400 x203
Sentex Communications, mike@sentex.net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada
--------------0092DAC1F1203FC0CDCBEF55
Content-Type: text/plain; charset=UTF-8;
name="nfs.txt"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
filename="nfs.txt"
MTM6MjE6MTUuNDU3NDg0IElQICh0b3MgMHgwLCB0dGwgNjQsIGlkIDU2MDk3LCBvZmZzZXQg
MCwgZmxhZ3MgW0RGXSwgcHJvdG8gVENQICg2KSwgbGVuZ3RoIDE3NikKICAgIDE5Mi4xNjgu
Y2xpZW50LjkzNyA+IDE5Mi4xNjguc2VydmVyLjIwNDk6IEZsYWdzIFtQLl0sIGNrc3VtIDB4
OGNhOCAoY29ycmVjdCksIHNlcSA4NzUxMzI5Mjo4NzUxMzQxNiwgYWNrIDEwOTYyMSwgd2lu
IDU2MzIsIG9wdGlvbnMgW25vcCxub3AsVFMgdmFsIDEwNDg2ODMgZWNyIDUwMDEwMzg3Ml0s
IGxlbmd0aCAxMjQ6IE5GUyByZXF1ZXN0IHhpZCA0Njg3MzM4NTAgMTIwIGNvbW1pdCBmaCBV
bmtub3duL0IzMEJGNDFFREU0OEVDNUMwQTAwMApCMDAwMDAwMDAwMDg0N0Q4MjAxMDAwMDAw
MDAwMDAwMDAwMCAwIGJ5dGVzIEAgMAoxMzoyMToxNS40NTc1MDIgSVAgKHRvcyAweDAsIHR0
bCA2NCwgaWQgMCwgb2Zmc2V0IDAsIGZsYWdzIFtERl0sIHByb3RvIFRDUCAoNiksIGxlbmd0
aCAyMDgsIGJhZCBja3N1bSAwICgtPmQzY2UpISkKICAgIDE5Mi4xNjguc2VydmVyLjIwNDkg
PiAxOTIuMTY4LmNsaWVudC45Mzc6IEZsYWdzIFtQLl0sIGNrc3VtIDB4NjcxYyAoaW5jb3Jy
ZWN0IC0+IDB4NDEwOSksIHNlcSAxMDk2MjE6MTA5Nzc3LCBhY2sgODc1MTM0MTYsIHdpbiAy
OTEyNywgb3B0aW9ucyBbbm9wLG5vcCxUUyB2YWwgNTAwMTAzODczIGVjciAxMDQ4NjgzXSwg
bGVuZ3RoIDE1NjogTkZTIHJlcGx5IHhpZCA0Njg3MzM4NTAgcmVwbHkgb2sgMTUyIGNvbW1p
dCBQUkU6IHN6IDQxNjAyMgo1MjggbXRpbWUgMTUyMjg2MjQ3NS40NTY5MDQwMDAgY3RpbWUg
MTUyMjg2MjQ3NS40NTY5MDQwMDAgUE9TVDogUkVHIDY0NCBpZHMgMjg3NjcvMCBzeiA0MTYw
MjI1MjggbmxpbmsgMSByZGV2IDEwMi84MzAzNDEyMTYgZnNpZCAxZWY0MGJiMyBmaWxlaWQg
YiBhL20vY3RpbWUgMTUyMjg2MjM1OS4xMDIxNzcwMDAgMTUyMjg2MjQ3NS40NTY5MDQwMDAg
MTUyMjg2MjQ3NS40NTY5MDQwMDAKMTM6MjE6MTUuNDYyNTA0IElQICh0b3MgMHgwLCB0dGwg
NjQsIGlkIDU2MDk4LCBvZmZzZXQgMCwgZmxhZ3MgW0RGXSwgcHJvdG8gVENQICg2KSwgbGVu
Z3RoIDE2NCkKICAgIDE5Mi4xNjguY2xpZW50LjkzNyA+IDE5Mi4xNjguc2VydmVyLjIwNDk6
IEZsYWdzIFtQLl0sIGNrc3VtIDB4YTNiYSAoY29ycmVjdCksIHNlcSA4NzUxMzQxNjo4NzUx
MzUyOCwgYWNrIDEwOTc3Nywgd2luIDU2MzIsIG9wdGlvbnMgW25vcCxub3AsVFMgdmFsIDEw
NDg2ODQgZWNyIDUwMDEwMzg3M10sIGxlbmd0aCAxMTI6IE5GUyByZXF1ZXN0IHhpZCA0ODU1
MTEwNjYgMTA4IGdldGF0dHIgZmggVW5rbm93bi9CMzBCRjQxRURFNDhFQzVDMEEwMAowOTAw
MDAwMDAwMDA2RDdEODIwMTAwMDAwMDAwMDAwMDAwMDAKMTM6MjE6MTUuNDYyNTM1IElQICh0
b3MgMHgwLCB0dGwgNjQsIGlkIDAsIG9mZnNldCAwLCBmbGFncyBbREZdLCBwcm90byBUQ1Ag
KDYpLCBsZW5ndGggMTY4LCBiYWQgY2tzdW0gMCAoLT5kM2Y2KSEpCiAgICAxOTIuMTY4LnNl
cnZlci4yMDQ5ID4gMTkyLjE2OC5jbGllbnQuOTM3OiBGbGFncyBbUC5dLCBja3N1bSAweDY2
ZjQgKGluY29ycmVjdCAtPiAweGNjYjIpLCBzZXEgMTA5Nzc3OjEwOTg5MywgYWNrIDg3NTEz
NTI4LCB3aW4gMjkxMjcsIG9wdGlvbnMgW25vcCxub3AsVFMgdmFsIDUwMDEwMzg3OCBlY3Ig
MTA0ODY4NF0sIGxlbmd0aCAxMTY6IE5GUyByZXBseSB4aWQgNDg1NTExMDY2IHJlcGx5IG9r
IDExMiBnZXRhdHRyIFJFRyA2NDQgaWRzIDIKODc2Ny8wIHN6IDQwOTYwMDAwMDAKMTM6MjE6
MTUuNDYyNjY5IElQICh0b3MgMHgwLCB0dGwgNjQsIGlkIDU2MDk5LCBvZmZzZXQgMCwgZmxh
Z3MgW0RGXSwgcHJvdG8gVENQICg2KSwgbGVuZ3RoIDE2NCkKICAgIDE5Mi4xNjguY2xpZW50
LjkzNyA+IDE5Mi4xNjguc2VydmVyLjIwNDk6IEZsYWdzIFtQLl0sIGNrc3VtIDB4ODlkMSAo
Y29ycmVjdCksIHNlcSA4NzUxMzUyODo4NzUxMzY0MCwgYWNrIDEwOTg5Mywgd2luIDU2MzIs
IG9wdGlvbnMgW25vcCxub3AsVFMgdmFsIDEwNDg2ODQgZWNyIDUwMDEwMzg3OF0sIGxlbmd0
aCAxMTI6IE5GUyByZXF1ZXN0IHhpZCA1MDIyODgyODIgMTA4IGdldGF0dHIgZmggVW5rbm93
bi9CMzBCRjQxRURFNDhFQzVDMEEwMAowQTAwMDAwMDAwMDA4NDdEODIwMTAwMDAwMDAwMDAw
MDAwMDAKMTM6MjE6MTUuNDYyNzEzIElQICh0b3MgMHgwLCB0dGwgNjQsIGlkIDAsIG9mZnNl
dCAwLCBmbGFncyBbREZdLCBwcm90byBUQ1AgKDYpLCBsZW5ndGggMTY4LCBiYWQgY2tzdW0g
MCAoLT5kM2Y2KSEpCiAgICAxOTIuMTY4LnNlcnZlci4yMDQ5ID4gMTkyLjE2OC5jbGllbnQu
OTM3OiBGbGFncyBbUC5dLCBja3N1bSAweDY2ZjQgKGluY29ycmVjdCAtPiAweDEyZmQpLCBz
ZXEgMTA5ODkzOjExMDAwOSwgYWNrIDg3NTEzNjQwLCB3aW4gMjkxMjcsIG9wdGlvbnMgW25v
cCxub3AsVFMgdmFsIDUwMDEwMzg3OCBlY3IgMTA0ODY4NF0sIGxlbmd0aCAxMTY6IE5GUyBy
ZXBseSB4aWQgNTAyMjg4MjgyIHJlcGx5IG9rIDExMiBnZXRhdHRyIFJFRyA2NDQgaWRzIDIK
ODc2Ny8wIHN6IDMwNDUKMTM6MjE6MTUuNDYyODQ1IElQICh0b3MgMHgwLCB0dGwgNjQsIGlk
IDU2MTAwLCBvZmZzZXQgMCwgZmxhZ3MgW0RGXSwgcHJvdG8gVENQICg2KSwgbGVuZ3RoIDE4
OCkKICAgIDE5Mi4xNjguY2xpZW50LjkzNyA+IDE5Mi4xNjguc2VydmVyLjIwNDk6IEZsYWdz
IFtQLl0sIGNrc3VtIDB4ZTJkYiAoY29ycmVjdCksIHNlcSA4NzUxMzY0MDo4NzUxMzc3Niwg
YWNrIDExMDAwOSwgd2luIDU2MzIsIG9wdGlvbnMgW25vcCxub3AsVFMgdmFsIDEwNDg2ODQg
ZWNyIDUwMDEwMzg3OF0sIGxlbmd0aCAxMzY6IE5GUyByZXF1ZXN0IHhpZCA1MTkwNjU0OTgg
MTMyIHJlYWRkaXJwbHVzIGZoIFVua25vd24vQjMwQkY0MUVERTQ4RUM1QwowQTAwMDgwMDAw
MDAwMDAwNTQ3RDgyMDEwMDAwMDAwMDAwMDAwMDAwIDQwOTYgYnl0ZXMgQCAwIG1heCAzMjc2
OCB2ZXJmIDAwMDAwMDAwNTc0NjdhMDAKMTM6MjE6MTUuNDYyOTA3IElQICh0b3MgMHgwLCB0
dGwgNjQsIGlkIDAsIG9mZnNldCAwLCBmbGFncyBbREZdLCBwcm90byBUQ1AgKDYpLCBsZW5n
dGggOTY4LCBiYWQgY2tzdW0gMCAoLT5kMGQ2KSEpCiAgICAxOTIuMTY4LnNlcnZlci4yMDQ5
ID4gMTkyLjE2OC5jbGllbnQuOTM3OiBGbGFncyBbUC5dLCBja3N1bSAweDZhMTQgKGluY29y
cmVjdCAtPiAweGM4M2IpLCBzZXEgMTEwMDA5OjExMDkyNSwgYWNrIDg3NTEzNzc2LCB3aW4g
MjkxMjcsIG9wdGlvbnMgW25vcCxub3AsVFMgdmFsIDUwMDEwMzg3OCBlY3IgMTA0ODY4NF0s
IGxlbmd0aCA5MTY6IE5GUyByZXBseSB4aWQgNTE5MDY1NDk4IHJlcGx5IG9rIDkxMiByZWFk
ZGlycGx1cyBQT1NUOiBESVIKIDcwMCBpZHMgMjg3NjcvMCBzeiA1IG5saW5rIDIgcmRldiAw
LzkgZnNpZCAxZWY0MGJiMyBmaWxlaWQgOCBhL20vY3RpbWUgMTUyMjg2MjQ3NS40NjI4NzIw
MDAgMTUyMjg2MjM1OS4xMDIxODEwMDAgMTUyMjg2MjM1OS4xMDIxODEwMDAgdmVyZiAwMDAw
MDAwMDU3NDY3YTAwCjEzOjIxOjE1LjQ5OTYyNSBJUCAodG9zIDB4MCwgdHRsIDY0LCBpZCA1
NjEwMSwgb2Zmc2V0IDAsIGZsYWdzIFtERl0sIHByb3RvIFRDUCAoNiksIGxlbmd0aCA1MikK
ICAgIDE5Mi4xNjguY2xpZW50LjkzNyA+IDE5Mi4xNjguc2VydmVyLjIwNDk6IEZsYWdzIFsu
XSwgY2tzdW0gMHhlMjQyIChjb3JyZWN0KSwgc2VxIDg3NTEzNzc2LCBhY2sgMTEwOTI1LCB3
aW4gNTYzMiwgb3B0aW9ucyBbbm9wLG5vcCxUUyB2YWwgMTA0ODY5NCBlY3IgNTAwMTAzODc4
XSwgbGVuZ3RoIDAKMTM6MjE6MTcuNDc2NDE5IElQICh0b3MgMHgwLCB0dGwgNjQsIGlkIDU2
MTAyLCBvZmZzZXQgMCwgZmxhZ3MgW0RGXSwgcHJvdG8gVENQICg2KSwgbGVuZ3RoIDkwMDAp
CiAgICAxOTIuMTY4LmNsaWVudC45MzcgPiAxOTIuMTY4LnNlcnZlci4yMDQ5OiBGbGFncyBb
Ll0sIGNrc3VtIDB4ZDZmMSAoY29ycmVjdCksIHNlcSA4NzUxMzc3Njo4NzUyMjcyNCwgYWNr
IDExMDkyNSwgd2luIDU2MzIsIG9wdGlvbnMgW25vcCxub3AsVFMgdmFsIDEwNDkxODggZWNy
IDUwMDEwMzg3OF0sIGxlbmd0aCA4OTQ4OiBORlMgcmVxdWVzdCB4aWQgNTM1ODQyNzE0IDg5
NDQgd3JpdGUgZmggVW5rbm93bi9CMzBCRjQxRURFNDhFQzVDMEEwMDBCMDAwMDAwMDAwMDg0
N0Q4MjAxMDAwMDAwMDAwMDAwMDAwMCAxMzEwNzIgKDEzMTA3MikgYnl0ZXMgQCA0MTYwMjI1
MjggPHVuc3RhYmxlPgoxMzoyMToxNy40NzY0MjkgSVAgKHRvcyAweDAsIHR0bCA2NCwgaWQg
NTYxMDMsIG9mZnNldCAwLCBmbGFncyBbREZdLCBwcm90byBUQ1AgKDYpLCBsZW5ndGggOTAw
MCkKICAgIDE5Mi4xNjguY2xpZW50LjkzNyA+IDE5Mi4xNjguc2VydmVyLjIwNDk6IEZsYWdz
IFsuXSwgY2tzdW0gMHgwN2U4IChjb3JyZWN0KSwgc2VxIDg3NTIyNzI0Ojg3NTMxNjcyLCBh
Y2sgMTEwOTI1LCB3aW4gNTYzMiwgb3B0aW9ucyBbbm9wLG5vcCxUUyB2YWwgMTA0OTE4OCBl
Y3IgNTAwMTAzODc4XSwgbGVuZ3RoIDg5NDgKMTM6MjE6MTcuNDc2NDQ5IElQICh0b3MgMHgw
LCB0dGwgNjQsIGlkIDAsIG9mZnNldCAwLCBmbGFncyBbREZdLCBwcm90byBUQ1AgKDYpLCBs
ZW5ndGggNTIsIGJhZCBja3N1bSAwICgtPmQ0NmEpISkKICAgIDE5Mi4xNjguc2VydmVyLjIw
NDkgPiAxOTIuMTY4LmNsaWVudC45Mzc6IEZsYWdzIFsuXSwgY2tzdW0gMHg2NjgwIChpbmNv
cnJlY3QgLT4gMHgzN2RmKSwgc2VxIDExMDkyNSwgYWNrIDg3NTMxNjcyLCB3aW4gMjg4NDcs
IG9wdGlvbnMgW25vcCxub3AsVFMgdmFsIDUwMDEwNTg5MiBlY3IgMTA0OTE4OF0sIGxlbmd0
aCAwCjEzOjIxOjE3LjQ3NjQ1MSBJUCAodG9zIDB4MCwgdHRsIDY0LCBpZCA1NjEwNCwgb2Zm
c2V0IDAsIGZsYWdzIFtERl0sIHByb3RvIFRDUCAoNiksIGxlbmd0aCA5MDAwKQogICAgMTky
LjE2OC5jbGllbnQuOTM3ID4gMTkyLjE2OC5zZXJ2ZXIuMjA0OTogRmxhZ3MgWy5dLCBja3N1
bSAweDlhMGMgKGNvcnJlY3QpLCBzZXEgODc1MzE2NzI6ODc1NDA2MjAsIGFjayAxMTA5MjUs
IHdpbiA1NjMyLCBvcHRpb25zIFtub3Asbm9wLFRTIHZhbCAxMDQ5MTg4IGVjciA1MDAxMDM4
NzhdLCBsZW5ndGggODk0OAoxMzoyMToxNy40NzY0NTMgSVAgKHRvcyAweDAsIHR0bCA2NCwg
aWQgNTYxMDUsIG9mZnNldCAwLCBmbGFncyBbREZdLCBwcm90byBUQ1AgKDYpLCBsZW5ndGgg
OTAwMCkKICAgIDE5Mi4xNjguY2xpZW50LjkzNyA+IDE5Mi4xNjguc2VydmVyLjIwNDk6IEZs
YWdzIFsuXSwgY2tzdW0gMHg5Mjg1IChjb3JyZWN0KSwgc2VxIDg3NTQwNjIwOjg3NTQ5NTY4
LCBhY2sgMTEwOTI1LCB3aW4gNTYzMiwgb3B0aW9ucyBbbm9wLG5vcCxUUyB2YWwgMTA0OTE4
OCBlY3IgNTAwMTAzODc4XSwgbGVuZ3RoIDg5NDgKMTM6MjE6MTcuNDc2NDU1IElQICh0b3Mg
MHgwLCB0dGwgNjQsIGlkIDAsIG9mZnNldCAwLCBmbGFncyBbREZdLCBwcm90byBUQ1AgKDYp
LCBsZW5ndGggNTIsIGJhZCBja3N1bSAwICgtPmQ0NmEpISkKICAgIDE5Mi4xNjguc2VydmVy
LjIwNDkgPiAxOTIuMTY4LmNsaWVudC45Mzc6IEZsYWdzIFsuXSwgY2tzdW0gMHg2NjgwIChp
bmNvcnJlY3QgLT4gMHhmMzBlKSwgc2VxIDExMDkyNSwgYWNrIDg3NTQ5NTY4LCB3aW4gMjg1
NjcsIG9wdGlvbnMgW25vcCxub3AsVFMgdmFsIDUwMDEwNTg5MiBlY3IgMTA0OTE4OF0sIGxl
bmd0aCAwCjEzOjIxOjE3LjQ3NjQ1NSBJUCAodG9zIDB4MCwgdHRsIDY0LCBpZCA1NjEwNiwg
b2Zmc2V0IDAsIGZsYWdzIFtERl0sIHByb3RvIFRDUCAoNiksIGxlbmd0aCA5MDAwKQogICAg
MTkyLjE2OC5jbGllbnQuOTM3ID4gMTkyLjE2OC5zZXJ2ZXIuMjA0OTogRmxhZ3MgWy5dLCBj
a3N1bSAweDFlYmUgKGNvcnJlY3QpLCBzZXEgODc1NDk1Njg6ODc1NTg1MTYsIGFjayAxMTA5
MjUsIHdpbiA1NjMyLCBvcHRpb25zIFtub3Asbm9wLFRTIHZhbCAxMDQ5MTg4IGVjciA1MDAx
MDM4NzhdLCBsZW5ndGggODk0OAoxMzoyMToxNy40NzY0NTcgSVAgKHRvcyAweDAsIHR0bCA2
NCwgaWQgNTYxMDcsIG9mZnNldCAwLCBmbGFncyBbREZdLCBwcm90byBUQ1AgKDYpLCBsZW5n
dGggOTAwMCkKICAgIDE5Mi4xNjguY2xpZW50LjkzNyA+IDE5Mi4xNjguc2VydmVyLjIwNDk6
IEZsYWdzIFsuXSwgY2tzdW0gMHgwYjE3IChjb3JyZWN0KSwgc2VxIDg3NTU4NTE2Ojg3NTY3
NDY0LCBhY2sgMTEwOTI1LCB3aW4gNTYzMiwgb3B0aW9ucyBbbm9wLG5vcCxUUyB2YWwgMTA0
OTE4OCBlY3IgNTAwMTAzODc4XSwgbGVuZ3RoIDg5NDgKMTM6MjE6MTcuNDc2NDU5IElQICh0
b3MgMHgwLCB0dGwgNjQsIGlkIDAsIG9mZnNldCAwLCBmbGFncyBbREZdLCBwcm90byBUQ1Ag
KDYpLCBsZW5ndGggNTIsIGJhZCBja3N1bSAwICgtPmQ0NmEpISkKICAgIDE5Mi4xNjguc2Vy
dmVyLjIwNDkgPiAxOTIuMTY4LmNsaWVudC45Mzc6IEZsYWdzIFsuXSwgY2tzdW0gMHg2Njgw
IChpbmNvcnJlY3QgLT4gMHhhZTNkKSwgc2VxIDExMDkyNSwgYWNrIDg3NTY3NDY0LCB3aW4g
MjgyODgsIG9wdGlvbnMgW25vcCxub3AsVFMgdmFsIDUwMDEwNTg5MiBlY3IgMTA0OTE4OF0s
IGxlbmd0aCAwCjEzOjIxOjE3LjQ3NjQ2MCBJUCAodG9zIDB4MCwgdHRsIDY0LCBpZCA1NjEw
OCwgb2Zmc2V0IDAsIGZsYWdzIFtERl0sIHByb3RvIFRDUCAoNiksIGxlbmd0aCA5MDAwKQog
ICAgMTkyLjE2OC5jbGllbnQuOTM3ID4gMTkyLjE2OC5zZXJ2ZXIuMjA0OTogRmxhZ3MgWy5d
LCBja3N1bSAweDBlZTMgKGNvcnJlY3QpLCBzZXEgODc1Njc0NjQ6ODc1NzY0MTIsIGFjayAx
MTA5MjUsIHdpbiA1NjMyLCBvcHRpb25zIFtub3Asbm9wLFRTIHZhbCAxMDQ5MTg4IGVjciA1
MDAxMDM4NzhdLCBsZW5ndGggODk0OAoxMzoyMToxNy40NzY0NjMgSVAgKHRvcyAweDAsIHR0
bCA2NCwgaWQgNTYxMDksIG9mZnNldCAwLCBmbGFncyBbREZdLCBwcm90byBUQ1AgKDYpLCBs
ZW5ndGggOTAwMCkKICAgIDE5Mi4xNjguY2xpZW50LjkzNyA+IDE5Mi4xNjguc2VydmVyLjIw
NDk6IEZsYWdzIFsuXSwgY2tzdW0gMHgzNjU3IChjb3JyZWN0KSwgc2VxIDg3NTc2NDEyOjg3
NTg1MzYwLCBhY2sgMTEwOTI1LCB3aW4gNTYzMiwgb3B0aW9ucyBbbm9wLG5vcCxUUyB2YWwg
MTA0OTE4OCBlY3IgNTAwMTAzODc4XSwgbGVuZ3RoIDg5NDgKMTM6MjE6MTcuNDc2NDY1IElQ
ICh0b3MgMHgwLCB0dGwgNjQsIGlkIDAsIG9mZnNldCAwLCBmbGFncyBbREZdLCBwcm90byBU
Q1AgKDYpLCBsZW5ndGggNTIsIGJhZCBja3N1bSAwICgtPmQ0NmEpISkKICAgIDE5Mi4xNjgu
c2VydmVyLjIwNDkgPiAxOTIuMTY4LmNsaWVudC45Mzc6IEZsYWdzIFsuXSwgY2tzdW0gMHg2
NjgwIChpbmNvcnJlY3QgLT4gMHg2OTZkKSwgc2VxIDExMDkyNSwgYWNrIDg3NTg1MzYwLCB3
aW4gMjgwMDgsIG9wdGlvbnMgW25vcCxub3AsVFMgdmFsIDUwMDEwNTg5MiBlY3IgMTA0OTE4
OF0sIGxlbmd0aCAwCjEzOjIxOjE3LjQ3NjQ2NiBJUCAodG9zIDB4MCwgdHRsIDY0LCBpZCA1
NjExMCwgb2Zmc2V0IDAsIGZsYWdzIFtERl0sIHByb3RvIFRDUCAoNiksIGxlbmd0aCA5MDAw
KQogICAgMTkyLjE2OC5jbGllbnQuOTM3ID4gMTkyLjE2OC5zZXJ2ZXIuMjA0OTogRmxhZ3Mg
Wy5dLCBja3N1bSAweDI1MzkgKGNvcnJlY3QpLCBzZXEgODc1ODUzNjA6ODc1OTQzMDgsIGFj
ayAxMTA5MjUsIHdpbiA1NjMyLCBvcHRpb25zIFtub3Asbm9wLFRTIHZhbCAxMDQ5MTg4IGVj
ciA1MDAxMDM4NzhdLCBsZW5ndGggODk0OAoxMzoyMToxNy40NzY0ODMgSVAgKHRvcyAweDAs
IHR0bCA2NCwgaWQgNTYxMTEsIG9mZnNldCAwLCBmbGFncyBbREZdLCBwcm90byBUQ1AgKDYp
LCBsZW5ndGggOTAwMCkKICAgIDE5Mi4xNjguY2xpZW50LjkzNyA+IDE5Mi4xNjguc2VydmVy
LjIwNDk6IEZsYWdzIFsuXSwgY2tzdW0gMHgzMzk5IChjb3JyZWN0KSwgc2VxIDg3NTk0MzA4
Ojg3NjAzMjU2LCBhY2sgMTEwOTI1LCB3aW4gNTYzMiwgb3B0aW9ucyBbbm9wLG5vcCxUUyB2
YWwgMTA0OTE4OCBlY3IgNTAwMTAzODc4XSwgbGVuZ3RoIDg5NDgKMTM6MjE6MTcuNDc2NDg1
IElQICh0b3MgMHgwLCB0dGwgNjQsIGlkIDAsIG9mZnNldCAwLCBmbGFncyBbREZdLCBwcm90
byBUQ1AgKDYpLCBsZW5ndGggNTIsIGJhZCBja3N1bSAwICgtPmQ0NmEpISkKICAgIDE5Mi4x
Njguc2VydmVyLjIwNDkgPiAxOTIuMTY4LmNsaWVudC45Mzc6IEZsYWdzIFsuXSwgY2tzdW0g
MHg2NjgwIChpbmNvcnJlY3QgLT4gMHgyMDNlKSwgc2VxIDExMDkyNSwgYWNrIDg3NjAzMjU2
LCB3aW4gMjg4NDcsIG9wdGlvbnMgW25vcCxub3AsVFMgdmFsIDUwMDEwNTg5MiBlY3IgMTA0
OTE4OF0sIGxlbmd0aCAwCjEzOjIxOjE3LjQ3NjU2NyBJUCAodG9zIDB4MCwgdHRsIDY0LCBp
ZCA1NjExMiwgb2Zmc2V0IDAsIGZsYWdzIFtERl0sIHByb3RvIFRDUCAoNiksIGxlbmd0aCA5
MDAwKQogICAgMTkyLjE2OC5jbGllbnQuOTM3ID4gMTkyLjE2OC5zZXJ2ZXIuMjA0OTogRmxh
Z3MgWy5dLCBja3N1bSAweGY3OTkgKGNvcnJlY3QpLCBzZXEgODc2MDMyNTY6ODc2MTIyMDQs
IGFjayAxMTA5MjUsIHdpbiA1NjMyLCBvcHRpb25zIFtub3Asbm9wLFRTIHZhbCAxMDQ5MTg4
IGVjciA1MDAxMDU4OTJdLCBsZW5ndGggODk0OAoxMzoyMToxNy40NzY1NzEgSVAgKHRvcyAw
eDAsIHR0bCA2NCwgaWQgNTYxMTMsIG9mZnNldCAwLCBmbGFncyBbREZdLCBwcm90byBUQ1Ag
KDYpLCBsZW5ndGggOTAwMCkKICAgIDE5Mi4xNjguY2xpZW50LjkzNyA+IDE5Mi4xNjguc2Vy
dmVyLjIwNDk6IEZsYWdzIFsuXSwgY2tzdW0gMHg2YTEyIChjb3JyZWN0KSwgc2VxIDg3NjEy
MjA0Ojg3NjIxMTUyLCBhY2sgMTEwOTI1LCB3aW4gNTYzMiwgb3B0aW9ucyBbbm9wLG5vcCxU
UyB2YWwgMTA0OTE4OCBlY3IgNTAwMTA1ODkyXSwgbGVuZ3RoIDg5NDgKMTM6MjE6MTcuNDc2
NTc0IElQICh0b3MgMHgwLCB0dGwgNjQsIGlkIDAsIG9mZnNldCAwLCBmbGFncyBbREZdLCBw
cm90byBUQ1AgKDYpLCBsZW5ndGggNTIsIGJhZCBja3N1bSAwICgtPmQ0NmEpISkKICAgIDE5
Mi4xNjguc2VydmVyLjIwNDkgPiAxOTIuMTY4LmNsaWVudC45Mzc6IEZsYWdzIFsuXSwgY2tz
dW0gMHg2NjgwIChpbmNvcnJlY3QgLT4gMHhkYjZkKSwgc2VxIDExMDkyNSwgYWNrIDg3NjIx
MTUyLCB3aW4gMjg1NjcsIG9wdGlvbnMgW25vcCxub3AsVFMgdmFsIDUwMDEwNTg5MiBlY3Ig
MTA0OTE4OF0sIGxlbmd0aCAwCjEzOjIxOjE3LjQ3NjU3NSBJUCAodG9zIDB4MCwgdHRsIDY0
LCBpZCA1NjExNCwgb2Zmc2V0IDAsIGZsYWdzIFtERl0sIHByb3RvIFRDUCAoNiksIGxlbmd0
aCA5MDAwKQogICAgMTkyLjE2OC5jbGllbnQuOTM3ID4gMTkyLjE2OC5zZXJ2ZXIuMjA0OTog
RmxhZ3MgWy5dLCBja3N1bSAweDFmM2MgKGNvcnJlY3QpLCBzZXEgODc2MjExNTI6ODc2MzAx
MDAsIGFjayAxMTA5MjUsIHdpbiA1NjMyLCBvcHRpb25zIFtub3Asbm9wLFRTIHZhbCAxMDQ5
MTg4IGVjciA1MDAxMDU4OTJdLCBsZW5ndGggODk0OAoxMzoyMToxNy40NzY1NzggSVAgKHRv
cyAweDAsIHR0bCA2NCwgaWQgNTYxMTUsIG9mZnNldCAwLCBmbGFncyBbREZdLCBwcm90byBU
Q1AgKDYpLCBsZW5ndGggOTAwMCkKICAgIDE5Mi4xNjguY2xpZW50LjkzNyA+IDE5Mi4xNjgu
c2VydmVyLjIwNDk6IEZsYWdzIFsuXSwgY2tzdW0gMHhhMThjIChjb3JyZWN0KSwgc2VxIDg3
NjMwMTAwOjg3NjM5MDQ4LCBhY2sgMTEwOTI1LCB3aW4gNTYzMiwgb3B0aW9ucyBbbm9wLG5v
cCxUUyB2YWwgMTA0OTE4OCBlY3IgNTAwMTA1ODkyXSwgbGVuZ3RoIDg5NDgKMTM6MjE6MTcu
NDc2NTgwIElQICh0b3MgMHgwLCB0dGwgNjQsIGlkIDAsIG9mZnNldCAwLCBmbGFncyBbREZd
LCBwcm90byBUQ1AgKDYpLCBsZW5ndGggNTIsIGJhZCBja3N1bSAwICgtPmQ0NmEpISkK
--------------0092DAC1F1203FC0CDCBEF55--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?369fab06-6213-ba87-cc66-c9829e8a76a0>