Date: Tue, 14 Dec 2010 19:20:13 -0500 From: Keith Potter <kpotter@cap-press.com> To: <freebsd-stable@freebsd.org>, <kpotter@cap-press.com> Subject: 8.2-PRERELEASE, ZFS Slows Machine to a Crawl Message-ID: <4596b291f711df77f624773afad3811d@cap-press.com>
next in thread | raw e-mail | index | archive | help
--=_97b141643bab6cbf973d27e216a96e96
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset=UTF-8;
format=flowed
Greetings! We are setting up a new server using 8.2-PRERELEASE (as of
12/12/10) and we are encountering performance issues with ZFS. The
system boots from a ZFS volume consisting of two Western Digital 2TB
RE4 drives in a ZFS mirror. The rest of the system consists of an
Intel-based Tyan motherboard, 2 x Xeon 5520, and 12GB ECC ram. I have
attached the dmesg output to this email as a text file for more
hardware details.
The performance degradation occurs when copying large amounts of data
to or from the ZFS volume. It happens both when copying from the
network onto the volume, and when copying from the volume back onto
itself. When copying a large (22GB) file, the disk throughput
according to `zpool iostat -v` is only about 30-35 MB/s, but yet the
system is extremely slow interactively. SSH-ing into the machine takes
a good 5-10 seconds just to get a password prompt, and running simple
commands like `ls` and `top` takes several seconds to respond.
Compression is enabled for some volumes in the pool, but not for any of
the volumes that have been copied to or from. CPU load stays very low
(<5%) during all of this.
We have tried tuning some of the ZFS paramters. The default for
vfs.zfs.arc_max was about 11GB, but we have tried setting it as low as
400MB with no change. vfs.numvnodes has typically been below 10000
during all of this, but even such, we have toyed with the
kern.maxvnodes parameter (ranging from 10000 to 500000) with no effect.
The only thing that had any real effect was setting
vfs.zfs.txg.write_limit_override to 500000000 or so, which helped
slightly, but the system was still very slow to respond.
When not under IO load the system is very fast, running a buildworld
in 13 minutes. Unfortunately, I am not sure whether the issue existed
under 8.1-RELEASE because we did not copy any large files before it was
updated. I've attached the outputs of `zpool status` and `zpool iostat
-v` below:
pool: zroot
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror ONLINE 0 0 0
gpt/disk0 ONLINE 0 0 0
gpt/disk1 ONLINE 0 0 0
errors: No known data errors
pool used avail read write read write
------------- ----- ----- ----- ----- ----- -----
zroot 240G 1.58T 0 1 6.40K 3.18K
mirror 240G 1.58T 0 1 6.40K 3.18K
gpt/disk0 - - 0 0 5.64K 3.21K
gpt/disk1 - - 0 0 4.75K 3.21K
------------- ----- ----- ----- ----- ----- -----
I greatly appreciate any input!
--=_97b141643bab6cbf973d27e216a96e96
Content-Transfer-Encoding: base64
Content-Type: text/plain; charset=UTF-8;
name=dmesg.txt
Content-Disposition: attachment;
filename=dmesg.txt
Y2FtOiB1c2luZyBtaW5pbXVtIHNjc2lfZGVsYXkgKDEwMG1zKQ0KQ29weXJpZ2h0IChjKSAxOTky
LTIwMTAgVGhlIEZyZWVCU0QgUHJvamVjdC4NCkNvcHlyaWdodCAoYykgMTk3OSwgMTk4MCwgMTk4
MywgMTk4NiwgMTk4OCwgMTk4OSwgMTk5MSwgMTk5MiwgMTk5MywgMTk5NA0KICAgICAgICBUaGUg
UmVnZW50cyBvZiB0aGUgVW5pdmVyc2l0eSBvZiBDYWxpZm9ybmlhLiBBbGwgcmlnaHRzIHJlc2Vy
dmVkLg0KRnJlZUJTRCBpcyBhIHJlZ2lzdGVyZWQgdHJhZGVtYXJrIG9mIFRoZSBGcmVlQlNEIEZv
dW5kYXRpb24uDQpGcmVlQlNEIDguMi1QUkVSRUxFQVNFICMwOiBTdW4gRGVjIDEyIDE2OjA2OjQ4
IEVTVCAyMDEwDQogICAgcm9vdEBibGFoOi91c3Ivb2JqL3Vzci9zcmMvc3lzL1hFT04gYW1kNjQN
ClRpbWVjb3VudGVyICJpODI1NCIgZnJlcXVlbmN5IDExOTMxODIgSHogcXVhbGl0eSAwDQpDUFU6
IEludGVsKFIpIFhlb24oUikgQ1BVICAgICAgICAgICBFNTUyMCAgQCAyLjI3R0h6ICgyMjY2Ljc2
LU1IeiBLOC1jbGFzcyBDUFUpDQogIE9yaWdpbiA9ICJHZW51aW5lSW50ZWwiICBJZCA9IDB4MTA2
YTUgIEZhbWlseSA9IDYgIE1vZGVsID0gMWEgIFN0ZXBwaW5nID0gNQ0KICBGZWF0dXJlcz0weGJm
ZWJmYmZmPEZQVSxWTUUsREUsUFNFLFRTQyxNU1IsUEFFLE1DRSxDWDgsQVBJQyxTRVAsTVRSUixQ
R0UsTUNBLENNT1YsUEFULFBTRTM2LENMRkxVU0gsRFRTLEFDUEksTU1YLEZYU1IsU1NFLFNTRTIs
U1MsSFRULFRNLFBCRT4NCiAgRmVhdHVyZXMyPTB4OWNlM2JkPFNTRTMsRFRFUzY0LE1PTixEU19D
UEwsVk1YLEVTVCxUTTIsU1NTRTMsQ1gxNix4VFBSLFBEQ00sRENBLFNTRTQuMSxTU0U0LjIsUE9Q
Q05UPg0KICBBTUQgRmVhdHVyZXM9MHgyODEwMDgwMDxTWVNDQUxMLE5YLFJEVFNDUCxMTT4NCiAg
QU1EIEZlYXR1cmVzMj0weDE8TEFIRj4NCiAgVFNDOiBQLXN0YXRlIGludmFyaWFudA0KcmVhbCBt
ZW1vcnkgID0gMTI4ODQ5MDE4ODggKDEyMjg4IE1CKQ0KYXZhaWwgbWVtb3J5ID0gMTIzNzAxNjk4
NTYgKDExNzk3IE1CKQ0KQUNQSSBBUElDIFRhYmxlOiA8MDMxNzEwIEFQSUMxNjE3Pg0KRnJlZUJT
RC9TTVA6IE11bHRpcHJvY2Vzc29yIFN5c3RlbSBEZXRlY3RlZDogMTYgQ1BVcw0KRnJlZUJTRC9T
TVA6IDIgcGFja2FnZShzKSB4IDQgY29yZShzKSB4IDIgU01UIHRocmVhZHMNCiBjcHUwIChCU1Ap
OiBBUElDIElEOiAgMA0KIGNwdTEgKEFQKTogQVBJQyBJRDogIDENCiBjcHUyIChBUCk6IEFQSUMg
SUQ6ICAyDQogY3B1MyAoQVApOiBBUElDIElEOiAgMw0KIGNwdTQgKEFQKTogQVBJQyBJRDogIDQN
CiBjcHU1IChBUCk6IEFQSUMgSUQ6ICA1DQogY3B1NiAoQVApOiBBUElDIElEOiAgNg0KIGNwdTcg
KEFQKTogQVBJQyBJRDogIDcNCiBjcHU4IChBUCk6IEFQSUMgSUQ6IDE2DQogY3B1OSAoQVApOiBB
UElDIElEOiAxNw0KIGNwdTEwIChBUCk6IEFQSUMgSUQ6IDE4DQogY3B1MTEgKEFQKTogQVBJQyBJ
RDogMTkNCiBjcHUxMiAoQVApOiBBUElDIElEOiAyMA0KIGNwdTEzIChBUCk6IEFQSUMgSUQ6IDIx
DQogY3B1MTQgKEFQKTogQVBJQyBJRDogMjINCiBjcHUxNSAoQVApOiBBUElDIElEOiAyMw0KaW9h
cGljMCA8VmVyc2lvbiAyLjA+IGlycXMgMC0yMyBvbiBtb3RoZXJib2FyZA0KaW9hcGljMSA8VmVy
c2lvbiAyLjA+IGlycXMgMjQtNDcgb24gbW90aGVyYm9hcmQNCmtiZDEgYXQga2JkbXV4MA0KYWNw
aTA6IDwwMzE3MTAgWFNEVDE2MTc+IG9uIG1vdGhlcmJvYXJkDQphY3BpMDogW0lUSFJFQURdDQph
Y3BpMDogUG93ZXIgQnV0dG9uIChmaXhlZCkNCmFjcGkwOiByZXNlcnZhdGlvbiBvZiAwLCBhMDAw
MCAoMykgZmFpbGVkDQphY3BpMDogcmVzZXJ2YXRpb24gb2YgMTAwMDAwLCBiZmYwMDAwMCAoMykg
ZmFpbGVkDQpUaW1lY291bnRlciAiQUNQSS1mYXN0IiBmcmVxdWVuY3kgMzU3OTU0NSBIeiBxdWFs
aXR5IDEwMDANCmFjcGlfdGltZXIwOiA8MjQtYml0IHRpbWVyIGF0IDMuNTc5NTQ1TUh6PiBwb3J0
IDB4ODA4LTB4ODBiIG9uIGFjcGkwDQpjcHUwOiA8QUNQSSBDUFU+IG9uIGFjcGkwDQpBQ1BJIFdh
cm5pbmc6IEluY29ycmVjdCBjaGVja3N1bSBpbiB0YWJsZSBbT0VNQl0gLSAweEFELCBzaG91bGQg
YmUgMHhBQSAoMjAxMDEwMTMvdGJ1dGlscy0zNTQpDQpjcHUxOiA8QUNQSSBDUFU+IG9uIGFjcGkw
DQpjcHUyOiA8QUNQSSBDUFU+IG9uIGFjcGkwDQpjcHUzOiA8QUNQSSBDUFU+IG9uIGFjcGkwDQpj
cHU0OiA8QUNQSSBDUFU+IG9uIGFjcGkwDQpjcHU1OiA8QUNQSSBDUFU+IG9uIGFjcGkwDQpjcHU2
OiA8QUNQSSBDUFU+IG9uIGFjcGkwDQpjcHU3OiA8QUNQSSBDUFU+IG9uIGFjcGkwDQpjcHU4OiA8
QUNQSSBDUFU+IG9uIGFjcGkwDQpjcHU5OiA8QUNQSSBDUFU+IG9uIGFjcGkwDQpjcHUxMDogPEFD
UEkgQ1BVPiBvbiBhY3BpMA0KY3B1MTE6IDxBQ1BJIENQVT4gb24gYWNwaTANCmNwdTEyOiA8QUNQ
SSBDUFU+IG9uIGFjcGkwDQpjcHUxMzogPEFDUEkgQ1BVPiBvbiBhY3BpMA0KY3B1MTQ6IDxBQ1BJ
IENQVT4gb24gYWNwaTANCmNwdTE1OiA8QUNQSSBDUFU+IG9uIGFjcGkwDQpwY2liMDogPEFDUEkg
SG9zdC1QQ0kgYnJpZGdlPiBwb3J0IDB4Y2Y4LTB4Y2ZmIG9uIGFjcGkwDQpwY2kwOiA8QUNQSSBQ
Q0kgYnVzPiBvbiBwY2liMA0KcGNpYjE6IDxBQ1BJIFBDSS1QQ0kgYnJpZGdlPiBhdCBkZXZpY2Ug
MS4wIG9uIHBjaTANCnBjaTEwOiA8QUNQSSBQQ0kgYnVzPiBvbiBwY2liMQ0KcGNpYjI6IDxBQ1BJ
IFBDSS1QQ0kgYnJpZGdlPiBhdCBkZXZpY2UgMy4wIG9uIHBjaTANCnBjaTk6IDxBQ1BJIFBDSSBi
dXM+IG9uIHBjaWIyDQpwY2liMzogPEFDUEkgUENJLVBDSSBicmlkZ2U+IGF0IGRldmljZSA3LjAg
b24gcGNpMA0KcGNpODogPEFDUEkgUENJIGJ1cz4gb24gcGNpYjMNCnBjaWI0OiA8UENJLVBDSSBi
cmlkZ2U+IGF0IGRldmljZSA4LjAgb24gcGNpMA0KcGNpNzogPFBDSSBidXM+IG9uIHBjaWI0DQpw
Y2liNTogPFBDSS1QQ0kgYnJpZGdlPiBhdCBkZXZpY2UgOS4wIG9uIHBjaTANCnBjaTY6IDxQQ0kg
YnVzPiBvbiBwY2liNQ0KcGNpYjY6IDxQQ0ktUENJIGJyaWRnZT4gYXQgZGV2aWNlIDEwLjAgb24g
cGNpMA0KcGNpNTogPFBDSSBidXM+IG9uIHBjaWI2DQpwY2kwOiA8YmFzZSBwZXJpcGhlcmFsLCBp
bnRlcnJ1cHQgY29udHJvbGxlcj4gYXQgZGV2aWNlIDIwLjAgKG5vIGRyaXZlciBhdHRhY2hlZCkN
CnBjaTA6IDxiYXNlIHBlcmlwaGVyYWwsIGludGVycnVwdCBjb250cm9sbGVyPiBhdCBkZXZpY2Ug
MjAuMSAobm8gZHJpdmVyIGF0dGFjaGVkKQ0KcGNpMDogPGJhc2UgcGVyaXBoZXJhbCwgaW50ZXJy
dXB0IGNvbnRyb2xsZXI+IGF0IGRldmljZSAyMC4yIChubyBkcml2ZXIgYXR0YWNoZWQpDQpwY2kw
OiA8YmFzZSBwZXJpcGhlcmFsLCBpbnRlcnJ1cHQgY29udHJvbGxlcj4gYXQgZGV2aWNlIDIwLjMg
KG5vIGRyaXZlciBhdHRhY2hlZCkNCnBjaTA6IDxiYXNlIHBlcmlwaGVyYWw+IGF0IGRldmljZSAy
Mi4wIChubyBkcml2ZXIgYXR0YWNoZWQpDQpwY2kwOiA8YmFzZSBwZXJpcGhlcmFsPiBhdCBkZXZp
Y2UgMjIuMSAobm8gZHJpdmVyIGF0dGFjaGVkKQ0KcGNpMDogPGJhc2UgcGVyaXBoZXJhbD4gYXQg
ZGV2aWNlIDIyLjIgKG5vIGRyaXZlciBhdHRhY2hlZCkNCnBjaTA6IDxiYXNlIHBlcmlwaGVyYWw+
IGF0IGRldmljZSAyMi4zIChubyBkcml2ZXIgYXR0YWNoZWQpDQpwY2kwOiA8YmFzZSBwZXJpcGhl
cmFsPiBhdCBkZXZpY2UgMjIuNCAobm8gZHJpdmVyIGF0dGFjaGVkKQ0KcGNpMDogPGJhc2UgcGVy
aXBoZXJhbD4gYXQgZGV2aWNlIDIyLjUgKG5vIGRyaXZlciBhdHRhY2hlZCkNCnBjaTA6IDxiYXNl
IHBlcmlwaGVyYWw+IGF0IGRldmljZSAyMi42IChubyBkcml2ZXIgYXR0YWNoZWQpDQpwY2kwOiA8
YmFzZSBwZXJpcGhlcmFsPiBhdCBkZXZpY2UgMjIuNyAobm8gZHJpdmVyIGF0dGFjaGVkKQ0KdWhj
aTA6IDxJbnRlbCA4MjgwMUpJIChJQ0gxMCkgVVNCIGNvbnRyb2xsZXIgVVNCLUQ+IHBvcnQgMHg5
NDAwLTB4OTQxZiBpcnEgMTYgYXQgZGV2aWNlIDI2LjAgb24gcGNpMA0KdWhjaTA6IFtJVEhSRUFE
XQ0KdWhjaTA6IExlZ1N1cCA9IDB4MmYwMA0KdXNidXMwOiA8SW50ZWwgODI4MDFKSSAoSUNIMTAp
IFVTQiBjb250cm9sbGVyIFVTQi1EPiBvbiB1aGNpMA0KdWhjaTE6IDxJbnRlbCA4MjgwMUpJIChJ
Q0gxMCkgVVNCIGNvbnRyb2xsZXIgVVNCLUU+IHBvcnQgMHg5NDgwLTB4OTQ5ZiBpcnEgMjEgYXQg
ZGV2aWNlIDI2LjEgb24gcGNpMA0KdWhjaTE6IFtJVEhSRUFEXQ0KdWhjaTE6IExlZ1N1cCA9IDB4
MmYwMA0KdXNidXMxOiA8SW50ZWwgODI4MDFKSSAoSUNIMTApIFVTQiBjb250cm9sbGVyIFVTQi1F
PiBvbiB1aGNpMQ0KdWhjaTI6IDxJbnRlbCA4MjgwMUpJIChJQ0gxMCkgVVNCIGNvbnRyb2xsZXIg
VVNCLUY+IHBvcnQgMHg5ODAwLTB4OTgxZiBpcnEgMTkgYXQgZGV2aWNlIDI2LjIgb24gcGNpMA0K
dWhjaTI6IFtJVEhSRUFEXQ0KdWhjaTI6IExlZ1N1cCA9IDB4MmYwMA0KdXNidXMyOiA8SW50ZWwg
ODI4MDFKSSAoSUNIMTApIFVTQiBjb250cm9sbGVyIFVTQi1GPiBvbiB1aGNpMg0KZWhjaTA6IDxJ
bnRlbCA4MjgwMUpJIChJQ0gxMCkgVVNCIDIuMCBjb250cm9sbGVyIFVTQi1CPiBtZW0gMHhmYmNm
NjAwMC0weGZiY2Y2M2ZmIGlycSAxOCBhdCBkZXZpY2UgMjYuNyBvbiBwY2kwDQplaGNpMDogW0lU
SFJFQURdDQp1c2J1czM6IEVIQ0kgdmVyc2lvbiAxLjANCnVzYnVzMzogPEludGVsIDgyODAxSkkg
KElDSDEwKSBVU0IgMi4wIGNvbnRyb2xsZXIgVVNCLUI+IG9uIGVoY2kwDQpwY2liNzogPEFDUEkg
UENJLVBDSSBicmlkZ2U+IGlycSAxNyBhdCBkZXZpY2UgMjguMCBvbiBwY2kwDQpwY2k0OiA8QUNQ
SSBQQ0kgYnVzPiBvbiBwY2liNw0KcGNpYjg6IDxBQ1BJIFBDSS1QQ0kgYnJpZGdlPiBpcnEgMTcg
YXQgZGV2aWNlIDI4LjQgb24gcGNpMA0KcGNpMzogPEFDUEkgUENJIGJ1cz4gb24gcGNpYjgNCmVt
MDogPEludGVsKFIpIFBSTy8xMDAwIE5ldHdvcmsgQ29ubmVjdGlvbiA3LjEuOD4gcG9ydCAweGVj
MDAtMHhlYzFmIG1lbSAweGZiZWUwMDAwLTB4ZmJlZmZmZmYsMHhmYmVkYzAwMC0weGZiZWRmZmZm
IGlycSAxNiBhdCBkZXZpY2UgMC4wIG9uIHBjaTMNCmVtMDogVXNpbmcgTVNJWCBpbnRlcnJ1cHRz
IHdpdGggMyB2ZWN0b3JzDQplbTA6IFtJVEhSRUFEXQ0KZW0wOiBbSVRIUkVBRF0NCmVtMDogW0lU
SFJFQURdDQplbTA6IEV0aGVybmV0IGFkZHJlc3M6IDAwOmUwOjgxOmJhOmJmOmU3DQpwY2liOTog
PEFDUEkgUENJLVBDSSBicmlkZ2U+IGlycSAxNiBhdCBkZXZpY2UgMjguNSBvbiBwY2kwDQpwY2ky
OiA8QUNQSSBQQ0kgYnVzPiBvbiBwY2liOQ0KZW0xOiA8SW50ZWwoUikgUFJPLzEwMDAgTmV0d29y
ayBDb25uZWN0aW9uIDcuMS44PiBwb3J0IDB4ZGMwMC0weGRjMWYgbWVtIDB4ZmJkZTAwMDAtMHhm
YmRmZmZmZiwweGZiZGRjMDAwLTB4ZmJkZGZmZmYgaXJxIDE3IGF0IGRldmljZSAwLjAgb24gcGNp
Mg0KZW0xOiBVc2luZyBNU0lYIGludGVycnVwdHMgd2l0aCAzIHZlY3RvcnMNCmVtMTogW0lUSFJF
QURdDQplbTE6IFtJVEhSRUFEXQ0KZW0xOiBbSVRIUkVBRF0NCmVtMTogRXRoZXJuZXQgYWRkcmVz
czogMDA6ZTA6ODE6YmE6YmY6ZTgNCnVoY2kzOiA8SW50ZWwgODI4MDFKSSAoSUNIMTApIFVTQiBj
b250cm9sbGVyIFVTQi1BPiBwb3J0IDB4OTg4MC0weDk4OWYgaXJxIDIzIGF0IGRldmljZSAyOS4w
IG9uIHBjaTANCnVoY2kzOiBbSVRIUkVBRF0NCnVoY2kzOiBMZWdTdXAgPSAweDJmMDANCnVzYnVz
NDogPEludGVsIDgyODAxSkkgKElDSDEwKSBVU0IgY29udHJvbGxlciBVU0ItQT4gb24gdWhjaTMN
CnVoY2k0OiA8SW50ZWwgODI4MDFKSSAoSUNIMTApIFVTQiBjb250cm9sbGVyIFVTQi1CPiBwb3J0
IDB4OWMwMC0weDljMWYgaXJxIDE5IGF0IGRldmljZSAyOS4xIG9uIHBjaTANCnVoY2k0OiBbSVRI
UkVBRF0NCnVoY2k0OiBMZWdTdXAgPSAweDJmMDANCnVzYnVzNTogPEludGVsIDgyODAxSkkgKElD
SDEwKSBVU0IgY29udHJvbGxlciBVU0ItQj4gb24gdWhjaTQNCnVoY2k1OiA8SW50ZWwgODI4MDFK
SSAoSUNIMTApIFVTQiBjb250cm9sbGVyIFVTQi1DPiBwb3J0IDB4YTAwMC0weGEwMWYgaXJxIDE4
IGF0IGRldmljZSAyOS4yIG9uIHBjaTANCnVoY2k1OiBbSVRIUkVBRF0NCnVoY2k1OiBMZWdTdXAg
PSAweDJmMDANCnVzYnVzNjogPEludGVsIDgyODAxSkkgKElDSDEwKSBVU0IgY29udHJvbGxlciBV
U0ItQz4gb24gdWhjaTUNCmVoY2kxOiA8SW50ZWwgODI4MDFKSSAoSUNIMTApIFVTQiAyLjAgY29u
dHJvbGxlciBVU0ItQT4gbWVtIDB4ZmJjZjgwMDAtMHhmYmNmODNmZiBpcnEgMjMgYXQgZGV2aWNl
IDI5Ljcgb24gcGNpMA0KZWhjaTE6IFtJVEhSRUFEXQ0KdXNidXM3OiBFSENJIHZlcnNpb24gMS4w
DQp1c2J1czc6IDxJbnRlbCA4MjgwMUpJIChJQ0gxMCkgVVNCIDIuMCBjb250cm9sbGVyIFVTQi1B
PiBvbiBlaGNpMQ0KcGNpYjEwOiA8QUNQSSBQQ0ktUENJIGJyaWRnZT4gYXQgZGV2aWNlIDMwLjAg
b24gcGNpMA0KcGNpMTogPEFDUEkgUENJIGJ1cz4gb24gcGNpYjEwDQp2Z2FwY2kwOiA8VkdBLWNv
bXBhdGlibGUgZGlzcGxheT4gcG9ydCAweGNjMDAtMHhjYzdmIG1lbSAweGZiMDAwMDAwLTB4ZmI3
ZmZmZmYsMHhmYWZlMDAwMC0weGZhZmZmZmZmIGlycSAxNiBhdCBkZXZpY2UgMC4wIG9uIHBjaTEN
CmlzYWIwOiA8UENJLUlTQSBicmlkZ2U+IGF0IGRldmljZSAzMS4wIG9uIHBjaTANCmlzYTA6IDxJ
U0EgYnVzPiBvbiBpc2FiMA0KYXRhcGNpMDogPEludGVsIElDSDEwIFNBVEEzMDAgY29udHJvbGxl
cj4gcG9ydCAweGFjMDAtMHhhYzA3LDB4YTg4MC0weGE4ODMsMHhhODAwLTB4YTgwNywweGE0ODAt
MHhhNDgzLDB4YTQwMC0weGE0MGYsMHhhMDgwLTB4YTA4ZiBpcnEgMTkgYXQgZGV2aWNlIDMxLjIg
b24gcGNpMA0KYXRhcGNpMDogW0lUSFJFQURdDQphdGEyOiA8QVRBIGNoYW5uZWwgMD4gb24gYXRh
cGNpMA0KYXRhMjogW0lUSFJFQURdDQphdGEzOiA8QVRBIGNoYW5uZWwgMT4gb24gYXRhcGNpMA0K
YXRhMzogW0lUSFJFQURdDQpwY2kwOiA8c2VyaWFsIGJ1cywgU01CdXM+IGF0IGRldmljZSAzMS4z
IChubyBkcml2ZXIgYXR0YWNoZWQpDQphdGFwY2kxOiA8SW50ZWwgSUNIMTAgU0FUQTMwMCBjb250
cm9sbGVyPiBwb3J0IDB4YmMwMC0weGJjMDcsMHhiODgwLTB4Yjg4MywweGI4MDAtMHhiODA3LDB4
YjQ4MC0weGI0ODMsMHhiNDAwLTB4YjQwZiwweGIwODAtMHhiMDhmIGlycSAxOSBhdCBkZXZpY2Ug
MzEuNSBvbiBwY2kwDQphdGFwY2kxOiBbSVRIUkVBRF0NCmF0YTQ6IDxBVEEgY2hhbm5lbCAwPiBv
biBhdGFwY2kxDQphdGE0OiBbSVRIUkVBRF0NCmF0YTU6IDxBVEEgY2hhbm5lbCAxPiBvbiBhdGFw
Y2kxDQphdGE1OiBbSVRIUkVBRF0NCmFjcGlfYnV0dG9uMDogPFBvd2VyIEJ1dHRvbj4gb24gYWNw
aTANCmF0cnRjMDogPEFUIHJlYWx0aW1lIGNsb2NrPiBwb3J0IDB4NzAtMHg3MSBpcnEgOCBvbiBh
Y3BpMA0KdWFydDA6IDwxNjU1MCBvciBjb21wYXRpYmxlPiBwb3J0IDB4M2Y4LTB4M2ZmIGlycSA0
IGZsYWdzIDB4MTAgb24gYWNwaTANCnVhcnQwOiBbRklMVEVSXQ0KdWFydDE6IDwxNjU1MCBvciBj
b21wYXRpYmxlPiBwb3J0IDB4MmY4LTB4MmZmIGlycSAzIG9uIGFjcGkwDQp1YXJ0MTogW0ZJTFRF
Ul0NCmFjcGlfaHBldDA6IDxIaWdoIFByZWNpc2lvbiBFdmVudCBUaW1lcj4gaW9tZW0gMHhmZWQw
MDAwMC0weGZlZDAwM2ZmIG9uIGFjcGkwDQpUaW1lY291bnRlciAiSFBFVCIgZnJlcXVlbmN5IDE0
MzE4MTgwIEh6IHF1YWxpdHkgOTAwDQpxcGkwOiA8UVBJIHN5c3RlbSBidXM+IG9uIG1vdGhlcmJv
YXJkDQpwY2liMTE6IDxRUEkgSG9zdC1QQ0kgYnJpZGdlPiBwY2lidXMgMjU1IG9uIHFwaTANCnBj
aTI1NTogPFBDSSBidXM+IG9uIHBjaWIxMQ0KcGNpYjEyOiA8UVBJIEhvc3QtUENJIGJyaWRnZT4g
cGNpYnVzIDI1NCBvbiBxcGkwDQpwY2kyNTQ6IDxQQ0kgYnVzPiBvbiBwY2liMTINCm9ybTA6IDxJ
U0EgT3B0aW9uIFJPTT4gYXQgaW9tZW0gMHhjMDAwMC0weGM3ZmZmIG9uIGlzYTANCnNjMDogPFN5
c3RlbSBjb25zb2xlPiBhdCBmbGFncyAweDEwMCBvbiBpc2EwDQpzYzA6IENHQSA8MTYgdmlydHVh
bCBjb25zb2xlcywgZmxhZ3M9MHgzMDA+DQp2Z2EwOiA8R2VuZXJpYyBJU0EgVkdBPiBhdCBwb3J0
IDB4M2QwLTB4M2RiIGlvbWVtIDB4YjgwMDAtMHhiZmZmZiBvbiBpc2EwDQphdGtiZGMwOiA8S2V5
Ym9hcmQgY29udHJvbGxlciAoaTgwNDIpPiBhdCBwb3J0IDB4NjAsMHg2NCBvbiBpc2EwDQphdGti
ZDA6IDxBVCBLZXlib2FyZD4gaXJxIDEgb24gYXRrYmRjMA0Ka2JkMCBhdCBhdGtiZDANCmF0a2Jk
MDogW0dJQU5ULUxPQ0tFRF0NCmF0a2JkMDogW0lUSFJFQURdDQpwcGMwOiBjYW5ub3QgcmVzZXJ2
ZSBJL08gcG9ydCByYW5nZQ0KZXN0MDogPEVuaGFuY2VkIFNwZWVkU3RlcCBGcmVxdWVuY3kgQ29u
dHJvbD4gb24gY3B1MA0KcDR0Y2MwOiA8Q1BVIEZyZXF1ZW5jeSBUaGVybWFsIENvbnRyb2w+IG9u
IGNwdTANCmVzdDE6IDxFbmhhbmNlZCBTcGVlZFN0ZXAgRnJlcXVlbmN5IENvbnRyb2w+IG9uIGNw
dTENCnA0dGNjMTogPENQVSBGcmVxdWVuY3kgVGhlcm1hbCBDb250cm9sPiBvbiBjcHUxDQplc3Qy
OiA8RW5oYW5jZWQgU3BlZWRTdGVwIEZyZXF1ZW5jeSBDb250cm9sPiBvbiBjcHUyDQpwNHRjYzI6
IDxDUFUgRnJlcXVlbmN5IFRoZXJtYWwgQ29udHJvbD4gb24gY3B1Mg0KZXN0MzogPEVuaGFuY2Vk
IFNwZWVkU3RlcCBGcmVxdWVuY3kgQ29udHJvbD4gb24gY3B1Mw0KcDR0Y2MzOiA8Q1BVIEZyZXF1
ZW5jeSBUaGVybWFsIENvbnRyb2w+IG9uIGNwdTMNCmVzdDQ6IDxFbmhhbmNlZCBTcGVlZFN0ZXAg
RnJlcXVlbmN5IENvbnRyb2w+IG9uIGNwdTQNCnA0dGNjNDogPENQVSBGcmVxdWVuY3kgVGhlcm1h
bCBDb250cm9sPiBvbiBjcHU0DQplc3Q1OiA8RW5oYW5jZWQgU3BlZWRTdGVwIEZyZXF1ZW5jeSBD
b250cm9sPiBvbiBjcHU1DQpwNHRjYzU6IDxDUFUgRnJlcXVlbmN5IFRoZXJtYWwgQ29udHJvbD4g
b24gY3B1NQ0KZXN0NjogPEVuaGFuY2VkIFNwZWVkU3RlcCBGcmVxdWVuY3kgQ29udHJvbD4gb24g
Y3B1Ng0KcDR0Y2M2OiA8Q1BVIEZyZXF1ZW5jeSBUaGVybWFsIENvbnRyb2w+IG9uIGNwdTYNCmVz
dDc6IDxFbmhhbmNlZCBTcGVlZFN0ZXAgRnJlcXVlbmN5IENvbnRyb2w+IG9uIGNwdTcNCnA0dGNj
NzogPENQVSBGcmVxdWVuY3kgVGhlcm1hbCBDb250cm9sPiBvbiBjcHU3DQplc3Q4OiA8RW5oYW5j
ZWQgU3BlZWRTdGVwIEZyZXF1ZW5jeSBDb250cm9sPiBvbiBjcHU4DQpwNHRjYzg6IDxDUFUgRnJl
cXVlbmN5IFRoZXJtYWwgQ29udHJvbD4gb24gY3B1OA0KZXN0OTogPEVuaGFuY2VkIFNwZWVkU3Rl
cCBGcmVxdWVuY3kgQ29udHJvbD4gb24gY3B1OQ0KcDR0Y2M5OiA8Q1BVIEZyZXF1ZW5jeSBUaGVy
bWFsIENvbnRyb2w+IG9uIGNwdTkNCmVzdDEwOiA8RW5oYW5jZWQgU3BlZWRTdGVwIEZyZXF1ZW5j
eSBDb250cm9sPiBvbiBjcHUxMA0KcDR0Y2MxMDogPENQVSBGcmVxdWVuY3kgVGhlcm1hbCBDb250
cm9sPiBvbiBjcHUxMA0KZXN0MTE6IDxFbmhhbmNlZCBTcGVlZFN0ZXAgRnJlcXVlbmN5IENvbnRy
b2w+IG9uIGNwdTExDQpwNHRjYzExOiA8Q1BVIEZyZXF1ZW5jeSBUaGVybWFsIENvbnRyb2w+IG9u
IGNwdTExDQplc3QxMjogPEVuaGFuY2VkIFNwZWVkU3RlcCBGcmVxdWVuY3kgQ29udHJvbD4gb24g
Y3B1MTINCnA0dGNjMTI6IDxDUFUgRnJlcXVlbmN5IFRoZXJtYWwgQ29udHJvbD4gb24gY3B1MTIN
CmVzdDEzOiA8RW5oYW5jZWQgU3BlZWRTdGVwIEZyZXF1ZW5jeSBDb250cm9sPiBvbiBjcHUxMw0K
cDR0Y2MxMzogPENQVSBGcmVxdWVuY3kgVGhlcm1hbCBDb250cm9sPiBvbiBjcHUxMw0KZXN0MTQ6
IDxFbmhhbmNlZCBTcGVlZFN0ZXAgRnJlcXVlbmN5IENvbnRyb2w+IG9uIGNwdTE0DQpwNHRjYzE0
OiA8Q1BVIEZyZXF1ZW5jeSBUaGVybWFsIENvbnRyb2w+IG9uIGNwdTE0DQplc3QxNTogPEVuaGFu
Y2VkIFNwZWVkU3RlcCBGcmVxdWVuY3kgQ29udHJvbD4gb24gY3B1MTUNCnA0dGNjMTU6IDxDUFUg
RnJlcXVlbmN5IFRoZXJtYWwgQ29udHJvbD4gb24gY3B1MTUNClpGUyBmaWxlc3lzdGVtIHZlcnNp
b24gNA0KWkZTIHN0b3JhZ2UgcG9vbCB2ZXJzaW9uIDE1DQpUaW1lY291bnRlcnMgdGljayBldmVy
eSAxLjAwMCBtc2VjDQp2Ym94ZHJ2OiBmQXN5bmM9MCBvZmZNaW49MHg1ZDQgb2ZmTWF4PTB4ODc4
DQp1c2J1czA6IDEyTWJwcyBGdWxsIFNwZWVkIFVTQiB2MS4wDQp1c2J1czE6IDEyTWJwcyBGdWxs
IFNwZWVkIFVTQiB2MS4wDQp1c2J1czI6IDEyTWJwcyBGdWxsIFNwZWVkIFVTQiB2MS4wDQp1c2J1
czM6IDQ4ME1icHMgSGlnaCBTcGVlZCBVU0IgdjIuMA0KdXNidXM0OiAxMk1icHMgRnVsbCBTcGVl
ZCBVU0IgdjEuMA0KdXNidXM1OiAxMk1icHMgRnVsbCBTcGVlZCBVU0IgdjEuMA0KdXNidXM2OiAx
Mk1icHMgRnVsbCBTcGVlZCBVU0IgdjEuMA0KdXNidXM3OiA0ODBNYnBzIEhpZ2ggU3BlZWQgVVNC
IHYyLjANCmFkNDogMTkwNzcyOU1CIDxXREMgV0QyMDAzRllZUy0wMlcwQjAgMDEuMDFEMDE+IGF0
IGF0YTItbWFzdGVyIFVETUExMDAgU0FUQSAzR2Ivcw0KdWdlbjAuMTogPEludGVsPiBhdCB1c2J1
czANCnVodWIwOiA8SW50ZWwgVUhDSSByb290IEhVQiwgY2xhc3MgOS8wLCByZXYgMS4wMC8xLjAw
LCBhZGRyIDE+IG9uIHVzYnVzMA0KdWdlbjEuMTogPEludGVsPiBhdCB1c2J1czENCnVodWIxOiA8
SW50ZWwgVUhDSSByb290IEhVQiwgY2xhc3MgOS8wLCByZXYgMS4wMC8xLjAwLCBhZGRyIDE+IG9u
IHVzYnVzMQ0KdWdlbjIuMTogPEludGVsPiBhdCB1c2J1czINCnVodWIyOiA8SW50ZWwgVUhDSSBy
b290IEhVQiwgY2xhc3MgOS8wLCByZXYgMS4wMC8xLjAwLCBhZGRyIDE+IG9uIHVzYnVzMg0KdWdl
bjMuMTogPEludGVsPiBhdCB1c2J1czMNCnVodWIzOiA8SW50ZWwgRUhDSSByb290IEhVQiwgY2xh
c3MgOS8wLCByZXYgMi4wMC8xLjAwLCBhZGRyIDE+IG9uIHVzYnVzMw0KdWdlbjQuMTogPEludGVs
PiBhdCB1c2J1czQNCnVodWI0OiA8SW50ZWwgVUhDSSByb290IEhVQiwgY2xhc3MgOS8wLCByZXYg
MS4wMC8xLjAwLCBhZGRyIDE+IG9uIHVzYnVzNA0KdWdlbjUuMTogPEludGVsPiBhdCB1c2J1czUN
CnVodWI1OiA8SW50ZWwgVUhDSSByb290IEhVQiwgY2xhc3MgOS8wLCByZXYgMS4wMC8xLjAwLCBh
ZGRyIDE+IG9uIHVzYnVzNQ0KdWdlbjYuMTogPEludGVsPiBhdCB1c2J1czYNCnVodWI2OiA8SW50
ZWwgVUhDSSByb290IEhVQiwgY2xhc3MgOS8wLCByZXYgMS4wMC8xLjAwLCBhZGRyIDE+IG9uIHVz
YnVzNg0KdWdlbjcuMTogPEludGVsPiBhdCB1c2J1czcNCnVodWI3OiA8SW50ZWwgRUhDSSByb290
IEhVQiwgY2xhc3MgOS8wLCByZXYgMi4wMC8xLjAwLCBhZGRyIDE+IG9uIHVzYnVzNw0KYWQ2OiAx
OTA3NzI5TUIgPFdEQyBXRDIwMDNGWVlTLTAyVzBCMCAwMS4wMUQwMT4gYXQgYXRhMy1tYXN0ZXIg
VURNQTEwMCBTQVRBIDNHYi9zDQpTTVA6IEFQIENQVSAjMSBMYXVuY2hlZCENClNNUDogQVAgQ1BV
ICMxMSBMYXVuY2hlZCENClNNUDogQVAgQ1BVICM0IExhdW5jaGVkIQ0KU01QOiBBUCBDUFUgIzEz
IExhdW5jaGVkIQ0KU01QOiBBUCBDUFUgIzMgTGF1bmNoZWQhDQpTTVA6IEFQIENQVSAjMTIgTGF1
bmNoZWQhDQpTTVA6IEFQIENQVSAjMiBMYXVuY2hlZCENClNNUDogQVAgQ1BVICMxMCBMYXVuY2hl
ZCENClNNUDogQVAgQ1BVICM2IExhdW5jaGVkIQ0KU01QOiBBUCBDUFUgIzkgTGF1bmNoZWQhDQpT
TVA6IEFQIENQVSAjNSBMYXVuY2hlZCENClNNUDogQVAgQ1BVICMxNCBMYXVuY2hlZCENClNNUDog
QVAgQ1BVICM3IExhdW5jaGVkIQ0KU01QOiBBUCBDUFUgIzggTGF1bmNoZWQhDQpTTVA6IEFQIENQ
VSAjMTUgTGF1bmNoZWQhDQpSb290IG1vdW50IHdhaXRpbmcgZm9yOiB1c2J1czcgdXNidXM2IHVz
YnVzNSB1c2J1czQgdXNidXMzIHVzYnVzMiB1c2J1czEgdXNidXMwDQp1aHViMDogMiBwb3J0cyB3
aXRoIDIgcmVtb3ZhYmxlLCBzZWxmIHBvd2VyZWQNCnVodWIxOiAyIHBvcnRzIHdpdGggMiByZW1v
dmFibGUsIHNlbGYgcG93ZXJlZA0KdWh1YjI6IDIgcG9ydHMgd2l0aCAyIHJlbW92YWJsZSwgc2Vs
ZiBwb3dlcmVkDQp1aHViNDogMiBwb3J0cyB3aXRoIDIgcmVtb3ZhYmxlLCBzZWxmIHBvd2VyZWQN
CnVodWI1OiAyIHBvcnRzIHdpdGggMiByZW1vdmFibGUsIHNlbGYgcG93ZXJlZA0KdWh1YjY6IDIg
cG9ydHMgd2l0aCAyIHJlbW92YWJsZSwgc2VsZiBwb3dlcmVkDQpSb290IG1vdW50IHdhaXRpbmcg
Zm9yOiB1c2J1czcgdXNidXMzDQpSb290IG1vdW50IHdhaXRpbmcgZm9yOiB1c2J1czcgdXNidXMz
DQp1aHViMzogNiBwb3J0cyB3aXRoIDYgcmVtb3ZhYmxlLCBzZWxmIHBvd2VyZWQNCnVodWI3OiA2
IHBvcnRzIHdpdGggNiByZW1vdmFibGUsIHNlbGYgcG93ZXJlZA0KdWdlbjcuMjogPEFtZXJpY2Fu
IE1lZ2F0cmVuZHMgSW5jLj4gYXQgdXNidXM3DQpUcnlpbmcgdG8gbW91bnQgcm9vdCBmcm9tIHpm
czp6cm9vdA0KdmJveG5ldDA6IEV0aGVybmV0IGFkZHJlc3M6IDBhOjAwOjI3OjAwOjAwOjAwDQpl
bTE6IGxpbmsgc3RhdGUgY2hhbmdlZCB0byBVUA0KDQo=
--=_97b141643bab6cbf973d27e216a96e96--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4596b291f711df77f624773afad3811d>