Date: Sun, 11 Jun 2023 08:54:21 -0700 From: Scott Gasch <scott.gasch@gmail.com> To: freebsd-questions <freebsd-questions@freebsd.org> Subject: Swap filling up, suspect kernel memory issue? Message-ID: <CABYAQkR69K=S0tCpWeKNdNR3-mvZwtr_68957_A_-35CbxBfFg@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
--00000000000012c3a805fddc9f0e
Content-Type: text/plain; charset="UTF-8"
I am running a 13.2-RELEASE GENERIC kernel and seeing a pattern where,
after about 10 days of uptime, my swap begins to fill up.
# *swapinfo -h*
Device Size Used Avail Capacity
/dev/ada0p3 48G 4.3G 44G 9%
/dev/ada1p3 48G 4.3G 44G 9%
/dev/ada2p3 48G 4.3G 44G 9%
Total 144G 13G 131G 9%
# *vmstat -h*
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr ad0 ad1 in sy cs us sy
id
1 0 45 598G 17G 19K 2 0 0 20K 2.3K 0 0 1206 48K 19K 3 1
96
I cannot find a usermode culprit; the sum of the size of process swaps is
nowhere near the amount of space reported by swapinfo:
# */usr/bin/top -w -o swap*
last pid: 88600; load averages: 1.15, 1.00, 0.88 up
11+11:21:55 08:41:54
352 processes: 3 running, 347 sleeping, 2 zombie
CPU: 5.9% user, 0.0% nice, 1.1% system, 0.0% interrupt, 92.9% idle
Mem: 7812M Active, 13G Inact, 65G Laundry, 22G Wired, 744M Buf, 17G Free
ARC: 9571M Total, 1507M MFU, 5037M MRU, 27M Anon, 92M Header, 2907M Other
4620M Compressed, 12G Uncompressed, 2.57:1 Ratio
Swap: 144G Total, 13G Used, 131G Free, 9% Inuse
PID USERNAME THR PRI NICE SIZE RES SWAP STATE C TIME WCPU
COMMAND
2097 jupyter 1 20 0 118M 10M 12M kqread 17 0:09 0.00%
python3.9
85444 770 1 20 0 179M 52K 11M kqread 21 0:00 0.00%
postgres
85441 770 1 20 0 179M 52K 11M kqread 12 0:00 0.00%
postgres
85439 770 1 20 0 179M 52K 11M kqread 19 0:00 0.00%
postgres
9886 www 1 52 0 367M 52K 10M accept 10 0:00 0.00%
php-fpm
9887 www 1 52 0 367M 52K 10M accept 0 0:00 0.00%
php-fpm
9885 www 1 52 0 367M 52K 10M accept 8 0:00 0.00%
php-fpm
9883 www 1 52 0 367M 52K 10M accept 12 0:00 0.00%
php-fpm
9881 www 1 52 0 367M 52K 10M accept 0 0:00 0.00%
php-fpm
9880 www 1 52 0 367M 52K 10M accept 3 0:00 0.00%
php-fpm
9882 www 1 52 0 367M 52K 10M accept 3 0:00 0.00%
php-fpm
9876 www 1 52 0 367M 52K 10M accept 22 0:00 0.00%
php-fpm
9875 www 1 52 0 367M 52K 10M accept 5 0:00 0.00%
php-fpm
9878 www 1 52 0 367M 52K 10M accept 7 0:00 0.00%
php-fpm
9874 www 1 52 0 367M 52K 10M accept 1 0:00 0.00%
php-fpm
9872 www 1 52 0 367M 52K 10M accept 1 0:00 0.00%
php-fpm
9873 www 1 52 0 367M 52K 10M accept 3 0:00 0.00%
php-fpm
9871 www 1 52 0 367M 52K 10M accept 13 0:00 0.00%
php-fpm
9870 www 1 52 0 367M 52K 10M accept 2 0:00 0.00%
php-fpm
57411 770 1 20 0 179M 696K 9108K kqread 10 0:01
0.00% postgres
54978 www 1 20 0 32M 852K 8336K accept 17 0:08
0.02% httpd
9639 770 1 20 0 179M 572K 8060K kqread 6 0:02
0.00% postgres
8350 770 1 20 0 176M 560K 7744K kqread 18 0:02
0.00% postgres
8349 770 1 20 0 176M 668K 7640K kqread 5 0:00
0.00% postgres
8354 770 1 20 0 177M 3248K 5280K kqread 5 0:00
0.00% postgres
8353 770 1 20 0 177M 3452K 5172K kqread 2 0:03
0.00% postgres
5949 smmsp 1 20 0 18M 964K 3324K pause 17 0:00
0.00% sendmail
1956 root 1 52 0 18M 4096B 3024K lockf 14 0:00
0.00% <saslauthd>
5984 scott 1 20 0 15M 4096B 3020K wait 20 0:00
0.00% <bash>
1968 root 1 52 0 18M 4096B 2976K accept 12 0:00
0.00% <saslauthd>
There are not "that many" processes so this is not "death by a thousand
cuts":
# *ps -aux | wc -l*
372
I have a suspicion that this is related to the wireguard kmod simply
because I run wireguard in a vnet jail and didn't observe this problem
until setting that up. But I don't have any evidence.
I've tried to mitigate this via swapoff -a. This works once but the next
day swap will be back, even fuller. I've been doing regular reboots to fix
this but would like to get to the bottom of it. If left alone, swap will
fill up and the machine will get into a "not quite hung" but unusable and
useless state.
Am I off base with my suspicion that this is kernel mode memory? Can
someone teach me how to diagnose the status of kernel mode memory heap?
Thx,
Scott
--00000000000012c3a805fddc9f0e
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: base64
PGRpdiBkaXI9Imx0ciI+PGRpdj5JIGFtIHJ1bm5pbmcgYSAxMy4yLVJFTEVBU0UgR0VORVJJQyBr
ZXJuZWwgYW5kIHNlZWluZyBhIHBhdHRlcm4gd2hlcmUsIGFmdGVyIGFib3V0IDEwIGRheXMgb2Yg
dXB0aW1lLCBteSBzd2FwIGJlZ2lucyB0byBmaWxsIHVwLsKgwqA8L2Rpdj48ZGl2Pjxicj48L2Rp
dj48ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFjZSI+I8KgPGI+c3dhcGluZm8gLWg8L2I+PC9mb250
PjwvZGl2PjxkaXY+PGZvbnQgZmFjZT0ibW9ub3NwYWNlIj5EZXZpY2UgwqAgwqAgwqAgwqAgwqAg
wqAgwqBTaXplIMKgIMKgIFVzZWQgwqAgwqBBdmFpbCBDYXBhY2l0eTxicj4vZGV2L2FkYTBwMyDC
oCDCoCDCoCDCoCDCoDQ4RyDCoCDCoCA0LjNHIMKgIMKgIMKgNDRHIMKgIMKgIDklPGJyPi9kZXYv
YWRhMXAzIMKgIMKgIMKgIMKgIMKgNDhHIMKgIMKgIDQuM0cgwqAgwqAgwqA0NEcgwqAgwqAgOSU8
YnI+L2Rldi9hZGEycDMgwqAgwqAgwqAgwqAgwqA0OEcgwqAgwqAgNC4zRyDCoCDCoCDCoDQ0RyDC
oCDCoCA5JTxicj5Ub3RhbCDCoCDCoCDCoCDCoCDCoCDCoCDCoCAxNDRHIMKgIMKgIMKgMTNHIMKg
IMKgIDEzMUcgwqAgwqAgOSU8L2ZvbnQ+PGJyPjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+PGZv
bnQgZmFjZT0ibW9ub3NwYWNlIj4jwqA8Yj52bXN0YXQgLWg8L2I+PGJyPsKgcHJvY3MgwqAgwqBt
ZW1vcnkgwqAgwqBwYWdlIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgZGlza3MgwqAg
wqAgZmF1bHRzIMKgIMKgIMKgIGNwdTxicj7CoHIgwqBiIMKgdyDCoGF2bSDCoGZyZSDCoGZsdCDC
oHJlIMKgcGkgwqBwbyDCoCBmciDCoCBzciBhZDAgYWQxIMKgIGluIMKgIHN5IMKgIGNzIHVzIHN5
IGlkPGJyPsKgMSDCoDAgNDUgNTk4RyDCoDE3RyDCoDE5SyDCoCAyIMKgIDAgwqAgMCDCoDIwSyAy
LjNLIMKgIDAgwqAgMCAxMjA2IMKgNDhLIMKgMTlLIMKgMyDCoDEgOTY8L2ZvbnQ+PGJyPjwvZGl2
PjxkaXY+PGJyPjwvZGl2PjxkaXY+SSBjYW5ub3QgZmluZCBhIHVzZXJtb2RlwqBjdWxwcml0OyB0
aGUgc3VtIG9mIHRoZSBzaXplIG9mIHByb2Nlc3Mgc3dhcHMgaXMgbm93aGVyZSBuZWFyIHRoZSBh
bW91bnQgb2Ygc3BhY2UgcmVwb3J0ZWQgYnkgc3dhcGluZm86PC9kaXY+PGRpdj48YnI+PC9kaXY+
PGRpdj48Zm9udCBmYWNlPSJtb25vc3BhY2UiPiPCoDxiPi91c3IvYmluL3RvcCAtdyAtbyBzd2Fw
PC9iPjwvZm9udD48L2Rpdj48ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFjZSI+bGFzdCBwaWQ6IDg4
NjAwOyDCoGxvYWQgYXZlcmFnZXM6IMKgMS4xNSwgwqAxLjAwLCDCoDAuODggwqAgwqAgwqAgwqAg
wqAgwqAgwqAgwqAgwqB1cCAxMSsxMToyMTo1NSDCoDA4OjQxOjU0PC9mb250PjwvZGl2Pjxmb250
IGZhY2U9Im1vbm9zcGFjZSI+MzUyIHByb2Nlc3NlczogMyBydW5uaW5nLCAzNDcgc2xlZXBpbmcs
IDIgem9tYmllPGJyPkNQVTogwqA1LjklIHVzZXIsIMKgMC4wJSBuaWNlLCDCoDEuMSUgc3lzdGVt
LCDCoDAuMCUgaW50ZXJydXB0LCA5Mi45JSBpZGxlPGJyPk1lbTogNzgxMk0gQWN0aXZlLCAxM0cg
SW5hY3QsIDY1RyBMYXVuZHJ5LCAyMkcgV2lyZWQsIDc0NE0gQnVmLCAxN0cgRnJlZTxicj5BUkM6
IDk1NzFNIFRvdGFsLCAxNTA3TSBNRlUsIDUwMzdNIE1SVSwgMjdNIEFub24sIDkyTSBIZWFkZXIs
IDI5MDdNIE90aGVyPGJyPsKgIMKgIMKgNDYyME0gQ29tcHJlc3NlZCwgMTJHIFVuY29tcHJlc3Nl
ZCwgMi41NzoxIFJhdGlvPGJyPlN3YXA6IDE0NEcgVG90YWwsIDEzRyBVc2VkLCAxMzFHIEZyZWUs
IDklIEludXNlPGJyPjxicj7CoCBQSUQgVVNFUk5BTUUgwqAgwqBUSFIgUFJJIE5JQ0UgwqAgU0la
RSDCoCDCoFJFUyBTV0FQIFNUQVRFIMKgIMKgQyDCoCBUSU1FIMKgIMKgV0NQVSBDT01NQU5EPGJy
PsKgMjA5NyBqdXB5dGVyIMKgIMKgIMKgIDEgwqAyMCDCoCDCoDAgwqAgMTE4TSDCoCDCoDEwTSDC
oDEyTSBrcXJlYWQgwqAxNyDCoCAwOjA5IMKgIDAuMDAlIHB5dGhvbjMuOTxicj44NTQ0NCDCoCDC
oDc3MCDCoCDCoCDCoCDCoDEgwqAyMCDCoCDCoDAgwqAgMTc5TSDCoCDCoDUySyDCoDExTSBrcXJl
YWQgwqAyMSDCoCAwOjAwIMKgIDAuMDAlIHBvc3RncmVzPGJyPjg1NDQxIMKgIMKgNzcwIMKgIMKg
IMKgIMKgMSDCoDIwIMKgIMKgMCDCoCAxNzlNIMKgIMKgNTJLIMKgMTFNIGtxcmVhZCDCoDEyIMKg
IDA6MDAgwqAgMC4wMCUgcG9zdGdyZXM8YnI+ODU0MzkgwqAgwqA3NzAgwqAgwqAgwqAgwqAxIMKg
MjAgwqAgwqAwIMKgIDE3OU0gwqAgwqA1MksgwqAxMU0ga3FyZWFkIMKgMTkgwqAgMDowMCDCoCAw
LjAwJSBwb3N0Z3Jlczxicj7CoDk4ODYgd3d3IMKgIMKgIMKgIMKgIMKgIDEgwqA1MiDCoCDCoDAg
wqAgMzY3TSDCoCDCoDUySyDCoDEwTSBhY2NlcHQgwqAxMCDCoCAwOjAwIMKgIDAuMDAlIHBocC1m
cG08YnI+wqA5ODg3IHd3dyDCoCDCoCDCoCDCoCDCoCAxIMKgNTIgwqAgwqAwIMKgIDM2N00gwqAg
wqA1MksgwqAxME0gYWNjZXB0IMKgIDAgwqAgMDowMCDCoCAwLjAwJSBwaHAtZnBtPGJyPsKgOTg4
NSB3d3cgwqAgwqAgwqAgwqAgwqAgMSDCoDUyIMKgIMKgMCDCoCAzNjdNIMKgIMKgNTJLIMKgMTBN
IGFjY2VwdCDCoCA4IMKgIDA6MDAgwqAgMC4wMCUgcGhwLWZwbTxicj7CoDk4ODMgd3d3IMKgIMKg
IMKgIMKgIMKgIDEgwqA1MiDCoCDCoDAgwqAgMzY3TSDCoCDCoDUySyDCoDEwTSBhY2NlcHQgwqAx
MiDCoCAwOjAwIMKgIDAuMDAlIHBocC1mcG08YnI+wqA5ODgxIHd3dyDCoCDCoCDCoCDCoCDCoCAx
IMKgNTIgwqAgwqAwIMKgIDM2N00gwqAgwqA1MksgwqAxME0gYWNjZXB0IMKgIDAgwqAgMDowMCDC
oCAwLjAwJSBwaHAtZnBtPGJyPsKgOTg4MCB3d3cgwqAgwqAgwqAgwqAgwqAgMSDCoDUyIMKgIMKg
MCDCoCAzNjdNIMKgIMKgNTJLIMKgMTBNIGFjY2VwdCDCoCAzIMKgIDA6MDAgwqAgMC4wMCUgcGhw
LWZwbTxicj7CoDk4ODIgd3d3IMKgIMKgIMKgIMKgIMKgIDEgwqA1MiDCoCDCoDAgwqAgMzY3TSDC
oCDCoDUySyDCoDEwTSBhY2NlcHQgwqAgMyDCoCAwOjAwIMKgIDAuMDAlIHBocC1mcG08YnI+wqA5
ODc2IHd3dyDCoCDCoCDCoCDCoCDCoCAxIMKgNTIgwqAgwqAwIMKgIDM2N00gwqAgwqA1MksgwqAx
ME0gYWNjZXB0IMKgMjIgwqAgMDowMCDCoCAwLjAwJSBwaHAtZnBtPGJyPsKgOTg3NSB3d3cgwqAg
wqAgwqAgwqAgwqAgMSDCoDUyIMKgIMKgMCDCoCAzNjdNIMKgIMKgNTJLIMKgMTBNIGFjY2VwdCDC
oCA1IMKgIDA6MDAgwqAgMC4wMCUgcGhwLWZwbTxicj7CoDk4Nzggd3d3IMKgIMKgIMKgIMKgIMKg
IDEgwqA1MiDCoCDCoDAgwqAgMzY3TSDCoCDCoDUySyDCoDEwTSBhY2NlcHQgwqAgNyDCoCAwOjAw
IMKgIDAuMDAlIHBocC1mcG08YnI+wqA5ODc0IHd3dyDCoCDCoCDCoCDCoCDCoCAxIMKgNTIgwqAg
wqAwIMKgIDM2N00gwqAgwqA1MksgwqAxME0gYWNjZXB0IMKgIDEgwqAgMDowMCDCoCAwLjAwJSBw
aHAtZnBtPGJyPsKgOTg3MiB3d3cgwqAgwqAgwqAgwqAgwqAgMSDCoDUyIMKgIMKgMCDCoCAzNjdN
IMKgIMKgNTJLIMKgMTBNIGFjY2VwdCDCoCAxIMKgIDA6MDAgwqAgMC4wMCUgcGhwLWZwbTxicj7C
oDk4NzMgd3d3IMKgIMKgIMKgIMKgIMKgIDEgwqA1MiDCoCDCoDAgwqAgMzY3TSDCoCDCoDUySyDC
oDEwTSBhY2NlcHQgwqAgMyDCoCAwOjAwIMKgIDAuMDAlIHBocC1mcG08YnI+wqA5ODcxIHd3dyDC
oCDCoCDCoCDCoCDCoCAxIMKgNTIgwqAgwqAwIMKgIDM2N00gwqAgwqA1MksgwqAxME0gYWNjZXB0
IMKgMTMgwqAgMDowMCDCoCAwLjAwJSBwaHAtZnBtPGJyPsKgOTg3MCB3d3cgwqAgwqAgwqAgwqAg
wqAgMSDCoDUyIMKgIMKgMCDCoCAzNjdNIMKgIMKgNTJLIMKgMTBNIGFjY2VwdCDCoCAyIMKgIDA6
MDAgwqAgMC4wMCUgcGhwLWZwbTxicj41NzQxMSDCoCDCoDc3MCDCoCDCoCDCoCDCoDEgwqAyMCDC
oCDCoDAgwqAgMTc5TSDCoCA2OTZLIDkxMDhLIGtxcmVhZCDCoDEwIMKgIDA6MDEgwqAgMC4wMCUg
cG9zdGdyZXM8YnI+NTQ5Nzggd3d3IMKgIMKgIMKgIMKgIMKgIDEgwqAyMCDCoCDCoDAgwqAgwqAz
Mk0gwqAgODUySyA4MzM2SyBhY2NlcHQgwqAxNyDCoCAwOjA4IMKgIDAuMDIlIGh0dHBkPGJyPsKg
OTYzOSDCoCDCoDc3MCDCoCDCoCDCoCDCoDEgwqAyMCDCoCDCoDAgwqAgMTc5TSDCoCA1NzJLIDgw
NjBLIGtxcmVhZCDCoCA2IMKgIDA6MDIgwqAgMC4wMCUgcG9zdGdyZXM8YnI+wqA4MzUwIMKgIMKg
NzcwIMKgIMKgIMKgIMKgMSDCoDIwIMKgIMKgMCDCoCAxNzZNIMKgIDU2MEsgNzc0NEsga3FyZWFk
IMKgMTggwqAgMDowMiDCoCAwLjAwJSBwb3N0Z3Jlczxicj7CoDgzNDkgwqAgwqA3NzAgwqAgwqAg
wqAgwqAxIMKgMjAgwqAgwqAwIMKgIDE3Nk0gwqAgNjY4SyA3NjQwSyBrcXJlYWQgwqAgNSDCoCAw
OjAwIMKgIDAuMDAlIHBvc3RncmVzPGJyPsKgODM1NCDCoCDCoDc3MCDCoCDCoCDCoCDCoDEgwqAy
MCDCoCDCoDAgwqAgMTc3TSDCoDMyNDhLIDUyODBLIGtxcmVhZCDCoCA1IMKgIDA6MDAgwqAgMC4w
MCUgcG9zdGdyZXM8YnI+wqA4MzUzIMKgIMKgNzcwIMKgIMKgIMKgIMKgMSDCoDIwIMKgIMKgMCDC
oCAxNzdNIMKgMzQ1MksgNTE3Mksga3FyZWFkIMKgIDIgwqAgMDowMyDCoCAwLjAwJSBwb3N0Z3Jl
czxicj7CoDU5NDkgc21tc3AgwqAgwqAgwqAgwqAgMSDCoDIwIMKgIMKgMCDCoCDCoDE4TSDCoCA5
NjRLIDMzMjRLIHBhdXNlIMKgIDE3IMKgIDA6MDAgwqAgMC4wMCUgc2VuZG1haWw8YnI+wqAxOTU2
IHJvb3QgwqAgwqAgwqAgwqAgwqAxIMKgNTIgwqAgwqAwIMKgIMKgMThNIMKgNDA5NkIgMzAyNEsg
bG9ja2YgwqAgMTQgwqAgMDowMCDCoCAwLjAwJSAmbHQ7c2FzbGF1dGhkJmd0Ozxicj7CoDU5ODQg
c2NvdHQgwqAgwqAgwqAgwqAgMSDCoDIwIMKgIMKgMCDCoCDCoDE1TSDCoDQwOTZCIDMwMjBLIHdh
aXQgwqAgwqAyMCDCoCAwOjAwIMKgIDAuMDAlICZsdDtiYXNoJmd0Ozxicj48L2ZvbnQ+PGRpdj48
Zm9udCBmYWNlPSJtb25vc3BhY2UiPsKgMTk2OCByb290IMKgIMKgIMKgIMKgIMKgMSDCoDUyIMKg
IMKgMCDCoCDCoDE4TSDCoDQwOTZCIDI5NzZLIGFjY2VwdCDCoDEyIMKgIDA6MDAgwqAgMC4wMCUg
Jmx0O3Nhc2xhdXRoZCZndDs8L2ZvbnQ+PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5UaGVyZSBh
cmUgbm90ICZxdW90O3RoYXQgbWFueSZxdW90OyBwcm9jZXNzZXMgc28gdGhpcyBpcyBub3QgJnF1
b3Q7ZGVhdGggYnkgYSB0aG91c2FuZCBjdXRzJnF1b3Q7OjwvZGl2PjxkaXY+PGJyPjwvZGl2Pjxk
aXY+PGZvbnQgZmFjZT0ibW9ub3NwYWNlIj4jwqA8Yj5wcyAtYXV4IHwgd2MgLWw8L2I+PGJyPsKg
IMKgIMKgMzcyPGJyPjwvZm9udD48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PkkgaGF2ZSBhIHN1
c3BpY2lvbiB0aGF0IHRoaXMgaXMgcmVsYXRlZCB0byB0aGUgd2lyZWd1YXJkIGttb2Qgc2ltcGx5
IGJlY2F1c2UgSSBydW4gd2lyZWd1YXJkIGluIGEgdm5ldCBqYWlsIGFuZCBkaWRuJiMzOTt0IG9i
c2VydmUgdGhpcyBwcm9ibGVtIHVudGlsIHNldHRpbmcgdGhhdCB1cC7CoCBCdXQgSSBkb24mIzM5
O3QgaGF2ZSBhbnkgZXZpZGVuY2UuPC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5JJiMzOTt2ZSB0
cmllZCB0byBtaXRpZ2F0ZSB0aGlzIHZpYSBzd2Fwb2ZmIC1hLsKgIFRoaXMgd29ya3Mgb25jZSBi
dXQgdGhlIG5leHQgZGF5IHN3YXAgd2lsbCBiZSBiYWNrLCBldmVuIGZ1bGxlci7CoCBJJiMzOTt2
ZSBiZWVuIGRvaW5nIHJlZ3VsYXIgcmVib290cyB0byBmaXggdGhpcyBidXQgd291bGQgbGlrZSB0
byBnZXQgdG8gdGhlIGJvdHRvbSBvZiBpdC7CoCBJZiBsZWZ0IGFsb25lLCBzd2FwIHdpbGwgZmls
bCB1cCBhbmQgdGhlIG1hY2hpbmUgd2lsbCBnZXQgaW50byBhICZxdW90O25vdCBxdWl0ZSBodW5n
JnF1b3Q7IGJ1dCB1bnVzYWJsZSBhbmQgdXNlbGVzcyBzdGF0ZS48L2Rpdj48ZGl2Pjxicj48L2Rp
dj48ZGl2PkFtIEkgb2ZmIGJhc2Ugd2l0aCBteSBzdXNwaWNpb24gdGhhdCB0aGlzIGlzIGtlcm5l
bCBtb2RlIG1lbW9yeT/CoCBDYW4gc29tZW9uZSB0ZWFjaCBtZSBob3cgdG8gZGlhZ25vc2UgdGhl
IHN0YXR1cyBvZiBrZXJuZWwgbW9kZSBtZW1vcnkgaGVhcD88L2Rpdj48ZGl2Pjxicj48L2Rpdj48
ZGl2PlRoeCw8L2Rpdj48ZGl2PlNjb3R0PC9kaXY+PC9kaXY+DQo=
--00000000000012c3a805fddc9f0e--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CABYAQkR69K=S0tCpWeKNdNR3-mvZwtr_68957_A_-35CbxBfFg>