Date: Wed, 23 Sep 2020 00:37:27 +0200 From: Peter <pmc@citylink.dinoex.sub.org> To: Konstantin Belousov <kostikbel@gmail.com> Cc: Mark Johnston <markj@freebsd.org>, freebsd-stable@freebsd.org Subject: Re: How to free used Swap-Space? (from errno=8) Message-ID: <20200922223727.GA84038@gate.oper.dinoex.org> In-Reply-To: <20200922210332.GF2570@kib.kiev.ua> References: <20200922160801.GA19535@gate.oper.dinoex.org> <20200922163319.GA70673@raichu> <20200922173107.GA27670@gate.oper.dinoex.org> <20200922180901.GC70673@raichu> <20200922191149.GA47828@gate.oper.dinoex.org> <20200922210332.GF2570@kib.kiev.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
--k+w/mQv8wyuph6w0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
On Wed, Sep 23, 2020 at 12:03:32AM +0300, Konstantin Belousov wrote:
! On Tue, Sep 22, 2020 at 09:11:49PM +0200, Peter wrote:
! > So what happens then is this:
! >
! > $ file scc.e
! > scc.e: ELF 32-bit LSB executable, Intel 80386, version 1
! > (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1,
! > for FreeBSD 9.3 (903504), stripped
! >
! > $ ./scc.e
! > ELF interpreter /libexec/ld-elf.so.1 not found, error 8
! > Abort trap
! >
! > And this will cost about some (hundred?) kB of swapspace every time it
! > happens. And they do not go away again, neither can the concerned jail
! > do fully die again.
! In what sense it 'costs' ?
Well that amount memory gets occupied. Forever, that is, until
poweroff/reset.
! Can you show exact sequence of commands and outputs that demostrate your
! point ? What type of filesystem the binaries live on ?
Oh, I didn't care. Originally on ZFS. When I tried to reproduce it,
most likely on an NFS-4 share, as I didn't bother to put it anywhere
special.
! I want to reproduce it locally.
Yes that's great! Lets see which info You are lacking.
Here we are now on my desktop box (mostly same machine, same
configuration, i5-3570, 11.4-p3, amd64).
I explicitely removed all the files that do not get installed
when /etc/src.conf contains the "WITHOUT_LIB32=", but I have the
COMPAT_FREEBSD32 still in the kernel.
Now I fetch such an old R9.3/i386 binary from my backups, and
drop it into some NFS filesystem:
(That binary is only 4kB, I just attach it here, if you wanna try
you can straightaway use that one - in normal operation it just
converts some words stdin to stdout).
admin@disp:510:1/ext/Repos$ dir usr2sys
-rwxr-xr-x 1 bin bin 4316 Apr 7 2016 usr2sys
admin@disp:511:1/ext/Repos$ file usr2sys
usr2sys: ELF 32-bit LSB executable, Intel 80386, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 9.3 (903504), stripped
admin@disp:513:1/ext/Repos$ mount | grep Repos
edge-e:/ext/Repos on /ext/Repos (nfs, nfsv4acls)
admin@disp:514:1/ext/Repos$ top | cat
Mem: 952M Active, 1687M Inact, 419M Laundry, 4423M Wired, 774M Buf, 348M Free
ARC: 1940M Total, 1378M MFU, 172M MRU, 2492K Anon, 48M Header, 340M Other
1134M Compressed, 2749M Uncompressed, 2.43:1 Ratio
Swap: 20G Total, 36M Used, 20G Free
As we see, this machine has 8 Gig installed and currently about no swap
used. Now watch what happens:
epos$ ./usr2sys
ELF interpreter /libexec/ld-elf.so.1 not found, error 8
Abort trap
admin@disp:519:1/ext/Repos$ for i in `seq 1000`
> do ./usr2sys
> done
ELF interpreter /libexec/ld-elf.so.1 not found, error 8
Abort trap
...
admin@disp:514:1/ext/Repos$ top | cat
Mem: 1010M Active, 1807M Inact, 419M Laundry, 4523M Wired, 774M Buf, 69M Free
ARC: 1940M Total, 1383M MFU, 166M MRU, 2503K Anon, 48M Header, 340M Other
1134M Compressed, 2750M Uncompressed, 2.43:1 Ratio
Swap: 20G Total, 36M Used, 20G Free
The free memory has already disappeared!
admin@disp:521:1/ext/Repos$ for i in `seq 5000`; do ./usr2sys ; done
...
admin@disp:522:1/ext/Repos$ top | cat
Mem: 2154M Active, 78M Inact, 787M Laundry, 4722M Wired, 774M Buf, 89M Free
ARC: 1753M Total, 1273M MFU, 97M MRU, 2653K Anon, 39M Header, 340M Other
953M Compressed, 2445M Uncompressed, 2.56:1 Ratio
Swap: 20G Total, 358M Used, 20G Free, 1% Inuse
Now the swapspace starts filling.
Lets see if the placement filesystem makes any difference and go onto UFS:
admin@disp:525:1/ext/Repos$ su -
Password:
root@disp:~ # cp /ext/Repos/usr2sys /var
root@disp:~ # dir /var/usr2sys
-rwxr-xr-x 1 bin bin 4316 Sep 22 23:55 /var/usr2sys
root@disp:~ # mount | grep /var
/dev/ada0p5 on /var (ufs, local, soft-updates)
admin@disp:527:1/var$ ./usr2sys
ELF interpreter /libexec/ld-elf.so.1 not found, error 8
Abort trap
admin@disp:521:1/ext/Repos$ for i in `seq 5000`; do ./usr2sys ; done
ELF interpreter /libexec/ld-elf.so.1 not found, error 8
Abort trap
...
Ahh, that runs a LOT faster now than on the NFS!
admin@disp:529:1/var$ top | cat
Mem: 1546M Active, 67M Inact, 934M Laundry, 5121M Wired, 774M Buf, 161M Free
ARC: 1646M Total, 1159M MFU, 107M MRU, 2686K Anon, 37M Header, 340M Other
849M Compressed, 2257M Uncompressed, 2.66:1 Ratio
Swap: 20G Total, 1658M Used, 18G Free, 8% Inuse
But memory leakage is similar to worse.
admin@disp:530:1/var$ df tmp
Filesystem 1K-blocks Used Avail Capacity Mounted on
zdesk/var/tmp 24747504 231052 24516452 1% /var/tmp
admin@disp:531:1/var$ cp usr2sys tmp
admin@disp:532:1/var$ cd tmp
admin@disp:533:1/var/tmp$ ./usr2sys
ELF interpreter /libexec/ld-elf.so.1 not found, error 8
Abort trap
admin@disp:534:1/var/tmp$ for i in `seq 5000`; do ./usr2sys ; done
...
You can see this is now a ZFS, and the behaviour is basically the same:
Mem: 1497M Active, 5292K Inact, 803M Laundry, 5313M Wired, 774M Buf, 212M Free
ARC: 1432M Total, 963M MFU, 105M MRU, 2511K Anon, 21M Header, 341M Other
650M Compressed, 1482M Uncompressed, 2.28:1 Ratio
Swap: 20G Total, 2791M Used, 17G Free, 13% Inuse
Now we have 3 gig eaten up, so we loose approx. 1 gig per 5000
aborted invocations.
And these do not go get freed anymore. One could continue this now, and
probably at some point the system will crash (and probably, beforehand
it will do some OOM killing), but I don't want to try that out now,
I'll just reboot now.
regards,
PMc
--k+w/mQv8wyuph6w0
Content-Type: application/octet-stream
Content-Disposition: attachment; filename=usr2sys
Content-Transfer-Encoding: base64
f0VMRgEBAQkAAAAAAAAAAAIAAwABAAAAoIQECDQAAACkDAAAAAAAADQAIAAIACgAGwAaAAYA
AAA0AAAANIAECDSABAgAAQAAAAEAAAUAAAAEAAAAAwAAADQBAAA0gQQINIEECBUAAAAVAAAA
BAAAAAEAAAABAAAAAAAAAACABAgAgAQIbAgAAGwIAAAFAAAAABAAAAEAAABsCAAAbJgECGyY
BAgcAQAAMAEAAAYAAAAAEAAAAgAAAIAIAACAmAQIgJgECNAAAADQAAAABgAAAAQAAAAEAAAA
TAEAAEyBBAhMgQQIGAAAABgAAAAEAAAABAAAAFDldGQgCAAAIIgECCCIBAgUAAAAFAAAAAQA
AAAEAAAAUeV0ZAAAAAAAAAAAAAAAAAAAAAAAAAAABgAAAAQAAAAvbGliZXhlYy9sZC1lbGYu
c28uMQAAAAAIAAAABAAAAAEAAABGcmVlQlNEAFDJDQADAAAADwAAAAYAAAAJAAAACAAAAAAA
AAAAAAAADQAAAA4AAAADAAAAAgAAAAEAAAAMAAAACwAAAAcAAAAFAAAACgAAAAQAAAAAAAAA
AAAAAAMAAAAKAAAAAQAAAAUAAABhICA7CgAAAA0AAAAAAAAAIMdZYabdo2u745J8vFB2nhXT
qs0AAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAIAAAADsAAAAAAAAAOQAAABIAAABzAAAA
AAAAACsAAAASAAAATAAAAAAAAAAFAAAAEgAAAGsAAAAAAAAAQgAAABIAAABWAAAAAAAAAKwA
AAASAAAAdQAAAAAAAABDAAAAEgAAAEIAAAAAAAAAjAAAABIAAABHAAAAAAAAAMIAAAASAAAA
XgAAAJCZBAgEAAAAEQAYACoAAACYmQQIBAAAABEAGAB6AAAAnJkECAAAAAAQAPH/HwAAAHyZ
BAgEAAAAEQAXADIAAACImQQIBAAAABEAGAAAX0p2X1JlZ2lzdGVyQ2xhc3NlcwBsaWJjLnNv
LjcAX19wcm9nbmFtZQBlbnZpcm9uAF9fc3RkaW5wAHByaW50ZgBnZXRjAHB1dHMAX2luaXRf
dGxzAHB1dGNoYXIAX19pc3RocmVhZGVkAF9fc3JnZXQAYXRleGl0AF9lbmQARkJTRF8xLjAA
AAAAAAIAAgACAAIAAgACAAIAAgACAAEAAQABAAIAAAABAAEAFQAAABAAAAAAAAAAsCh6BwAA
AgB/AAAAAAAAAIiZBAgFDgAAkJkECAUKAABcmQQIBwIAAGCZBAgHAwAAZJkECAcEAABomQQI
BwUAAGyZBAgHBgAAcJkECAcHAAB0mQQIBwgAAHiZBAgHCQAAg+wM6JgBAADo0wMAAIPEDMMA
AAD/NVSZBAj/JViZBAgAAAAA/yVcmQQIaAAAAADp4P////8lYJkECGgIAAAA6dD/////JWSZ
BAhoEAAAAOnA/////yVomQQIaBgAAADpsP////8lbJkECGggAAAA6aD/////JXCZBAhoKAAA
AOmQ/////yV0mQQIaDAAAADpgP////8leJkECGg4AAAA6XD///8AAAAAAAAAAAAAAAAx7VWJ
5YPk8I1FCIPsBFD/dQRS6AgAAADMkJCQkJCQkFWJ5VdWU4PsDKGYmQQIi10Mi3UQhcChmJkE
CI18ngQPRMeF26OYmQQIfiyLFoXSdCaJFXyZBAgPtgKEwHQZQpA8L6F8mQQID0TCo3yZBAgP
tgJChMB16biAmAQIhcB0NItFCIkEJOj5/v//xwQkBIgECOjt/v//6LT+//+JfCQIiXQkBIkc
JOi0AQAAiQQk6BD////o2/7//+vQkJCQkJBVieWD7AiAPZSZBAgAdAzrHIPABKOEmQQI/9Kh
hJkECIsQhdJ168YFlJkECAHJw5BVieWD7AihfJgECIXAdBK4AAAAAIXAdAnHBCR8mAQI/9DJ
w5CQkJCQkJCQkJCQkJBVieWD7BiJXfSJdfiJffyLXQi+AAAAAIA7IXUIg8MBvgEAAAAPtgM8
KnUrgHsBAHVFhfZ0EMcEJCEAAACNdCYA6E/+///HBCQQiAQI6HP+///p3QAAADwudRwPtkMB
hMAPhM0AAAA8KnUMgHsCAGaQD4S9AAAAid/8uf////+4AAAAAPKu99GNQf+NVBj/gDoqdU2N
RBj+gDgudSXGAACF9nQMxwQkIQAAAOjl/f//iVwkBMcEJBSIBAjolf3//+tyxgIAhfZ0DccE
JCEAAACQ6L/9//+JHCTo5/3//+tUkIX2dCPHBCQhAAAAkI10JgDon/3//4lcJATHBCQciAQI
6E/9///rHIlcJATHBCQciAQI6D39///HBCQhAAAA6HH9//+JXCQExwQkFIgECOgh/f//i130
i3X4i338iexdw41MJASD5PD/cfxVieVWU1GB7AwEAAC7AAAAAI219Pv//+s9g/kKdBiD+Sx0
E4iMHfT7//+DwwHrJ420JgAAAACF234cxoQd9Pv//wCJNCTobP7//7sAAAAAjbQmAAAAAIM9
kJkECAB1OaGImQQIi1AEg+oBiVAEhdJ5FaGImQQIiQQk6Lz8//+JwesljXQmAIsViJkECIsC
D7YIg8ABiQLrD6GImQQIiQQk6MX8//+JwYP5IHQJg/n/D4Vk////xwQkAAAAAOiZ/P//kJCQ
kJBVieVTg+wEoWyYBAiD+P90HzHbjbYAAAAAjbwnAAAAAP/Qi4NomAQIg+sEg/j/dfBYW13D
g+wM6FT9//+DxAzDYWxsACVzLmFsbAoAJXMsAAEbAzsQAAAAAQAAAID8//8sAAAAFAAAAAAA
AAABelIAAXwIARsMBASIAQAAGAAAABwAAABM/P//GQAAAABDDgRChQINBQAAAAAAAAD/////
AAAAAP////8AAAAAAAAAAAEAAAAVAAAADAAAAPCDBAgNAAAABIgECAQAAABkgQQI9f7/b7SB
BAgFAAAA2IIECAYAAADogQQICgAAAIgAAAALAAAAEAAAABUAAAAAAAAAAwAAAFCZBAgCAAAA
QAAAABQAAAARAAAAFwAAALCDBAgRAAAAoIMECBIAAAAQAAAAEwAAAAgAAAD+//9vgIMECP//
/28BAAAA8P//b2CDBAgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAACAmAQIAAAAAAAAAAAahAQIKoQECDqEBAhKhAQIWoQECGqEBAh6hAQIioQECBuI
BAgAAAAAeJgECAAkRnJlZUJTRDogcmVsZW5nLzkuMS9saWIvY3N1L2kzODYtZWxmL2NydDFf
cy5TIDIxNzM4MyAyMDExLTAxLTEzIDIzOjAwOjIyWiBraWIgJAAAJEZyZWVCU0Q6IHN0YWJs
ZS85L2xpYi9jc3UvaTM4Ni1lbGYvY3J0MV9jLmMgMjQ1Nzc3IDIwMTMtMDEtMjIgMDc6MzI6
MjZaIGtpYiAkACRGcmVlQlNEOiByZWxlbmcvOS4xL2xpYi9jc3UvY29tbW9uL2NydGJyYW5k
LmMgMjE3Mzc1IDIwMTEtMDEtMTMgMjA6NDQ6MzFaIGRpbSAkAEdDQzogKEdOVSkgNC4yLjEg
MjAwNzA4MzEgcGF0Y2hlZCBbRnJlZUJTRF0AACRGcmVlQlNEOiByZWxlbmcvOS4xL2xpYi9j
c3UvaTM4Ni1lbGYvY3J0aS5TIDIxNzEwNSAyMDExLTAxLTA3IDE2OjA3OjUxWiBraWIgJAAA
R0NDOiAoR05VKSA0LjIuMSAyMDA3MDgzMSBwYXRjaGVkIFtGcmVlQlNEXQAAR0NDOiAoR05V
KSA0LjIuMSAyMDA3MDgzMSBwYXRjaGVkIFtGcmVlQlNEXQAAR0NDOiAoR05VKSA0LjIuMSAy
MDA3MDgzMSBwYXRjaGVkIFtGcmVlQlNEXQAAJEZyZWVCU0Q6IHJlbGVuZy85LjEvbGliL2Nz
dS9pMzg2LWVsZi9jcnRuLlMgMjE3MTA1IDIwMTEtMDEtMDcgMTY6MDc6NTFaIGtpYiAkAAAu
c2hzdHJ0YWIALmludGVycAAubm90ZS5BQkktdGFnAC5nbnUuaGFzaAAuZHluc3ltAC5keW5z
dHIALmdudS52ZXJzaW9uAC5nbnUudmVyc2lvbl9yAC5yZWwuZHluAC5yZWwucGx0AC5pbml0
AC50ZXh0AC5maW5pAC5yb2RhdGEALmVoX2ZyYW1lX2hkcgAuZWhfZnJhbWUALmN0b3JzAC5k
dG9ycwAuamNyAC5keW5hbWljAC5nb3QucGx0AC5kYXRhAC5ic3MALmNvbW1lbnQAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACwAAAAEAAAACAAAANIEECDQB
AAAVAAAAAAAAAAAAAAABAAAAAAAAABMAAAAHAAAAAgAAAEyBBAhMAQAAGAAAAAAAAAAAAAAA
BAAAAAAAAAAlAAAABQAAAAIAAABkgQQIZAEAAFAAAAAFAAAAAAAAAAQAAAAEAAAAIQAAAPb/
/28CAAAAtIEECLQBAAA0AAAABQAAAAAAAAAEAAAABAAAACsAAAALAAAAAgAAAOiBBAjoAQAA
8AAAAAYAAAABAAAABAAAABAAAAAzAAAAAwAAAAIAAADYggQI2AIAAIgAAAAAAAAAAAAAAAEA
AAAAAAAAOwAAAP///28CAAAAYIMECGADAAAeAAAABQAAAAAAAAACAAAAAgAAAEgAAAD+//9v
AgAAAICDBAiAAwAAIAAAAAYAAAABAAAABAAAAAAAAABXAAAACQAAAAIAAACggwQIoAMAABAA
AAAFAAAAAAAAAAQAAAAIAAAAYAAAAAkAAAACAAAAsIMECLADAABAAAAABQAAAAwAAAAEAAAA
CAAAAGkAAAABAAAABgAAAPCDBAjwAwAAEQAAAAAAAAAAAAAABAAAAAAAAABkAAAAAQAAAAYA
AAAEhAQIBAQAAJAAAAAAAAAAAAAAAAQAAAAEAAAAbwAAAAEAAAAGAAAAoIQECKAEAABkAwAA
AAAAAAAAAAAQAAAAAAAAAHUAAAABAAAABgAAAASIBAgECAAADAAAAAAAAAAAAAAABAAAAAAA
AAB7AAAAAQAAADIAAAAQiAQIEAgAABAAAAAAAAAAAAAAAAEAAAABAAAAgwAAAAEAAAACAAAA
IIgECCAIAAAUAAAAAAAAAAAAAAAEAAAAAAAAAJEAAAABAAAAAgAAADSIBAg0CAAAOAAAAAAA
AAAAAAAABAAAAAAAAACbAAAAAQAAAAMAAABsmAQIbAgAAAgAAAAAAAAAAAAAAAQAAAAAAAAA
ogAAAAEAAAADAAAAdJgECHQIAAAIAAAAAAAAAAAAAAAEAAAAAAAAAKkAAAABAAAAAwAAAHyY
BAh8CAAABAAAAAAAAAAAAAAABAAAAAAAAACuAAAABgAAAAMAAACAmAQIgAgAANAAAAAGAAAA
AAAAAAQAAAAIAAAAtwAAAAEAAAADAAAAUJkECFAJAAAsAAAAAAAAAAAAAAAEAAAABAAAAMAA
AAABAAAAAwAAAHyZBAh8CQAADAAAAAAAAAAAAAAABAAAAAAAAADGAAAACAAAAAMAAACImQQI
iAkAABQAAAAAAAAAAAAAAAgAAAAAAAAAywAAAAEAAAAAAAAAAAAAAIgJAABGAgAAAAAAAAAA
AAABAAAAAAAAAAEAAAADAAAAAAAAAAAAAADOCwAA1AAAAAAAAAAAAAAAAQAAAAAAAAA=
--k+w/mQv8wyuph6w0--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20200922223727.GA84038>