Date: Sat, 8 May 1999 14:23:56 -0500 (CDT) From: Kevin Day <toasty@nfs.dragondata.com> To: dillon@apollo.backplane.com (Matthew Dillon) Cc: toasty@home.dragondata.com (Kevin Day), current@FreeBSD.ORG Subject: Re: -current NFS crash (out of mbuf clusters) Message-ID: <199905081923.OAA14425@nfs.dragondata.com> In-Reply-To: <199905081911.MAA55730@apollo.backplane.com> from Matthew Dillon at "May 8, 1999 12:11:14 pm"
next in thread | previous in thread | raw e-mail | index | archive | help
> :> netstat -m -M vmcore.XX -N kernel.XX
> :>
> :
> :1014/2144 mbufs in use:
> : 714 mbufs allocated to data
> : 300 mbufs allocated to packet headers
> :638/1324/1536 mbuf clusters in use (current/peak/max)
> :2916 Kbytes allocated to network (48% in use)
> :0 requests for memory denied
> :0 requests for memory delayed
> :0 calls to protocol drain routines
> :
> :What does this tell you?
> :
> :Kevin
>
> It tells me your userbase is out of control :-) From the looks
> of it, hundreds of cron jobs are starting up simultaniously
> and overloading some system resource.
>
Yeah, I wrote a patch to cron for a while that wouldn't allow that to
happen, but it didn't apply cleanly to 4.0's cron, so I'm going to go check
why. :)
(It staggered the requests, only allow x to run per quantum)
> I would also recommend:
>
> vmstat -m -M vmcore.XX -N kernel.XX
>
Memory statistics by bucket size
Size In Use Free Requests HighWater Couldfree
16 883 141 82052816 1280 0
32 7569 9711 561955 640 253
64 27568 1744 87910913 320 253
128 1526 202 80132397 160 1021415
256 20096 3616 244895 80 151
512 78 2 1911 40 0
1K 277 135 13673 20 314
2K 32 12 173 10 67
4K 6 1 4585 5 0
8K 0 1 2 5 0
16K 3 0 3 5 0
32K 4 0 4 5 0
64K 5 0 5 5 0
128K 1 0 1 5 0
256K 1 0 1 5 0
512K 0 0 2 5 0
Memory usage type by bucket size
Size Type(s)
16 devbuf, temp, proc, sysctl, rman, soname, pcb, vnodes, ether_multi,
routetbl, isa_devlist, atkbddev, devbuf, temp, proc, sysctl, rman,
soname, pcb, vnodes, ether_multi, routetbl, isa_devlist, atkbddev,
devbuf, temp, proc, sysctl, rman, soname, pcb, vnodes, ether_multi,
routetbl
32 kld, sigio, devbuf, temp, pgrp, subproc, sysctl, SWAP, soname, pcb,
cluster_save buffer, vnodes, ifaddr, ether_multi, routetbl, in_multi,
NFS req, kld, sigio, devbuf, temp, pgrp, subproc, sysctl, SWAP,
soname, pcb, cluster_save buffer, vnodes, ifaddr, ether_multi,
routetbl, in_multi, NFS req, kld, sigio, devbuf, temp, pgrp, subproc,
sysctl, SWAP, soname, pcb, cluster_save buffer, vnodes, ifaddr,
ether_multi, routetbl, in_multi, NFS req
64 file, lockf, namecache, devbuf, temp, session, rman, soname, pcb,
cluster_save buffer, vnodes, ifaddr, ether_multi, routetbl, NFS req,
file, lockf, namecache, devbuf, temp, session, rman, soname, pcb,
cluster_save buffer, vnodes, ifaddr, ether_multi, routetbl, NFS req,
file, lockf, namecache, devbuf, temp, session, rman, soname, pcb,
cluster_save buffer, vnodes, ifaddr, ether_multi, routetbl, NFS req
128 isadev, kld, timecounter, file desc, zombie, namecache, devbuf, temp,
cred, ttys, soname, vnodes, ifaddr, routetbl, ZONE, isadev, kld,
timecounter, file desc, zombie, namecache, devbuf, temp, cred, ttys,
soname, vnodes, ifaddr, routetbl, ZONE, isadev, kld, timecounter,
file desc, zombie, namecache, devbuf, temp, cred, ttys, soname,
vnodes, ifaddr, routetbl, ZONE
256 file desc, devbuf, temp, proc, subproc, vnodes, ifaddr, routetbl,
NFS srvsock, NFS daemon, FFS node, file desc, devbuf, temp, proc,
subproc, vnodes, ifaddr, routetbl, NFS srvsock, NFS daemon, FFS node,
file desc, devbuf, temp, proc, subproc, vnodes, ifaddr, routetbl,
NFS srvsock, NFS daemon, FFS node
512 file desc, devbuf, temp, ioctlops, BIO buffer, mount, NFSV3 diroff,
UFS mount, isa_devlist, file desc, devbuf, temp, ioctlops,
BIO buffer, mount, NFSV3 diroff, UFS mount, isa_devlist, file desc,
devbuf, temp, ioctlops, BIO buffer, mount, NFSV3 diroff, UFS mount
1K devbuf, temp, proc, BIO buffer, NQNFS Lease, devbuf, temp, proc,
BIO buffer, NQNFS Lease, devbuf, temp, proc, BIO buffer, NQNFS Lease
2K devbuf, temp, pcb, BIO buffer, UFS mount, mbuf, isa_devlist, devbuf,
temp, pcb, BIO buffer, UFS mount, mbuf, isa_devlist, devbuf, temp,
pcb, BIO buffer, UFS mount
4K devbuf, temp, UFS mount, devbuf, temp, UFS mount, devbuf, temp,
UFS mount
8K temp, temp, temp
16K devbuf, devbuf, devbuf
32K devbuf, temp, MSDOSFS mount, devbuf, temp, MSDOSFS mount, devbuf,
temp, MSDOSFS mount
64K ISOFS mount, NFS hash, UFS ihash, UFS quota, VM pgdata, ISOFS mount,
NFS hash, UFS ihash, UFS quota, VM pgdata, ISOFS mount, NFS hash,
UFS ihash, UFS quota, VM pgdata
128K namecache, namecache, namecache
256K SWAP, SWAP, SWAP
512K temp, temp, temp
Memory statistics by type Type Kern
Type InUse MemUse HighUse Limit Requests Limit Limit Size(s)
isadev 16 2K 2K 40960K 16 0 0 128
ISOFS mount 1 64K 64K 40960K 1 0 0 64K
kld 4 1K 1K 40960K 4 0 0 32,128
timecounter 5 1K 1K 40960K 5 0 0 128
file desc 570 74K 74K 40960K 234498 0 0 128,256,512
file 2444 153K 154K 40960K 82740308 0 0 64
sigio 2 1K 1K 40960K 5 0 0 32
zombie 5 1K 14K 40960K 231587 0 0 128
lockf 17 2K 4K 40960K 27419 0 0 64
namecache 24510 1665K 1809K 40960K 279237 0 0 64,128,128K
devbuf 332 156K 156K 40960K 384 0 0 16,32,64,128,256,512,1K,2K,4K,16K,32K
temp 953 94K 994K 40960K 212165 0 0 16,32,64,128,256,512,1K,2K,4K,8K,32K,512K
pgrp 276 9K 10K 40960K 28030 0 0 32
session 259 17K 18K 40960K 26357 0 0 64
proc 118 5K 5K 40960K 5113 0 0 16,256,1K
subproc 1336 88K 100K 40960K 492804 0 0 32,256
cred 356 45K 53K 40960K 79650970 0 0 128
sysctl 0 0K 1K 40960K 1614 0 0 16,32
SWAP 2 141K 141K 40960K 2 0 0 32,256K
ioctlops 0 0K 1K 40960K 1 0 0 512
rman 33 2K 2K 40960K 50 0 0 16,64
ttys 245 31K 85K 40960K 7850 0 0 128
soname 4 1K 1K 40960K 1778822 0 0 16,32,64,128
pcb 459 12K 12K 40960K 665954 0 0 16,32,64,2K
BIO buffer 294 320K 403K 40960K 13047 0 0 512,1K,2K
cluster_save buffer 0 0K 1K 40960K 2828 0 0 32,64
mount 4 2K 2K 40960K 4 0 0 512
vnodes 5214 168K 475K 40960K 79653054 0 0 16,32,64,128,256
MSDOSFS mount 1 32K 32K 40960K 1 0 0 32K
ifaddr 58 8K 8K 40960K 58 0 0 32,64,128,256
ether_multi 12 1K 1K 40960K 12 0 0 16,32,64
routetbl 521 76K 107K 40960K 3024 0 0 16,32,64,128,256
in_multi 3 1K 1K 40960K 3 0 0 32
NFS srvsock 2 1K 1K 40960K 2 0 0 256
NFS req 347 18K 20K 40960K 4694818 0 0 32,64
NFS daemon 1 1K 1K 40960K 1 0 0 256
NFSV3 diroff 11 6K 6K 40960K 11 0 0 512
NQNFS Lease 1 1K 1K 40960K 1 0 0 1K
NFS hash 1 64K 64K 40960K 1 0 0 64K
FFS node 19577 4895K 5811K 40960K 173220 0 0 256
UFS ihash 1 64K 64K 40960K 1 0 0 64K
UFS quota 1 64K 64K 40960K 1 0 0 64K
UFS mount 3 7K 7K 40960K 3 0 0 512,2K,4K
VM pgdata 1 64K 64K 40960K 1 0 0 64K
ZONE 18 3K 3K 40960K 18 0 0 128
mbuf 1 2K 2K 40960K 1 0 0 2K
isa_devlist 29 3K 3K 40960K 29 0 0 16,512,2K
atkbddev 1 1K 1K 40960K 1 0 0 16
isadev 16 2K 2K 40960K 16 0 0 128
ISOFS mount 1 64K 64K 40960K 1 0 0 64K
kld 4 1K 1K 40960K 4 0 0 32,128
timecounter 5 1K 1K 40960K 5 0 0 128
file desc 570 74K 74K 40960K 234498 0 0 128,256,512
file 2444 153K 154K 40960K 82740308 0 0 64
sigio 2 1K 1K 40960K 5 0 0 32
zombie 5 1K 14K 40960K 231587 0 0 128
lockf 17 2K 4K 40960K 27419 0 0 64
namecache 24510 1665K 1809K 40960K 279237 0 0 64,128,128K
devbuf 332 156K 156K 40960K 384 0 0 16,32,64,128,256,512,1K,2K,4K,16K,32K
temp 953 94K 994K 40960K 212165 0 0 16,32,64,128,256,512,1K,2K,4K,8K,32K,512K
pgrp 276 9K 10K 40960K 28030 0 0 32
session 259 17K 18K 40960K 26357 0 0 64
proc 118 5K 5K 40960K 5113 0 0 16,256,1K
subproc 1336 88K 100K 40960K 492804 0 0 32,256
cred 356 45K 53K 40960K 79650970 0 0 128
sysctl 0 0K 1K 40960K 1614 0 0 16,32
SWAP 2 141K 141K 40960K 2 0 0 32,256K
ioctlops 0 0K 1K 40960K 1 0 0 512
rman 33 2K 2K 40960K 50 0 0 16,64
ttys 245 31K 85K 40960K 7850 0 0 128
soname 4 1K 1K 40960K 1778822 0 0 16,32,64,128
pcb 459 12K 12K 40960K 665954 0 0 16,32,64,2K
BIO buffer 294 320K 403K 40960K 13047 0 0 512,1K,2K
cluster_save buffer 0 0K 1K 40960K 2828 0 0 32,64
mount 4 2K 2K 40960K 4 0 0 512
vnodes 5214 168K 475K 40960K 79653054 0 0 16,32,64,128,256
MSDOSFS mount 1 32K 32K 40960K 1 0 0 32K
ifaddr 58 8K 8K 40960K 58 0 0 32,64,128,256
ether_multi 12 1K 1K 40960K 12 0 0 16,32,64
routetbl 521 76K 107K 40960K 3024 0 0 16,32,64,128,256
in_multi 3 1K 1K 40960K 3 0 0 32
NFS srvsock 2 1K 1K 40960K 2 0 0 256
NFS req 347 18K 20K 40960K 4694818 0 0 32,64
NFS daemon 1 1K 1K 40960K 1 0 0 256
NFSV3 diroff 11 6K 6K 40960K 11 0 0 512
NQNFS Lease 1 1K 1K 40960K 1 0 0 1K
NFS hash 1 64K 64K 40960K 1 0 0 64K
FFS node 19577 4895K 5811K 40960K 173220 0 0 256
UFS ihash 1 64K 64K 40960K 1 0 0 64K
UFS quota 1 64K 64K 40960K 1 0 0 64K
UFS mount 3 7K 7K 40960K 3 0 0 512,2K,4K
VM pgdata 1 64K 64K 40960K 1 0 0 64K
ZONE 18 3K 3K 40960K 18 0 0 128
mbuf 1 2K 2K 40960K 1 0 0 2K
isa_devlist 29 3K 3K 40960K 29 0 0 16,512,2K
atkbddev 1 1K 1K 40960K 1 0 0 16
isadev 16 2K 2K 40960K 16 0 0 128
ISOFS mount 1 64K 64K 40960K 1 0 0 64K
kld 4 1K 1K 40960K 4 0 0 32,128
timecounter 5 1K 1K 40960K 5 0 0 128
file desc 570 74K 74K 40960K 234498 0 0 128,256,512
file 2444 153K 154K 40960K 82740308 0 0 64
sigio 2 1K 1K 40960K 5 0 0 32
zombie 5 1K 14K 40960K 231587 0 0 128
lockf 17 2K 4K 40960K 27419 0 0 64
namecache 24510 1665K 1809K 40960K 279237 0 0 64,128,128K
devbuf 332 156K 156K 40960K 384 0 0 16,32,64,128,256,512,1K,2K,4K,16K,32K
temp 953 94K 994K 40960K 212165 0 0 16,32,64,128,256,512,1K,2K,4K,8K,32K,512K
pgrp 276 9K 10K 40960K 28030 0 0 32
session 259 17K 18K 40960K 26357 0 0 64
proc 118 5K 5K 40960K 5113 0 0 16,256,1K
subproc 1336 88K 100K 40960K 492804 0 0 32,256
cred 356 45K 53K 40960K 79650970 0 0 128
sysctl 0 0K 1K 40960K 1614 0 0 16,32
SWAP 2 141K 141K 40960K 2 0 0 32,256K
ioctlops 0 0K 1K 40960K 1 0 0 512
rman 33 2K 2K 40960K 50 0 0 16,64
ttys 245 31K 85K 40960K 7850 0 0 128
soname 4 1K 1K 40960K 1778822 0 0 16,32,64,128
pcb 459 12K 12K 40960K 665954 0 0 16,32,64,2K
BIO buffer 294 320K 403K 40960K 13047 0 0 512,1K,2K
cluster_save buffer 0 0K 1K 40960K 2828 0 0 32,64
mount 4 2K 2K 40960K 4 0 0 512
vnodes 5214 168K 475K 40960K 79653054 0 0 16,32,64,128,256
MSDOSFS mount 1 32K 32K 40960K 1 0 0 32K
ifaddr 58 8K 8K 40960K 58 0 0 32,64,128,256
ether_multi 12 1K 1K 40960K 12 0 0 16,32,64
routetbl 521 76K 107K 40960K 3024 0 0 16,32,64,128,256
in_multi 3 1K 1K 40960K 3 0 0 32
NFS srvsock 2 1K 1K 40960K 2 0 0 256
NFS req 347 18K 20K 40960K 4694818 0 0 32,64
NFS daemon 1 1K 1K 40960K 1 0 0 256
NFSV3 diroff 11 6K 6K 40960K 11 0 0 512
NQNFS Lease 1 1K 1K 40960K 1 0 0 1K
NFS hash 1 64K 64K 40960K 1 0 0 64K
FFS node 19577 4895K 5811K 40960K 173220 0 0 256
UFS ihash 1 64K 64K 40960K 1 0 0 64K
UFS quota 1 64K 64K 40960K 1 0 0 64K
UFS mount 3 7K 7K 40960K 3 0 0 512,2K,4K
VM pgdata 1 64K 64K 40960K 1 0 0 64K
ZONE 18 3K 3K 40960K 18 0 0 128
Memory Totals: In Use Free Requests
25040K 1516K 752769977
> It is possible that the machine was attacked from the outside since you
> are allowing eggdrops to be run. An IP spoofing attack can eat a
> considerable amount of KVM due to temporary routes and, in fact, run
> it out, leaving no memory left for mbufs. If so, this will show up
> in the vmstat.
Yeah, I've run into this a few times, which usually ended in "panic:
kern_kmem_map too small".
>
> A quick side note on eggdrops: We allowed them at BEST.COM, but
> after four years our machines and networks were getting attacked
> virtually every day by IRC bozos. Also, the users who tend to run
> eggdrops also tend to be stupid - often logging in from compromised
> machines, so we also had a huge problem with these user's accounts being
> compromised. We eventually gave up and banned bots entirely. Things have
> been a whole lot quieter since.
>
Yeah, we see our problems, but I'm heavy handed with my cluebat, when it's
intentional misdoings. In eggdrop user's defense, the instructions for
eggdrop tells them to do */10 for the minutes, so they're only doing what
they're told. I had a few discussions with Paul Vixie about possible
solutions within cron to prevent the forkbombing every 10 mins/hour. We
couldn't come up with a great solution, but what I wrote seems to work well.
(If you want a diff, i'd be happy to provide it)
> Another thing you can do in regards to the cron jobs is go through
> all your user's crons, many of which are probably running bot check
> scripts every 10 minutes, and adjust them to run only once an hour,
> plus scramble the 'minute' so they do not all run simultaniously.
> I've seen IRC bozos setup cron jobs that run botcheck once a minute.
> We gave them one warning, and if they did not heed it we kicked them off.
>
Another cron patch was to make the minimum interval 10 minutes for user
jobs. :)
However, this exact same server, with the exact same cron jobs and eggdrops
ran fine on 2.2.8 like this. The only reason I was motivated to fix cron,
was that it caused the nfs server to nearly lock up every 10 minutes.
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-current" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199905081923.OAA14425>