From owner-freebsd-current Sat May 8 12:24: 6 1999 Delivered-To: freebsd-current@freebsd.org Received: from nfs.dragondata.com (power.dragondata.com [204.137.237.7]) by hub.freebsd.org (Postfix) with ESMTP id BE1AB156A8 for ; Sat, 8 May 1999 12:24:01 -0700 (PDT) (envelope-from toasty@nfs.dragondata.com) Received: (from toasty@localhost) by nfs.dragondata.com (8.9.3/8.9.3) id OAA14425; Sat, 8 May 1999 14:23:57 -0500 (CDT) (envelope-from toasty) From: Kevin Day Message-Id: <199905081923.OAA14425@nfs.dragondata.com> Subject: Re: -current NFS crash (out of mbuf clusters) In-Reply-To: <199905081911.MAA55730@apollo.backplane.com> from Matthew Dillon at "May 8, 1999 12:11:14 pm" To: dillon@apollo.backplane.com (Matthew Dillon) Date: Sat, 8 May 1999 14:23:56 -0500 (CDT) Cc: toasty@home.dragondata.com (Kevin Day), current@FreeBSD.ORG X-Mailer: ELM [version 2.4ME+ PL54 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-freebsd-current@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG > :> netstat -m -M vmcore.XX -N kernel.XX > :> > : > :1014/2144 mbufs in use: > : 714 mbufs allocated to data > : 300 mbufs allocated to packet headers > :638/1324/1536 mbuf clusters in use (current/peak/max) > :2916 Kbytes allocated to network (48% in use) > :0 requests for memory denied > :0 requests for memory delayed > :0 calls to protocol drain routines > : > :What does this tell you? > : > :Kevin > > It tells me your userbase is out of control :-) From the looks > of it, hundreds of cron jobs are starting up simultaniously > and overloading some system resource. > Yeah, I wrote a patch to cron for a while that wouldn't allow that to happen, but it didn't apply cleanly to 4.0's cron, so I'm going to go check why. :) (It staggered the requests, only allow x to run per quantum) > I would also recommend: > > vmstat -m -M vmcore.XX -N kernel.XX > Memory statistics by bucket size Size In Use Free Requests HighWater Couldfree 16 883 141 82052816 1280 0 32 7569 9711 561955 640 253 64 27568 1744 87910913 320 253 128 1526 202 80132397 160 1021415 256 20096 3616 244895 80 151 512 78 2 1911 40 0 1K 277 135 13673 20 314 2K 32 12 173 10 67 4K 6 1 4585 5 0 8K 0 1 2 5 0 16K 3 0 3 5 0 32K 4 0 4 5 0 64K 5 0 5 5 0 128K 1 0 1 5 0 256K 1 0 1 5 0 512K 0 0 2 5 0 Memory usage type by bucket size Size Type(s) 16 devbuf, temp, proc, sysctl, rman, soname, pcb, vnodes, ether_multi, routetbl, isa_devlist, atkbddev, devbuf, temp, proc, sysctl, rman, soname, pcb, vnodes, ether_multi, routetbl, isa_devlist, atkbddev, devbuf, temp, proc, sysctl, rman, soname, pcb, vnodes, ether_multi, routetbl 32 kld, sigio, devbuf, temp, pgrp, subproc, sysctl, SWAP, soname, pcb, cluster_save buffer, vnodes, ifaddr, ether_multi, routetbl, in_multi, NFS req, kld, sigio, devbuf, temp, pgrp, subproc, sysctl, SWAP, soname, pcb, cluster_save buffer, vnodes, ifaddr, ether_multi, routetbl, in_multi, NFS req, kld, sigio, devbuf, temp, pgrp, subproc, sysctl, SWAP, soname, pcb, cluster_save buffer, vnodes, ifaddr, ether_multi, routetbl, in_multi, NFS req 64 file, lockf, namecache, devbuf, temp, session, rman, soname, pcb, cluster_save buffer, vnodes, ifaddr, ether_multi, routetbl, NFS req, file, lockf, namecache, devbuf, temp, session, rman, soname, pcb, cluster_save buffer, vnodes, ifaddr, ether_multi, routetbl, NFS req, file, lockf, namecache, devbuf, temp, session, rman, soname, pcb, cluster_save buffer, vnodes, ifaddr, ether_multi, routetbl, NFS req 128 isadev, kld, timecounter, file desc, zombie, namecache, devbuf, temp, cred, ttys, soname, vnodes, ifaddr, routetbl, ZONE, isadev, kld, timecounter, file desc, zombie, namecache, devbuf, temp, cred, ttys, soname, vnodes, ifaddr, routetbl, ZONE, isadev, kld, timecounter, file desc, zombie, namecache, devbuf, temp, cred, ttys, soname, vnodes, ifaddr, routetbl, ZONE 256 file desc, devbuf, temp, proc, subproc, vnodes, ifaddr, routetbl, NFS srvsock, NFS daemon, FFS node, file desc, devbuf, temp, proc, subproc, vnodes, ifaddr, routetbl, NFS srvsock, NFS daemon, FFS node, file desc, devbuf, temp, proc, subproc, vnodes, ifaddr, routetbl, NFS srvsock, NFS daemon, FFS node 512 file desc, devbuf, temp, ioctlops, BIO buffer, mount, NFSV3 diroff, UFS mount, isa_devlist, file desc, devbuf, temp, ioctlops, BIO buffer, mount, NFSV3 diroff, UFS mount, isa_devlist, file desc, devbuf, temp, ioctlops, BIO buffer, mount, NFSV3 diroff, UFS mount 1K devbuf, temp, proc, BIO buffer, NQNFS Lease, devbuf, temp, proc, BIO buffer, NQNFS Lease, devbuf, temp, proc, BIO buffer, NQNFS Lease 2K devbuf, temp, pcb, BIO buffer, UFS mount, mbuf, isa_devlist, devbuf, temp, pcb, BIO buffer, UFS mount, mbuf, isa_devlist, devbuf, temp, pcb, BIO buffer, UFS mount 4K devbuf, temp, UFS mount, devbuf, temp, UFS mount, devbuf, temp, UFS mount 8K temp, temp, temp 16K devbuf, devbuf, devbuf 32K devbuf, temp, MSDOSFS mount, devbuf, temp, MSDOSFS mount, devbuf, temp, MSDOSFS mount 64K ISOFS mount, NFS hash, UFS ihash, UFS quota, VM pgdata, ISOFS mount, NFS hash, UFS ihash, UFS quota, VM pgdata, ISOFS mount, NFS hash, UFS ihash, UFS quota, VM pgdata 128K namecache, namecache, namecache 256K SWAP, SWAP, SWAP 512K temp, temp, temp Memory statistics by type Type Kern Type InUse MemUse HighUse Limit Requests Limit Limit Size(s) isadev 16 2K 2K 40960K 16 0 0 128 ISOFS mount 1 64K 64K 40960K 1 0 0 64K kld 4 1K 1K 40960K 4 0 0 32,128 timecounter 5 1K 1K 40960K 5 0 0 128 file desc 570 74K 74K 40960K 234498 0 0 128,256,512 file 2444 153K 154K 40960K 82740308 0 0 64 sigio 2 1K 1K 40960K 5 0 0 32 zombie 5 1K 14K 40960K 231587 0 0 128 lockf 17 2K 4K 40960K 27419 0 0 64 namecache 24510 1665K 1809K 40960K 279237 0 0 64,128,128K devbuf 332 156K 156K 40960K 384 0 0 16,32,64,128,256,512,1K,2K,4K,16K,32K temp 953 94K 994K 40960K 212165 0 0 16,32,64,128,256,512,1K,2K,4K,8K,32K,512K pgrp 276 9K 10K 40960K 28030 0 0 32 session 259 17K 18K 40960K 26357 0 0 64 proc 118 5K 5K 40960K 5113 0 0 16,256,1K subproc 1336 88K 100K 40960K 492804 0 0 32,256 cred 356 45K 53K 40960K 79650970 0 0 128 sysctl 0 0K 1K 40960K 1614 0 0 16,32 SWAP 2 141K 141K 40960K 2 0 0 32,256K ioctlops 0 0K 1K 40960K 1 0 0 512 rman 33 2K 2K 40960K 50 0 0 16,64 ttys 245 31K 85K 40960K 7850 0 0 128 soname 4 1K 1K 40960K 1778822 0 0 16,32,64,128 pcb 459 12K 12K 40960K 665954 0 0 16,32,64,2K BIO buffer 294 320K 403K 40960K 13047 0 0 512,1K,2K cluster_save buffer 0 0K 1K 40960K 2828 0 0 32,64 mount 4 2K 2K 40960K 4 0 0 512 vnodes 5214 168K 475K 40960K 79653054 0 0 16,32,64,128,256 MSDOSFS mount 1 32K 32K 40960K 1 0 0 32K ifaddr 58 8K 8K 40960K 58 0 0 32,64,128,256 ether_multi 12 1K 1K 40960K 12 0 0 16,32,64 routetbl 521 76K 107K 40960K 3024 0 0 16,32,64,128,256 in_multi 3 1K 1K 40960K 3 0 0 32 NFS srvsock 2 1K 1K 40960K 2 0 0 256 NFS req 347 18K 20K 40960K 4694818 0 0 32,64 NFS daemon 1 1K 1K 40960K 1 0 0 256 NFSV3 diroff 11 6K 6K 40960K 11 0 0 512 NQNFS Lease 1 1K 1K 40960K 1 0 0 1K NFS hash 1 64K 64K 40960K 1 0 0 64K FFS node 19577 4895K 5811K 40960K 173220 0 0 256 UFS ihash 1 64K 64K 40960K 1 0 0 64K UFS quota 1 64K 64K 40960K 1 0 0 64K UFS mount 3 7K 7K 40960K 3 0 0 512,2K,4K VM pgdata 1 64K 64K 40960K 1 0 0 64K ZONE 18 3K 3K 40960K 18 0 0 128 mbuf 1 2K 2K 40960K 1 0 0 2K isa_devlist 29 3K 3K 40960K 29 0 0 16,512,2K atkbddev 1 1K 1K 40960K 1 0 0 16 isadev 16 2K 2K 40960K 16 0 0 128 ISOFS mount 1 64K 64K 40960K 1 0 0 64K kld 4 1K 1K 40960K 4 0 0 32,128 timecounter 5 1K 1K 40960K 5 0 0 128 file desc 570 74K 74K 40960K 234498 0 0 128,256,512 file 2444 153K 154K 40960K 82740308 0 0 64 sigio 2 1K 1K 40960K 5 0 0 32 zombie 5 1K 14K 40960K 231587 0 0 128 lockf 17 2K 4K 40960K 27419 0 0 64 namecache 24510 1665K 1809K 40960K 279237 0 0 64,128,128K devbuf 332 156K 156K 40960K 384 0 0 16,32,64,128,256,512,1K,2K,4K,16K,32K temp 953 94K 994K 40960K 212165 0 0 16,32,64,128,256,512,1K,2K,4K,8K,32K,512K pgrp 276 9K 10K 40960K 28030 0 0 32 session 259 17K 18K 40960K 26357 0 0 64 proc 118 5K 5K 40960K 5113 0 0 16,256,1K subproc 1336 88K 100K 40960K 492804 0 0 32,256 cred 356 45K 53K 40960K 79650970 0 0 128 sysctl 0 0K 1K 40960K 1614 0 0 16,32 SWAP 2 141K 141K 40960K 2 0 0 32,256K ioctlops 0 0K 1K 40960K 1 0 0 512 rman 33 2K 2K 40960K 50 0 0 16,64 ttys 245 31K 85K 40960K 7850 0 0 128 soname 4 1K 1K 40960K 1778822 0 0 16,32,64,128 pcb 459 12K 12K 40960K 665954 0 0 16,32,64,2K BIO buffer 294 320K 403K 40960K 13047 0 0 512,1K,2K cluster_save buffer 0 0K 1K 40960K 2828 0 0 32,64 mount 4 2K 2K 40960K 4 0 0 512 vnodes 5214 168K 475K 40960K 79653054 0 0 16,32,64,128,256 MSDOSFS mount 1 32K 32K 40960K 1 0 0 32K ifaddr 58 8K 8K 40960K 58 0 0 32,64,128,256 ether_multi 12 1K 1K 40960K 12 0 0 16,32,64 routetbl 521 76K 107K 40960K 3024 0 0 16,32,64,128,256 in_multi 3 1K 1K 40960K 3 0 0 32 NFS srvsock 2 1K 1K 40960K 2 0 0 256 NFS req 347 18K 20K 40960K 4694818 0 0 32,64 NFS daemon 1 1K 1K 40960K 1 0 0 256 NFSV3 diroff 11 6K 6K 40960K 11 0 0 512 NQNFS Lease 1 1K 1K 40960K 1 0 0 1K NFS hash 1 64K 64K 40960K 1 0 0 64K FFS node 19577 4895K 5811K 40960K 173220 0 0 256 UFS ihash 1 64K 64K 40960K 1 0 0 64K UFS quota 1 64K 64K 40960K 1 0 0 64K UFS mount 3 7K 7K 40960K 3 0 0 512,2K,4K VM pgdata 1 64K 64K 40960K 1 0 0 64K ZONE 18 3K 3K 40960K 18 0 0 128 mbuf 1 2K 2K 40960K 1 0 0 2K isa_devlist 29 3K 3K 40960K 29 0 0 16,512,2K atkbddev 1 1K 1K 40960K 1 0 0 16 isadev 16 2K 2K 40960K 16 0 0 128 ISOFS mount 1 64K 64K 40960K 1 0 0 64K kld 4 1K 1K 40960K 4 0 0 32,128 timecounter 5 1K 1K 40960K 5 0 0 128 file desc 570 74K 74K 40960K 234498 0 0 128,256,512 file 2444 153K 154K 40960K 82740308 0 0 64 sigio 2 1K 1K 40960K 5 0 0 32 zombie 5 1K 14K 40960K 231587 0 0 128 lockf 17 2K 4K 40960K 27419 0 0 64 namecache 24510 1665K 1809K 40960K 279237 0 0 64,128,128K devbuf 332 156K 156K 40960K 384 0 0 16,32,64,128,256,512,1K,2K,4K,16K,32K temp 953 94K 994K 40960K 212165 0 0 16,32,64,128,256,512,1K,2K,4K,8K,32K,512K pgrp 276 9K 10K 40960K 28030 0 0 32 session 259 17K 18K 40960K 26357 0 0 64 proc 118 5K 5K 40960K 5113 0 0 16,256,1K subproc 1336 88K 100K 40960K 492804 0 0 32,256 cred 356 45K 53K 40960K 79650970 0 0 128 sysctl 0 0K 1K 40960K 1614 0 0 16,32 SWAP 2 141K 141K 40960K 2 0 0 32,256K ioctlops 0 0K 1K 40960K 1 0 0 512 rman 33 2K 2K 40960K 50 0 0 16,64 ttys 245 31K 85K 40960K 7850 0 0 128 soname 4 1K 1K 40960K 1778822 0 0 16,32,64,128 pcb 459 12K 12K 40960K 665954 0 0 16,32,64,2K BIO buffer 294 320K 403K 40960K 13047 0 0 512,1K,2K cluster_save buffer 0 0K 1K 40960K 2828 0 0 32,64 mount 4 2K 2K 40960K 4 0 0 512 vnodes 5214 168K 475K 40960K 79653054 0 0 16,32,64,128,256 MSDOSFS mount 1 32K 32K 40960K 1 0 0 32K ifaddr 58 8K 8K 40960K 58 0 0 32,64,128,256 ether_multi 12 1K 1K 40960K 12 0 0 16,32,64 routetbl 521 76K 107K 40960K 3024 0 0 16,32,64,128,256 in_multi 3 1K 1K 40960K 3 0 0 32 NFS srvsock 2 1K 1K 40960K 2 0 0 256 NFS req 347 18K 20K 40960K 4694818 0 0 32,64 NFS daemon 1 1K 1K 40960K 1 0 0 256 NFSV3 diroff 11 6K 6K 40960K 11 0 0 512 NQNFS Lease 1 1K 1K 40960K 1 0 0 1K NFS hash 1 64K 64K 40960K 1 0 0 64K FFS node 19577 4895K 5811K 40960K 173220 0 0 256 UFS ihash 1 64K 64K 40960K 1 0 0 64K UFS quota 1 64K 64K 40960K 1 0 0 64K UFS mount 3 7K 7K 40960K 3 0 0 512,2K,4K VM pgdata 1 64K 64K 40960K 1 0 0 64K ZONE 18 3K 3K 40960K 18 0 0 128 Memory Totals: In Use Free Requests 25040K 1516K 752769977 > It is possible that the machine was attacked from the outside since you > are allowing eggdrops to be run. An IP spoofing attack can eat a > considerable amount of KVM due to temporary routes and, in fact, run > it out, leaving no memory left for mbufs. If so, this will show up > in the vmstat. Yeah, I've run into this a few times, which usually ended in "panic: kern_kmem_map too small". > > A quick side note on eggdrops: We allowed them at BEST.COM, but > after four years our machines and networks were getting attacked > virtually every day by IRC bozos. Also, the users who tend to run > eggdrops also tend to be stupid - often logging in from compromised > machines, so we also had a huge problem with these user's accounts being > compromised. We eventually gave up and banned bots entirely. Things have > been a whole lot quieter since. > Yeah, we see our problems, but I'm heavy handed with my cluebat, when it's intentional misdoings. In eggdrop user's defense, the instructions for eggdrop tells them to do */10 for the minutes, so they're only doing what they're told. I had a few discussions with Paul Vixie about possible solutions within cron to prevent the forkbombing every 10 mins/hour. We couldn't come up with a great solution, but what I wrote seems to work well. (If you want a diff, i'd be happy to provide it) > Another thing you can do in regards to the cron jobs is go through > all your user's crons, many of which are probably running bot check > scripts every 10 minutes, and adjust them to run only once an hour, > plus scramble the 'minute' so they do not all run simultaniously. > I've seen IRC bozos setup cron jobs that run botcheck once a minute. > We gave them one warning, and if they did not heed it we kicked them off. > Another cron patch was to make the minimum interval 10 minutes for user jobs. :) However, this exact same server, with the exact same cron jobs and eggdrops ran fine on 2.2.8 like this. The only reason I was motivated to fix cron, was that it caused the nfs server to nearly lock up every 10 minutes. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-current" in the body of the message