Date: Tue, 29 Jul 2003 21:04:54 -0400 From: Don Bowman <don@sandvine.com> To: Don Bowman <don@sandvine.com>, 'Robert Watson' <rwatson@freebsd.org>, Dave Dolson <ddolson@sandvine.com> Cc: "'freebsd-stable@freebsd.org'" <freebsd-stable@freebsd.org> Subject: RE: kernel deadlock Message-ID: <FE045D4D9F7AED4CBFF1B3B813C853370274206C@mail.sandvine.com>
next in thread | raw e-mail | index | archive | help
From: Don Bowman [mailto:don@sandvine.com]
>
> From: Robert Watson [mailto:rwatson@freebsd.org]
> > On Tue, 29 Jul 2003, Dave Dolson wrote:
> >
> > > To follow up, I've discovered that the system has
> exhausted its "FFS
> > > node" malloc type.
> ...
> >
> > Some problems with this have turned up in -CURRENT on large-memory
> > machines where some of the scaling factors have been off. In
>
> We currently have kern.maxvnodes=70354 set (automatically
> scaled). This
> is a 1GB box.
>
> I will try re-running the test with less.
>
> when it hits kern.maxvnodes, what will it do?
After applying the fixes from RELENG_4 for kern/52425,
I can still easily reproduce this hang without low memory.
Further debugging shows that vnlru process is waiting on
vlrup. This line is shown below. ie vnlru_nowhere is being
incremented ever 3 seconds.
static void
vnlru_proc(void)
{
...
s = splbio();
for (;;) {
...
if (done == 0) {
vnlru_nowhere++;
tsleep(vnlruproc, PPAUSE, "vlrup", hz * 3);
}
}
splx(s);
syncher is in vlruwk wait from getnewvnode().
lots of other processes waiting on ffsvgt.
this implies that vlrureclaim() was unable to free anything.
i have maxvnode = 35k. as soon as i hit this value, my system locked
up [bash on serial shell non-responsive, serial driver echos chars,
can drop into ddb]. Processes which don't use filesystem seem to continue
to run ok.
A couple of procs are waiting on inode: env, cron. These never come
out of waiting for it.
suggestions?
db> ps
pid proc addr uid ppid pgrp flag stat wmesg wchan cmd
649 dc35a8a0 e0a32000 0 641 641 004104 3 ffsvgt c03698a8 atrun
648 dc35a3c0 e0e36000 0 647 648 000014 3 vlruwk c0364c90 cron
647 dc35b740 e03d4000 0 135 135 000004 3 ppwait dc35b740 cron
646 dc35b0c0 e03ee000 0 635 101 004004 3 inode c368ee00 env
645 dc35ad80 e03f1000 0 212 644 004006 3 ffsvgt c03698a8 grep
644 dc35aa40 e0400000 0 212 644 004006 3 ffsvgt c03698a8 sysctl
641 dc35a080 e0e4c000 0 640 641 004084 3 wait dc35a080 sh
640 dc35a220 e0e39000 0 135 135 000084 3 piperd e037c5c0 cron
635 dc35a560 e0e32000 0 101 101 004084 3 piperd e037cd40 sh
456 dc35abe0 e03fc000 0 133 456 4004004 3 ffsvgt c03698a8 tclsh83
212 dc35bdc0 e0392000 0 199 212 004086 3 wait dc35bdc0 bash
199 dc35c440 e036e000 0 1 199 004186 3 wait dc35c440 login
187 dc35c2a0 e0376000 0 1 7 000086 3 select c037c460 snmpd
169 dc35af20 e03e7000 0 1 169 000084 3 nanslp c0364970
siocontrol
163 dc35b260 e03e2000 0 1 163 000084 3 nanslp c0364970 wddt
143 dc35b400 e03dd000 25 1 143 2000184 3 pause e03dd260
sendmail
140 dc35b5a0 e03d9000 0 1 140 000184 3 select c037c460 sendmail
137 dc35b8e0 e03d0000 0 1 137 000184 3 select c037c460 sshd
135 dc35ba80 e03c2000 0 1 135 000004 3 inode c35f4400 cron
133 dc35bc20 e0397000 0 1 133 000084 3 select c037c460 inetd
124 dc35bf60 e0382000 0 1 124 000084 3 select c037c460 syslogd
101 dc35c100 e037e000 0 1 101 000084 3 wait dc35c100 dhclient
6 dc35c5e0 defd1000 0 0 0 000204 3 vlrup dc35c5e0 vnlru
5 dc35c780 defce000 0 0 0 000204 3 syncer c037c388 syncer
4 dc35c920 defcb000 0 0 0 000204 3 psleep c0364b3c
bufdaemon
3 dc35cac0 defc8000 0 0 0 000204 3 psleep c0373280 vmdaemon
2 dc35cc60 defc5000 0 0 0 000204 3 psleep c0352118
pagedaemon
1 dc35ce00 dc361000 0 0 1 004284 3 wait dc35ce00 init
0 c037b760 c040e000 0 0 0 000204 3 sched c037b760 swapper
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?FE045D4D9F7AED4CBFF1B3B813C853370274206C>