From owner-freebsd-stable@FreeBSD.ORG Thu Nov 4 09:52:57 2004 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 51DAC16A4EA for ; Thu, 4 Nov 2004 09:52:57 +0000 (GMT) Received: from park.rambler.ru (park.rambler.ru [81.19.64.101]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5E83143D4C for ; Thu, 4 Nov 2004 09:52:55 +0000 (GMT) (envelope-from is@rambler-co.ru) Received: from is.park.rambler.ru (is.park.rambler.ru [81.19.64.102]) by park.rambler.ru (8.12.6/8.12.6) with ESMTP id iA49qmis021538; Thu, 4 Nov 2004 12:52:48 +0300 (MSK) (envelope-from is@rambler-co.ru) Date: Thu, 4 Nov 2004 12:52:48 +0300 (MSK) From: Igor Sysoev X-X-Sender: is@is.park.rambler.ru To: Uwe Doering In-Reply-To: <4189666A.9020500@geminix.org> Message-ID: <20041104124616.S92154@is.park.rambler.ru> References: <4168578F.7060706@geminix.org> <20041103191641.K63546@is.park.rambler.ru> <4189666A.9020500@geminix.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: stable@freebsd.org Subject: Re: vnode_pager_putpages errors and DOS? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Nov 2004 09:52:57 -0000 On Thu, 4 Nov 2004, Uwe Doering wrote: > Igor Sysoev wrote: > > On Sat, 9 Oct 2004, Uwe Doering wrote: > >>[...] > >>I wonder whether the unresponsiveness is actually just the result of the > >>kernel spending most of the time in printf(), generating warning > >>messages. vnode_pager_generic_putpages() doesn't return any error in > >>case of a write failure, so the caller (syncer in this case) isn't aware > >>that the paging out failed, that is, it is supposed to carry on as if > >>nothing happened. > >> > >>So how about limiting the number of warnings to one per second? UFS has > >>similar code in order to curb "file system full" and the like. Please > >>consider trying the attached patch, which applies cleanly to 4-STABLE. > >>It won't make the actual application causing these errors any happier, > >>but it may eliminate the DoS aspect of the issue. > > > > I have just tried your patch. To test I ran the program from > > http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/67919 > > > > The patch allows me to login on machine while the system reports about > > "vnode_pager_putpages: I/O error 28". However, the file system access is > > very limited and after some time the system became unresponsible. > > Limited file system access is to be expected, since > vnode_pager_putpages() keeps the number of dirty buffers > ('numdirtybuffers') near its upper limit ('hidirtybuffers'). However, > the unresponsiveness may be caused by another shortcoming I found in the > meantime. > > When 'numdirtybuffers' is greater or equal 'hidirtybuffers', function > bwillwrite() will block until 'numdirtybuffers' drops below some > threshold value. bwillwrite() gets called in a number of places that > deal with writing data to disk. > > Two of these places are dofilewrite() (which is in turn called by > write() and pwrite()) and writev(). There, bwillwrite() gets called if > the file descriptor is of type DTYPE_VNODE. Now, this unfortunately > doesn't take into account that ttys, including pseudo ttys, and even > /dev/null and friends, are character device nodes and therefore vnodes > as well, but have nothing to do with writing data to disk. That is, in > case of heavy disk write activity, write attempts to these device nodes > get blocked, too! With the consequence that the system appears to > become unresponsive at the shell prompt, or reacts very sporadic. Even > daemonized processes that happen to log data to /dev/null (on stdout & > stderr, for example) will block. > > What we need here is an additional test that makes sure that in case of > a character device bwillwrite() gets called only if the device is in > fact a disk. Please consider trying out the attached patch. It will > not reduce the heavy disk activity (which is, after all, legitimate), > but it is supposed to enable you to operate the system at shell level > and kill the offending process, or do whatever is necessary to resolve > the problem. I've tried your patch from second email (it requires to include for devsw and D_DISK): the system also became unresponsible. The main problem is that I could not kill the offending process - it stuck in biowr state. Igor Sysoev http://sysoev.ru/en/