From owner-freebsd-fs@FreeBSD.ORG Wed Sep 22 13:22:03 2010 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 74F791065696 for ; Wed, 22 Sep 2010 13:22:03 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from people.fsn.hu (people.fsn.hu [195.228.252.137]) by mx1.freebsd.org (Postfix) with ESMTP id 270EB8FC1F for ; Wed, 22 Sep 2010 13:22:02 +0000 (UTC) Received: by people.fsn.hu (Postfix, from userid 1001) id 98F5C459FE4; Wed, 22 Sep 2010 15:22:01 +0200 (CEST) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000043, version=1.2.2 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MF-ACE0E1EA [pR: 11.1327] X-CRM114-CacheID: sfid-20100922_15215_DD0E7287 X-CRM114-Status: Good ( pR: 11.1327 ) X-DSPAM-Result: Whitelisted X-DSPAM-Processed: Wed Sep 22 15:22:01 2010 X-DSPAM-Confidence: 0.8516 X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 4c9a02f972594415313006 X-DSPAM-Factors: 27, From*Attila Nagy , 0.00058, wrote+>, 0.00213, To*FreeBSD.org, 0.00268, wrote, 0.00389, >+I, 0.00490, a+>, 0.00760, >+The, 0.00760, (it, 0.00789, 12+38, 0.01000, system+>, 0.01000, that+>, 0.01000, 8+STABLE, 0.01000, STABLE, 0.01000, >+1, 0.01000, >+2, 0.01000, seems+to, 0.01000, mostly, 0.01000, mass, 0.01000, Nagy, 0.01000, but+>, 0.01000, why+the, 0.01000, a+while, 0.99000, Received*FreeBSD.org>, 0.01000, enabled, 0.01000, (for, 0.01000, this+machine, 0.01000, X-Spambayes-Classification: ham; 0.00 Received: from japan.t-online.private (japan.t-online.co.hu [195.228.243.99]) by people.fsn.hu (Postfix) with ESMTPSA id BB511459FC7 for ; Wed, 22 Sep 2010 15:21:56 +0200 (CEST) Message-ID: <4C9A02F1.6020604@fsn.hu> Date: Wed, 22 Sep 2010 15:21:53 +0200 From: Attila Nagy User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.9) Gecko/20100921 Thunderbird/3.1.4 MIME-Version: 1.0 To: freebsd-fs@FreeBSD.org References: <4C99DC90.70208@fsn.hu> In-Reply-To: <4C99DC90.70208@fsn.hu> Content-Type: text/plain; charset=ISO-8859-2; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: Re: zcolli (zcollide) state, what does znode dying means? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Sep 2010 13:22:03 -0000 On 09/22/10 12:38, Attila Nagy wrote: > I have a machine, which is heavily hammered with file system > operations, running a very recent 8-STABLE. > The symptom is that everything works fine for a few minutes, then a > lot of processes get into zcolli state (according to top). At that > there there are two outcomes: > 1. the disks calm down for a while (for long seconds, there is no, or > very small amount of IO, verified with gstat), top shows nearly 100% > system, a lot of processes are on the run queue (load is in the sky, > around 300 and 1000), all operations stop, top refreshes, but I can't > really execute new programs, then suddenly the zcolli states change > and the IO resumes and the run queue decreases. > 2. the system remains in this state, after 5-10 minutes there is still > no change, only a reset helps (doesn't even react to CTRL-ALT-DEL, but > running programs, like top still refreshes, but no disk IO can be made) It turned out that due to a restart prefetch got enabled. On this machine it made so much extra IO (it does mostly random reads) that it could livelock itself. The only thing I don't understand is why the IO ceased during the mass zcollide period, that seems to be a wait for something scenario (sometimes endlessly), which is bad.