From owner-freebsd-questions@FreeBSD.ORG Sat Feb 20 21:48:49 2010 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 929EF106566B for ; Sat, 20 Feb 2010 21:48:49 +0000 (UTC) (envelope-from dan@dan.emsphone.com) Received: from email1.allantgroup.com (email1.emsphone.com [199.67.51.115]) by mx1.freebsd.org (Postfix) with ESMTP id 5A5188FC0C for ; Sat, 20 Feb 2010 21:48:49 +0000 (UTC) Received: from dan.emsphone.com (dan.emsphone.com [199.67.51.101]) by email1.allantgroup.com (8.14.0/8.14.0) with ESMTP id o1KLmmAZ046864 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Sat, 20 Feb 2010 15:48:48 -0600 (CST) (envelope-from dan@dan.emsphone.com) Received: from dan.emsphone.com (smmsp@localhost [127.0.0.1]) by dan.emsphone.com (8.14.4/8.14.3) with ESMTP id o1KLmlXV012465 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Sat, 20 Feb 2010 15:48:48 -0600 (CST) (envelope-from dan@dan.emsphone.com) Received: (from dan@localhost) by dan.emsphone.com (8.14.4/8.14.3/Submit) id o1KLmlvm012437; Sat, 20 Feb 2010 15:48:47 -0600 (CST) (envelope-from dan) Date: Sat, 20 Feb 2010 15:48:45 -0600 From: Dan Nelson To: Boris Samorodov Message-ID: <20100220214845.GI70798@dan.emsphone.com> References: <97752307@ipt.ru> <20100220202614.GG70798@dan.emsphone.com> <88797383@ipt.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <88797383@ipt.ru> X-OS: FreeBSD 7.2-STABLE User-Agent: Mutt/1.5.20 (2009-06-14) X-Virus-Scanned: clamav-milter 0.95.3 at email1.allantgroup.com X-Virus-Status: Clean X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.0.2 (email1.allantgroup.com [199.67.51.78]); Sat, 20 Feb 2010 15:48:48 -0600 (CST) X-Scanned-By: MIMEDefang 2.45 Cc: freebsd-questions@freebsd.org Subject: Re: strange disk activity X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Feb 2010 21:48:49 -0000 In the last episode (Feb 20), Boris Samorodov said: > On Sat, 20 Feb 2010 14:26:15 -0600 Dan Nelson wrote: > > In the last episode (Feb 20), Boris Samorodov said: > > > Hello List, > > > > > > I've got a very strange disk activity: > > > ----- > > > % iostat -xw60 da0 > > > extended device statistics > > > device r/s w/s kr/s kw/s wait svc_t b > > > da0 43.2 204.4 971.9 10917.2 0 30.7 30 > > > extended device statistics > > > device r/s w/s kr/s kw/s wait svc_t b > > > da0 5.1 274.6 72.7 15206.2 0 50.4 27 > > > > > > If I'm not mistaken it's approx. 15Mb/sec. Which means more than 1000 Tb a > > > day! > > > > > > How can I find which program is trashing the disk? The system is > > > FreeBSD-7.0 with 15 jails. Thanks for your help. > > > Something like this would be a good start: > > > dtrace -n 'syscall::write:entry { @dist[pid,execname] = sum(arg2); }' > > > Let it run a few seconds, then hit ^C, and it will print the total bytes > > written by each process during that period. You may also have to trace > > writev and pwrite, since FreeBSD's dtrace doesn't include the sysinfo > > provider that would let you count them all at once. > > > Data written to swap or mmap'ped files won't register here, but most > > heavy writes aren't done with mmap. > > Thanks, Dan! Dtrace is really a way to go. > > Unfortunately dtrace is not available at 7.0. That means that > the system should be updated to 7-STABLE. > > Are there other possibilities meanwhile? ktrace -d -i -p 0 ; sleep 10 ; ktrace -C , then run kdump -m64 and search for large numbers of writes in the output. "-d -i -p0" selects all current and future children of pid 0, which will trace all processes. "-m64" limits the I/O dump size to 64 bytes. -- Dan Nelson dnelson@allantgroup.com