From owner-freebsd-stable@FreeBSD.ORG Fri Aug 13 16:01:11 2010 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BE0A610656A5 for ; Fri, 13 Aug 2010 16:01:11 +0000 (UTC) (envelope-from oberman@es.net) Received: from mailgw.es.net (mail1.es.net [IPv6:2001:400:201:1::2]) by mx1.freebsd.org (Postfix) with ESMTP id CAA698FC2C for ; Fri, 13 Aug 2010 16:01:10 +0000 (UTC) Received: from ptavv.es.net (ptavv.es.net [IPv6:2001:400:910::29]) by mailgw.es.net (8.14.3/8.14.3) with ESMTP id o7DG19IX004097 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT) for ; Fri, 13 Aug 2010 09:01:09 -0700 Received: from ptavv.es.net (localhost [127.0.0.1]) by ptavv.es.net (Tachyon Server) with ESMTP id 8BDDA1CC3A for ; Fri, 13 Aug 2010 09:01:09 -0700 (PDT) To: stable@freebsd.org Date: Fri, 13 Aug 2010 09:01:09 -0700 From: "Kevin Oberman" Message-Id: <20100813160109.8BDDA1CC3A@ptavv.es.net> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.0.10011, 1.0.148, 0.0.0000 definitions=2010-08-13_05:2010-08-13, 2010-08-13, 1970-01-01 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 ipscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx engine=5.0.0-1005130000 definitions=main-1008130105 Cc: Subject: Inconsistent IO performance X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Aug 2010 16:01:11 -0000 For some time I have seen very odd issues with IO performance on 8-Stable. Going back to November of last year when 8.0 was released, I see variations of up to 22% in identical operations. This is not a degradation as the performance moves up and down. This is a very simplistic case. I have two identical disks (Fujitsu 80G) on a ThinkPad T43 with a 2 GHz CPU and 2G RAM. I run the command: dd bs=516096 if=/dev/ad0 of=/dev/ad2 I do this in single user mode immediately after a boot with no disks mounted for write. Just a 'boot -s', ,Enter> to start the shell, and the dd. I would expect very consistent performance from run to run, but I don't get it. Here are the results since 8.0 was released: Date Xfer rate Kernel date 12/4/09 19,242,573 Nov. 26 kernel (8.0-stable) 12/9/09 18,304,565 Dec. 6 kernel 12/17/0923,676,086 1/5/10 18,648,609 1/14/10 23,488,540 Jan. 6 kernel 1/21/10 19,551,680 Jan. 15 kernel 1/27/10 21,176,385 Jan. 21 kernel 2/5/10 22,387,745 2/11/10 23,387,894 2/17/10 20,412,172 Feb. 16 kernel 2/25/10 22,049,128 3/4/10 22,099,624 Mar. 3 kernel 3/17/10 20,334,896 Mar. 3 kernel 3/31/10 21,655,213 Mar. 25 kernel 4/8/10 19,673,170 4/14/10 22,235,518 4/30/10 21,262,223 Apr. 14 kernel 6/3/10 22,838,125 May 24 kernel 6/17/10 18,481,270 6/28/10 20,958,356 7/8/10 19,698,282 June 28 kernel 7/21/10 23,330,556 7/28/10 20,544,392 July 24 kernel (8.1-stable) 8/13/10 22,093,259 Aug. 9 kernel Note the dramatic differences even on the same kernel. For the December 6 kernel, for example, I see a maximum of 23,676,086 and a minimum of just 18,304,565. ???? Can anyone explain what might be causing such a dramatic difference? I should also note that the system was consistent back in V6 and V7 days. Consistently slow, but consistent. 17.5M was the norm in V6 and 18.0M in V7. The performance jumped to about 19M in March of 09 and jumped to its current speeds with 8.0. So performance has greatly improved to where the slowest times are better than the fastest prior to March of 09. Just very inconsistent. I don't know that anything is wrong, but I'd love to understand why this is happening. -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: oberman@es.net Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751