From owner-freebsd-hackers@FreeBSD.ORG Wed Jan 16 00:06:20 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 070A2153 for ; Wed, 16 Jan 2013 00:06:20 +0000 (UTC) (envelope-from dieterbsd@gmail.com) Received: from mail-ie0-f176.google.com (mail-ie0-f176.google.com [209.85.223.176]) by mx1.freebsd.org (Postfix) with ESMTP id D6F266F5 for ; Wed, 16 Jan 2013 00:06:19 +0000 (UTC) Received: by mail-ie0-f176.google.com with SMTP id 13so1387601iea.7 for ; Tue, 15 Jan 2013 16:06:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=9Q06EAFIL9n7zKrWxXjz4evGrL/yXEq+8F/usvqL82k=; b=y1jjGDbQOtC5eD2yZnaD3XkMiANd829QZODLwPXFyuSoRRMHyPr2c2bfzF9OwCMK/o CGKvnY7v4wQn1jQb1iYA+AJqqgF+rKzBFaEN6qKBM0U3NjV/pGMoGXfjix5o8kcZiQZ2 ScWe7HDOucxlYog64+RhsnEtqdfL/qVyG5gdV4LU7zvP++NSjqPsUydSfDqhe7uzHREX a4OaR9StD03lob/MiC39ATqPPrOyZp6IK3zVX/r4yQ21TKtd+zNA5FD4jdIc09ksel7x Li8GvF8HvNo3/EuVJql6iEMpjevyK7hP/KeYYG7x/2H6649xGCWAeaK3CbEjJyMQhSo2 t/wQ== MIME-Version: 1.0 Received: by 10.50.151.211 with SMTP id us19mr3408858igb.84.1358294777605; Tue, 15 Jan 2013 16:06:17 -0800 (PST) Received: by 10.64.107.196 with HTTP; Tue, 15 Jan 2013 16:06:17 -0800 (PST) Date: Tue, 15 Jan 2013 16:06:17 -0800 Message-ID: Subject: Re: IBM blade server abysmal disk write performances From: Dieter BSD To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jan 2013 00:06:20 -0000 Karim writes: > dd to the > raw drive and no compression/encryption or some other features, just a > naive boot off a live 9.1 CD then dd (see below). The following results > have been gathered on the FreeBSD 9.1 system: > > # dd if=/dev/zero of=toto count=100 > 100+0 records in > 100+0 records out > 51200 bytes transferred in 1.057507 secs (48416 bytes/sec) By "raw drive" I meant something like dd if=/dev/zero of=/dev/da0 bs=1m count=1000 "of=toto" implies that you are using a filesystem. (FFS? ZFS? other?) Matthew writes: >> But dmesg claims "Command Queueing enabled", so you should be >> getting more than one op per rev, and writes should be fast. > > Queueing would only help if your load threw multiple ops at the drive > before waiting for any of them to complete. I'd expect a dd to a raw > device to throw a single, wait for it to return complete, then throw > the next, leading to no more than 1 op per rev. I see a huge speedup from NCQ on both raw disks and with FFS/su. Without NCQ I only get 6% of the expected performance, even with a large blocksize. The kernel must be doing write-behind even to a raw disk, otherwise waiting for write(2) to return before issuing the next write would slow it down as Matthew suggests. Writing an entire 3TB disk (raw disk, no fs) gives: 21378.98 real 2.00 user 440.98 sys or 140 MB/s (133 MiB/s) on slow controller in PCIe x1 slot. The same test on the same make & model disk on a much faster controller in the chipset takes over 10x as long because FreeBSD does not support NCQ on that controller. :-( Karim's data sure looks like 1 op per rev. Either it isn't really doing NCQ or the filesystem is doing something to keep NCQ from being effective. For example, mounting the fs with the sync option would probably have that effect.