From owner-freebsd-hackers@FreeBSD.ORG Thu Jan 17 18:43:53 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 8D375D0E for ; Thu, 17 Jan 2013 18:43:53 +0000 (UTC) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from wojtek.tensor.gdynia.pl (wojtek.tensor.gdynia.pl [188.252.31.196]) by mx1.freebsd.org (Postfix) with ESMTP id 045599B5 for ; Thu, 17 Jan 2013 18:43:52 +0000 (UTC) Received: from wojtek.tensor.gdynia.pl (localhost [127.0.0.1]) by wojtek.tensor.gdynia.pl (8.14.5/8.14.5) with ESMTP id r0HIhphY000441 for ; Thu, 17 Jan 2013 19:43:51 +0100 (CET) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from localhost (wojtek@localhost) by wojtek.tensor.gdynia.pl (8.14.5/8.14.5/Submit) with ESMTP id r0HIhpRR000438 for ; Thu, 17 Jan 2013 19:43:51 +0100 (CET) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Date: Thu, 17 Jan 2013 19:43:51 +0100 (CET) From: Wojciech Puchar To: freebsd-hackers@freebsd.org Subject: stupid UFS behaviour on random writes Message-ID: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.7 (wojtek.tensor.gdynia.pl [127.0.0.1]); Thu, 17 Jan 2013 19:43:51 +0100 (CET) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jan 2013 18:43:53 -0000 create 10GB file (on 2GB RAM machine, with some swap used to make sure little cache would be available for filesystem. dd if=/dev/zero of=file bs=1m count=10k block size is 32KB, fragment size 4k now test random read access to it (10 threads) randomio test 10 0 0 4096 normal result on such not so fast disk in my laptop. 118.5 | 118.5 5.8 82.3 383.2 85.6 | 0.0 inf nan 0.0 nan 138.4 | 138.4 3.9 72.2 499.7 76.1 | 0.0 inf nan 0.0 nan 142.9 | 142.9 5.4 69.9 297.7 60.9 | 0.0 inf nan 0.0 nan 133.9 | 133.9 4.3 74.1 480.1 75.1 | 0.0 inf nan 0.0 nan 138.4 | 138.4 5.1 72.1 380.0 71.3 | 0.0 inf nan 0.0 nan 145.9 | 145.9 4.7 68.8 419.3 69.6 | 0.0 inf nan 0.0 nan systat shows 4kB I/O size. all is fine. BUT random 4kB writes randomio test 10 1 0 4096 total | read: latency (ms) | write: latency (ms) iops | iops min avg max sdev | iops min avg max sdev --------+-----------------------------------+---------------------------------- 38.5 | 0.0 inf nan 0.0 nan | 38.5 9.0 166.5 1156.8 261.5 44.0 | 0.0 inf nan 0.0 nan | 44.0 0.1 251.2 2616.7 492.7 44.0 | 0.0 inf nan 0.0 nan | 44.0 7.6 178.3 1895.4 330.0 45.0 | 0.0 inf nan 0.0 nan | 45.0 0.0 239.8 3457.4 522.3 45.5 | 0.0 inf nan 0.0 nan | 45.5 0.1 249.8 5126.7 621.0 results are horrific. systat shows 32kB I/O, gstat shows half are reads half are writes. Why UFS need to read full block, change one 4kB part and then write back, instead of just writing 4kB part?