From owner-freebsd-fs@FreeBSD.ORG Thu Aug 9 16:07:53 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C3C141065673 for ; Thu, 9 Aug 2012 16:07:53 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.3.230]) by mx1.freebsd.org (Postfix) with ESMTP id 4C49C8FC14 for ; Thu, 9 Aug 2012 16:07:52 +0000 (UTC) Received: from dcave.digsys.bg (dcave.digsys.bg [192.92.129.5]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.5/8.14.5) with ESMTP id q79FdZoA032835 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Thu, 9 Aug 2012 18:39:36 +0300 (EEST) (envelope-from daniel@digsys.bg) Message-ID: <5023D9B7.20001@digsys.bg> Date: Thu, 09 Aug 2012 18:39:35 +0300 From: Daniel Kalchev User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:10.0.6esrpre) Gecko/20120728 Thunderbird/10.0.6 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit Subject: ZFS scrub CPU bound? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Aug 2012 16:07:53 -0000 I have built myself an server with SSD only drives. That thing goes "fast", like during scrub # zpool status pool: system state: ONLINE scan: scrub in progress since Thu Aug 9 18:30:46 2012 87.1G scanned out of 193G at 602M/s, 0h3m to go 0 repaired, 45.06% done config: NAME STATE READ WRITE CKSUM system ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 da0p3 ONLINE 0 0 0 da1p3 ONLINE 0 0 0 da2p3 ONLINE 0 0 0 da3p3 ONLINE 0 0 0 But, what I wonder is top -S # top -S last pid: 7912; load averages: 9.89, 3.26, 1.29 up 0+06:03:59 18:32:04 146 processes: 3 running, 142 sleeping, 1 waiting CPU: 0.4% user, 0.0% nice, 51.4% system, 0.8% interrupt, 47.3% idle Mem: 2171M Active, 1541M Inact, 5407M Wired, 25M Cache, 416K Buf, 22G Free Swap: 8192M Total, 8192M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 32 155 ki31 0K 512K CPU31 31 190.2H 1818.46% idle 0 root 268 -8 0 0K 4288K - 0 141:25 1261.33% kernel 4 root 4 -8 - 0K 80K CPU30 30 9:13 95.17% zfskern 13 root 3 -8 - 0K 48K - 4 5:37 42.97% geom 12 root 66 -84 - 0K 1056K WAIT 0 6:19 30.86% intr [...] It seems that zfskern will top to 100%. This is an 32 core system, and as you see scrub, at 600MB/sec is able to eat 16 cores (from 2x 2.2 GHz Opteron 6274). There is high load on geom as well... but geom does go over 100% CPU, so I suppose it scales. What might be there in ZFS code that is single-CPU bound? freebsd 9-stable. Daniel