From owner-freebsd-performance@FreeBSD.ORG Wed Feb 11 19:33:29 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DFB0A1065672 for ; Wed, 11 Feb 2009 19:33:29 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: from yw-out-2324.google.com (yw-out-2324.google.com [74.125.46.29]) by mx1.freebsd.org (Postfix) with ESMTP id 951A38FC1A for ; Wed, 11 Feb 2009 19:33:29 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: by yw-out-2324.google.com with SMTP id 2so215111ywt.13 for ; Wed, 11 Feb 2009 11:33:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:message-id:subject:from:to:cc :content-type:content-transfer-encoding; bh=Ql9SFdS+eRPTvVdLKhM+D/CNHBwOmgWG1Bu92el2roE=; b=VO6TR032AEV61CLe5vcwi/lzqt75tgCusT8UWPci0ZJN/sjpYtH9Qhl4v1ABuH7ux2 X+G8AUHZg+ztadNk94V3M1eLMZcOrN6uLbvHc5EfYGV049qqlPr1JUl8h5ucMPD/Ycsj ne0cERzIKdcD4yvmCkKqpmBV0bol7IjBvkTqQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=PglEH2fherK4iWho+83gIvkco1pMuRcyo6crsLX/Renf8+58glgQPuneOWWXqy8nMq ZAIcDEDVMK3o32piqKnJuBn7/PL+ziSkVKlhkAHtcx0tQ5iqQ5a6FSs6aXoGUi5fzT6U BJY1lo7G07EgpUmUqukWLNXhmpugJv+vlon+U= MIME-Version: 1.0 Sender: ivoras@gmail.com Received: by 10.142.214.5 with SMTP id m5mr2375851wfg.266.1234378965332; Wed, 11 Feb 2009 11:02:45 -0800 (PST) In-Reply-To: <49927151.2030100@mawer.org> References: <499165F3.6050803@sebster.com> <49918DA6.4020608@sebster.com> <49918E0A.1060500@sebster.com> <49927151.2030100@mawer.org> Date: Wed, 11 Feb 2009 20:02:45 +0100 X-Google-Sender-Auth: 419056312391f1fa Message-ID: <9bbcef730902111102j5ef13491md4aa887de83e55f4@mail.gmail.com> From: Ivan Voras To: Antony Mawer Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org Subject: Re: FreeBSD 7.1 disk performance issue on ESXi 3.5 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Feb 2009 19:33:30 -0000 2009/2/11 Antony Mawer : > How would one go about gathering data on such a scenario to help improve > this? We were planning a project involving VMware deployments with FreeBSD > 7.1 systems in the near future, but if performance is that bad it is likely > to be a show stopper. I have now tested it under ESXi 3.5, and here's what I find: In FreeBSD 7.1 amd64, 4 vCPUs performance for dbench is : 1 proc : 155 MB/s, 2 proc: 175 MB/s, 4 proc: 188 MB/s The same performance *as reported by VMWare's Infrastructure Client* ("performance" tab): around 50 MB/s in all cases Visual inspection of drives' LED indicators (2 drive 10k RPM RAID0 hw array) confirms constant activity. In Ubuntu 8.10 amd64, 4 vCPUs, performance for dbench is : 1 proc: 375 MB/s, 2 proc: 660 MB/s, 4 proc: 1055 MB/s (sic!) The same performance *as reported by VMWare Infrastructure Client*: around 25 MB/s in all cases (sic!) Visual inspection of drives: very sporadic activity The maximum performance expected from this array is around 150 MB/s *at peaks* - there is physically no way it can go above this, so I judge the above measurements bogus. This is all very strange. Someone here is caching more than it should be, and it looks like it's VMWare. It doesn't look as clock skew in the guests since "iostat 1" et al work at about 1sec wallclock time. The "visual inspection" oddity inspired me to do another benchmark: Bonnie++ reports: For FreeBSD: write: 52 MB/s, rewrite: 21 MB/s, read: 45 MB/s For Linux: write: 141 MB/s, rewrite: 55 MB/s, read: 168 MB/s VMWare's Infrastructure Client agrees with these performance measurements in both cases, and drives are blinking as expected. As previously demonstrated by me and others, Linux usually has significantly better file system performance in the non-virtualized case, so the difference could be simply increased by the virtualization.