From owner-freebsd-hackers@freebsd.org Tue Sep 15 15:29:28 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4D6E1A0424E for ; Tue, 15 Sep 2015 15:29:28 +0000 (UTC) (envelope-from tobias.oberstein@gmail.com) Received: from mail-wi0-x230.google.com (mail-wi0-x230.google.com [IPv6:2a00:1450:400c:c05::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CFB111DD7 for ; Tue, 15 Sep 2015 15:29:27 +0000 (UTC) (envelope-from tobias.oberstein@gmail.com) Received: by wiclk2 with SMTP id lk2so34313523wic.0 for ; Tue, 15 Sep 2015 08:29:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-type:content-transfer-encoding; bh=Dag8xS+WvHrntn/7P3s3jo8SUfS8LofuKHdIkSg6KDk=; b=miAhIZcEAZh3cGIUHZx6ahm3lSj84VuaptRErEbwAi6+l9iW9pygSHPbgshCCF5ruz f9/GaACtItDg6db8fZvUyhJaVlN4huJ0bq6/TMMopP3jBK7fU/g8lLFuuEBkDQsDGl1a Hs8xIIhZFZKqAms6feEDzrGa6XyHygPX2M/RlRPI8GQ5wprCRqp+jBNpoMGwdF5KZ7vw aQbTV+MkOfbyXAo+4AThfS7+JHU+zhCcoh8jyR/IpQdu2m+MYhNioxqrKSF7T8N0NTyM 7n8pROq07Bw7wMBK7c4he1CkNAefrJhLZkFAZFadYzubxHHHgOnYOujCRmAwS7z+XCw+ izzg== X-Received: by 10.180.9.112 with SMTP id y16mr8703432wia.28.1442330964970; Tue, 15 Sep 2015 08:29:24 -0700 (PDT) Received: from [192.168.1.140] (ppp-82-135-65-249.dynamic.mnet-online.de. [82.135.65.249]) by smtp.googlemail.com with ESMTPSA id go5sm20500666wib.3.2015.09.15.08.29.23 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 15 Sep 2015 08:29:23 -0700 (PDT) Subject: Re: NVMe performance 4x slower than expected To: CadSoftware , freebsd-hackers@freebsd.org References: <551C5A82.2090306@gmail.com> <20150401212303.GB2379@kib.kiev.ua> <5526EA33.6090004@gmail.com> <5527F554.2030806@gmail.com> <1442330273152-6039235.post@n5.nabble.com> From: Tobias Oberstein Message-ID: <55F83952.3070407@gmail.com> Date: Tue, 15 Sep 2015 17:29:22 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <1442330273152-6039235.post@n5.nabble.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Sep 2015 15:29:28 -0000 > I believe FreeBSD 10.2-RELEASE includes an updated nvme (4) driver. Do these > changes fix the performance issues described in this thread? I don't know. The problems with the machine we've deployed however run deeper than "just NVMe": RAM: 3TB .. the FBSD kernel patches that fixed this might be in 10.2 .. dunno amount of PCIx resources: the box has 8x NVMe, 2x SAS controller, 2x dual-port 10GbE .. FBSD freaked out due to this .. there were kernel patches also .. again no clue if those are in mainline now And finally, the bummer: NUMA performance. The box has 4 sockets, 48 cores, and we are moving to 64 cores. My unscientific impression was, that FBSD isn't yet able to cope with that. Anyway, in the meantime, we've moved from SLES 12 to Ubuntu 15.04 ( Linux kernel 3.19.3) .. due to recent Linux developments: blkmq and such. 8kB random write performance now is now at >1 million IOPS (over a test duration of 10h and a 1TB dataset at block device level - RAID-0 over 8x 2TB NVMe). That's nearly 40% more than on SLES 12, and the CPU load is lower also. So we can saturate the hardware IOPS wise. Cheers, /Tobias