From owner-freebsd-hackers@FreeBSD.ORG Thu Apr 2 14:12:52 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7AD22F2E; Thu, 2 Apr 2015 14:12:52 +0000 (UTC) Received: from mail-wi0-x230.google.com (mail-wi0-x230.google.com [IPv6:2a00:1450:400c:c05::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 04E31AB2; Thu, 2 Apr 2015 14:12:52 +0000 (UTC) Received: by wixm2 with SMTP id m2so56813743wix.0; Thu, 02 Apr 2015 07:12:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=lP3U3n8N6n/PwWkdivTiMAgDcWYjbF/TX/IOEYFNHN0=; b=fp0kSviDgo0hjVWTDQlEHhqQGDRWAvGcDE31plecXZ0ZRRQaTqIDKwF62Up/jX2vfU XJwP1q/OP1onyEUFNZMYTlbeOjf4I0pHp7Q2lLCk/z879A0aVIffJbNjuO/5gxbO+aPs jc5czE+oXOZz+ugRmv77BfpCV5xDAH/OziYZAoN3U/F64b5y2/mi+0JJ0CSQq0MbXu4e 8UUSXD0eYeV5JK1M18zru11ZyRRIookurdRcXyXh43cC9/oE0VFzrFbjefMmTw8Gg+c9 T+CyT5Qq5VLMMtgRMhaFjOjWZutKzYiI6dj7td1deFispkoQZsWazt9rR8vCERoDhbq7 RU7g== X-Received: by 10.194.187.236 with SMTP id fv12mr95977834wjc.131.1427983969519; Thu, 02 Apr 2015 07:12:49 -0700 (PDT) Received: from [192.168.43.73] ([89.15.239.102]) by mx.google.com with ESMTPSA id dj4sm7443820wjc.13.2015.04.02.07.12.48 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 02 Apr 2015 07:12:48 -0700 (PDT) Message-ID: <551D4E5F.9090400@gmail.com> Date: Thu, 02 Apr 2015 16:12:47 +0200 From: Tobias Oberstein User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: Jim Harris Subject: Re: NVMe performance 4x slower than expected References: <551BC57D.5070101@gmail.com> <551C5A82.2090306@gmail.com> <20150401212303.GB2379@kib.kiev.ua> <551C6B62.7080205@gmail.com> In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Cc: Konstantin Belousov , "freebsd-hackers@freebsd.org" , Michael Fuckner , Alan Somers X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Apr 2015 14:12:52 -0000 > You can also try a debug tunable that is in the nvme driver. > > hw.nvme.per_cpu_io_queues=0 I have rerun tests with kernel that has INVARIANTS off, and above sysctl in loader.conf. Results are the same. vmstat now: root@s4l-zfs:~/oberstet # vmstat -ia | grep nvme irq371: nvme0 8 0 irq372: nvme0 7478 0 irq373: nvme1 8 0 irq374: nvme1 7612 0 irq375: nvme2 8 0 irq376: nvme2 7695 0 irq377: nvme3 7 0 irq378: nvme3 7716 0 irq379: nvme4 8 0 irq380: nvme4 7622 0 irq381: nvme5 7 0 irq382: nvme5 7561 0 irq383: nvme6 8 0 irq384: nvme6 7609 0 irq385: nvme7 7 0 irq386: nvme7 15373417 1174 === I was advised (off list) to run tests against a pure ramdisk. Here are results from a single socket E3: https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd_ramdisk.md#xeon-e3-machine and here are results for the 48 core box https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd_ramdisk.md#48-core-big-machine Performance with this box is 1/10 on this test compared to single socket E3! Something is severely wrong. It seems, there might be multiple issues (not only NVMe). And this is after already running with 3 patches to make it even boot. Well. I'm running out of ideas to try, and also patience with the users waiting for this box =( /Tobias