From owner-freebsd-scsi@freebsd.org Sat Jul 1 07:58:28 2017 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BD7BBD9F341 for ; Sat, 1 Jul 2017 07:58:28 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id A41807D574 for ; Sat, 1 Jul 2017 07:58:28 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: by mailman.ysv.freebsd.org (Postfix) id A082BD9F340; Sat, 1 Jul 2017 07:58:28 +0000 (UTC) Delivered-To: scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A0190D9F33F for ; Sat, 1 Jul 2017 07:58:28 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 421847D573; Sat, 1 Jul 2017 07:58:28 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mh0.gentlemail.de (ezra.dcm1.omnilan.net [IPv6:2a00:e10:2800::a135]) by mx0.gentlemail.de (8.14.5/8.14.5) with ESMTP id v617wOdX060491; Sat, 1 Jul 2017 09:58:24 +0200 (CEST) (envelope-from freebsd@omnilan.de) Received: from titan.inop.mo1.omnilan.net (s1.omnilan.de [217.91.127.234]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id 0B5F35A8; Sat, 1 Jul 2017 09:58:23 +0200 (CEST) Message-ID: <5957561F.7030906@omnilan.de> Date: Sat, 01 Jul 2017 09:58:23 +0200 From: Harry Schmalzbauer Organization: OmniLAN User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; de-DE; rv:1.9.2.8) Gecko/20100906 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: Alexander Motin , scsi@freebsd.org Subject: Re: not a reproducable regression [Was :Re: bhyve ahcich0: Timeout on slot 0 port 0,, regression with stable/11->releng/11.1] References: <5956A3DF.8060109@omnilan.de> <5956A4E0.3030108@omnilan.de> In-Reply-To: <5956A4E0.3030108@omnilan.de> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]); Sat, 01 Jul 2017 09:58:24 +0200 (CEST) X-Milter: Spamilter (Reciever: mx0.gentlemail.de; Sender-ip: ; Sender-helo: mh0.gentlemail.de; ) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Jul 2017 07:58:28 -0000 Bezüglich Harry Schmalzbauer's Nachricht vom 30.06.2017 21:22 (localtime): > Bezüglich Harry Schmalzbauer's Nachricht vom 30.06.2017 21:17 (localtime): >> Hello, >> >> on releng/11.1 I noticed a severe performace degradation during file >> unlinking in a FreeBSD guest. Host was running quiet recent stable/11 >> before. >> >> On the host, the vm is started with >> ahci,hd:/dev/adaN >> >> The guest attaches: >> ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 >> ada0: ACS-2 ATA SATA 3.x device >> >> The geust has very high Sys-load during unlinking (50-75%@2 cores). >> Also, the host logs these errors: > Sorry, not the host logs these errors, but the guest, obviously. > After a reboot, I couldn't reproduce the timeouts, nor the completion time multiplication. It takes ~8 minutes like it alwas was… Sorry for late falsifying. I'm still wondering what these timeout error codes translate to. And I can still see very high CPU load in the guest during unlinking (obj-tree of usr/src). While systat reports less than 10irqs/s for AHCI0, "intr" consumes 30-800% at times (with 4 vCPUs). "bufdaemon" and "rm" consume 10-100% each. So total average is about 2 completely loaded cores for 'rm' running in a FreeBSD guest. Any hints regarding timeout codes or how to trace where CPU cycles drain are highly appreciated! Thanks, -harry