From owner-freebsd-testing@FreeBSD.ORG Tue May 26 16:06:26 2015 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 51647DB2; Tue, 26 May 2015 16:06:26 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: from mail-wi0-x234.google.com (mail-wi0-x234.google.com [IPv6:2a00:1450:400c:c05::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DBD40330; Tue, 26 May 2015 16:06:25 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: by wichy4 with SMTP id hy4so87693808wic.1; Tue, 26 May 2015 09:06:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=kVywcyXNcNW7zyOD4+QYRPocdK7dprygcYkQWa6cKEM=; b=bfKiiDBoOFU+DwrBxZojLWnVRqlXfofVGqMCG7wyHK0qtF8js8bSM3K7tcaPdK3tYg KwL1PxvWNY9VoS1ucNK99Acz24Wgut++wugpvoicF1sgCm/oSBgBI8rhn673odKIXJAG KtMtm6MiVDwQVR9zs3COA1/gReBIJ2c42n8vwf8WUpu5RkX6D0hQ/4/RnaEixXswypYY pp08EzMzslcr1WpW1C+Loh1F/PtWM6SlM0UIIYSa2llaOCKhilCCfkcnverccCSvOO7X vf5h0kJqCUdeZH0buw0aXMEmXk1hHZXF9HPGsqxPAY6itS8G2MWh8vD0diq10PhuqfY7 JAMQ== MIME-Version: 1.0 X-Received: by 10.180.90.202 with SMTP id by10mr17948316wib.62.1432656384309; Tue, 26 May 2015 09:06:24 -0700 (PDT) Received: by 10.27.52.18 with HTTP; Tue, 26 May 2015 09:06:24 -0700 (PDT) In-Reply-To: <9FBE053A-971A-433E-9B88-D62CF9273CFB@gmail.com> References: <28845361-E624-4CB9-B255-1B2C1FD0DF28@gmail.com> <4BEE0A54-3AD2-42DB-AF4E-2B0A5E34DF4E@gmail.com> <553F3CA7.9090601@FreeBSD.org> <9FBE053A-971A-433E-9B88-D62CF9273CFB@gmail.com> Date: Tue, 26 May 2015 09:06:24 -0700 Message-ID: Subject: Re: cam(4) timeouts in bhyve/kyua runs up on Jenkins? From: Neel Natu To: Garrett Cooper Cc: Alexander Motin , FreeBSD Current , "freebsd-testing@freebsd.org" , Peter Grehan Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 May 2015 16:06:26 -0000 Hi NGie, On Mon, May 25, 2015 at 1:40 PM, Garrett Cooper wro= te: > On Apr 28, 2015, at 0:54, Alexander Motin wrote: > >> Hi. >> >> On 27.04.2015 21:17, Garrett Cooper wrote: >>> On Apr 27, 2015, at 11:16, Garrett Cooper >>> wrote: >>> >>>> I was looking at the console log for the latest kyua run and I=E2=80= =99ve >>>> noticed that it=E2=80=99s timing out a bit more [1] than it was >>>> previously [2]. I=E2=80=99ve seen some of your commits recently to cam= (4) >>>> dealing with bhyve =E2=80=94 has there been a performance regression >>>> there? Thanks! -NGie >>>> >>>> 1. >>>> https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests2/940/console >>>> 2. >>>> https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests2/983/console >>> >>> (Sorry for not being more explicit for the archives) These are the >>> timeouts I=E2=80=99m referring to: >>> >>> ahcich0: is 00000000 cs 00000000 ss 1f000000 rs 1f000000 tfd 50 >>> serr 00000000 cmd 1000dc17 (ada0:ahcich0:0:0:0): >>> WRITE_FPDMA_QUEUED. ACB: 61 08 a8 54 1e 40 00 00 00 00 00 00 >>> (ada0:ahcich0:0:0:0): CAM status: Command timeout >>> (ada0:ahcich0:0:0:0): Retrying command Can you try to reproduce with a virtio-blk instead of ahci-hd? It will help narrow down the problem because both ahci-hd and virtio-blk emulations share a common backend to read/write to the disk. best Neel >> >> Last time I was more working on bhyve host disk emulation, rather then >> on cam(4) running on guest. Considering that, what guest and what host >> versions are you running? Is there any other load on host except this VM >> that could cause I/O delays high enough to trigger timeouts? What are >> you using to back the virtual disk (file, zvol, ...)? > > I have no idea what the Jenkins slaves are running in terms of configurat= ion/version/etc. You=E2=80=99ll have to ask jenkins-admin@=E2=80=A6 > Thanks! > -NGie