From owner-freebsd-current@FreeBSD.ORG Tue May 26 16:16:26 2015 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9BF4D16B; Tue, 26 May 2015 16:16:26 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-pa0-x22a.google.com (mail-pa0-x22a.google.com [IPv6:2607:f8b0:400e:c03::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 681FF814; Tue, 26 May 2015 16:16:26 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: by pacwv17 with SMTP id wv17so96202659pac.2; Tue, 26 May 2015 09:16:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=NQMoZEdSLexEl/ST0uhsZWU66e2x5j31lGf1bLMt7z8=; b=iGWwb3kDHaoMLSguGK3GxyWphHXRgFccYMZ0SMiKxJJ/P+STShdkXL8kkgeGXYzEJm NIj6ACjxmglqqS46aouINDEvohiyP8oR/xkrqY0PEU10nYNoj0Ir6q4earNnOUvcYpB5 4WiVxGeFmYtgu2TVgFSZCfoDldbOO4TOgYOeByFO9R418DxWH2HwHxUrhwy9TSTKGAbl 5kWsySUUq5XDj1kaPjPIBj67VVx6q51Pp77tolB+cIZDqoe3dToS6aeCQtfsPeGigRQJ apq3WdRnqAMHq7kKdCBROMjzpSd5ryNdOEY47ukV4JSv3DeXTcdsANE8C5gyRzfhvxmV S/GA== X-Received: by 10.70.56.4 with SMTP id w4mr50731740pdp.148.1432656985893; Tue, 26 May 2015 09:16:25 -0700 (PDT) Received: from mavbook.mavhome.dp.ua (he85-184.ewnet.net. [72.52.85.184]) by mx.google.com with ESMTPSA id u10sm13497294pbs.30.2015.05.26.09.16.24 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 26 May 2015 09:16:24 -0700 (PDT) Sender: Alexander Motin Message-ID: <55649C57.1080906@FreeBSD.org> Date: Tue, 26 May 2015 09:16:23 -0700 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: Neel Natu , Garrett Cooper CC: FreeBSD Current , "freebsd-testing@freebsd.org" , Peter Grehan Subject: Re: cam(4) timeouts in bhyve/kyua runs up on Jenkins? References: <28845361-E624-4CB9-B255-1B2C1FD0DF28@gmail.com> <4BEE0A54-3AD2-42DB-AF4E-2B0A5E34DF4E@gmail.com> <553F3CA7.9090601@FreeBSD.org> <9FBE053A-971A-433E-9B88-D62CF9273CFB@gmail.com> In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 May 2015 16:16:26 -0000 On 26.05.2015 09:06, Neel Natu wrote: > On Mon, May 25, 2015 at 1:40 PM, Garrett Cooper wrote: >> On Apr 28, 2015, at 0:54, Alexander Motin wrote: >>> On 27.04.2015 21:17, Garrett Cooper wrote: >>>> On Apr 27, 2015, at 11:16, Garrett Cooper >>>> wrote: >>>> >>>>> I was looking at the console log for the latest kyua run and I’ve >>>>> noticed that it’s timing out a bit more [1] than it was >>>>> previously [2]. I’ve seen some of your commits recently to cam(4) >>>>> dealing with bhyve — has there been a performance regression >>>>> there? Thanks! -NGie >>>>> >>>>> 1. >>>>> https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests2/940/console >>>>> 2. >>>>> https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests2/983/console >>>> >>>> (Sorry for not being more explicit for the archives) These are the >>>> timeouts I’m referring to: >>>> >>>> ahcich0: is 00000000 cs 00000000 ss 1f000000 rs 1f000000 tfd 50 >>>> serr 00000000 cmd 1000dc17 (ada0:ahcich0:0:0:0): >>>> WRITE_FPDMA_QUEUED. ACB: 61 08 a8 54 1e 40 00 00 00 00 00 00 >>>> (ada0:ahcich0:0:0:0): CAM status: Command timeout >>>> (ada0:ahcich0:0:0:0): Retrying command > > Can you try to reproduce with a virtio-blk instead of ahci-hd? > > It will help narrow down the problem because both ahci-hd and > virtio-blk emulations share a common backend to read/write to the > disk. virtio-blk has no command timeouts, so it may be problematic to notice delayed commands there, unless they stuck forever. This won't be fair comparison. Though if not use TRIM, virtio-blk should be more efficient now, as it is supposed to. >>> Last time I was more working on bhyve host disk emulation, rather then >>> on cam(4) running on guest. Considering that, what guest and what host >>> versions are you running? Is there any other load on host except this VM >>> that could cause I/O delays high enough to trigger timeouts? What are >>> you using to back the virtual disk (file, zvol, ...)? >> >> I have no idea what the Jenkins slaves are running in terms of configuration/version/etc. You’ll have to ask jenkins-admin@… >> Thanks! >> -NGie -- Alexander Motin