From owner-freebsd-current@FreeBSD.ORG Wed Apr 15 04:35:12 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6AB2B106566B for ; Wed, 15 Apr 2009 04:35:12 +0000 (UTC) (envelope-from artemb@gmail.com) Received: from mail-ew0-f171.google.com (mail-ew0-f171.google.com [209.85.219.171]) by mx1.freebsd.org (Postfix) with ESMTP id BEFB38FC08 for ; Wed, 15 Apr 2009 04:35:11 +0000 (UTC) (envelope-from artemb@gmail.com) Received: by ewy19 with SMTP id 19so2857590ewy.43 for ; Tue, 14 Apr 2009 21:35:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:message-id:subject:from:to:cc :content-type:content-transfer-encoding; bh=dS+frVOQo9zQdwASbfvloVa9jGTkyUHJi+nYKND5yh0=; b=rHBtPdEzFm8Da91Knfe0Krthd5L0mWjZAD3ktQhhjDHBkuX/tOK68L+GnwmwwfKhEt Fbmz9Zo9G4TxMHTBjNsbnxaovJf3QsoOch+JatevbBLlJj9I/q/z3t6HZHkUo+Uc5ksV g714RTFs0mQwwE4yHyInrZ7VhgatxBIe0rs1o= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=iuxTfkFWaqJ/dTmlWdVghoYxDJA9wNScaCyTdjfEifyYEz8z+Pc1bYZGzNxnW3Aerx oz/Lx0e0TrmFrQNBWp49sJsfT48QqKPbTbE2PeJpAlRSEq0KtWEr+QoErk+C8+UNRoqD GYkO2sxQeBuAGJP7NBFwjtWGHUFxvt4QceGSo= MIME-Version: 1.0 Sender: artemb@gmail.com Received: by 10.210.66.13 with SMTP id o13mr3665072eba.96.1239770110492; Tue, 14 Apr 2009 21:35:10 -0700 (PDT) In-Reply-To: References: <49C2CFF6.8070608@egr.msu.edu> <08D7DC2A-68BE-47B6-8D5D-5DE6B48F87E5@wanderview.com> Date: Tue, 14 Apr 2009 21:35:10 -0700 X-Google-Sender-Auth: 295a3e47273c6058 Message-ID: From: Artem Belevich To: Ben Kelly Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-current@freebsd.org Subject: Re: [patch] zfs livelock and thread priorities X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Apr 2009 04:35:12 -0000 I'll give it a try in a few days. I'll let you know how it went. BTW, now that you're tinkering with ZFS threads and priorities, whould you by any chance have any idea why zfs scrub is so painfully slow on -current? When I start scrub on my -stable box, it pretty much runs full speed -- I can see disks under load all the time. However on -current scrub seems to run in small bursts. Disks get busy for a second or so and then things get quiet for about five seconds or so and this pattern repeats over and over. --Artem On Tue, Apr 14, 2009 at 7:32 PM, Ben Kelly wrote: > On Apr 14, 2009, at 11:50 AM, Ben Kelly wrote: >> >> On Apr 13, 2009, at 7:36 PM, Artem Belevich wrote: >>> >>> Tried your patch that used PRIBIO+{1,2} for priorities with -current >>> r191008 and the kernel died with "spinlock held too long" panic. >>> Actually, there apparently were two instances of panic on different >>> cores.. >>> >>> Here's output of "alltrace" and "ps" after the crash: >>> http://pastebin.com/f140f4596 >>> >>> I've reverted the change and kernel booted just fine. >>> >>> The box is quad-core with two ZFS pools -- one single-disk and another >>> one is a two-disk mirror. Freebsd is installed on UFS partitions, ZFS >>> is used for user stuff only. >> >> Thanks for the report! >> >> I don't have a lot of time to look at this today, but it appears that >> there is a race condition on SMP machines when setting the priority >> immediately after the kproc is spawned. =A0As a quick hack I tried addin= g a >> pause between the kproc_create() and the sched_prio(). =A0Can you try th= is >> patch? >> >> >> =A0http://www.wanderview.com/svn/public/misc/zfs_livelock/zfs_thread_pri= ority.diff >> >> I'll try to take a closer look at this later in the week. > > Sorry for replying to my own e-mail, but I've updated the patch again wit= h a > less hackish approach. =A0(At the same URL above.) =A0I added a new > kproc_create_priority() function to set the priority of the new thread > before its first scheduled. =A0This should avoid any SMP races with setti= ng > the priority from an external thread. > > If you would be willing to try the test again with this new patch I would > appreciate it. > > Thanks! > > - Ben >