From owner-freebsd-fs@FreeBSD.ORG Sun Jun 13 00:08:03 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 33859106564A; Sun, 13 Jun 2010 00:08:03 +0000 (UTC) (envelope-from hiroshi@soupacific.com) Received: from mail.soupacific.com (mail.soupacific.com [211.19.53.201]) by mx1.freebsd.org (Postfix) with ESMTP id F29338FC0C; Sun, 13 Jun 2010 00:08:02 +0000 (UTC) Received: from [127.0.0.1] (unknown [192.168.1.239]) by mail.soupacific.com (Postfix) with ESMTP id 9EDD26B8E5; Sat, 12 Jun 2010 23:59:26 +0000 (UTC) Message-ID: <4C14215D.9090304@soupacific.com> Date: Sun, 13 Jun 2010 09:07:57 +0900 From: "hiroshi@soupacific.com" User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4 MIME-Version: 1.0 To: Mikolaj Golub References: <4BCFA4C2.6000109@soupacific.com><4BCFB1C5.5000908@soupacific.com> <4BD01800.9040901@soupacific.com><4BD0438B.5080308@soupacific.com> <4BD0E432.1000108@soupacific.com><20100423061521.GC1670@garage.freebsd.pl><4BD17B0D.5080601@soupacific.com> <4C10B526.4040908@soupacific.com><20100612104336.GA2253@garage.freebsd.pl><4C1372E0.1000903@soupacific.com><20100612142311.GF2253@garage.freebsd.pl><4C139F9C.2090305@soupacific.com> <86iq5oc82y.fsf@kopusha.home.net> In-Reply-To: <86iq5oc82y.fsf@kopusha.home.net> Content-Type: text/plain; charset=ISO-2022-JP Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, Pawel Jakub Dawidek Subject: Re: FreeBSD 8.1 and HAST X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Jun 2010 00:08:03 -0000 Thanks > It would be good to have all.log enabled in newsyslog.conf and provide the > output from there so all lines are in one log and it is clear which message > appeared earlier. Also the logs from the primary could be useful too. Could you tell me how to modify newsyslog.conf to one log file ? >BTW, this message lies that it is from the disk thread. It is from the >send thread too. See the attached patch. I did, After your your instruction for newsyslog.conf. I will send primary and secondary both log files. Thanks Hiroshi From owner-freebsd-fs@FreeBSD.ORG Sun Jun 13 00:36:37 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6837B106567C for ; Sun, 13 Jun 2010 00:36:37 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta04.emeryville.ca.mail.comcast.net (qmta04.emeryville.ca.mail.comcast.net [76.96.30.40]) by mx1.freebsd.org (Postfix) with ESMTP id 4B2A38FC19 for ; Sun, 13 Jun 2010 00:36:36 +0000 (UTC) Received: from omta23.emeryville.ca.mail.comcast.net ([76.96.30.90]) by qmta04.emeryville.ca.mail.comcast.net with comcast id VBal1e0011wfjNsA4CccPw; Sun, 13 Jun 2010 00:36:36 +0000 Received: from koitsu.dyndns.org ([98.248.46.159]) by omta23.emeryville.ca.mail.comcast.net with comcast id VCcb1e0053S48mS8jCcb8t; Sun, 13 Jun 2010 00:36:36 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 32A9C9B418; Sat, 12 Jun 2010 17:36:35 -0700 (PDT) Date: Sat, 12 Jun 2010 17:36:35 -0700 From: Jeremy Chadwick To: hiroshi@soupacific.com Message-ID: <20100613003635.GA60012@icarus.home.lan> References: <4BD0E432.1000108@soupacific.com> <20100423061521.GC1670@garage.freebsd.pl> <4BD17B0D.5080601@soupacific.com> <4C10B526.4040908@soupacific.com> <20100612104336.GA2253@garage.freebsd.pl> <4C1372E0.1000903@soupacific.com> <20100612142311.GF2253@garage.freebsd.pl> <4C139F9C.2090305@soupacific.com> <86iq5oc82y.fsf@kopusha.home.net> <4C14215D.9090304@soupacific.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4C14215D.9090304@soupacific.com> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-fs@freebsd.org, Pawel Jakub Dawidek Subject: Re: FreeBSD 8.1 and HAST X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Jun 2010 00:36:37 -0000 On Sun, Jun 13, 2010 at 09:07:57AM +0900, hiroshi@soupacific.com wrote: > > It would be good to have all.log enabled in newsyslog.conf and provide the > > output from there so all lines are in one log and it is clear which message > > appeared earlier. Also the logs from the primary could be useful too. > > Could you tell me how to modify newsyslog.conf to one log file ? He meant syslog.conf. The procedure: - Edit /etc/syslog.conf and uncomment the line that looks like this: #*.* /var/log/all.log - touch /var/log/all.log - chmod 600 /var/log/all.log - chown root:wheel /var/log/all.log - /etc/rc.d/syslogd reload Going forward you should have a file called /var/log/all.log that contains literally all syslog data. newsyslog is already set up on FreeBSD to automatically rotate this log file at midnight every day, so there's no need to mess with that piece. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Sun Jun 13 00:42:49 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A01A5106564A for ; Sun, 13 Jun 2010 00:42:49 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 54BB38FC0A for ; Sun, 13 Jun 2010 00:42:48 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AvsEADfGE0yDaFvG/2dsb2JhbACefnG9M4UaBA X-IronPort-AV: E=Sophos;i="4.53,409,1272859200"; d="scan'208";a="79746927" Received: from amazon.cs.uoguelph.ca ([131.104.91.198]) by esa-jnhn-pri.mail.uoguelph.ca with ESMTP; 12 Jun 2010 20:42:45 -0400 Received: from localhost (localhost.localdomain [127.0.0.1]) by amazon.cs.uoguelph.ca (Postfix) with ESMTP id 70DEE210156; Sat, 12 Jun 2010 20:42:47 -0400 (EDT) X-Virus-Scanned: amavisd-new at amazon.cs.uoguelph.ca Received: from amazon.cs.uoguelph.ca ([127.0.0.1]) by localhost (amazon.cs.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UXkJO28OFyZZ; Sat, 12 Jun 2010 20:42:46 -0400 (EDT) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by amazon.cs.uoguelph.ca (Postfix) with ESMTP id A717B210119; Sat, 12 Jun 2010 20:42:46 -0400 (EDT) Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id o5D0x2g12507; Sat, 12 Jun 2010 20:59:02 -0400 (EDT) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Sat, 12 Jun 2010 20:59:02 -0400 (EDT) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: Andrey Simonenko In-Reply-To: <20100611103803.GA1855@pm513-1.comsys.ntu-kpi.kiev.ua> Message-ID: References: <20100611103803.GA1855@pm513-1.comsys.ntu-kpi.kiev.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org Subject: Re: Testers: NFSv3 support for pxeboot for nfs diskless root X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Jun 2010 00:42:49 -0000 On Fri, 11 Jun 2010, Andrey Simonenko wrote: > On Wed, Jun 09, 2010 at 07:38:24PM -0400, Rick Macklem wrote: >> I put 3 patches (you need to apply them all) here: >> http://people.freebsd.org/~rmacklem/nfsdiskless-patches/ >> >> They convert lib/libstand/nfs.c and pxeboot to use NFSv3 instead >> of NFSv2 (unless built with OLD_NFSV2 defined). Initial test >> reports have been good. (one has it working ok and the other has >> a problem in an area not related to the patches, it appears) >> >> So, if others are interested in testing these, it would be >> appreciated, rick > > Shouldn't return values from malloc() calls be checked? > Also additional checks for NULL values before free() calls can be removed, > at least this will reduce size of code. There is PR/83424 related to this. > I have updated the patch with these changes. Thanks, rick From owner-freebsd-fs@FreeBSD.ORG Sun Jun 13 07:50:30 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9A01E1065670 for ; Sun, 13 Jun 2010 07:50:30 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello089077043238.chello.pl [89.77.43.238]) by mx1.freebsd.org (Postfix) with ESMTP id D2E368FC14 for ; Sun, 13 Jun 2010 07:50:29 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 8009745CA6; Sun, 13 Jun 2010 09:50:27 +0200 (CEST) Received: from localhost (gate.wheel.pl [10.0.0.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 3B421456B1; Sun, 13 Jun 2010 09:50:21 +0200 (CEST) Date: Sun, 13 Jun 2010 09:50:11 +0200 From: Pawel Jakub Dawidek To: Jeremy Chadwick Message-ID: <20100613074921.GB1320@garage.freebsd.pl> References: <20100423061521.GC1670@garage.freebsd.pl> <4BD17B0D.5080601@soupacific.com> <4C10B526.4040908@soupacific.com> <20100612104336.GA2253@garage.freebsd.pl> <4C1372E0.1000903@soupacific.com> <20100612142311.GF2253@garage.freebsd.pl> <4C139F9C.2090305@soupacific.com> <86iq5oc82y.fsf@kopusha.home.net> <4C14215D.9090304@soupacific.com> <20100613003635.GA60012@icarus.home.lan> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="XMCwj5IQnwKtuyBG" Content-Disposition: inline In-Reply-To: <20100613003635.GA60012@icarus.home.lan> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT amd64 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=4.5 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: FreeBSD 8.1 and HAST X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Jun 2010 07:50:30 -0000 --XMCwj5IQnwKtuyBG Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jun 12, 2010 at 05:36:35PM -0700, Jeremy Chadwick wrote: > On Sun, Jun 13, 2010 at 09:07:57AM +0900, hiroshi@soupacific.com wrote: > > > It would be good to have all.log enabled in newsyslog.conf and provid= e the > > > output from there so all lines are in one log and it is clear which m= essage > > > appeared earlier. Also the logs from the primary could be useful too. > >=20 > > Could you tell me how to modify newsyslog.conf to one log file ? >=20 > He meant syslog.conf. The procedure: >=20 > - Edit /etc/syslog.conf and uncomment the line that looks like this: >=20 > #*.* /var/log/all.log >=20 > - touch /var/log/all.log > - chmod 600 /var/log/all.log > - chown root:wheel /var/log/all.log > - /etc/rc.d/syslogd reload >=20 > Going forward you should have a file called /var/log/all.log that > contains literally all syslog data. newsyslog is already set up on > FreeBSD to automatically rotate this log file at midnight every day, > so there's no need to mess with that piece. Alternatively you could start hastd in foreground (-F option). --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --XMCwj5IQnwKtuyBG Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkwUjbIACgkQForvXbEpPzQ/vwCdEhAPI4HGE6ecLYCtseQiR44/ xaIAnjLzKLP+r5lC23ggkf0jRUc2Xc6H =01VN -----END PGP SIGNATURE----- --XMCwj5IQnwKtuyBG-- From owner-freebsd-fs@FreeBSD.ORG Sun Jun 13 08:44:21 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 03F8C1065674; Sun, 13 Jun 2010 08:44:21 +0000 (UTC) (envelope-from hiroshi@soupacific.com) Received: from mail.soupacific.com (mail.soupacific.com [211.19.53.201]) by mx1.freebsd.org (Postfix) with ESMTP id 25A778FC1D; Sun, 13 Jun 2010 08:44:19 +0000 (UTC) Received: from [127.0.0.1] (unknown [192.168.1.239]) by mail.soupacific.com (Postfix) with ESMTP id 663EC6BAAC; Sun, 13 Jun 2010 08:35:43 +0000 (UTC) Message-ID: <4C149A5C.3070401@soupacific.com> Date: Sun, 13 Jun 2010 17:44:12 +0900 From: "hiroshi@soupacific.com" User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4 MIME-Version: 1.0 To: Pawel Jakub Dawidek References: <20100423061521.GC1670@garage.freebsd.pl> <4BD17B0D.5080601@soupacific.com> <4C10B526.4040908@soupacific.com> <20100612104336.GA2253@garage.freebsd.pl> <4C1372E0.1000903@soupacific.com> <20100612142311.GF2253@garage.freebsd.pl> <4C139F9C.2090305@soupacific.com> <86iq5oc82y.fsf@kopusha.home.net> <4C14215D.9090304@soupacific.com> <20100613003635.GA60012@icarus.home.lan> <20100613074921.GB1320@garage.freebsd.pl> In-Reply-To: <20100613074921.GB1320@garage.freebsd.pl> Content-Type: multipart/mixed; boundary="------------070809040105010607030206" Cc: freebsd-fs@freebsd.org Subject: Re: FreeBSD 8.1 and HAST X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Jun 2010 08:44:21 -0000 This is a multi-part message in MIME format. --------------070809040105010607030206 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Good morning ! here are the file all.log each. I hope this might help you some idea. Thanks Hiorhis --------------070809040105010607030206 Content-Type: text/plain; name="slaveall.log" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="slaveall.log" Jun 13 17:33:20 sv01B syslogd: restart Jun 13 17:34:10 sv01B kernel: ugen0.2: at usbus0 (disconnected) Jun 13 17:34:10 sv01B kernel: ukbd0: at uhub0, port 3, addr 2 (disconnected) Jun 13 17:35:00 sv01B /usr/sbin/cron[812]: (root) CMD (/usr/libexec/atrun) Jun 13 17:35:29 sv01B kernel: ugen0.2: at usbus0 Jun 13 17:35:29 sv01B kernel: ukbd0: on usbus0 Jun 13 17:35:29 sv01B kernel: kbd2 at ukbd0 Jun 13 17:35:49 sv01B hastd: Accepting connection to tcp4://0.0.0.0:8457. Jun 13 17:35:49 sv01B hastd: Connection from tcp4://192.168.0.241:8457 to tcp4://192.168.0.240:17726. Jun 13 17:35:49 sv01B hastd: tcp4://192.168.0.240:17726: resource=zfshast Jun 13 17:35:49 sv01B hastd: [zfshast] (init) We act as init for the resource and not as secondary as requested by tcp4://192.168.0.240:17726. Jun 13 17:35:49 sv01B kernel: Jun 13 17:35:49 sv01B hastd: [zfshast] (init) We act as init for the resource and not as secondary as requested by tcp4://192.168.0.240:17726. Jun 13 17:35:53 sv01B hastd: [zfshast] (init) Role changed to secondary. Jun 13 17:35:54 sv01B hastd: Accepting connection to tcp4://0.0.0.0:8457. Jun 13 17:35:54 sv01B hastd: Connection from tcp4://192.168.0.241:8457 to tcp4://192.168.0.240:24260. Jun 13 17:35:54 sv01B hastd: tcp4://192.168.0.240:24260: resource=zfshast Jun 13 17:35:54 sv01B hastd: [zfshast] (secondary) Initial connection from tcp4://192.168.0.240:24260. Jun 13 17:35:54 sv01B hastd: [zfshast] (secondary) Incoming connection from tcp4://192.168.0.240:24260 configured. Jun 13 17:35:54 sv01B hastd: Accepting connection to tcp4://0.0.0.0:8457. Jun 13 17:35:54 sv01B hastd: Connection from tcp4://192.168.0.241:8457 to tcp4://192.168.0.240:51988. Jun 13 17:35:54 sv01B hastd: tcp4://192.168.0.240:51988: resource=zfshast Jun 13 17:35:54 sv01B hastd: [zfshast] (secondary) Outgoing connection to tcp4://192.168.0.240:51988 configured. Jun 13 17:35:54 sv01B hastd: [zfshast] (secondary) Obtained info about /dev/ad4p4. Jun 13 17:35:54 sv01B hastd: [zfshast] (secondary) Locked /dev/ad4p4. Jun 13 17:35:54 sv01B hastd: [zfshast] (secondary) recv: Taking free request. Jun 13 17:35:54 sv01B hastd: [zfshast] (secondary) disk: Taking request. Jun 13 17:35:54 sv01B hastd: [zfshast] (secondary) send: Taking request. Jun 13 17:35:54 sv01B hastd: [zfshast] (secondary) send: No requests, waiting. Jun 13 17:35:54 sv01B hastd: [zfshast] (secondary) disk: No requests, waiting. Jun 13 17:35:54 sv01B hastd: [zfshast] (secondary) recv: (0x8011f52e0) Got request. Jun 13 17:35:54 sv01B hastd: [zfshast] (secondary) recv: (0x8011f52e0) Got request header: WRITE(0, 131072). Jun 13 17:35:54 sv01B hastd: [zfshast] (secondary) recv: (0x8011f52e0) Moving request to the disk queue. Jun 13 17:35:54 sv01B hastd: [zfshast] (secondary) recv: Taking free request. Jun 13 17:35:54 sv01B hastd: [zfshast] (secondary) recv: (0x8011f5290) Got request. Jun 13 17:35:54 sv01B hastd: [zfshast] (secondary) Local activemap cleared. Jun 13 17:35:54 sv01B hastd: [zfshast] (secondary) disk: (0x8011f52e0) Got request: WRITE(0, 131072). Jun 13 17:35:54 sv01B hastd: [zfshast] (secondary) disk: (0x8011f52e0) Moving request to the send queue. Jun 13 17:35:54 sv01B hastd: [zfshast] (secondary) disk: Taking request. Jun 13 17:35:54 sv01B hastd: [zfshast] (secondary) disk: No requests, waiting. Jun 13 17:35:54 sv01B hastd: [zfshast] (secondary) send: (0x8011f52e0) Got request: WRITE(0, 131072). Jun 13 17:35:54 sv01B hastd: [zfshast] (secondary) send: (0x8011f52e0) Moving request to the free queue. Jun 13 17:35:54 sv01B hastd: [zfshast] (secondary) send: Taking request. Jun 13 17:35:54 sv01B hastd: [zfshast] (secondary) send: No requests, waiting. Jun 13 17:35:59 sv01B hastd: [zfshast] (secondary) Unable to receive request header. : Socket is not connected. Jun 13 17:35:59 sv01B kernel: Jun 13 17:35:59 sv01B hastd: [zfshast] (secondary) Unable to receive request header. : Socket is not connected. Jun 13 17:35:59 sv01B hastd: Accepting connection to tcp4://0.0.0.0:8457. Jun 13 17:35:59 sv01B hastd: Connection from tcp4://192.168.0.241:8457 to tcp4://192.168.0.240:21359. Jun 13 17:35:59 sv01B hastd: tcp4://192.168.0.240:21359: resource=zfshast Jun 13 17:35:59 sv01B hastd: [zfshast] (secondary) Initial connection from tcp4://192.168.0.240:21359. Jun 13 17:35:59 sv01B hastd: [zfshast] (secondary) Worker process exists (pid=822), stopping it. Jun 13 17:35:59 sv01B hastd: [zfshast] (secondary) Worker process exited ungracefully (pid=822, exitcode=75). Jun 13 17:35:59 sv01B kernel: Jun 13 17:35:59 sv01B hastd: [zfshast] (secondary) Worker process exited ungracefully (pid=822, exitcode=75). Jun 13 17:35:59 sv01B hastd: [zfshast] (secondary) Incoming connection from tcp4://192.168.0.240:21359 configured. Jun 13 17:35:59 sv01B hastd: Accepting connection to tcp4://0.0.0.0:8457. Jun 13 17:35:59 sv01B hastd: Connection from tcp4://192.168.0.241:8457 to tcp4://192.168.0.240:20778. Jun 13 17:35:59 sv01B hastd: tcp4://192.168.0.240:20778: resource=zfshast Jun 13 17:35:59 sv01B hastd: [zfshast] (secondary) Outgoing connection to tcp4://192.168.0.240:20778 configured. Jun 13 17:35:59 sv01B hastd: [zfshast] (secondary) Obtained info about /dev/ad4p4. Jun 13 17:35:59 sv01B hastd: [zfshast] (secondary) Locked /dev/ad4p4. Jun 13 17:35:59 sv01B hastd: [zfshast] (secondary) recv: Taking free request. Jun 13 17:35:59 sv01B hastd: [zfshast] (secondary) disk: Taking request. Jun 13 17:35:59 sv01B hastd: [zfshast] (secondary) disk: No requests, waiting. Jun 13 17:35:59 sv01B hastd: [zfshast] (secondary) recv: (0x8011f32e0) Got request. Jun 13 17:35:59 sv01B hastd: [zfshast] (secondary) send: Taking request. Jun 13 17:35:59 sv01B hastd: [zfshast] (secondary) send: No requests, waiting. Jun 13 17:35:59 sv01B hastd: [zfshast] (secondary) recv: (0x8011f32e0) Got request header: WRITE(0, 131072). Jun 13 17:35:59 sv01B hastd: [zfshast] (secondary) recv: (0x8011f32e0) Moving request to the disk queue. Jun 13 17:35:59 sv01B hastd: [zfshast] (secondary) recv: Taking free request. Jun 13 17:35:59 sv01B hastd: [zfshast] (secondary) recv: (0x8011f3290) Got request. Jun 13 17:35:59 sv01B hastd: [zfshast] (secondary) Local activemap cleared. Jun 13 17:35:59 sv01B hastd: [zfshast] (secondary) disk: (0x8011f32e0) Got request: WRITE(0, 131072). Jun 13 17:35:59 sv01B hastd: [zfshast] (secondary) disk: (0x8011f32e0) Moving request to the send queue. Jun 13 17:35:59 sv01B hastd: [zfshast] (secondary) disk: Taking request. Jun 13 17:35:59 sv01B hastd: [zfshast] (secondary) disk: No requests, waiting. Jun 13 17:35:59 sv01B hastd: [zfshast] (secondary) send: (0x8011f32e0) Got request: WRITE(0, 131072). Jun 13 17:35:59 sv01B hastd: [zfshast] (secondary) send: (0x8011f32e0) Moving request to the free queue. Jun 13 17:35:59 sv01B hastd: [zfshast] (secondary) send: Taking request. Jun 13 17:35:59 sv01B hastd: [zfshast] (secondary) send: No requests, waiting. Jun 13 17:36:04 sv01B hastd: [zfshast] (secondary) Unable to receive request header. : Socket is not connected. Jun 13 17:36:04 sv01B kernel: Jun 13 17:36:04 sv01B hastd: [zfshast] (secondary) Unable to receive request header. : Socket is not connected. Jun 13 17:36:04 sv01B hastd: Accepting connection to tcp4://0.0.0.0:8457. Jun 13 17:36:04 sv01B hastd: Connection from tcp4://192.168.0.241:8457 to tcp4://192.168.0.240:54823. Jun 13 17:36:04 sv01B hastd: tcp4://192.168.0.240:54823: resource=zfshast Jun 13 17:36:04 sv01B hastd: [zfshast] (secondary) Initial connection from tcp4://192.168.0.240:54823. Jun 13 17:36:04 sv01B hastd: [zfshast] (secondary) Worker process exists (pid=823), stopping it. Jun 13 17:36:04 sv01B hastd: [zfshast] (secondary) Worker process exited ungracefully (pid=823, exitcode=75). Jun 13 17:36:04 sv01B kernel: Jun 13 17:36:04 sv01B hastd: [zfshast] (secondary) Worker process exited ungracefully (pid=823, exitcode=75). Jun 13 17:36:04 sv01B hastd: [zfshast] (secondary) Incoming connection from tcp4://192.168.0.240:54823 configured. Jun 13 17:36:04 sv01B hastd: Accepting connection to tcp4://0.0.0.0:8457. Jun 13 17:36:04 sv01B hastd: Connection from tcp4://192.168.0.241:8457 to tcp4://192.168.0.240:37734. Jun 13 17:36:04 sv01B hastd: tcp4://192.168.0.240:37734: resource=zfshast Jun 13 17:36:04 sv01B hastd: [zfshast] (secondary) Outgoing connection to tcp4://192.168.0.240:37734 configured. Jun 13 17:36:04 sv01B hastd: [zfshast] (secondary) Obtained info about /dev/ad4p4. Jun 13 17:36:04 sv01B hastd: [zfshast] (secondary) Locked /dev/ad4p4. Jun 13 17:36:04 sv01B hastd: [zfshast] (secondary) recv: Taking free request. Jun 13 17:36:04 sv01B hastd: [zfshast] (secondary) disk: Taking request. Jun 13 17:36:04 sv01B hastd: [zfshast] (secondary) send: Taking request. Jun 13 17:36:04 sv01B hastd: [zfshast] (secondary) send: No requests, waiting. Jun 13 17:36:04 sv01B hastd: [zfshast] (secondary) disk: No requests, waiting. Jun 13 17:36:04 sv01B hastd: [zfshast] (secondary) recv: (0x8011f32e0) Got request. Jun 13 17:36:04 sv01B hastd: [zfshast] (secondary) recv: (0x8011f32e0) Got request header: WRITE(0, 131072). Jun 13 17:36:04 sv01B hastd: [zfshast] (secondary) recv: (0x8011f32e0) Moving request to the disk queue. Jun 13 17:36:04 sv01B hastd: [zfshast] (secondary) recv: Taking free request. Jun 13 17:36:04 sv01B hastd: [zfshast] (secondary) recv: (0x8011f3290) Got request. Jun 13 17:36:04 sv01B hastd: [zfshast] (secondary) Local activemap cleared. Jun 13 17:36:04 sv01B hastd: [zfshast] (secondary) disk: (0x8011f32e0) Got request: WRITE(0, 131072). Jun 13 17:36:04 sv01B hastd: [zfshast] (secondary) disk: (0x8011f32e0) Moving request to the send queue. Jun 13 17:36:04 sv01B hastd: [zfshast] (secondary) disk: Taking request. Jun 13 17:36:04 sv01B hastd: [zfshast] (secondary) disk: No requests, waiting. Jun 13 17:36:04 sv01B hastd: [zfshast] (secondary) send: (0x8011f32e0) Got request: WRITE(0, 131072). Jun 13 17:36:04 sv01B hastd: [zfshast] (secondary) send: (0x8011f32e0) Moving request to the free queue. Jun 13 17:36:04 sv01B hastd: [zfshast] (secondary) send: Taking request. Jun 13 17:36:04 sv01B hastd: [zfshast] (secondary) send: No requests, waiting. Jun 13 17:36:09 sv01B hastd: [zfshast] (secondary) Unable to receive request header. : Socket is not connected. Jun 13 17:36:09 sv01B kernel: Jun 13 17:36:09 sv01B hastd: [zfshast] (secondary) Unable to receive request header. : Socket is not connected. Jun 13 17:36:09 sv01B hastd: Accepting connection to tcp4://0.0.0.0:8457. Jun 13 17:36:09 sv01B hastd: Connection from tcp4://192.168.0.241:8457 to tcp4://192.168.0.240:47054. Jun 13 17:36:09 sv01B hastd: tcp4://192.168.0.240:47054: resource=zfshast Jun 13 17:36:09 sv01B hastd: [zfshast] (secondary) Initial connection from tcp4://192.168.0.240:47054. Jun 13 17:36:09 sv01B hastd: [zfshast] (secondary) Worker process exists (pid=824), stopping it. Jun 13 17:36:09 sv01B hastd: [zfshast] (secondary) Worker process exited ungracefully (pid=824, exitcode=75). Jun 13 17:36:09 sv01B kernel: Jun 13 17:36:09 sv01B hastd: [zfshast] (secondary) Worker process exited ungracefully (pid=824, exitcode=75). Jun 13 17:36:09 sv01B hastd: [zfshast] (secondary) Incoming connection from tcp4://192.168.0.240:47054 configured. Jun 13 17:36:09 sv01B hastd: Accepting connection to tcp4://0.0.0.0:8457. Jun 13 17:36:09 sv01B hastd: Connection from tcp4://192.168.0.241:8457 to tcp4://192.168.0.240:26039. Jun 13 17:36:09 sv01B hastd: tcp4://192.168.0.240:26039: resource=zfshast Jun 13 17:36:09 sv01B hastd: [zfshast] (secondary) Outgoing connection to tcp4://192.168.0.240:26039 configured. Jun 13 17:36:09 sv01B hastd: [zfshast] (secondary) Obtained info about /dev/ad4p4. Jun 13 17:36:09 sv01B hastd: [zfshast] (secondary) Locked /dev/ad4p4. Jun 13 17:36:09 sv01B hastd: [zfshast] (secondary) recv: Taking free request. Jun 13 17:36:09 sv01B hastd: [zfshast] (secondary) disk: Taking request. Jun 13 17:36:09 sv01B hastd: [zfshast] (secondary) disk: No requests, waiting. Jun 13 17:36:09 sv01B hastd: [zfshast] (secondary) recv: (0x8011f32e0) Got request. Jun 13 17:36:09 sv01B hastd: [zfshast] (secondary) send: Taking request. Jun 13 17:36:09 sv01B hastd: [zfshast] (secondary) send: No requests, waiting. Jun 13 17:36:09 sv01B hastd: [zfshast] (secondary) recv: (0x8011f32e0) Got request header: WRITE(0, 131072). Jun 13 17:36:09 sv01B hastd: [zfshast] (secondary) recv: (0x8011f32e0) Moving request to the disk queue. Jun 13 17:36:09 sv01B hastd: [zfshast] (secondary) recv: Taking free request. Jun 13 17:36:09 sv01B hastd: [zfshast] (secondary) recv: (0x8011f3290) Got request. Jun 13 17:36:09 sv01B hastd: [zfshast] (secondary) Local activemap cleared. Jun 13 17:36:09 sv01B hastd: [zfshast] (secondary) disk: (0x8011f32e0) Got request: WRITE(0, 131072). Jun 13 17:36:09 sv01B hastd: [zfshast] (secondary) disk: (0x8011f32e0) Moving request to the send queue. Jun 13 17:36:09 sv01B hastd: [zfshast] (secondary) disk: Taking request. Jun 13 17:36:09 sv01B hastd: [zfshast] (secondary) send: (0x8011f32e0) Got request: WRITE(0, 131072). Jun 13 17:36:09 sv01B hastd: [zfshast] (secondary) disk: No requests, waiting. Jun 13 17:36:09 sv01B hastd: [zfshast] (secondary) send: (0x8011f32e0) Moving request to the free queue. Jun 13 17:36:09 sv01B hastd: [zfshast] (secondary) send: Taking request. Jun 13 17:36:09 sv01B hastd: [zfshast] (secondary) send: No requests, waiting. Jun 13 17:36:14 sv01B hastd: [zfshast] (secondary) Unable to receive request header. : Socket is not connected. Jun 13 17:36:14 sv01B kernel: Jun 13 17:36:14 sv01B hastd: [zfshast] (secondary) Unable to receive request header. : Socket is not connected. Jun 13 17:36:14 sv01B hastd: Accepting connection to tcp4://0.0.0.0:8457. Jun 13 17:36:14 sv01B hastd: Connection from tcp4://192.168.0.241:8457 to tcp4://192.168.0.240:58513. Jun 13 17:36:14 sv01B hastd: tcp4://192.168.0.240:58513: resource=zfshast Jun 13 17:36:14 sv01B hastd: [zfshast] (secondary) Initial connection from tcp4://192.168.0.240:58513. Jun 13 17:36:14 sv01B hastd: [zfshast] (secondary) Worker process exists (pid=825), stopping it. Jun 13 17:36:14 sv01B hastd: [zfshast] (secondary) Worker process exited ungracefully (pid=825, exitcode=75). Jun 13 17:36:14 sv01B kernel: Jun 13 17:36:14 sv01B hastd: [zfshast] (secondary) Worker process exited ungracefully (pid=825, exitcode=75). Jun 13 17:36:14 sv01B hastd: [zfshast] (secondary) Incoming connection from tcp4://192.168.0.240:58513 configured. Jun 13 17:36:14 sv01B hastd: Accepting connection to tcp4://0.0.0.0:8457. Jun 13 17:36:14 sv01B hastd: Connection from tcp4://192.168.0.241:8457 to tcp4://192.168.0.240:48961. Jun 13 17:36:14 sv01B hastd: tcp4://192.168.0.240:48961: resource=zfshast Jun 13 17:36:14 sv01B hastd: [zfshast] (secondary) Outgoing connection to tcp4://192.168.0.240:48961 configured. Jun 13 17:36:14 sv01B hastd: [zfshast] (secondary) Obtained info about /dev/ad4p4. Jun 13 17:36:14 sv01B hastd: [zfshast] (secondary) Locked /dev/ad4p4. Jun 13 17:36:14 sv01B hastd: [zfshast] (secondary) recv: Taking free request. Jun 13 17:36:14 sv01B hastd: [zfshast] (secondary) disk: Taking request. Jun 13 17:36:14 sv01B hastd: [zfshast] (secondary) disk: No requests, waiting. Jun 13 17:36:14 sv01B hastd: [zfshast] (secondary) recv: (0x8011f32e0) Got request. Jun 13 17:36:14 sv01B hastd: [zfshast] (secondary) send: Taking request. Jun 13 17:36:14 sv01B hastd: [zfshast] (secondary) send: No requests, waiting. Jun 13 17:36:14 sv01B hastd: [zfshast] (secondary) recv: (0x8011f32e0) Got request header: WRITE(0, 131072). Jun 13 17:36:14 sv01B hastd: [zfshast] (secondary) recv: (0x8011f32e0) Moving request to the disk queue. Jun 13 17:36:14 sv01B hastd: [zfshast] (secondary) recv: Taking free request. Jun 13 17:36:14 sv01B hastd: [zfshast] (secondary) recv: (0x8011f3290) Got request. Jun 13 17:36:14 sv01B hastd: [zfshast] (secondary) Local activemap cleared. Jun 13 17:36:14 sv01B hastd: [zfshast] (secondary) disk: (0x8011f32e0) Got request: WRITE(0, 131072). Jun 13 17:36:14 sv01B hastd: [zfshast] (secondary) disk: (0x8011f32e0) Moving request to the send queue. Jun 13 17:36:14 sv01B hastd: [zfshast] (secondary) disk: Taking request. Jun 13 17:36:14 sv01B hastd: [zfshast] (secondary) disk: No requests, waiting. Jun 13 17:36:14 sv01B hastd: [zfshast] (secondary) send: (0x8011f32e0) Got request: WRITE(0, 131072). Jun 13 17:36:14 sv01B hastd: [zfshast] (secondary) send: (0x8011f32e0) Moving request to the free queue. Jun 13 17:36:14 sv01B hastd: [zfshast] (secondary) send: Taking request. Jun 13 17:36:14 sv01B hastd: [zfshast] (secondary) send: No requests, waiting. --------------070809040105010607030206 Content-Type: text/plain; name="masterall.log" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="masterall.log" Jun 13 16:21:02 sv01A syslogd: restart Jun 13 16:21:30 sv01A kernel: ugen2.2: at usbus2 (disconnected) Jun 13 16:21:30 sv01A kernel: ukbd0: at uhub2, port 2, addr 2 (disconnected) Jun 13 16:22:00 sv01A /usr/sbin/cron[915]: (operator) CMD (/usr/libexec/save-entropy) Jun 13 16:23:58 sv01A kernel: ugen2.2: at usbus2 Jun 13 16:23:58 sv01A kernel: ukbd0: on usbus2 Jun 13 16:23:58 sv01A kernel: kbd2 at ukbd0 Jun 13 16:25:00 sv01A /usr/sbin/cron[937]: (root) CMD (/usr/libexec/atrun) Jun 13 16:25:02 sv01A hastd: [zfshast] (init) Role changed to primary. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) Obtained info about /dev/ad4p4. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) Locked /dev/ad4p4. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) getsockopt(SO_ERROR) returned error: Connection refused. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) Unable to connect to 192.168.0.241: Connection refused. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) Device hast/zfshast created. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: Taking free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd510) Got free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd510) Waiting for request from the kernel. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd510) Request received from the kernel: READ(975038899200, 512). Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd510) Moving request to the send queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: Taking free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd4c0) Got free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd4c0) Waiting for request from the kernel. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd510) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) remote_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) remote_guard: Checking connections. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) remote_guard: Reconnecting to 192.168.0.241. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd510) Moving request to the done queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd510) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) getsockopt(SO_ERROR) returned error: Connection refused. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) Unable to connect to 192.168.0.241: Connection refused. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) remote_guard: Reconnect to 192.168.0.241 failed. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd510) Moving request to the free queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd4c0) Request received from the kernel: READ(65536, 8192). Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd4c0) Moving request to the send queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: Taking free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd470) Got free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd470) Waiting for request from the kernel. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd4c0) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) remote_recv: No requests, waiting. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd4c0) Moving request to the done queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd4c0) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd4c0) Moving request to the free queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd470) Request received from the kernel: READ(8192, 8192). Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd470) Moving request to the send queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: Taking free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd420) Got free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd420) Waiting for request from the kernel. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd470) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd470) Moving request to the done queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd470) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd470) Moving request to the free queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd420) Request received from the kernel: READ(0, 8192). Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd420) Moving request to the send queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: Taking free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd3d0) Got free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd3d0) Waiting for request from the kernel. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd420) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd420) Moving request to the done queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd420) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd420) Moving request to the free queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd3d0) Request received from the kernel: READ(262144, 8192). Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd3d0) Moving request to the send queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: Taking free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd380) Got free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd380) Waiting for request from the kernel. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd3d0) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd3d0) Moving request to the done queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd3d0) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd3d0) Moving request to the free queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd380) Request received from the kernel: READ(65536, 8192). Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd380) Moving request to the send queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: Taking free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd330) Got free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd330) Waiting for request from the kernel. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd380) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd380) Moving request to the done queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd380) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd380) Moving request to the free queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd330) Request received from the kernel: READ(8192, 8192). Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd330) Moving request to the send queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: Taking free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd2e0) Got free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd2e0) Waiting for request from the kernel. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd330) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd330) Moving request to the done queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd330) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd330) Moving request to the free queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd2e0) Request received from the kernel: READ(0, 8192). Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd2e0) Moving request to the send queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: Taking free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd290) Got free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd290) Waiting for request from the kernel. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd2e0) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd2e0) Moving request to the done queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd2e0) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd2e0) Moving request to the free queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd290) Request received from the kernel: READ(262144, 8192). Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd290) Moving request to the send queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: Taking free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd240) Got free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd240) Waiting for request from the kernel. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd290) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd290) Moving request to the done queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd290) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd290) Moving request to the free queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd240) Request received from the kernel: READ(32768, 512). Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd240) Moving request to the send queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: Taking free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd1f0) Got free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd1f0) Waiting for request from the kernel. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd240) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd240) Moving request to the done queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd240) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd240) Moving request to the free queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd1f0) Request received from the kernel: READ(0, 512). Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd1f0) Moving request to the send queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: Taking free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd1a0) Got free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd1a0) Waiting for request from the kernel. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd1f0) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd1f0) Moving request to the done queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd1f0) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd1f0) Moving request to the free queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd1a0) Request received from the kernel: READ(1024, 512). Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd1a0) Moving request to the send queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: Taking free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd150) Got free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd150) Waiting for request from the kernel. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd1a0) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd1a0) Moving request to the done queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd1a0) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd1a0) Moving request to the free queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd150) Request received from the kernel: READ(8192, 512). Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd150) Moving request to the send queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: Taking free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd100) Got free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd100) Waiting for request from the kernel. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd150) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd150) Moving request to the done queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd150) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd150) Moving request to the free queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd100) Request received from the kernel: READ(65536, 512). Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd100) Moving request to the send queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: Taking free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd0b0) Got free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd0b0) Waiting for request from the kernel. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd100) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd100) Moving request to the done queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd100) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd100) Moving request to the free queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd0b0) Request received from the kernel: READ(0, 512). Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd0b0) Moving request to the send queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: Taking free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd060) Got free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd060) Waiting for request from the kernel. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd0b0) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd0b0) Moving request to the done queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd0b0) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd0b0) Moving request to the free queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd060) Request received from the kernel: READ(512, 512). Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fd060) Moving request to the send queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: Taking free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fbfb0) Got free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fbfb0) Waiting for request from the kernel. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd060) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fd060) Moving request to the done queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd060) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fd060) Moving request to the free queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fbfb0) Request received from the kernel: READ(0, 512). Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fbfb0) Moving request to the send queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: Taking free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fbf60) Got free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fbf60) Waiting for request from the kernel. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fbfb0) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fbfb0) Moving request to the done queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fbfb0) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fbfb0) Moving request to the free queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fbf60) Request received from the kernel: READ(0, 512). Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fbf60) Moving request to the send queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: Taking free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fbf10) Got free request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_recv: (0x8012fbf10) Waiting for request from the kernel. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fbf60) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fbf60) Moving request to the done queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) local_send: Taking request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fbf60) Got request. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: (0x8012fbf60) Moving request to the free queue. Jun 13 16:25:02 sv01A hastd: [zfshast] (primary) ggate_send: Taking request. Jun 13 16:25:07 sv01A hastd: [zfshast] (primary) remote_guard: Checking connections. Jun 13 16:25:07 sv01A hastd: [zfshast] (primary) remote_guard: Reconnecting to 192.168.0.241. Jun 13 16:25:07 sv01A hastd: [zfshast] (primary) getsockopt(SO_ERROR) returned error: Connection refused. Jun 13 16:25:07 sv01A hastd: [zfshast] (primary) Unable to connect to 192.168.0.241: Connection refused. Jun 13 16:25:07 sv01A hastd: [zfshast] (primary) remote_guard: Reconnect to 192.168.0.241 failed. Jun 13 16:25:08 sv01A kernel: ugen2.2: at usbus2 (disconnected) Jun 13 16:25:08 sv01A kernel: ukbd0: at uhub2, port 2, addr 2 (disconnected) Jun 13 16:25:12 sv01A hastd: [zfshast] (primary) remote_guard: Checking connections. Jun 13 16:25:12 sv01A hastd: [zfshast] (primary) remote_guard: Reconnecting to 192.168.0.241. Jun 13 16:25:12 sv01A hastd: [zfshast] (primary) getsockopt(SO_ERROR) returned error: Connection refused. Jun 13 16:25:12 sv01A hastd: [zfshast] (primary) Unable to connect to 192.168.0.241: Connection refused. Jun 13 16:25:12 sv01A hastd: [zfshast] (primary) remote_guard: Reconnect to 192.168.0.241 failed. Jun 13 16:25:17 sv01A hastd: [zfshast] (primary) remote_guard: Checking connections. Jun 13 16:25:17 sv01A hastd: [zfshast] (primary) remote_guard: Reconnecting to 192.168.0.241. Jun 13 16:25:17 sv01A hastd: [zfshast] (primary) getsockopt(SO_ERROR) returned error: Connection refused. Jun 13 16:25:17 sv01A hastd: [zfshast] (primary) Unable to connect to 192.168.0.241: Connection refused. Jun 13 16:25:17 sv01A hastd: [zfshast] (primary) remote_guard: Reconnect to 192.168.0.241 failed. Jun 13 16:25:22 sv01A hastd: [zfshast] (primary) remote_guard: Checking connections. Jun 13 16:25:22 sv01A hastd: [zfshast] (primary) remote_guard: Reconnecting to 192.168.0.241. Jun 13 16:25:22 sv01A hastd: [zfshast] (primary) getsockopt(SO_ERROR) returned error: Connection refused. Jun 13 16:25:22 sv01A hastd: [zfshast] (primary) Unable to connect to 192.168.0.241: Connection refused. Jun 13 16:25:22 sv01A hastd: [zfshast] (primary) remote_guard: Reconnect to 192.168.0.241 failed. Jun 13 16:25:27 sv01A hastd: [zfshast] (primary) remote_guard: Checking connections. Jun 13 16:25:27 sv01A hastd: [zfshast] (primary) remote_guard: Reconnecting to 192.168.0.241. Jun 13 16:25:27 sv01A hastd: [zfshast] (primary) getsockopt(SO_ERROR) returned error: Connection refused. Jun 13 16:25:27 sv01A hastd: [zfshast] (primary) Unable to connect to 192.168.0.241: Connection refused. Jun 13 16:25:27 sv01A hastd: [zfshast] (primary) remote_guard: Reconnect to 192.168.0.241 failed. Jun 13 16:25:32 sv01A hastd: [zfshast] (primary) remote_guard: Checking connections. Jun 13 16:25:32 sv01A hastd: [zfshast] (primary) remote_guard: Reconnecting to 192.168.0.241. Jun 13 16:25:32 sv01A hastd: [zfshast] (primary) Remote node acts as init for the resource and not as secondary. Jun 13 16:25:32 sv01A hastd: [zfshast] (primary) remote_guard: Reconnect to 192.168.0.241 failed. Jun 13 16:25:37 sv01A hastd: [zfshast] (primary) remote_guard: Checking connections. Jun 13 16:25:37 sv01A hastd: [zfshast] (primary) remote_guard: Reconnecting to 192.168.0.241. Jun 13 16:25:37 sv01A hastd: [zfshast] (primary) Connected to 192.168.0.241. Jun 13 16:25:37 sv01A hastd: [zfshast] (primary) Successfully reconnected to 192.168.0.241. Jun 13 16:25:37 sv01A hastd: [zfshast] (primary) Synchronization started. 975039365120 bytes to go. Jun 13 16:25:37 sv01A hastd: [zfshast] (primary) sync: Taking free request. Jun 13 16:25:37 sv01A hastd: [zfshast] (primary) sync: (0x8012fbec0) Got free request. Jun 13 16:25:37 sv01A hastd: [zfshast] (primary) sync: (0x8012fbec0) Sending sync request: READ(0, 131072). Jun 13 16:25:37 sv01A hastd: [zfshast] (primary) sync: (0x8012fbec0) Moving request to the send queue. Jun 13 16:25:37 sv01A hastd: [zfshast] (primary) local_send: (0x8012fbec0) Got request. Jun 13 16:25:37 sv01A hastd: [zfshast] (primary) local_send: Taking request. Jun 13 16:25:37 sv01A hastd: [zfshast] (primary) sync: (0x8012fbec0) Sending sync request: WRITE(0, 131072). Jun 13 16:25:37 sv01A hastd: [zfshast] (primary) sync: (0x8012fbec0) Moving request to the send queue. Jun 13 16:25:37 sv01A hastd: [zfshast] (primary) sync: (0x8012fbec0) Moving request to the send queues. Jun 13 16:25:37 sv01A hastd: [zfshast] (primary) remote_send: (0x8012fbec0) Got request. Jun 13 16:25:37 sv01A hastd: [zfshast] (primary) remote_send: (0x8012fbec0) Moving request to the recv queue. Jun 13 16:25:37 sv01A hastd: [zfshast] (primary) remote_send: Taking request. Jun 13 16:25:37 sv01A hastd: [zfshast] (primary) Header contains no 'seq' field. Jun 13 16:25:37 sv01A kernel: Jun 13 16:25:37 sv01A hastd: [zfshast] (primary) Header contains no 'seq' field. Jun 13 16:25:42 sv01A hastd: [zfshast] (primary) Unable to receive reply header: Resource temporarily unavailable. Jun 13 16:25:42 sv01A kernel: Jun 13 16:25:42 sv01A hastd: [zfshast] (primary) Unable to receive reply header: Resource temporarily unavailable. Jun 13 16:25:42 sv01A hastd: [zfshast] (primary) Closing old incoming connection to 192.168.0.241. Jun 13 16:25:42 sv01A hastd: [zfshast] (primary) Closing old outgoing connection to 192.168.0.241. Jun 13 16:25:42 sv01A hastd: [zfshast] (primary) remote_recv: No requests, waiting. Jun 13 16:25:42 sv01A hastd: [zfshast] (primary) Unable to write synchronization data: Invalid argument. Jun 13 16:25:42 sv01A hastd: [zfshast] (primary) Unable to write synchronization data: Invalid argument. Jun 13 16:25:42 sv01A kernel: Jun 13 16:25:42 sv01A hastd: [zfshast] (primary) Unable to write synchronization data: Invalid argument. Jun 13 16:25:42 sv01A hastd: [zfshast] (primary) sync: (0x8012fbec0) Moving request to the free queue. Jun 13 16:25:42 sv01A hastd: [zfshast] (primary) remote_guard: Checking connections. Jun 13 16:25:42 sv01A hastd: [zfshast] (primary) remote_guard: Reconnecting to 192.168.0.241. Jun 13 16:25:42 sv01A hastd: [zfshast] (primary) Connected to 192.168.0.241. Jun 13 16:25:42 sv01A hastd: [zfshast] (primary) Successfully reconnected to 192.168.0.241. Jun 13 16:25:42 sv01A hastd: [zfshast] (primary) Synchronization started. 975039365120 bytes to go. Jun 13 16:25:42 sv01A hastd: [zfshast] (primary) sync: Taking free request. Jun 13 16:25:42 sv01A hastd: [zfshast] (primary) sync: (0x8012fbe70) Got free request. Jun 13 16:25:42 sv01A hastd: [zfshast] (primary) sync: (0x8012fbe70) Sending sync request: READ(0, 131072). Jun 13 16:25:42 sv01A hastd: [zfshast] (primary) sync: (0x8012fbe70) Moving request to the send queue. Jun 13 16:25:42 sv01A hastd: [zfshast] (primary) local_send: (0x8012fbe70) Got request. Jun 13 16:25:42 sv01A hastd: [zfshast] (primary) local_send: Taking request. Jun 13 16:25:42 sv01A hastd: [zfshast] (primary) sync: (0x8012fbe70) Sending sync request: WRITE(0, 131072). Jun 13 16:25:42 sv01A hastd: [zfshast] (primary) sync: (0x8012fbe70) Moving request to the send queue. Jun 13 16:25:42 sv01A hastd: [zfshast] (primary) sync: (0x8012fbe70) Moving request to the send queues. Jun 13 16:25:42 sv01A hastd: [zfshast] (primary) remote_send: (0x8012fbe70) Got request. Jun 13 16:25:42 sv01A hastd: [zfshast] (primary) remote_send: (0x8012fbe70) Moving request to the recv queue. Jun 13 16:25:42 sv01A hastd: [zfshast] (primary) remote_send: Taking request. Jun 13 16:25:42 sv01A hastd: [zfshast] (primary) Header contains no 'seq' field. Jun 13 16:25:42 sv01A kernel: Jun 13 16:25:42 sv01A hastd: [zfshast] (primary) Header contains no 'seq' field. Jun 13 16:25:47 sv01A hastd: [zfshast] (primary) Unable to receive reply header: Resource temporarily unavailable. Jun 13 16:25:47 sv01A kernel: Jun 13 16:25:47 sv01A hastd: [zfshast] (primary) Unable to receive reply header: Resource temporarily unavailable. Jun 13 16:25:47 sv01A hastd: [zfshast] (primary) Closing old incoming connection to 192.168.0.241. Jun 13 16:25:47 sv01A hastd: [zfshast] (primary) Closing old outgoing connection to 192.168.0.241. Jun 13 16:25:47 sv01A hastd: [zfshast] (primary) remote_recv: No requests, waiting. Jun 13 16:25:47 sv01A hastd: [zfshast] (primary) remote_guard: Checking connections. Jun 13 16:25:47 sv01A hastd: [zfshast] (primary) remote_guard: Reconnecting to 192.168.0.241. Jun 13 16:25:47 sv01A hastd: [zfshast] (primary) Unable to write synchronization data: Invalid argument. Jun 13 16:25:47 sv01A hastd: [zfshast] (primary) Unable to write synchronization data: Invalid argument. Jun 13 16:25:47 sv01A kernel: Jun 13 16:25:47 sv01A hastd: [zfshast] (primary) Unable to write synchronization data: Invalid argument. Jun 13 16:25:47 sv01A hastd: [zfshast] (primary) sync: (0x8012fbe70) Moving request to the free queue. Jun 13 16:25:47 sv01A hastd: [zfshast] (primary) Connected to 192.168.0.241. Jun 13 16:25:47 sv01A hastd: [zfshast] (primary) Successfully reconnected to 192.168.0.241. Jun 13 16:25:47 sv01A hastd: [zfshast] (primary) Synchronization started. 975039365120 bytes to go. Jun 13 16:25:47 sv01A hastd: [zfshast] (primary) sync: Taking free request. Jun 13 16:25:47 sv01A hastd: [zfshast] (primary) sync: (0x8012fbe20) Got free request. Jun 13 16:25:47 sv01A hastd: [zfshast] (primary) sync: (0x8012fbe20) Sending sync request: READ(0, 131072). Jun 13 16:25:47 sv01A hastd: [zfshast] (primary) sync: (0x8012fbe20) Moving request to the send queue. Jun 13 16:25:47 sv01A hastd: [zfshast] (primary) local_send: (0x8012fbe20) Got request. Jun 13 16:25:47 sv01A hastd: [zfshast] (primary) local_send: Taking request. Jun 13 16:25:47 sv01A hastd: [zfshast] (primary) sync: (0x8012fbe20) Sending sync request: WRITE(0, 131072). Jun 13 16:25:47 sv01A hastd: [zfshast] (primary) sync: (0x8012fbe20) Moving request to the send queue. Jun 13 16:25:47 sv01A hastd: [zfshast] (primary) sync: (0x8012fbe20) Moving request to the send queues. Jun 13 16:25:47 sv01A hastd: [zfshast] (primary) remote_send: (0x8012fbe20) Got request. Jun 13 16:25:47 sv01A hastd: [zfshast] (primary) remote_send: (0x8012fbe20) Moving request to the recv queue. Jun 13 16:25:47 sv01A hastd: [zfshast] (primary) remote_send: Taking request. Jun 13 16:25:47 sv01A hastd: [zfshast] (primary) Header contains no 'seq' field. Jun 13 16:25:47 sv01A kernel: Jun 13 16:25:47 sv01A hastd: [zfshast] (primary) Header contains no 'seq' field. Jun 13 16:25:52 sv01A hastd: [zfshast] (primary) Unable to receive reply header: Resource temporarily unavailable. Jun 13 16:25:52 sv01A kernel: Jun 13 16:25:52 sv01A hastd: [zfshast] (primary) Unable to receive reply header: Resource temporarily unavailable. Jun 13 16:25:52 sv01A hastd: [zfshast] (primary) Closing old incoming connection to 192.168.0.241. Jun 13 16:25:52 sv01A hastd: [zfshast] (primary) Closing old outgoing connection to 192.168.0.241. Jun 13 16:25:52 sv01A hastd: [zfshast] (primary) remote_recv: No requests, waiting. Jun 13 16:25:52 sv01A hastd: [zfshast] (primary) remote_guard: Checking connections. Jun 13 16:25:52 sv01A hastd: [zfshast] (primary) remote_guard: Reconnecting to 192.168.0.241. Jun 13 16:25:52 sv01A hastd: [zfshast] (primary) Unable to write synchronization data: Invalid argument. Jun 13 16:25:52 sv01A hastd: [zfshast] (primary) Unable to write synchronization data: Invalid argument. Jun 13 16:25:52 sv01A kernel: Jun 13 16:25:52 sv01A hastd: [zfshast] (primary) Unable to write synchronization data: Invalid argument. Jun 13 16:25:52 sv01A hastd: [zfshast] (primary) sync: (0x8012fbe20) Moving request to the free queue. Jun 13 16:25:52 sv01A hastd: [zfshast] (primary) Connected to 192.168.0.241. Jun 13 16:25:52 sv01A hastd: [zfshast] (primary) Successfully reconnected to 192.168.0.241. Jun 13 16:25:52 sv01A hastd: [zfshast] (primary) Synchronization started. 975039365120 bytes to go. Jun 13 16:25:52 sv01A hastd: [zfshast] (primary) sync: Taking free request. Jun 13 16:25:52 sv01A hastd: [zfshast] (primary) sync: (0x8012fbdd0) Got free request. Jun 13 16:25:52 sv01A hastd: [zfshast] (primary) sync: (0x8012fbdd0) Sending sync request: READ(0, 131072). Jun 13 16:25:52 sv01A hastd: [zfshast] (primary) sync: (0x8012fbdd0) Moving request to the send queue. Jun 13 16:25:52 sv01A hastd: [zfshast] (primary) local_send: (0x8012fbdd0) Got request. Jun 13 16:25:52 sv01A hastd: [zfshast] (primary) local_send: Taking request. Jun 13 16:25:52 sv01A hastd: [zfshast] (primary) sync: (0x8012fbdd0) Sending sync request: WRITE(0, 131072). Jun 13 16:25:52 sv01A hastd: [zfshast] (primary) sync: (0x8012fbdd0) Moving request to the send queue. Jun 13 16:25:52 sv01A hastd: [zfshast] (primary) sync: (0x8012fbdd0) Moving request to the send queues. Jun 13 16:25:52 sv01A hastd: [zfshast] (primary) remote_send: (0x8012fbdd0) Got request. Jun 13 16:25:52 sv01A hastd: [zfshast] (primary) remote_send: (0x8012fbdd0) Moving request to the recv queue. Jun 13 16:25:52 sv01A hastd: [zfshast] (primary) remote_send: Taking request. Jun 13 16:25:52 sv01A hastd: [zfshast] (primary) Header contains no 'seq' field. Jun 13 16:25:52 sv01A kernel: Jun 13 16:25:52 sv01A hastd: [zfshast] (primary) Header contains no 'seq' field. Jun 13 16:25:57 sv01A hastd: [zfshast] (primary) Unable to receive reply header: Resource temporarily unavailable. Jun 13 16:25:57 sv01A kernel: Jun 13 16:25:57 sv01A hastd: [zfshast] (primary) Unable to receive reply header: Resource temporarily unavailable. Jun 13 16:25:57 sv01A hastd: [zfshast] (primary) Closing old incoming connection to 192.168.0.241. Jun 13 16:25:57 sv01A hastd: [zfshast] (primary) Closing old outgoing connection to 192.168.0.241. Jun 13 16:25:57 sv01A hastd: [zfshast] (primary) remote_recv: No requests, waiting. Jun 13 16:25:57 sv01A hastd: [zfshast] (primary) remote_guard: Checking connections. Jun 13 16:25:57 sv01A hastd: [zfshast] (primary) remote_guard: Reconnecting to 192.168.0.241. Jun 13 16:25:57 sv01A hastd: [zfshast] (primary) Unable to write synchronization data: Invalid argument. Jun 13 16:25:57 sv01A hastd: [zfshast] (primary) Unable to write synchronization data: Invalid argument. Jun 13 16:25:57 sv01A kernel: Jun 13 16:25:57 sv01A hastd: [zfshast] (primary) Unable to write synchronization data: Invalid argument. Jun 13 16:25:57 sv01A hastd: [zfshast] (primary) sync: (0x8012fbdd0) Moving request to the free queue. Jun 13 16:25:57 sv01A hastd: [zfshast] (primary) Connected to 192.168.0.241. Jun 13 16:25:57 sv01A hastd: [zfshast] (primary) Successfully reconnected to 192.168.0.241. Jun 13 16:25:57 sv01A hastd: [zfshast] (primary) Synchronization started. 975039365120 bytes to go. Jun 13 16:25:57 sv01A hastd: [zfshast] (primary) sync: Taking free request. Jun 13 16:25:57 sv01A hastd: [zfshast] (primary) sync: (0x8012fbd80) Got free request. Jun 13 16:25:57 sv01A hastd: [zfshast] (primary) sync: (0x8012fbd80) Sending sync request: READ(0, 131072). Jun 13 16:25:57 sv01A hastd: [zfshast] (primary) sync: (0x8012fbd80) Moving request to the send queue. Jun 13 16:25:57 sv01A hastd: [zfshast] (primary) local_send: (0x8012fbd80) Got request. Jun 13 16:25:57 sv01A hastd: [zfshast] (primary) local_send: Taking request. Jun 13 16:25:57 sv01A hastd: [zfshast] (primary) sync: (0x8012fbd80) Sending sync request: WRITE(0, 131072). Jun 13 16:25:57 sv01A hastd: [zfshast] (primary) sync: (0x8012fbd80) Moving request to the send queue. Jun 13 16:25:57 sv01A hastd: [zfshast] (primary) sync: (0x8012fbd80) Moving request to the send queues. Jun 13 16:25:57 sv01A hastd: [zfshast] (primary) remote_send: (0x8012fbd80) Got request. Jun 13 16:25:57 sv01A hastd: [zfshast] (primary) remote_send: (0x8012fbd80) Moving request to the recv queue. Jun 13 16:25:57 sv01A hastd: [zfshast] (primary) remote_send: Taking request. Jun 13 16:25:57 sv01A hastd: [zfshast] (primary) Header contains no 'seq' field. Jun 13 16:25:57 sv01A kernel: Jun 13 16:25:57 sv01A hastd: [zfshast] (primary) Header contains no 'seq' field. Jun 13 16:26:02 sv01A hastd: [zfshast] (primary) Unable to receive reply header: Resource temporarily unavailable. Jun 13 16:26:02 sv01A kernel: Jun 13 16:26:02 sv01A hastd: [zfshast] (primary) Unable to receive reply header: Resource temporarily unavailable. Jun 13 16:26:02 sv01A hastd: [zfshast] (primary) Closing old incoming connection to 192.168.0.241. Jun 13 16:26:02 sv01A hastd: [zfshast] (primary) Closing old outgoing connection to 192.168.0.241. Jun 13 16:26:02 sv01A hastd: [zfshast] (primary) remote_recv: No requests, waiting. Jun 13 16:26:02 sv01A hastd: [zfshast] (primary) Unable to write synchronization data: Invalid argument. Jun 13 16:26:02 sv01A hastd: [zfshast] (primary) Unable to write synchronization data: Invalid argument. Jun 13 16:26:02 sv01A kernel: Jun 13 16:26:02 sv01A hastd: [zfshast] (primary) Unable to write synchronization data: Invalid argument. Jun 13 16:26:02 sv01A hastd: [zfshast] (primary) sync: (0x8012fbd80) Moving request to the free queue. Jun 13 16:26:02 sv01A hastd: [zfshast] (primary) remote_guard: Checking connections. Jun 13 16:26:02 sv01A hastd: [zfshast] (primary) remote_guard: Reconnecting to 192.168.0.241. Jun 13 16:26:02 sv01A hastd: [zfshast] (primary) Connected to 192.168.0.241. Jun 13 16:26:02 sv01A hastd: [zfshast] (primary) Successfully reconnected to 192.168.0.241. Jun 13 16:26:02 sv01A hastd: [zfshast] (primary) Synchronization started. 975039365120 bytes to go. Jun 13 16:26:02 sv01A hastd: [zfshast] (primary) sync: Taking free request. Jun 13 16:26:02 sv01A hastd: [zfshast] (primary) sync: (0x8012fbd30) Got free request. Jun 13 16:26:02 sv01A hastd: [zfshast] (primary) sync: (0x8012fbd30) Sending sync request: READ(0, 131072). Jun 13 16:26:02 sv01A hastd: [zfshast] (primary) sync: (0x8012fbd30) Moving request to the send queue. Jun 13 16:26:02 sv01A hastd: [zfshast] (primary) local_send: (0x8012fbd30) Got request. Jun 13 16:26:02 sv01A hastd: [zfshast] (primary) local_send: Taking request. Jun 13 16:26:02 sv01A hastd: [zfshast] (primary) sync: (0x8012fbd30) Sending sync request: WRITE(0, 131072). Jun 13 16:26:02 sv01A hastd: [zfshast] (primary) sync: (0x8012fbd30) Moving request to the send queue. Jun 13 16:26:02 sv01A hastd: [zfshast] (primary) sync: (0x8012fbd30) Moving request to the send queues. Jun 13 16:26:02 sv01A hastd: [zfshast] (primary) remote_send: (0x8012fbd30) Got request. Jun 13 16:26:02 sv01A hastd: [zfshast] (primary) remote_send: (0x8012fbd30) Moving request to the recv queue. Jun 13 16:26:02 sv01A hastd: [zfshast] (primary) remote_send: Taking request. Jun 13 16:26:02 sv01A hastd: [zfshast] (primary) Header contains no 'seq' field. Jun 13 16:26:02 sv01A kernel: Jun 13 16:26:02 sv01A hastd: [zfshast] (primary) Header contains no 'seq' field. Jun 13 16:26:07 sv01A hastd: [zfshast] (primary) Unable to receive reply header: Resource temporarily unavailable. Jun 13 16:26:07 sv01A kernel: Jun 13 16:26:07 sv01A hastd: [zfshast] (primary) Unable to receive reply header: Resource temporarily unavailable. Jun 13 16:26:07 sv01A hastd: [zfshast] (primary) Closing old incoming connection to 192.168.0.241. Jun 13 16:26:07 sv01A hastd: [zfshast] (primary) Closing old outgoing connection to 192.168.0.241. Jun 13 16:26:07 sv01A hastd: [zfshast] (primary) remote_recv: No requests, waiting. Jun 13 16:26:07 sv01A hastd: [zfshast] (primary) remote_guard: Checking connections. Jun 13 16:26:07 sv01A hastd: [zfshast] (primary) remote_guard: Reconnecting to 192.168.0.241. Jun 13 16:26:07 sv01A hastd: [zfshast] (primary) Unable to write synchronization data: Invalid argument. Jun 13 16:26:07 sv01A hastd: [zfshast] (primary) Unable to write synchronization data: Invalid argument. Jun 13 16:26:07 sv01A kernel: Jun 13 16:26:07 sv01A hastd: [zfshast] (primary) Unable to write synchronization data: Invalid argument. Jun 13 16:26:07 sv01A hastd: [zfshast] (primary) sync: (0x8012fbd30) Moving request to the free queue. Jun 13 16:26:08 sv01A hastd: [zfshast] (primary) Connected to 192.168.0.241. Jun 13 16:26:08 sv01A hastd: [zfshast] (primary) Successfully reconnected to 192.168.0.241. Jun 13 16:26:08 sv01A hastd: [zfshast] (primary) Synchronization started. 975039365120 bytes to go. Jun 13 16:26:08 sv01A hastd: [zfshast] (primary) sync: Taking free request. Jun 13 16:26:08 sv01A hastd: [zfshast] (primary) sync: (0x8012fbce0) Got free request. Jun 13 16:26:08 sv01A hastd: [zfshast] (primary) sync: (0x8012fbce0) Sending sync request: READ(0, 131072). Jun 13 16:26:08 sv01A hastd: [zfshast] (primary) sync: (0x8012fbce0) Moving request to the send queue. Jun 13 16:26:08 sv01A hastd: [zfshast] (primary) local_send: (0x8012fbce0) Got request. Jun 13 16:26:08 sv01A hastd: [zfshast] (primary) local_send: Taking request. Jun 13 16:26:08 sv01A hastd: [zfshast] (primary) sync: (0x8012fbce0) Sending sync request: WRITE(0, 131072). Jun 13 16:26:08 sv01A hastd: [zfshast] (primary) sync: (0x8012fbce0) Moving request to the send queue. Jun 13 16:26:08 sv01A hastd: [zfshast] (primary) sync: (0x8012fbce0) Moving request to the send queues. Jun 13 16:26:08 sv01A hastd: [zfshast] (primary) remote_send: (0x8012fbce0) Got request. Jun 13 16:26:08 sv01A hastd: [zfshast] (primary) remote_send: (0x8012fbce0) Moving request to the recv queue. Jun 13 16:26:08 sv01A hastd: [zfshast] (primary) remote_send: Taking request. Jun 13 16:26:08 sv01A hastd: [zfshast] (primary) Header contains no 'seq' field. Jun 13 16:26:08 sv01A kernel: Jun 13 16:26:08 sv01A hastd: [zfshast] (primary) Header contains no 'seq' field. Jun 13 16:26:13 sv01A hastd: [zfshast] (primary) Unable to receive reply header: Resource temporarily unavailable. Jun 13 16:26:13 sv01A kernel: Jun 13 16:26:13 sv01A hastd: [zfshast] (primary) Unable to receive reply header: Resource temporarily unavailable. Jun 13 16:26:13 sv01A hastd: [zfshast] (primary) Closing old incoming connection to 192.168.0.241. Jun 13 16:26:13 sv01A hastd: [zfshast] (primary) Closing old outgoing connection to 192.168.0.241. Jun 13 16:26:13 sv01A hastd: [zfshast] (primary) remote_recv: No requests, waiting. Jun 13 16:26:13 sv01A hastd: [zfshast] (primary) remote_guard: Checking connections. Jun 13 16:26:13 sv01A hastd: [zfshast] (primary) remote_guard: Reconnecting to 192.168.0.241. Jun 13 16:26:13 sv01A hastd: [zfshast] (primary) Unable to write synchronization data: Invalid argument. Jun 13 16:26:13 sv01A hastd: [zfshast] (primary) Unable to write synchronization data: Invalid argument. Jun 13 16:26:13 sv01A kernel: Jun 13 16:26:13 sv01A hastd: [zfshast] (primary) Unable to write synchronization data: Invalid argument. Jun 13 16:26:13 sv01A hastd: [zfshast] (primary) sync: (0x8012fbce0) Moving request to the free queue. Jun 13 16:26:13 sv01A hastd: [zfshast] (primary) Connected to 192.168.0.241. Jun 13 16:26:13 sv01A hastd: [zfshast] (primary) Successfully reconnected to 192.168.0.241. Jun 13 16:26:13 sv01A hastd: [zfshast] (primary) Synchronization started. 975039365120 bytes to go. Jun 13 16:26:13 sv01A hastd: [zfshast] (primary) sync: Taking free request. Jun 13 16:26:13 sv01A hastd: [zfshast] (primary) sync: (0x8012fbc90) Got free request. Jun 13 16:26:13 sv01A hastd: [zfshast] (primary) sync: (0x8012fbc90) Sending sync request: READ(0, 131072). Jun 13 16:26:13 sv01A hastd: [zfshast] (primary) sync: (0x8012fbc90) Moving request to the send queue. Jun 13 16:26:13 sv01A hastd: [zfshast] (primary) local_send: (0x8012fbc90) Got request. Jun 13 16:26:13 sv01A hastd: [zfshast] (primary) local_send: Taking request. Jun 13 16:26:13 sv01A hastd: [zfshast] (primary) sync: (0x8012fbc90) Sending sync request: WRITE(0, 131072). Jun 13 16:26:13 sv01A hastd: [zfshast] (primary) sync: (0x8012fbc90) Moving request to the send queue. Jun 13 16:26:13 sv01A hastd: [zfshast] (primary) sync: (0x8012fbc90) Moving request to the send queues. Jun 13 16:26:13 sv01A hastd: [zfshast] (primary) remote_send: (0x8012fbc90) Got request. Jun 13 16:26:13 sv01A hastd: [zfshast] (primary) remote_send: (0x8012fbc90) Moving request to the recv queue. Jun 13 16:26:13 sv01A hastd: [zfshast] (primary) remote_send: Taking request. Jun 13 16:26:13 sv01A hastd: [zfshast] (primary) Header contains no 'seq' field. Jun 13 16:26:13 sv01A kernel: Jun 13 16:26:13 sv01A hastd: [zfshast] (primary) Header contains no 'seq' field. Jun 13 16:26:17 sv01A kernel: ugen2.2: at usbus2 Jun 13 16:26:17 sv01A kernel: ukbd0: on usbus2 Jun 13 16:26:17 sv01A kernel: kbd2 at ukbd0 Jun 13 16:26:18 sv01A hastd: [zfshast] (primary) Unable to receive reply header: Resource temporarily unavailable. Jun 13 16:26:18 sv01A kernel: Jun 13 16:26:18 sv01A hastd: [zfshast] (primary) Unable to receive reply header: Resource temporarily unavailable. Jun 13 16:26:18 sv01A hastd: [zfshast] (primary) Closing old incoming connection to 192.168.0.241. Jun 13 16:26:18 sv01A hastd: [zfshast] (primary) Closing old outgoing connection to 192.168.0.241. Jun 13 16:26:18 sv01A hastd: [zfshast] (primary) remote_recv: No requests, waiting. Jun 13 16:26:18 sv01A hastd: [zfshast] (primary) remote_guard: Checking connections. Jun 13 16:26:18 sv01A hastd: [zfshast] (primary) remote_guard: Reconnecting to 192.168.0.241. Jun 13 16:26:18 sv01A hastd: [zfshast] (primary) Unable to write synchronization data: Invalid argument. Jun 13 16:26:18 sv01A hastd: [zfshast] (primary) Unable to write synchronization data: Invalid argument. Jun 13 16:26:18 sv01A kernel: Jun 13 16:26:18 sv01A hastd: [zfshast] (primary) Unable to write synchronization data: Invalid argument. Jun 13 16:26:18 sv01A hastd: [zfshast] (primary) sync: (0x8012fbc90) Moving request to the free queue. Jun 13 16:26:18 sv01A hastd: [zfshast] (primary) Connected to 192.168.0.241. Jun 13 16:26:18 sv01A hastd: [zfshast] (primary) Successfully reconnected to 192.168.0.241. Jun 13 16:26:18 sv01A hastd: [zfshast] (primary) Synchronization started. 975039365120 bytes to go. Jun 13 16:26:18 sv01A hastd: [zfshast] (primary) sync: Taking free request. Jun 13 16:26:18 sv01A hastd: [zfshast] (primary) sync: (0x8012fbc40) Got free request. Jun 13 16:26:18 sv01A hastd: [zfshast] (primary) sync: (0x8012fbc40) Sending sync request: READ(0, 131072). Jun 13 16:26:18 sv01A hastd: [zfshast] (primary) sync: (0x8012fbc40) Moving request to the send queue. Jun 13 16:26:18 sv01A hastd: [zfshast] (primary) local_send: (0x8012fbc40) Got request. Jun 13 16:26:18 sv01A hastd: [zfshast] (primary) local_send: Taking request. Jun 13 16:26:18 sv01A hastd: [zfshast] (primary) sync: (0x8012fbc40) Sending sync request: WRITE(0, 131072). Jun 13 16:26:18 sv01A hastd: [zfshast] (primary) sync: (0x8012fbc40) Moving request to the send queue. Jun 13 16:26:18 sv01A hastd: [zfshast] (primary) sync: (0x8012fbc40) Moving request to the send queues. Jun 13 16:26:18 sv01A hastd: [zfshast] (primary) remote_send: (0x8012fbc40) Got request. Jun 13 16:26:18 sv01A hastd: [zfshast] (primary) remote_send: (0x8012fbc40) Moving request to the recv queue. Jun 13 16:26:18 sv01A hastd: [zfshast] (primary) remote_send: Taking request. Jun 13 16:26:18 sv01A hastd: [zfshast] (primary) Header contains no 'seq' field. Jun 13 16:26:18 sv01A kernel: Jun 13 16:26:18 sv01A hastd: [zfshast] (primary) Header contains no 'seq' field. --------------070809040105010607030206-- From owner-freebsd-fs@FreeBSD.ORG Sun Jun 13 09:47:13 2010 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 52432106564A for ; Sun, 13 Jun 2010 09:47:13 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello089077043238.chello.pl [89.77.43.238]) by mx1.freebsd.org (Postfix) with ESMTP id 8EF408FC17 for ; Sun, 13 Jun 2010 09:47:11 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id B17C545C9B; Sun, 13 Jun 2010 11:47:09 +0200 (CEST) Received: from localhost (gate.wheel.pl [10.0.0.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id F20E945C8A; Sun, 13 Jun 2010 11:47:02 +0200 (CEST) Date: Sun, 13 Jun 2010 11:46:52 +0200 From: Pawel Jakub Dawidek To: Alexander Leidinger Message-ID: <20100613094652.GC1320@garage.freebsd.pl> References: <20100610162629.38992mazf0sfdqg0@webmail.leidinger.net> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="3Pql8miugIZX0722" Content-Disposition: inline In-Reply-To: <20100610162629.38992mazf0sfdqg0@webmail.leidinger.net> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT amd64 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=4.5 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.4 Cc: fs@freebsd.org Subject: Re: CFT: periodic scrubbing of ZFS pools X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Jun 2010 09:47:13 -0000 --3Pql8miugIZX0722 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Jun 10, 2010 at 04:26:29PM +0200, Alexander Leidinger wrote: > Hi, >=20 > as there seems to be interest in a periodic script to scrub zpools, I =20 > modified my monthly-POC into a daily script with parameters for which =20 > pools to scrub, how many days between scrubs (even different per pool, = =20 > if required), and several error checks (non-existing pool specified, =20 > scrub in progress). >=20 > You can find it at > http://www.Leidinger.net/FreeBSD/current-patches/600.scrub-zfs >=20 > Please put it into /etc/periodic/daily and test it. Possible =20 > periodic.conf variables are: > daily_scrub_zfs_enable=3D"YES" > daily_scrub_zfs_pools=3D"name1 name2 name3" # all if unset or empty > daily scrub_zfs_default_threshold=3D"" # default: 30 > daily_scrub_zfs__threshold=3D"" >=20 > If there is no specific threshold for a pool (=3D days between scrubs), = =20 > the default threshold is used. Patch looks good. The only minor issue I found is that in: _last_scrub=3D$(zpool history ${pool} | grep scrub | tail -1 |\ cut -d ' ' -f 1) grep is definitely too weak. If there is a pool or even dataset with 'scrub' word in it, it will be confused. Also when someone stops the scrub it will be stored in history as 'zpool scrub -s '. I proposed something like this: egrep '^[0-9\.\:\-]{19} zpool scrub '${pool}'$' Currently there is no way to tell if 'zpool scrub -s' stopped a scrub or if there was no scrub running. In newer versions of ZFS, it has a property which tells when the last scrub finished. Once we update ZFS we could take advantage of it. All in all, with better grep I'd like to see it committed. --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --3Pql8miugIZX0722 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkwUqQwACgkQForvXbEpPzTvuACfSsFubrO5blWrHiOWKBfzU+iR 9wgAoJ7pXLwDJ9BRdGWWM+bU9lcgzQdS =XiCJ -----END PGP SIGNATURE----- --3Pql8miugIZX0722-- From owner-freebsd-fs@FreeBSD.ORG Sun Jun 13 10:24:19 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DC108106566B for ; Sun, 13 Jun 2010 10:24:19 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello089077043238.chello.pl [89.77.43.238]) by mx1.freebsd.org (Postfix) with ESMTP id 1EB3E8FC19 for ; Sun, 13 Jun 2010 10:24:18 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 891DD45CBA; Sun, 13 Jun 2010 12:24:17 +0200 (CEST) Received: from localhost (gate.wheel.pl [10.0.0.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id A825545C99; Sun, 13 Jun 2010 12:24:11 +0200 (CEST) Date: Sun, 13 Jun 2010 12:24:01 +0200 From: Pawel Jakub Dawidek To: "hiroshi@soupacific.com" Message-ID: <20100613102401.GE1320@garage.freebsd.pl> References: <4C10B526.4040908@soupacific.com> <20100612104336.GA2253@garage.freebsd.pl> <4C1372E0.1000903@soupacific.com> <20100612142311.GF2253@garage.freebsd.pl> <4C139F9C.2090305@soupacific.com> <86iq5oc82y.fsf@kopusha.home.net> <4C14215D.9090304@soupacific.com> <20100613003635.GA60012@icarus.home.lan> <20100613074921.GB1320@garage.freebsd.pl> <4C149A5C.3070401@soupacific.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="pyE8wggRBhVBcj8z" Content-Disposition: inline In-Reply-To: <4C149A5C.3070401@soupacific.com> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT amd64 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=4.5 tests=ALL_TRUSTED,BAYES_00, TO_ADDRESS_EQ_REAL autolearn=ham version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: FreeBSD 8.1 and HAST X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Jun 2010 10:24:19 -0000 --pyE8wggRBhVBcj8z Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Jun 13, 2010 at 05:44:12PM +0900, hiroshi@soupacific.com wrote: > Good morning ! >=20 > here are the file all.log each. >=20 > I hope this might help you some idea. [...] Much better now. > Jun 13 16:25:37 sv01A hastd: [zfshast] (primary) Header contains no 'seq'= field. This is the most important bit from the primary node. The header either does not contain 'seq' field or this field is 0. It can only be 0 if you have old kernel. With recent kernel geom_gate.ko was modified to start seq at 1, so this should not happen. Can verify that your kernel is in sync with your userland? --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --pyE8wggRBhVBcj8z Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkwUscAACgkQForvXbEpPzQtmwCgzACI8bLGzY1aAS0e9RpJDpyG bEAAn0Aw8TPJokbwfpsN86Dp3IinJuIM =ifgd -----END PGP SIGNATURE----- --pyE8wggRBhVBcj8z-- From owner-freebsd-fs@FreeBSD.ORG Sun Jun 13 10:56:36 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A290C1065670; Sun, 13 Jun 2010 10:56:36 +0000 (UTC) (envelope-from to.my.trociny@gmail.com) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id A79F08FC12; Sun, 13 Jun 2010 10:56:35 +0000 (UTC) Received: by bwz2 with SMTP id 2so1896329bwz.13 for ; Sun, 13 Jun 2010 03:56:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:to:cc:subject:references :x-comment-to:date:in-reply-to:message-id:user-agent:mime-version :content-type; bh=NMyagLLt8sAUrK2s6SrJw1A0jka1U0jNqph/0pTaXpM=; b=pjBMfwojMS8hvMr6mf1WvWuWQdpGd/TSeTzeNlNjBEuk3nVKkRRTnq6hAGx4yfog1T aWBklUR8PM/Od26PCcnY1IO1f57GgfujJ09+youqsvBlnxEPzNzvuG8vNV+yTBFEP+5j SyuOQK3siLjIQ9BaBBbhMC/PhTqb7F6RBhUvY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:cc:subject:references:x-comment-to:date:in-reply-to :message-id:user-agent:mime-version:content-type; b=OmJHmaw8ruZFrapSTzscDIhS6v5UiUGA9lilAIXZeo1mCEwOCMqUvVnw8XHuNri5Mr 3hsTsfFucxkcqFqOwdDxBGIS40dCcoGdj7MIv5/XNztrI4e5Za6z9207MyskEiScKFvW dIFCzjCwYl+sULsIoXQc9uzJRNPUzXmzitegw= Received: by 10.204.83.211 with SMTP id g19mr3214976bkl.44.1276426594569; Sun, 13 Jun 2010 03:56:34 -0700 (PDT) Received: from localhost ([95.69.160.52]) by mx.google.com with ESMTPS id v3sm14165596bkz.10.2010.06.13.03.56.33 (version=TLSv1/SSLv3 cipher=RC4-MD5); Sun, 13 Jun 2010 03:56:33 -0700 (PDT) From: Mikolaj Golub To: Pawel Jakub Dawidek References: <20100423061521.GC1670@garage.freebsd.pl> <4BD17B0D.5080601@soupacific.com> <4C10B526.4040908@soupacific.com> <20100612104336.GA2253@garage.freebsd.pl> <4C1372E0.1000903@soupacific.com> <20100612142311.GF2253@garage.freebsd.pl> <4C139F9C.2090305@soupacific.com> <86iq5oc82y.fsf@kopusha.home.net> <4C14215D.9090304@soupacific.com> <20100613003635.GA60012@icarus.home.lan> <20100613074921.GB1320@garage.freebsd.pl> <4C149A5C.3070401@soupacific.com> X-Comment-To: hiroshi@soupacific.com Date: Sun, 13 Jun 2010 13:56:31 +0300 In-Reply-To: <4C149A5C.3070401@soupacific.com> (hiroshi@soupacific.com's message of "Sun, 13 Jun 2010 17:44:12 +0900") Message-ID: <86ljajw68w.fsf@kopusha.home.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.2 (berkeley-unix) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Cc: freebsd-fs@freebsd.org Subject: Re: FreeBSD 8.1 and HAST X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Jun 2010 10:56:36 -0000 --=-=-= Just a note, not related to this problem: On Sun, 13 Jun 2010 17:44:12 +0900 hiroshi@soupacific.com wrote: h> Jun 13 17:35:54 sv01B hastd: Accepting connection to tcp4://0.0.0.0:8457. h> Jun 13 17:35:54 sv01B hastd: Connection from tcp4://192.168.0.241:8457 to tcp4://192.168.0.240:24260. It looks like the addresses in this message are swapped. Also I have noticed one typo in another log message and the issue with send thread logging as if it is disk thread (I reported about this earlier in this thread). Please look at the attached patch. -- Mikolaj Golub --=-=-= Content-Type: text/x-patch Content-Disposition: inline; filename=hastd.log.patch Index: sbin/hastd/hastd.c =================================================================== --- sbin/hastd/hastd.c (revision 208960) +++ sbin/hastd/hastd.c (working copy) @@ -200,7 +200,7 @@ listen_accept(void) proto_local_address(conn, laddr, sizeof(laddr)); proto_remote_address(conn, raddr, sizeof(raddr)); - pjdlog_info("Connection from %s to %s.", laddr, raddr); + pjdlog_info("Connection from %s to %s.", raddr, laddr); /* Error in setting timeout is not critical, but why should it fail? */ if (proto_timeout(conn, HAST_TIMEOUT) < 0) @@ -286,7 +286,7 @@ listen_accept(void) if (token != NULL && memcmp(token, res->hr_token, sizeof(res->hr_token)) != 0) { pjdlog_error("Token received from %s doesn't match.", raddr); - nv_add_stringf(nverr, "errmsg", "Toke doesn't match."); + nv_add_stringf(nverr, "errmsg", "Token doesn't match."); goto fail; } /* Index: sbin/hastd/secondary.c =================================================================== --- sbin/hastd/secondary.c (revision 208960) +++ sbin/hastd/secondary.c (working copy) @@ -687,7 +687,7 @@ send_thread(void *arg) pjdlog_exit(EX_TEMPFAIL, "Unable to send reply."); } nv_free(nvout); - pjdlog_debug(2, "disk: (%p) Moving request to the free queue.", + pjdlog_debug(2, "send: (%p) Moving request to the free queue.", hio); nv_free(hio->hio_nv); hio->hio_error = 0; --=-=-=-- From owner-freebsd-fs@FreeBSD.ORG Sun Jun 13 10:57:22 2010 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 135C51065688; Sun, 13 Jun 2010 10:57:22 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from mail.ebusiness-leidinger.de (mail.ebusiness-leidinger.de [217.11.53.44]) by mx1.freebsd.org (Postfix) with ESMTP id C18138FC19; Sun, 13 Jun 2010 10:57:21 +0000 (UTC) Received: from outgoing.leidinger.net (pD9E2C8F8.dip.t-dialin.net [217.226.200.248]) by mail.ebusiness-leidinger.de (Postfix) with ESMTPSA id A720584405C; Sun, 13 Jun 2010 12:57:18 +0200 (CEST) Received: from unknown (unknown [192.168.2.110]) by outgoing.leidinger.net (Postfix) with ESMTP id CBAFD53C7; Sun, 13 Jun 2010 12:57:14 +0200 (CEST) Date: Sun, 13 Jun 2010 12:57:16 +0200 From: Alexander Leidinger To: Pawel Jakub Dawidek Message-ID: <20100613125716.000078df@unknown> In-Reply-To: <20100613094652.GC1320@garage.freebsd.pl> References: <20100610162629.38992mazf0sfdqg0@webmail.leidinger.net> <20100613094652.GC1320@garage.freebsd.pl> X-Mailer: Claws Mail 3.7.2cvs15 (GTK+ 2.16.0; i586-pc-mingw32msvc) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-EBL-MailScanner-Information: Please contact the ISP for more information X-EBL-MailScanner-ID: A720584405C.A7001 X-EBL-MailScanner: Found to be clean X-EBL-MailScanner-SpamCheck: not spam, spamhaus-ZEN, SpamAssassin (not cached, score=-1, required 6, autolearn=disabled, ALL_TRUSTED -1.00) X-EBL-MailScanner-From: alexander@leidinger.net X-EBL-MailScanner-Watermark: 1277031439.38518@dvqJJXicoKqQg75APDwRQA X-EBL-Spam-Status: No Cc: fs@freebsd.org Subject: Re: CFT: periodic scrubbing of ZFS pools X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Jun 2010 10:57:22 -0000 On Sun, 13 Jun 2010 11:46:52 +0200 Pawel Jakub Dawidek wrote: > I proposed something like this: > > egrep '^[0-9\.\:\-]{19} zpool scrub '${pool}'$' > > Currently there is no way to tell if 'zpool scrub -s' stopped a scrub > or if there was no scrub running. In newer versions of ZFS, it has a > property which tells when the last scrub finished. Once we update ZFS > we could take advantage of it. > > All in all, with better grep I'd like to see it committed. I have a look at getting enough time to write the corresponding man-page stuff on Tuesday, change the script as you suggested for this (and another change for the case when there was no scrub run ever before), and then commit it. Thanks for the suggestion, Alexander. From owner-freebsd-fs@FreeBSD.ORG Sun Jun 13 11:26:38 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 610AE1065670; Sun, 13 Jun 2010 11:26:38 +0000 (UTC) (envelope-from hiroshi@soupacific.com) Received: from mail.soupacific.com (mail.soupacific.com [211.19.53.201]) by mx1.freebsd.org (Postfix) with ESMTP id 2AFFD8FC0A; Sun, 13 Jun 2010 11:26:37 +0000 (UTC) Received: from [127.0.0.1] (unknown [192.168.1.239]) by mail.soupacific.com (Postfix) with ESMTP id 95A876BB33; Sun, 13 Jun 2010 11:18:02 +0000 (UTC) Message-ID: <4C14C066.7050505@soupacific.com> Date: Sun, 13 Jun 2010 20:26:30 +0900 From: "hiroshi@soupacific.com" User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4 MIME-Version: 1.0 To: Pawel Jakub Dawidek References: <4C10B526.4040908@soupacific.com> <20100612104336.GA2253@garage.freebsd.pl> <4C1372E0.1000903@soupacific.com> <20100612142311.GF2253@garage.freebsd.pl> <4C139F9C.2090305@soupacific.com> <86iq5oc82y.fsf@kopusha.home.net> <4C14215D.9090304@soupacific.com> <20100613003635.GA60012@icarus.home.lan> <20100613074921.GB1320@garage.freebsd.pl> <4C149A5C.3070401@soupacific.com> <20100613102401.GE1320@garage.freebsd.pl> In-Reply-To: <20100613102401.GE1320@garage.freebsd.pl> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: FreeBSD 8.1 and HAST X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Jun 2010 11:26:38 -0000 > >> Jun 13 16:25:37 sv01A hastd: [zfshast] (primary) Header contains no 'seq' field. > > This is the most important bit from the primary node. > > The header either does not contain 'seq' field or this field is 0. It > can only be 0 if you have old kernel. With recent kernel geom_gate.ko > was modified to start seq at 1, so this should not happen. > > Can verify that your kernel is in sync with your userland? > I did csup, builworld buildkernel and instalkernel . geom_gate.ko and uanme -a are same date! Stable-supfile *default release=cvs tag=RELENG_8 g_gate.c and g_gate.h Apr 18 What else I should check ? > Can verify that your kernel is in sync with your userland? Is there any other way to check ? Thanks Hiroshi From owner-freebsd-fs@FreeBSD.ORG Sun Jun 13 12:41:10 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 84F611065670 for ; Sun, 13 Jun 2010 12:41:10 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta13.emeryville.ca.mail.comcast.net (qmta13.emeryville.ca.mail.comcast.net [76.96.27.243]) by mx1.freebsd.org (Postfix) with ESMTP id 679758FC21 for ; Sun, 13 Jun 2010 12:41:09 +0000 (UTC) Received: from omta11.emeryville.ca.mail.comcast.net ([76.96.30.36]) by qmta13.emeryville.ca.mail.comcast.net with comcast id VQez1e0050mlR8UADQh9xA; Sun, 13 Jun 2010 12:41:09 +0000 Received: from koitsu.dyndns.org ([98.248.46.159]) by omta11.emeryville.ca.mail.comcast.net with comcast id VQh71e0063S48mS8XQh85K; Sun, 13 Jun 2010 12:41:08 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 9B14C9B418; Sun, 13 Jun 2010 05:41:07 -0700 (PDT) Date: Sun, 13 Jun 2010 05:41:07 -0700 From: Jeremy Chadwick To: hiroshi@soupacific.com Message-ID: <20100613124107.GA77002@icarus.home.lan> References: <4C1372E0.1000903@soupacific.com> <20100612142311.GF2253@garage.freebsd.pl> <4C139F9C.2090305@soupacific.com> <86iq5oc82y.fsf@kopusha.home.net> <4C14215D.9090304@soupacific.com> <20100613003635.GA60012@icarus.home.lan> <20100613074921.GB1320@garage.freebsd.pl> <4C149A5C.3070401@soupacific.com> <20100613102401.GE1320@garage.freebsd.pl> <4C14C066.7050505@soupacific.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4C14C066.7050505@soupacific.com> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-fs@freebsd.org, Pawel Jakub Dawidek Subject: Re: FreeBSD 8.1 and HAST X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Jun 2010 12:41:10 -0000 On Sun, Jun 13, 2010 at 08:26:30PM +0900, hiroshi@soupacific.com wrote: > > > > >>Jun 13 16:25:37 sv01A hastd: [zfshast] (primary) Header contains no 'seq' field. > > > >This is the most important bit from the primary node. > > > >The header either does not contain 'seq' field or this field is 0. It > >can only be 0 if you have old kernel. With recent kernel geom_gate.ko > >was modified to start seq at 1, so this should not happen. > > > >Can verify that your kernel is in sync with your userland? > > > > I did csup, builworld buildkernel and instalkernel . > geom_gate.ko and uanme -a are same date! This is not verifying that the kernel and userland are in sync. geom_gate.ko is installed via "installkernel". You should never just buildkernel/installkernel/reboot; you should do everything, including world. The procedure to follow is described in /usr/src/Makefile. I've included it below, just for posterity. # 1. `cd /usr/src' (or to the directory containing your source tree). # 2. `make buildworld' # 3. `make buildkernel KERNCONF=YOUR_KERNEL_HERE' (default is GENERIC). # 4. `make installkernel KERNCONF=YOUR_KERNEL_HERE' (default is GENERIC). # [steps 3. & 4. can be combined by using the "kernel" target] # 5. `reboot' (in single user mode: boot -s from the loader prompt). # 6. `mergemaster -p' # 7. `make installworld' # 8. `make delete-old' # 9. `mergemaster' (you may wish to use -U or -ai). # 10. `reboot' # 11. `make delete-old-libs' (in case no 3rd party program uses them anymore) Please do not skip/avoid the reboot-into-single-user-mode step. If you don't have physical access the machine or OOB (out-of-band, e.g. serial console, IPMI, LOM, etc.) access, you can "live dangerously" by doing the installworld step after installkernel then rebooting. But be aware there are situations where some files in /libexec or /usr/libexec may not get updated, which can be immediately devastating. I've seen this happen on a couple of occasions. Considered yourself warned. :-) -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Sun Jun 13 14:42:49 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 183931065674; Sun, 13 Jun 2010 14:42:49 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 5AA518FC13; Sun, 13 Jun 2010 14:42:47 +0000 (UTC) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id o5DEggSl032257 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 13 Jun 2010 17:42:42 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4) with ESMTP id o5DEggru083594; Sun, 13 Jun 2010 17:42:42 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4/Submit) id o5DEgekc083593; Sun, 13 Jun 2010 17:42:40 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 13 Jun 2010 17:42:40 +0300 From: Kostik Belousov To: Attilio Rao Message-ID: <20100613144240.GV13238@deviant.kiev.zoral.com.ua> References: <20100603143501.GA3176@a91-153-117-195.elisa-laajakaista.fi> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="3jK+0sHr6j/jwA0V" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-2.3 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_50, DNS_FROM_OPENWHOIS autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-fs@freebsd.org, Jaakko Heinonen Subject: Re: syncer vnode leak because of nmount() race X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Jun 2010 14:42:49 -0000 --3jK+0sHr6j/jwA0V Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jun 04, 2010 at 05:00:49PM +0200, Attilio Rao wrote: > 2010/6/3 Jaakko Heinonen : > > > > Hi, > > > > I have been playing with devfs and stress2 recently and I was able to > > trigger a syncer vnode leak with devfs.sh. As far as I can tell it is a > > nmount(2) (initial) mount race against update mount. The code in > > vfs_domount() is protected with vfs_busy()/vfs_unbusy() but it doesn't > > protect against update mounts. The protection by Giant is defeated > > because devfs_root() may sleep due to sx_xlock(). vfs_domount() calls > > VFS_ROOT() before allocating the syncer vnode. VI_MOUNT vnode flag > > doesn't provide sufficient protection against update mounts either. >=20 > Thanks for this bug report. > I think that, luckilly, it is not a very common condition to have the > mount still in flight and get updates... :) > However, I think that the real breakage here is that the check on > mnt->mnt_syncer is done lockless and it is unsafe. It might really be > protected by the mount interlock but that's tricky because > vfs_allocate_syncvnode() sleeps. Also re-checking the condition (after > the allocation takes place) is not too simple here. Is it ? I think that the patch below would fix the issue, by syncronizing mnt_syncer by syncer mutex. On the other hand, I prefer the jh' patch, because it seemingly disallows parallel updates of the mount point. I believe that mp->mnt_vnodecovered should be stable after the call to vfs_busy() succeeded. > I found also this bug when rewriting the syncer and I resolved that by > using a separate flag for that (in my case it was simpler and more > beneficial actually for some other reasons, but you may do the same > thing with a mnt_kern_flag entry). > If you have no time I can work actively on the patch, even if I'm > confident, luckilly, this problem is very hard to happen in the real > life. >=20 > Additively, note that vfs_busy() here is not intended to protect > against such situations but against unmount. > Also, I'm very unsure about what Giant protection might bring here. > IMHO we might probabilly remove it at all as long as all the sleeping > point present make it completely unuseful if anything (and I don't see > a reason why it may be needed). >=20 > > PS. vfs_unbusy(9) manual page is out of date after r184554 and IMO > > =9A =9Avfs_busy(9) manual page is misleading because it talks about > > =9A =9Asynchronizing access to a mount point. >=20 > May you be more precise on what is misleading please? diff --git a/sys/kern/vfs_mount.c b/sys/kern/vfs_mount.c index 088d939..053bd68 100644 --- a/sys/kern/vfs_mount.c +++ b/sys/kern/vfs_mount.c @@ -1034,14 +1034,10 @@ vfs_domount( else mp->mnt_kern_flag &=3D ~MNTK_ASYNC; MNT_IUNLOCK(mp); - if ((mp->mnt_flag & MNT_RDONLY) =3D=3D 0) { - if (mp->mnt_syncer =3D=3D NULL) - error =3D vfs_allocate_syncvnode(mp); - } else { - if (mp->mnt_syncer !=3D NULL) - vrele(mp->mnt_syncer); - mp->mnt_syncer =3D NULL; - } + if ((mp->mnt_flag & MNT_RDONLY) =3D=3D 0) + error =3D vfs_allocate_syncvnode(mp); + else + vfs_deallocate_syncvnode(mp); vfs_unbusy(mp); VI_LOCK(vp); vp->v_iflag &=3D ~VI_MOUNT; diff --git a/sys/kern/vfs_subr.c b/sys/kern/vfs_subr.c index ff6744a..6e4bb12 100644 --- a/sys/kern/vfs_subr.c +++ b/sys/kern/vfs_subr.c @@ -3400,12 +3400,31 @@ vfs_allocate_syncvnode(struct mount *mp) /* XXX - vn_syncer_add_to_worklist() also grabs and drops sync_mtx. */ mtx_lock(&sync_mtx); sync_vnode_count++; + if (mp->mnt_syncer =3D=3D NULL) { + mp->mnt_syncer =3D vp; + vp =3D NULL; + } mtx_unlock(&sync_mtx); BO_UNLOCK(bo); - mp->mnt_syncer =3D vp; + if (vp !=3D NULL) + vgone(vp); return (0); } =20 +void +vfs_deallocate_syncvnode(struct mount *mp) +{ + struct vnode *vp; + + mtx_lock(&sync_mtx); + vp =3D mp->mnt_syncer; + if (vp !=3D NULL) + mp->mnt_syncer =3D NULL; + mtx_unlock(&sync_mtx); + if (vp !=3D NULL) + vrele(vp); +} + /* * Do a lazy sync of the filesystem. */ @@ -3484,15 +3503,16 @@ sync_reclaim(struct vop_reclaim_args *ap) =20 bo =3D &vp->v_bufobj; BO_LOCK(bo); - vp->v_mount->mnt_syncer =3D NULL; + mtx_lock(&sync_mtx); + if (vp->v_mount->mnt_syncer =3D=3D vp) + vp->v_mount->mnt_syncer =3D NULL; if (bo->bo_flag & BO_ONWORKLST) { - mtx_lock(&sync_mtx); LIST_REMOVE(bo, bo_synclist); syncer_worklist_len--; sync_vnode_count--; - mtx_unlock(&sync_mtx); bo->bo_flag &=3D ~BO_ONWORKLST; } + mtx_unlock(&sync_mtx); BO_UNLOCK(bo); =20 return (0); diff --git a/sys/sys/mount.h b/sys/sys/mount.h index 20dcf64..dcff206 100644 --- a/sys/sys/mount.h +++ b/sys/sys/mount.h @@ -731,6 +731,7 @@ int vfs_busy(struct mount *, int); int vfs_export /* process mount export info */ (struct mount *, struct export_args *); int vfs_allocate_syncvnode(struct mount *); +void vfs_deallocate_syncvnode(struct mount *); int vfs_donmount(struct thread *td, int fsflags, struct uio *fsoptions); void vfs_getnewfsid(struct mount *); struct cdev *vfs_getrootfsid(struct mount *); --3jK+0sHr6j/jwA0V Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (FreeBSD) iEYEARECAAYFAkwU7l8ACgkQC3+MBN1Mb4hljgCfaImXDvro2UDwP2GHdnHVvgDS ziYAoNtaKSPCKsLL89I3OmV//8arTa9N =BR4w -----END PGP SIGNATURE----- --3jK+0sHr6j/jwA0V-- From owner-freebsd-fs@FreeBSD.ORG Sun Jun 13 17:23:26 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DE2EE1065674 for ; Sun, 13 Jun 2010 17:23:26 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello089077043238.chello.pl [89.77.43.238]) by mx1.freebsd.org (Postfix) with ESMTP id 6DD448FC1F for ; Sun, 13 Jun 2010 17:23:25 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 10A5A45DD8; Sun, 13 Jun 2010 19:23:23 +0200 (CEST) Received: from localhost (chello089077043238.chello.pl [89.77.43.238]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id DE8C3456B1; Sun, 13 Jun 2010 19:23:16 +0200 (CEST) Date: Sun, 13 Jun 2010 19:23:08 +0200 From: Pawel Jakub Dawidek To: "hiroshi@soupacific.com" Message-ID: <20100613172308.GA2201@garage.freebsd.pl> References: <4C1372E0.1000903@soupacific.com> <20100612142311.GF2253@garage.freebsd.pl> <4C139F9C.2090305@soupacific.com> <86iq5oc82y.fsf@kopusha.home.net> <4C14215D.9090304@soupacific.com> <20100613003635.GA60012@icarus.home.lan> <20100613074921.GB1320@garage.freebsd.pl> <4C149A5C.3070401@soupacific.com> <20100613102401.GE1320@garage.freebsd.pl> <4C14C066.7050505@soupacific.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="wRRV7LY7NUeQGEoC" Content-Disposition: inline In-Reply-To: <4C14C066.7050505@soupacific.com> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT amd64 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-0.6 required=4.5 tests=BAYES_00,RCVD_IN_SORBS_DUL, TO_ADDRESS_EQ_REAL autolearn=no version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: FreeBSD 8.1 and HAST X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Jun 2010 17:23:26 -0000 --wRRV7LY7NUeQGEoC Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Jun 13, 2010 at 08:26:30PM +0900, hiroshi@soupacific.com wrote: > What else I should check ? [...] Could you paste the output of: # sysctl kern.geom.gate --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --wRRV7LY7NUeQGEoC Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkwVE/wACgkQForvXbEpPzST9gCfY0o8l8o6g4AlGYEb6QTCXr3T ed8AoNADSTRB9yCQ9KQHUz6AyGvaOtfA =f7hZ -----END PGP SIGNATURE----- --wRRV7LY7NUeQGEoC-- From owner-freebsd-fs@FreeBSD.ORG Sun Jun 13 22:09:33 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0DA711065675; Sun, 13 Jun 2010 22:09:33 +0000 (UTC) (envelope-from hiroshi@soupacific.com) Received: from mail.soupacific.com (mail.soupacific.com [211.19.53.201]) by mx1.freebsd.org (Postfix) with ESMTP id CFD3F8FC0C; Sun, 13 Jun 2010 22:09:32 +0000 (UTC) Received: from [127.0.0.1] (unknown [192.168.1.239]) by mail.soupacific.com (Postfix) with ESMTP id BEA626BD18; Sun, 13 Jun 2010 22:00:57 +0000 (UTC) Message-ID: <4C155714.7090201@soupacific.com> Date: Mon, 14 Jun 2010 07:09:24 +0900 From: "hiroshi@soupacific.com" User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4 MIME-Version: 1.0 To: Pawel Jakub Dawidek References: <4C1372E0.1000903@soupacific.com> <20100612142311.GF2253@garage.freebsd.pl> <4C139F9C.2090305@soupacific.com> <86iq5oc82y.fsf@kopusha.home.net> <4C14215D.9090304@soupacific.com> <20100613003635.GA60012@icarus.home.lan> <20100613074921.GB1320@garage.freebsd.pl> <4C149A5C.3070401@soupacific.com> <20100613102401.GE1320@garage.freebsd.pl> <4C14C066.7050505@soupacific.com> <20100613172308.GA2201@garage.freebsd.pl> In-Reply-To: <20100613172308.GA2201@garage.freebsd.pl> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: FreeBSD 8.1 and HAST X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Jun 2010 22:09:33 -0000 > Could you paste the output of: > > # sysctl kern.geom.gate > Both 8.1 and 9.0-Current returns same #sysctl kern.geom.gate sysctl: unknown oid 'kern.geom.gate' Thanks Hiroshi From owner-freebsd-fs@FreeBSD.ORG Sun Jun 13 22:25:31 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 606051065670; Sun, 13 Jun 2010 22:25:31 +0000 (UTC) (envelope-from hiroshi@soupacific.com) Received: from mail.soupacific.com (mail.soupacific.com [211.19.53.201]) by mx1.freebsd.org (Postfix) with ESMTP id 2DC738FC16; Sun, 13 Jun 2010 22:25:30 +0000 (UTC) Received: from [127.0.0.1] (unknown [192.168.1.239]) by mail.soupacific.com (Postfix) with ESMTP id 711E46BD28; Sun, 13 Jun 2010 22:16:56 +0000 (UTC) Message-ID: <4C155AD3.3000601@soupacific.com> Date: Mon, 14 Jun 2010 07:25:23 +0900 From: "hiroshi@soupacific.com" User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4 MIME-Version: 1.0 To: Pawel Jakub Dawidek References: <4C1372E0.1000903@soupacific.com> <20100612142311.GF2253@garage.freebsd.pl> <4C139F9C.2090305@soupacific.com> <86iq5oc82y.fsf@kopusha.home.net> <4C14215D.9090304@soupacific.com> <20100613003635.GA60012@icarus.home.lan> <20100613074921.GB1320@garage.freebsd.pl> <4C149A5C.3070401@soupacific.com> <20100613102401.GE1320@garage.freebsd.pl> <4C14C066.7050505@soupacific.com> <20100613172308.GA2201@garage.freebsd.pl> In-Reply-To: <20100613172308.GA2201@garage.freebsd.pl> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: FreeBSD 8.1 and HAST X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Jun 2010 22:25:31 -0000 Sorry ignore last mail's reply ! > > Could you paste the output of: > > # sysctl kern.geom.gate > Both 8.1 and 9.0 returns kern.geom.gate.maxunits: 256 kern.geom.gate.debug: 0 Is this O.K. ? Thanks Hiroshi From owner-freebsd-fs@FreeBSD.ORG Mon Jun 14 07:28:21 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 410811065673; Mon, 14 Jun 2010 07:28:21 +0000 (UTC) (envelope-from to.my.trociny@gmail.com) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id 465958FC08; Mon, 14 Jun 2010 07:28:20 +0000 (UTC) Received: by bwz2 with SMTP id 2so2513059bwz.13 for ; Mon, 14 Jun 2010 00:28:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:to:cc:subject:references :x-comment-to:date:in-reply-to:message-id:user-agent:mime-version :content-type; bh=r4lW3Dxwyfb4GSQX9fkvZYS9nqRDoBHOPZkGg4w2cCM=; b=wjzDCQ6ksyS8yYm4Vm/Hrq46hayw6hqh7T+tHGjwpQCsIGEqzvgiUMmHWK5jdaB2Qf eL70RqSanutCUKDxG6511XGzGl/npWa3u9zga5fXSOu82snA5xNCN3vNmV53Kx1ujhYv JGwUrGfHiDElZHMN+y9o2XrahgEmAWUAIR0JE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:cc:subject:references:x-comment-to:date:in-reply-to :message-id:user-agent:mime-version:content-type; b=NIA1Xh6kRERQ76kMWYJ6g+9iEM4EzmAzfCtctnWfJ99JHxaI7+Qg6jpL81OpZ5Sqkp EB6IqJMbBZ8vniX9cIRhaG5JojE6Qgphii/SrK7bVsGhLVELIVSSY8FCnkL6L7MlbFM5 J0jwOQMlXzoYJmImXkZ9wwlQPaHw3t0sF+w3s= Received: by 10.204.160.156 with SMTP id n28mr4039738bkx.1.1276500499239; Mon, 14 Jun 2010 00:28:19 -0700 (PDT) Received: from localhost ([95.69.160.52]) by mx.google.com with ESMTPS id z17sm18299426bkx.18.2010.06.14.00.28.17 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 14 Jun 2010 00:28:17 -0700 (PDT) From: Mikolaj Golub To: Pawel Jakub Dawidek References: <4C10B526.4040908@soupacific.com> <20100612104336.GA2253@garage.freebsd.pl> <4C1372E0.1000903@soupacific.com> <20100612142311.GF2253@garage.freebsd.pl> <4C139F9C.2090305@soupacific.com> <86iq5oc82y.fsf@kopusha.home.net> <4C14215D.9090304@soupacific.com> <20100613003635.GA60012@icarus.home.lan> <20100613074921.GB1320@garage.freebsd.pl> <4C149A5C.3070401@soupacific.com> <20100613102401.GE1320@garage.freebsd.pl> X-Comment-To: Pawel Jakub Dawidek Date: Mon, 14 Jun 2010 10:28:15 +0300 In-Reply-To: <20100613102401.GE1320@garage.freebsd.pl> (Pawel Jakub Dawidek's message of "Sun, 13 Jun 2010 12:24:01 +0200") Message-ID: <86eigavzsg.fsf@kopusha.home.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.2 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: freebsd-fs@freebsd.org Subject: Re: FreeBSD 8.1 and HAST X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2010 07:28:21 -0000 On Sun, 13 Jun 2010 12:24:01 +0200 Pawel Jakub Dawidek wrote: >> Jun 13 16:25:37 sv01A hastd: [zfshast] (primary) Header contains no 'seq' field. PJD> This is the most important bit from the primary node. PJD> The header either does not contain 'seq' field or this field is 0. It PJD> can only be 0 if you have old kernel. With recent kernel geom_gate.ko PJD> was modified to start seq at 1, so this should not happen. I am a bit confused how this seq is supposed to work. For sync thread. I have set up hast on 8-STABLE (before I used it on 9-CURRENT only) and have the same issue as hiroshi@ does. I have added pjdlog_debug(2, "remote_send: seq is %llu.", (uint64_t)ggio->gctl_seq); after nv_add_uint64(nv, (uint64_t)ggio->gctl_seq, "seq"); in primary/remote_send thread and observe the following: Jun 14 00:12:54 zhuzha hastd: [storage] (primary) Unable to receive reply header: Resource temporarily unavailable. Jun 14 00:12:54 zhuzha kernel: Jun 14 00:12:54 zhuzha hastd: [storage] (primary) Unable to receive reply header: Resource temporarily unavailable. Jun 14 00:12:54 zhuzha hastd: [storage] (primary) Closing old incoming connection to tcp4://192.168.120.6. Jun 14 00:12:54 zhuzha hastd: [storage] (primary) Closing old outgoing connection to tcp4://192.168.120.6. Jun 14 00:12:54 zhuzha hastd: [storage] (primary) remote_recv: No requests, waiting. Jun 14 00:12:54 zhuzha hastd: [storage] (primary) remote_guard: Checking connections. Jun 14 00:12:54 zhuzha hastd: [storage] (primary) remote_guard: Reconnecting to tcp4://192.168.120.6. Jun 14 00:12:54 zhuzha hastd: [storage] (primary) Unable to write synchronization data: Invalid argument. Jun 14 00:12:54 zhuzha kernel: Jun 14 00:12:54 zhuzha hastd: [storage] (primary) Unable to write synchronization data: Invalid argument. Jun 14 00:12:54 zhuzha hastd: [storage] (primary) sync: (0x284f0080) Moving request to the free queue. Jun 14 00:12:54 zhuzha hastd: [storage] (primary) Connected to tcp4://192.168.120.6. Jun 14 00:12:54 zhuzha hastd: [storage] (primary) Successfully reconnected to tcp4://192.168.120.6. Jun 14 00:12:54 zhuzha hastd: [storage] (primary) Synchronization started. 10485760 bytes to go. Jun 14 00:12:54 zhuzha hastd: [storage] (primary) sync: Taking free request. Jun 14 00:12:54 zhuzha hastd: [storage] (primary) sync: (0x284f0040) Got free request. Jun 14 00:12:54 zhuzha hastd: [storage] (primary) sync: (0x284f0040) Sending sync request: READ(0, 131072). Jun 14 00:12:54 zhuzha hastd: [storage] (primary) sync: (0x284f0040) Moving request to the send queue. Jun 14 00:12:54 zhuzha hastd: [storage] (primary) local_send: (0x284f0040) Got request. Jun 14 00:12:54 zhuzha hastd: [storage] (primary) local_send: Taking request. Jun 14 00:12:54 zhuzha hastd: [storage] (primary) sync: (0x284f0040) Sending sync request: WRITE(0, 131072). Jun 14 00:12:54 zhuzha hastd: [storage] (primary) sync: (0x284f0040) Moving request to the send queue. Jun 14 00:12:54 zhuzha hastd: [storage] (primary) sync: (0x284f0040) Moving request to the send queues. Jun 14 00:12:54 zhuzha hastd: [storage] (primary) remote_send: (0x284f0040) Got request. Jun 14 00:12:54 zhuzha hastd: [storage] (primary) remote_send: seq is 0. Jun 14 00:12:54 zhuzha hastd: [storage] (primary) remote_send: (0x284f0040) Moving request to the recv queue. Jun 14 00:12:54 zhuzha hastd: [storage] (primary) remote_send: Taking request. Jun 14 00:12:54 zhuzha hastd: [storage] (primary) Header contains no 'seq' field. Jun 14 00:12:54 zhuzha kernel: Jun 14 00:12:54 zhuzha hastd: [storage] (primary) Header contains no 'seq' field. This lasts for some time (usually more then a half of a hour) and at some moment seq is set to 1 (but one time I saw when it was set to 19) and after this sychroniztion to secondary starts to work (seq is increasing by one for the next request). Jun 14 00:29:23 zhuzha hastd: [storage] (primary) Unable to receive reply header: Resource tempor arily unavailable. Jun 14 00:29:23 zhuzha kernel: Jun 14 00:29:23 zhuzha hastd: [storage] (primary) Unable to receiv e reply header: Resource temporarily unavailable. Jun 14 00:29:23 zhuzha hastd: [storage] (primary) Closing old incoming connection to tcp4://192.1 68.120.6. Jun 14 00:29:23 zhuzha hastd: [storage] (primary) Closing old outgoing connection to tcp4://192.1 68.120.6. Jun 14 00:29:23 zhuzha hastd: [storage] (primary) remote_recv: No requests, waiting. Jun 14 00:29:23 zhuzha hastd: [storage] (primary) remote_guard: Checking connections. Jun 14 00:29:23 zhuzha hastd: [storage] (primary) remote_guard: Reconnecting to tcp4://192.168.12 0.6. Jun 14 00:29:23 zhuzha hastd: [storage] (primary) Unable to write synchronization data: Invalid a rgument. Jun 14 00:29:23 zhuzha kernel: Jun 14 00:29:23 zhuzha hastd: [storage] (primary) Unable to write synchronization data: Invalid argument. Jun 14 00:29:23 zhuzha hastd: [storage] (primary) sync: (0x28441580) Moving request to the free q ueue. Jun 14 00:29:23 zhuzha hastd: [storage] (primary) Connected to tcp4://192.168.120.6. Jun 14 00:29:23 zhuzha hastd: [storage] (primary) Successfully reconnected to tcp4://192.168.120. 6. Jun 14 00:29:23 zhuzha hastd: [storage] (primary) Synchronization started. 10485760 bytes to go. Jun 14 00:29:23 zhuzha hastd: [storage] (primary) sync: Taking free request. Jun 14 00:29:23 zhuzha hastd: [storage] (primary) sync: (0x284f1640) Got free request. Jun 14 00:29:23 zhuzha hastd: [storage] (primary) sync: (0x284f1640) Sending sync request: READ(0 , 131072). Jun 14 00:29:23 zhuzha hastd: [storage] (primary) sync: (0x284f1640) Moving request to the send q ueue. Jun 14 00:29:23 zhuzha hastd: [storage] (primary) local_send: (0x284f1640) Got request. Jun 14 00:29:23 zhuzha hastd: [storage] (primary) local_send: Taking request. Jun 14 00:29:23 zhuzha hastd: [storage] (primary) sync: (0x284f1640) Sending sync request: WRITE( 0, 131072). Jun 14 00:29:23 zhuzha hastd: [storage] (primary) sync: (0x284f1640) Moving request to the send q ueue. Jun 14 00:29:23 zhuzha hastd: [storage] (primary) sync: (0x284f1640) Moving request to the send q ueues. Jun 14 00:29:23 zhuzha hastd: [storage] (primary) remote_send: (0x284f1640) Got request. Jun 14 00:29:23 zhuzha hastd: [storage] (primary) remote_send: seq is 1. Jun 14 00:29:23 zhuzha hastd: [storage] (primary) remote_send: (0x284f1640) Moving request to the recv queue. Jun 14 00:29:23 zhuzha hastd: [storage] (primary) remote_send: Taking request. Jun 14 00:29:24 zhuzha hastd: [storage] (primary) remote_recv: No requests, waiting. Jun 14 00:29:24 zhuzha hastd: [storage] (primary) sync: (0x284f1640) Moving request to the free queue. Jun 14 00:29:24 zhuzha hastd: [storage] (primary) sync: Taking free request. Jun 14 00:29:24 zhuzha hastd: [storage] (primary) sync: (0x284f15c0) Got free request. Jun 14 00:29:24 zhuzha hastd: [storage] (primary) sync: (0x284f15c0) Sending sync request: READ(131072, 131072). Jun 14 00:29:24 zhuzha hastd: [storage] (primary) sync: (0x284f15c0) Moving request to the send queue. Jun 14 00:29:24 zhuzha hastd: [storage] (primary) local_send: (0x284f15c0) Got request. Jun 14 00:29:24 zhuzha hastd: [storage] (primary) local_send: Taking request. Jun 14 00:29:24 zhuzha hastd: [storage] (primary) sync: (0x284f15c0) Sending sync request: WRITE(131072, 131072). Jun 14 00:29:24 zhuzha hastd: [storage] (primary) sync: (0x284f15c0) Moving request to the send queue. Jun 14 00:29:24 zhuzha hastd: [storage] (primary) sync: (0x284f15c0) Moving request to the send queues. Jun 14 00:29:24 zhuzha hastd: [storage] (primary) remote_send: (0x284f15c0) Got request. Jun 14 00:29:24 zhuzha hastd: [storage] (primary) remote_send: seq is 2. Jun 14 00:29:24 zhuzha hastd: [storage] (primary) remote_send: (0x284f15c0) Moving request to the recv queue. ... Adding seq logging on CURRENT I observe seq having the same value 2779096485 (0xa5a5a5a5): Jun 14 03:35:45 lolek hastd: [test] (primary) remote_send: (0x287f44c0) Got request. Jun 14 03:35:45 lolek hastd: [test] (primary) remote_send: seq is 2779096485. Jun 14 03:35:45 lolek hastd: [test] (primary) remote_send: (0x287f44c0) Moving request to the recv queue. So it looks like seq is just contents of unitialized memory and this works under CURRENT only because memory is initialized to 0xa5 by malloc here. This test program returns "*p is 0xa5a5a5a5" under my CURRENT and "*p is 0x0" under 8-STABLE. #include #include #include int main(int argc, char *argv[]) { unsigned long *p; p = malloc(sizeof(*p)); if (p == NULL) err(1,"malloc failed"); printf ("*p is 0x%lx\n", *p); return 0; } -- Mikolaj Golub From owner-freebsd-fs@FreeBSD.ORG Mon Jun 14 08:01:10 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 07F6D106566C; Mon, 14 Jun 2010 08:01:10 +0000 (UTC) (envelope-from hiroshi@soupacific.com) Received: from mail.soupacific.com (mail.soupacific.com [211.19.53.201]) by mx1.freebsd.org (Postfix) with ESMTP id C66BA8FC1A; Mon, 14 Jun 2010 08:01:09 +0000 (UTC) Received: from [127.0.0.1] (unknown [192.168.1.239]) by mail.soupacific.com (Postfix) with ESMTP id BCC386BF70; Mon, 14 Jun 2010 07:52:34 +0000 (UTC) Message-ID: <4C15E1C3.9030907@soupacific.com> Date: Mon, 14 Jun 2010 17:01:07 +0900 From: "hiroshi@soupacific.com" User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4 MIME-Version: 1.0 To: Mikolaj Golub References: <4C10B526.4040908@soupacific.com><20100612104336.GA2253@garage.freebsd.pl><4C1372E0.1000903@soupacific.com><20100612142311.GF2253@garage.freebsd.pl><4C139F9C.2090305@soupacific.com> <86iq5oc82y.fsf@kopusha.home.net><4C14215D.9090304@soupacific.com><20100613003635.GA60012@icarus.home.lan><20100613074921.GB1320@garage.freebsd.pl><4C149A5C.3070401@soupacific.com><20100613102401.GE1320@garage.freebsd.pl> <86eigavzsg.fsf@kopusha.home.net> In-Reply-To: <86eigavzsg.fsf@kopusha.home.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, Pawel Jakub Dawidek Subject: Re: FreeBSD 8.1 and HAST X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2010 08:01:10 -0000 > So it looks like seq is just contents of unitialized memory and this works > under CURRENT only because memory is initialized to 0xa5 by malloc here. > > This test program returns "*p is 0xa5a5a5a5" under my CURRENT and "*p is 0x0" > under 8-STABLE. Great ! Yes it returns exact same as you ! my 8.1 0x0 and 9.0 0xa5a5a5a5 ! Ooops ! What shall I do for this !!! Thanks Hiroshi From owner-freebsd-fs@FreeBSD.ORG Mon Jun 14 09:51:01 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EFEE0106566C for ; Mon, 14 Jun 2010 09:51:01 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello089077043238.chello.pl [89.77.43.238]) by mx1.freebsd.org (Postfix) with ESMTP id 344578FC17 for ; Mon, 14 Jun 2010 09:50:59 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 7A04645CD9; Mon, 14 Jun 2010 11:50:57 +0200 (CEST) Received: from localhost (pdawidek.wheel.pl [10.0.1.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 37FC245C9F; Mon, 14 Jun 2010 11:50:52 +0200 (CEST) Date: Mon, 14 Jun 2010 11:50:44 +0200 From: Pawel Jakub Dawidek To: Mikolaj Golub Message-ID: <20100614095044.GH1721@garage.freebsd.pl> References: <4C1372E0.1000903@soupacific.com> <20100612142311.GF2253@garage.freebsd.pl> <4C139F9C.2090305@soupacific.com> <86iq5oc82y.fsf@kopusha.home.net> <4C14215D.9090304@soupacific.com> <20100613003635.GA60012@icarus.home.lan> <20100613074921.GB1320@garage.freebsd.pl> <4C149A5C.3070401@soupacific.com> <20100613102401.GE1320@garage.freebsd.pl> <86eigavzsg.fsf@kopusha.home.net> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="82evfD9Ogz2JrdWZ" Content-Disposition: inline In-Reply-To: <86eigavzsg.fsf@kopusha.home.net> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT amd64 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=4.5 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: FreeBSD 8.1 and HAST X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2010 09:51:02 -0000 --82evfD9Ogz2JrdWZ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Jun 14, 2010 at 10:28:15AM +0300, Mikolaj Golub wrote: >=20 > On Sun, 13 Jun 2010 12:24:01 +0200 Pawel Jakub Dawidek wrote: >=20 > >> Jun 13 16:25:37 sv01A hastd: [zfshast] (primary) Header contains no '= seq' field. >=20 > PJD> This is the most important bit from the primary node. >=20 > PJD> The header either does not contain 'seq' field or this field is 0. = It > PJD> can only be 0 if you have old kernel. With recent kernel geom_gate.= ko > PJD> was modified to start seq at 1, so this should not happen. >=20 > I am a bit confused how this seq is supposed to work. For sync thread. I = have > set up hast on 8-STABLE (before I used it on 9-CURRENT only) and have the= same > issue as hiroshi@ does. I have added=20 >=20 > pjdlog_debug(2, "remote_send: seq is %llu.", (uint64_t)ggio->gctl_seq); >=20 > after=20 >=20 > nv_add_uint64(nv, (uint64_t)ggio->gctl_seq, "seq"); >=20 > in primary/remote_send thread and observe the following: [...] Could you find where exactly it looses proper value? I found that in ggate_recv_thread() after ioctl(2), gctl_seq has expected value, but I'm not setup to test it further quickly. --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --82evfD9Ogz2JrdWZ Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkwV+3QACgkQForvXbEpPzQhxQCfaHqWrpagqptbkh6xcfZoNKHr UwoAoJQfuEetO+eKuiGtmRBt2x46ubg1 =VmKM -----END PGP SIGNATURE----- --82evfD9Ogz2JrdWZ-- From owner-freebsd-fs@FreeBSD.ORG Mon Jun 14 11:06:50 2010 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 649951065673 for ; Mon, 14 Jun 2010 11:06:50 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 51E1C8FC1F for ; Mon, 14 Jun 2010 11:06:50 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o5EB6oAx078500 for ; Mon, 14 Jun 2010 11:06:50 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o5EB6nle078498 for freebsd-fs@FreeBSD.org; Mon, 14 Jun 2010 11:06:49 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 14 Jun 2010 11:06:49 GMT Message-Id: <201006141106.o5EB6nle078498@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2010 11:06:50 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/147420 fs [ufs] [panic] ufs_dirbad, nullfs, jail panic (corrupt o kern/147292 fs [nfs] [patch] readahead missing in nfs client options o kern/146708 fs [ufs] [panic] Kernel panic in softdep_disk_write_compl o kern/146528 fs [zfs] Severe memory leak in ZFS on i386 o kern/146502 fs [nfs] FreeBSD 8 NFS Client Connection to Server o kern/146375 fs [nfs] [patch] Typos in macro variables names in sys/fs o kern/145778 fs [zfs] [panic] panic in zfs_fuid_map_id (known issue fi s kern/145712 fs [zfs] cannot offline two drives in a raidz2 configurat s kern/145424 fs [zfs] [patch] move source closer to v15 o kern/145411 fs [xfs] [panic] Kernel panics shortly after mounting an o kern/145309 fs [disklabel]: Editing disk label invalidates the whole o kern/145272 fs [zfs] [panic] Panic during boot when accessing zfs on o kern/145246 fs [ufs] dirhash in 7.3 gratuitously frees hashes when it o kern/145238 fs [zfs] [panic] kernel panic on zpool clear tank o kern/145229 fs [zfs] Vast differences in ZFS ARC behavior between 8.0 o kern/145189 fs [nfs] nfsd performs abysmally under load o kern/144929 fs [ufs] [lor] vfs_bio.c + ufs_dirhash.c o kern/144458 fs [nfs] [patch] nfsd fails as a kld p kern/144447 fs [zfs] sharenfs fsunshare() & fsshare_main() non functi o kern/144416 fs [panic] Kernel panic on online filesystem optimization s kern/144415 fs [zfs] [panic] kernel panics on boot after zfs crash o kern/144234 fs [zfs] Cannot boot machine with recent gptzfsboot code o kern/143825 fs [nfs] [panic] Kernel panic on NFS client o kern/143345 fs [ext2fs] [patch] extfs minor header cleanups to better o kern/143212 fs [nfs] NFSv4 client strange work ... o kern/143184 fs [zfs] [lor] zfs/bufwait LOR o kern/142924 fs [ext2fs] [patch] Small cleanup for the inode struct in o kern/142914 fs [zfs] ZFS performance degradation over time o kern/142878 fs [zfs] [vfs] lock order reversal o kern/142597 fs [ext2fs] ext2fs does not work on filesystems with real o kern/142489 fs [zfs] [lor] allproc/zfs LOR o kern/142466 fs Update 7.2 -> 8.0 on Raid 1 ends with screwed raid [re o kern/142401 fs [ntfs] [patch] Minor updates to NTFS from NetBSD o kern/142306 fs [zfs] [panic] ZFS drive (from OSX Leopard) causes two o kern/142068 fs [ufs] BSD labels are got deleted spontaneously o kern/141897 fs [msdosfs] [panic] Kernel panic. msdofs: file name leng o kern/141463 fs [nfs] [panic] Frequent kernel panics after upgrade fro o kern/141305 fs [zfs] FreeBSD ZFS+sendfile severe performance issues ( o kern/141091 fs [patch] [nullfs] fix panics with DIAGNOSTIC enabled o kern/141086 fs [nfs] [panic] panic("nfs: bioread, not dir") on FreeBS o kern/141010 fs [zfs] "zfs scrub" fails when backed by files in UFS2 o kern/140888 fs [zfs] boot fail from zfs root while the pool resilveri o kern/140661 fs [zfs] [patch] /boot/loader fails to work on a GPT/ZFS- o kern/140640 fs [zfs] snapshot crash o kern/140134 fs [msdosfs] write and fsck destroy filesystem integrity o kern/140068 fs [smbfs] [patch] smbfs does not allow semicolon in file o kern/139725 fs [zfs] zdb(1) dumps core on i386 when examining zpool c o kern/139715 fs [zfs] vfs.numvnodes leak on busy zfs o bin/139651 fs [nfs] mount(8): read-only remount of NFS volume does n o kern/139597 fs [patch] [tmpfs] tmpfs initializes va_gen but doesn't u o kern/139564 fs [zfs] [panic] 8.0-RC1 - Fatal trap 12 at end of shutdo o kern/139407 fs [smbfs] [panic] smb mount causes system crash if remot o kern/139363 fs [nfs] diskless root nfs mount from non FreeBSD server o kern/138790 fs [zfs] ZFS ceases caching when mem demand is high o kern/138421 fs [ufs] [patch] remove UFS label limitations o kern/138202 fs mount_msdosfs(1) see only 2Gb f kern/137037 fs [zfs] [hang] zfs rollback on root causes FreeBSD to fr o kern/136968 fs [ufs] [lor] ufs/bufwait/ufs (open) o kern/136945 fs [ufs] [lor] filedesc structure/ufs (poll) o kern/136944 fs [ffs] [lor] bufwait/snaplk (fsync) o kern/136873 fs [ntfs] Missing directories/files on NTFS volume o kern/136865 fs [nfs] [patch] NFS exports atomic and on-the-fly atomic o kern/136470 fs [nfs] Cannot mount / in read-only, over NFS o kern/135546 fs [zfs] zfs.ko module doesn't ignore zpool.cache filenam o kern/135469 fs [ufs] [panic] kernel crash on md operation in ufs_dirb o kern/135050 fs [zfs] ZFS clears/hides disk errors on reboot o kern/134491 fs [zfs] Hot spares are rather cold... o kern/133676 fs [smbfs] [panic] umount -f'ing a vnode-based memory dis o kern/133614 fs [panic] panic: ffs_truncate: read-only filesystem o kern/133174 fs [msdosfs] [patch] msdosfs must support utf-encoded int f kern/133150 fs [zfs] Page fault with ZFS on 7.1-RELEASE/amd64 while w o kern/132960 fs [ufs] [panic] panic:ffs_blkfree: freeing free frag o kern/132397 fs reboot causes filesystem corruption (failure to sync b o kern/132331 fs [ufs] [lor] LOR ufs and syncer o kern/132237 fs [msdosfs] msdosfs has problems to read MSDOS Floppy o kern/132145 fs [panic] File System Hard Crashes o kern/131441 fs [unionfs] [nullfs] unionfs and/or nullfs not combineab o kern/131360 fs [nfs] poor scaling behavior of the NFS server under lo o kern/131342 fs [nfs] mounting/unmounting of disks causes NFS to fail o bin/131341 fs makefs: error "Bad file descriptor" on the mount poin o kern/130920 fs [msdosfs] cp(1) takes 100% CPU time while copying file o kern/130229 fs [iconv] usermount fails on fs that need iconv o kern/130210 fs [nullfs] Error by check nullfs o kern/129760 fs [nfs] after 'umount -f' of a stale NFS share FreeBSD l o kern/129488 fs [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: o kern/129231 fs [ufs] [patch] New UFS mount (norandom) option - mostly o kern/129152 fs [panic] non-userfriendly panic when trying to mount(8) o kern/129059 fs [zfs] [patch] ZFS bootloader whitelistable via WITHOUT f kern/128829 fs smbd(8) causes periodic panic on 7-RELEASE o kern/127420 fs [gjournal] [panic] Journal overflow on gmirrored gjour o kern/127029 fs [panic] mount(8): trying to mount a write protected zi o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file o kern/125895 fs [ffs] [panic] kernel: panic: ffs_blkfree: freeing free s kern/125738 fs [zfs] [request] SHA256 acceleration in ZFS p kern/124621 fs [ext3] [patch] Cannot mount ext2fs partition f bin/124424 fs [zfs] zfs(8): zfs list -r shows strange snapshots' siz o kern/123939 fs [msdosfs] corrupts new files o kern/122380 fs [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o bin/121898 fs [nullfs] pwd(1)/getcwd(2) fails with Permission denied o bin/121779 fs [ufs] snapinfo(8) (and related tools?) only work for t o bin/121366 fs [zfs] [patch] Automatic disk scrubbing from periodic(8 o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha f kern/120991 fs [panic] [fs] [snapshot] System crashes when manipulati o kern/120483 fs [ntfs] [patch] NTFS filesystem locking changes o kern/120482 fs [ntfs] [patch] Sync style changes between NetBSD and F f kern/119735 fs [zfs] geli + ZFS + samba starting on boot panics 7.0-B o kern/118912 fs [2tb] disk sizing/geometry problem with large array o kern/118713 fs [minidump] [patch] Display media size required for a k o bin/118249 fs mv(1): moving a directory changes its mtime o kern/118107 fs [ntfs] [panic] Kernel panic when accessing a file at N o bin/117315 fs [smbfs] mount_smbfs(8) and related options can't mount o kern/117314 fs [ntfs] Long-filename only NTFS fs'es cause kernel pani o kern/117158 fs [zfs] zpool scrub causes panic if geli vdevs detach on o bin/116980 fs [msdosfs] [patch] mount_msdosfs(8) resets some flags f o conf/116931 fs lack of fsck_cd9660 prevents mounting iso images with o kern/116913 fs [ffs] [panic] ffs_blkfree: freeing free block p kern/116608 fs [msdosfs] [patch] msdosfs fails to check mount options o kern/116583 fs [ffs] [hang] System freezes for short time when using o kern/116170 fs [panic] Kernel panic when mounting /tmp o kern/115645 fs [snapshots] [panic] lockmgr: thread 0xc4c00d80, not ex o bin/115361 fs [zfs] mount(8) gets into a state where it won't set/un o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/113852 fs [smbfs] smbfs does not properly implement DFS referral o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/111843 fs [msdosfs] Long Names of files are incorrectly created o kern/111782 fs [ufs] dump(8) fails horribly for large filesystems s bin/111146 fs [2tb] fsck(8) fails on 6T filesystem o kern/109024 fs [msdosfs] mount_msdosfs: msdosfs_iconv: Operation not o kern/109010 fs [msdosfs] can't mv directory within fat32 file system o bin/107829 fs [2TB] fdisk(8): invalid boundary checking in fdisk / w o kern/106107 fs [ufs] left-over fsck_snapshot after unfinished backgro o kern/106030 fs [ufs] [panic] panic in ufs from geom when a dead disk o kern/104406 fs [ufs] Processes get stuck in "ufs" state under persist o kern/104133 fs [ext2fs] EXT2FS module corrupts EXT2/3 filesystems o kern/103035 fs [ntfs] Directories in NTFS mounted disc images appear o kern/101324 fs [smbfs] smbfs sometimes not case sensitive when it's s o kern/99290 fs [ntfs] mount_ntfs ignorant of cluster sizes o kern/97377 fs [ntfs] [patch] syntax cleanup for ntfs_ihash.c o kern/95222 fs [iso9660] File sections on ISO9660 level 3 CDs ignored o kern/94849 fs [ufs] rename on UFS filesystem is not atomic o kern/94769 fs [ufs] Multiple file deletions on multi-snapshotted fil o kern/94733 fs [smbfs] smbfs may cause double unlock o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/92272 fs [ffs] [hang] Filling a filesystem while creating a sna f kern/91568 fs [ufs] [panic] writing to UFS/softupdates DVD media in o kern/91134 fs [smbfs] [patch] Preserve access and modification time a kern/90815 fs [smbfs] [patch] SMBFS with character conversions somet o kern/88657 fs [smbfs] windows client hang when browsing a samba shar o kern/88266 fs [smbfs] smbfs does not implement UIO_NOCOPY and sendfi o kern/87859 fs [smbfs] System reboot while umount smbfs. o kern/86587 fs [msdosfs] rm -r /PATH fails with lots of small files o kern/85326 fs [smbfs] [panic] saving a file via samba to an overquot o kern/84589 fs [2TB] 5.4-STABLE unresponsive during background fsck 2 o kern/80088 fs [smbfs] Incorrect file time setting on NTFS mounted vi o kern/73484 fs [ntfs] Kernel panic when doing `ls` from the client si o bin/73019 fs [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino o kern/71774 fs [ntfs] NTFS cannot "see" files on a WinXP filesystem o kern/68978 fs [panic] [ufs] crashes with failing hard disk, loose po o kern/65920 fs [nwfs] Mounted Netware filesystem behaves strange o kern/65901 fs [smbfs] [patch] smbfs fails fsx write/truncate-down/tr o kern/61503 fs [smbfs] mount_smbfs does not work as non-root o kern/55617 fs [smbfs] Accessing an nsmb-mounted drive via a smb expo o kern/53137 fs [ffs] [panic] background fscking causing ffs_valloc pa o kern/51685 fs [hang] Unbounded inode allocation causes kernel to loc o kern/51583 fs [nullfs] [patch] allow to work with devices and socket o kern/36566 fs [smbfs] System reboot with dead smb mount and umount o kern/33464 fs [ufs] soft update inconsistencies after system crash o kern/18874 fs [2TB] 32bit NFS servers export wrong negative values t 174 problems total. From owner-freebsd-fs@FreeBSD.ORG Mon Jun 14 13:16:46 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 38ABE1065675; Mon, 14 Jun 2010 13:16:46 +0000 (UTC) (envelope-from hiroshi@soupacific.com) Received: from mail.soupacific.com (mail.soupacific.com [211.19.53.201]) by mx1.freebsd.org (Postfix) with ESMTP id 070818FC17; Mon, 14 Jun 2010 13:16:45 +0000 (UTC) Received: from [127.0.0.1] (unknown [192.168.1.239]) by mail.soupacific.com (Postfix) with ESMTP id 444316B127; Mon, 14 Jun 2010 13:08:12 +0000 (UTC) Message-ID: <4C162BBB.4010806@soupacific.com> Date: Mon, 14 Jun 2010 22:16:43 +0900 From: "hiroshi@soupacific.com" User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4 MIME-Version: 1.0 To: Pawel Jakub Dawidek References: <4C1372E0.1000903@soupacific.com> <20100612142311.GF2253@garage.freebsd.pl> <4C139F9C.2090305@soupacific.com> <86iq5oc82y.fsf@kopusha.home.net> <4C14215D.9090304@soupacific.com> <20100613003635.GA60012@icarus.home.lan> <20100613074921.GB1320@garage.freebsd.pl> <4C149A5C.3070401@soupacific.com> <20100613102401.GE1320@garage.freebsd.pl> <86eigavzsg.fsf@kopusha.home.net> <20100614095044.GH1721@garage.freebsd.pl> In-Reply-To: <20100614095044.GH1721@garage.freebsd.pl> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: FreeBSD 8.1 and HAST X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2010 13:16:46 -0000 I put pjdlog_debug as below; remote_send_thread(void *arg) { ...... for (;;) { pjdlog_debug(2, "remote_send: Taking request."); QUEUE_TAKE1(hio, send, ncomp); pjdlog_debug(2, "remote_send: (%p) Got request.", hio); ggio = &hio->hio_ggio; pjdlog_debug(2, "remote_send 1: seq is %lu", (uint64_t)ggio->gctl_seq); It's always return 0 ! Where is &hio->hio_ggio setting ? geom side ? Please let me know what shall I do ? Thanks Hiroshi From owner-freebsd-fs@FreeBSD.ORG Mon Jun 14 15:07:58 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BE56F1065672; Mon, 14 Jun 2010 15:07:58 +0000 (UTC) (envelope-from to.my.trociny@gmail.com) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id C6D558FC15; Mon, 14 Jun 2010 15:07:57 +0000 (UTC) Received: by bwz2 with SMTP id 2so2861266bwz.13 for ; Mon, 14 Jun 2010 08:07:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:to:cc:subject:references :x-comment-to:date:in-reply-to:message-id:user-agent:mime-version :content-type; bh=/NVGcIRNnO4e91QtM3WCWjkhgKG5ZtiE6x4QIPwb+v0=; b=NSLuFBoffTCx+B852xSwLAOfAiH8OSayCRe0FWcui6QsWR4CS4pd/vSstswAuy9GgQ 9FdCBngDV8GhvDR50LnTh/xNPRotWcq/Gf3Xu9uud+bRLHxlLqZcsE36K8sELleEUG+U Zw7qbpkPfkBeaUjyH+uEpAxvteHajCN/fLzzU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:cc:subject:references:x-comment-to:date:in-reply-to :message-id:user-agent:mime-version:content-type; b=QPUmoYUlbndQrIghfPALcoRdZWobcdstpITVOCPhGUy1pSaxcYfSVNHNfrktBboRXh fgw4PXuEmsdd4EMG+k+mTS6Mu9pS6mOEMOSePF1620GoihlbaET+7BweqNDxUqRbDM0Z +81LSvcqH534MeP/gDbFYAlCHFXWWvqFJVAtc= Received: by 10.204.34.130 with SMTP id l2mr4367813bkd.164.1276528076720; Mon, 14 Jun 2010 08:07:56 -0700 (PDT) Received: from localhost ([95.69.160.52]) by mx.google.com with ESMTPS id v14sm19887393bkz.8.2010.06.14.08.07.54 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 14 Jun 2010 08:07:55 -0700 (PDT) From: Mikolaj Golub To: Pawel Jakub Dawidek References: <4C1372E0.1000903@soupacific.com> <20100612142311.GF2253@garage.freebsd.pl> <4C139F9C.2090305@soupacific.com> <86iq5oc82y.fsf@kopusha.home.net> <4C14215D.9090304@soupacific.com> <20100613003635.GA60012@icarus.home.lan> <20100613074921.GB1320@garage.freebsd.pl> <4C149A5C.3070401@soupacific.com> <20100613102401.GE1320@garage.freebsd.pl> <86eigavzsg.fsf@kopusha.home.net> <20100614095044.GH1721@garage.freebsd.pl> X-Comment-To: Pawel Jakub Dawidek Date: Mon, 14 Jun 2010 18:07:51 +0300 In-Reply-To: <20100614095044.GH1721@garage.freebsd.pl> (Pawel Jakub Dawidek's message of "Mon, 14 Jun 2010 11:50:44 +0200") Message-ID: <868w6hwt2w.fsf@kopusha.home.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.2 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: freebsd-fs@freebsd.org Subject: Re: FreeBSD 8.1 and HAST X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2010 15:07:58 -0000 On Mon, 14 Jun 2010 11:50:44 +0200 Pawel Jakub Dawidek wrote: PJD> On Mon, Jun 14, 2010 at 10:28:15AM +0300, Mikolaj Golub wrote: >> >> On Sun, 13 Jun 2010 12:24:01 +0200 Pawel Jakub Dawidek wrote: >> >> >> Jun 13 16:25:37 sv01A hastd: [zfshast] (primary) Header contains no 'seq' field. >> >> PJD> This is the most important bit from the primary node. >> >> PJD> The header either does not contain 'seq' field or this field is 0. It >> PJD> can only be 0 if you have old kernel. With recent kernel geom_gate.ko >> PJD> was modified to start seq at 1, so this should not happen. >> >> I am a bit confused how this seq is supposed to work. For sync thread. I have >> set up hast on 8-STABLE (before I used it on 9-CURRENT only) and have the same >> issue as hiroshi@ does. I have added >> >> pjdlog_debug(2, "remote_send: seq is %llu.", (uint64_t)ggio->gctl_seq); >> >> after >> >> nv_add_uint64(nv, (uint64_t)ggio->gctl_seq, "seq"); >> >> in primary/remote_send thread and observe the following: PJD> [...] PJD> Could you find where exactly it looses proper value? PJD> I found that in ggate_recv_thread() after ioctl(2), gctl_seq has PJD> expected value, but I'm not setup to test it further quickly. I suppose ggate_recv_thread() is ok but as I wrote earlier I am concerned about sync thread. I have added additional prints and how it looks: Jun 14 17:47:50 zhuzha hastd: [storage] (primary) Device hast/storage recovered. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) init_environment: hio: Ox28441580; seq: 0. ... Jun 14 17:47:50 zhuzha hastd: [storage] (primary) init_environment: hio: Ox284f2600; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) init_environment: hio: Ox284f2640; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) ggate_recv: Taking free request. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) ggate_recv: (0x284f2640) Got free request. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) ggate_recv 1: hio: Ox284f2640; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) ggate_recv: (0x284f2640) Waiting for request from the kernel. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) ggate_recv 2: hio: Ox284f2640; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) remote_send: Taking request. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) local_send: Taking request. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) ggate_send: Taking request. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) remote_recv: No requests, waiting. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) remote_guard: Checking connections. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) Synchronization started. 10485760 bytes to go. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) remote_guard: Connection to tcp4://192.168.120.6 is ok. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync: Taking free request. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync: (0x284f2600) Got free request. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync 1: hio: Ox284f2600; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync 2: hio: Ox284f2600; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync 3: hio: Ox284f2600; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync: (0x284f2600) Sending sync request: READ(0, 131072). Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync: (0x284f2600) Moving request to the send queue. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync 4: hio: Ox284f2600; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync 5: hio: Ox284f2600; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) local_send: (0x284f2600) Got request. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) local_send 1: hio: Ox284f2600; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) local_send: Taking request. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync 6: hio: Ox284f2600; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync 7: hio: Ox284f2600; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync: (0x284f2600) Sending sync request: WRITE(0, 131072). Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync: (0x284f2600) Moving request to the send queue. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync 8: hio: Ox284f2600; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync: (0x284f2600) Moving request to the send queues. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) sync 9: hio: Ox284f2600; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) remote_send: (0x284f2600) Got request. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) remote_send 1: hio: Ox284f2600; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) remote_send 3: hio: Ox284f2600; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) remote_send: seq is 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) remote_send: (0x284f2600) Moving request to the recv queue. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) remote_send 7: hio: Ox284f2600; seq: 0. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) remote_send: Taking request. Jun 14 17:47:50 zhuzha hastd: [storage] (primary) Header contains no 'seq' field. Jun 14 17:47:50 zhuzha kernel: Jun 14 17:47:50 zhuzha hastd: [storage] (primary) Header contains no 'seq' field. So ggate_recv takes free just initilized hio (0x284f2640) from the free queue and is waiting for the data from the kernel. At this time sync thread starts syncronization, takes another just initilized hio (0x284f2600) from the free queue and puts it to remote_send, so the request is sent with seq == 0. -- Mikolaj Golub From owner-freebsd-fs@FreeBSD.ORG Mon Jun 14 15:38:03 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BEFE51065687 for ; Mon, 14 Jun 2010 15:38:03 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello089077043238.chello.pl [89.77.43.238]) by mx1.freebsd.org (Postfix) with ESMTP id 01F898FC13 for ; Mon, 14 Jun 2010 15:38:02 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 3F97045E9D; Mon, 14 Jun 2010 17:38:00 +0200 (CEST) Received: from localhost (pdawidek.wheel.pl [10.0.1.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 5468E45E94; Mon, 14 Jun 2010 17:37:54 +0200 (CEST) Date: Mon, 14 Jun 2010 17:37:46 +0200 From: Pawel Jakub Dawidek To: Mikolaj Golub Message-ID: <20100614153746.GN1721@garage.freebsd.pl> References: <4C139F9C.2090305@soupacific.com> <86iq5oc82y.fsf@kopusha.home.net> <4C14215D.9090304@soupacific.com> <20100613003635.GA60012@icarus.home.lan> <20100613074921.GB1320@garage.freebsd.pl> <4C149A5C.3070401@soupacific.com> <20100613102401.GE1320@garage.freebsd.pl> <86eigavzsg.fsf@kopusha.home.net> <20100614095044.GH1721@garage.freebsd.pl> <868w6hwt2w.fsf@kopusha.home.net> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="kK1uqZGE6pgsGNyR" Content-Disposition: inline In-Reply-To: <868w6hwt2w.fsf@kopusha.home.net> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT amd64 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=4.5 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: FreeBSD 8.1 and HAST X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2010 15:38:03 -0000 --kK1uqZGE6pgsGNyR Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Jun 14, 2010 at 06:07:51PM +0300, Mikolaj Golub wrote: > So ggate_recv takes free just initilized hio (0x284f2640) from the free q= ueue > and is waiting for the data from the kernel. At this time sync thread sta= rts > syncronization, takes another just initilized hio (0x284f2600) from the f= ree > queue and puts it to remote_send, so the request is sent with seq =3D=3D = 0. Right, sync code was broken. For sync requests gctl_seq is irrelevant, so could you guys test this patch: http://people.freebsd.org/~pjd/patches/hastd_primary.c.2.patch --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --kK1uqZGE6pgsGNyR Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkwWTMoACgkQForvXbEpPzTX6gCfU54R5Yj4q4dDgNm40pC5J+tT xeoAoOh2oxb1afAjHbW+diL+ddq5LNjO =bMtb -----END PGP SIGNATURE----- --kK1uqZGE6pgsGNyR-- From owner-freebsd-fs@FreeBSD.ORG Mon Jun 14 15:59:16 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 94EE6106566C; Mon, 14 Jun 2010 15:59:16 +0000 (UTC) (envelope-from to.my.trociny@gmail.com) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id ACFEB8FC08; Mon, 14 Jun 2010 15:59:15 +0000 (UTC) Received: by bwz2 with SMTP id 2so2900784bwz.13 for ; Mon, 14 Jun 2010 08:59:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:to:cc:subject:references :x-comment-to:date:in-reply-to:message-id:user-agent:mime-version :content-type; bh=CoYklJlOntJ0OslxTLC7dO8orzgOXjY99JlUyOHYlhs=; b=KgOAbRWtysCEh6vlUG97hkwvhEUyZYEjRYIX00YKL2azKje7y8zAXAaqd2BXk51tIA YUFEkshEuUztMgHwCcewOfDBd9czGmk3ZFP1jgYVYzeCj2DDySyABlBEDmvdERmtRqCy diyPYk2NbqiD6It86+sUrg0t4JqlTs7VyxOO4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:cc:subject:references:x-comment-to:date:in-reply-to :message-id:user-agent:mime-version:content-type; b=BvKRASAth2qQeuZihBvjU1LTXKxJZjC7SJEDeRC/qWgwYyHNtd1bVLcS/bQoWE6JWV 5/7VYytGphMyWhLnZqb9uJqlQ2d1R3M/97MKNlQnZikZcQ1IxXpUC99o8lCN8PsmLyK7 cOKcs/whwCNABc/OM4P9oPJ8BuO58oWnpool8= Received: by 10.204.79.222 with SMTP id q30mr4426542bkk.93.1276531154584; Mon, 14 Jun 2010 08:59:14 -0700 (PDT) Received: from localhost ([95.69.160.52]) by mx.google.com with ESMTPS id f13sm20067684bka.5.2010.06.14.08.59.13 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 14 Jun 2010 08:59:13 -0700 (PDT) From: Mikolaj Golub To: Pawel Jakub Dawidek References: <4C139F9C.2090305@soupacific.com> <86iq5oc82y.fsf@kopusha.home.net> <4C14215D.9090304@soupacific.com> <20100613003635.GA60012@icarus.home.lan> <20100613074921.GB1320@garage.freebsd.pl> <4C149A5C.3070401@soupacific.com> <20100613102401.GE1320@garage.freebsd.pl> <86eigavzsg.fsf@kopusha.home.net> <20100614095044.GH1721@garage.freebsd.pl> <868w6hwt2w.fsf@kopusha.home.net> <20100614153746.GN1721@garage.freebsd.pl> X-Comment-To: Pawel Jakub Dawidek Date: Mon, 14 Jun 2010 18:59:12 +0300 In-Reply-To: <20100614153746.GN1721@garage.freebsd.pl> (Pawel Jakub Dawidek's message of "Mon, 14 Jun 2010 17:37:46 +0200") Message-ID: <86zkyxvc4v.fsf@kopusha.home.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.2 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: freebsd-fs@freebsd.org Subject: Re: FreeBSD 8.1 and HAST X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2010 15:59:16 -0000 On Mon, 14 Jun 2010 17:37:46 +0200 Pawel Jakub Dawidek wrote: PJD> Right, sync code was broken. For sync requests gctl_seq is irrelevant, PJD> so could you guys test this patch: PJD> http://people.freebsd.org/~pjd/patches/hastd_primary.c.2.patch It works for me. No issues with synchronization on 8-STABLE now. Also have been running it on CURRENT. -- Mikolaj Golub From owner-freebsd-fs@FreeBSD.ORG Mon Jun 14 16:27:32 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8000E1065670; Mon, 14 Jun 2010 16:27:32 +0000 (UTC) (envelope-from hiroshi@soupacific.com) Received: from mail.soupacific.com (mail.soupacific.com [211.19.53.201]) by mx1.freebsd.org (Postfix) with ESMTP id 4EA388FC0C; Mon, 14 Jun 2010 16:27:32 +0000 (UTC) Received: from [127.0.0.1] (unknown [192.168.1.239]) by mail.soupacific.com (Postfix) with ESMTP id 4434B6B296; Mon, 14 Jun 2010 16:18:58 +0000 (UTC) Message-ID: <4C165871.6060609@soupacific.com> Date: Tue, 15 Jun 2010 01:27:29 +0900 From: "hiroshi@soupacific.com" User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4 MIME-Version: 1.0 To: Pawel Jakub Dawidek References: <4C139F9C.2090305@soupacific.com> <86iq5oc82y.fsf@kopusha.home.net> <4C14215D.9090304@soupacific.com> <20100613003635.GA60012@icarus.home.lan> <20100613074921.GB1320@garage.freebsd.pl> <4C149A5C.3070401@soupacific.com> <20100613102401.GE1320@garage.freebsd.pl> <86eigavzsg.fsf@kopusha.home.net> <20100614095044.GH1721@garage.freebsd.pl> <868w6hwt2w.fsf@kopusha.home.net> <20100614153746.GN1721@garage.freebsd.pl> In-Reply-To: <20100614153746.GN1721@garage.freebsd.pl> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: FreeBSD 8.1 and HAST X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2010 16:27:32 -0000 Good Morning ! Thanks pjd@ ! > http://people.freebsd.org/~pjd/patches/hastd_primary.c.2.patch > It works fine now! I sincry appreciate your great works and fastest responce! I believe that HAST will change the world ! Thanks a lot. Hiroshi Katayama From owner-freebsd-fs@FreeBSD.ORG Mon Jun 14 21:47:21 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 44B32106564A for ; Mon, 14 Jun 2010 21:47:21 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello089077043238.chello.pl [89.77.43.238]) by mx1.freebsd.org (Postfix) with ESMTP id 7DE528FC0A for ; Mon, 14 Jun 2010 21:47:20 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 0B8E145E91; Mon, 14 Jun 2010 23:47:17 +0200 (CEST) Received: from localhost (chello089077043238.chello.pl [89.77.43.238]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 8367245D8D; Mon, 14 Jun 2010 23:47:11 +0200 (CEST) Date: Mon, 14 Jun 2010 23:46:50 +0200 From: Pawel Jakub Dawidek To: Mikolaj Golub Message-ID: <20100614214650.GA2498@garage.freebsd.pl> References: <20100612104336.GA2253@garage.freebsd.pl> <4C1372E0.1000903@soupacific.com> <20100612142311.GF2253@garage.freebsd.pl> <4C139F9C.2090305@soupacific.com> <86iq5oc82y.fsf@kopusha.home.net> <4C14215D.9090304@soupacific.com> <20100613003635.GA60012@icarus.home.lan> <20100613074921.GB1320@garage.freebsd.pl> <4C149A5C.3070401@soupacific.com> <86ljajw68w.fsf@kopusha.home.net> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="oyUTqETQ0mS9luUI" Content-Disposition: inline In-Reply-To: <86ljajw68w.fsf@kopusha.home.net> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT amd64 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-0.6 required=4.5 tests=BAYES_00,RCVD_IN_SORBS_DUL autolearn=no version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: FreeBSD 8.1 and HAST X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2010 21:47:21 -0000 --oyUTqETQ0mS9luUI Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Jun 13, 2010 at 01:56:31PM +0300, Mikolaj Golub wrote: >=20 > Just a note, not related to this problem: >=20 > On Sun, 13 Jun 2010 17:44:12 +0900 hiroshi@soupacific.com wrote: >=20 > h> Jun 13 17:35:54 sv01B hastd: Accepting connection to tcp4://0.0.0.0:8= 457. > h> Jun 13 17:35:54 sv01B hastd: Connection from tcp4://192.168.0.241:845= 7 to tcp4://192.168.0.240:24260. >=20 > It looks like the addresses in this message are swapped. Also I have noti= ced > one typo in another log message and the issue with send thread logging as= if > it is disk thread (I reported about this earlier in this thread). Please = look > at the attached patch. Committed, thanks! --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --oyUTqETQ0mS9luUI Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkwWo0kACgkQForvXbEpPzSkHwCfaaZsGUfH4xlrezdVTgWRkA/Q ICMAoOOw0/H17LDfTLggeRGsxeL7c2co =+PnE -----END PGP SIGNATURE----- --oyUTqETQ0mS9luUI-- From owner-freebsd-fs@FreeBSD.ORG Mon Jun 14 21:49:08 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 808A01065674 for ; Mon, 14 Jun 2010 21:49:08 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello089077043238.chello.pl [89.77.43.238]) by mx1.freebsd.org (Postfix) with ESMTP id B88D88FC15 for ; Mon, 14 Jun 2010 21:49:07 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id E8C3B45C8A; Mon, 14 Jun 2010 23:49:05 +0200 (CEST) Received: from localhost (chello089077043238.chello.pl [89.77.43.238]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 94762456B1; Mon, 14 Jun 2010 23:49:00 +0200 (CEST) Date: Mon, 14 Jun 2010 23:48:51 +0200 From: Pawel Jakub Dawidek To: "hiroshi@soupacific.com" Message-ID: <20100614214851.GB2498@garage.freebsd.pl> References: <4C14215D.9090304@soupacific.com> <20100613003635.GA60012@icarus.home.lan> <20100613074921.GB1320@garage.freebsd.pl> <4C149A5C.3070401@soupacific.com> <20100613102401.GE1320@garage.freebsd.pl> <86eigavzsg.fsf@kopusha.home.net> <20100614095044.GH1721@garage.freebsd.pl> <868w6hwt2w.fsf@kopusha.home.net> <20100614153746.GN1721@garage.freebsd.pl> <4C165871.6060609@soupacific.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="9zSXsLTf0vkW971A" Content-Disposition: inline In-Reply-To: <4C165871.6060609@soupacific.com> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT amd64 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-0.6 required=4.5 tests=BAYES_00,RCVD_IN_SORBS_DUL, TO_ADDRESS_EQ_REAL autolearn=no version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: FreeBSD 8.1 and HAST X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2010 21:49:08 -0000 --9zSXsLTf0vkW971A Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jun 15, 2010 at 01:27:29AM +0900, hiroshi@soupacific.com wrote: > Good Morning ! >=20 > Thanks pjd@ ! >=20 > > http://people.freebsd.org/~pjd/patches/hastd_primary.c.2.patch > > > It works fine now! >=20 > I sincry appreciate your great works and fastest responce! Thank you guys for all the help with tracking it down and testing the fix, which is now committed to HEAD. > I believe that HAST will change the world ! :) --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --9zSXsLTf0vkW971A Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkwWo8IACgkQForvXbEpPzTbxgCeI4Ssl9OCUBhlC0iDZzr0Wjai 5KUAn06Td7ABkVadTM4bnng2dfu3VuyB =VeHK -----END PGP SIGNATURE----- --9zSXsLTf0vkW971A-- From owner-freebsd-fs@FreeBSD.ORG Mon Jun 14 21:54:02 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1ED8F106564A for ; Mon, 14 Jun 2010 21:54:02 +0000 (UTC) (envelope-from thomas@gibfest.dk) Received: from mail.tyknet.dk (mail.tyknet.dk [213.150.42.155]) by mx1.freebsd.org (Postfix) with ESMTP id 802588FC16 for ; Mon, 14 Jun 2010 21:54:01 +0000 (UTC) Received: from webmail.tyknet.dk (mail.skabet.cn.dom [10.16.255.155]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.tyknet.dk (Postfix) with ESMTPSA id 7D031639A36 for ; Mon, 14 Jun 2010 23:34:18 +0200 (CEST) X-DKIM: OpenDKIM Filter v1.1.2 mail.tyknet.dk 7D031639A36 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=gibfest.dk; s=default; t=1276551258; bh=sTCB1gAeHIePmfgfN5JdW+kYKlU/y2QLxgpKJREQNqU=; h=MIME-Version:Content-Type:Content-Transfer-Encoding:Date:From:To: Subject:In-Reply-To:References:Message-ID; b=P5LKxT5c3R5azx8xwQ2dHV9YXbTOB3XljMAb+FboR3SdYF70/beLhOAGokP5dp7SC XfA6lzD7ie5F28Lto6rscElGCqr2BkzyOlJNK1qVPQPUNMSgEy0b8Nu3r6SpZXe3Mq u/FG6jZ1YpdSFkpSass33e2MC34yJ+gLuO1/MY7Q= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Date: Mon, 14 Jun 2010 23:34:18 +0200 From: Thomas Steen Rasmussen To: In-Reply-To: <20100614153746.GN1721@garage.freebsd.pl> References: <4C139F9C.2090305@soupacific.com> <86iq5oc82y.fsf@kopusha.home.net> <4C14215D.9090304@soupacific.com> <20100613003635.GA60012@icarus.home.lan> <20100613074921.GB1320@garage.freebsd.pl> <4C149A5C.3070401@soupacific.com> <20100613102401.GE1320@garage.freebsd.pl> <86eigavzsg.fsf@kopusha.home.net> <20100614095044.GH1721@garage.freebsd.pl> <868w6hwt2w.fsf@kopusha.home.net> <20100614153746.GN1721@garage.freebsd.pl> Message-ID: <714dcb527e912e5805a77a8262abfec8@10.16.255.155> X-Sender: thomas@gibfest.dk User-Agent: RoundCube Webmail/0.4-beta Subject: Re: FreeBSD 8.1 and HAST X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2010 21:54:02 -0000 On Mon, 14 Jun 2010 17:37:46 +0200, Pawel Jakub Dawidek wrote: > On Mon, Jun 14, 2010 at 06:07:51PM +0300, Mikolaj Golub wrote: >> So ggate_recv takes free just initilized hio (0x284f2640) from the free >> queue >> and is waiting for the data from the kernel. At this time sync thread >> starts >> syncronization, takes another just initilized hio (0x284f2600) from the >> free >> queue and puts it to remote_send, so the request is sent with seq == 0. > > Right, sync code was broken. For sync requests gctl_seq is irrelevant, > so could you guys test this patch: > > http://people.freebsd.org/~pjd/patches/hastd_primary.c.2.patch Hello, I am trying out HAST for the first time and I believe I may be seeing this problem - will this fix be committed in time for 8.1 ? Thank you Thomas Steen Rasmussen From owner-freebsd-fs@FreeBSD.ORG Tue Jun 15 00:30:04 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8063E106564A for ; Tue, 15 Jun 2010 00:30:04 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 5689F8FC0A for ; Tue, 15 Jun 2010 00:30:04 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o5F0U4tN070899 for ; Tue, 15 Jun 2010 00:30:04 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o5F0U44l070894; Tue, 15 Jun 2010 00:30:04 GMT (envelope-from gnats) Date: Tue, 15 Jun 2010 00:30:04 GMT Message-Id: <201006150030.o5F0U44l070894@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: dfilter@FreeBSD.ORG (dfilter service) Cc: Subject: Re: kern/144458: commit references a PR X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: dfilter service List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2010 00:30:04 -0000 The following reply was made to PR kern/144458; it has been noted by GNATS. From: dfilter@FreeBSD.ORG (dfilter service) To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/144458: commit references a PR Date: Tue, 15 Jun 2010 00:25:16 +0000 (UTC) Author: rmacklem Date: Tue Jun 15 00:25:04 2010 New Revision: 209191 URL: http://svn.freebsd.org/changeset/base/209191 Log: Add MODULE_DEPEND() macros to the experimental NFS client and server so that the modules will load when kernels are built with none of the NFS* configuration options specified. I believe this resolves the problems reported by PR kern/144458 and the email on freebsd-stable@ posted by Dmitry Pryanishnikov on June 13. Tested by: kib PR: kern/144458 Reviewed by: kib MFC after: 1 week Modified: head/sys/fs/nfsclient/nfs_clport.c head/sys/fs/nfsserver/nfs_nfsdport.c Modified: head/sys/fs/nfsclient/nfs_clport.c ============================================================================== --- head/sys/fs/nfsclient/nfs_clport.c Mon Jun 14 23:51:35 2010 (r209190) +++ head/sys/fs/nfsclient/nfs_clport.c Tue Jun 15 00:25:04 2010 (r209191) @@ -1275,4 +1275,6 @@ DECLARE_MODULE(nfscl, nfscl_mod, SI_SUB_ /* So that loader and kldload(2) can find us, wherever we are.. */ MODULE_VERSION(nfscl, 1); MODULE_DEPEND(nfscl, nfscommon, 1, 1, 1); +MODULE_DEPEND(nfscl, krpc, 1, 1, 1); +MODULE_DEPEND(nfscl, nfssvc, 1, 1, 1); Modified: head/sys/fs/nfsserver/nfs_nfsdport.c ============================================================================== --- head/sys/fs/nfsserver/nfs_nfsdport.c Mon Jun 14 23:51:35 2010 (r209190) +++ head/sys/fs/nfsserver/nfs_nfsdport.c Tue Jun 15 00:25:04 2010 (r209191) @@ -3147,4 +3147,6 @@ DECLARE_MODULE(nfsd, nfsd_mod, SI_SUB_VF MODULE_VERSION(nfsd, 1); MODULE_DEPEND(nfsd, nfscommon, 1, 1, 1); MODULE_DEPEND(nfsd, nfslockd, 1, 1, 1); +MODULE_DEPEND(nfsd, krpc, 1, 1, 1); +MODULE_DEPEND(nfsd, nfssvc, 1, 1, 1); _______________________________________________ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Tue Jun 15 01:08:08 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CBCEA1065670; Tue, 15 Jun 2010 01:08:08 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id A37108FC1A; Tue, 15 Jun 2010 01:08:08 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o5F188ZK004884; Tue, 15 Jun 2010 01:08:08 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o5F188IZ004880; Tue, 15 Jun 2010 01:08:08 GMT (envelope-from linimon) Date: Tue, 15 Jun 2010 01:08:08 GMT Message-Id: <201006150108.o5F188IZ004880@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/147790: [zfs] zfs set acl(mode|inherit) fails on existing zfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2010 01:08:08 -0000 Old Synopsis: zfs set acl(mode|inherit) fails on existing zfs New Synopsis: [zfs] zfs set acl(mode|inherit) fails on existing zfs Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Tue Jun 15 01:07:52 UTC 2010 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=147790 From owner-freebsd-fs@FreeBSD.ORG Tue Jun 15 07:53:39 2010 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A05D81065676 for ; Tue, 15 Jun 2010 07:53:39 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from mail.ebusiness-leidinger.de (mail.ebusiness-leidinger.de [217.11.53.44]) by mx1.freebsd.org (Postfix) with ESMTP id 30C038FC2D for ; Tue, 15 Jun 2010 07:53:38 +0000 (UTC) Received: from outgoing.leidinger.net (pD954FFFD.dip.t-dialin.net [217.84.255.253]) by mail.ebusiness-leidinger.de (Postfix) with ESMTPSA id 551C184405C; Tue, 15 Jun 2010 09:53:34 +0200 (CEST) Received: from webmail.leidinger.net (webmail.leidinger.net [192.168.1.102]) by outgoing.leidinger.net (Postfix) with ESMTP id 3351A5562; Tue, 15 Jun 2010 09:53:31 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=Leidinger.net; s=outgoing-alex; t=1276588411; bh=JJ9yoxs57KBQ8WSg75eOExej7vA8ElMFt22vdF+7wm4=; h=Message-ID:Date:From:To:Cc:Subject:References:In-Reply-To: MIME-Version:Content-Type:Content-Transfer-Encoding; b=w4MWHE3AB4YFkR71lJBREUenk1betU6+x6Abc0RC+bPm2y/N2q477b7123HHMmW+S d7udqVZcuw8b2tuy8F+4W9fUUx10BV+z/rJX6CO121/StIvxsZfucvtMlkmqao6+j1 32l4g6FeMAkzlHrzpTR6gtRwe7/Lvm87Xmy0sD+Rs/U311U5YMjzQkzFxkoJZWU6WD F2xybahdeebp2BAQcu+JoTNBEyQs/cRiy6f/z2B70g9bl0+iI4+znSdxjhGqc/wrFm noUw1MYPwzucFTXrxb6kvapl/L9hIPPrlj2qGoXntMVT/A4h6uGyGUhl9LCugGRxpQ kCvYaELYq7Iuw== Received: (from www@localhost) by webmail.leidinger.net (8.14.4/8.13.8/Submit) id o5F7rUdh011672; Tue, 15 Jun 2010 09:53:30 +0200 (CEST) (envelope-from Alexander@Leidinger.net) Received: from pslux.ec.europa.eu (pslux.ec.europa.eu [158.169.9.14]) by webmail.leidinger.net (Horde Framework) with HTTP; Tue, 15 Jun 2010 09:53:30 +0200 Message-ID: <20100615095330.89843jrd13kffdkw@webmail.leidinger.net> Date: Tue, 15 Jun 2010 09:53:30 +0200 From: Alexander Leidinger To: jhell References: <20100609162627.11355zjzwnf7nj8k@webmail.leidinger.net> <4C0FAE2A.7050103@dataix.net> <4C0FB1DE.9080508@dataix.net> <20100610115324.10161biomkjndvy8@webmail.leidinger.net> <20100610173825.164930ekkryr5tes@webmail.leidinger.net> <4C1138D0.7070901@dataix.net> <20100611104219.51344ag1ah7br4kk@webmail.leidinger.net> <4C137ACE.9080900@dataix.net> In-Reply-To: <4C137ACE.9080900@dataix.net> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Dynamic Internet Messaging Program (DIMP) H3 (1.1.4) X-EBL-MailScanner-Information: Please contact the ISP for more information X-EBL-MailScanner-ID: 551C184405C.A693E X-EBL-MailScanner: Found to be clean X-EBL-MailScanner-SpamCheck: not spam, spamhaus-ZEN, SpamAssassin (not cached, score=-1.023, required 6, autolearn=disabled, ALL_TRUSTED -1.00, DKIM_SIGNED 0.10, DKIM_VALID -0.10, DKIM_VALID_AU -0.10, TW_ZF 0.08) X-EBL-MailScanner-From: alexander@leidinger.net X-EBL-MailScanner-Watermark: 1277193216.95304@D+Z2CRnq1/W9p1apuzjCrQ X-EBL-Spam-Status: No Cc: fs@freebsd.org Subject: Re: Do we want a periodic script for a zfs scrub? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2010 07:53:39 -0000 Quoting jhell (from Sat, 12 Jun 2010 08:17:18 -0400): > On 06/11/2010 04:42, Alexander Leidinger wrote: > : #!/bin/sh > : > : lastscrub=$(zpool history exports |grep scrub |tail -1 |cut -f1 -d.) > : todayjul=$(date -j -f "%Y-%m-%d" "+%j" $(date "+%Y-%m-%d")) > : scrubjul=$(date -j -f "%Y-%m-%d" "+%j" $lastscrub) > : > : echo $lastscrub Last Scrub From zpool history > : echo $todayjul Today converted to julian > : echo $scrubjul Last scrub converted to julian > : > : expired=$(($todayjul-$scrubjul)) >> >> Apart from the fact that we can do this with one $(( ))... what happens >> if/when time_t is extended to 64 bits on 32 bit platforms? Can we get >> into trouble with the shell-arithmetic or not? It depends upon the >> bit-size of the shell integers, and the signedness of them. Jilles (our >> shell maintainer) suggested also to use the seconds since epoch and I >> asked him the same question. I'm waiting for an answer from him. >> > > I do not think this would be a problem for the script as the script is > relying on date for the conversion except for the subtraction that is > taking place. > > If there was a problem then I would believe it would have to be > corrected in date(1) & possibly sh(1), I could be wrong though. The question is if the integer arithmetic in shell is supposed/allowed to handle 64bit unsigned integers on 32bit machines or not. > Another situation that had come to mind is when & certainly being > possible that the time on the system could drop to before the last scrub > has taken place causing (using the above script for example) todayjul to > be less than scrubjul and in that case would output a negative integer > that would skew results until the system time has been restored to its > correct & current date & time. I think there are more places which are screwed up when the time is not correct, so I will not say "hey, yeah! let's add some code for this" now (I see no problem if there is no scrub until the date of the machine is fixed). Bye, Alexander. -- Be sure to evaluate the bird-hand/bush ratio. http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 From owner-freebsd-fs@FreeBSD.ORG Tue Jun 15 08:42:21 2010 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DEE3B106566B for ; Tue, 15 Jun 2010 08:42:21 +0000 (UTC) (envelope-from se@FreeBSD.org) Received: from smtp107.plus.mail.re1.yahoo.com (smtp107.plus.mail.re1.yahoo.com [69.147.102.70]) by mx1.freebsd.org (Postfix) with SMTP id 800B58FC12 for ; Tue, 15 Jun 2010 08:42:21 +0000 (UTC) Received: (qmail 16398 invoked from network); 15 Jun 2010 08:15:40 -0000 Received: from [10.141.19.34] (se@88.128.88.17 with plain) by smtp107.plus.mail.re1.yahoo.com with SMTP; 15 Jun 2010 01:15:32 -0700 PDT X-Yahoo-SMTP: iDf2N9.swBDAhYEh7VHfpgq0lnq. X-YMail-OSG: h8nvFesVM1k1GSt3kSinb83T3dgRmAUe7aelWCO1FlGIvvX 1KQtP_ZW6e.V7wTaV3I_WB8aW2aLoWVy47cddsFDEB2uHodY7YBnsbKhcbDs GrFMEVNyDiXg.WrpZ1SY398vAwb.aEq.1hkouPs._8VqfIt8WddbuX8AhDNU Fqye6gZOHZyobHohAk34YR5tfm11uKPRtN6Sza.UrEFC_p.aFk0Bj_9p1RIs ZXZBVtiKJyCXUTXDY7.TaQDYqt8V28m1uZlIXWiPHSkJe7ny6Mx4KqKYk8BM sBP5HBeTM0Gx_xnNEaEKOj5XvR9jSi7zCiZEZAzzYhdk3b5WJ7QgGjyFsR8N d3v9OYXoqXCt9VMBU7iWivCLNpjuH7TdVKFM2sQrAc9wSSrzH3LlEThlg40S P8OM- X-Yahoo-Newman-Property: ymail-5 Message-ID: <4C173693.9020603@FreeBSD.org> Date: Tue, 15 Jun 2010 10:15:15 +0200 From: Stefan Esser User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.8.1.18) Gecko/20081105 Thunderbird/2.0.0.18 ThunderBrowse/3.2.2.1 Mnenhy/0.7.5.666 MIME-Version: 1.0 To: Alexander Leidinger References: <20100609162627.11355zjzwnf7nj8k@webmail.leidinger.net> <4C0FAE2A.7050103@dataix.net> <4C0FB1DE.9080508@dataix.net> <20100610115324.10161biomkjndvy8@webmail.leidinger.net> <20100610173825.164930ekkryr5tes@webmail.leidinger.net> <4C1138D0.7070901@dataix.net> <20100611104219.51344ag1ah7br4kk@webmail.leidinger.net> <4C137ACE.9080900@dataix.net> <20100615095330.89843jrd13kffdkw@webmail.leidinger.net> In-Reply-To: <20100615095330.89843jrd13kffdkw@webmail.leidinger.net> X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: 7bit Cc: fs@freebsd.org Subject: Re: Do we want a periodic script for a zfs scrub? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2010 08:42:22 -0000 Am 15.06.2010 09:53, schrieb Alexander Leidinger: > Quoting jhell (from Sat, 12 Jun 2010 08:17:18 -0400): > >> On 06/11/2010 04:42, Alexander Leidinger wrote: >> : #!/bin/sh >> : >> : lastscrub=$(zpool history exports |grep scrub |tail -1 |cut -f1 -d.) >> : todayjul=$(date -j -f "%Y-%m-%d" "+%j" $(date "+%Y-%m-%d")) >> : scrubjul=$(date -j -f "%Y-%m-%d" "+%j" $lastscrub) >> : >> : echo $lastscrub Last Scrub From zpool history >> : echo $todayjul Today converted to julian >> : echo $scrubjul Last scrub converted to julian >> : >> : expired=$(($todayjul-$scrubjul)) >>> >>> Apart from the fact that we can do this with one $(( ))... what happens >>> if/when time_t is extended to 64 bits on 32 bit platforms? Can we get >>> into trouble with the shell-arithmetic or not? It depends upon the >>> bit-size of the shell integers, and the signedness of them. Jilles (our >>> shell maintainer) suggested also to use the seconds since epoch and I >>> asked him the same question. I'm waiting for an answer from him. >>> >> >> I do not think this would be a problem for the script as the script is >> relying on date for the conversion except for the subtraction that is >> taking place. >> >> If there was a problem then I would believe it would have to be >> corrected in date(1) & possibly sh(1), I could be wrong though. > > The question is if the integer arithmetic in shell is supposed/allowed > to handle 64bit unsigned integers on 32bit machines or not. I extended expr and test (standalone binaries and /bin/sh builtins) to handle 64 bit on i386 a looong time ago (and I also added overflow checks to all calculations). But then it was decided, that POSIX demands 32 bit range on 32 bit machines, so support for 64 bit range was made conditional on a switch (expr -e if memory serves me right ...). So yes, 64 bit range is available on all platforms. Best regards, STefan From owner-freebsd-fs@FreeBSD.ORG Tue Jun 15 08:59:36 2010 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4D3191065674; Tue, 15 Jun 2010 08:59:36 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from mail.ebusiness-leidinger.de (mail.ebusiness-leidinger.de [217.11.53.44]) by mx1.freebsd.org (Postfix) with ESMTP id E45C98FC0A; Tue, 15 Jun 2010 08:59:35 +0000 (UTC) Received: from outgoing.leidinger.net (pD954FFFD.dip.t-dialin.net [217.84.255.253]) by mail.ebusiness-leidinger.de (Postfix) with ESMTPSA id CD7BA84405C; Tue, 15 Jun 2010 10:59:32 +0200 (CEST) Received: from webmail.leidinger.net (webmail.leidinger.net [192.168.1.102]) by outgoing.leidinger.net (Postfix) with ESMTP id 42E25556A; Tue, 15 Jun 2010 10:59:29 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=Leidinger.net; s=outgoing-alex; t=1276592369; bh=RiMQktv1e2AyVlYfP0VrqQ0aYovi9RxbpCcspeBa2RI=; h=Message-ID:Date:From:To:Cc:Subject:References:In-Reply-To: MIME-Version:Content-Type:Content-Transfer-Encoding; b=XAkt37rhlrPAA175M4maysU3jlgSnw8RwBMMC44/+lnd8Xy1eolPxEgfqGb5ltGUV nm6thgFYZSEhP01z5RaIaLsazPTM+ljKu5q5LzWSqpSqLjy+YpNzusUp2ydmLypgcA TjEIChrIBYLoAvNS1BuVeu0++NY5mFTrFviGZR+Upq7TSI765upEYiuDZ7KqTq02Tq kkq8FrojfRp90anCBTHPR/+Fh20aq34xV1GQ44wNhm1lNJpuDClXbOTx7uJpcpHJdo hQIT/Vc5VcCq5pwLdnRiLzQfTpn1czSz9lg7hZWW62yVJ/rAA/JWguk4M7MuPH8g08 6yTXBtWEi2yHA== Received: (from www@localhost) by webmail.leidinger.net (8.14.4/8.13.8/Submit) id o5F8xSF0027130; Tue, 15 Jun 2010 10:59:28 +0200 (CEST) (envelope-from Alexander@Leidinger.net) Received: from pslux.ec.europa.eu (pslux.ec.europa.eu [158.169.9.14]) by webmail.leidinger.net (Horde Framework) with HTTP; Tue, 15 Jun 2010 10:59:28 +0200 Message-ID: <20100615105928.8033401bkc5krfuo@webmail.leidinger.net> Date: Tue, 15 Jun 2010 10:59:28 +0200 From: Alexander Leidinger To: Stefan Esser References: <20100609162627.11355zjzwnf7nj8k@webmail.leidinger.net> <4C0FAE2A.7050103@dataix.net> <4C0FB1DE.9080508@dataix.net> <20100610115324.10161biomkjndvy8@webmail.leidinger.net> <20100610173825.164930ekkryr5tes@webmail.leidinger.net> <4C1138D0.7070901@dataix.net> <20100611104219.51344ag1ah7br4kk@webmail.leidinger.net> <4C137ACE.9080900@dataix.net> <20100615095330.89843jrd13kffdkw@webmail.leidinger.net> <4C173693.9020603@FreeBSD.org> In-Reply-To: <4C173693.9020603@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Dynamic Internet Messaging Program (DIMP) H3 (1.1.4) X-EBL-MailScanner-Information: Please contact the ISP for more information X-EBL-MailScanner-ID: CD7BA84405C.A88B8 X-EBL-MailScanner: Found to be clean X-EBL-MailScanner-SpamCheck: not spam, spamhaus-ZEN, SpamAssassin (not cached, score=-1.023, required 6, autolearn=disabled, ALL_TRUSTED -1.00, DKIM_SIGNED 0.10, DKIM_VALID -0.10, DKIM_VALID_AU -0.10, TW_ZF 0.08) X-EBL-MailScanner-From: alexander@leidinger.net X-EBL-MailScanner-Watermark: 1277197174.05752@jTgekAsDF+gtZY30sEuINw X-EBL-Spam-Status: No Cc: fs@FreeBSD.org Subject: Re: Do we want a periodic script for a zfs scrub? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2010 08:59:36 -0000 Quoting Stefan Esser (from Tue, 15 Jun 2010 10:15:15 +0200): > Am 15.06.2010 09:53, schrieb Alexander Leidinger: >> The question is if the integer arithmetic in shell is supposed/allowed >> to handle 64bit unsigned integers on 32bit machines or not. > > I extended expr and test (standalone binaries and /bin/sh builtins) to > handle 64 bit on i386 a looong time ago (and I also added overflow > checks to all calculations). But then it was decided, that POSIX demands > 32 bit range on 32 bit machines, so support for 64 bit range was made > conditional on a switch (expr -e if memory serves me right ...). So yes, > 64 bit range is available on all platforms. Great, I converted to expr -e and used seconds since epoch to calculate the date difference. Bye, Alexander. -- Anything worth doing is worth overdoing. http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 From owner-freebsd-fs@FreeBSD.ORG Tue Jun 15 09:00:18 2010 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1A58E106566C; Tue, 15 Jun 2010 09:00:18 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from mail.ebusiness-leidinger.de (mail.ebusiness-leidinger.de [217.11.53.44]) by mx1.freebsd.org (Postfix) with ESMTP id B8CA38FC08; Tue, 15 Jun 2010 09:00:17 +0000 (UTC) Received: from outgoing.leidinger.net (pD954FFFD.dip.t-dialin.net [217.84.255.253]) by mail.ebusiness-leidinger.de (Postfix) with ESMTPSA id 6C4A1844077; Tue, 15 Jun 2010 11:00:13 +0200 (CEST) Received: from webmail.leidinger.net (webmail.leidinger.net [192.168.1.102]) by outgoing.leidinger.net (Postfix) with ESMTP id 8C7D5556C; Tue, 15 Jun 2010 11:00:10 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=Leidinger.net; s=outgoing-alex; t=1276592410; bh=yZOxHtkqbtR7qCD4625VrzAqsmOgfkMGcMrbpjE7RHM=; h=Message-ID:Date:From:To:Cc:Subject:References:In-Reply-To: MIME-Version:Content-Type:Content-Transfer-Encoding; b=wjurrV+Ox4XhIqmTrOq4zRLH4zdufSI81JaIbxbZR/gheH/K2nmbvQqH9D0AV4NRe gA16EPMVRJk85up17qaZ2LA85ZGyF1amZ2I65VyayTWjeXws8kbe/gRO2z2rPAdrQ7 itkjjRNPA/UQRu7/y5x1TB+v3eOUOjuTd8h3PZk8Ruuc6l7VSz2T1Bdwf4tlxr+yk9 ESuSH0Tw8TVUZGuY6bWBwkGT9NrjxdSi6TwHiNHmuxGqYi+9R/i0Z9ZlSKNmJHmJj3 1Iaw7SKn5F2iOe2Nsbko1HoMC4zTU3ZMOGkAsq+u+aIHTiQWpgGAh+pbw+DYOyK3eY OnRfiY+BCc5LQ== Received: (from www@localhost) by webmail.leidinger.net (8.14.4/8.13.8/Submit) id o5F90AUi027498; Tue, 15 Jun 2010 11:00:10 +0200 (CEST) (envelope-from Alexander@Leidinger.net) Received: from pslux.ec.europa.eu (pslux.ec.europa.eu [158.169.9.14]) by webmail.leidinger.net (Horde Framework) with HTTP; Tue, 15 Jun 2010 11:00:10 +0200 Message-ID: <20100615110010.29831wb37ppb8izo@webmail.leidinger.net> Date: Tue, 15 Jun 2010 11:00:10 +0200 From: Alexander Leidinger To: Pawel Jakub Dawidek References: <20100610162629.38992mazf0sfdqg0@webmail.leidinger.net> <20100613094652.GC1320@garage.freebsd.pl> In-Reply-To: <20100613094652.GC1320@garage.freebsd.pl> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Dynamic Internet Messaging Program (DIMP) H3 (1.1.4) X-EBL-MailScanner-Information: Please contact the ISP for more information X-EBL-MailScanner-ID: 6C4A1844077.A732E X-EBL-MailScanner: Found to be clean X-EBL-MailScanner-SpamCheck: not spam, spamhaus-ZEN, SpamAssassin (not cached, score=-1.1, required 6, autolearn=disabled, ALL_TRUSTED -1.00, DKIM_SIGNED 0.10, DKIM_VALID -0.10, DKIM_VALID_AU -0.10) X-EBL-MailScanner-From: alexander@leidinger.net X-EBL-MailScanner-Watermark: 1277197216.30362@jaC8xDuLDvrCnD3M0D+z/Q X-EBL-Spam-Status: No Cc: fs@FreeBSD.org Subject: Re: CFT: periodic scrubbing of ZFS pools X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2010 09:00:18 -0000 Quoting Pawel Jakub Dawidek (from Sun, 13 Jun 2010 11:46:52 +0200): > All in all, with better grep I'd like to see it committed. Done. Bye, Alexander. -- Q: Why don't lawyers go to the beach? A: The cats keep trying to bury them. http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 From owner-freebsd-fs@FreeBSD.ORG Tue Jun 15 09:45:09 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CAA6A1065673; Tue, 15 Jun 2010 09:45:09 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from mail.ebusiness-leidinger.de (mail.ebusiness-leidinger.de [217.11.53.44]) by mx1.freebsd.org (Postfix) with ESMTP id 7200C8FC24; Tue, 15 Jun 2010 09:45:09 +0000 (UTC) Received: from outgoing.leidinger.net (pD954FFFD.dip.t-dialin.net [217.84.255.253]) by mail.ebusiness-leidinger.de (Postfix) with ESMTPSA id 47C6784405C; Tue, 15 Jun 2010 11:45:00 +0200 (CEST) Received: from webmail.leidinger.net (webmail.leidinger.net [192.168.1.102]) by outgoing.leidinger.net (Postfix) with ESMTP id 29FE85572; Tue, 15 Jun 2010 11:44:57 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=Leidinger.net; s=outgoing-alex; t=1276595097; bh=URMtmW4Wlsd4IQ9WwTwKENWM6Rmz6n3pXMlwSN3iwv4=; h=Message-ID:Date:From:To:Cc:Subject:References:In-Reply-To: MIME-Version:Content-Type:Content-Transfer-Encoding; b=FQRElRh33UTFNRD9byAHT9yslrtkNrqvucfwm2X1CiKGZLhtIMbih3/KfMxT38JXk rObYyVfCJVyCcWV9/AMwz6aBPg+3yFg3sYldnwZGc5bX8X+++nrRRbXVIiyTFP67fz pvLkc/pAIaOebxvFcfj0P7Ymw9uV6PNqX++Yt6idu8WXz4ln+2nwOVIA8F8wbQL2S3 bLxP/ZP9PgUY7SLfmUx9LkE/ivEN3T+wXmVhwD3bqjlr0BI0qLcVW+nUyWsNCGh+fc s4nnk4FQLo2wiN56GHW6s3LPMFZz4gncrUYFd1RwKEmWxmEf6aYeWYDlnNDL3okwK+ iYMU4SUSQ51gg== Received: (from www@localhost) by webmail.leidinger.net (8.14.4/8.13.8/Submit) id o5F9iuAM037819; Tue, 15 Jun 2010 11:44:56 +0200 (CEST) (envelope-from Alexander@Leidinger.net) Received: from pslux.ec.europa.eu (pslux.ec.europa.eu [158.169.9.14]) by webmail.leidinger.net (Horde Framework) with HTTP; Tue, 15 Jun 2010 11:44:56 +0200 Message-ID: <20100615114456.49994h9g40qhcnms@webmail.leidinger.net> Date: Tue, 15 Jun 2010 11:44:56 +0200 From: Alexander Leidinger To: Ivan Voras References: <20100610162629.38992mazf0sfdqg0@webmail.leidinger.net> <20100611172033.42001s90ahe57oe8@webmail.leidinger.net> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Dynamic Internet Messaging Program (DIMP) H3 (1.1.4) X-EBL-MailScanner-Information: Please contact the ISP for more information X-EBL-MailScanner-ID: 47C6784405C.A63C2 X-EBL-MailScanner: Found to be clean X-EBL-MailScanner-SpamCheck: not spam, spamhaus-ZEN, SpamAssassin (not cached, score=-1.1, required 6, autolearn=disabled, ALL_TRUSTED -1.00, DKIM_SIGNED 0.10, DKIM_VALID -0.10, DKIM_VALID_AU -0.10) X-EBL-MailScanner-From: alexander@leidinger.net X-EBL-MailScanner-Watermark: 1277199901.0497@vhb/sByYpcK3+uPDlMOTJQ X-EBL-Spam-Status: No Cc: freebsd-fs@freebsd.org Subject: Re: CFT: periodic scrubbing of ZFS pools X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2010 09:45:10 -0000 Quoting Ivan Voras (from Fri, 11 Jun 2010 18:11:45 +0200): > On 11 June 2010 17:20, Alexander Leidinger wrote: >> Quoting Ivan Voras (from Fri, 11 Jun 2010 14:04:24 >> +0200): > >>> Fairly good and useful, but could you add a small check of "zpool >>> status" information before scrubbing that would a) complain LOUDLY AND >>> VISIBLY if a previous scrub failed and b) skip issuing a new scrub >>> command if there is such an error, to avoid stressing possibly broken >>> hardware? >> >> Can you please provide an example of such a failed scrub? > > You should probably treat any status message that doesn't have "none > requested" or "scrub completed with 0 errors..." as failed. I prefer to make it explicit instead of failing on everything which is unknown. > I could setup a gnop device with errors to prove it if you'd like :) Would be great. Bye, Alexander. -- "My two friends have my share of the money, I'll have to speak to them." -- Vito Corleone, "Chapter 14", page 197 http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 From owner-freebsd-fs@FreeBSD.ORG Tue Jun 15 13:21:47 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 188DF1065677 for ; Tue, 15 Jun 2010 13:21:47 +0000 (UTC) (envelope-from thomas@gibfest.dk) Received: from mail.tyknet.dk (mail.tyknet.dk [213.150.42.155]) by mx1.freebsd.org (Postfix) with ESMTP id C74CA8FC0A for ; Tue, 15 Jun 2010 13:21:46 +0000 (UTC) Received: from [10.32.67.5] (fw.int.webpartner.dk [213.150.34.98]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mail.tyknet.dk (Postfix) with ESMTPSA id 8BD82639A36 for ; Tue, 15 Jun 2010 15:21:45 +0200 (CEST) X-DKIM: OpenDKIM Filter v1.1.2 mail.tyknet.dk 8BD82639A36 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=gibfest.dk; s=default; t=1276608105; bh=FsRLNer3EeefviOn7v0cboN9ydLsst/WNH65MDs9zJc=; h=Message-ID:Date:From:MIME-Version:To:Subject:Content-Type: Content-Transfer-Encoding; b=fywIpFuhFntyPeDdVb/2WaMEUHxpd5EImNurvxHJZneL9p27oCA/nn+bC9RrnrhVZ TDui9d5DpSD424Ivrt+O0ZcZS2tDXENjWbSrUIeQdQ/sdZSj8IAy4UazPOvQW1AKFc RYF7NN8Npmz7hvuPl5T0l8tmMwQVdqUTws6kq5HQ= Message-ID: <4C177E69.3020204@gibfest.dk> Date: Tue, 15 Jun 2010 15:21:45 +0200 From: Thomas Steen Rasmussen User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.9) Gecko/20100317 Lightning/1.0b1 Thunderbird/3.0.4 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: ZFS l2arc and HAST ? newbie question X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2010 13:21:47 -0000 Hello list, I am playing with HAST in order to build some redundant storage for a mailserver, using ZFS as the filesystem. I have the following zpool layout before stating the HAST experiments: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz2 ONLINE 0 0 0 label/hd4 ONLINE 0 0 0 label/hd5 ONLINE 0 0 0 label/hd6 ONLINE 0 0 0 label/hd7 ONLINE 0 0 0 logs ONLINE 0 0 0 mirror ONLINE 0 0 0 label/ssd0s1 ONLINE 0 0 0 label/ssd1s1 ONLINE 0 0 0 cache label/ssd0s2 ONLINE 0 0 0 label/ssd1s2 ONLINE 0 0 0 As I understand it, to accomplish this with HAST I will need to make a HAST resource for each physical disk, like so: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz2 ONLINE 0 0 0 hast/hahd4 ONLINE 0 0 0 hast/hahd5 ONLINE 0 0 0 hast/hahd6 ONLINE 0 0 0 hast/hahd7 ONLINE 0 0 0 But what about slog and cache devices, currently on SSD disks for performance reasons ? It doesn't really make sense to synchronize a cache disk over the network, does it ? Could I build the zpool with the SSD disks directly (without HAST) and would ZFS survive an export/import on the other host, when the cache disks are suddently different ? I am thinking cache only here, not slog. Do SSD l2arc / slog even make any sense when I am "deliberately" slowing down the filsystem with network redundancy anyway ? Oh, and is there any problems using labels for HAST devices ? My controller likes to give new device names to disks now and then, and it has been a blessing to use labels instead of device names, so I'd like to continue doing that when using HAST. If needed, any testing on my part will unfortunately have to wait a couple of days for the MFC of the HAST fix from yesterday, as the SEQ issue is preventing me from further experiments with HAST for now. Thank you for any input, and _THANK YOU_ for the work on both ZFS and HAST, their combined awesomeness is reaching epic proportions. Best regards Thomas Steen Rasmussen From owner-freebsd-fs@FreeBSD.ORG Tue Jun 15 14:04:18 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AA8A31065672 for ; Tue, 15 Jun 2010 14:04:18 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta06.emeryville.ca.mail.comcast.net (qmta06.emeryville.ca.mail.comcast.net [76.96.30.56]) by mx1.freebsd.org (Postfix) with ESMTP id 92DBB8FC23 for ; Tue, 15 Jun 2010 14:04:18 +0000 (UTC) Received: from omta15.emeryville.ca.mail.comcast.net ([76.96.30.71]) by qmta06.emeryville.ca.mail.comcast.net with comcast id WDFk1e0031Y3wxoA6E4Jls; Tue, 15 Jun 2010 14:04:18 +0000 Received: from koitsu.dyndns.org ([98.248.46.159]) by omta15.emeryville.ca.mail.comcast.net with comcast id WE4H1e0023S48mS8bE4H7r; Tue, 15 Jun 2010 14:04:17 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id D0C479B418; Tue, 15 Jun 2010 07:04:16 -0700 (PDT) Date: Tue, 15 Jun 2010 07:04:16 -0700 From: Jeremy Chadwick To: Thomas Steen Rasmussen Message-ID: <20100615140416.GA48384@icarus.home.lan> References: <4C177E69.3020204@gibfest.dk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4C177E69.3020204@gibfest.dk> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-fs@freebsd.org Subject: Re: ZFS l2arc and HAST ? newbie question X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2010 14:04:18 -0000 On Tue, Jun 15, 2010 at 03:21:45PM +0200, Thomas Steen Rasmussen wrote: > But what about slog and cache devices, currently on SSD disks for > performance reasons ? It doesn't really make sense to synchronize > a cache disk over the network, does it ? > > Could I build the zpool with the SSD disks directly (without > HAST) and would ZFS survive an export/import on the other host, > when the cache disks are suddently different ? I am thinking cache > only here, not slog. Taken from zpool(8): If a read error is encountered on a cache device, that read I/O is reissued to the original storage pool device, which might be part of a mirrored or raidz configuration. The content of the cache devices is considered volatile, as is the case with other system caches. So I would say use the local system's SSDs for "cache" and leave the rest for HAST. With regards to "Intent Logs" -- I simply do not know. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Tue Jun 15 14:55:59 2010 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E80E9106566C for ; Tue, 15 Jun 2010 14:55:59 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from fallbackmx07.syd.optusnet.com.au (fallbackmx07.syd.optusnet.com.au [211.29.132.9]) by mx1.freebsd.org (Postfix) with ESMTP id C1F1F8FC1C for ; Tue, 15 Jun 2010 14:55:58 +0000 (UTC) Received: from mail04.syd.optusnet.com.au (mail04.syd.optusnet.com.au [211.29.132.185]) by fallbackmx07.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id o5FDAhjT007155 for ; Tue, 15 Jun 2010 23:10:43 +1000 Received: from c122-106-175-69.carlnfd1.nsw.optusnet.com.au (c122-106-175-69.carlnfd1.nsw.optusnet.com.au [122.106.175.69]) by mail04.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id o5FDAaVv021171 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 15 Jun 2010 23:10:38 +1000 Date: Tue, 15 Jun 2010 23:10:36 +1000 (EST) From: Bruce Evans X-X-Sender: bde@delplex.bde.org To: Stefan Esser In-Reply-To: <4C173693.9020603@FreeBSD.org> Message-ID: <20100615221954.R38827@delplex.bde.org> References: <20100609162627.11355zjzwnf7nj8k@webmail.leidinger.net> <4C0FAE2A.7050103@dataix.net> <4C0FB1DE.9080508@dataix.net> <20100610115324.10161biomkjndvy8@webmail.leidinger.net> <20100610173825.164930ekkryr5tes@webmail.leidinger.net> <4C1138D0.7070901@dataix.net> <20100611104219.51344ag1ah7br4kk@webmail.leidinger.net> <4C137ACE.9080900@dataix.net> <20100615095330.89843jrd13kffdkw@webmail.leidinger.net> <4C173693.9020603@FreeBSD.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Alexander Leidinger , fs@freebsd.org Subject: Re: Do we want a periodic script for a zfs scrub? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2010 14:56:00 -0000 On Tue, 15 Jun 2010, Stefan Esser wrote: > Am 15.06.2010 09:53, schrieb Alexander Leidinger: >> Quoting jhell (from Sat, 12 Jun 2010 08:17:18 -0400): >> >>> On 06/11/2010 04:42, Alexander Leidinger wrote: >>> : #!/bin/sh >>> : >>> : lastscrub=$(zpool history exports |grep scrub |tail -1 |cut -f1 -d.) >>> : todayjul=$(date -j -f "%Y-%m-%d" "+%j" $(date "+%Y-%m-%d")) >>> : scrubjul=$(date -j -f "%Y-%m-%d" "+%j" $lastscrub) >>> : >>> : echo $lastscrub Last Scrub From zpool history >>> : echo $todayjul Today converted to julian >>> : echo $scrubjul Last scrub converted to julian >>> : >>> : expired=$(($todayjul-$scrubjul)) >>>> >>>> Apart from the fact that we can do this with one $(( ))... what happens >>>> if/when time_t is extended to 64 bits on 32 bit platforms? Can we get >>>> into trouble with the shell-arithmetic or not? It depends upon the >>>> bit-size of the shell integers, and the signedness of them. Jilles (our >>>> shell maintainer) suggested also to use the seconds since epoch and I >>>> asked him the same question. I'm waiting for an answer from him. >>> >>> I do not think this would be a problem for the script as the script is >>> relying on date for the conversion except for the subtraction that is >>> taking place. >>> >>> If there was a problem then I would believe it would have to be >>> corrected in date(1) & possibly sh(1), I could be wrong though. >> >> The question is if the integer arithmetic in shell is supposed/allowed >> to handle 64bit unsigned integers on 32bit machines or not. Of course it is not required. It is not required even on 64-bit machines. Only signed long integer arithmetic is required in POSIX.1-2001. signed long may have any size with >= 31 value bits. Thus only 31 value bits and 1 sign bit are required on all machines. Longs should be twice as long as the register width on all machines, so they should be 64 bits on 32-bit machines and 128 bits on 64-bit machines, but this is normally broken; normally, 32-bit machines normally have 32-bit longs with 31 value bits, while 64 bit machines normally have 64-bit longs with 63 value bits; thus the shell normally handles only 31-bit unsigned integers on 32-bit machines and 63-bit unsigned integers on 64-bit machines. time_t may have any arithmetic type in C90/99 or POSIX (with POSIX's broken representation in POSIX and any representation in C), but it is normally signed integer, so if it is extended to 64 bits then it probably wouldn't need 64-bit unsigned integers anyway. > I extended expr and test (standalone binaries and /bin/sh builtins) to > handle 64 bit on i386 a looong time ago (and I also added overflow > checks to all calculations). But then it was decided, that POSIX demands > 32 bit range on 32 bit machines, so support for 64 bit range was made > conditional on a switch (expr -e if memory serves me right ...). So yes, > 64 bit range is available on all platforms. Hmm, POSIX doesn't require this brokenness for the shell, though it requires practically equivalent brokenness for all operands and option- arguments generally: it requires "all" integer arithmetic to use signed long variables and "all" floating point arithmetic to use double variables, with Standard C semantics. signed long may have any size, but normally has 31 or 63 value bits as above. It is unclear how a variable can ever be floating point. Then it contradicts^Woverrides "all" with certain extensions for the shell. However, these extensions are almost useless since they mainly make a difference when the signed long arithmetic would overflow, in which case the behaviour is undefined (?) so it can DTRT using any extension that it wants. Bruce From owner-freebsd-fs@FreeBSD.ORG Tue Jun 15 15:54:07 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 851ED1065673 for ; Tue, 15 Jun 2010 15:54:07 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello089077043238.chello.pl [89.77.43.238]) by mx1.freebsd.org (Postfix) with ESMTP id D5DCD8FC08 for ; Tue, 15 Jun 2010 15:54:06 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 1D32C45C8A; Tue, 15 Jun 2010 17:54:05 +0200 (CEST) Received: from localhost (chello089077043238.chello.pl [89.77.43.238]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 4B5684569A; Tue, 15 Jun 2010 17:53:59 +0200 (CEST) Date: Tue, 15 Jun 2010 17:53:48 +0200 From: Pawel Jakub Dawidek To: Thomas Steen Rasmussen Message-ID: <20100615155348.GA1922@garage.freebsd.pl> References: <4C177E69.3020204@gibfest.dk> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="45Z9DzgjV8m4Oswq" Content-Disposition: inline In-Reply-To: <4C177E69.3020204@gibfest.dk> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT amd64 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-0.6 required=4.5 tests=BAYES_00,RCVD_IN_SORBS_DUL autolearn=no version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: ZFS l2arc and HAST ? newbie question X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2010 15:54:07 -0000 --45Z9DzgjV8m4Oswq Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jun 15, 2010 at 03:21:45PM +0200, Thomas Steen Rasmussen wrote: > Hello list, >=20 > I am playing with HAST in order to build some redundant storage > for a mailserver, using ZFS as the filesystem. > I have the following zpool layout before stating the HAST experiments: >=20 > NAME STATE READ WRITE CKSUM > tank ONLINE 0 0 0 > raidz2 ONLINE 0 0 0 > label/hd4 ONLINE 0 0 0 > label/hd5 ONLINE 0 0 0 > label/hd6 ONLINE 0 0 0 > label/hd7 ONLINE 0 0 0 > logs ONLINE 0 0 0 > mirror ONLINE 0 0 0 > label/ssd0s1 ONLINE 0 0 0 > label/ssd1s1 ONLINE 0 0 0 > cache > label/ssd0s2 ONLINE 0 0 0 > label/ssd1s2 ONLINE 0 0 0 >=20 > As I understand it, to accomplish this with HAST I will need to make a > HAST resource for each physical disk, like so: >=20 > NAME STATE READ WRITE CKSUM > tank ONLINE 0 0 0 > raidz2 ONLINE 0 0 0 > hast/hahd4 ONLINE 0 0 0 > hast/hahd5 ONLINE 0 0 0 > hast/hahd6 ONLINE 0 0 0 > hast/hahd7 ONLINE 0 0 0 >=20 > But what about slog and cache devices, currently on SSD disks for > performance reasons ? It doesn't really make sense to synchronize > a cache disk over the network, does it ? No, it doesn't. Cache is forgotten on import anyway, so don't bother. You have to be careful, though, because you probably need to remove old cache device from the pool after import on secondary and add local disk. > Could I build the zpool with the SSD disks directly (without > HAST) and would ZFS survive an export/import on the other host, > when the cache disks are suddently different ? I am thinking cache > only here, not slog. It simply won't find cache disks, you will need to do what I described above. > Do SSD l2arc / slog even make any sense when I am "deliberately" > slowing down the filsystem with network redundancy anyway ? Forget about HAST for L2ARC. In case of SLOG it can still be faster over the network than pool with local SATA disks without SLOG. As usual the best way to verify this is to test it for your workload:) > Oh, and is there any problems using labels for HAST devices ? My > controller likes to give new device names to disks now and then, > and it has been a blessing to use labels instead of device names, > so I'd like to continue doing that when using HAST. Use labeled providers in hast.conf, but there is no need to label HAST providers (/dev/hast/). --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --45Z9DzgjV8m4Oswq Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkwXogwACgkQForvXbEpPzRS7wCg0I8Z5JIeEzGlDKG58PhhcCGg YScAoO8oqrTiQigGDyHDm1gH1U4G8uW0 =Lkt6 -----END PGP SIGNATURE----- --45Z9DzgjV8m4Oswq-- From owner-freebsd-fs@FreeBSD.ORG Tue Jun 15 17:20:11 2010 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EAA021065670 for ; Tue, 15 Jun 2010 17:20:11 +0000 (UTC) (envelope-from fty@employees.org) Received: from tyhouse.portbridge.com (tyhouse.portbridge.com [209.170.129.155]) by mx1.freebsd.org (Postfix) with ESMTP id 9FB008FC1B for ; Tue, 15 Jun 2010 17:20:11 +0000 (UTC) Received: from [10.116.32.157] (rtp-fty-87112.cisco.com [10.116.32.157]) by tyhouse.portbridge.com (8.14.3/8.14.3) with ESMTP id o5FHCR4B003636 for ; Tue, 15 Jun 2010 13:12:28 -0400 (EDT) (envelope-from fty@employees.org) Message-ID: <4C17B3A9.5000302@employees.org> Date: Tue, 15 Jun 2010 13:08:57 -0400 From: Frank Terhaar-Yonkers User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.10) Gecko/20100612 Lightning/1.0b1 Shredder/3.0.6pre ThunderBrowse/3.2.8.1 MIME-Version: 1.0 To: fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: need to remove an active pool component X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2010 17:20:12 -0000 Howdy - It seems I painted myself into a corner. I originally had ad16 as a hot spare to the raidz. Sometime back I used it temporarily because one of the disks in the raidz went offline (ended up being a bad SATA cable) and when repaired swapped the original back in. Intending to put ad16 back as the hotspare I used the wrong command(s) and added it to the pool instead - doh! Anyway, by the time I realized this files were already created on it and now I can't remove it. From reading, it seems that the ability to remove active elements from a pool is a long ways off. So what I was thinking was a bit of a hack to facilitate removing the disk: 1) create a sysctl variable to inhibit write to a particular disk in a pool. 2) parse the meta-data on the disk to get a list of files, then move off, then move back so the file ends up back in the pool but on the raidz. 3) When empty, I should be able to remove the disk - yes? I don't have the space elsewhere to back this thing up and reload it, which would be the easiest thing. Opinions? TIA, - Frank -------------------- tynas(toot)~ ->zs pool: typool state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM typool ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad4 ONLINE 0 0 0 ad6 ONLINE 0 0 0 ad8 ONLINE 0 0 0 ad14 ONLINE 0 0 0 ad16 ONLINE 0 0 0 errors: No known data errors tynas(toot)~ ->zpool remove typool ad16 cannot remove ad16: only inactive hot spares or cache devices can be removed Exit 1 tynas(toot)~ ->zpool detach typool ad16 cannot detach ad16: only applicable to mirror and replacing vdevs Exit 255 tynas(toot)~ ->zpool offline typool ad16 cannot offline ad16: no valid replicas Exit 1 -- \\\\////\\\\////\\\\\////\\\\\////\\\\////\\\\ Frank Terhaar-Yonkers W4FTY TRA 8325/L3 Apex, North Carolina fty@employees.org From owner-freebsd-fs@FreeBSD.ORG Tue Jun 15 18:13:13 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7CE631065675; Tue, 15 Jun 2010 18:13:13 +0000 (UTC) (envelope-from ticso@cicely7.cicely.de) Received: from raven.bwct.de (raven.bwct.de [85.159.14.73]) by mx1.freebsd.org (Postfix) with ESMTP id A1E038FC08; Tue, 15 Jun 2010 18:13:12 +0000 (UTC) Received: from mail.cicely.de ([10.1.1.37]) by raven.bwct.de (8.13.4/8.13.4) with ESMTP id o5FIDAwq028533 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 15 Jun 2010 20:13:10 +0200 (CEST) (envelope-from ticso@cicely7.cicely.de) Received: from cicely7.cicely.de (cicely7.cicely.de [10.1.1.9]) by mail.cicely.de (8.14.3/8.14.3) with ESMTP id o5FICxiA085339 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 15 Jun 2010 20:12:59 +0200 (CEST) (envelope-from ticso@cicely7.cicely.de) Received: from cicely7.cicely.de (localhost [127.0.0.1]) by cicely7.cicely.de (8.14.2/8.14.2) with ESMTP id o5FICxdB016622; Tue, 15 Jun 2010 20:12:59 +0200 (CEST) (envelope-from ticso@cicely7.cicely.de) Received: (from ticso@localhost) by cicely7.cicely.de (8.14.2/8.14.2/Submit) id o5FICwMP016621; Tue, 15 Jun 2010 20:12:58 +0200 (CEST) (envelope-from ticso) Date: Tue, 15 Jun 2010 20:12:58 +0200 From: Bernd Walter To: Pawel Jakub Dawidek Message-ID: <20100615181258.GO87112@cicely7.cicely.de> References: <4C177E69.3020204@gibfest.dk> <20100615155348.GA1922@garage.freebsd.pl> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20100615155348.GA1922@garage.freebsd.pl> X-Operating-System: FreeBSD cicely7.cicely.de 7.0-STABLE i386 User-Agent: Mutt/1.5.11 X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED=-1, BAYES_00=-1.9, T_RP_MATCHES_RCVD=-0.01 autolearn=unavailable version=3.3.0 X-Spam-Checker-Version: SpamAssassin 3.3.0 (2010-01-18) on spamd.cicely.de Cc: freebsd-fs@freebsd.org Subject: Re: ZFS l2arc and HAST ? newbie question X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: ticso@cicely.de List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2010 18:13:13 -0000 On Tue, Jun 15, 2010 at 05:53:48PM +0200, Pawel Jakub Dawidek wrote: > On Tue, Jun 15, 2010 at 03:21:45PM +0200, Thomas Steen Rasmussen wrote: > > Hello list, > > > > I am playing with HAST in order to build some redundant storage > > for a mailserver, using ZFS as the filesystem. > > I have the following zpool layout before stating the HAST experiments: > > > > NAME STATE READ WRITE CKSUM > > tank ONLINE 0 0 0 > > raidz2 ONLINE 0 0 0 > > label/hd4 ONLINE 0 0 0 > > label/hd5 ONLINE 0 0 0 > > label/hd6 ONLINE 0 0 0 > > label/hd7 ONLINE 0 0 0 > > logs ONLINE 0 0 0 > > mirror ONLINE 0 0 0 > > label/ssd0s1 ONLINE 0 0 0 > > label/ssd1s1 ONLINE 0 0 0 > > cache > > label/ssd0s2 ONLINE 0 0 0 > > label/ssd1s2 ONLINE 0 0 0 > > > > As I understand it, to accomplish this with HAST I will need to make a > > HAST resource for each physical disk, like so: > > > > NAME STATE READ WRITE CKSUM > > tank ONLINE 0 0 0 > > raidz2 ONLINE 0 0 0 > > hast/hahd4 ONLINE 0 0 0 > > hast/hahd5 ONLINE 0 0 0 > > hast/hahd6 ONLINE 0 0 0 > > hast/hahd7 ONLINE 0 0 0 > > > > But what about slog and cache devices, currently on SSD disks for > > performance reasons ? It doesn't really make sense to synchronize > > a cache disk over the network, does it ? > > No, it doesn't. Cache is forgotten on import anyway, so don't bother. > You have to be careful, though, because you probably need to remove old > cache device from the pool after import on secondary and add local disk. Unless conditions have changed a missing cache device seem to be without any sign of problem. I have been running with USB-sticks for L2ARC and also tested with missing USB devices at boot time (without reimporting). Even unplugging a device during access wasn't a problem - at least not with ZFS. ZFS didn't even complain when I had a blocked USB and every access to a specific cache device just timed out - in fact I didn't even notice any slowness. Pawel, do you know if there is any chance that ZFS can boot with warm L2ARC? I is mentioned in several articles, but my system starts with empty cache. My system is already quite old, so probably it is already in. > > Could I build the zpool with the SSD disks directly (without > > HAST) and would ZFS survive an export/import on the other host, > > when the cache disks are suddently different ? I am thinking cache > > only here, not slog. > > It simply won't find cache disks, you will need to do what I described > above. > > > Do SSD l2arc / slog even make any sense when I am "deliberately" > > slowing down the filsystem with network redundancy anyway ? > > Forget about HAST for L2ARC. In case of SLOG it can still be faster over > the network than pool with local SATA disks without SLOG. As usual the > best way to verify this is to test it for your workload:) > > > Oh, and is there any problems using labels for HAST devices ? My > > controller likes to give new device names to disks now and then, > > and it has been a blessing to use labels instead of device names, > > so I'd like to continue doing that when using HAST. > > Use labeled providers in hast.conf, but there is no need to label HAST > providers (/dev/hast/). I'm very interested to hear about your results with HAST/ZFS combo because I have a possible use case. -- B.Walter http://www.bwct.de Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm. From owner-freebsd-fs@FreeBSD.ORG Tue Jun 15 18:19:35 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0B81E106566B for ; Tue, 15 Jun 2010 18:19:35 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-iw0-f182.google.com (mail-iw0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id C50F88FC0A for ; Tue, 15 Jun 2010 18:19:34 +0000 (UTC) Received: by iwn7 with SMTP id 7so6781297iwn.13 for ; Tue, 15 Jun 2010 11:19:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type; bh=hsdPYnOnjCHGaJODOKjv46tetGZv1QO3k3DP+8VlBEc=; b=Ur0NT45kMrt2bZoP39fEXho4v5zeg6z8vofZOyvKhOVcFkQgflkGybNkEoAZbirgUY Kwk/iXgEAmYccUIKe6OjD7rJKv/gp7MyWPbNCJn218JgHfskfeoKQp++dxAfj2ByEpal rkPTV5M/STJ433/+Xlp5Z+qAH8+R0HO1Oo3dM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=ihsljqHyzihOKAnqPOYSRJf5jVIbFhGeAPq/oXFo2vTkYpYw7xdjg3+YoSIlDFcSXR MYB5kJeZlbzLie5eIxtPboRoSKzRHcOJFjYjcyjSgn/77WYc6tKHIZbbc4vj2bmlmn43 S6YSLewD9aaYG7rpNR/yBsO6ZqnwgEvZ/TURI= MIME-Version: 1.0 Received: by 10.231.178.162 with SMTP id bm34mr7613833ibb.86.1276625973985; Tue, 15 Jun 2010 11:19:33 -0700 (PDT) Received: by 10.231.16.137 with HTTP; Tue, 15 Jun 2010 11:19:33 -0700 (PDT) In-Reply-To: <20100615181258.GO87112@cicely7.cicely.de> References: <4C177E69.3020204@gibfest.dk> <20100615155348.GA1922@garage.freebsd.pl> <20100615181258.GO87112@cicely7.cicely.de> Date: Tue, 15 Jun 2010 11:19:33 -0700 Message-ID: From: Freddie Cash To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: ZFS l2arc and HAST ? newbie question X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2010 18:19:35 -0000 On Tue, Jun 15, 2010 at 11:12 AM, Bernd Walter wrote: > Pawel, do you know if there is any chance that ZFS can boot with warm > L2ARC? It is mentioned in several articles, but my system starts with empty > cache. My system is already quite old, so probably it is already in. > There's been nothing mentioned about it on the zfs-discuss mailing list, and I haven't seen it blogged about or mentioned in the changelogs for ZFSv14+, so I don't think this has been implemented in ZFS yet. I'm very interested to hear about your results with HAST/ZFS combo > because I have a possible use case. I've only used it in testing via VirtualBox so far. Our intended use is to create a fail-over storage network (1 server at building A, 1 server at building B, connected via gigabit fibre) for a virtualisation setup. In testing inside VirtualBox, it works nicely. Just waiting for our fibre link to go live to test it on real hardware. :) -- Freddie Cash fjwcash@gmail.com From owner-freebsd-fs@FreeBSD.ORG Tue Jun 15 18:31:01 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A34101065670; Tue, 15 Jun 2010 18:31:01 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-pv0-f182.google.com (mail-pv0-f182.google.com [74.125.83.182]) by mx1.freebsd.org (Postfix) with ESMTP id 62E7E8FC15; Tue, 15 Jun 2010 18:31:01 +0000 (UTC) Received: by pvg2 with SMTP id 2so346692pvg.13 for ; Tue, 15 Jun 2010 11:31:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:from:date:to:cc :subject:message-id:reply-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=muwMOg2LVCJEZUkzK+ArnxyisIwx5fRcjcN9HmEx7Jc=; b=djPpnaQIjPG/AaCrcx+bOcLmjqhIZ0ustPfHnEsuzS/qqCCQACnMVFWQwDHqVPlQu/ gbBLOlilmAjXIPz4MEvweJibav8RclLApqsIQr24zU1vBIl4Edt3P4vS7es7i32M2wGo X0q32Cy3wytS0CGxwwbgQRVIR9hdQc2bcM/oo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=huzVZ/Ig6zCa9Oeoa3VZfUU7pMCabqtjpezaSrcCjMg27pCPZRLrTsQdOmB1fkdWOY hLSU449Vesy9lQp+2lF2FxLv/4bWC2d3UeOwhe1hTwFvHdUcxtnGYg3/e+DrykR4DMcc bpDsBbo+gGFDMiS0rERMzabqT0YKUEgZvwUps= Received: by 10.141.4.17 with SMTP id g17mr6039039rvi.166.1276626660787; Tue, 15 Jun 2010 11:31:00 -0700 (PDT) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id b12sm6056157rvn.10.2010.06.15.11.30.59 (version=TLSv1/SSLv3 cipher=RC4-MD5); Tue, 15 Jun 2010 11:31:00 -0700 (PDT) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Tue, 15 Jun 2010 11:30:18 -0700 From: Pyun YongHyeon Date: Tue, 15 Jun 2010 11:30:18 -0700 To: Anders Nordby Message-ID: <20100615183018.GI4257@michelle.cdnetworks.com> References: <20100610081710.GA64350@server.vk2pj.dyndns.org> <20100610110609.GA87243@fupp.net> <20100610114831.GB71432@icarus.home.lan> <20100610130307.GA33285@fupp.net> <20100610133859.GA74094@icarus.home.lan> <20100611031809.GA93666@icarus.home.lan> <20100611163314.GA84574@fupp.net> <20100611175805.GE13776@michelle.cdnetworks.com> <20100611230120.GA89356@fupp.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20100611230120.GA89356@fupp.net> User-Agent: Mutt/1.4.2.3i Cc: freebsd-fs@freebsd.org, Peter Jeremy Subject: Re: Odd network issues on ZFS based NFS server X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2010 18:31:01 -0000 On Sat, Jun 12, 2010 at 01:01:20AM +0200, Anders Nordby wrote: > Hi, > > On Fri, Jun 11, 2010 at 10:58:05AM -0700, Pyun YongHyeon wrote: > >> Let me know if there's anything bge related I can try/test. It might > >> take a day or two or more. Customer is sort of getting annoyed by these > >> problems, so the room for testing is getting smaller. But of course I > >> want to help get a fix for this. > > Show me dmesg output to know which bge(4) controller you had. And > > show me output of "netstat -ndI bge0". Some bge(4) controllers > > supports detailed MAC counters and these are exported via sysctl. > > If your controller is one of these controller, you can check the > > statistics of controller with "sysctl dev.bge.0.stat" and post it > > if you can see them. > > Since running on rue NIC I didn't retry bge again. But I did not reboot > since I had problems last time either, I just changed NIC from bge1 to > ue0. So I'm not sure if these numbers are interesting or if I should > retry using a bge NIC, but here goes: > > anders@unixfile:~$ grep ^bge1 /var/run/dmesg.boot > bge1: mem > 0xfdce0000-0xfdceffff irq 26 at device 1.1 on pci3 > bge1: Ethernet address: 00:16:35:03:e6:3e > bge1: [ITHREAD] > anders@unixfile:~$ netstat -ndI bge1 > Name Mtu Network Address Ipkts Ierrs Idrop > Opkts Oerrs Coll Drop > bge1* 1500 00:16:35:03:e6:3e 21417404 0 0 > 20313076 0 0 0 > anders@unixfile:~$ sysctl dev.bge.1.stats > dev.bge.1.stats.FramesDroppedDueToFilters: 0 > dev.bge.1.stats.DmaWriteQueueFull: 34 > dev.bge.1.stats.DmaWriteHighPriQueueFull: 0 > dev.bge.1.stats.NoMoreRxBDs: 0 > dev.bge.1.stats.InputDiscards: 0 > dev.bge.1.stats.InputErrors: 0 > dev.bge.1.stats.RecvThresholdHit: 12086131 > dev.bge.1.stats.DmaReadQueueFull: 957280 > dev.bge.1.stats.DmaReadHighPriQueueFull: 4835 > dev.bge.1.stats.SendDataCompQueueFull: 0 > dev.bge.1.stats.RingSetSendProdIndex: 20515417 > dev.bge.1.stats.RingStatusUpdate: 20492506 > dev.bge.1.stats.Interrupts: 20492506 > dev.bge.1.stats.AvoidedInterrupts: 0 > dev.bge.1.stats.SendThresholdHit: 0 > dev.bge.1.stats.rx.Octets: 0 > dev.bge.1.stats.rx.Fragments: 0 > dev.bge.1.stats.rx.UcastPkts: 0 This MAC statistics look odd. netstat(1) showed you had 21417404 input packets and 20313076 output packets. But these counters show all 0s. I have no idea how this can happen. Did the controller ever work? If the controller used to work before, I suspect hardware failure(NIC part). > dev.bge.1.stats.rx.MulticastPkts: 0 > dev.bge.1.stats.rx.FCSErrors: 0 > dev.bge.1.stats.rx.AlignmentErrors: 0 > dev.bge.1.stats.rx.xonPauseFramesReceived: 0 > dev.bge.1.stats.rx.xoffPauseFramesReceived: 0 > dev.bge.1.stats.rx.ControlFramesReceived: 0 > dev.bge.1.stats.rx.xoffStateEntered: 0 > dev.bge.1.stats.rx.FramesTooLong: 0 > dev.bge.1.stats.rx.Jabbers: 0 > dev.bge.1.stats.rx.UndersizePkts: 0 > dev.bge.1.stats.rx.inRangeLengthError: 0 > dev.bge.1.stats.rx.outRangeLengthError: 0 > dev.bge.1.stats.tx.Octets: 0 > dev.bge.1.stats.tx.Collisions: 0 > dev.bge.1.stats.tx.XonSent: 0 > dev.bge.1.stats.tx.XoffSent: 0 > dev.bge.1.stats.tx.flowControlDone: 0 > dev.bge.1.stats.tx.InternalMacTransmitErrors: 0 > dev.bge.1.stats.tx.SingleCollisionFrames: 0 > dev.bge.1.stats.tx.MultipleCollisionFrames: 0 > dev.bge.1.stats.tx.DeferredTransmissions: 0 > dev.bge.1.stats.tx.ExcessiveCollisions: 0 > dev.bge.1.stats.tx.LateCollisions: 0 > dev.bge.1.stats.tx.UcastPkts: 0 > dev.bge.1.stats.tx.MulticastPkts: 0 > dev.bge.1.stats.tx.BroadcastPkts: 0 > dev.bge.1.stats.tx.CarrierSenseErrors: 0 > dev.bge.1.stats.tx.Discards: 0 > dev.bge.1.stats.tx.Errors: 0 > > Regards, > > -- > Anders. From owner-freebsd-fs@FreeBSD.ORG Tue Jun 15 20:04:08 2010 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DDC9C106567C for ; Tue, 15 Jun 2010 20:04:08 +0000 (UTC) (envelope-from se@FreeBSD.org) Received: from smtp102.plus.mail.re1.yahoo.com (smtp102.plus.mail.re1.yahoo.com [69.147.102.65]) by mx1.freebsd.org (Postfix) with SMTP id 86F1D8FC17 for ; Tue, 15 Jun 2010 20:04:08 +0000 (UTC) Received: (qmail 17146 invoked from network); 15 Jun 2010 20:04:07 -0000 Received: from [192.168.2.111] (se@188.60.144.187 with plain) by smtp102.plus.mail.re1.yahoo.com with SMTP; 15 Jun 2010 13:04:06 -0700 PDT X-Yahoo-SMTP: iDf2N9.swBDAhYEh7VHfpgq0lnq. X-YMail-OSG: Febbu1IVM1laKRoqB84w8PVq.XKlNeHNULIxkz7avSjLRYn tlOXMOS1PNbBTF1dVSgS35kORkWgG.s9ndjvuGnB2Fm4g2Qee7u9jsR1y0Gi RzqoN385ubOg6rohk5b1iT7JoTpnoChsRhtgNx8VMfjnjVat7x_j7pxjuZbT KQwh5onouEcuSvJgmmtuP707cnG8SigHFBbujCvGDaygUqmp5sw2QN7hjaaE fNuKFeCSIL9T5MYZAczUQE7AKCoowV_Bi_E1sGrgvD3_VrcIOMUeqYpcweYc 0AsbYhVHWDXKBifWDjgtyNzK42nc.DtzSJBbXwwZB57GkDHoWHfvwLVUUGWG _fvayCA-- X-Yahoo-Newman-Property: ymail-3 Message-ID: <4C17DCB1.20700@FreeBSD.org> Date: Tue, 15 Jun 2010 22:04:01 +0200 From: Stefan Esser User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.8.1.18) Gecko/20081105 Thunderbird/2.0.0.18 ThunderBrowse/3.2.2.1 Mnenhy/0.7.5.666 MIME-Version: 1.0 To: Bruce Evans References: <20100609162627.11355zjzwnf7nj8k@webmail.leidinger.net> <4C0FAE2A.7050103@dataix.net> <4C0FB1DE.9080508@dataix.net> <20100610115324.10161biomkjndvy8@webmail.leidinger.net> <20100610173825.164930ekkryr5tes@webmail.leidinger.net> <4C1138D0.7070901@dataix.net> <20100611104219.51344ag1ah7br4kk@webmail.leidinger.net> <4C137ACE.9080900@dataix.net> <20100615095330.89843jrd13kffdkw@webmail.leidinger.net> <4C173693.9020603@FreeBSD.org> <20100615221954.R38827@delplex.bde.org> In-Reply-To: <20100615221954.R38827@delplex.bde.org> X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: 7bit Cc: Alexander Leidinger , fs@freebsd.org Subject: Re: Do we want a periodic script for a zfs scrub? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2010 20:04:09 -0000 Am 15.06.2010 15:10, schrieb Bruce Evans: > On Tue, 15 Jun 2010, Stefan Esser wrote: >> I extended expr and test (standalone binaries and /bin/sh builtins) to >> handle 64 bit on i386 a looong time ago (and I also added overflow >> checks to all calculations). But then it was decided, that POSIX demands >> 32 bit range on 32 bit machines, so support for 64 bit range was made >> conditional on a switch (expr -e if memory serves me right ...). So yes, >> 64 bit range is available on all platforms. > > Hmm, POSIX doesn't require this brokenness for the shell, though it > requires practically equivalent brokenness for all operands and option- > arguments generally: it requires "all" integer arithmetic to use > signed long variables and "all" floating point arithmetic to use double > variables, with Standard C semantics. signed long may have any size, > but normally has 31 or 63 value bits as above. [...] Well, 64 bit arithmetic was generally used in expr for 22 months, (10 July 2000 to 10 May 2002) after I had added support (and range checks, on your advise ;-) ). Then Garrett Wollman switched off 64 bit range by default, to be only optionally enabled by "-e". Then EXPR_COMPAT was made to have the same effect, and a few months later support for configuration of a system default by a symlink "/etc/compat-FreeBSD-4-util -> x,y" where one of the comma separated values that symlink "links to" can be "expr" to enable expr compat mode (and thus -e). Ain't that ugly? I have not checked (and being far away from my FreeBSD systems can not easily check), whether this is still in place in -CURRENT ... Maybe it is time to be tossed out, since FreeBSD-4 compatibility is not a big goal in -CURRENT anymore, I assume ... Anyway, implementation details of expr are far away from the topic of this thread and I'll let the issue rest. But I'll take a note to check whether check_utility_compat(3) really exists in -CURRENT and what use it might have today. It adds startup time to every single invocation of /bin/sh, test (?) and expr and I doubt that a lot of people know this feature exists. Best regards, STefan -------------------------- Excerpt from SVN log for "expr", showing commits that affected the 64 bit numeric range: Revision 106065 - (view) (annotate) - [select for diffs] Modified Mon Oct 28 00:15:43 2002 UTC (7 years, 7 months ago) by wollman File length: 12834 byte(s) Diff to previous 96382 Create a small library function, check_utility_compat(3), to determine whether a named utility should behave in FreeBSD 4.x-compatible mode or in a standard mode (default standard). The configuration is done malloc(3)-style, with either an environment variable or a symlink. Update expr(1) to use this new interface. Revision 96382 - (view) (annotate) - [select for diffs] Modified Sat May 11 03:08:12 2002 UTC (8 years, 1 month ago) by wollman File length: 12797 byte(s) Diff to previous 96367 EXPR_COMPAT should imply -e, since there is no way to specify it otherwise, and -e reflects the historic behavior of FreeBSD's expr. Revision 96367 - (view) (annotate) - [select for diffs] Modified Fri May 10 22:59:29 2002 UTC (8 years, 1 month ago) by wollman File length: 12784 byte(s) Diff to previous 95278 The response to my POSIX interpretation request says that `expr' is required to be oblivious to overflow and to use the data type `long'. (Division by zero is undefined in ISO C so it's still OK to check for it here.) Add a new `-e' flag to get the old, more useful behavior. -------------------------- Revision 62926 - (view) (annotate) - [select for diffs] Modified Mon Jul 10 21:30:55 2000 UTC (9 years, 11 months ago) by se File length: 8761 byte(s) Diff to previous 50471 Extend numeric operations to support 64 bit numbers. From owner-freebsd-fs@FreeBSD.ORG Tue Jun 15 21:16:23 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 56A9E1065678 for ; Tue, 15 Jun 2010 21:16:23 +0000 (UTC) (envelope-from csalgau-br@bitdefender.com) Received: from mail.bitdefender.com (mail.bitdefender.com [91.199.104.2]) by mx1.freebsd.org (Postfix) with ESMTP id 882C38FC14 for ; Tue, 15 Jun 2010 21:16:22 +0000 (UTC) Received: (qmail 3430 invoked from network); 15 Jun 2010 23:49:40 +0300 Received: from 86-124-180-164.iasi.fiberlink.ro (HELO CSalgau3.mshome.net) (csalgau@bitdefender.com@86.124.180.164) by mail.bitdefender.com with AES256-SHA encrypted SMTP; 15 Jun 2010 23:49:39 +0300 Date: Tue, 15 Jun 2010 23:49:37 +0300 From: Mihai-Catalin Salgau Organization: S.C. Bitdefender S.R.L. X-Priority: 3 (Normal) Message-ID: <687948190.20100615234937@bitdefender.com> To: freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BitDefender-Spam: No (0) X-BitDefender-SpamStamp: v1, build 2.8.85.96964, bayes score: 500(0), pbayes score: 241(0), neunet score: 0(0), flags: [NN_LARGISH_BIGGISH2; NN_LEGIT_BITDEFENDER], SQMD: 0d99278ed5a27dbdad297834d9901ea0.fuzzy.fzrbl.org, total: 0(775) X-BitDefender-Scanner: Clean, Agent: BitDefender qmail 3.1.0 on elfie.dsd.hq, sigver: 7.32231 Subject: NTFS streams X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2010 21:16:23 -0000 I'm looking for a way of listing NTFS streams while using=20 /usr/src/sys/fs/ntfs/. The man page for mount_ntfs only says how you can read streams, but=20 not how you can list them. NTFS-3G does have a way of listing them, but relies on extended=20 attributes, but the Linux interface is different from the FreeBSD one and nobody ported it yet. Any help appreciated. Thanks --=20 Mihai-Catalin Salgau From owner-freebsd-fs@FreeBSD.ORG Wed Jun 16 00:21:19 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 912F61065678; Wed, 16 Jun 2010 00:21:19 +0000 (UTC) (envelope-from brucec@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 6859C8FC1C; Wed, 16 Jun 2010 00:21:19 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o5G0LJvC041279; Wed, 16 Jun 2010 00:21:19 GMT (envelope-from brucec@freefall.freebsd.org) Received: (from brucec@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o5G0LJix041275; Wed, 16 Jun 2010 00:21:19 GMT (envelope-from brucec) Date: Wed, 16 Jun 2010 00:21:19 GMT Message-Id: <201006160021.o5G0LJix041275@freefall.freebsd.org> To: brucec@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: brucec@FreeBSD.org Cc: Subject: Re: kern/147890: [regression] ufs-related lock problem in RELENG_8 (18.04.2010 -> 20.04.2010) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jun 2010 00:21:19 -0000 Synopsis: [regression] ufs-related lock problem in RELENG_8 (18.04.2010 -> 20.04.2010) Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: brucec Responsible-Changed-When: Wed Jun 16 00:20:49 UTC 2010 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=147890 From owner-freebsd-fs@FreeBSD.ORG Wed Jun 16 02:34:22 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 67AB01065679 for ; Wed, 16 Jun 2010 02:34:22 +0000 (UTC) (envelope-from spork@bway.net) Received: from xena.bway.net (xena.bway.net [216.220.96.26]) by mx1.freebsd.org (Postfix) with ESMTP id 0B3788FC08 for ; Wed, 16 Jun 2010 02:34:21 +0000 (UTC) Received: (qmail 53023 invoked by uid 0); 16 Jun 2010 02:34:21 -0000 Received: from unknown (HELO ?10.3.2.41?) (spork@96.57.144.66) by smtp.bway.net with (DHE-RSA-AES256-SHA encrypted) SMTP; 16 Jun 2010 02:34:21 -0000 Date: Tue, 15 Jun 2010 22:34:20 -0400 (EDT) From: Charles Sprickman X-X-Sender: spork@hotlap.local To: freebsd-fs@freebsd.org Message-ID: User-Agent: Alpine 2.00 (OSX 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII Subject: zfs panic "evicting znode" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jun 2010 02:34:22 -0000 Howdy, I have a box running 8.0-RELEASE that recently started panicing every few hours with the following message: panic: evicting znode 0xa1eafe80 cpuid = 0 Uptime: 30m56s Physical memory: 2034 MB Dumping 569 MB (counts down to 458, then the box freezes hard) The dump doesn't finish, so there's nothing for savecore to grab. It's a very basic zfs config - two scsi drives in a mirror: [root@h21 /usr/local/etc/pdns]# zpool status pool: zroot state: ONLINE scrub: scrub completed after 0h5m with 0 errors on Tue Jun 15 21:32:11 2010 config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror ONLINE 0 0 0 gpt/disk0 ONLINE 0 0 0 gpt/disk1 ONLINE 0 0 0 errors: No known data errors This is running on an older dual-xeon (1.8GHz/32 bit) supermicro 1U server w/2GB of RAM. The following zfs tunables are set in loader.conf: [root@h21 /usr/local/etc/pdns]# cat /boot/loader.conf zfs_load="YES" vm.kmem_size_max="1000M" vm.kmem_size="1000M" vfs.zfs.arc_max="400M" vfs.root.mountfrom="zfs:zroot" Google is showing me nothing on this panic except for hits on the source code that actuall contains the panic message. Any hints as to what this means? Here's zdb output: [root@h21 /usr/local/adm/bin]# zdb zroot version=13 name='zroot' state=0 txg=267691 pool_guid=12059945251392529754 hostid=1898595607 hostname='h21.biglist.com' vdev_tree type='root' id=0 guid=12059945251392529754 children[0] type='mirror' id=0 guid=14682767316808875040 metaslab_array=23 metaslab_shift=30 ashift=9 asize=142515896320 is_log=0 children[0] type='disk' id=0 guid=11600930948623447097 path='/dev/gpt/disk0' whole_disk=0 children[1] type='disk' id=1 guid=4279842263738814989 path='/dev/gpt/disk1' whole_disk=0 Assertion failed: (rwlp->rw_count == 0), file /usr/src/cddl/lib/libzpool/../../../cddl/contrib/opensolaris/lib/libzpool/common/kernel.c, line 203. Abort trap: 6 (core dumped) Note: the coredump on zdb only occurs if the pool's name is specified, otherwise the output is identical. I'm guessing this is just an error in zdb since it happens on every 8.0 box I've got. Any help is appreciated - we're really looking to get zfs in production, but this problem is a bit odd, and it's not like we can fsck to fix any possible problems with the fs. Thanks, Charles From owner-freebsd-fs@FreeBSD.ORG Wed Jun 16 09:59:36 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E6DB21065678; Wed, 16 Jun 2010 09:59:36 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 6323F8FC14; Wed, 16 Jun 2010 09:59:35 +0000 (UTC) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id o5G9xVMx054844 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 16 Jun 2010 12:59:31 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4) with ESMTP id o5G9xVQH004053; Wed, 16 Jun 2010 12:59:31 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4/Submit) id o5G9xVQO004052; Wed, 16 Jun 2010 12:59:31 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 16 Jun 2010 12:59:31 +0300 From: Kostik Belousov To: brucec@freebsd.org Message-ID: <20100616095931.GP13238@deviant.kiev.zoral.com.ua> References: <201006160021.o5G0LJix041275@freefall.freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201006160021.o5G0LJix041275@freefall.freebsd.org> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-2.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_40, DNS_FROM_OPENWHOIS autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-fs@freebsd.org, freebsd-bugs@freebsd.org Subject: Re: kern/147890: [regression] ufs-related lock problem in RELENG_8 (18.04.2010 -> 20.04.2010) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jun 2010 09:59:37 -0000 Submitter should read http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-deadlocks.html and provide the listed information. From owner-freebsd-fs@FreeBSD.ORG Wed Jun 16 13:10:03 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EC9601065675 for ; Wed, 16 Jun 2010 13:10:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id C2FEB8FC0A for ; Wed, 16 Jun 2010 13:10:03 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o5GDA3mC033153 for ; Wed, 16 Jun 2010 13:10:03 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o5GDA3xe033152; Wed, 16 Jun 2010 13:10:03 GMT (envelope-from gnats) Date: Wed, 16 Jun 2010 13:10:03 GMT Message-Id: <201006161310.o5GDA3xe033152@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Bruce Cran Cc: Subject: Re: kern/147890: [ufs] [regression] ufs-related lock problem in RELENG_8 (18.04.2010 -> 20.04.2010) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Bruce Cran List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jun 2010 13:10:04 -0000 The following reply was made to PR kern/147890; it has been noted by GNATS. From: Bruce Cran To: bug-followup@freebsd.org, lynx.ripe@gmail.com Cc: Subject: Re: kern/147890: [ufs] [regression] ufs-related lock problem in RELENG_8 (18.04.2010 -> 20.04.2010) Date: Wed, 16 Jun 2010 14:03:30 +0100 Could you run the commands shown in http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers- handbook/kerneldebug-deadlocks.html to get more information to help debug the problem please? -- Bruce From owner-freebsd-fs@FreeBSD.ORG Wed Jun 16 13:25:20 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DAB51106564A for ; Wed, 16 Jun 2010 13:25:20 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello089077043238.chello.pl [89.77.43.238]) by mx1.freebsd.org (Postfix) with ESMTP id 70D4E8FC0A for ; Wed, 16 Jun 2010 13:25:20 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id BBF0B45C9F; Wed, 16 Jun 2010 15:25:18 +0200 (CEST) Received: from localhost (pdawidek.wheel.pl [10.0.1.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 8FB1545684; Wed, 16 Jun 2010 15:25:13 +0200 (CEST) Date: Wed, 16 Jun 2010 15:25:04 +0200 From: Pawel Jakub Dawidek To: Charles Sprickman Message-ID: <20100616132504.GK1739@garage.freebsd.pl> References: Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="IJAclU0AInkryoed" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT amd64 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=4.5 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: zfs panic "evicting znode" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jun 2010 13:25:20 -0000 --IJAclU0AInkryoed Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jun 15, 2010 at 10:34:20PM -0400, Charles Sprickman wrote: > Howdy, >=20 > I have a box running 8.0-RELEASE that recently started panicing every few= =20 > hours with the following message: >=20 > panic: evicting znode 0xa1eafe80 Do you have at least backtrace from DDB? What kind of workload do you have? I was no able to find a way to trigger znode_evict_error() call (this is what you're seeing). You could try changing '#if 1' to '#if 0' in zfs_znode.c:znode_evict_error(), but I don't think this code path was ever tested, because as I said, I was unable to trigger it. --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --IJAclU0AInkryoed Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkwY0LAACgkQForvXbEpPzSAmwCgyTDV4q7zAYqQa/DKnekBlbKF AV8AnipSZtaS5hwVyt8sBq1Ms64yMOWh =2OPg -----END PGP SIGNATURE----- --IJAclU0AInkryoed-- From owner-freebsd-fs@FreeBSD.ORG Wed Jun 16 22:20:04 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E34E9106564A for ; Wed, 16 Jun 2010 22:20:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id B6D9F8FC08 for ; Wed, 16 Jun 2010 22:20:03 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o5GMK3xs002901 for ; Wed, 16 Jun 2010 22:20:03 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o5GMK3uM002900; Wed, 16 Jun 2010 22:20:03 GMT (envelope-from gnats) Date: Wed, 16 Jun 2010 22:20:03 GMT Message-Id: <201006162220.o5GMK3uM002900@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Dmitry Pryanishnikov Cc: Subject: Re: kern/147890: [ufs] [regression] ufs-related lock problem in RELENG_8 (18.04.2010 -> 20.04.2010) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Dmitry Pryanishnikov List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jun 2010 22:20:04 -0000 The following reply was made to PR kern/147890; it has been noted by GNATS. From: Dmitry Pryanishnikov To: Bruce Cran Cc: bug-followup@freebsd.org Subject: Re: kern/147890: [ufs] [regression] ufs-related lock problem in RELENG_8 (18.04.2010 -> 20.04.2010) Date: Thu, 17 Jun 2010 01:12:17 +0300 Hello! 2010/6/16 Bruce Cran : > Could you run the commands shown in > http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-deadlocks.html > to get more information to help debug the problem please? Done - I've rebuilt the kernel from the sources as of 20.04.2010 with the missing debugging options, repeated the problem, turned ddb(4) capture buffer on, issued recommended ddb commands, then called 'doadump'; then rebooted and extracted capture buffer contents from the crashdump using ddb(8). BTW, is there a way to get this information _without repeating the problem_ - using only a crashdump from it's previous occurrence? All data required for analysis should already be there. I suspect not every deadlock is easy repeatable (like this one)... db> ps pid ppid pgrp uid state wmesg wchan cmd 2143 1846 2143 0 S+ ufs 0xcd466dc8 sync 2131 1 1911 1000 S ufs 0xcd466dc8 initial thread 1979 1 1911 1000 S select 0xc804fd64 initial thread 1968 1 1918 1000 S select 0xcdb825e4 initial thread 1965 1943 1939 1000 S (threaded) soffice.bin 100193 S select 0xc75c2e64 soffice.bin 100192 S accept 0xcdb93b82 soffice.bin 100191 S ucond 0xcdb83500 soffice.bin 100190 S ucond 0xcdb83540 soffice.bin 100167 S select 0xcdb83264 initial thread 1962 1928 1911 1000 S select 0xc80503a4 initial thread 1961 1928 1911 1000 S select 0xcdb832a4 initial thread 1960 1928 1911 1000 S select 0xc80505e4 initial thread 1953 1 1911 1000 S select 0xc8050da4 xfce4-settings-help 1952 1 1951 1000 S ufs 0xcddc65b8 initial thread 1943 1939 1939 1000 S wait 0xc80802a8 sh 1939 1 1939 1000 Ss wait 0xc80dad48 sh 1935 1 1911 1000 S select 0xcdb82e64 xfsettingsd 1932 1 1911 1000 S (threaded) xfdesktop 100149 S select 0xcdb829e4 initial thread 1930 1 1911 1000 S (threaded) Thunar 100196 S piperd 0xc804d930 Thunar 100128 S select 0xc75cf124 initial thread 1928 1 1911 1000 S select 0xc8050764 initial thread 1926 1 1911 1000 S select 0xc8050164 initial thread 1920 1 1918 1000 S select 0xc8050b24 initial thread 1918 1 1918 1000 Ss select 0xc8050924 dbus-daemon 1917 1 1911 1000 S select 0xc75b8824 dbus-launch 1911 1842 1911 1000 Ss select 0xc75b91a4 initial thread 1862 1828 1862 0 S+ ttyin 0xc7379870 bash 1846 1829 1846 0 S+ wait 0xc7688550 bash 1842 1836 1842 0 Ss wait 0xc781c550 xdm 1839 1836 1839 0 Ss select 0xc80507e4 initial thread 1837 1 1 0 S ttydcd 0xc7390490 getty 1836 1 1 0 S pause 0xc80d7058 xdm 1835 1 1835 0 Ss+ ttyin 0xc71f7a70 getty 1834 1 1834 0 Ss+ ttyin 0xc71f7e70 getty 1833 1 1833 0 Ss+ ttyin 0xc7379070 getty 1832 1 1832 0 Ss+ ttyin 0xc7379470 getty 1831 1 1831 0 Ss+ ttyin 0xc71f7c70 getty 1830 1 1830 0 Ss+ ttyin 0xc7379270 getty 1829 1 1829 0 Ss+ wait 0xc768aaa0 login 1828 1 1828 0 Ss+ wait 0xc80d7d48 login 1802 1 1802 0 Ss select 0xc7394de4 inetd 1770 1 1770 0 Ss select 0xc804f524 moused 1724 1 1724 0 Ss nanslp 0xc07cf224 cron 1717 1 1717 25 Ss pause 0xc7c42058 sendmail 1713 1 1713 0 Ss select 0xc7577ce4 sendmail 1707 1 1707 0 Ss select 0xc75cf8e4 sshd 1621 1 1621 556 Ss select 0xc80501a4 dbus-daemon 1416 1 1416 0 Ss select 0xc75b7324 ntpd 1380 1374 1374 0 S piperd 0xc804d188 logger 1379 1374 1374 0 S select 0xc7577aa4 ssh 1374 1 1374 0 Ss wait 0xc7f6a7f8 sh 1200 1 1200 53 Ss (threaded) named 100140 S kqread 0xc703e380 named 100139 S ucond 0xc75c4e40 named 100138 S ucond 0xc75c4b00 named 100137 S ucond 0xc75990c0 named 100136 S ucond 0xc7394d80 named 100135 S ucond 0xc7381800 named 100121 S sigwait 0xe9633be4 named 1101 1 1101 0 Ss select 0xc75b79e4 syslogd 700 1 700 0 Ss select 0xc75cf2a4 devd 100 1 100 0 Ss pause 0xc71b6850 adjkerntz 20 0 0 0 SL sdflush 0xc0949920 [softdepflush] 19 0 0 0 SL vlruwt 0xc759f550 [vnlru] 18 0 0 0 SL ufs 0xcd466dc8 [syncer] 17 0 0 0 SL psleep 0xc093b348 [bufdaemon] 16 0 0 0 SL pgzero 0xc094a5f4 [pagezero] 15 0 0 0 SL psleep 0xc094a21c [vmdaemon] 14 0 0 0 SL psleep 0xc094a1e4 [pagedaemon] 9 0 0 0 SL ccb_scan 0xc07b8dd4 [xpt_thrd] 8 0 0 0 SL - 0xc71f763c [fdc0] 7 0 0 0 SL - 0xc72a0000 [fw0_probe] 13 0 0 0 SL (threaded) usb 100053 D - 0xc7284d0c [usbus4] 100052 D - 0xc7284cdc [usbus4] 100051 D - 0xc7284cac [usbus4] 100050 D - 0xc7284c7c [usbus4] 100049 D - 0xc726ddac [usbus3] 100048 D - 0xc726dd7c [usbus3] 100047 D - 0xc726dd4c [usbus3] 100046 D - 0xc726dd1c [usbus3] 100045 D - 0xc725cdac [usbus2] 100044 D - 0xc725cd7c [usbus2] 100043 D - 0xc725cd4c [usbus2] 100042 D - 0xc725cd1c [usbus2] 100040 D - 0xc7248dac [usbus1] 100039 D - 0xc7248d7c [usbus1] 100038 D - 0xc7248d4c [usbus1] 100037 D - 0xc7248d1c [usbus1] 100035 D - 0xc721ddac [usbus0] 100034 D - 0xc721dd7c [usbus0] 100033 D - 0xc721dd4c [usbus0] 100032 D - 0xc721dd1c [usbus0] 6 0 0 0 SL crypto_r 0xc094962c [crypto returns] 5 0 0 0 SL crypto_w 0xc0949608 [crypto] 12 0 0 0 SL - 0xc07cf084 [yarrow] 4 0 0 0 SL - 0xc07cce64 [g_down] 3 0 0 0 SL - 0xc07cce60 [g_up] 2 0 0 0 SL - 0xc07cce58 [g_event] 11 0 0 0 WL (threaded) intr 100172 I [irq256: vgapci0] 100060 I [swi0: uart] 100059 I [irq12: psm0] 100058 I [irq1: atkbd0] 100056 I [irq14: ata0] 100041 I [irq18: fwohci0+] 100036 I [irq19: uhci1++] 100031 I [irq23: uhci0 ehci0] 100029 I [irq16: atapci0+] 100028 I [irq257: hdac0] 100027 I [irq9: acpi0] 100025 I [swi6: task queue] 100024 I [swi6: Giant taskq] 100019 I [swi5: +] 100018 I [swi2: cambio] 100011 I [swi4: clock] 100010 I [swi4: clock] 100009 I [swi4: clock] 100008 I [swi4: clock] 100007 I [swi3: vm] 100006 I [swi1: netisr 0] 10 0 0 0 RL (threaded) idle 100005 Run CPU 0 [idle: cpu0] 100004 CanRun [idle: cpu1] 100003 Run CPU 2 [idle: cpu2] 100002 Run CPU 3 [idle: cpu3] 1 0 1 0 SLs wait 0xc7043000 [init] 0 0 0 0 RLs (threaded) kernel 100120 D - 0xc75991c0 [dummynet] 100061 Run CPU 1 [deadlkres] 100054 D - 0xc727ecc0 [fw0_taskq] 100030 D - 0xc7204300 [em0 taskq] 100026 D - 0xc71a4ac0 [kqueue taskq] 100023 D - 0xc71a5040 [acpi_task_2] 100022 D - 0xc71a5040 [acpi_task_1] 100021 D - 0xc71a5040 [acpi_task_0] 100020 D - 0xc71a5080 [thread taskq] 100000 D sched 0xc07ccf00 [swapper] db> trace Tracing pid 0 tid 100061 td 0xc7385000 kdb_enter(c0740afa,c0740afa,c073a07e,c6bcbcd0,1,...) at 0xc053618a = kdb_enter+0x3a panic(c073a07e,c07166d9,c7c47940,3574e,c7c425d8,...) at 0xc05077f6 = panic+0x136 deadlkres(0,c6bcbd38,c073bc0b,343,c07ccf00,...) at 0xc04c4a1a = deadlkres+0x27a fork_exit(c04c47a0,0,c6bcbd38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6bcbd70, ebp = 0 --- db> show pcpu cpuid = 1 dynamic pcpu = 0x60a4f00 curthread = 0xc7385000: pid 0 "deadlkres" curpcb = 0xc6bcbd90 fpcurthread = none idlethread = 0xc70446f0: pid 10 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: db> show allpcpu Current CPU: 1 cpuid = 0 dynamic pcpu = 0x86cf00 curthread = 0xc70444a0: pid 10 "idle: cpu0" curpcb = 0xc6883d90 fpcurthread = none idlethread = 0xc70444a0: pid 10 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: cpuid = 1 dynamic pcpu = 0x60a4f00 curthread = 0xc7385000: pid 0 "deadlkres" curpcb = 0xc6bcbd90 fpcurthread = none idlethread = 0xc70446f0: pid 10 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: cpuid = 2 dynamic pcpu = 0x60a7f00 curthread = 0xc7044940: pid 10 "idle: cpu2" curpcb = 0xc687dd90 fpcurthread = none idlethread = 0xc7044940: pid 10 "idle: cpu2" APIC ID = 2 currentldt = 0x50 spin locks held: cpuid = 3 dynamic pcpu = 0x60aaf00 curthread = 0xc7044b90: pid 10 "idle: cpu3" curpcb = 0xc687ad90 fpcurthread = none idlethread = 0xc7044b90: pid 10 "idle: cpu3" APIC ID = 3 currentldt = 0x50 spin locks held: db> show locks db> show alllocks Process 2131 (thunar-vfs-mime-cle) thread 0xc7c47940 (100108) Process 1952 (gam_server) thread 0xc7385b90 (100092) Process 18 (syncer) thread 0xc7383000 (100067) db> show lockedvnods Locked vnodes db> alltrace Tracing command sync pid 2143 tid 100114 td 0xc7c43940 sched_switch(c7c43940,0,104,191,28b4be6b,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,50,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7c43940,0,c0745097,260,50,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(cd466dc8,50,c07393ba,4,0,...) at 0xc0540f63 = sleepq_wait+0x63 __lockmgr_args(cd466dc8,80100,cd466e30,0,0,...) at 0xc04f087f = __lockmgr_args+0xb1f ffs_lock(e95fbb74,c054944b,c0763314,80100,cd466d70,...) at 0xc067e9f1 = ffs_lock+0xa1 VOP_LOCK1_APV(c07a47c0,e95fbb74,c7c439e4,c07b7a20,cd466d70,...) at 0xc070c945 = VOP_LOCK1_APV+0xb5 _vn_lock(cd466d70,80100,c074e6ce,82b,4,...) at 0xc05a2fd8 = _vn_lock+0x78 vget(cd466d70,80100,c7c43940,39a,c763b870,...) at 0xc05969db = vget+0xbb qsync(c763b870,0,c0761f41,552,4,...) at 0xc0687db7 = qsync+0x197 ffs_sync(c763b870,2,c074fa1e,93,c763b870,...) at 0xc067a139 = ffs_sync+0x349 sync(c7c43940,e95fbcf8,c,c,286,...) at 0xc059d2e5 = sync+0x145 syscall(e95fbd38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (36, FreeBSD ELF32, sync), eip = 0x480d6f5f, esp = 0xbfbfec4c, ebp = 0xbfbfec58 --- Tracing command thunar-vfs-mime-cle pid 2131 tid 100108 td 0xc7c47940 sched_switch(c7c47940,0,104,191,2fe57af9,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,50,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7c47940,0,c0745097,260,50,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(cd466dc8,50,c07393ba,4,0,...) at 0xc0540f63 = sleepq_wait+0x63 __lockmgr_args(cd466dc8,400400,cd466e30,0,0,...) at 0xc04f087f = __lockmgr_args+0xb1f ffs_lock(e95e3878,e95e3878,c04f808c,400400,cd466d70,...) at 0xc067e9f1 = ffs_lock+0xa1 VOP_LOCK1_APV(c07a47c0,e95e3878,e95e3920,c07b7a20,cd466d70,...) at 0xc070c945 = VOP_LOCK1_APV+0xb5 _vn_lock(cd466d70,400400,c076358b,15c,e95e38bc,...) at 0xc05a2fd8 = _vn_lock+0x78 ufs_accessx(e95e3920,c0772d8f,0,1,cd466d70,...) at 0xc068bce4 = ufs_accessx+0xa4 VOP_ACCESSX_APV(c07a47c0,e95e3920,cddc6560,c07b7780,cd466d70,...) at 0xc070abd5 = VOP_ACCESSX_APV+0xc5 ufs_delete_denied(c7c47940,48f4da,80000,e95e39c4,e95e39d4,...) at 0xc06859e4 = ufs_delete_denied+0xb4 ufs_lookup_(0,e95e3a0c,c070b115,e95e3a2c,c0772e9e,...) at 0xc06862f1 = ufs_lookup_+0x811 ufs_lookup(e95e3a2c,c0772e9e,cd466d70,cd466d70,e95e3bf8,...) at 0xc068661e = ufs_lookup+0x1e VOP_CACHEDLOOKUP_APV(c07a47c0,e95e3a2c,e95e3bf8,e95e3be4,caefd180,...) at 0xc070b115 = VOP_CACHEDLOOKUP_APV+0xc5 vfs_cache_lookup(e95e3ab8,c074f40e,c07b7660,200000,e95e3bcc,...) at 0xc0583d66 = vfs_cache_lookup+0xd6 VOP_LOOKUP_APV(c07a47c0,e95e3ab8,c074dc99,30b,e95e3be4,...) at 0xc070dac5 = VOP_LOOKUP_APV+0xe5 lookup(e95e3bcc,c7819000,0,c5,c7c42550,...) at 0xc058ad13 = lookup+0x5c3 namei(e95e3bcc,c7c479e4,c090d068,c07906bc,c076408d,...) at 0xc058ba22 = namei+0x4c2 kern_renameat(c7c47940,ffffff9c,28528280,ffffff9c,2852d3c0,...) at 0xc059aa97 = kern_renameat+0x57 kern_rename(c7c47940,28528280,2852d3c0,0,e95e3d2c,...) at 0xc059ae46 = kern_rename+0x36 rename(c7c47940,e95e3cf8,c0770332,c07484d2,282,...) at 0xc059ae79 = rename+0x29 syscall(e95e3d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (128, FreeBSD ELF32, rename), eip = 0x281bea9b, esp = 0xbfbfe35c, ebp = 0xbfbfebd8 --- Tracing command gqview pid 1979 tid 100180 td 0xc80db000 sched_switch(c80db000,0,104,191,1c15aa7b,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c80db000,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(eb716aac,c04f811a,c804fd50,0,c80db000,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_timedwait_sig(c804fd64,0,eb716adc,101,0,...) at 0xc0540daa = sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c804fd64,c804fd50,bb9,621,eb716b70,...) at 0xc04c5690 = _cv_timedwait_sig+0x250 seltdwait(eb716c60,eb716c68,52f,c80db000,eb716b60,...) at 0xc054b5ba = seltdwait+0x8a poll(c80db000,eb716cf8,c0770332,c0747b95,200282,...) at 0xc054ba80 = poll+0x300 syscall(eb716d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip = 0x28b27b0f, esp = 0xbfbfea9c, ebp = 0xbfbfeab8 --- Tracing command gconfd-2 pid 1968 tid 100147 td 0xc8025940 sched_switch(c8025940,0,104,191,5bd77f79,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c8025940,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(e96a1aac,c04f811a,cdb825d0,0,c8025940,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_timedwait_sig(cdb825e4,0,e96a1adc,101,0,...) at 0xc0540daa = sleepq_timedwait_sig+0x1a _cv_timedwait_sig(cdb825e4,cdb825d0,7531,621,e96a1b90,...) at 0xc04c5690 = _cv_timedwait_sig+0x250 seltdwait(e96a1c60,e96a1c68,52f,c8025940,e96a1b60,...) at 0xc054b5ba = seltdwait+0x8a poll(c8025940,e96a1cf8,c0770332,c0747b95,286,...) at 0xc054ba80 = poll+0x300 syscall(e96a1d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip = 0x28463b0f, esp = 0xbfbfea1c, ebp = 0xbfbfea38 --- Tracing command soffice.bin pid 1965 tid 100193 td 0xcdb9e250 sched_switch(cdb9e250,0,104,191,dbfe1e6b,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(cdb9e250,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c04f811a,c75c2e50,0,c073f15b,cdb9e250,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c75c2e64,0,eb73dadc,101,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _cv_wait_sig(c75c2e64,c75c2e50,c074742e,621,eb73db68,...) at 0xc04c5ce0 = _cv_wait_sig+0x240 seltdwait(c7fc202c,c074742e,52f,cdb9e250,eb73db60,...) at 0xc054b5d2 = seltdwait+0xa2 poll(cdb9e250,eb73dcf8,c0770332,c072d816,286,...) at 0xc054ba80 = poll+0x300 syscall(eb73dd38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip = 0x2a8a1b0f, esp = 0xbf6fbebc, ebp = 0xbf6fbed8 --- Tracing command soffice.bin pid 1965 tid 100192 td 0xcdb9e4a0 sched_switch(cdb9e4a0,0,104,191,c99bad73,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,58,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(cdb9e4a0,0,c0745097,1a0,58,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(cdb93b82,58,c0747ac1,100,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _sleep(cdb93b82,c093b1b0,158,c0747ac1,0,...) at 0xc0510b84 = _sleep+0x354 kern_accept(cdb9e4a0,8,0,0,0,...) at 0xc05751f0 = kern_accept+0x120 accept(cdb9e4a0,eb73acf8,c0770332,c0748721,286,...) at 0xc0575661 = accept+0x41 syscall(eb73ad38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (30, FreeBSD ELF32, accept), eip = 0x2a89209f, esp = 0xbf7fc52c, ebp = 0xbf7fc548 --- Tracing command soffice.bin pid 1965 tid 100191 td 0xcdb9e6f0 sched_switch(cdb9e6f0,0,104,191,b926aaea,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(cdb9e6f0,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(d6f06,c053fd30,cdb9e6f0,0,100,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_timedwait_sig(cdb83500,0,c074235e,100,0,...) at 0xc0540daa = sleepq_timedwait_sig+0x1a _sleep(cdb83500,c07d2a44,100,c074235e,d6f06,...) at 0xc0510b4e = _sleep+0x31e __umtx_op_cv_wait(cdb9e6f0,eb737cf8,eb737d2c,c06f41e5,cdb9e6f0,...) at 0xc051ff78 = __umtx_op_cv_wait+0x558 _umtx_op(cdb9e6f0,eb737cf8,c0770332,c0747f0c,202,...) at 0xc051b0c7 = _umtx_op+0x27 syscall(eb737d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x2a808227, esp = 0xbf8fddbc, ebp = 0xbf8fdde8 --- Tracing command soffice.bin pid 1965 tid 100190 td 0xcdb9e940 sched_switch(cdb9e940,0,104,191,646d55a2,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(cdb9e940,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(2711,c053fd30,cdb9e940,1,100,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_timedwait_sig(cdb83540,0,c074235e,100,0,...) at 0xc0540daa = sleepq_timedwait_sig+0x1a _sleep(cdb83540,c07d2084,100,c074235e,2711,...) at 0xc0510b4e = _sleep+0x31e __umtx_op_cv_wait(cdb9e940,eb734cf8,eb734d2c,c06f41e5,cdb9e940,...) at 0xc051ff78 = __umtx_op_cv_wait+0x558 _umtx_op(cdb9e940,eb734cf8,c0770332,c0747f0c,202,...) at 0xc051b0c7 = _umtx_op+0x27 syscall(eb734d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x2a808227, esp = 0xbf9feeac, ebp = 0xbf9feed8 --- Tracing command soffice.bin pid 1965 tid 100167 td 0xc8027940 sched_switch(c8027940,0,104,191,ba406799,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c8027940,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(e96e7aac,c04f811a,cdb83250,0,c8027940,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_timedwait_sig(cdb83264,0,e96e7adc,101,0,...) at 0xc0540daa = sleepq_timedwait_sig+0x1a _cv_timedwait_sig(cdb83264,cdb83250,a7ade,621,e96e7b88,...) at 0xc04c5690 = _cv_timedwait_sig+0x250 seltdwait(e96e7c60,e96e7c68,52f,c8027940,e96e7b60,...) at 0xc054b5ba = seltdwait+0x8a poll(c8027940,e96e7cf8,c0770332,c0747b95,200286,...) at 0xc054ba80 = poll+0x300 syscall(e96e7d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip = 0x2a8a1b0f, esp = 0xbfbfe4bc, ebp = 0xbfbfe4d8 --- Tracing command xfce4-xkb-plugin pid 1962 tid 100156 td 0xc80836f0 sched_switch(c80836f0,0,104,191,5b37d11a,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c80836f0,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c04f811a,c8050390,0,c073f15b,c80836f0,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c80503a4,0,e96c6adc,101,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _cv_wait_sig(c80503a4,c8050390,c074742e,621,e96c6b70,...) at 0xc04c5ce0 = _cv_wait_sig+0x240 seltdwait(c75bea2c,c074742e,52f,c80836f0,e96c6b60,...) at 0xc054b5d2 = seltdwait+0xa2 poll(c80836f0,e96c6cf8,c0770332,c072d816,282,...) at 0xc054ba80 = poll+0x300 syscall(e96c6d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip = 0x28ad3b0f, esp = 0xbfbfe99c, ebp = 0xbfbfe9b8 --- Tracing command xfce4-mixer-plugin pid 1961 tid 100173 td 0xc80dd250 sched_switch(c80dd250,0,104,191,413877a,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c80dd250,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c04f811a,cdb83290,0,c073f15b,c80dd250,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(cdb832a4,0,eb701adc,101,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _cv_wait_sig(cdb832a4,cdb83290,c074742e,621,eb701b68,...) at 0xc04c5ce0 = _cv_wait_sig+0x240 seltdwait(c758352c,c074742e,52f,c80dd250,eb701b60,...) at 0xc054b5d2 = seltdwait+0xa2 poll(c80dd250,eb701cf8,c0770332,c072d816,282,...) at 0xc054ba80 = poll+0x300 syscall(eb701d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip = 0x2900cb0f, esp = 0xbfbfe99c, ebp = 0xbfbfe9b8 --- Tracing command xfce4-menu-plugin pid 1960 tid 100104 td 0xc781e250 sched_switch(c781e250,0,104,191,4d4ebc5e,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c781e250,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c04f811a,c80505d0,0,c073f15b,c781e250,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c80505e4,0,e95abadc,101,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _cv_wait_sig(c80505e4,c80505d0,c074742e,621,e95abb70,...) at 0xc04c5ce0 = _cv_wait_sig+0x240 seltdwait(c75c522c,c074742e,52f,c781e250,e95abb60,...) at 0xc054b5d2 = seltdwait+0xa2 poll(c781e250,e95abcf8,c0770332,c072d816,282,...) at 0xc054ba80 = poll+0x300 syscall(e95abd38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip = 0x28de2b0f, esp = 0xbfbfe9ac, ebp = 0xbfbfe9c8 --- Tracing command xfce4-settings-help pid 1953 tid 100158 td 0xc8083250 sched_switch(c8083250,0,104,191,5b3922a3,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c8083250,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c04f811a,c8050d90,0,c073f15b,c8083250,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c8050da4,0,e96ccadc,101,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _cv_wait_sig(c8050da4,c8050d90,c074742e,621,e96ccb80,...) at 0xc04c5ce0 = _cv_wait_sig+0x240 seltdwait(c7fc562c,c074742e,52f,c8083250,e96ccb60,...) at 0xc054b5d2 = seltdwait+0xa2 poll(c8083250,e96cccf8,c0770332,c072d816,282,...) at 0xc054ba80 = poll+0x300 syscall(e96ccd38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip = 0x288b3b0f, esp = 0xbfbfea3c, ebp = 0xbfbfea58 --- Tracing command gam_server pid 1952 tid 100092 td 0xc7385b90 sched_switch(c7385b90,0,104,191,2fe5a1ff,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,50,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7385b90,0,c0745097,260,cddc65b8,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(cddc65b8,50,c07393ba,4,1,...) at 0xc0540f63 = sleepq_wait+0x63 __lockmgr_args(cddc65b8,200100,cddc6620,0,0,...) at 0xc04f0215 = __lockmgr_args+0x4b5 ffs_lock(e957a7e0,c054944b,c074db65,200100,cddc6560,...) at 0xc067e9f1 = ffs_lock+0xa1 VOP_LOCK1_APV(c07a47c0,e957a7e0,c7385c34,c07b7a20,cddc6560,...) at 0xc070c945 = VOP_LOCK1_APV+0xb5 _vn_lock(cddc6560,200100,c074e6ce,82b,4,...) at 0xc05a2fd8 = _vn_lock+0x78 vget(cddc6560,200100,c7385b90,50,0,...) at 0xc05969db = vget+0xbb vfs_hash_get(c763b870,48f4da,200000,c7385b90,e957a960,...) at 0xc0589b5d = vfs_hash_get+0xed ffs_vgetf(c763b870,48f4da,200000,e957a960,0,...) at 0xc0679149 = ffs_vgetf+0x49 ffs_vget(c763b870,48f4da,200000,e957a960,e957a970,...) at 0xc067968e = ffs_vget+0x2e ufs_lookup_(0,e957a9a8,c070b115,e957a9c8,c0772e9e,...) at 0xc06865a9 = ufs_lookup_+0xac9 ufs_lookup(e957a9c8,c0772e9e,cd466d70,cd466d70,e957ab8c,...) at 0xc068661e = ufs_lookup+0x1e VOP_CACHEDLOOKUP_APV(c07a47c0,e957a9c8,e957ab8c,e957ab78,caefd180,...) at 0xc070b115 = VOP_CACHEDLOOKUP_APV+0xc5 vfs_cache_lookup(e957aa54,c074f40e,c07b7660,200000,e957ab60,...) at 0xc0583d66 = vfs_cache_lookup+0xd6 VOP_LOOKUP_APV(c07a47c0,e957aa54,c074dc99,30b,e957ab78,...) at 0xc070dac5 = VOP_LOOKUP_APV+0xe5 lookup(e957ab60,c7692c00,0,c5,c7688000,...) at 0xc058ad13 = lookup+0x5c3 namei(e957ab60,c802a5a0,c757ac00,e957ac14,c04d8bde,...) at 0xc058ba22 = namei+0x4c2 kern_statat_vnhook(c7385b90,0,ffffff9c,2846ad00,0,...) at 0xc059b3f2 = kern_statat_vnhook+0x72 kern_statat(c7385b90,0,ffffff9c,2846ad00,0,...) at 0xc059b54c = kern_statat+0x3c kern_stat(c7385b90,2846ad00,0,e957ac1c,c7385b90,...) at 0xc059b696 = kern_stat+0x36 stat(c7385b90,e957acf8,c0770332,c07483c6,246,...) at 0xc059b73f = stat+0x2f syscall(e957ad38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (188, FreeBSD ELF32, stat), eip = 0x28386a23, esp = 0xbfbfe98c, ebp = 0xbfbfea28 --- Tracing command sh pid 1943 tid 100148 td 0xc80256f0 sched_switch(c80256f0,0,104,191,e36a908,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,5c,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c80256f0,0,c0745097,1a0,5c,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c80802a8,5c,c0747a51,100,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _sleep(c80802a8,c8080330,15c,c0747a51,0,...) at 0xc0510b84 = _sleep+0x354 kern_wait(c80256f0,ffffffff,e96a4c78,2,0,...) at 0xc04dbde6 = kern_wait+0xb46 wait4(c80256f0,e96a4cf8,c0770332,c07482ab,216,...) at 0xc04dbe5b = wait4+0x3b syscall(e96a4d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2815dbdb, esp = 0xbfbfe9ec, ebp = 0xbfbfea08 --- Tracing command sh pid 1939 tid 100184 td 0xc80844a0 sched_switch(c80844a0,0,104,191,c21d95d3,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,5c,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c80844a0,0,c0745097,1a0,5c,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c80dad48,5c,c0747a51,100,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _sleep(c80dad48,c80dadd0,15c,c0747a51,0,...) at 0xc0510b84 = _sleep+0x354 kern_wait(c80844a0,ffffffff,eb722c78,2,0,...) at 0xc04dbde6 = kern_wait+0xb46 wait4(c80844a0,eb722cf8,c0770332,c0747a30,216,...) at 0xc04dbe5b = wait4+0x3b syscall(eb722d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2815dbdb, esp = 0xbfbfe99c, ebp = 0xbfbfe9b8 --- Tracing command xfsettingsd pid 1935 tid 100075 td 0xc7681250 sched_switch(c7681250,0,104,191,b7150b60,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7681250,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c04f811a,cdb82e50,0,c073f15b,c7681250,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(cdb82e64,0,e9536adc,101,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _cv_wait_sig(cdb82e64,cdb82e50,c074742e,621,e9536b70,...) at 0xc04c5ce0 = _cv_wait_sig+0x240 seltdwait(c7fc492c,c074742e,52f,c7681250,e9536b60,...) at 0xc054b5d2 = seltdwait+0xa2 poll(c7681250,e9536cf8,c0770332,c072d816,282,...) at 0xc054ba80 = poll+0x300 syscall(e9536d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip = 0x287c0b0f, esp = 0xbfbfeacc, ebp = 0xbfbfeae8 --- Tracing command xfdesktop pid 1932 tid 100149 td 0xc80254a0 sched_switch(c80254a0,0,104,191,3355019d,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c80254a0,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(e96a7aac,c04f811a,cdb829d0,0,c80254a0,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_timedwait_sig(cdb829e4,0,e96a7adc,101,0,...) at 0xc0540daa = sleepq_timedwait_sig+0x1a _cv_timedwait_sig(cdb829e4,cdb829d0,3e9,621,e96a7b88,...) at 0xc04c5690 = _cv_timedwait_sig+0x250 seltdwait(e96a7c60,e96a7c68,52f,c80254a0,e96a7b60,...) at 0xc054b5ba = seltdwait+0x8a poll(c80254a0,e96a7cf8,c0770332,c0747b95,286,...) at 0xc054ba80 = poll+0x300 syscall(e96a7d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip = 0x28eb2b0f, esp = 0xbfbfe6bc, ebp = 0xbfbfe6d8 --- Tracing command Thunar pid 1930 tid 100196 td 0xc781f6f0 sched_switch(c781f6f0,0,104,191,2c27b007,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,4c,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c781f6f0,0,c0745097,1a0,4c,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c804d930,4c,c0747637,100,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _sleep(c804d930,c804daa0,14c,c0747637,0,...) at 0xc0510b84 = _sleep+0x354 pipe_read(c76f4690,eb746c5c,caefd180,0,c781f6f0,...) at 0xc054efa7 = pipe_read+0x3f7 dofileread(eb746c5c,ffffffff,ffffffff,0,c76f4690,...) at 0xc054bf26 = dofileread+0x96 kern_readv(c781f6f0,d,eb746c5c,eb746c7c,1,...) at 0xc054c258 = kern_readv+0x58 read(c781f6f0,eb746cf8,c,c,200286,...) at 0xc054c36f = read+0x4f syscall(eb746d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x28a68593, esp = 0xbf5faeec, ebp = 0xbf5faf08 --- Tracing command Thunar pid 1930 tid 100128 td 0xc7f70940 sched_switch(c7f70940,0,104,191,6c161109,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7f70940,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(e964faac,c04f811a,c75cf110,0,c7f70940,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_timedwait_sig(c75cf124,0,e964fadc,101,0,...) at 0xc0540daa = sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c75cf124,c75cf110,3e9,621,e964fb80,...) at 0xc04c5690 = _cv_timedwait_sig+0x250 seltdwait(e964fc60,e964fc68,52f,c7f70940,e964fb60,...) at 0xc054b5ba = seltdwait+0x8a poll(c7f70940,e964fcf8,c0770332,c0747b95,200286,...) at 0xc054ba80 = poll+0x300 syscall(e964fd38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip = 0x28a0eb0f, esp = 0xbfbfeadc, ebp = 0xbfbfeaf8 --- Tracing command xfce4-panel pid 1928 tid 100159 td 0xc8083000 sched_switch(c8083000,0,104,191,5b37bb60,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c8083000,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(e96cfaac,c04f811a,c8050750,0,c8083000,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_timedwait_sig(c8050764,0,e96cfadc,101,0,...) at 0xc0540daa = sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c8050764,c8050750,3db,621,e96cfb78,...) at 0xc04c5690 = _cv_timedwait_sig+0x250 seltdwait(e96cfc60,e96cfc68,52f,c8083000,e96cfb60,...) at 0xc054b5ba = seltdwait+0x8a poll(c8083000,e96cfcf8,c0770332,c0747b95,200282,...) at 0xc054ba80 = poll+0x300 syscall(e96cfd38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip = 0x28d53b0f, esp = 0xbfbfea3c, ebp = 0xbfbfea58 --- Tracing command xfwm4 pid 1926 tid 100071 td 0xc7681b90 sched_switch(c7681b90,0,104,191,5b378290,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7681b90,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c04f811a,c8050150,0,c073f15b,c7681b90,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c8050164,0,e9525adc,101,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _cv_wait_sig(c8050164,c8050150,c074742e,621,e9525b78,...) at 0xc04c5ce0 = _cv_wait_sig+0x240 seltdwait(c75c592c,c074742e,52f,c7681b90,e9525b60,...) at 0xc054b5d2 = seltdwait+0xa2 poll(c7681b90,e9525cf8,c0770332,c072d816,282,...) at 0xc054ba80 = poll+0x300 syscall(e9525d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip = 0x28de9b0f, esp = 0xbfbfe6ac, ebp = 0xbfbfe6c8 --- Tracing command xfconfd pid 1920 tid 100141 td 0xc8026940 sched_switch(c8026940,0,104,191,40a1f3b2,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c8026940,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c04f811a,c8050b10,0,c073f15b,c8026940,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c8050b24,0,e968fadc,101,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _cv_wait_sig(c8050b24,c8050b10,c074742e,621,e968fb70,...) at 0xc04c5ce0 = _cv_wait_sig+0x240 seltdwait(c75c5e2c,c074742e,52f,c8026940,e968fb60,...) at 0xc054b5d2 = seltdwait+0xa2 poll(c8026940,e968fcf8,c0770332,c07737a6,286,...) at 0xc054ba80 = poll+0x300 syscall(e968fd38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip = 0x283e8b0f, esp = 0xbfbfe9ec, ebp = 0xbfbfea08 --- Tracing command dbus-daemon pid 1918 tid 100072 td 0xc7681940 sched_switch(c7681940,0,104,191,b454ff12,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7681940,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c04f811a,c8050910,0,c073f15b,c7681940,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c8050924,0,e9529adc,101,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _cv_wait_sig(c8050924,c8050910,c074742e,621,e9529bb8,...) at 0xc04c5ce0 = _cv_wait_sig+0x240 seltdwait(c808962c,c074742e,52f,c7681940,e9529b60,...) at 0xc054b5d2 = seltdwait+0xa2 poll(c7681940,e9529cf8,c0770332,c0747f0c,282,...) at 0xc054ba80 = poll+0x300 syscall(e9529d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip = 0x281a3b0f, esp = 0xbfbfe7ac, ebp = 0xbfbfe7c8 --- Tracing command dbus-launch pid 1917 tid 100183 td 0xc80846f0 sched_switch(c80846f0,0,104,191,3a99ea08,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c80846f0,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c04f811a,c75b8810,0,c073f15b,c80846f0,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c75b8824,0,eb71fa80,101,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _cv_wait_sig(c75b8824,c75b8810,c074742e,621,c76915e8,...) at 0xc04c5ce0 = _cv_wait_sig+0x240 seltdwait(c76915e8,d8,caefd180,c80846f0,c074b1b4,...) at 0xc054b5d2 = seltdwait+0xa2 kern_select(c80846f0,6,bfbfeac8,0,bfbfe8c8,0,20,bfbfe600,cdb76000) at 0xc054c884 = kern_select+0x504 select(c80846f0,eb71fcf8,c0770332,c072d816,282,...) at 0xc054cbd6 = select+0x66 syscall(eb71fd38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x282f6513, esp = 0xbfbfe5fc, ebp = 0xbfbfe628 --- Tracing command xfce4-session pid 1911 tid 100179 td 0xc80db250 sched_switch(c80db250,0,104,191,b715754d,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c80db250,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c04f811a,c75b9190,0,c073f15b,c80db250,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c75b91a4,0,eb713adc,101,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _cv_wait_sig(c75b91a4,c75b9190,c074742e,621,eb713ba8,...) at 0xc04c5ce0 = _cv_wait_sig+0x240 seltdwait(c783272c,c074742e,52f,c80db250,eb713b60,...) at 0xc054b5d2 = seltdwait+0xa2 poll(c80db250,eb713cf8,c0770332,c072d816,286,...) at 0xc054ba80 = poll+0x300 syscall(eb713d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip = 0x28df6b0f, esp = 0xbfbfeadc, ebp = 0xbfbfeaf8 --- Tracing command bash pid 1862 tid 100091 td 0xc7680000 sched_switch(c7680000,0,104,191,eb8a2f02,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7680000,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c0745097,160,0,c7680000,c7680000,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c7379870,0,c0749892,101,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _cv_wait_sig(c7379870,c07ce9f0,c07491e9,4fd,0,...) at 0xc04c5ce0 = _cv_wait_sig+0x240 tty_wait(c7379800,c7379870,e9576c5c,1,0,...) at 0xc0559721 = tty_wait+0x71 ttydisc_read(c7379800,e9576c5c,0,9e,0,...) at 0xc0560e8c = ttydisc_read+0x22c ttydev_read(c738fa00,e9576c5c,0,0,1,...) at 0xc055b0ca = ttydev_read+0xaa devfs_read_f(c7f4fa10,e9576c5c,caefd780,0,c7680000,...) at 0xc04a6c1e = devfs_read_f+0x7e dofileread(e9576c5c,ffffffff,ffffffff,0,c7f4fa10,...) at 0xc054bf26 = dofileread+0x96 kern_readv(c7680000,0,e9576c5c,e9576c7c,1,...) at 0xc054c258 = kern_readv+0x58 read(c7680000,e9576cf8,c0770332,c074792b,246,...) at 0xc054c36f = read+0x4f syscall(e9576d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x8127ca3, esp = 0xbfbfe08c, ebp = 0xbfbfe0b8 --- Tracing command bash pid 1846 tid 100088 td 0xc7680250 sched_switch(c7680250,0,104,191,1efe8d41,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,5c,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7680250,0,c0745097,1a0,5c,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c7688550,5c,c0747a51,100,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _sleep(c7688550,c76885d8,15c,c0747a51,0,...) at 0xc0510b84 = _sleep+0x354 kern_wait(c7680250,ffffffff,e956ac78,6,0,...) at 0xc04dbde6 = kern_wait+0xb46 wait4(c7680250,e956acf8,c0770332,c074792b,286,...) at 0xc04dbe5b = wait4+0x3b syscall(e956ad38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x80e26df, esp = 0xbfbfea2c, ebp = 0xbfbfea48 --- Tracing command xdm pid 1842 tid 100095 td 0xc76bd6f0 sched_switch(c76bd6f0,0,104,191,cfae5dd4,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,5c,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c76bd6f0,0,c0745097,1a0,5c,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c781c550,5c,c0747a51,100,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _sleep(c781c550,c781c5d8,15c,c0747a51,0,...) at 0xc0510b84 = _sleep+0x354 kern_wait(c76bd6f0,ffffffff,e9590c78,0,0,...) at 0xc04dbde6 = kern_wait+0xb46 wait4(c76bd6f0,e9590cf8,c0770332,c074792b,282,...) at 0xc04dbe5b = wait4+0x3b syscall(e9590d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2831fbdb, esp = 0xbfbfe8cc, ebp = 0xbfbfe8e8 --- Tracing command Xorg pid 1839 tid 100081 td 0xc7680940 sched_switch(c7680940,0,104,191,5b384353,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7680940,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(e954ea50,c04f811a,c80507d0,0,c7680940,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_timedwait_sig(c80507e4,0,e954ea80,101,0,...) at 0xc0540daa = sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c80507e4,c80507d0,3b737,621,c8085038,...) at 0xc04c5690 = _cv_timedwait_sig+0x250 seltdwait(e954ec2c,e954ec34,c7fcb900,c7680940,c074b1b4,...) at 0xc054b5ba = seltdwait+0x8a kern_select(c7680940,1c,81e8a00,0,0,e954ec74,20,f3,7c830) at 0xc054c884 = kern_select+0x504 select(c7680940,e954ecf8,c0770332,c0747a40,200246,...) at 0xc054cbd6 = select+0x66 syscall(e954ed38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x285ad513, esp = 0xbfbfeb0c, ebp = 0xbfbfed78 --- Tracing command getty pid 1837 tid 100160 td 0xc8082b90 sched_switch(c8082b90,0,104,191,1cef9f31,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c8082b90,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c04f811a,c7390404,0,c073f15b,c8082b90,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c7390490,0,e96d296c,101,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _cv_wait_sig(c7390490,c7390404,c07491e9,4fd,c7390400,...) at 0xc04c5ce0 = _cv_wait_sig+0x240 tty_wait(c7390400,c7390490,0,4e,caf0d560,...) at 0xc0559721 = tty_wait+0x71 ttydev_open(c738e600,3,2000,c8082b90,e96d2a08,...) at 0xc055b4a2 = ttydev_open+0x372 devfs_open(e96d2ac0,c0772e58,caf0d560,0,e96d2ba8,...) at 0xc04a8768 = devfs_open+0xe8 VOP_OPEN_APV(c07862c0,e96d2ac0,e96d2a4c,c054944b,0,...) at 0xc070afc5 = VOP_OPEN_APV+0xc5 vn_open_cred(e96d2ba8,e96d2c60,1,0,c7041400,...) at 0xc05a29a6 = vn_open_cred+0x456 vn_open(e96d2ba8,e96d2c60,1,c76f4000,c054944b,...) at 0xc05a2acb = vn_open+0x3b kern_openat(c8082b90,ffffff9c,804fbc0,0,3,...) at 0xc05a1947 = kern_openat+0xe7 kern_open(c8082b90,804fbc0,0,2,3,...) at 0xc05a1e45 = kern_open+0x35 open(c8082b90,e96d2cf8,c,c,282,...) at 0xc05a1e80 = open+0x30 syscall(e96d2d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (5, FreeBSD ELF32, open), eip = 0x2817ea83, esp = 0xbfbfedac, ebp = 0xbfbfedd8 --- Tracing command xdm pid 1836 tid 100168 td 0xc80ddb90 sched_switch(c80ddb90,0,104,191,c8a6f9b1,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,68,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c80ddb90,0,c0745097,1a0,68,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c80d7058,68,c0740d48,100,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _sleep(c80d7058,c80d7088,168,c0740d48,0,...) at 0xc0510b84 = _sleep+0x354 kern_sigsuspend(c80ddb90,0,0,0,0,...) at 0xc050b7ae = kern_sigsuspend+0xae sigsuspend(c80ddb90,e96eacf8,c0770332,c074792b,246,...) at 0xc050b8bd = sigsuspend+0x4d syscall(e96ead38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (4, FreeBSD ELF32, write), eip = 0x28307d9b, esp = 0xbfbfe85c, ebp = 0xbfbfe918 --- Tracing command getty pid 1835 tid 100078 td 0xc76c1250 sched_switch(c76c1250,0,104,191,1cef0682,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c76c1250,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c0745097,160,0,c76c1250,c76c1250,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c71f7a70,0,c0749892,101,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _cv_wait_sig(c71f7a70,c07ce9f0,c07491e9,4fd,0,...) at 0xc04c5ce0 = _cv_wait_sig+0x240 tty_wait(c71f7a00,c71f7a70,e9542c5c,1,0,...) at 0xc0559721 = tty_wait+0x71 ttydisc_read(c71f7a00,e9542c5c,0,9e,0,...) at 0xc0560e8c = ttydisc_read+0x22c ttydev_read(c738f300,e9542c5c,0,0,1,...) at 0xc055b0ca = ttydev_read+0xaa devfs_read_f(c7691428,e9542c5c,c7041400,0,c76c1250,...) at 0xc04a6c1e = devfs_read_f+0x7e dofileread(e9542c5c,ffffffff,ffffffff,0,c7691428,...) at 0xc054bf26 = dofileread+0x96 kern_readv(c76c1250,0,e9542c5c,e9542c7c,1,...) at 0xc054c258 = kern_readv+0x58 read(c76c1250,e9542cf8,c0770332,c0752f28,246,...) at 0xc054c36f = read+0x4f syscall(e9542d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f593, esp = 0xbfbfedbc, ebp = 0xbfbfede8 --- Tracing command getty pid 1834 tid 100085 td 0xc76c06f0 sched_switch(c76c06f0,0,104,191,1d0ddcfc,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c76c06f0,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c0745097,160,0,c76c06f0,c76c06f0,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c71f7e70,0,c0749892,101,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _cv_wait_sig(c71f7e70,c07ce9f0,c07491e9,4fd,0,...) at 0xc04c5ce0 = _cv_wait_sig+0x240 tty_wait(c71f7e00,c71f7e70,e955ec5c,1,0,...) at 0xc0559721 = tty_wait+0x71 ttydisc_read(c71f7e00,e955ec5c,0,9e,0,...) at 0xc0560e8c = ttydisc_read+0x22c ttydev_read(c738f400,e955ec5c,0,0,1,...) at 0xc055b0ca = ttydev_read+0xaa devfs_read_f(c767b5b0,e955ec5c,c7041400,0,c76c06f0,...) at 0xc04a6c1e = devfs_read_f+0x7e dofileread(e955ec5c,ffffffff,ffffffff,0,c767b5b0,...) at 0xc054bf26 = dofileread+0x96 kern_readv(c76c06f0,0,e955ec5c,e955ec7c,1,...) at 0xc054c258 = kern_readv+0x58 read(c76c06f0,e955ecf8,c0770332,c0752f28,246,...) at 0xc054c36f = read+0x4f syscall(e955ed38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f593, esp = 0xbfbfedbc, ebp = 0xbfbfede8 --- Tracing command getty pid 1833 tid 100089 td 0xc76c0250 sched_switch(c76c0250,0,104,191,1ce387c1,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c76c0250,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c0745097,160,0,c76c0250,c76c0250,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c7379070,0,c0749892,101,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _cv_wait_sig(c7379070,c07ce9f0,c07491e9,4fd,0,...) at 0xc04c5ce0 = _cv_wait_sig+0x240 tty_wait(c7379000,c7379070,e956ec5c,1,0,...) at 0xc0559721 = tty_wait+0x71 ttydisc_read(c7379000,e956ec5c,0,9e,0,...) at 0xc0560e8c = ttydisc_read+0x22c ttydev_read(c738f500,e956ec5c,0,0,1,...) at 0xc055b0ca = ttydev_read+0xaa devfs_read_f(c8085930,e956ec5c,c7041400,0,c76c0250,...) at 0xc04a6c1e = devfs_read_f+0x7e dofileread(e956ec5c,ffffffff,ffffffff,0,c8085930,...) at 0xc054bf26 = dofileread+0x96 kern_readv(c76c0250,0,e956ec5c,e956ec7c,1,...) at 0xc054c258 = kern_readv+0x58 read(c76c0250,e956ecf8,c0770332,c0752f28,246,...) at 0xc054c36f = read+0x4f syscall(e956ed38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f593, esp = 0xbfbfedbc, ebp = 0xbfbfede8 --- Tracing command getty pid 1832 tid 100083 td 0xc76c0940 sched_switch(c76c0940,0,104,191,1ceaf1fb,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c76c0940,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c0745097,160,0,c76c0940,c76c0940,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c7379470,0,c0749892,101,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _cv_wait_sig(c7379470,c07ce9f0,c07491e9,4fd,0,...) at 0xc04c5ce0 = _cv_wait_sig+0x240 tty_wait(c7379400,c7379470,e9556c5c,1,0,...) at 0xc0559721 = tty_wait+0x71 ttydisc_read(c7379400,e9556c5c,0,9e,0,...) at 0xc0560e8c = ttydisc_read+0x22c ttydev_read(c738f600,e9556c5c,0,0,1,...) at 0xc055b0ca = ttydev_read+0xaa devfs_read_f(c76f4070,e9556c5c,c7041400,0,c76c0940,...) at 0xc04a6c1e = devfs_read_f+0x7e dofileread(e9556c5c,ffffffff,ffffffff,0,c76f4070,...) at 0xc054bf26 = dofileread+0x96 kern_readv(c76c0940,0,e9556c5c,e9556c7c,1,...) at 0xc054c258 = kern_readv+0x58 read(c76c0940,e9556cf8,c0770332,c0752f28,246,...) at 0xc054c36f = read+0x4f syscall(e9556d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f593, esp = 0xbfbfedbc, ebp = 0xbfbfede8 --- Tracing command getty pid 1831 tid 100127 td 0xc7f70b90 sched_switch(c7f70b90,0,104,191,1d0d5c35,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7f70b90,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c0745097,160,0,c7f70b90,c7f70b90,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c71f7c70,0,c0749892,101,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _cv_wait_sig(c71f7c70,c07ce9f0,c07491e9,4fd,0,...) at 0xc04c5ce0 = _cv_wait_sig+0x240 tty_wait(c71f7c00,c71f7c70,e964bc5c,1,0,...) at 0xc0559721 = tty_wait+0x71 ttydisc_read(c71f7c00,e964bc5c,0,9e,0,...) at 0xc0560e8c = ttydisc_read+0x22c ttydev_read(c738f700,e964bc5c,0,0,1,...) at 0xc055b0ca = ttydev_read+0xaa devfs_read_f(c7691af0,e964bc5c,c7041400,0,c7f70b90,...) at 0xc04a6c1e = devfs_read_f+0x7e dofileread(e964bc5c,ffffffff,ffffffff,0,c7691af0,...) at 0xc054bf26 = dofileread+0x96 kern_readv(c7f70b90,0,e964bc5c,e964bc7c,1,...) at 0xc054c258 = kern_readv+0x58 read(c7f70b90,e964bcf8,c0770332,c0752f28,246,...) at 0xc054c36f = read+0x4f syscall(e964bd38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f593, esp = 0xbfbfedbc, ebp = 0xbfbfede8 --- Tracing command getty pid 1830 tid 100130 td 0xc7f704a0 sched_switch(c7f704a0,0,104,191,1cfa1c5b,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7f704a0,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c0745097,160,0,c7f704a0,c7f704a0,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c7379270,0,c0749892,101,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _cv_wait_sig(c7379270,c07ce9f0,c07491e9,4fd,0,...) at 0xc04c5ce0 = _cv_wait_sig+0x240 tty_wait(c7379200,c7379270,e9657c5c,1,0,...) at 0xc0559721 = tty_wait+0x71 ttydisc_read(c7379200,e9657c5c,0,9e,0,...) at 0xc0560e8c = ttydisc_read+0x22c ttydev_read(c738f800,e9657c5c,0,0,1,...) at 0xc055b0ca = ttydev_read+0xaa devfs_read_f(c767b380,e9657c5c,c7041400,0,c7f704a0,...) at 0xc04a6c1e = devfs_read_f+0x7e dofileread(e9657c5c,ffffffff,ffffffff,0,c767b380,...) at 0xc054bf26 = dofileread+0x96 kern_readv(c7f704a0,0,e9657c5c,e9657c7c,1,...) at 0xc054c258 = kern_readv+0x58 read(c7f704a0,e9657cf8,c0770332,c0752f28,246,...) at 0xc054c36f = read+0x4f syscall(e9657d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f593, esp = 0xbfbfedbc, ebp = 0xbfbfede8 --- Tracing command login pid 1829 tid 100094 td 0xc76bd940 sched_switch(c76bd940,0,104,191,4cf2740b,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,5c,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c76bd940,0,c0745097,1a0,5c,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c768aaa0,5c,c0747a51,100,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _sleep(c768aaa0,c768ab28,15c,c0747a51,0,...) at 0xc0510b84 = _sleep+0x354 kern_wait(c76bd940,736,e9583c78,0,0,...) at 0xc04dbde6 = kern_wait+0xb46 wait4(c76bd940,e9583cf8,c,c,216,...) at 0xc04dbe5b = wait4+0x3b syscall(e9583d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x28120bdb, esp = 0xbfbfed2c, ebp = 0xbfbfed48 --- Tracing command login pid 1828 tid 100163 td 0xc80824a0 sched_switch(c80824a0,0,104,191,5be7b466,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,5c,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c80824a0,0,c0745097,1a0,5c,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c80d7d48,5c,c0747a51,100,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _sleep(c80d7d48,c80d7dd0,15c,c0747a51,0,...) at 0xc0510b84 = _sleep+0x354 kern_wait(c80824a0,746,e96dbc78,0,0,...) at 0xc04dbde6 = kern_wait+0xb46 wait4(c80824a0,e96dbcf8,c,c,216,...) at 0xc04dbe5b = wait4+0x3b syscall(e96dbd38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x28120bdb, esp = 0xbfbfed2c, ebp = 0xbfbfed48 --- Tracing command inetd pid 1802 tid 100103 td 0xc781e4a0 sched_switch(c781e4a0,0,104,191,d72096,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c781e4a0,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c04f811a,c7394dd0,0,c073f15b,c781e4a0,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c7394de4,0,e95a8a80,101,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _cv_wait_sig(c7394de4,c7394dd0,c074742e,621,c767b0a8,...) at 0xc04c5ce0 = _cv_wait_sig+0x240 seltdwait(c767b0a8,58,c7041400,c781e4a0,c0765b58,...) at 0xc054b5d2 = seltdwait+0xa2 kern_select(c781e4a0,7,bfbfdfc0,0,0,0,20,bfbfdf80,c781a000) at 0xc054c884 = kern_select+0x504 select(c781e4a0,e95a8cf8,c0770332,c0747aa4,202,...) at 0xc054cbd6 = select+0x66 syscall(e95a8d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x281a3513, esp = 0xbfbfdf7c, ebp = 0xbfbfee08 --- Tracing command moused pid 1770 tid 100090 td 0xc76c0000 sched_switch(c76c0000,0,104,191,ab40f8dd,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c76c0000,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c04f811a,c804f510,0,c073f15b,c76c0000,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c804f524,0,e9572a80,101,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _cv_wait_sig(c804f524,c804f510,c074742e,621,c7edb690,...) at 0xc04c5ce0 = _cv_wait_sig+0x240 seltdwait(c7edb690,58,c7041400,c76c0000,0,...) at 0xc054b5d2 = seltdwait+0xa2 kern_select(c76c0000,6,bfbfea34,0,0,0,20,bfbfe9c0,c76bb000) at 0xc054c884 = kern_select+0x504 select(c76c0000,e9572cf8,c0770332,c0752f28,246,...) at 0xc054cbd6 = select+0x66 syscall(e9572d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x281ac513, esp = 0xbfbfe9bc, ebp = 0xbfbfeb58 --- Tracing command cron pid 1724 tid 100097 td 0xc76bd250 sched_switch(c76bd250,0,104,191,9e5cb557,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,5c,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c76bd250,0,c0745097,1a0,5c,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(dea9,c053fd30,c76bd250,1,100,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_timedwait_sig(c07cf224,5c,c0741eb4,100,0,...) at 0xc0540daa = sleepq_timedwait_sig+0x1a _sleep(c07cf224,0,15c,c0741eb4,dea9,...) at 0xc0510b4e = _sleep+0x31e kern_nanosleep(c76bd250,e9596c68,e9596c70,39,0,...) at 0xc0517f21 = kern_nanosleep+0xc1 nanosleep(c76bd250,e9596cf8,c0770332,c0747f0c,206,...) at 0xc051995f = nanosleep+0x6f syscall(e9596d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x2817bf17, esp = 0xbfbfec8c, ebp = 0xbfbfecb8 --- Tracing command sendmail pid 1717 tid 100110 td 0xc7c474a0 sched_switch(c7c474a0,0,104,191,e0648a6b,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,68,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7c474a0,0,c0745097,1a0,68,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c7c42058,68,c0740d48,100,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _sleep(c7c42058,c7c42088,168,c0740d48,0,...) at 0xc0510b84 = _sleep+0x354 kern_sigsuspend(c7c474a0,0,0,0,0,...) at 0xc050b7ae = kern_sigsuspend+0xae sigsuspend(c7c474a0,e95ebcf8,c0770332,c074792b,202,...) at 0xc050b8bd = sigsuspend+0x4d syscall(e95ebd38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (4, FreeBSD ELF32, write), eip = 0x2832fd9b, esp = 0xbfbfda9c, ebp = 0xbfbfdac8 --- Tracing command sendmail pid 1713 tid 100087 td 0xc76c04a0 sched_switch(c76c04a0,0,104,191,95606d23,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c76c04a0,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(e9566a50,c04f811a,c7577cd0,0,c76c04a0,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_timedwait_sig(c7577ce4,0,e9566a80,101,0,...) at 0xc0540daa = sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c7577ce4,c7577cd0,1389,621,c80ea188,...) at 0xc04c5690 = _cv_timedwait_sig+0x250 seltdwait(e9566c2c,e9566c34,c75cb480,c76c04a0,c159b014,...) at 0xc054b5ba = seltdwait+0x8a kern_select(c76c04a0,8,bfbfd8bc,0,0,e9566c74,20,5,0) at 0xc054c884 = kern_select+0x504 select(c76c04a0,e9566cf8,c0770332,c0747e6b,202,...) at 0xc054cbd6 = select+0x66 syscall(e9566d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x283d6513, esp = 0xbfbfd02c, ebp = 0xbfbfdac8 --- Tracing command sshd pid 1707 tid 100086 td 0xc76804a0 sched_switch(c76804a0,0,104,191,9bb47272,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c76804a0,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c04f811a,c75cf8d0,0,c073f15b,c76804a0,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c75cf8e4,0,e9562a80,101,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _cv_wait_sig(c75cf8e4,c75cf8d0,c074742e,621,c7674b60,...) at 0xc04c5ce0 = _cv_wait_sig+0x240 seltdwait(c7674b60,58,c75cb180,c76804a0,c076500d,...) at 0xc054b5d2 = seltdwait+0xa2 kern_select(c76804a0,5,286090ac,0,0,0,20,bfbfe2a0,c76887f8) at 0xc054c884 = kern_select+0x504 select(c76804a0,e9562cf8,c,c,206,...) at 0xc054cbd6 = select+0x66 syscall(e9562d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x284a9513, esp = 0xbfbfe29c, ebp = 0xbfbfee38 --- Tracing command dbus-daemon pid 1621 tid 100170 td 0xc80dd6f0 sched_switch(c80dd6f0,0,104,191,850a5cc,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c80dd6f0,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c04f811a,c8050190,0,c073f15b,c80dd6f0,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c80501a4,0,e96f0adc,101,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _cv_wait_sig(c80501a4,c8050190,c074742e,621,e96f0b78,...) at 0xc04c5ce0 = _cv_wait_sig+0x240 seltdwait(c7589e2c,c074742e,52f,c80dd6f0,e96f0b60,...) at 0xc054b5d2 = seltdwait+0xa2 poll(c80dd6f0,e96f0cf8,c0770332,c0747f0c,282,...) at 0xc054ba80 = poll+0x300 syscall(e96f0d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip = 0x281a3b0f, esp = 0xbfbfe95c, ebp = 0xbfbfe978 --- Tracing command ntpd pid 1416 tid 100113 td 0xc7c43b90 sched_switch(c7c43b90,0,104,191,6810e037,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7c43b90,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c04f811a,c75b7310,0,c073f15b,c7c43b90,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c75b7324,0,e95f7a80,101,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _cv_wait_sig(c75b7324,c75b7310,c074742e,621,c76f4d90,...) at 0xc04c5ce0 = _cv_wait_sig+0x240 seltdwait(c76f4d90,58,c7041400,c7c43b90,0,...) at 0xc054b5d2 = seltdwait+0xa2 kern_select(c7c43b90,20,bfbfed18,0,0,0,20,bfbfecf0,c7c417f8) at 0xc054c884 = kern_select+0x504 select(c7c43b90,e95f7cf8,16,16,212,...) at 0xc054cbd6 = select+0x66 syscall(e95f7d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x28354513, esp = 0xbfbfecec, ebp = 0xbfbfedb8 --- Tracing command logger pid 1380 tid 100076 td 0xc7681000 sched_switch(c7681000,0,104,191,bc02b312,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,4c,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7681000,0,c0745097,1a0,4c,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c804d188,4c,c0747637,100,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _sleep(c804d188,c804d2f8,14c,c0747637,0,...) at 0xc0510b84 = _sleep+0x354 pipe_read(c7edb850,e953ac5c,c7041400,0,c7681000,...) at 0xc054efa7 = pipe_read+0x3f7 dofileread(e953ac5c,ffffffff,ffffffff,0,c7edb850,...) at 0xc054bf26 = dofileread+0x96 kern_readv(c7681000,0,e953ac5c,e953ac7c,1,...) at 0xc054c258 = kern_readv+0x58 read(c7681000,e953acf8,c0770332,c0747c21,216,...) at 0xc054c36f = read+0x4f syscall(e953ad38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x2817d593, esp = 0xbfbfe93c, ebp = 0xbfbfe958 --- Tracing command ssh pid 1379 tid 100155 td 0xc8083940 sched_switch(c8083940,0,104,191,bdba131a,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c8083940,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c04f811a,c7577a90,0,c073f15b,c8083940,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c7577aa4,0,e96b9a80,101,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _cv_wait_sig(c7577aa4,c7577a90,c074742e,621,c767bb98,...) at 0xc04c5ce0 = _cv_wait_sig+0x240 seltdwait(c767bb98,58,c7f7db00,c8083940,246,...) at 0xc054b5d2 = seltdwait+0xa2 kern_select(c8083940,5,285010b0,285010b4,0,0,20,bfbfbf30,c807f000) at 0xc054c884 = kern_select+0x504 select(c8083940,e96b9cf8,c0770332,c073d197,206,...) at 0xc054cbd6 = select+0x66 syscall(e96b9d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x28381513, esp = 0xbfbfbf2c, ebp = 0xbfbfbfd8 --- Tracing command sh pid 1374 tid 100129 td 0xc7f706f0 sched_switch(c7f706f0,0,104,191,db29b8dc,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,5c,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7f706f0,0,c0745097,1a0,5c,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c7f6a7f8,5c,c0747a51,100,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _sleep(c7f6a7f8,c7f6a880,15c,c0747a51,0,...) at 0xc0510b84 = _sleep+0x354 kern_wait(c7f706f0,ffffffff,e9653c78,2,0,...) at 0xc04dbde6 = kern_wait+0xb46 wait4(c7f706f0,e9653cf8,c0770332,c0747a30,216,...) at 0xc04dbe5b = wait4+0x3b syscall(e9653d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2815dbdb, esp = 0xbfbfea4c, ebp = 0xbfbfea68 --- Tracing command named pid 1200 tid 100140 td 0xc8026b90 sched_switch(c8026b90,0,104,191,6e86129a,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,58,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c8026b90,0,c0745097,1a0,58,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c703e380,58,c073b7b7,100,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _sleep(c703e380,c703e380,158,c073b7b7,0,...) at 0xc0510b84 = _sleep+0x354 kern_kevent(c8026b90,9,0,40,e968cc5c,...) at 0xc04d8514 = kern_kevent+0x364 kevent(c8026b90,e968ccf8,c0770332,c072d816,246,...) at 0xc04d8e4b = kevent+0x19b syscall(e968cd38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (363, FreeBSD ELF32, kevent), eip = 0x2849f07b, esp = 0xbf4f9e0c, ebp = 0xbf4f9f98 --- Tracing command named pid 1200 tid 100139 td 0xc8027000 sched_switch(c8027000,0,104,191,a96622fc,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c8027000,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(119d12,c053fd30,c8027000,2,100,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_timedwait_sig(c75c4e40,0,c074235e,100,0,...) at 0xc0540daa = sleepq_timedwait_sig+0x1a _sleep(c75c4e40,c07d18cc,100,c074235e,119d12,...) at 0xc0510b4e = _sleep+0x31e __umtx_op_cv_wait(c8027000,e9689cf8,e9689d2c,c06f41e5,c8027000,...) at 0xc051ff78 = __umtx_op_cv_wait+0x558 _umtx_op(c8027000,e9689cf8,c0770332,c0747f0c,202,...) at 0xc051b0c7 = _umtx_op+0x27 syscall(e9689d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x283c8227, esp = 0xbf5faddc, ebp = 0xbf5fae08 --- Tracing command named pid 1200 tid 100138 td 0xc8027250 sched_switch(c8027250,0,104,191,fd997d29,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c8027250,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c75c4b00,0,c074235e,100,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _sleep(c75c4b00,c07d1a6c,100,c074235e,0,...) at 0xc0510b84 = _sleep+0x354 __umtx_op_cv_wait(c8027250,e9686cf8,e9686d2c,c06f41e5,c8027250,...) at 0xc051fe96 = __umtx_op_cv_wait+0x476 _umtx_op(c8027250,e9686cf8,c0770332,c0747b73,202,...) at 0xc051b0c7 = _umtx_op+0x27 syscall(e9686d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x283c8227, esp = 0xbf6fbe9c, ebp = 0xbf6fbec8 --- Tracing command named pid 1200 tid 100137 td 0xc80274a0 sched_switch(c80274a0,0,104,191,6a100e2d,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c80274a0,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c75990c0,0,c074235e,100,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _sleep(c75990c0,c07d1a6c,100,c074235e,0,...) at 0xc0510b84 = _sleep+0x354 __umtx_op_cv_wait(c80274a0,e9683cf8,e9683d2c,c06f41e5,c80274a0,...) at 0xc051fe96 = __umtx_op_cv_wait+0x476 _umtx_op(c80274a0,e9683cf8,c0770332,c0747b73,202,...) at 0xc051b0c7 = _umtx_op+0x27 syscall(e9683d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x283c8227, esp = 0xbf7fce9c, ebp = 0xbf7fcec8 --- Tracing command named pid 1200 tid 100136 td 0xc80276f0 sched_switch(c80276f0,0,104,191,fdbd72ee,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c80276f0,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c7394d80,0,c074235e,100,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _sleep(c7394d80,c07d1a6c,100,c074235e,0,...) at 0xc0510b84 = _sleep+0x354 __umtx_op_cv_wait(c80276f0,e9680cf8,e9680d2c,c06f41e5,c80276f0,...) at 0xc051fe96 = __umtx_op_cv_wait+0x476 _umtx_op(c80276f0,e9680cf8,c0770332,c0747b73,202,...) at 0xc051b0c7 = _umtx_op+0x27 syscall(e9680d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x283c8227, esp = 0xbf8fde9c, ebp = 0xbf8fdec8 --- Tracing command named pid 1200 tid 100135 td 0xc7385940 sched_switch(c7385940,0,104,191,6e8997e7,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7385940,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c7381800,0,c074235e,100,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _sleep(c7381800,c07d1a6c,100,c074235e,0,...) at 0xc0510b84 = _sleep+0x354 __umtx_op_cv_wait(c7385940,e967dcf8,e967dd2c,c06f41e5,c7385940,...) at 0xc051fe96 = __umtx_op_cv_wait+0x476 _umtx_op(c7385940,e967dcf8,c0770332,c0747b73,202,...) at 0xc051b0c7 = _umtx_op+0x27 syscall(e967dd38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x283c8227, esp = 0xbf9fee9c, ebp = 0xbf9feec8 --- Tracing command named pid 1200 tid 100121 td 0xc7f71b90 sched_switch(c7f71b90,0,104,191,4dc27c2b,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,68,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7f71b90,0,c0745097,1a0,68,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(e9633be4,68,c0740d4e,100,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _sleep(e9633be4,c7f6bdd0,168,c0740d4e,0,...) at 0xc0510b84 = _sleep+0x354 kern_sigtimedwait(c7f71b90,4003,0,0,0,...) at 0xc050bb4c = kern_sigtimedwait+0x28c sigwait(c7f71b90,e9633cf8,c0770332,c0748721,200282,...) at 0xc050bfb7 = sigwait+0x77 syscall(e9633d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (429, FreeBSD ELF32, sigwait), eip = 0x284146db, esp = 0xbfbfec9c, ebp = 0xbfbfecc8 --- Tracing command syslogd pid 1101 tid 100123 td 0xc7f716f0 sched_switch(c7f716f0,0,104,191,ddc5cff9,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7f716f0,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c04f811a,c75b79d0,0,c073f15b,c7f716f0,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c75b79e4,0,e963ba80,101,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _cv_wait_sig(c75b79e4,c75b79d0,c074742e,621,c7675e00,...) at 0xc04c5ce0 = _cv_wait_sig+0x240 seltdwait(c7675e00,58,c7041400,c7f716f0,0,...) at 0xc054b5d2 = seltdwait+0xa2 kern_select(c7f716f0,b,282310ac,0,0,0,20,bfbfe260,c7f6b7f8) at 0xc054c884 = kern_select+0x504 select(c7f716f0,e963bcf8,c0770332,c0747a40,246,...) at 0xc054cbd6 = select+0x66 syscall(e963bd38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x28192513, esp = 0xbfbfe25c, ebp = 0xbfbfedc8 --- Tracing command devd pid 700 tid 100101 td 0xc781e940 sched_switch(c781e940,0,104,191,a3e23ce5,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c781e940,0,c0745097,1a0,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c04f811a,c75cf290,0,c073f15b,c781e940,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c75cf2a4,0,e95a2a80,101,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _cv_wait_sig(c75cf2a4,c75cf290,c074742e,621,c767b578,...) at 0xc04c5ce0 = _cv_wait_sig+0x240 seltdwait(c767b578,58,c7041400,c781e940,c05494ec,...) at 0xc054b5d2 = seltdwait+0xa2 kern_select(c781e940,6,bfbfe9a0,0,0,0,20,bfbfe970,c781a550) at 0xc054c884 = kern_select+0x504 select(c781e940,e95a2cf8,c0770332,c07484d2,206,...) at 0xc054cbd6 = select+0x66 syscall(e95a2d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x8088343, esp = 0xbfbfe96c, ebp = 0xbfbfee48 --- Tracing command adjkerntz pid 100 tid 100073 td 0xc76816f0 sched_switch(c76816f0,0,104,191,5022b0eb,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,68,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c76816f0,0,c0745097,1a0,68,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c71b6850,68,c0740d48,100,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _sleep(c71b6850,c71b6880,168,c0740d48,0,...) at 0xc0510b84 = _sleep+0x354 kern_sigsuspend(c76816f0,0,0,0,0,...) at 0xc050b7ae = kern_sigsuspend+0xae sigsuspend(c76816f0,e952dcf8,c0770332,c0747e6b,202,...) at 0xc050b8bd = sigsuspend+0x4d syscall(e952dd38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (4, FreeBSD ELF32, write), eip = 0x280d6d9b, esp = 0xbfbfed6c, ebp = 0xbfbfee38 --- Tracing command softdepflush pid 20 tid 100069 td 0xc727d940 sched_switch(c727d940,0,104,191,ebf6f65,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,44,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c727d940,0,c0745097,283,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_timedwait(c0949920,44,c0761839,0,0,...) at 0xc0540e6b = sleepq_timedwait+0x6b _sleep(c0949920,c09498c4,44,c0761839,3e8,...) at 0xc0510b69 = _sleep+0x339 softdep_flush(0,e7332d38,c073bc0b,343,c759f2a8,...) at 0xc0674b44 = softdep_flush+0x244 fork_exit(c0674900,0,e7332d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7332d70, ebp = 0 --- Tracing command vnlru pid 19 tid 100068 td 0xc727db90 sched_switch(c727db90,0,104,191,de62f5e,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,50,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c727db90,0,c0745097,283,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_timedwait(c759f550,50,c074f6cf,0,0,...) at 0xc0540e6b = sleepq_timedwait+0x6b _sleep(c759f550,c093b5d8,250,c074f6cf,3e8,...) at 0xc0510b69 = _sleep+0x339 vnlru_proc(0,e732fd38,c073bc0b,343,c759f550,...) at 0xc0599157 = vnlru_proc+0xe7 fork_exit(c0599070,0,e732fd38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe732fd70, ebp = 0 --- Tracing command syncer pid 18 tid 100067 td 0xc7383000 sched_switch(c7383000,0,104,191,e660f504,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,50,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7383000,0,c0745097,260,50,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(cd466dc8,50,c07393ba,4,0,...) at 0xc0540f63 = sleepq_wait+0x63 __lockmgr_args(cd466dc8,80100,cd466e30,0,0,...) at 0xc04f087f = __lockmgr_args+0xb1f ffs_lock(e732cb34,c054944b,c0763314,80100,cd466d70,...) at 0xc067e9f1 = ffs_lock+0xa1 VOP_LOCK1_APV(c07a47c0,e732cb34,c73830a4,c07b7a20,cd466d70,...) at 0xc070c945 = VOP_LOCK1_APV+0xb5 _vn_lock(cd466d70,80100,c074e6ce,82b,4,...) at 0xc05a2fd8 = _vn_lock+0x78 vget(cd466d70,80100,c7383000,39a,c763b870,...) at 0xc05969db = vget+0xbb qsync(c763b870,0,c0761f41,552,4,...) at 0xc0687db7 = qsync+0x197 ffs_sync(c763b870,3,c074e6ce,d6b,c763b870,...) at 0xc067a139 = ffs_sync+0x349 sync_fsync(e732cc7c,c0772c36,c76b8778,e732cc7c,c76b86b8,...) at 0xc059812f = sync_fsync+0x18f VOP_FSYNC_APV(c0799bc0,e732cc7c,c074e6ce,6a5,c7383000,...) at 0xc070a545 = VOP_FSYNC_APV+0xc5 sync_vnode(c093b618,c093b604,3e8,6d4,4e20,...) at 0xc059840b = sync_vnode+0x16b sched_sync(0,e732cd38,c073bc0b,343,c759f7f8,...) at 0xc0598753 = sched_sync+0x273 fork_exit(c05984e0,0,e732cd38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe732cd70, ebp = 0 --- Tracing command bufdaemon pid 17 tid 100066 td 0xc7383250 sched_switch(c7383250,0,104,191,3b094933,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,44,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7383250,0,c0745097,283,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_timedwait(c093b348,44,c074cb15,0,0,...) at 0xc0540e6b = sleepq_timedwait+0x6b _sleep(c093b348,c093b34c,44,c074cb15,3e8,...) at 0xc0510b69 = _sleep+0x339 buf_daemon(0,e7329d38,c073bc0b,343,c759faa0,...) at 0xc05807b8 = buf_daemon+0x138 fork_exit(c0580680,0,e7329d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7329d70, ebp = 0 --- Tracing command pagezero pid 16 tid 100065 td 0xc73834a0 sched_switch(c73834a0,0,104,191,d7d993b1,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c73834a0,0,c0745097,283,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_timedwait(c094a5f4,0,c07678c2,0,0,...) at 0xc0540e6b = sleepq_timedwait+0x6b _sleep(c094a5f4,c094a1b0,0,c07678c2,493e0,...) at 0xc0510b69 = _sleep+0x339 vm_pagezero(0,e7326d38,c073bc0b,343,c759fd48,...) at 0xc06b42dc = vm_pagezero+0xdc fork_exit(c06b4200,0,e7326d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7326d70, ebp = 0 --- Tracing command vmdaemon pid 15 tid 100064 td 0xc73836f0 sched_switch(c73836f0,0,104,191,237688ca,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,68,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c73836f0,0,c0745097,260,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c094a21c,68,c074cb15,0,0,...) at 0xc0540f63 = sleepq_wait+0x63 _sleep(c094a21c,c094a220,68,c074cb15,0,...) at 0xc0510b9b = _sleep+0x36b vm_daemon(0,e7323d38,c073bc0b,343,c70432a8,...) at 0xc06ae699 = vm_daemon+0x59 fork_exit(c06ae640,0,e7323d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7323d70, ebp = 0 --- Tracing command pagedaemon pid 14 tid 100063 td 0xc7383940 sched_switch(c7383940,0,104,191,a04fab1a,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,44,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7383940,0,c0745097,283,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_timedwait(c094a1e4,44,c074cb15,0,0,...) at 0xc0540e6b = sleepq_timedwait+0x6b _sleep(c094a1e4,c094a1b0,44,c074cb15,1388,...) at 0xc0510b69 = _sleep+0x339 vm_pageout(0,e7320d38,c073bc0b,343,c7043550,...) at 0xc06af79b = vm_pageout+0x2bb fork_exit(c06af4e0,0,e7320d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe7320d70, ebp = 0 --- Tracing command xpt_thrd pid 9 tid 100062 td 0xc7383b90 sched_switch(c7383b90,0,104,191,1e90655d,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,4c,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7383b90,0,c0745097,260,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c07b8dd4,4c,c072889c,0,0,...) at 0xc0540f63 = sleepq_wait+0x63 _sleep(c07b8dd4,c07b8e18,4c,c072889c,0,...) at 0xc0510b9b = _sleep+0x36b xpt_scanner_thread(0,c6bced38,c073bc0b,343,c70437f8,...) at 0xc043fb87 = xpt_scanner_thread+0x47 fork_exit(c043fb40,0,c6bced38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6bced70, ebp = 0 --- Tracing command fdc0 pid 8 tid 100057 td 0xc72446f0 sched_switch(c72446f0,0,104,191,14b3b3f0,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,4c,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c72446f0,0,c0745097,283,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_timedwait(c71f763c,4c,c0ca82e8,0,0,...) at 0xc0540e6b = sleepq_timedwait+0x6b _sleep(c71f763c,c71f76f0,4c,c0ca82e8,3e8,...) at 0xc0510b69 = _sleep+0x339 fdc_thread(c71f7600,c6bbed38,c073bc0b,343,c7043aa0,...) at 0xc0ca610d = fdc_thread+0x89d fork_exit(c0ca5870,c71f7600,c6bbed38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6bbed70, ebp = 0 --- Tracing command fw0_probe pid 7 tid 100055 td 0xc7244b90 sched_switch(c7244b90,0,104,191,415a8d7b,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,5c,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7244b90,0,c0745097,1a0,5c,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c72a0000,5c,c0d5ba7e,100,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _sleep(c72a0000,c72a4488,15c,c0d5ba7e,0,...) at 0xc0510b84 = _sleep+0x354 fw_bus_probe_thread(c72a0000,c6b9ed38,c073bc0b,343,c7043d48,...) at 0xc0d4eaab = fw_bus_probe_thread+0x9b fork_exit(c0d4ea10,c72a0000,c6b9ed38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b9ed70, ebp = 0 --- Tracing command usb pid 13 tid 100053 td 0xc727c250 sched_switch(c727c250,0,104,191,1d799992,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c727c250,0,c0745097,260,c727c250,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c7284d0c,0,c6b94cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63 _cv_wait(c7284d0c,c7284dac,c0b6b100,6c,c7284d14,...) at 0xc04c62c0 = _cv_wait+0x240 usb_process(c7284d04,c6b94d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10 = usb_process+0x170 fork_exit(c0b61aa0,c7284d04,c6b94d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b94d70, ebp = 0 --- Tracing command usb pid 13 tid 100052 td 0xc727c4a0 sched_switch(c727c4a0,0,104,191,e6af0580,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c727c4a0,0,c0745097,260,c727c4a0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c7284cdc,0,c6b91cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63 _cv_wait(c7284cdc,c7284dac,c0b6b100,6c,c7284ce4,...) at 0xc04c62c0 = _cv_wait+0x240 usb_process(c7284cd4,c6b91d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10 = usb_process+0x170 fork_exit(c0b61aa0,c7284cd4,c6b91d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b91d70, ebp = 0 --- Tracing command usb pid 13 tid 100051 td 0xc727c6f0 sched_switch(c727c6f0,0,104,191,1d7457f7,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c727c6f0,0,c0745097,260,c727c6f0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c7284cac,0,c6b8ecbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63 _cv_wait(c7284cac,c7284dac,c0b6b100,6c,c7284cb4,...) at 0xc04c62c0 = _cv_wait+0x240 usb_process(c7284ca4,c6b8ed38,c073bc0b,343,c71b6000,...) at 0xc0b61c10 = usb_process+0x170 fork_exit(c0b61aa0,c7284ca4,c6b8ed38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b8ed70, ebp = 0 --- Tracing command usb pid 13 tid 100050 td 0xc727c940 sched_switch(c727c940,0,104,191,1d744207,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c727c940,0,c0745097,260,c727c940,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c7284c7c,0,c6b8bcbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63 _cv_wait(c7284c7c,c7284dac,c0b6b100,6c,c7284c84,...) at 0xc04c62c0 = _cv_wait+0x240 usb_process(c7284c74,c6b8bd38,c073bc0b,343,c71b6000,...) at 0xc0b61c10 = usb_process+0x170 fork_exit(c0b61aa0,c7284c74,c6b8bd38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b8bd70, ebp = 0 --- Tracing command usb pid 13 tid 100049 td 0xc727cb90 sched_switch(c727cb90,0,104,191,1d7428a5,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c727cb90,0,c0745097,260,c727cb90,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c726ddac,0,c6b86cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63 _cv_wait(c726ddac,c726de4c,c0b6b100,6c,c726ddb4,...) at 0xc04c62c0 = _cv_wait+0x240 usb_process(c726dda4,c6b86d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10 = usb_process+0x170 fork_exit(c0b61aa0,c726dda4,c6b86d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b86d70, ebp = 0 --- Tracing command usb pid 13 tid 100048 td 0xc727d000 sched_switch(c727d000,0,104,191,63226d58,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c727d000,0,c0745097,260,c727d000,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c726dd7c,0,c6b83cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63 _cv_wait(c726dd7c,c726de4c,c0b6b100,6c,c726dd84,...) at 0xc04c62c0 = _cv_wait+0x240 usb_process(c726dd74,c6b83d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10 = usb_process+0x170 fork_exit(c0b61aa0,c726dd74,c6b83d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b83d70, ebp = 0 --- Tracing command usb pid 13 tid 100047 td 0xc727d250 sched_switch(c727d250,0,104,191,1d6f01d4,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c727d250,0,c0745097,260,c727d250,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c726dd4c,0,c6b80cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63 _cv_wait(c726dd4c,c726de4c,c0b6b100,6c,c726dd54,...) at 0xc04c62c0 = _cv_wait+0x240 usb_process(c726dd44,c6b80d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10 = usb_process+0x170 fork_exit(c0b61aa0,c726dd44,c6b80d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b80d70, ebp = 0 --- Tracing command usb pid 13 tid 100046 td 0xc727d4a0 sched_switch(c727d4a0,0,104,191,1d6ee90b,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c727d4a0,0,c0745097,260,c727d4a0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c726dd1c,0,c6b7dcbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63 _cv_wait(c726dd1c,c726de4c,c0b6b100,6c,c726dd24,...) at 0xc04c62c0 = _cv_wait+0x240 usb_process(c726dd14,c6b7dd38,c073bc0b,343,c71b6000,...) at 0xc0b61c10 = usb_process+0x170 fork_exit(c0b61aa0,c726dd14,c6b7dd38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b7dd70, ebp = 0 --- Tracing command usb pid 13 tid 100045 td 0xc71bd940 sched_switch(c71bd940,0,104,191,1d6ecd9f,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c71bd940,0,c0745097,260,c71bd940,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c725cdac,0,c6b79cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63 _cv_wait(c725cdac,c725ce4c,c0b6b100,6c,c725cdb4,...) at 0xc04c62c0 = _cv_wait+0x240 usb_process(c725cda4,c6b79d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10 = usb_process+0x170 fork_exit(c0b61aa0,c725cda4,c6b79d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b79d70, ebp = 0 --- Tracing command usb pid 13 tid 100044 td 0xc71bdb90 sched_switch(c71bdb90,0,104,191,6505c938,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c71bdb90,0,c0745097,260,c71bdb90,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c725cd7c,0,c6b76cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63 _cv_wait(c725cd7c,c725ce4c,c0b6b100,6c,c725cd84,...) at 0xc04c62c0 = _cv_wait+0x240 usb_process(c725cd74,c6b76d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10 = usb_process+0x170 fork_exit(c0b61aa0,c725cd74,c6b76d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b76d70, ebp = 0 --- Tracing command usb pid 13 tid 100043 td 0xc7243000 sched_switch(c7243000,0,104,191,1d699d50,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7243000,0,c0745097,260,c7243000,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c725cd4c,0,c6b73cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63 _cv_wait(c725cd4c,c725ce4c,c0b6b100,6c,c725cd54,...) at 0xc04c62c0 = _cv_wait+0x240 usb_process(c725cd44,c6b73d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10 = usb_process+0x170 fork_exit(c0b61aa0,c725cd44,c6b73d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b73d70, ebp = 0 --- Tracing command usb pid 13 tid 100042 td 0xc7243250 sched_switch(c7243250,0,104,191,1d69877b,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7243250,0,c0745097,260,c7243250,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c725cd1c,0,c6b70cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63 _cv_wait(c725cd1c,c725ce4c,c0b6b100,6c,c725cd24,...) at 0xc04c62c0 = _cv_wait+0x240 usb_process(c725cd14,c6b70d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10 = usb_process+0x170 fork_exit(c0b61aa0,c725cd14,c6b70d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b70d70, ebp = 0 --- Tracing command usb pid 13 tid 100040 td 0xc72436f0 sched_switch(c72436f0,0,104,191,1d69708f,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c72436f0,0,c0745097,260,c72436f0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c7248dac,0,c6b69cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63 _cv_wait(c7248dac,c7248e4c,c0b6b100,6c,c7248db4,...) at 0xc04c62c0 = _cv_wait+0x240 usb_process(c7248da4,c6b69d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10 = usb_process+0x170 fork_exit(c0b61aa0,c7248da4,c6b69d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b69d70, ebp = 0 --- Tracing command usb pid 13 tid 100039 td 0xc7243940 sched_switch(c7243940,0,104,191,650581e3,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7243940,0,c0745097,260,c7243940,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c7248d7c,0,c6b66cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63 _cv_wait(c7248d7c,c7248e4c,c0b6b100,6c,c7248d84,...) at 0xc04c62c0 = _cv_wait+0x240 usb_process(c7248d74,c6b66d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10 = usb_process+0x170 fork_exit(c0b61aa0,c7248d74,c6b66d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b66d70, ebp = 0 --- Tracing command usb pid 13 tid 100038 td 0xc7243b90 sched_switch(c7243b90,0,104,191,1d63f67e,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7243b90,0,c0745097,260,c7243b90,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c7248d4c,0,c6b63cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63 _cv_wait(c7248d4c,c7248e4c,c0b6b100,6c,c7248d54,...) at 0xc04c62c0 = _cv_wait+0x240 usb_process(c7248d44,c6b63d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10 = usb_process+0x170 fork_exit(c0b61aa0,c7248d44,c6b63d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b63d70, ebp = 0 --- Tracing command usb pid 13 tid 100037 td 0xc7244000 sched_switch(c7244000,0,104,191,1d63dff5,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7244000,0,c0745097,260,c7244000,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c7248d1c,0,c6b60cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63 _cv_wait(c7248d1c,c7248e4c,c0b6b100,6c,c7248d24,...) at 0xc04c62c0 = _cv_wait+0x240 usb_process(c7248d14,c6b60d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10 = usb_process+0x170 fork_exit(c0b61aa0,c7248d14,c6b60d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b60d70, ebp = 0 --- Tracing command usb pid 13 tid 100035 td 0xc72444a0 sched_switch(c72444a0,0,104,191,1d63c44a,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c72444a0,0,c0745097,260,c72444a0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c721ddac,0,c6b59cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63 _cv_wait(c721ddac,c721de4c,c0b6b100,6c,c721ddb4,...) at 0xc04c62c0 = _cv_wait+0x240 usb_process(c721dda4,c6b59d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10 = usb_process+0x170 fork_exit(c0b61aa0,c721dda4,c6b59d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b59d70, ebp = 0 --- Tracing command usb pid 13 tid 100034 td 0xc71bc000 sched_switch(c71bc000,0,104,191,6322a5c5,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c71bc000,0,c0745097,260,c71bc000,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c721dd7c,0,c6b56cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63 _cv_wait(c721dd7c,c721de4c,c0b6b100,6c,c721dd84,...) at 0xc04c62c0 = _cv_wait+0x240 usb_process(c721dd74,c6b56d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10 = usb_process+0x170 fork_exit(c0b61aa0,c721dd74,c6b56d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b56d70, ebp = 0 --- Tracing command usb pid 13 tid 100033 td 0xc71bc250 sched_switch(c71bc250,0,104,191,1d5e122f,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c71bc250,0,c0745097,260,c71bc250,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c721dd4c,0,c6b53cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63 _cv_wait(c721dd4c,c721de4c,c0b6b100,6c,c721dd54,...) at 0xc04c62c0 = _cv_wait+0x240 usb_process(c721dd44,c6b53d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10 = usb_process+0x170 fork_exit(c0b61aa0,c721dd44,c6b53d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b53d70, ebp = 0 --- Tracing command usb pid 13 tid 100032 td 0xc71bc4a0 sched_switch(c71bc4a0,0,104,191,1d5dface,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c71bc4a0,0,c0745097,260,c71bc4a0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c721dd1c,0,c6b50cbc,1,0,...) at 0xc0540f63 = sleepq_wait+0x63 _cv_wait(c721dd1c,c721de4c,c0b6b100,6c,c721dd24,...) at 0xc04c62c0 = _cv_wait+0x240 usb_process(c721dd14,c6b50d38,c073bc0b,343,c71b6000,...) at 0xc0b61c10 = usb_process+0x170 fork_exit(c0b61aa0,c721dd14,c6b50d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b50d70, ebp = 0 --- Tracing command crypto returns pid 6 tid 100017 td 0xc71b7940 sched_switch(c71b7940,0,104,191,1d5dd82d,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,5c,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c71b7940,0,c0745097,260,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c094962c,5c,c075f35c,0,0,...) at 0xc0540f63 = sleepq_wait+0x63 _sleep(c094962c,c094963c,5c,c075f35c,0,...) at 0xc0510b9b = _sleep+0x36b crypto_ret_proc(0,c68a8d38,c073bc0b,343,c71b62a8,...) at 0xc06507bc = crypto_ret_proc+0x1cc fork_exit(c06505f0,0,c68a8d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc68a8d70, ebp = 0 --- Tracing command crypto pid 5 tid 100016 td 0xc71b7b90 sched_switch(c71b7b90,0,104,191,1d5dbc82,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,5c,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c71b7b90,0,c0745097,260,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c0949608,5c,c075f560,0,0,...) at 0xc0540f63 = sleepq_wait+0x63 _sleep(c0949608,c0949618,5c,c075f560,0,...) at 0xc0510b9b = _sleep+0x36b crypto_proc(0,c68a5d38,c073bc0b,343,c71b6550,...) at 0xc0651c91 = crypto_proc+0x311 fork_exit(c0651980,0,c68a5d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc68a5d70, ebp = 0 --- Tracing command yarrow pid 12 tid 100015 td 0xc7046250 sched_switch(c7046250,0,104,191,73a5fb5a,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7046250,0,c0745097,283,2,...) at 0xc054028f = sleepq_switch+0x15f sleepq_timedwait(c07cf084,0,c0bc5e24,2,0,...) at 0xc0540e6b = sleepq_timedwait+0x6b _sleep(c07cf084,0,0,c0bc5e24,64,...) at 0xc0510b69 = _sleep+0x339 pause(c0bc5e24,64,c0bc5e26,111,0,...) at 0xc0510cf7 = pause+0x47 random_kthread(0,c68a2d38,c073bc0b,343,c7042000,...) at 0xc0bbf0a6 = random_kthread+0x1b6 fork_exit(c0bbeef0,0,c68a2d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc68a2d70, ebp = 0 --- Tracing command g_down pid 4 tid 100014 td 0xc70464a0 sched_switch(c70464a0,0,104,191,23c1cf6a,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,4c,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c70464a0,0,c0745097,260,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c07cce64,4c,c0758b0d,0,0,...) at 0xc0540f63 = sleepq_wait+0x63 _sleep(c07cce64,c07ccdc8,24c,c0758b0d,0,...) at 0xc0510b9b = _sleep+0x36b g_io_schedule_down(c70464a0,0,c07380d3,74,0,...) at 0xc04af026 = g_io_schedule_down+0x56 g_down_procbody(0,c689fd38,c073bc0b,343,c70422a8,...) at 0xc04af68d = g_down_procbody+0x8d fork_exit(c04af600,0,c689fd38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc689fd70, ebp = 0 --- Tracing command g_up pid 3 tid 100013 td 0xc70466f0 sched_switch(c70466f0,0,104,191,23de6e8a,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,4c,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c70466f0,0,c0745097,260,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c07cce60,4c,c0758b0d,0,0,...) at 0xc0540f63 = sleepq_wait+0x63 _sleep(c07cce60,c07ccde8,24c,c0758b0d,0,...) at 0xc0510b9b = _sleep+0x36b g_io_schedule_up(c70466f0,0,c07380d3,5d,0,...) at 0xc04aefbe = g_io_schedule_up+0x11e g_up_procbody(0,c689cd38,c073bc0b,343,c7042550,...) at 0xc04af71d = g_up_procbody+0x8d fork_exit(c04af690,0,c689cd38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc689cd70, ebp = 0 --- Tracing command g_event pid 2 tid 100012 td 0xc7046940 sched_switch(c7046940,0,104,191,6c429b16,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,4c,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7046940,0,c0745097,283,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_timedwait(c07cce58,4c,c0758b0d,0,0,...) at 0xc0540e6b = sleepq_timedwait+0x6b _sleep(c07cce58,0,4c,c0758b0d,64,...) at 0xc0510b69 = _sleep+0x339 g_event_procbody(0,c6899d38,c073bc0b,343,c70427f8,...) at 0xc04af7eb = g_event_procbody+0xcb fork_exit(c04af720,0,c6899d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6899d70, ebp = 0 --- Tracing command intr pid 11 tid 100172 td 0xc781d6f0 sched_switch(c781d6f0,0,109,191,498f3eb,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(109,0,c073bec7,52d,c71ea570,...) at 0xc0510680 = mi_switch+0x200 ithread_loop(c7fce100,eb6fed38,c073bc0b,343,c7042aa0,...) at 0xc04e0f96 = ithread_loop+0x1f6 fork_exit(c04e0da0,c7fce100,eb6fed38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xeb6fed70, ebp = 0 --- Tracing command intr pid 11 tid 100060 td 0xc7385250 fork_trampoline() at 0xc06d6b78 = fork_trampoline Tracing command intr pid 11 tid 100059 td 0xc73854a0 sched_switch(c73854a0,0,109,191,a41e83fa,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(109,0,c073bec7,52d,c7040cf0,...) at 0xc0510680 = mi_switch+0x200 ithread_loop(c737e7e0,c6bc4d38,c073bc0b,343,c7042aa0,...) at 0xc04e0f96 = ithread_loop+0x1f6 fork_exit(c04e0da0,c737e7e0,c6bc4d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6bc4d70, ebp = 0 --- Tracing command intr pid 11 tid 100058 td 0xc73856f0 sched_switch(c73856f0,0,109,191,f1f97bb8,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(109,0,c073bec7,52d,c7041370,...) at 0xc0510680 = mi_switch+0x200 ithread_loop(c737e9a0,c6bc1d38,c073bc0b,343,c7042aa0,...) at 0xc04e0f96 = ithread_loop+0x1f6 fork_exit(c04e0da0,c737e9a0,c6bc1d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6bc1d70, ebp = 0 --- Tracing command intr pid 11 tid 100056 td 0xc7244940 sched_switch(c7244940,0,109,191,23dde220,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(109,0,c073bec7,52d,c7040bf0,...) at 0xc0510680 = mi_switch+0x200 ithread_loop(c736e120,c6ba4d38,c073bc0b,343,c7042aa0,...) at 0xc04e0f96 = ithread_loop+0x1f6 fork_exit(c04e0da0,c736e120,c6ba4d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6ba4d70, ebp = 0 --- Tracing command intr pid 11 tid 100041 td 0xc72434a0 sched_switch(c72434a0,0,109,191,1d325785,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(109,0,c073bec7,52d,c70409f0,...) at 0xc0510680 = mi_switch+0x200 ithread_loop(c71f4b80,c6b6dd38,c073bc0b,343,c7042aa0,...) at 0xc04e0f96 = ithread_loop+0x1f6 fork_exit(c04e0da0,c71f4b80,c6b6dd38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b6dd70, ebp = 0 --- Tracing command intr pid 11 tid 100036 td 0xc7244250 sched_switch(c7244250,0,109,191,ff498233,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(109,0,c073bec7,52d,c7040970,...) at 0xc0510680 = mi_switch+0x200 ithread_loop(c71f49b0,c6b5dd38,c073bc0b,343,c7042aa0,...) at 0xc04e0f96 = ithread_loop+0x1f6 fork_exit(c04e0da0,c71f49b0,c6b5dd38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b5dd70, ebp = 0 --- Tracing command intr pid 11 tid 100031 td 0xc71bc6f0 fork_trampoline() at 0xc06d6b78 = fork_trampoline Tracing command intr pid 11 tid 100029 td 0xc71bcb90 fork_trampoline() at 0xc06d6b78 = fork_trampoline Tracing command intr pid 11 tid 100028 td 0xc71bd000 sched_switch(c71bd000,0,109,191,f83d1055,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(109,0,c073bec7,52d,c71dadf0,...) at 0xc0510680 = mi_switch+0x200 ithread_loop(c71ec000,c690bd38,c073bc0b,343,c7042aa0,...) at 0xc04e0f96 = ithread_loop+0x1f6 fork_exit(c04e0da0,c71ec000,c690bd38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc690bd70, ebp = 0 --- Tracing command intr pid 11 tid 100027 td 0xc71bd250 fork_trampoline() at 0xc06d6b78 = fork_trampoline Tracing command intr pid 11 tid 100025 td 0xc71bd6f0 sched_switch(c71bd6f0,0,109,191,a3db484d,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(109,0,c073bec7,52d,c71537f0,...) at 0xc0510680 = mi_switch+0x200 ithread_loop(c70832d0,c68c0d38,c073bc0b,343,c7042aa0,...) at 0xc04e0f96 = ithread_loop+0x1f6 fork_exit(c04e0da0,c70832d0,c68c0d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc68c0d70, ebp = 0 --- Tracing command intr pid 11 tid 100024 td 0xc70846f0 fork_trampoline() at 0xc06d6b78 = fork_trampoline Tracing command intr pid 11 tid 100019 td 0xc71b74a0 sched_switch(c71b74a0,0,109,191,9cffb807,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(109,0,c073bec7,52d,c71538f0,...) at 0xc0510680 = mi_switch+0x200 ithread_loop(c7083310,c68aed38,c073bc0b,343,c7042aa0,...) at 0xc04e0f96 = ithread_loop+0x1f6 fork_exit(c04e0da0,c7083310,c68aed38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc68aed70, ebp = 0 --- Tracing command intr pid 11 tid 100018 td 0xc71b76f0 sched_switch(c71b76f0,0,109,191,f12cfc2e,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(109,0,c073bec7,52d,c71539f0,...) at 0xc0510680 = mi_switch+0x200 ithread_loop(c7083340,c68abd38,c073bc0b,343,c7042aa0,...) at 0xc04e0f96 = ithread_loop+0x1f6 fork_exit(c04e0da0,c7083340,c68abd38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc68abd70, ebp = 0 --- Tracing command intr pid 11 tid 100011 td 0xc7046b90 sched_switch(c7046b90,0,109,191,30988d8a,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(109,0,c073bec7,52d,c7040470,...) at 0xc0510680 = mi_switch+0x200 ithread_loop(c7083790,c6896d38,c073bc0b,343,c7042aa0,...) at 0xc04e0f96 = ithread_loop+0x1f6 fork_exit(c04e0da0,c7083790,c6896d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6896d70, ebp = 0 --- Tracing command intr pid 11 tid 100010 td 0xc7084000 sched_switch(c7084000,0,109,191,7170b6,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(109,0,c073bec7,52d,c70404f0,...) at 0xc0510680 = mi_switch+0x200 ithread_loop(c70837a0,c6893d38,c073bc0b,343,c7042aa0,...) at 0xc04e0f96 = ithread_loop+0x1f6 fork_exit(c04e0da0,c70837a0,c6893d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6893d70, ebp = 0 --- Tracing command intr pid 11 tid 100009 td 0xc7084250 sched_switch(c7084250,0,109,191,77674170,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(109,0,c073bec7,52d,c7040570,...) at 0xc0510680 = mi_switch+0x200 ithread_loop(c70837b0,c6890d38,c073bc0b,343,c7042aa0,...) at 0xc04e0f96 = ithread_loop+0x1f6 fork_exit(c04e0da0,c70837b0,c6890d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6890d70, ebp = 0 --- Tracing command intr pid 11 tid 100008 td 0xc70844a0 sched_switch(c70844a0,0,109,191,78233802,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(109,0,c073bec7,52d,c70405f0,...) at 0xc0510680 = mi_switch+0x200 ithread_loop(c70837c0,c688dd38,c073bc0b,343,c7042aa0,...) at 0xc04e0f96 = ithread_loop+0x1f6 fork_exit(c04e0da0,c70837c0,c688dd38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc688dd70, ebp = 0 --- Tracing command intr pid 11 tid 100007 td 0xc7044000 fork_trampoline() at 0xc06d6b78 = fork_trampoline Tracing command intr pid 11 tid 100006 td 0xc7044250 sched_switch(c7044250,0,109,191,6e898b90,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(109,0,c073bec7,52d,c70406f0,...) at 0xc0510680 = mi_switch+0x200 ithread_loop(c70837e0,c6887d38,c073bc0b,343,c7042aa0,...) at 0xc04e0f96 = ithread_loop+0x1f6 fork_exit(c04e0da0,c70837e0,c6887d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6887d70, ebp = 0 --- Tracing command idle pid 10 tid 100005 td 0xc70444a0 cpustop_handler(1,c6883c28,c06f4456,80008cce,369e99,...) at 0xc06e7712 = cpustop_handler+0x32 ipi_nmi_handler(80008cce,369e99,0,0,c7042d48,...) at 0xc06e779f = ipi_nmi_handler+0x2f trap(c6883c34) at 0xc06f4456 = trap+0x36 calltrap() at 0xc06d6b0b = calltrap+0x6 --- trap 0x13, eip = 0xc0c23ce5, esp = 0xc6883c74, ebp = 0xc6883c74 --- acpi_cpu_c1(0,ffffffff,0,c0529eb1,c07e1490,...) at 0xc0c23ce5 = acpi_cpu_c1+0x5 acpi_cpu_idle(c6883cb4,c06e25bb,1,c6883cf8,c052a1ae,...) at 0xc0c1ca51 = acpi_cpu_idle+0x1c1 cpu_idle_acpi(1,c6883cf8,c052a1ae,1,c6883cd4,...) at 0xc06e36fb = cpu_idle_acpi+0x1b cpu_idle(1,c6883cd4,c0742824,9fa,c70444a0,...) at 0xc06e25bb = cpu_idle+0x1b sched_idletd(0,c6883d38,c073bc0b,343,c7042d48,...) at 0xc052a1ae = sched_idletd+0x23e fork_exit(c0529f70,0,c6883d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6883d70, ebp = 0 --- Tracing command idle pid 10 tid 100004 td 0xc70446f0 sched_switch(c70446f0,0,60c,18c,776735b2,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(60c,0,c0742824,806,1,...) at 0xc0510680 = mi_switch+0x200 sched_preempt(c70446f0,1,1f4,c6880c74,c06d722e,...) at 0xc05284af = sched_preempt+0x9f ipi_bitmap_handler(c07d0008,28,c0730028,c7081200,c7081218,...) at 0xc06e63b4 = ipi_bitmap_handler+0x34 Xipi_intr_bitmap_handler() at 0xc06d722e = Xipi_intr_bitmap_handler+0x2e --- interrupt, eip = 0xc0c23ce5, esp = 0xc6880c74, ebp = 0xc6880c74 --- acpi_cpu_c1(0,ffffffff,1,c0529eb1,c07e1490,...) at 0xc0c23ce5 = acpi_cpu_c1+0x5 acpi_cpu_idle(c6880cb4,c06e25bb,0,c6880cf8,c052a1ae,...) at 0xc0c1ca51 = acpi_cpu_idle+0x1c1 cpu_idle_acpi(0,c6880cf8,c052a1ae,0,c6880cd4,...) at 0xc06e36fb = cpu_idle_acpi+0x1b cpu_idle(0,c6880cd4,c0742824,9fa,c70446f0,...) at 0xc06e25bb = cpu_idle+0x1b sched_idletd(0,c6880d38,c073bc0b,343,c7042d48,...) at 0xc052a1ae = sched_idletd+0x23e fork_exit(c0529f70,0,c6880d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6880d70, ebp = 0 --- Tracing command idle pid 10 tid 100003 td 0xc7044940 cpustop_handler(4,c687dc28,c06f4456,c0742824,747,...) at 0xc06e7712 = cpustop_handler+0x32 ipi_nmi_handler(c0742824,747,c687dbdc,2,c7042d48,...) at 0xc06e779f = ipi_nmi_handler+0x2f trap(c687dc34) at 0xc06f4456 = trap+0x36 calltrap() at 0xc06d6b0b = calltrap+0x6 --- trap 0x13, eip = 0xc0c23ce5, esp = 0xc687dc74, ebp = 0xc687dc74 --- acpi_cpu_c1(8,ffff0028,2,2,c07d3d00,...) at 0xc0c23ce5 = acpi_cpu_c1+0x5 acpi_cpu_idle(c687dcb4,c06e25bb,1,c687dcf8,c052a1ae,...) at 0xc0c1ca51 = acpi_cpu_idle+0x1c1 cpu_idle_acpi(1,c687dcf8,c052a1ae,1,c687dcd4,...) at 0xc06e36fb = cpu_idle_acpi+0x1b cpu_idle(1,c687dcd4,c0742824,9fa,c7044940,...) at 0xc06e25bb = cpu_idle+0x1b sched_idletd(0,c687dd38,c073bc0b,343,c7042d48,...) at 0xc052a1ae = sched_idletd+0x23e fork_exit(c0529f70,0,c687dd38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc687dd70, ebp = 0 --- Tracing command idle pid 10 tid 100002 td 0xc7044b90 cpustop_handler(8,c687ac28,c06f4456,c0742824,747,...) at 0xc06e7712 = cpustop_handler+0x32 ipi_nmi_handler(c0742824,747,c04f7cf4,3,c7042d48,...) at 0xc06e779f = ipi_nmi_handler+0x2f trap(c687ac34) at 0xc06f4456 = trap+0x36 calltrap() at 0xc06d6b0b = calltrap+0x6 --- trap 0x13, eip = 0xc0c23ce5, esp = 0xc687ac74, ebp = 0xc687ac74 --- acpi_cpu_c1(0,ffffffff,3,c0529eb1,c07e1490,...) at 0xc0c23ce5 = acpi_cpu_c1+0x5 acpi_cpu_idle(c687acb4,c06e25bb,1,c687acf8,c052a1ae,...) at 0xc0c1ca51 = acpi_cpu_idle+0x1c1 cpu_idle_acpi(1,c687acf8,c052a1ae,1,c687acd4,...) at 0xc06e36fb = cpu_idle_acpi+0x1b cpu_idle(1,c687acd4,c0742824,9fa,c7044b90,...) at 0xc06e25bb = cpu_idle+0x1b sched_idletd(0,c687ad38,c073bc0b,343,c7042d48,...) at 0xc052a1ae = sched_idletd+0x23e fork_exit(c0529f70,0,c687ad38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc687ad70, ebp = 0 --- Tracing command init pid 1 tid 100001 td 0xc7046000 sched_switch(c7046000,0,104,191,2c2544f6,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,5c,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7046000,0,c0745097,1a0,5c,...) at 0xc054028f = sleepq_switch+0x15f sleepq_catch_signals(c0745097,160,0,100,100,...) at 0xc0540377 = sleepq_catch_signals+0xb7 sleepq_wait_sig(c7043000,5c,c0747a51,100,0,...) at 0xc0540eb7 = sleepq_wait_sig+0x17 _sleep(c7043000,c7043088,15c,c0747a51,0,...) at 0xc0510b84 = _sleep+0x354 kern_wait(c7046000,ffffffff,c6876c78,0,0,...) at 0xc04dbde6 = kern_wait+0xb46 wait4(c7046000,c6876cf8,c0770332,c0747868,282,...) at 0xc04dbe5b = wait4+0x3b syscall(c6876d38) at 0xc06f41e5 = syscall+0x1f5 Xint0x80_syscall() at 0xc06d6b70 = Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x8054b2b, esp = 0xbfbfe90c, ebp = 0xbfbfe928 --- Tracing command kernel pid 0 tid 100120 td 0xc781d940 sched_switch(c781d940,0,104,191,78233d12,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c781d940,0,c0745097,260,c781d940,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c75991c0,0,c0741770,c0758b0d,0,...) at 0xc0540f63 = sleepq_wait+0x63 msleep_spin(c75991c0,c75991d8,c0758b0d,0,c073f15b,...) at 0xc0510f4d = msleep_spin+0x21d taskqueue_thread_loop(c7f5dd20,e9630d38,c073bc0b,343,c07ccf00,...) at 0xc0542864 = taskqueue_thread_loop+0x94 fork_exit(c05427d0,c7f5dd20,e9630d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe9630d70, ebp = 0 --- Tracing command kernel pid 0 tid 100061 td 0xc7385000 kdb_enter(c0740afa,c0740afa,c073a07e,c6bcbcd0,1,...) at 0xc053618a = kdb_enter+0x3a panic(c073a07e,c07166d9,c7c47940,3574e,c7c425d8,...) at 0xc05077f6 = panic+0x136 deadlkres(0,c6bcbd38,c073bc0b,343,c07ccf00,...) at 0xc04c4a1a = deadlkres+0x27a fork_exit(c04c47a0,0,c6bcbd38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6bcbd70, ebp = 0 --- Tracing command kernel pid 0 tid 100054 td 0xc727c000 sched_switch(c727c000,0,104,191,717c5466,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c727c000,0,c0745097,260,c727c000,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c727ecc0,0,c0741770,c0758b0d,0,...) at 0xc0540f63 = sleepq_wait+0x63 msleep_spin(c727ecc0,c727ecd8,c0758b0d,0,c073f15b,...) at 0xc0510f4d = msleep_spin+0x21d taskqueue_thread_loop(c72a449c,c6b9bd38,c073bc0b,343,c07ccf00,...) at 0xc0542864 = taskqueue_thread_loop+0x94 fork_exit(c05427d0,c72a449c,c6b9bd38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b9bd70, ebp = 0 --- Tracing command kernel pid 0 tid 100030 td 0xc71bc940 sched_switch(c71bc940,0,104,191,4eb8cfaa,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c71bc940,0,c0745097,260,c71bc940,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c7204300,0,c0741770,c0758b0d,0,...) at 0xc0540f63 = sleepq_wait+0x63 msleep_spin(c7204300,c7204318,c0758b0d,0,c073f15b,...) at 0xc0510f4d = msleep_spin+0x21d taskqueue_thread_loop(c7209520,c6b49d38,c073bc0b,343,c07ccf00,...) at 0xc0542864 = taskqueue_thread_loop+0x94 fork_exit(c05427d0,c7209520,c6b49d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6b49d70, ebp = 0 --- Tracing command kernel pid 0 tid 100026 td 0xc71bd4a0 sched_switch(c71bd4a0,0,104,191,1d88da60,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c71bd4a0,0,c0745097,260,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c71a4ac0,0,c0758b0d,0,0,...) at 0xc0540f63 = sleepq_wait+0x63 _sleep(c71a4ac0,c71a4ad8,0,c0758b0d,0,...) at 0xc0510b9b = _sleep+0x36b taskqueue_thread_loop(c07cd7c4,c68c3d38,c073bc0b,343,c07ccf00,...) at 0xc054288a = taskqueue_thread_loop+0xba fork_exit(c05427d0,c07cd7c4,c68c3d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc68c3d70, ebp = 0 --- Tracing command kernel pid 0 tid 100023 td 0xc7084940 sched_switch(c7084940,0,104,191,1d88c428,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7084940,0,c0745097,260,c7084940,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c71a5040,0,c0741770,c0758b0d,0,...) at 0xc0540f63 = sleepq_wait+0x63 msleep_spin(c71a5040,c71a5058,c0758b0d,0,c073f15b,...) at 0xc0510f4d = msleep_spin+0x21d taskqueue_thread_loop(c0c3623c,c68bad38,c073bc0b,343,c07ccf00,...) at 0xc0542864 = taskqueue_thread_loop+0x94 fork_exit(c05427d0,c0c3623c,c68bad38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc68bad70, ebp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xc7084b90 sched_switch(c7084b90,0,104,191,1d88afbb,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c7084b90,0,c0745097,260,c7084b90,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c71a5040,0,c0741770,c0758b0d,0,...) at 0xc0540f63 = sleepq_wait+0x63 msleep_spin(c71a5040,c71a5058,c0758b0d,0,c073f15b,...) at 0xc0510f4d = msleep_spin+0x21d taskqueue_thread_loop(c0c3623c,c68b7d38,c073bc0b,343,c07ccf00,...) at 0xc0542864 = taskqueue_thread_loop+0x94 fork_exit(c05427d0,c0c3623c,c68b7d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc68b7d70, ebp = 0 --- Tracing command kernel pid 0 tid 100021 td 0xc71b7000 sched_switch(c71b7000,0,104,191,1d88982d,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c71b7000,0,c0745097,260,c71b7000,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c71a5040,0,c0741770,c0758b0d,0,...) at 0xc0540f63 = sleepq_wait+0x63 msleep_spin(c71a5040,c71a5058,c0758b0d,0,c073f15b,...) at 0xc0510f4d = msleep_spin+0x21d taskqueue_thread_loop(c0c3623c,c68b4d38,c073bc0b,343,c07ccf00,...) at 0xc0542864 = taskqueue_thread_loop+0x94 fork_exit(c05427d0,c0c3623c,c68b4d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc68b4d70, ebp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xc71b7250 sched_switch(c71b7250,0,104,191,4427f86c,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,0,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c71b7250,0,c0745097,260,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_wait(c71a5080,0,c0758b0d,0,0,...) at 0xc0540f63 = sleepq_wait+0x63 _sleep(c71a5080,c71a5098,0,c0758b0d,0,...) at 0xc0510b9b = _sleep+0x36b taskqueue_thread_loop(c07e1688,c68b1d38,c073bc0b,343,c07ccf00,...) at 0xc054288a = taskqueue_thread_loop+0xba fork_exit(c05427d0,c07e1688,c68b1d38) at 0xc04dddf8 = fork_exit+0xb8 fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc68b1d70, ebp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xc07cd1b0 sched_switch(c07cd1b0,0,104,191,66059445,...) at 0xc05292c0 = sched_switch+0x330 mi_switch(104,0,c0745097,1eb,44,...) at 0xc0510680 = mi_switch+0x200 sleepq_switch(c07cd1b0,0,c0745097,283,0,...) at 0xc054028f = sleepq_switch+0x15f sleepq_timedwait(c07ccf00,44,c07430b5,0,0,...) at 0xc0540e6b = sleepq_timedwait+0x6b _sleep(c07ccf00,0,44,c07430b5,2710,...) at 0xc0510b69 = _sleep+0x339 scheduler(0,101ec00,101ec00,101e000,1025000,...) at 0xc069e96e = scheduler+0x23e mi_startup() at 0xc04c2aa6 = mi_startup+0x96 begin() at 0xc043adc5 = begin+0x2c -- Sincerely, Dmitry nic-hdl: LYNX-RIPE From owner-freebsd-fs@FreeBSD.ORG Thu Jun 17 03:45:54 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AA3981065687; Thu, 17 Jun 2010 03:45:54 +0000 (UTC) (envelope-from hiroshi@soupacific.com) Received: from mail.soupacific.com (mail.soupacific.com [211.19.53.201]) by mx1.freebsd.org (Postfix) with ESMTP id 76A238FC0C; Thu, 17 Jun 2010 03:45:54 +0000 (UTC) Received: from [127.0.0.1] (unknown [192.168.1.239]) by mail.soupacific.com (Postfix) with ESMTP id 819EB6C2B5; Thu, 17 Jun 2010 03:37:24 +0000 (UTC) Message-ID: <4C199A70.2060207@soupacific.com> Date: Thu, 17 Jun 2010 12:45:52 +0900 From: "hiroshi@soupacific.com" User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4 MIME-Version: 1.0 To: Pawel Jakub Dawidek References: <4C14215D.9090304@soupacific.com> <20100613003635.GA60012@icarus.home.lan> <20100613074921.GB1320@garage.freebsd.pl> <4C149A5C.3070401@soupacific.com> <20100613102401.GE1320@garage.freebsd.pl> <86eigavzsg.fsf@kopusha.home.net> <20100614095044.GH1721@garage.freebsd.pl> <868w6hwt2w.fsf@kopusha.home.net> <20100614153746.GN1721@garage.freebsd.pl> <4C165871.6060609@soupacific.com> <20100614214851.GB2498@garage.freebsd.pl> In-Reply-To: <20100614214851.GB2498@garage.freebsd.pl> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: FreeBSD 8.1 and HAST X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jun 2010 03:45:54 -0000 Hi folks ! Couple more question now ! Since my test environment is 1T bytes hdd, so it took long long time to finish first synchronize dirty bytes. When manual start HAST on each node, then it will start synchronize dirty bytes. During synchronization, set rc.conf hast_enable="YES" ucarp_enable="YES" After reboot, In ucarp_up.sh, zpool import -f poolname shows error as ZFS pool import from resource failed: cannot import 'hasthoem': pool is formatted using a newer ZFS version. After completion of synchronization, then switch slave as master. using /etc/vip_up.sh Same ZFS pool import error message appears Also hastctl status shows status: degaraded Mr. Mikolaj Golub ! Did you have this problem before ? Where shoul I check ? Thanks Hiroshi From owner-freebsd-fs@FreeBSD.ORG Thu Jun 17 07:28:46 2010 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0F426106564A for ; Thu, 17 Jun 2010 07:28:46 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 9D8528FC15 for ; Thu, 17 Jun 2010 07:28:45 +0000 (UTC) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id o5H7Sfb5066122 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 17 Jun 2010 10:28:41 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4) with ESMTP id o5H7Sfod011220; Thu, 17 Jun 2010 10:28:41 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4/Submit) id o5H7SfUf011219; Thu, 17 Jun 2010 10:28:41 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 17 Jun 2010 10:28:41 +0300 From: Kostik Belousov To: Dmitry Pryanishnikov Message-ID: <20100617072841.GY13238@deviant.kiev.zoral.com.ua> References: <201006162220.o5GMK3uM002900@freefall.freebsd.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="AQDb5/OyJPRJL1y9" Content-Disposition: inline In-Reply-To: <201006162220.o5GMK3uM002900@freefall.freebsd.org> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-2.3 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_50, DNS_FROM_OPENWHOIS autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-fs@FreeBSD.org Subject: Re: kern/147890: [ufs] [regression] ufs-related lock problem in RELENG_8 (18.04.2010 -> 20.04.2010) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jun 2010 07:28:46 -0000 --AQDb5/OyJPRJL1y9 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Jun 16, 2010 at 10:20:03PM +0000, Dmitry Pryanishnikov wrote: > The following reply was made to PR kern/147890; it has been noted by GNAT= S. >=20 > From: Dmitry Pryanishnikov > To: Bruce Cran > Cc: bug-followup@freebsd.org > Subject: Re: kern/147890: [ufs] [regression] ufs-related lock problem in= =20 > RELENG_8 (18.04.2010 -> 20.04.2010) > Date: Thu, 17 Jun 2010 01:12:17 +0300 >=20 > Hello! > =20 > 2010/6/16 Bruce Cran : > > Could you run the commands shown in > > http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/k= erneldebug-deadlocks.html > > to get more information to help debug the problem please? > =20 > Done - I've rebuilt the kernel from the sources as of 20.04.2010 > with the missing debugging options, repeated the problem, turned I suspect you do not. In particular, DEBUG_VFS_LOCKS, DEBUG_LOCKS and WITNESS seems to be missed. > ddb(4) capture buffer on, issued recommended ddb commands, then called > 'doadump'; then rebooted and extracted capture buffer contents from > the crashdump using ddb(8). Due to missed options, we cannot trace the lock chains and see why the issue happens. > =20 > BTW, is there a way to get this information _without repeating the > problem_ - using only a crashdump from it's previous occurrence? All > data required for analysis should already be there. I suspect not > every deadlock is easy repeatable (like this one)... It is, but you have to understand how locks work, and what you are looking for. > =20 > db> ps > =20 > pid ppid pgrp uid state wmesg wchan cmd > 2143 1846 2143 0 S+ ufs 0xcd466dc8 sync > 2131 1 1911 1000 S ufs 0xcd466dc8 initial thread =2E.. > 18 0 0 0 SL ufs 0xcd466dc8 [syncer] =2E... > Tracing command syncer pid 18 tid 100067 td 0xc7383000 > sched_switch(c7383000,0,104,191,e660f504,...) at 0xc05292c0 =3D sched_sw= itch+0x330 > mi_switch(104,0,c0745097,1eb,50,...) at 0xc0510680 =3D mi_switch+0x200 > sleepq_switch(c7383000,0,c0745097,260,50,...) at 0xc054028f =3D > sleepq_switch+0x15f > sleepq_wait(cd466dc8,50,c07393ba,4,0,...) at 0xc0540f63 =3D sleepq_wait+= 0x63 > __lockmgr_args(cd466dc8,80100,cd466e30,0,0,...) at 0xc04f087f =3D > __lockmgr_args+0xb1f > ffs_lock(e732cb34,c054944b,c0763314,80100,cd466d70,...) at 0xc067e9f1 > =3D ffs_lock+0xa1 > VOP_LOCK1_APV(c07a47c0,e732cb34,c73830a4,c07b7a20,cd466d70,...) at > 0xc070c945 =3D VOP_LOCK1_APV+0xb5 > _vn_lock(cd466d70,80100,c074e6ce,82b,4,...) at 0xc05a2fd8 =3D _vn_lock+0= x78 > vget(cd466d70,80100,c7383000,39a,c763b870,...) at 0xc05969db =3D vget+0x= bb > qsync(c763b870,0,c0761f41,552,4,...) at 0xc0687db7 =3D qsync+0x197 > ffs_sync(c763b870,3,c074e6ce,d6b,c763b870,...) at 0xc067a139 =3D ffs_syn= c+0x349 > sync_fsync(e732cc7c,c0772c36,c76b8778,e732cc7c,c76b86b8,...) at > 0xc059812f =3D sync_fsync+0x18f > VOP_FSYNC_APV(c0799bc0,e732cc7c,c074e6ce,6a5,c7383000,...) at > 0xc070a545 =3D VOP_FSYNC_APV+0xc5 > sync_vnode(c093b618,c093b604,3e8,6d4,4e20,...) at 0xc059840b =3D sync_vn= ode+0x16b > sched_sync(0,e732cd38,c073bc0b,343,c759f7f8,...) at 0xc0598753 =3D > sched_sync+0x273 > fork_exit(c05984e0,0,e732cd38) at 0xc04dddf8 =3D fork_exit+0xb8 > fork_trampoline() at 0xc06d6b80 =3D fork_trampoline+0x8 > --- trap 0, eip =3D 0, esp =3D 0xe732cd70, ebp =3D 0 --- Do you have custoom kernel build, in particular, are quotas compiled in ? What are the drives ? Having to pull the generic information about the problematic system does not make debugging faster. --AQDb5/OyJPRJL1y9 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (FreeBSD) iEYEARECAAYFAkwZzqkACgkQC3+MBN1Mb4h5BACeMbGnVuUNASRzA72BMrr00WVw BegAn3ZcY0q722VOo0G9xItKOttxiuVl =ryzd -----END PGP SIGNATURE----- --AQDb5/OyJPRJL1y9-- From owner-freebsd-fs@FreeBSD.ORG Thu Jun 17 07:42:32 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 10C4D1065676 for ; Thu, 17 Jun 2010 07:42:32 +0000 (UTC) (envelope-from spork@bway.net) Received: from xena.bway.net (xena.bway.net [216.220.96.26]) by mx1.freebsd.org (Postfix) with ESMTP id A603E8FC1A for ; Thu, 17 Jun 2010 07:42:31 +0000 (UTC) Received: (qmail 5149 invoked by uid 0); 17 Jun 2010 07:42:29 -0000 Received: from unknown (HELO ?10.3.2.41?) (spork@96.57.144.66) by smtp.bway.net with (DHE-RSA-AES256-SHA encrypted) SMTP; 17 Jun 2010 07:42:29 -0000 Date: Thu, 17 Jun 2010 03:42:28 -0400 (EDT) From: Charles Sprickman X-X-Sender: spork@hotlap.local To: Pawel Jakub Dawidek In-Reply-To: <20100616132504.GK1739@garage.freebsd.pl> Message-ID: References: <20100616132504.GK1739@garage.freebsd.pl> User-Agent: Alpine 2.00 (OSX 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org Subject: Re: zfs panic "evicting znode" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jun 2010 07:42:32 -0000 On Wed, 16 Jun 2010, Pawel Jakub Dawidek wrote: > On Tue, Jun 15, 2010 at 10:34:20PM -0400, Charles Sprickman wrote: >> Howdy, >> >> I have a box running 8.0-RELEASE that recently started panicing every few >> hours with the following message: >> >> panic: evicting znode 0xa1eafe80 > > Do you have at least backtrace from DDB? Nope. The box had been stable and then paniced at least two times that I'm aware of. On the last one, I drove to the datacenter to see what it dumped on the console. I just built and installed a kernel with ddb/kdb and will reboot next time I'm there. > What kind of workload do you have? The box is essentially a development machine, and after it started acting up, I moved the jails to another host, so for this panic and the one before it, it was essentially idle. > I was no able to find a way to trigger znode_evict_error() call (this is > what you're seeing). You could try changing '#if 1' to '#if 0' in > zfs_znode.c:znode_evict_error(), but I don't think this code path was > ever tested, because as I said, I was unable to trigger it. What is this area of the code doing? Is this part of some type of consistency check that zfs performs? Does it in any way indicate that there's bad metadata that a scrub cannot detect? Thanks, Charles > -- > Pawel Jakub Dawidek http://www.wheelsystems.com > pjd@FreeBSD.org http://www.FreeBSD.org > FreeBSD committer Am I Evil? Yes, I Am! > From owner-freebsd-fs@FreeBSD.ORG Thu Jun 17 08:32:46 2010 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 59B341065670; Thu, 17 Jun 2010 08:32:46 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 539688FC14; Thu, 17 Jun 2010 08:32:45 +0000 (UTC) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id o5H8WdGx079158 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 17 Jun 2010 11:32:39 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4) with ESMTP id o5H8WdOP037644; Thu, 17 Jun 2010 11:32:39 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4/Submit) id o5H8WdCh037643; Thu, 17 Jun 2010 11:32:39 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 17 Jun 2010 11:32:39 +0300 From: Kostik Belousov To: Dmitry Pryanishnikov Message-ID: <20100617083239.GZ13238@deviant.kiev.zoral.com.ua> References: <201006162220.o5GMK3uM002900@freefall.freebsd.org> <20100617072841.GY13238@deviant.kiev.zoral.com.ua> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="pr2uf1USn+1lHhL+" Content-Disposition: inline In-Reply-To: <20100617072841.GY13238@deviant.kiev.zoral.com.ua> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-2.3 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_50, DNS_FROM_OPENWHOIS autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-fs@FreeBSD.org, trasz@FreeBSD.org Subject: Re: kern/147890: [ufs] [regression] ufs-related lock problem in RELENG_8 (18.04.2010 -> 20.04.2010) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jun 2010 08:32:46 -0000 --pr2uf1USn+1lHhL+ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Jun 17, 2010 at 10:28:41AM +0300, Kostik Belousov wrote: > On Wed, Jun 16, 2010 at 10:20:03PM +0000, Dmitry Pryanishnikov wrote: > > The following reply was made to PR kern/147890; it has been noted by GN= ATS. > >=20 > > From: Dmitry Pryanishnikov > > To: Bruce Cran > > Cc: bug-followup@freebsd.org > > Subject: Re: kern/147890: [ufs] [regression] ufs-related lock problem i= n=20 > > RELENG_8 (18.04.2010 -> 20.04.2010) > > Date: Thu, 17 Jun 2010 01:12:17 +0300 > >=20 > > Hello! > > =20 > > 2010/6/16 Bruce Cran : > > > Could you run the commands shown in > > > http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook= /kerneldebug-deadlocks.html > > > to get more information to help debug the problem please? > > =20 > > Done - I've rebuilt the kernel from the sources as of 20.04.2010 > > with the missing debugging options, repeated the problem, turned > I suspect you do not. In particular, DEBUG_VFS_LOCKS, DEBUG_LOCKS > and WITNESS seems to be missed. >=20 > > ddb(4) capture buffer on, issued recommended ddb commands, then called > > 'doadump'; then rebooted and extracted capture buffer contents from > > the crashdump using ddb(8). > Due to missed options, we cannot trace the lock chains and see why the > issue happens. >=20 > > =20 > > BTW, is there a way to get this information _without repeating the > > problem_ - using only a crashdump from it's previous occurrence? All > > data required for analysis should already be there. I suspect not > > every deadlock is easy repeatable (like this one)... > It is, but you have to understand how locks work, and what you are > looking for. >=20 > > =20 > > db> ps > > =20 > > pid ppid pgrp uid state wmesg wchan cmd > > 2143 1846 2143 0 S+ ufs 0xcd466dc8 sync > > 2131 1 1911 1000 S ufs 0xcd466dc8 initial thread > ... > > 18 0 0 0 SL ufs 0xcd466dc8 [syncer] > .... > > Tracing command syncer pid 18 tid 100067 td 0xc7383000 > > sched_switch(c7383000,0,104,191,e660f504,...) at 0xc05292c0 =3D sched_= switch+0x330 > > mi_switch(104,0,c0745097,1eb,50,...) at 0xc0510680 =3D mi_switch+0x200 > > sleepq_switch(c7383000,0,c0745097,260,50,...) at 0xc054028f =3D > > sleepq_switch+0x15f > > sleepq_wait(cd466dc8,50,c07393ba,4,0,...) at 0xc0540f63 =3D sleepq_wai= t+0x63 > > __lockmgr_args(cd466dc8,80100,cd466e30,0,0,...) at 0xc04f087f =3D > > __lockmgr_args+0xb1f > > ffs_lock(e732cb34,c054944b,c0763314,80100,cd466d70,...) at 0xc067e9f1 > > =3D ffs_lock+0xa1 > > VOP_LOCK1_APV(c07a47c0,e732cb34,c73830a4,c07b7a20,cd466d70,...) at > > 0xc070c945 =3D VOP_LOCK1_APV+0xb5 > > _vn_lock(cd466d70,80100,c074e6ce,82b,4,...) at 0xc05a2fd8 =3D _vn_lock= +0x78 > > vget(cd466d70,80100,c7383000,39a,c763b870,...) at 0xc05969db =3D vget+= 0xbb > > qsync(c763b870,0,c0761f41,552,4,...) at 0xc0687db7 =3D qsync+0x197 > > ffs_sync(c763b870,3,c074e6ce,d6b,c763b870,...) at 0xc067a139 =3D ffs_s= ync+0x349 > > sync_fsync(e732cc7c,c0772c36,c76b8778,e732cc7c,c76b86b8,...) at > > 0xc059812f =3D sync_fsync+0x18f > > VOP_FSYNC_APV(c0799bc0,e732cc7c,c074e6ce,6a5,c7383000,...) at > > 0xc070a545 =3D VOP_FSYNC_APV+0xc5 > > sync_vnode(c093b618,c093b604,3e8,6d4,4e20,...) at 0xc059840b =3D sync_= vnode+0x16b > > sched_sync(0,e732cd38,c073bc0b,343,c759f7f8,...) at 0xc0598753 =3D > > sched_sync+0x273 > > fork_exit(c05984e0,0,e732cd38) at 0xc04dddf8 =3D fork_exit+0xb8 > > fork_trampoline() at 0xc06d6b80 =3D fork_trampoline+0x8 > > --- trap 0, eip =3D 0, esp =3D 0xe732cd70, ebp =3D 0 --- > Do you have custoom kernel build, in particular, are quotas compiled in ? > What are the drives ? >=20 > Having to pull the generic information about the problematic system > does not make debugging faster. What _could_ happen in your case is the relock in ufs_accessx for QUOTA. ufs_delete_denied() calls VOP_ACCESSX for both parent directory and target vnode while parent directory is locked. If source directory is not exclusively locked, LOR can happen, since ufs_accessx has to unlock and lock directory vnode. I am not sure how many similar cases of VOP_ACCESS[X] is places in the problematic locations. Try the (untested) patch below. diff --git a/sys/ufs/ufs/ufs_lookup.c b/sys/ufs/ufs/ufs_lookup.c index 0030c52..c779491 100644 --- a/sys/ufs/ufs/ufs_lookup.c +++ b/sys/ufs/ufs/ufs_lookup.c @@ -77,6 +77,32 @@ SYSCTL_INT(_debug, OID_AUTO, dircheck, CTLFLAG_RW, &dirc= hk, 0, ""); /* true if old FS format...*/ #define OFSFMT(vp) ((vp)->v_mount->mnt_maxsymlinklen <=3D 0) =20 +#ifdef QUOTA +static int +ufs_lookup_upgrade_lock(struct vnode *vp) +{ + int error; + + ASSERT_VOP_LOCKED(vp, __FUNCTION__); + if (VOP_ISLOCKED(vp) =3D=3D LK_EXCLUSIVE) + return (0); + + error =3D 0; + + /* + * Upgrade vnode lock, since getinoquota() + * requires exclusive lock to modify inode. + */ + vhold(vp); + vn_lock(vp, LK_UPGRADE | LK_RETRY); + VI_LOCK(vp); + if (vp->v_iflag & VI_DOOMED) + error =3D ENOENT; + vdropl(vp); + return (error); +} +#endif + static int ufs_delete_denied(struct vnode *vdp, struct vnode *tdp, struct ucred *cred, struct thread *td) @@ -232,6 +258,13 @@ ufs_lookup_ino(struct vnode *vdp, struct vnode **vpp, = struct componentname *cnp, vnode_create_vobject(vdp, DIP(dp, i_size), cnp->cn_thread); =20 bmask =3D VFSTOUFS(vdp->v_mount)->um_mountp->mnt_stat.f_iosize - 1; +#ifdef QUOTA + if ((nameiop =3D=3D DELETE || nameiop =3D=3D RENAME) && (flags & ISLASTCN= )) { + error =3D ufs_lookup_upgrade_lock(vdp); + if (error !=3D 0) + return (error); + } +#endif =20 restart: bp =3D NULL; --pr2uf1USn+1lHhL+ Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (FreeBSD) iEYEARECAAYFAkwZ3acACgkQC3+MBN1Mb4hShQCeMwojotSEFN+ENxpWsRTP39uV SycAoI7g34eITBqfnVZ1tFhUqzDVtHro =lAI2 -----END PGP SIGNATURE----- --pr2uf1USn+1lHhL+-- From owner-freebsd-fs@FreeBSD.ORG Thu Jun 17 10:34:52 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 312EE1065673; Thu, 17 Jun 2010 10:34:52 +0000 (UTC) (envelope-from to.my.trociny@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.152]) by mx1.freebsd.org (Postfix) with ESMTP id 3F50D8FC1A; Thu, 17 Jun 2010 10:34:51 +0000 (UTC) Received: by fg-out-1718.google.com with SMTP id d23so1404475fga.13 for ; Thu, 17 Jun 2010 03:34:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:to:cc:subject:references :x-comment-to:date:in-reply-to:message-id:user-agent:mime-version :content-type; bh=E2oRHRH2nkfRVDbRKgcl4L9Bzb2Rft7EHfE7mQUDWoU=; b=XkexgFUWv6Y9gOjYSNePnYlsqyuvnOa0DUNbOBfhHrZ2UyCba7nY/Y/ucgApqPWYKM Evvgh4hAy//LGiCOHO/iPdr0G41qmZvda7hkPez/OvU8iTm4rBA3tXP77PX164FPQGgW WSMQ9IK2lgeI9O2ROYD73un+JG87GPDH45SCM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:cc:subject:references:x-comment-to:date:in-reply-to :message-id:user-agent:mime-version:content-type; b=esfoXW0JBB2F1/QYbjykfgjjkaefLBmGUE4etH/rDKT3ywc+cm7R4Ea31FE1JJ3TfG ue00Vz9zDRA1vodggI4UKeoMmgfOn78AZOarbUdNhSOch294YwBEzfGsya0vkuXEpQ56 E3n6IVbr6yhEbDyaiuPW085L+00iqYDF+jqMA= Received: by 10.204.137.216 with SMTP id x24mr7510280bkt.168.1276770890173; Thu, 17 Jun 2010 03:34:50 -0700 (PDT) Received: from localhost ([95.69.160.52]) by mx.google.com with ESMTPS id v3sm33487206bkz.16.2010.06.17.03.34.47 (version=TLSv1/SSLv3 cipher=RC4-MD5); Thu, 17 Jun 2010 03:34:48 -0700 (PDT) From: Mikolaj Golub To: "hiroshi\@soupacific.com" References: <4C14215D.9090304@soupacific.com> <20100613003635.GA60012@icarus.home.lan> <20100613074921.GB1320@garage.freebsd.pl> <4C149A5C.3070401@soupacific.com> <20100613102401.GE1320@garage.freebsd.pl> <86eigavzsg.fsf@kopusha.home.net> <20100614095044.GH1721@garage.freebsd.pl> <868w6hwt2w.fsf@kopusha.home.net> <20100614153746.GN1721@garage.freebsd.pl> <4C165871.6060609@soupacific.com> <20100614214851.GB2498@garage.freebsd.pl> <4C199A70.2060207@soupacific.com> X-Comment-To: hiroshi@soupacific.com Date: Thu, 17 Jun 2010 13:34:44 +0300 In-Reply-To: <4C199A70.2060207@soupacific.com> (hiroshi@soupacific.com's message of "Thu, 17 Jun 2010 12:45:52 +0900") Message-ID: <86631ilzgb.fsf@kopusha.home.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.2 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: freebsd-fs@freebsd.org, Pawel Jakub Dawidek Subject: Re: FreeBSD 8.1 and HAST X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jun 2010 10:34:52 -0000 On Thu, 17 Jun 2010 12:45:52 +0900 hiroshi@soupacific.com wrote: h> Hi folks ! h> Couple more question now ! Since my test environment is 1T bytes hdd, h> so it took long long time to finish first synchronize dirty bytes. h> When manual start HAST on each node, then it will start synchronize h> dirty bytes. During synchronization, h> set rc.conf h> hast_enable="YES" h> ucarp_enable="YES" h> After reboot, h> In ucarp_up.sh, zpool import -f poolname shows error as h> ZFS pool import from resource failed: cannot import 'hasthoem': pool h> is formatted using a newer ZFS version. h> After completion of synchronization, then switch slave as h> master. using /etc/vip_up.sh h> Same ZFS pool import error message appears h> Also h> hastctl status shows h> status: degaraded h> Mr. Mikolaj Golub ! Did you have this problem before ? I didn't. But as I wrote earlier I worked with HAST on CURRENT. So I just checked if there is some issue with ZFS pools on HAST 8-STABLE. I created 400Mb md device, configured HAST to use it and created zpool. Manual switching to failover: host1: zpool export -f storage hastctl role secondary storage host2: hastctl role primary all zpool import -f storage There were no any issues. Just a note, as I don't have many 8-STABLE boxes this HAST has been created between my host in the office and my laptop at home. Very nice to see that it works via WAN :-) So, can you try recreating ZFS pool to see if "zpool import" issue has gone? Please provide here hastctl status on both nodes and hastctl.cfg. Then we might ask you about logs :-). I would recommend playing at first without ucarp -- switching to failover manully. In this way you can be sure that HAST works and also be more familar how it works. After this you can try ucarp. I have not used ucarp myself -- I have been using our own application for failure detection and initiating switching to failover. -- Mikolaj Golub From owner-freebsd-fs@FreeBSD.ORG Thu Jun 17 11:19:06 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B78F11065675; Thu, 17 Jun 2010 11:19:06 +0000 (UTC) (envelope-from hiroshi@soupacific.com) Received: from mail.soupacific.com (mail.soupacific.com [211.19.53.201]) by mx1.freebsd.org (Postfix) with ESMTP id 7F1078FC25; Thu, 17 Jun 2010 11:19:06 +0000 (UTC) Received: from [127.0.0.1] (unknown [192.168.1.239]) by mail.soupacific.com (Postfix) with ESMTP id 993996C4F1; Thu, 17 Jun 2010 11:10:36 +0000 (UTC) Message-ID: <4C1A04A7.9050107@soupacific.com> Date: Thu, 17 Jun 2010 20:19:03 +0900 From: "hiroshi@soupacific.com" User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4 MIME-Version: 1.0 To: Mikolaj Golub References: <4C14215D.9090304@soupacific.com><20100613003635.GA60012@icarus.home.lan><20100613074921.GB1320@garage.freebsd.pl><4C149A5C.3070401@soupacific.com><20100613102401.GE1320@garage.freebsd.pl><86eigavzsg.fsf@kopusha.home.net><20100614095044.GH1721@garage.freebsd.pl><868w6hwt2w.fsf@kopusha.home.net><20100614153746.GN1721@garage.freebsd.pl><4C165871.6060609@soupacific.com><20100614214851.GB2498@garage.freebsd.pl><4C199A70.2060207@soupacific.com> <86631ilzgb.fsf@kopusha.home.net> In-Reply-To: <86631ilzgb.fsf@kopusha.home.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, Pawel Jakub Dawidek Subject: Re: FreeBSD 8.1 and HAST X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jun 2010 11:19:06 -0000 Huuum .... > I didn't. But as I wrote earlier I worked with HAST on CURRENT. So I just > checked if there is some issue with ZFS pools on HAST 8-STABLE. I created > 400Mb md device, configured HAST to use it and created zpool. Manual switching > to failover: > > host1: > zpool export -f storage > hastctl role secondary storage > > host2: > hastctl role primary all > zpool import -f storage > > There were no any issues. Almost same manner I did too. So I now put debug print to zpool to check version's value. It might take a while to finish. > > Just a note, as I don't have many 8-STABLE boxes this HAST has been created > between my host in the office and my laptop at home. Very nice to see that it > works via WAN :-) > Sound great ! After compilation I will check > So, can you try recreating ZFS pool to see if "zpool import" issue has gone? > > Please provide here hastctl status on both nodes and hastctl.cfg. Then we > might ask you about logs :-). > > I would recommend playing at first without ucarp -- switching to failover > manully. In this way you can be sure that HAST works and also be more familar > how it works. After this you can try ucarp. I have not used ucarp myself -- I > have been using our own application for failure detection and initiating > switching to failover. > I used to use CARP untill now. Thanks Hiroshi From owner-freebsd-fs@FreeBSD.ORG Thu Jun 17 14:33:38 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 41B14106566B for ; Thu, 17 Jun 2010 14:33:38 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello089077043238.chello.pl [89.77.43.238]) by mx1.freebsd.org (Postfix) with ESMTP id 67A928FC14 for ; Thu, 17 Jun 2010 14:33:36 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id D519C45E94; Thu, 17 Jun 2010 16:33:32 +0200 (CEST) Received: from localhost (pdawidek.wheel.pl [10.0.1.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id CA78145EEB; Thu, 17 Jun 2010 16:33:18 +0200 (CEST) Date: Thu, 17 Jun 2010 16:33:09 +0200 From: Pawel Jakub Dawidek To: "hiroshi@soupacific.com" Message-ID: <20100617143309.GE1672@garage.freebsd.pl> References: <20100613074921.GB1320@garage.freebsd.pl> <4C149A5C.3070401@soupacific.com> <20100613102401.GE1320@garage.freebsd.pl> <86eigavzsg.fsf@kopusha.home.net> <20100614095044.GH1721@garage.freebsd.pl> <868w6hwt2w.fsf@kopusha.home.net> <20100614153746.GN1721@garage.freebsd.pl> <4C165871.6060609@soupacific.com> <20100614214851.GB2498@garage.freebsd.pl> <4C199A70.2060207@soupacific.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="4Epv4kl9IRBfg3rk" Content-Disposition: inline In-Reply-To: <4C199A70.2060207@soupacific.com> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT amd64 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=4.5 tests=ALL_TRUSTED,BAYES_00, TO_ADDRESS_EQ_REAL autolearn=ham version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: FreeBSD 8.1 and HAST X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jun 2010 14:33:38 -0000 --4Epv4kl9IRBfg3rk Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Jun 17, 2010 at 12:45:52PM +0900, hiroshi@soupacific.com wrote: > Hi folks ! >=20 > Couple more question now ! Since my test environment is 1T bytes hdd, so= =20 > it took long long time to finish first synchronize dirty bytes. >=20 > When manual start HAST on each node, then it will start synchronize=20 > dirty bytes. During synchronization, > set rc.conf >=20 > hast_enable=3D"YES" > ucarp_enable=3D"YES" >=20 > After reboot, > In ucarp_up.sh, zpool import -f poolname shows error as >=20 > ZFS pool import from resource failed: cannot import 'hasthoem': pool is= =20 > formatted using a newer ZFS version. You probably created pool on FreeBSD-CURRENT and now trying to use it on 8-STABLE. There was a version bump from v13 to v14 at some point. You can see your pool version by running: # zpool get version --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --4Epv4kl9IRBfg3rk Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkwaMiQACgkQForvXbEpPzRpSwCgrQ2OQkbrRx4I+h27ApisL9/E FRIAoOmf3CZFtrwiB1wsGhzvjKaxVHJ1 =W+0R -----END PGP SIGNATURE----- --4Epv4kl9IRBfg3rk-- From owner-freebsd-fs@FreeBSD.ORG Thu Jun 17 16:01:39 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EE934106564A for ; Thu, 17 Jun 2010 16:01:39 +0000 (UTC) (envelope-from lynx.ripe@gmail.com) Received: from mail-yw0-f191.google.com (mail-yw0-f191.google.com [209.85.211.191]) by mx1.freebsd.org (Postfix) with ESMTP id A6CFC8FC08 for ; Thu, 17 Jun 2010 16:01:38 +0000 (UTC) Received: by ywh29 with SMTP id 29so97899ywh.27 for ; Thu, 17 Jun 2010 09:01:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=vVbnLD+A/xvULHoDjF89ER6ydX8NVwq1t6DrWrEJEpg=; b=wL62lRNloMu1iGZq31QFIs8eIqrgcVKTGeZglX5J7WJxSxaf8XCndAyiucizWFM56q ctLamELe7BIh0JoC70gJsjzQwkKXJakaRTJy9rMtutYkT9rNGcb7hXzLuiOhMOnO0EDb 5/Vg2cQyrdeUWwH7Zy5v/gi8AaF3vW6XOsF9Y= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=PYpYb8GJDbKeyTFmThvqZ9tyZMij3+FyylhxZqPDYd3J6bfrKhrK1IDsvpLX9bs32S vIdNqcDa+qs17J2TZ3lJwcef6yQ2dHcr0N4NZJP/po8wkiRI5PE+6vjeBN5fa5bAdNef jK36OmbBRV+Smf03+9HDAGdXPM9hMMQG5nnxw= MIME-Version: 1.0 Received: by 10.101.130.14 with SMTP id h14mr8835670ann.142.1276788924343; Thu, 17 Jun 2010 08:35:24 -0700 (PDT) Received: by 10.231.146.199 with HTTP; Thu, 17 Jun 2010 08:35:24 -0700 (PDT) In-Reply-To: References: <201006162220.o5GMK3uM002900@freefall.freebsd.org> <20100617072841.GY13238@deviant.kiev.zoral.com.ua> Date: Thu, 17 Jun 2010 18:35:24 +0300 Message-ID: From: Dmitry Pryanishnikov To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Subject: Fwd: kern/147890: [ufs] [regression] ufs-related lock problem in RELENG_8 (18.04.2010 -> 20.04.2010) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jun 2010 16:01:40 -0000 I'm sorry for not replying to all recipients... ---------- Forwarded message ---------- From: Dmitry Pryanishnikov Date: 2010/6/17 Subject: Re: kern/147890: [ufs] [regression] ufs-related lock problem in RELENG_8 (18.04.2010 -> 20.04.2010) To: Kostik Belousov Hello! 2010/6/17 Kostik Belousov : > On Wed, Jun 16, 2010 at 10:20:03PM +0000, Dmitry Pryanishnikov wrote: >> The following reply was made to PR kern/147890; it has been noted by GNA= TS. >> >> From: Dmitry Pryanishnikov >> To: Bruce Cran >> Cc: bug-followup@freebsd.org >> Subject: Re: kern/147890: [ufs] [regression] ufs-related lock problem in >> =A0 =A0 =A0 RELENG_8 (18.04.2010 -> 20.04.2010) >> Date: Thu, 17 Jun 2010 01:12:17 +0300 >> >> =A0Hello! >> >> =A02010/6/16 Bruce Cran : >> =A0> Could you run the commands shown in >> =A0> http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handboo= k/kerneldebug-deadlocks.html >> =A0> to get more information to help debug the problem please? >> >> =A0 =A0Done - I've rebuilt the kernel from the sources as of 20.04.2010 >> =A0with the missing debugging options, repeated the problem, turned > I suspect you do not. In particular, DEBUG_VFS_LOCKS, DEBUG_LOCKS > and WITNESS seems to be missed. =A0Actually they are present - here is the complete config of this kernel: # # DP, BSF8, 2010-06-16 ident =A0 =A0 =A0 =A0 =A0 LYNX makeoptions =A0 =A0 DEBUG=3D-g =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0#Build kernel= with gdb(1) debug symbols # This allows you to actually store this configuration file into # the kernel binary itself. See config(8) for more details. # options =A0 =A0 =A0 =A0 INCLUDE_CONFIG_FILE =A0 =A0 # Include this file in = kernel ##################################################################### # CPU OPTIONS cpu =A0 =A0 =A0 =A0 =A0 =A0 I686_CPU =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0# aka P= entium Pro(tm) ##################################################################### # Scheduler options: options =A0 =A0 =A0 =A0 SCHED_ULE ##################################################################### # SMP OPTIONS: options =A0 =A0 =A0 =A0 SMP =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 # Symme= tric MultiProcessor Kernel # SMP Debugging Options: options =A0 =A0 =A0 =A0 PREEMPTION =A0 =A0 =A0 =A0 =A0 =A0 =A0# Enable kern= el thread preemption options =A0 =A0 =A0 =A0 MUTEX_DEBUG options =A0 =A0 =A0 =A0 WITNESS device =A0 =A0 =A0 =A0 =A0apic =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0# I/O= apic ##################################################################### # COMPATIBILITY OPTIONS options =A0 =A0 =A0 =A0 COMPAT_FREEBSD4 options =A0 =A0 =A0 =A0 COMPAT_FREEBSD5 options =A0 =A0 =A0 =A0 COMPAT_FREEBSD6 options =A0 =A0 =A0 =A0 COMPAT_FREEBSD7 options =A0 =A0 =A0 =A0 SYSVSHM options =A0 =A0 =A0 =A0 SYSVSEM options =A0 =A0 =A0 =A0 SYSVMSG ##################################################################### # DEBUGGING OPTIONS options =A0 =A0 =A0 =A0 KDB options =A0 =A0 =A0 =A0 KDB_TRACE options =A0 =A0 =A0 =A0 DDB options =A0 =A0 =A0 =A0 DDB_NUMSYM options =A0 =A0 =A0 =A0 BREAK_TO_DEBUGGER =A0 =A0 =A0 # BREAK on a serial c= onsole goes to DDB options =A0 =A0 =A0 =A0 SYSCTL_DEBUG options =A0 =A0 =A0 =A0 KTRACE =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0#kernel t= racing options =A0 =A0 =A0 =A0 INVARIANTS options =A0 =A0 =A0 =A0 INVARIANT_SUPPORT options =A0 =A0 =A0 =A0 DEBUG_LOCKS options =A0 =A0 =A0 =A0 DEBUG_VFS_LOCKS options =A0 =A0 =A0 =A0 DIAGNOSTIC options =A0 =A0 =A0 =A0 DEADLKRES =A0 =A0 =A0 =A0 =A0 =A0 =A0 #software dea= dlock resolver thread ##################################################################### # NETWORKING OPTIONS options =A0 =A0 =A0 =A0 INET =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0#Intern= et communications protocols options =A0 =A0 =A0 =A0 INET6 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 #IPv6 com= munications protocols options =A0 =A0 =A0 =A0 IPSEC =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 #IP secur= ity (requires device crypto) options =A0 =A0 =A0 =A0 IPSEC_NAT_T =A0 =A0 =A0 =A0 =A0 =A0 #NAT-T support,= UDP encap of ESP device =A0 =A0 =A0 =A0 =A0ether =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 #Generi= c Ethernet device =A0 =A0 =A0 =A0 =A0loop =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0#Netw= ork loopback device device =A0 =A0 =A0 =A0 =A0bpf =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 #Berk= eley packet filter device =A0 =A0 =A0 =A0 =A0enc =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 #IPse= c interface options =A0 =A0 =A0 =A0 IPFIREWALL =A0 =A0 =A0 =A0 =A0 =A0 =A0#firewall options =A0 =A0 =A0 =A0 IPFIREWALL_VERBOSE =A0 =A0 =A0#enable logging to sy= slogd(8) options =A0 =A0 =A0 =A0 IPFIREWALL_VERBOSE_LIMIT=3D100 =A0 =A0#limit verbos= ity options =A0 =A0 =A0 =A0 IPFIREWALL_FORWARD =A0 =A0 =A0#packet destination c= hanges options =A0 =A0 =A0 =A0 IPFIREWALL_NAT =A0 =A0 =A0 =A0 =A0#ipfw kernel nat = support options =A0 =A0 =A0 =A0 LIBALIAS =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0# libalias = library, performing NAT options =A0 =A0 =A0 =A0 BPF_JITTER =A0 =A0 =A0 =A0 =A0 =A0 =A0#support for = BPF just-in-time compiler ##################################################################### # FILESYSTEM OPTIONS options =A0 =A0 =A0 =A0 FFS =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 #Fast f= ilesystem options =A0 =A0 =A0 =A0 SOFTUPDATES options =A0 =A0 =A0 =A0 UFS_EXTATTR options =A0 =A0 =A0 =A0 UFS_ACL options =A0 =A0 =A0 =A0 UFS_DIRHASH options =A0 =A0 =A0 =A0 UFS_GJOURNAL =A0 =A0 =A0 =A0 =A0 =A0# Gjournal-base= d UFS journaling support. options =A0 =A0 =A0 =A0 QUOTA =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 #enable d= isk quotas options =A0 =A0 =A0 =A0 SUIDDIR =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 #The suiddi= r hack ##################################################################### # POSIX P1003.1B options =A0 =A0 =A0 =A0 _KPOSIX_PRIORITY_SCHEDULING options =A0 =A0 =A0 =A0 P1003_1B_SEMAPHORES ##################################################################### # MISCELLANEOUS DEVICES AND OPTIONS device =A0 =A0 =A0 =A0 =A0pty =A0 =A0 =A0 =A0 =A0 =A0 #Pseudo ttys # Size of the kernel message buffer. =A0Should be N * pagesize. options =A0 =A0 =A0 =A0 MSGBUF_SIZE=3D131072 ##################################################################### # crypto subsystem device =A0 =A0 =A0 =A0 =A0crypto =A0 =A0 =A0 =A0 =A0# core crypto support ##################################################################### # HARDWARE DEVICE CONFIGURATION # Bus support. device =A0 =A0 =A0 =A0 =A0pci # The keyboard controller; it controls the keyboard and the PS/2 mouse. device =A0 =A0 =A0 =A0 =A0atkbdc =A0 =A0 =A0 =A0 =A0# AT keyboard controlle= r device =A0 =A0 =A0 =A0 =A0atkbd =A0 =A0 =A0 =A0 =A0 # AT keyboard device =A0 =A0 =A0 =A0 =A0psm =A0 =A0 =A0 =A0 =A0 =A0 # PS/2 mouse options =A0 =A0 =A0 =A0 KBD_INSTALL_CDEV =A0 =A0 =A0 =A0# install a CDEV en= try in /dev device =A0 =A0 =A0 =A0 =A0vga =A0 =A0 =A0 =A0 =A0 =A0 # VGA video card driv= er device =A0 =A0 =A0 =A0 =A0splash =A0 =A0 =A0 =A0 =A0# Splash screen and scr= een saver support options =A0 =A0 =A0 =A0 X86BIOS =A0 =A0 =A0 =A0 # x86 real mode BIOS emulat= or, required by atkbdc/dpms/vesa options =A0 =A0 =A0 =A0 PRINTF_BUFR_SIZE=3D128 =A0 =A0# Prevent printf outp= ut being interspersed. # The syscons console driver (SCO color console compatible). device =A0 =A0 =A0 =A0 =A0sc options =A0 =A0 =A0 =A0 SC_HISTORY_SIZE=3D20000 options =A0 =A0 =A0 =A0 SC_TWOBUTTON_MOUSE #options =A0 =A0 =A0 =A0TEKEN_UTF8 =A0 =A0 =A0 =A0 =A0 =A0 =A0# UTF-8 outpu= t handling #options =A0 =A0 =A0 =A0TEKEN_XTERM =A0 =A0 =A0 =A0 =A0 =A0 # xterm-style t= erminal emulation device =A0 =A0 =A0 =A0 =A0uart =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0# Gen= eric UART driver # ATA and ATAPI devices device =A0 =A0 =A0 =A0 =A0ata device =A0 =A0 =A0 =A0 =A0atadisk =A0 =A0 =A0 =A0 # ATA disk drives options =A0 =A0 =A0 =A0 ATA_STATIC_ID =A0 # Static device numbering # SCSI peripherals device =A0 =A0 =A0 =A0 =A0scbus =A0 =A0 =A0 =A0 =A0 # SCSI bus (required fo= r SCSI) device =A0 =A0 =A0 =A0 =A0da =A0 =A0 =A0 =A0 =A0 =A0 =A0# Direct Access (di= sks) device =A0 =A0 =A0 =A0 =A0cd =A0 =A0 =A0 =A0 =A0 =A0 =A0# CD device =A0 =A0 =A0 =A0 =A0pass =A0 =A0 =A0 =A0 =A0 =A0# Passthrough device = (direct SCSI access) # Sound options =A0 =A0 =A0 =A0 SND_PCM_64 =A0 =A0 =A0# Process 32bit samples throu= gh 64bit integer Also I've commented out 'debug.witness.watch=3D0' in /boot/loader.conf before booting this kernel. > >> =A0ddb(4) capture buffer on, issued recommended ddb commands, then calle= d >> =A0'doadump'; then rebooted and extracted capture buffer contents from >> =A0the crashdump using ddb(8). > Due to missed options, we cannot trace the lock chains and see why the > issue happens. =A0Please specify what options are still missing in my config. >> >> =A0db> ps >> >> =A0 =A0pid =A0ppid =A0pgrp =A0 uid =A0 state =A0 wmesg =A0 =A0 wchan =A0= =A0cmd >> =A0 2143 =A01846 =A02143 =A0 =A0 0 =A0S+ =A0 =A0 =A0ufs =A0 =A0 =A00xcd4= 66dc8 sync >> =A0 2131 =A0 =A0 1 =A01911 =A01000 =A0S =A0 =A0 =A0 ufs =A0 =A0 =A00xcd4= 66dc8 initial thread > ... >> =A0 =A0 18 =A0 =A0 0 =A0 =A0 0 =A0 =A0 0 =A0SL =A0 =A0 =A0ufs =A0 =A0 = =A00xcd466dc8 [syncer] > .... >> =A0Tracing command syncer pid 18 tid 100067 td 0xc7383000 >> =A0sched_switch(c7383000,0,104,191,e660f504,...) at 0xc05292c0 =3D sched= _switch+0x330 >> =A0mi_switch(104,0,c0745097,1eb,50,...) at 0xc0510680 =3D mi_switch+0x20= 0 >> =A0sleepq_switch(c7383000,0,c0745097,260,50,...) at 0xc054028f =3D >> =A0sleepq_switch+0x15f >> =A0sleepq_wait(cd466dc8,50,c07393ba,4,0,...) at 0xc0540f63 =3D sleepq_wa= it+0x63 >> =A0__lockmgr_args(cd466dc8,80100,cd466e30,0,0,...) at 0xc04f087f =3D >> =A0__lockmgr_args+0xb1f >> =A0ffs_lock(e732cb34,c054944b,c0763314,80100,cd466d70,...) at 0xc067e9f1 >> =A0=3D ffs_lock+0xa1 >> =A0VOP_LOCK1_APV(c07a47c0,e732cb34,c73830a4,c07b7a20,cd466d70,...) at >> =A00xc070c945 =3D VOP_LOCK1_APV+0xb5 >> =A0_vn_lock(cd466d70,80100,c074e6ce,82b,4,...) at 0xc05a2fd8 =3D _vn_loc= k+0x78 >> =A0vget(cd466d70,80100,c7383000,39a,c763b870,...) at 0xc05969db =3D vget= +0xbb >> =A0qsync(c763b870,0,c0761f41,552,4,...) at 0xc0687db7 =3D qsync+0x197 >> =A0ffs_sync(c763b870,3,c074e6ce,d6b,c763b870,...) at 0xc067a139 =3D ffs_= sync+0x349 >> =A0sync_fsync(e732cc7c,c0772c36,c76b8778,e732cc7c,c76b86b8,...) at >> =A00xc059812f =3D sync_fsync+0x18f >> =A0VOP_FSYNC_APV(c0799bc0,e732cc7c,c074e6ce,6a5,c7383000,...) at >> =A00xc070a545 =3D VOP_FSYNC_APV+0xc5 >> =A0sync_vnode(c093b618,c093b604,3e8,6d4,4e20,...) at 0xc059840b =3D sync= _vnode+0x16b >> =A0sched_sync(0,e732cd38,c073bc0b,343,c759f7f8,...) at 0xc0598753 =3D >> =A0sched_sync+0x273 >> =A0fork_exit(c05984e0,0,e732cd38) at 0xc04dddf8 =3D fork_exit+0xb8 >> =A0fork_trampoline() at 0xc06d6b80 =3D fork_trampoline+0x8 >> =A0--- trap 0, eip =3D 0, esp =3D 0xe732cd70, ebp =3D 0 --- > Do you have custoom kernel build, in particular, are quotas compiled in ? =A0I've posted my config during the initial PR submission, and re-posting updated version now. Yes, QUOTA compiled in. It is turned on for /usr. > What are the drives ? ad0: 152626MB at ata0-master UDMA100 ad14: 953869MB at ata7-master UDMA100 SATA 3Gb/= s dmitry@lynx$ mount -t ufs /dev/ad0s3a on / (ufs, local, synchronous) /dev/ad0s3d on /usr (ufs, local, with quotas, soft-updates) /dev/ad14s4a on /databig (ufs, local, soft-updates) > Having to pull the generic information about the problematic system > does not make debugging faster. =A0I'm ready to provide any additional information - just not sure what else is significant for this problem. -- Sincerely, Dmitry nic-hdl: LYNX-RIPE From owner-freebsd-fs@FreeBSD.ORG Thu Jun 17 21:30:19 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E007E106567B for ; Thu, 17 Jun 2010 21:30:19 +0000 (UTC) (envelope-from lynx.ripe@gmail.com) Received: from mail-iw0-f182.google.com (mail-iw0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 83BFD8FC14 for ; Thu, 17 Jun 2010 21:30:19 +0000 (UTC) Received: by iwn7 with SMTP id 7so409380iwn.13 for ; Thu, 17 Jun 2010 14:30:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=Tl4n1i1ny0uozZUvVB3KsVHqm97THAjRHZM6M2jnlPw=; b=PwnBvVkILFzCfe8+MMadtbU527X29qRDRGxEacH1/yZW1b1Q4qsa+hmYSMMnb6fHV5 DSnAb4OtMZAkxyvaQhFFhU0JbFYjkmTXEvcupPkLpA73TQKF6g1YWYRZU7+Kxm1ioKF9 FctSu+0yh8yRxjvxMA5g3ebfEE6zFup5CyFZw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=GEeMc5NIp3B2bEG32z88UycBLWmTGEclbPGma9gun0ixuWvHH8EtdMOA5JtqfWgkVU prhAVAFu2BZIxxk20hmUvIYwyYGOkFWmYZsju6/7DJ5s5O1FxPdZkHR6xBC8+Qdhaszo A/N9pboOiFUPrR9a29r99b0ryGhrtUSLZ93J0= MIME-Version: 1.0 Received: by 10.231.123.141 with SMTP id p13mr95011ibr.174.1276810089587; Thu, 17 Jun 2010 14:28:09 -0700 (PDT) Received: by 10.231.146.199 with HTTP; Thu, 17 Jun 2010 14:28:08 -0700 (PDT) In-Reply-To: <20100617072841.GY13238@deviant.kiev.zoral.com.ua> References: <201006162220.o5GMK3uM002900@freefall.freebsd.org> <20100617072841.GY13238@deviant.kiev.zoral.com.ua> Date: Fri, 18 Jun 2010 00:28:08 +0300 Message-ID: From: Dmitry Pryanishnikov To: Kostik Belousov Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org Subject: Re: kern/147890: [ufs] [regression] ufs-related lock problem in RELENG_8 (18.04.2010 -> 20.04.2010) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jun 2010 21:30:20 -0000 Hello! 2010/6/17 Kostik Belousov : >> =A0 =A0Done - I've rebuilt the kernel from the sources as of 20.04.2010 >> =A0with the missing debugging options, repeated the problem, turned > I suspect you do not. In particular, DEBUG_VFS_LOCKS, DEBUG_LOCKS > and WITNESS seems to be missed. Well, now I understand the reason of your suspicion ;) The problem is that not all ddb(8) output actually comes to the capture buffer. For the 'show lockedvnods' handler, only the initial 'Locked vnodes' gets printed via db_printf - rest of info gets output via vn_printf() - that is, plain printf()s, and doesn't go to the capture buffer. So I've booted the problem kernel (now the newer one - sources as of 14.06.2010), repeated the same procedure, however also took a photographs of the 'show lockedvnods' output: http://picasaweb.google.com/lynx.ripe/FreeBSDKern147890?feat=3Ddirectlink Here is the contents of the capture buffer for this run: db> ps pid ppid pgrp uid state wmesg wchan cmd 2072 1949 2072 0 S+ ufs 0xcb642308 sync 2038 1 1868 1000 S ufs 0xcb642308 initial thread 2012 1 1868 1000 S ufs 0xcb642308 initial thread 1949 1847 1949 0 S+ wait 0xc7036000 bash 1934 1846 1934 0 S+ ttyin 0xc7372070 bash 1931 1 1868 1000 S select 0xcafac664 initial thread 1929 1 1875 1000 S select 0xcaf8f624 initial thread 1926 1904 1900 1000 S (threaded) soffice.bin 100175 S accept 0xcb5c484a soffice.bin 100174 S ucond 0xc7599b40 soffice.bin 100173 S ucond 0xc75ba1c0 soffice.bin 100123 S select 0xcb4b9124 initial thread 1923 1885 1868 1000 S select 0xcaf8e064 initial thread 1922 1885 1868 1000 S select 0xc75b71e4 initial thread 1921 1885 1868 1000 S select 0xcaf8e7e4 initial thread 1912 1 1897 1000 S select 0xcb4b98a4 xfce4-settings-help 1904 1900 1900 1000 S wait 0xc7709000 sh 1900 1 1900 1000 Ss wait 0xc805d550 sh 1895 1 1894 1000 S ufs 0xcb8f1868 initial thread 1890 1 1868 1000 S (threaded) xfdesktop 100074 S select 0xc75cfd24 initial thread 1888 1 1868 1000 S select 0xc75cfda4 xfsettingsd 1887 1 1868 1000 S (threaded) Thunar 100178 S piperd 0xc7f02188 Thunar 100163 S select 0xcaf8f464 initial thread 1885 1 1868 1000 S select 0xc75b8ba4 initial thread 1883 1 1868 1000 S select 0xcaf8e8a4 initial thread 1877 1 1875 1000 S select 0xcaf8e2e4 initial thread 1875 1 1875 1000 Ss select 0xcaf8eda4 dbus-daemon 1874 1 1868 1000 S select 0xc75b78a4 dbus-launch 1868 1862 1868 1000 Ss select 0xc75cdb64 initial thread 1862 1854 1862 0 Ss wait 0xc805d2a8 xdm 1857 1854 1857 0 Ss select 0xcafac824 initial thread 1855 1 1 0 S ttydcd 0xc7390290 getty 1854 1 1 0 S pause 0xc7685af8 xdm 1853 1 1853 0 Ss+ ttyin 0xc7372270 getty 1852 1 1852 0 Ss+ ttyin 0xc7372670 getty 1851 1 1851 0 Ss+ ttyin 0xc71dfe70 getty 1850 1 1850 0 Ss+ ttyin 0xc71dfc70 getty 1849 1 1849 0 Ss+ ttyin 0xc71df670 getty 1848 1 1848 0 Ss+ ttyin 0xc71df870 getty 1847 1 1847 0 Ss+ wait 0xc76ea000 login 1846 1 1846 0 Ss+ wait 0xc7685550 login 1820 1 1820 0 Ss select 0xc75b7624 inetd 1788 1 1788 0 Ss select 0xc75cda64 moused 1742 1 1742 0 Ss nanslp 0xc07d0704 cron 1735 1 1735 25 Ss pause 0xc7686da0 sendmail 1731 1 1731 0 Ss select 0xc75cd9a4 sendmail 1725 1 1725 0 Ss select 0xc75b7224 sshd 1639 1 1639 556 Ss select 0xc73a00a4 dbus-daemon 1434 1 1434 0 Ss select 0xc75cd5a4 ntpd 1400 1390 1390 0 S piperd 0xc7f02930 logger 1399 1390 1390 0 S select 0xc75ba764 ssh 1390 1 1390 0 Ss wait 0xc70357f8 sh 1216 1 1216 53 Ss (threaded) named 100144 S kqread 0xc8096400 named 100143 S ucond 0xc7599ac0 named 100142 S ucond 0xc7599a80 named 100141 S ucond 0xc7599a40 named 100140 S ucond 0xc7599a00 named 100139 S ucond 0xc75999c0 named 100125 S sigwait 0xe9637be4 named 1117 1 1117 0 Ss select 0xc75ba8a4 syslogd 716 1 716 0 Ss select 0xc75993a4 devd 116 1 116 0 Ss pause 0xc7709850 adjkerntz 20 0 0 0 SL sdflush 0xc094aca0 [softdepflush] 19 0 0 0 SL vlruwt 0xc759f550 [vnlru] 18 0 0 0 SL ufs 0xcb642308 [syncer] 17 0 0 0 SL psleep 0xc093c808 [bufdaemon] 16 0 0 0 SL pgzero 0xc094b974 [pagezero] 15 0 0 0 SL psleep 0xc094b59c [vmdaemon] 14 0 0 0 SL psleep 0xc094b564 [pagedaemon] 9 0 0 0 SL ccb_scan 0xc07ba254 [xpt_thrd] 8 0 0 0 SL - 0xc737243c [fdc0] 7 0 0 0 SL - 0xc72a0000 [fw0_probe] 13 0 0 0 SL (threaded) usb 100053 D - 0xc7284d0c [usbus4] 100052 D - 0xc7284cdc [usbus4] 100051 D - 0xc7284cac [usbus4] 100050 D - 0xc7284c7c [usbus4] 100049 D - 0xc726ddac [usbus3] 100048 D - 0xc726dd7c [usbus3] 100047 D - 0xc726dd4c [usbus3] 100046 D - 0xc726dd1c [usbus3] 100045 D - 0xc725cdac [usbus2] 100044 D - 0xc725cd7c [usbus2] 100043 D - 0xc725cd4c [usbus2] 100042 D - 0xc725cd1c [usbus2] 100040 D - 0xc7248dac [usbus1] 100039 D - 0xc7248d7c [usbus1] 100038 D - 0xc7248d4c [usbus1] 100037 D - 0xc7248d1c [usbus1] 100035 D - 0xc721ddac [usbus0] 100034 D - 0xc721dd7c [usbus0] 100033 D - 0xc721dd4c [usbus0] 100032 D - 0xc721dd1c [usbus0] 12 0 0 0 SL - 0xc07d0564 [yarrow] 6 0 0 0 SL crypto_r 0xc094a9ac [crypto returns] 5 0 0 0 SL crypto_w 0xc094a988 [crypto] 4 0 0 0 SL - 0xc07ce2e4 [g_down] 3 0 0 0 SL - 0xc07ce2e0 [g_up] 2 0 0 0 SL - 0xc07ce2d8 [g_event] 11 0 0 0 WL (threaded) intr 100153 I [irq256: vgapci0] 100060 I [swi0: uart] 100059 I [irq12: psm0] 100058 I [irq1: atkbd0] 100056 I [irq14: ata0] 100041 I [irq18: fwohci0+] 100036 I [irq19: uhci1++] 100031 I [irq23: uhci0 ehci0] 100029 I [irq16: atapci0+] 100028 I [irq257: hdac0] 100027 I [irq9: acpi0] 100026 I [swi2: cambio] 100025 I [swi6: task queue] 100024 I [swi6: Giant taskq] 100019 I [swi5: +] 100011 I [swi1: netisr 0] 100010 I [swi4: clock] 100009 I [swi4: clock] 100008 I [swi4: clock] 100007 I [swi4: clock] 100006 I [swi3: vm] 10 0 0 0 RL (threaded) idle 100005 CanRun [idle: cpu0] 100004 Run CPU 1 [idle: cpu1] 100003 Run CPU 2 [idle: cpu2] 100002 Run CPU 3 [idle: cpu3] 1 0 1 0 SLs wait 0xc7043000 [init] 0 0 0 0 RLs (threaded) kernel 100116 D - 0xc75c5d80 [dummynet] 100061 Run CPU 0 [deadlkres] 100054 D - 0xc727ecc0 [fw0_taskq] 100030 D - 0xc7204300 [em0 taskq] 100023 D - 0xc71ab100 [thread taskq] 100022 D - 0xc71ab140 [acpi_task_2] 100021 D - 0xc71ab140 [acpi_task_1] 100020 D - 0xc71ab140 [acpi_task_0] 100018 D - 0xc71ab200 [kqueue taskq] 100000 D sched 0xc07ce380 [swapper] db> trace Tracing pid 0 tid 100061 td 0xc7385000 kdb_enter(c0741c4a,c0741c4a,c073b16c,c6bcbcd0,0,...) at 0xc0536cea =3D kdb_enter+0x3a panic(c073b16c,c071770c,c76f0a00,3cea7,c7035dd0,...) at 0xc0508226 =3D pani= c+0x136 deadlkres(0,c6bcbd38,c073cd0d,344,c07ce380,...) at 0xc04c5171 =3D deadlkres= +0x361 fork_exit(c04c4e10,0,c6bcbd38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6bcbd70, ebp =3D 0 --- db> show pcpu cpuid =3D 0 dynamic pcpu =3D 0x86ba80 curthread =3D 0xc7385000: pid 0 "deadlkres" curpcb =3D 0xc6bcbd90 fpcurthread =3D none idlethread =3D 0xc7044500: pid 10 "idle: cpu0" APIC ID =3D 0 currentldt =3D 0x50 spin locks held: db> show allpcpu Current CPU: 0 cpuid =3D 0 dynamic pcpu =3D 0x86ba80 curthread =3D 0xc7385000: pid 0 "deadlkres" curpcb =3D 0xc6bcbd90 fpcurthread =3D none idlethread =3D 0xc7044500: pid 10 "idle: cpu0" APIC ID =3D 0 currentldt =3D 0x50 spin locks held: cpuid =3D 1 dynamic pcpu =3D 0x60a3a80 curthread =3D 0xc7044780: pid 10 "idle: cpu1" curpcb =3D 0xc6880d90 fpcurthread =3D none idlethread =3D 0xc7044780: pid 10 "idle: cpu1" APIC ID =3D 1 currentldt =3D 0x50 spin locks held: cpuid =3D 2 dynamic pcpu =3D 0x60a6a80 curthread =3D 0xc7044a00: pid 10 "idle: cpu2" curpcb =3D 0xc687dd90 fpcurthread =3D none idlethread =3D 0xc7044a00: pid 10 "idle: cpu2" APIC ID =3D 2 currentldt =3D 0x50 spin locks held: cpuid =3D 3 dynamic pcpu =3D 0x60a9a80 curthread =3D 0xc7044c80: pid 10 "idle: cpu3" curpcb =3D 0xc687ad90 fpcurthread =3D none idlethread =3D 0xc7044c80: pid 10 "idle: cpu3" APIC ID =3D 3 currentldt =3D 0x50 spin locks held: db> show locks db> show locks db> show alllocks Process 2012 (thunar-vfs-mime-cle) thread 0xc7f09780 (100148) exclusive lockmgr ufs (ufs) r =3D 0 (0xcb8f1868) locked @ /databig/ftp/RELENG_8/src.100614/sys/kern/vfs_subr.c:2103 Process 1895 (gam_server) thread 0xc76ef000 (100117) shared lockmgr ufs (ufs) r =3D 0 (0xcb642308) locked @ /databig/ftp/RELENG_8/src.100614/sys/kern/vfs_subr.c:2103 Process 18 (syncer) thread 0xc7383000 (100067) exclusive lockmgr syncer (syncer) r =3D 0 (0xc76e2c70) locked @ /databig/ftp/RELENG_8/src.100614/sys/kern/vfs_subr.c:1713 db> show lockedvnods Locked vnodes db> alltrace Tracing command sync pid 2072 tid 100095 td 0xc770e280 sched_switch(c770e280,0,104,191,c7e039f1,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,50,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c770e280,0,c0746206,260,50,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_wait(cb642308,50,c073a476,4,0,...) at 0xc0541ab3 =3D sleepq_wait+0x6= 3 __lockmgr_args(cb642308,80100,cb642370,0,0,...) at 0xc04f0fcf =3D __lockmgr_args+0xb1f ffs_lock(e9594b74,c0549fcb,c07644b5,80100,cb6422b0,...) at 0xc067f711 =3D ffs_lock+0xa1 VOP_LOCK1_APV(c07a5c40,e9594b74,c770e324,c07b8ea0,cb6422b0,...) at 0xc070d945 =3D VOP_LOCK1_APV+0xb5 _vn_lock(cb6422b0,80100,c074f848,837,4,...) at 0xc05a3a78 =3D _vn_lock+0x78 vget(cb6422b0,80100,c770e280,39a,c763e870,...) at 0xc05975fb =3D vget+0xbb qsync(c763e870,0,c07630cd,55b,4,...) at 0xc0688af7 =3D qsync+0x197 ffs_sync(c763e870,2,c0750b8a,93,c763e870,...) at 0xc067ae39 =3D ffs_sync+0x= 349 sync(c770e280,e9594cf8,c,c,286,...) at 0xc059dc55 =3D sync+0x145 syscall(e9594d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (36, FreeBSD ELF32, sync), eip =3D 0x480d6f5f, esp =3D 0xbfbfec4c, ebp =3D 0xbfbfec58 --- Tracing command thunar-vfs-mime-cle pid 2038 tid 100104 td 0xc76f0a00 sched_switch(c76f0a00,0,104,191,76c00124,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,50,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c76f0a00,0,c0746206,260,50,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_wait(cb642308,50,c073a476,4,0,...) at 0xc0541ab3 =3D sleepq_wait+0x6= 3 __lockmgr_args(cb642308,400400,cb642370,0,0,...) at 0xc04f0fcf =3D __lockmgr_args+0xb1f ffs_lock(e95da998,e95da9b0,e95da984,400400,cb6422b0,...) at 0xc067f711 =3D ffs_lock+0xa1 VOP_LOCK1_APV(c07a5c40,e95da998,e95da9a0,c07b8ea0,cb6422b0,...) at 0xc070d945 =3D VOP_LOCK1_APV+0xb5 _vn_lock(cb6422b0,400400,c074ee13,2a7,e95dabc4,...) at 0xc05a3a78 =3D _vn_lock+0x78 lookup(e95dabac,c7039c00,0,ca,c7035d48,...) at 0xc058b877 =3D lookup+0x547 namei(e95dabac,5aa,e95daac0,c0549fcb,0,...) at 0xc058c602 =3D namei+0x4c2 vn_open_cred(e95dabac,e95dac60,180,0,c757aa00,...) at 0xc05a3080 =3D vn_open_cred+0x90 vn_open(e95dabac,e95dac60,180,c7693498,280b3000,...) at 0xc05a356b =3D vn_open+0x3b kern_openat(c76f0a00,ffffff9c,28528280,0,a03,...) at 0xc05a24a5 =3D kern_openat+0xf5 kern_open(c76f0a00,28528280,0,a02,180,...) at 0xc05a28e5 =3D kern_open+0x35 open(c76f0a00,e95dacf8,c0771550,c0748d04,206,...) at 0xc05a2920 =3D open+0x= 30 syscall(e95dad38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (5, FreeBSD ELF32, open), eip =3D 0x28253a83, esp =3D 0xbfbfe2cc, ebp =3D 0xbfbfe2f8 --- Tracing command thunar-vfs-mime-cle pid 2012 tid 100148 td 0xc7f09780 sched_switch(c7f09780,0,104,191,d403e0a3,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,50,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7f09780,0,c0746206,260,50,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_wait(cb642308,50,c073a476,4,0,...) at 0xc0541ab3 =3D sleepq_wait+0x6= 3 __lockmgr_args(cb642308,400400,cb642370,0,0,...) at 0xc04f0fcf =3D __lockmgr_args+0xb1f ffs_lock(e967f878,e967f878,c04f87dc,400400,cb6422b0,...) at 0xc067f711 =3D ffs_lock+0xa1 VOP_LOCK1_APV(c07a5c40,e967f878,e967f920,c07b8ea0,cb6422b0,...) at 0xc070d945 =3D VOP_LOCK1_APV+0xb5 _vn_lock(cb6422b0,400400,c076472c,15c,e967f8bc,...) at 0xc05a3a78 =3D _vn_lock+0x78 ufs_accessx(e967f920,c0773fad,0,1,cb6422b0,...) at 0xc068ca24 =3D ufs_acces= sx+0xa4 VOP_ACCESSX_APV(c07a5c40,e967f920,cb8f1810,c07b8c00,cb6422b0,...) at 0xc070bbd5 =3D VOP_ACCESSX_APV+0xc5 ufs_delete_denied(c7f09780,48f500,80000,e967f9c4,e967f9d4,...) at 0xc0686704 =3D ufs_delete_denied+0xb4 ufs_lookup_(0,e967fa0c,c070c115,e967fa2c,c07740bc,...) at 0xc0687011 =3D ufs_lookup_+0x811 ufs_lookup(e967fa2c,c07740bc,cb6422b0,cb6422b0,e967fbf8,...) at 0xc068735e =3D ufs_lookup+0x1e VOP_CACHEDLOOKUP_APV(c07a5c40,e967fa2c,e967fbf8,e967fbe4,c757aa00,...) at 0xc070c115 =3D VOP_CACHEDLOOKUP_APV+0xc5 vfs_cache_lookup(e967fab8,c0750588,c07b8ae0,200000,e967fbcc,...) at 0xc0584946 =3D vfs_cache_lookup+0xd6 VOP_LOOKUP_APV(c07a5c40,e967fab8,c074ee13,310,e967fbe4,...) at 0xc070eac5 =3D VOP_LOOKUP_APV+0xe5 lookup(e967fbcc,c76c0800,0,ca,c7f03d48,...) at 0xc058b8f3 =3D lookup+0x5c3 namei(e967fbcc,c7f09824,c090e5d8,c079193c,c076522e,...) at 0xc058c602 =3D namei+0x4c2 kern_renameat(c7f09780,ffffff9c,28528280,ffffff9c,2852d3c0,...) at 0xc059b5e7 =3D kern_renameat+0x57 kern_rename(c7f09780,28528280,2852d3c0,0,e967fd2c,...) at 0xc059b996 =3D kern_rename+0x36 rename(c7f09780,e967fcf8,c0771550,c0749641,286,...) at 0xc059b9c9 =3D renam= e+0x29 syscall(e967fd38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (128, FreeBSD ELF32, rename), eip =3D 0x281bea9b, esp =3D 0xbfbfe37c, ebp =3D 0xbfbfebf8 --- Tracing command bash pid 1949 tid 100103 td 0xc76f0c80 sched_switch(c76f0c80,0,104,191,c714be30,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,5c,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c76f0c80,0,c0746206,1a0,5c,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_catch_signals(c0746206,160,0,100,100,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c7036000,5c,c0748bc0,100,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _sleep(c7036000,c7036088,15c,c0748bc0,0,...) at 0xc05115b4 =3D _sleep+0x354 kern_wait(c76f0c80,ffffffff,e95d7c78,6,0,...) at 0xc04dc516 =3D kern_wait+0= xb46 wait4(c76f0c80,e95d7cf8,c0771550,c0748a9a,286,...) at 0xc04dc58b =3D wait4+= 0x3b syscall(e95d7d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip =3D 0x80e26df, esp =3D 0xbfbfea2c, ebp =3D 0xbfbfea48 --- Tracing command bash pid 1934 tid 100145 td 0xc7f0a000 sched_switch(c7f0a000,0,104,191,ca657a43,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7f0a000,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(c0746206,160,0,c7f0a000,c7f0a000,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c7372070,0,c074aa01,101,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _cv_wait_sig(c7372070,c07cfeb0,c074a358,4fd,0,...) at 0xc04c6390 =3D _cv_wait_sig+0x240 tty_wait(c7372000,c7372070,e9673c5c,1,0,...) at 0xc055a2c1 =3D tty_wait+0x7= 1 ttydisc_read(c7372000,e9673c5c,0,9e,0,...) at 0xc0561a2c =3D ttydisc_read+0= x22c ttydev_read(c738fa00,e9673c5c,0,0,1,...) at 0xc055bc6a =3D ttydev_read+0xaa devfs_read_f(c7f01150,e9673c5c,c7effa80,0,c7f0a000,...) at 0xc04a71ce =3D devfs_read_f+0x7e dofileread(e9673c5c,ffffffff,ffffffff,0,c7f01150,...) at 0xc054cab6 =3D dofileread+0x96 kern_readv(c7f0a000,0,e9673c5c,e9673c7c,1,...) at 0xc054cde8 =3D kern_readv= +0x58 read(c7f0a000,e9673cf8,c0771550,c0748a9a,246,...) at 0xc054ceff =3D read+0x= 4f syscall(e9673d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip =3D 0x8127ca3, esp =3D 0xbfbfe08c, ebp =3D 0xbfbfe0b8 --- Tracing command gqview pid 1931 tid 100092 td 0xc770ea00 sched_switch(c770ea00,0,104,191,6003f9a3,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c770ea00,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(e958baac,c04f886a,cafac650,0,c770ea00,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_timedwait_sig(cafac664,0,e958badc,101,0,...) at 0xc05418fa =3D sleepq_timedwait_sig+0x1a _cv_timedwait_sig(cafac664,cafac650,acd,621,e958bb70,...) at 0xc04c5d40 =3D _cv_timedwait_sig+0x250 seltdwait(e958bc60,e958bc68,52f,c770ea00,e958bb60,...) at 0xc054c14a =3D seltdwait+0x8a poll(c770ea00,e958bcf8,c0771550,c0748d04,200282,...) at 0xc054c610 =3D poll= +0x300 syscall(e958bd38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip =3D 0x28b27b0f, esp =3D 0xbfbfeacc, ebp =3D 0xbfbfeae8 --- Tracing command gconfd-2 pid 1929 tid 100127 td 0xc7c30c80 sched_switch(c7c30c80,0,104,191,f20a9a2e,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7c30c80,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(e963daac,c04f886a,caf8f610,0,c7c30c80,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_timedwait_sig(caf8f624,0,e963dadc,101,0,...) at 0xc05418fa =3D sleepq_timedwait_sig+0x1a _cv_timedwait_sig(caf8f624,caf8f610,7531,621,e963db90,...) at 0xc04c5d40 =3D _cv_timedwait_sig+0x250 seltdwait(e963dc60,e963dc68,52f,c7c30c80,e963db60,...) at 0xc054c14a =3D seltdwait+0x8a poll(c7c30c80,e963dcf8,c0771550,c0748d04,282,...) at 0xc054c610 =3D poll+0x= 300 syscall(e963dd38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip =3D 0x28463b0f, esp =3D 0xbfbfea3c, ebp =3D 0xbfbfea58 --- Tracing command soffice.bin pid 1926 tid 100175 td 0xc76d1a00 sched_switch(c76d1a00,0,104,191,1e9c7524,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,58,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c76d1a00,0,c0746206,1a0,58,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_catch_signals(c0746206,160,0,100,100,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(cb5c484a,58,c0748c30,100,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _sleep(cb5c484a,c093c670,158,c0748c30,0,...) at 0xc05115b4 =3D _sleep+0x354 kern_accept(c76d1a00,8,0,0,0,...) at 0xc0575da0 =3D kern_accept+0x120 accept(c76d1a00,eb6f8cf8,c0771550,c0749890,286,...) at 0xc0576211 =3D accep= t+0x41 syscall(eb6f8d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (30, FreeBSD ELF32, accept), eip =3D 0x2a89209f, esp =3D 0xbf7fc52c, ebp =3D 0xbf7fc548 --- Tracing command soffice.bin pid 1926 tid 100174 td 0xc76d1c80 sched_switch(c76d1c80,0,104,191,e08a1ae,...) at 0xc0529db0 =3D sched_switch= +0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c76d1c80,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(dbba1,c0540880,c76d1c80,1,100,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_timedwait_sig(c7599b40,0,c07434ae,100,0,...) at 0xc05418fa =3D sleepq_timedwait_sig+0x1a _sleep(c7599b40,c07d3c4c,100,c07434ae,dbba1,...) at 0xc051157e =3D _sleep+0= x31e __umtx_op_cv_wait(c76d1c80,eb6f5cf8,eb6f5d2c,c06f51e5,c76d1c80,...) at 0xc05209a8 =3D __umtx_op_cv_wait+0x558 _umtx_op(c76d1c80,eb6f5cf8,c0771550,c074907b,202,...) at 0xc051baf7 =3D _umtx_op+0x27 syscall(eb6f5d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (454, FreeBSD ELF32, _umtx_op), eip =3D 0x2a808227, esp =3D 0xbf8fddbc, ebp =3D 0xbf8fdde8 --- Tracing command soffice.bin pid 1926 tid 100173 td 0xc76ec000 sched_switch(c76ec000,0,104,191,32800bba,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c76ec000,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(2711,c0540880,c76ec000,0,100,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_timedwait_sig(c75ba1c0,0,c07434ae,100,0,...) at 0xc05418fa =3D sleepq_timedwait_sig+0x1a _sleep(c75ba1c0,c07d328c,100,c07434ae,2711,...) at 0xc051157e =3D _sleep+0x= 31e __umtx_op_cv_wait(c76ec000,eb6f2cf8,eb6f2d2c,c06f51e5,c76ec000,...) at 0xc05209a8 =3D __umtx_op_cv_wait+0x558 _umtx_op(c76ec000,eb6f2cf8,c0771550,c074907b,202,...) at 0xc051baf7 =3D _umtx_op+0x27 syscall(eb6f2d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (454, FreeBSD ELF32, _umtx_op), eip =3D 0x2a808227, esp =3D 0xbf9feeac, ebp =3D 0xbf9feed8 --- Tracing command soffice.bin pid 1926 tid 100123 td 0xc7f0a280 sched_switch(c7f0a280,0,104,191,af3a2e5c,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7f0a280,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(e9630aac,c04f886a,cb4b9110,0,c7f0a280,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_timedwait_sig(cb4b9124,0,e9630adc,101,0,...) at 0xc05418fa =3D sleepq_timedwait_sig+0x1a _cv_timedwait_sig(cb4b9124,cb4b9110,db97b,621,e9630b88,...) at 0xc04c5d40 =3D _cv_timedwait_sig+0x250 seltdwait(e9630c60,e9630c68,52f,c7f0a280,e9630b60,...) at 0xc054c14a =3D seltdwait+0x8a poll(c7f0a280,e9630cf8,c0771550,c0748d04,286,...) at 0xc054c610 =3D poll+0x= 300 syscall(e9630d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip =3D 0x2a8a1b0f, esp =3D 0xbfbfe4dc, ebp =3D 0xbfbfe4f8 --- Tracing command xfce4-xkb-plugin pid 1923 tid 100071 td 0xc768ba00 sched_switch(c768ba00,0,104,191,17bdb73e,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c768ba00,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(c04f886a,caf8e050,0,c074025d,c768ba00,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(caf8e064,0,e9525adc,101,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _cv_wait_sig(caf8e064,caf8e050,c074859d,621,e9525b70,...) at 0xc04c6390 =3D _cv_wait_sig+0x240 seltdwait(c75bc02c,c074859d,52f,c768ba00,e9525b60,...) at 0xc054c162 =3D seltdwait+0xa2 poll(c768ba00,e9525cf8,c0771550,c072e876,200286,...) at 0xc054c610 =3D poll= +0x300 syscall(e9525d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip =3D 0x28ad3b0f, esp =3D 0xbfbfe9bc, ebp =3D 0xbfbfe9d8 --- Tracing command xfce4-mixer-plugin pid 1922 tid 100149 td 0xc7f09500 sched_switch(c7f09500,0,104,191,65baabca,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7f09500,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(c04f886a,c75b71d0,0,c074025d,c7f09500,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c75b71e4,0,e9683adc,101,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _cv_wait_sig(c75b71e4,c75b71d0,c074859d,621,e9683b68,...) at 0xc04c6390 =3D _cv_wait_sig+0x240 seltdwait(c75c9d2c,c074859d,52f,c7f09500,e9683b60,...) at 0xc054c162 =3D seltdwait+0xa2 poll(c7f09500,e9683cf8,c0771550,c072e876,286,...) at 0xc054c610 =3D poll+0x= 300 syscall(e9683d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip =3D 0x2900cb0f, esp =3D 0xbfbfe9bc, ebp =3D 0xbfbfe9d8 --- Tracing command xfce4-menu-plugin pid 1921 tid 100114 td 0xc76ef500 sched_switch(c76ef500,0,104,191,f195f6f8,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c76ef500,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(c04f886a,caf8e7d0,0,c074025d,c76ef500,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(caf8e7e4,0,e9610adc,101,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _cv_wait_sig(caf8e7e4,caf8e7d0,c074859d,621,e9610b70,...) at 0xc04c6390 =3D _cv_wait_sig+0x240 seltdwait(c754bb2c,c074859d,52f,c76ef500,e9610b60,...) at 0xc054c162 =3D seltdwait+0xa2 poll(c76ef500,e9610cf8,c0771550,c072e876,282,...) at 0xc054c610 =3D poll+0x= 300 syscall(e9610d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip =3D 0x28de2b0f, esp =3D 0xbfbfe9cc, ebp =3D 0xbfbfe9e8 --- Tracing command xfce4-settings-help pid 1912 tid 100138 td 0xc8062780 sched_switch(c8062780,0,104,191,17beddc8,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c8062780,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(c04f886a,cb4b9890,0,c074025d,c8062780,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(cb4b98a4,0,e965eadc,101,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _cv_wait_sig(cb4b98a4,cb4b9890,c074859d,621,e965eb80,...) at 0xc04c6390 =3D _cv_wait_sig+0x240 seltdwait(c7582c2c,c074859d,52f,c8062780,e965eb60,...) at 0xc054c162 =3D seltdwait+0xa2 poll(c8062780,e965ecf8,c0771550,c072e876,200282,...) at 0xc054c610 =3D poll= +0x300 syscall(e965ed38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip =3D 0x288b3b0f, esp =3D 0xbfbfea5c, ebp =3D 0xbfbfea78 --- Tracing command sh pid 1904 tid 100096 td 0xc770e000 sched_switch(c770e000,0,104,191,27c4a3f2,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,5c,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c770e000,0,c0746206,1a0,5c,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_catch_signals(c0746206,160,0,100,100,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c7709000,5c,c0748bc0,100,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _sleep(c7709000,c7709088,15c,c0748bc0,0,...) at 0xc05115b4 =3D _sleep+0x354 kern_wait(c770e000,ffffffff,e9597c78,2,0,...) at 0xc04dc516 =3D kern_wait+0= xb46 wait4(c770e000,e9597cf8,c0771550,c074941a,216,...) at 0xc04dc58b =3D wait4+= 0x3b syscall(e9597d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip =3D 0x2815dbdb, esp =3D 0xbfbfea0c, ebp =3D 0xbfbfea28 --- Tracing command sh pid 1900 tid 100130 td 0xc7c30500 sched_switch(c7c30500,0,104,191,bca20692,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,5c,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7c30500,0,c0746206,1a0,5c,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_catch_signals(c0746206,160,0,100,100,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c805d550,5c,c0748bc0,100,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _sleep(c805d550,c805d5d8,15c,c0748bc0,0,...) at 0xc05115b4 =3D _sleep+0x354 kern_wait(c7c30500,ffffffff,e9646c78,2,0,...) at 0xc04dc516 =3D kern_wait+0= xb46 wait4(c7c30500,e9646cf8,c0771550,c0748b9f,216,...) at 0xc04dc58b =3D wait4+= 0x3b syscall(e9646d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip =3D 0x2815dbdb, esp =3D 0xbfbfe9bc, ebp =3D 0xbfbfe9d8 --- Tracing command gam_server pid 1895 tid 100117 td 0xc76ef000 sched_switch(c76ef000,0,104,191,d403fc84,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,50,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c76ef000,0,c0746206,260,cb8f1868,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_wait(cb8f1868,50,c073a476,4,1,...) at 0xc0541ab3 =3D sleepq_wait+0x6= 3 __lockmgr_args(cb8f1868,200100,cb8f18d0,0,0,...) at 0xc04f0965 =3D __lockmgr_args+0x4b5 ffs_lock(e961b7e0,c0549fcb,c074ecdf,200100,cb8f1810,...) at 0xc067f711 =3D ffs_lock+0xa1 VOP_LOCK1_APV(c07a5c40,e961b7e0,c76ef0a4,c07b8ea0,cb8f1810,...) at 0xc070d945 =3D VOP_LOCK1_APV+0xb5 _vn_lock(cb8f1810,200100,c074f848,837,4,...) at 0xc05a3a78 =3D _vn_lock+0x7= 8 vget(cb8f1810,200100,c76ef000,50,0,...) at 0xc05975fb =3D vget+0xbb vfs_hash_get(c763e870,48f500,200000,c76ef000,e961b960,...) at 0xc058a73d =3D vfs_hash_get+0xed ffs_vgetf(c763e870,48f500,200000,e961b960,0,...) at 0xc0679e49 =3D ffs_vget= f+0x49 ffs_vget(c763e870,48f500,200000,e961b960,e961b970,...) at 0xc067a38e =3D ffs_vget+0x2e ufs_lookup_(0,e961b9a8,c070c115,e961b9c8,c07740bc,...) at 0xc06872e1 =3D ufs_lookup_+0xae1 ufs_lookup(e961b9c8,c07740bc,cb6422b0,cb6422b0,e961bb8c,...) at 0xc068735e =3D ufs_lookup+0x1e VOP_CACHEDLOOKUP_APV(c07a5c40,e961b9c8,e961bb8c,e961bb78,c757aa00,...) at 0xc070c115 =3D VOP_CACHEDLOOKUP_APV+0xc5 vfs_cache_lookup(e961ba54,c0750588,c07b8ae0,200000,e961bb60,...) at 0xc0584946 =3D vfs_cache_lookup+0xd6 VOP_LOOKUP_APV(c07a5c40,e961ba54,c074ee13,310,e961bb78,...) at 0xc070eac5 =3D VOP_LOOKUP_APV+0xe5 lookup(e961bb60,c770f400,0,ca,c76ea550,...) at 0xc058b8f3 =3D lookup+0x5c3 namei(e961bb60,c80d43f0,cb4eb500,e961bc14,c04d930e,...) at 0xc058c602 =3D namei+0x4c2 kern_statat_vnhook(c76ef000,0,ffffff9c,2846ad00,0,...) at 0xc059bf42 =3D kern_statat_vnhook+0x72 kern_statat(c76ef000,0,ffffff9c,2846ad00,0,...) at 0xc059c09c =3D kern_stat= at+0x3c kern_stat(c76ef000,2846ad00,0,e961bc1c,c76ef000,...) at 0xc059c1e6 =3D kern_stat+0x36 stat(c76ef000,e961bcf8,c0771550,c0749535,246,...) at 0xc059c28f =3D stat+0x= 2f syscall(e961bd38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (188, FreeBSD ELF32, stat), eip =3D 0x28386a23, esp =3D 0xbfbfe9ac, ebp =3D 0xbfbfea48 --- Tracing command xfdesktop pid 1890 tid 100074 td 0xc768b280 sched_switch(c768b280,0,104,191,49e351fd,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c768b280,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(e9531aac,c04f886a,c75cfd10,0,c768b280,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_timedwait_sig(c75cfd24,0,e9531adc,101,0,...) at 0xc05418fa =3D sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c75cfd24,c75cfd10,3e9,621,e9531b88,...) at 0xc04c5d40 =3D _cv_timedwait_sig+0x250 seltdwait(e9531c60,e9531c68,52f,c768b280,e9531b60,...) at 0xc054c14a =3D seltdwait+0x8a poll(c768b280,e9531cf8,c0771550,c0748d04,286,...) at 0xc054c610 =3D poll+0x= 300 syscall(e9531d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip =3D 0x28eb2b0f, esp =3D 0xbfbfe6dc, ebp =3D 0xbfbfe6f8 --- Tracing command xfsettingsd pid 1888 tid 100100 td 0xc770d500 sched_switch(c770d500,0,104,191,869e1662,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c770d500,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(c04f886a,c75cfd90,0,c074025d,c770d500,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c75cfda4,0,e95ccadc,101,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _cv_wait_sig(c75cfda4,c75cfd90,c074859d,621,e95ccb70,...) at 0xc04c6390 =3D _cv_wait_sig+0x240 seltdwait(c809082c,c074859d,52f,c770d500,e95ccb60,...) at 0xc054c162 =3D seltdwait+0xa2 poll(c770d500,e95cccf8,c0771550,c072e876,286,...) at 0xc054c610 =3D poll+0x= 300 syscall(e95ccd38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip =3D 0x287c0b0f, esp =3D 0xbfbfeaec, ebp =3D 0xbfbfeb08 --- Tracing command Thunar pid 1887 tid 100178 td 0xc7c33a00 sched_switch(c7c33a00,0,104,191,74f3fb67,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,4c,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7c33a00,0,c0746206,1a0,4c,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_catch_signals(c0746206,160,0,100,100,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c7f02188,4c,c07487a6,100,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _sleep(c7f02188,c7f022f8,14c,c07487a6,0,...) at 0xc05115b4 =3D _sleep+0x354 pipe_read(c76bf700,eb702c5c,c757aa00,0,c7c33a00,...) at 0xc054fb37 =3D pipe_read+0x3f7 dofileread(eb702c5c,ffffffff,ffffffff,0,c76bf700,...) at 0xc054cab6 =3D dofileread+0x96 kern_readv(c7c33a00,d,eb702c5c,eb702c7c,1,...) at 0xc054cde8 =3D kern_readv= +0x58 read(c7c33a00,eb702cf8,c,c,200286,...) at 0xc054ceff =3D read+0x4f syscall(eb702d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip =3D 0x28a68593, esp =3D 0xbf6fbeec, ebp =3D 0xbf6fbf08 --- Tracing command Thunar pid 1887 tid 100163 td 0xc7f0aa00 sched_switch(c7f0aa00,0,104,191,6769d0b6,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7f0aa00,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(eb6ceaac,c04f886a,caf8f450,0,c7f0aa00,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_timedwait_sig(caf8f464,0,eb6ceadc,101,0,...) at 0xc05418fa =3D sleepq_timedwait_sig+0x1a _cv_timedwait_sig(caf8f464,caf8f450,3e9,621,eb6ceb80,...) at 0xc04c5d40 =3D _cv_timedwait_sig+0x250 seltdwait(eb6cec60,eb6cec68,52f,c7f0aa00,eb6ceb60,...) at 0xc054c14a =3D seltdwait+0x8a poll(c7f0aa00,eb6cecf8,c0771550,c0748d04,200282,...) at 0xc054c610 =3D poll= +0x300 syscall(eb6ced38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip =3D 0x28a0eb0f, esp =3D 0xbfbfeafc, ebp =3D 0xbfbfeb18 --- Tracing command xfce4-panel pid 1885 tid 100162 td 0xc7f0ac80 sched_switch(c7f0ac80,0,104,191,17bd5f75,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7f0ac80,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(eb6cbaac,c04f886a,c75b8b90,0,c7f0ac80,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_timedwait_sig(c75b8ba4,0,eb6cbadc,101,0,...) at 0xc05418fa =3D sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c75b8ba4,c75b8b90,3d8,621,eb6cbb78,...) at 0xc04c5d40 =3D _cv_timedwait_sig+0x250 seltdwait(eb6cbc60,eb6cbc68,52f,c7f0ac80,eb6cbb60,...) at 0xc054c14a =3D seltdwait+0x8a poll(c7f0ac80,eb6cbcf8,c0771550,c0748d04,282,...) at 0xc054c610 =3D poll+0x= 300 syscall(eb6cbd38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip =3D 0x28d53b0f, esp =3D 0xbfbfea5c, ebp =3D 0xbfbfea78 --- Tracing command xfwm4 pid 1883 tid 100136 td 0xc8062c80 sched_switch(c8062c80,0,104,191,17bd2a29,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c8062c80,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(c04f886a,caf8e890,0,c074025d,c8062c80,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(caf8e8a4,0,e9658adc,101,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _cv_wait_sig(caf8e8a4,caf8e890,c074859d,621,e9658b78,...) at 0xc04c6390 =3D _cv_wait_sig+0x240 seltdwait(c75c062c,c074859d,52f,c8062c80,e9658b60,...) at 0xc054c162 =3D seltdwait+0xa2 poll(c8062c80,e9658cf8,c0771550,c072e876,200282,...) at 0xc054c610 =3D poll= +0x300 syscall(e9658d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip =3D 0x28de9b0f, esp =3D 0xbfbfe6cc, ebp =3D 0xbfbfe6e8 --- Tracing command xfconfd pid 1877 tid 100098 td 0xc770da00 sched_switch(c770da00,0,104,191,6985b499,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c770da00,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(c04f886a,caf8e2d0,0,c074025d,c770da00,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(caf8e2e4,0,e959dadc,101,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _cv_wait_sig(caf8e2e4,caf8e2d0,c074859d,621,e959db70,...) at 0xc04c6390 =3D _cv_wait_sig+0x240 seltdwait(c782b52c,c074859d,52f,c770da00,e959db60,...) at 0xc054c162 =3D seltdwait+0xa2 poll(c770da00,e959dcf8,c0771550,c07749c4,282,...) at 0xc054c610 =3D poll+0x= 300 syscall(e959dd38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip =3D 0x283e8b0f, esp =3D 0xbfbfea0c, ebp =3D 0xbfbfea28 --- Tracing command dbus-daemon pid 1875 tid 100158 td 0xc8061280 sched_switch(c8061280,0,104,191,f765e0f5,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c8061280,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(c04f886a,caf8ed90,0,c074025d,c8061280,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(caf8eda4,0,eb6bfadc,101,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _cv_wait_sig(caf8eda4,caf8ed90,c074859d,621,eb6bfbb8,...) at 0xc04c6390 =3D _cv_wait_sig+0x240 seltdwait(c809292c,c074859d,52f,c8061280,eb6bfb60,...) at 0xc054c162 =3D seltdwait+0xa2 poll(c8061280,eb6bfcf8,c0771550,c074907b,282,...) at 0xc054c610 =3D poll+0x= 300 syscall(eb6bfd38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip =3D 0x281a3b0f, esp =3D 0xbfbfe7cc, ebp =3D 0xbfbfe7e8 --- Tracing command dbus-launch pid 1874 tid 100157 td 0xc7f09280 sched_switch(c7f09280,0,104,191,42e7299a,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7f09280,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(c04f886a,c75b7890,0,c074025d,c7f09280,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c75b78a4,0,eb6bca80,101,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _cv_wait_sig(c75b78a4,c75b7890,c074859d,621,c7f01508,...) at 0xc04c6390 =3D _cv_wait_sig+0x240 seltdwait(c7f01508,d8,c757aa00,c7f09280,c074c323,...) at 0xc054c162 =3D seltdwait+0xa2 kern_select(c7f09280,6,bfbfeae8,0,bfbfe8e8,0,20,bfbfe620,c7f037f8) at 0xc054d414 =3D kern_select+0x504 select(c7f09280,eb6bccf8,c0771550,c072e876,286,...) at 0xc054d766 =3D selec= t+0x66 syscall(eb6bcd38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip =3D 0x282f6513, esp =3D 0xbfbfe61c, ebp =3D 0xbfbfe648 --- Tracing command xfce4-session pid 1868 tid 100126 td 0xc7c31000 sched_switch(c7c31000,0,104,191,8a19ee3e,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7c31000,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(c04f886a,c75cdb50,0,c074025d,c7c31000,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c75cdb64,0,e963aadc,101,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _cv_wait_sig(c75cdb64,c75cdb50,c074859d,621,e963aba0,...) at 0xc04c6390 =3D _cv_wait_sig+0x240 seltdwait(c75d4a2c,c074859d,52f,c7c31000,e963ab60,...) at 0xc054c162 =3D seltdwait+0xa2 poll(c7c31000,e963acf8,c0771550,c072e876,282,...) at 0xc054c610 =3D poll+0x= 300 syscall(e963ad38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip =3D 0x28df6b0f, esp =3D 0xbfbfeafc, ebp =3D 0xbfbfeb18 --- Tracing command xdm pid 1862 tid 100131 td 0xc7c30280 sched_switch(c7c30280,0,104,191,1fe68fff,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,5c,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7c30280,0,c0746206,1a0,5c,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_catch_signals(c0746206,160,0,100,100,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c805d2a8,5c,c0748bc0,100,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _sleep(c805d2a8,c805d330,15c,c0748bc0,0,...) at 0xc05115b4 =3D _sleep+0x354 kern_wait(c7c30280,ffffffff,e9649c78,0,0,...) at 0xc04dc516 =3D kern_wait+0= xb46 wait4(c7c30280,e9649cf8,c0771550,c0748a9a,282,...) at 0xc04dc58b =3D wait4+= 0x3b syscall(e9649d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip =3D 0x2831fbdb, esp =3D 0xbfbfe8cc, ebp =3D 0xbfbfe8e8 --- Tracing command Xorg pid 1857 tid 100151 td 0xc8062280 sched_switch(c8062280,0,104,191,17be024d,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c8062280,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(c04f886a,cafac810,0,c074025d,c8062280,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(cafac824,0,e96a1a80,101,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _cv_wait_sig(cafac824,cafac810,c074859d,621,c8186d90,...) at 0xc04c6390 =3D _cv_wait_sig+0x240 seltdwait(c8186d90,58,c7f00000,c8062280,c074c323,...) at 0xc054c162 =3D seltdwait+0xa2 kern_select(c8062280,1c,81e8a00,0,0,0,20,bfbfeb10,cafa9d48) at 0xc054d414 =3D kern_select+0x504 select(c8062280,e96a1cf8,c0771550,c0748baf,200246,...) at 0xc054d766 =3D select+0x66 syscall(e96a1d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip =3D 0x285ad513, esp =3D 0xbfbfeb0c, ebp =3D 0xbfbfed78 --- Tracing command getty pid 1855 tid 100134 td 0xc8063280 sched_switch(c8063280,0,104,191,2a6c600,...) at 0xc0529db0 =3D sched_switch= +0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c8063280,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(c04f886a,c7390204,0,c074025d,c8063280,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c7390290,0,e96529e0,101,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _cv_wait_sig(c7390290,c7390204,c074a358,4fd,c7390200,...) at 0xc04c6390 =3D _cv_wait_sig+0x240 tty_wait(c7390200,c7390290,0,4e,caf87968,...) at 0xc055a2c1 =3D tty_wait+0x= 71 ttydev_open(c738e600,3,2000,c8063280,e9652a7c,...) at 0xc055c042 =3D ttydev_open+0x372 devfs_open(e9652b34,c0774076,caf87968,0,e9652bac,...) at 0xc04a8d18 =3D devfs_open+0xe8 VOP_OPEN_APV(c0787500,e9652b34,e9652ac0,c0549fcb,0,...) at 0xc070bfc5 =3D VOP_OPEN_APV+0xc5 vn_open_cred(e9652bac,e9652c60,1,0,c7041400,...) at 0xc05a3446 =3D vn_open_cred+0x456 vn_open(e9652bac,e9652c60,1,c7f018f8,28207000,...) at 0xc05a356b =3D vn_ope= n+0x3b kern_openat(c8063280,ffffff9c,804fbc0,0,3,...) at 0xc05a24a5 =3D kern_opena= t+0xf5 kern_open(c8063280,804fbc0,0,2,3,...) at 0xc05a28e5 =3D kern_open+0x35 open(c8063280,e9652cf8,c,c,282,...) at 0xc05a2920 =3D open+0x30 syscall(e9652d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (5, FreeBSD ELF32, open), eip =3D 0x2817ea83, esp =3D 0xbfbfedac, ebp =3D 0xbfbfedd8 --- Tracing command xdm pid 1854 tid 100075 td 0xc768b000 sched_switch(c768b000,0,104,191,afa0a08f,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,68,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c768b000,0,c0746206,1a0,68,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_catch_signals(c0746206,160,0,100,100,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c7685af8,68,c0741e98,100,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _sleep(c7685af8,c7685b28,168,c0741e98,0,...) at 0xc05115b4 =3D _sleep+0x354 kern_sigsuspend(c768b000,0,0,0,0,...) at 0xc050c1de =3D kern_sigsuspend+0xa= e sigsuspend(c768b000,e9535cf8,c0771550,c0748a9a,246,...) at 0xc050c2ed =3D sigsuspend+0x4d syscall(e9535d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (4, FreeBSD ELF32, write), eip =3D 0x28307d9b, esp =3D 0xbfbfe85c, ebp =3D 0xbfbfe918 --- Tracing command getty pid 1853 tid 100089 td 0xc7687000 sched_switch(c7687000,0,104,191,2bd2dd0,...) at 0xc0529db0 =3D sched_switch= +0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7687000,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(c0746206,160,0,c7687000,c7687000,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c7372270,0,c074aa01,101,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _cv_wait_sig(c7372270,c07cfeb0,c074a358,4fd,0,...) at 0xc04c6390 =3D _cv_wait_sig+0x240 tty_wait(c7372200,c7372270,e9582c5c,1,0,...) at 0xc055a2c1 =3D tty_wait+0x7= 1 ttydisc_read(c7372200,e9582c5c,0,9e,0,...) at 0xc0561a2c =3D ttydisc_read+0= x22c ttydev_read(c738f300,e9582c5c,0,0,1,...) at 0xc055bc6a =3D ttydev_read+0xaa devfs_read_f(c76bf690,e9582c5c,c7041400,0,c7687000,...) at 0xc04a71ce =3D devfs_read_f+0x7e dofileread(e9582c5c,ffffffff,ffffffff,0,c76bf690,...) at 0xc054cab6 =3D dofileread+0x96 kern_readv(c7687000,0,e9582c5c,e9582c7c,1,...) at 0xc054cde8 =3D kern_readv= +0x58 read(c7687000,e9582cf8,c0771550,c07540b1,246,...) at 0xc054ceff =3D read+0x= 4f syscall(e9582d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip =3D 0x2818f593, esp =3D 0xbfbfedbc, ebp =3D 0xbfbfede8 --- Tracing command getty pid 1852 tid 100082 td 0xc7687500 sched_switch(c7687500,0,104,191,2b65a7a,...) at 0xc0529db0 =3D sched_switch= +0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7687500,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(c0746206,160,0,c7687500,c7687500,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c7372670,0,c074aa01,101,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _cv_wait_sig(c7372670,c07cfeb0,c074a358,4fd,0,...) at 0xc04c6390 =3D _cv_wait_sig+0x240 tty_wait(c7372600,c7372670,e9551c5c,1,0,...) at 0xc055a2c1 =3D tty_wait+0x7= 1 ttydisc_read(c7372600,e9551c5c,0,9e,0,...) at 0xc0561a2c =3D ttydisc_read+0= x22c ttydev_read(c738f400,e9551c5c,0,0,1,...) at 0xc055bc6a =3D ttydev_read+0xaa devfs_read_f(c76dda10,e9551c5c,c7041400,0,c7687500,...) at 0xc04a71ce =3D devfs_read_f+0x7e dofileread(e9551c5c,ffffffff,ffffffff,0,c76dda10,...) at 0xc054cab6 =3D dofileread+0x96 kern_readv(c7687500,0,e9551c5c,e9551c7c,1,...) at 0xc054cde8 =3D kern_readv= +0x58 read(c7687500,e9551cf8,c0771550,c07540b1,246,...) at 0xc054ceff =3D read+0x= 4f syscall(e9551d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip =3D 0x2818f593, esp =3D 0xbfbfedbc, ebp =3D 0xbfbfede8 --- Tracing command getty pid 1851 tid 100084 td 0xc76f0000 sched_switch(c76f0000,0,104,191,2b7ea85,...) at 0xc0529db0 =3D sched_switch= +0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c76f0000,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(c0746206,160,0,c76f0000,c76f0000,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c71dfe70,0,c074aa01,101,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _cv_wait_sig(c71dfe70,c07cfeb0,c074a358,4fd,0,...) at 0xc04c6390 =3D _cv_wait_sig+0x240 tty_wait(c71dfe00,c71dfe70,e956dc5c,1,0,...) at 0xc055a2c1 =3D tty_wait+0x7= 1 ttydisc_read(c71dfe00,e956dc5c,0,9e,0,...) at 0xc0561a2c =3D ttydisc_read+0= x22c ttydev_read(c738f500,e956dc5c,0,0,1,...) at 0xc055bc6a =3D ttydev_read+0xaa devfs_read_f(c768c2a0,e956dc5c,c7041400,0,c76f0000,...) at 0xc04a71ce =3D devfs_read_f+0x7e dofileread(e956dc5c,ffffffff,ffffffff,0,c768c2a0,...) at 0xc054cab6 =3D dofileread+0x96 kern_readv(c76f0000,0,e956dc5c,e956dc7c,1,...) at 0xc054cde8 =3D kern_readv= +0x58 read(c76f0000,e956dcf8,c0771550,c07540b1,246,...) at 0xc054ceff =3D read+0x= 4f syscall(e956dd38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip =3D 0x2818f593, esp =3D 0xbfbfedbc, ebp =3D 0xbfbfede8 --- Tracing command getty pid 1850 tid 100137 td 0xc8062a00 sched_switch(c8062a00,0,104,191,2ab1ea0,...) at 0xc0529db0 =3D sched_switch= +0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c8062a00,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(c0746206,160,0,c8062a00,c8062a00,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c71dfc70,0,c074aa01,101,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _cv_wait_sig(c71dfc70,c07cfeb0,c074a358,4fd,0,...) at 0xc04c6390 =3D _cv_wait_sig+0x240 tty_wait(c71dfc00,c71dfc70,e965bc5c,1,0,...) at 0xc055a2c1 =3D tty_wait+0x7= 1 ttydisc_read(c71dfc00,e965bc5c,0,9e,0,...) at 0xc0561a2c =3D ttydisc_read+0= x22c ttydev_read(c738f600,e965bc5c,0,0,1,...) at 0xc055bc6a =3D ttydev_read+0xaa devfs_read_f(c7693d20,e965bc5c,c7041400,0,c8062a00,...) at 0xc04a71ce =3D devfs_read_f+0x7e dofileread(e965bc5c,ffffffff,ffffffff,0,c7693d20,...) at 0xc054cab6 =3D dofileread+0x96 kern_readv(c8062a00,0,e965bc5c,e965bc7c,1,...) at 0xc054cde8 =3D kern_readv= +0x58 read(c8062a00,e965bcf8,c0771550,c07540b1,246,...) at 0xc054ceff =3D read+0x= 4f syscall(e965bd38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip =3D 0x2818f593, esp =3D 0xbfbfedbc, ebp =3D 0xbfbfede8 --- Tracing command getty pid 1849 tid 100083 td 0xc76d3000 sched_switch(c76d3000,0,104,191,2c01837,...) at 0xc0529db0 =3D sched_switch= +0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c76d3000,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(c0746206,160,0,c76d3000,c76d3000,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c71df670,0,c074aa01,101,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _cv_wait_sig(c71df670,c07cfeb0,c074a358,4fd,0,...) at 0xc04c6390 =3D _cv_wait_sig+0x240 tty_wait(c71df600,c71df670,e9555c5c,1,0,...) at 0xc055a2c1 =3D tty_wait+0x7= 1 ttydisc_read(c71df600,e9555c5c,0,9e,0,...) at 0xc0561a2c =3D ttydisc_read+0= x22c ttydev_read(c738f700,e9555c5c,0,0,1,...) at 0xc055bc6a =3D ttydev_read+0xaa devfs_read_f(c7f01ce8,e9555c5c,c7041400,0,c76d3000,...) at 0xc04a71ce =3D devfs_read_f+0x7e dofileread(e9555c5c,ffffffff,ffffffff,0,c7f01ce8,...) at 0xc054cab6 =3D dofileread+0x96 kern_readv(c76d3000,0,e9555c5c,e9555c7c,1,...) at 0xc054cde8 =3D kern_readv= +0x58 read(c76d3000,e9555cf8,c0771550,c07540b1,246,...) at 0xc054ceff =3D read+0x= 4f syscall(e9555d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip =3D 0x2818f593, esp =3D 0xbfbfedbc, ebp =3D 0xbfbfede8 --- Tracing command getty pid 1848 tid 100102 td 0xc770d000 sched_switch(c770d000,0,104,191,2c9c469,...) at 0xc0529db0 =3D sched_switch= +0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c770d000,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(c0746206,160,0,c770d000,c770d000,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c71df870,0,c074aa01,101,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _cv_wait_sig(c71df870,c07cfeb0,c074a358,4fd,0,...) at 0xc04c6390 =3D _cv_wait_sig+0x240 tty_wait(c71df800,c71df870,e95d4c5c,1,0,...) at 0xc055a2c1 =3D tty_wait+0x7= 1 ttydisc_read(c71df800,e95d4c5c,0,9e,0,...) at 0xc0561a2c =3D ttydisc_read+0= x22c ttydev_read(c738f800,e95d4c5c,0,0,1,...) at 0xc055bc6a =3D ttydev_read+0xaa devfs_read_f(c7693e38,e95d4c5c,c7041400,0,c770d000,...) at 0xc04a71ce =3D devfs_read_f+0x7e dofileread(e95d4c5c,ffffffff,ffffffff,0,c7693e38,...) at 0xc054cab6 =3D dofileread+0x96 kern_readv(c770d000,0,e95d4c5c,e95d4c7c,1,...) at 0xc054cde8 =3D kern_readv= +0x58 read(c770d000,e95d4cf8,c0771550,c07540b1,246,...) at 0xc054ceff =3D read+0x= 4f syscall(e95d4d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip =3D 0x2818f593, esp =3D 0xbfbfedbc, ebp =3D 0xbfbfede8 --- Tracing command login pid 1847 tid 100119 td 0xc76eca00 sched_switch(c76eca00,0,104,191,43299c0d,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,5c,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c76eca00,0,c0746206,1a0,5c,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_catch_signals(c0746206,160,0,100,100,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c76ea000,5c,c0748bc0,100,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _sleep(c76ea000,c76ea088,15c,c0748bc0,0,...) at 0xc05115b4 =3D _sleep+0x354 kern_wait(c76eca00,79d,e9623c78,0,0,...) at 0xc04dc516 =3D kern_wait+0xb46 wait4(c76eca00,e9623cf8,c,c,216,...) at 0xc04dc58b =3D wait4+0x3b syscall(e9623d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip =3D 0x28120bdb, esp =3D 0xbfbfed2c, ebp =3D 0xbfbfed48 --- Tracing command login pid 1846 tid 100077 td 0xc7687a00 sched_switch(c7687a00,0,104,191,60b206f2,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,5c,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7687a00,0,c0746206,1a0,5c,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_catch_signals(c0746206,160,0,100,100,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c7685550,5c,c0748bc0,100,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _sleep(c7685550,c76855d8,15c,c0748bc0,0,...) at 0xc05115b4 =3D _sleep+0x354 kern_wait(c7687a00,78e,e953dc78,0,0,...) at 0xc04dc516 =3D kern_wait+0xb46 wait4(c7687a00,e953dcf8,c,c,216,...) at 0xc04dc58b =3D wait4+0x3b syscall(e953dd38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip =3D 0x28120bdb, esp =3D 0xbfbfed2c, ebp =3D 0xbfbfed48 --- Tracing command inetd pid 1820 tid 100099 td 0xc770d780 sched_switch(c770d780,0,104,191,e7e8e550,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c770d780,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(c04f886a,c75b7610,0,c074025d,c770d780,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c75b7624,0,e95a0a80,101,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _cv_wait_sig(c75b7624,c75b7610,c074859d,621,c768c230,...) at 0xc04c6390 =3D _cv_wait_sig+0x240 seltdwait(c768c230,58,c7041400,c770d780,4,...) at 0xc054c162 =3D seltdwait+= 0xa2 kern_select(c770d780,7,bfbfdfc0,0,0,0,20,bfbfdf80,c76eb7f8) at 0xc054d414 =3D kern_select+0x504 select(c770d780,e95a0cf8,c0771550,c0748c13,202,...) at 0xc054d766 =3D selec= t+0x66 syscall(e95a0d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip =3D 0x281a3513, esp =3D 0xbfbfdf7c, ebp =3D 0xbfbfee08 --- Tracing command moused pid 1788 tid 100081 td 0xc7687780 sched_switch(c7687780,0,104,191,ca9fbc48,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7687780,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(c04f886a,c75cda50,0,c074025d,c7687780,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c75cda64,0,e954da80,101,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _cv_wait_sig(c75cda64,c75cda50,c074859d,621,c768d818,...) at 0xc04c6390 =3D _cv_wait_sig+0x240 seltdwait(c768d818,58,c7041400,c7687780,0,...) at 0xc054c162 =3D seltdwait+= 0xa2 kern_select(c7687780,6,bfbfea34,0,0,0,20,bfbfe9c0,c76852a8) at 0xc054d414 =3D kern_select+0x504 select(c7687780,e954dcf8,c0771550,c07540b1,246,...) at 0xc054d766 =3D selec= t+0x66 syscall(e954dd38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip =3D 0x281ac513, esp =3D 0xbfbfe9bc, ebp =3D 0xbfbfeb58 --- Tracing command cron pid 1742 tid 100101 td 0xc770d280 sched_switch(c770d280,0,104,191,68d37c89,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,5c,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c770d280,0,c0746206,1a0,5c,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_catch_signals(e291,c0540880,c770d280,0,100,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_timedwait_sig(c07d0704,5c,c0743004,100,0,...) at 0xc05418fa =3D sleepq_timedwait_sig+0x1a _sleep(c07d0704,0,15c,c0743004,e291,...) at 0xc051157e =3D _sleep+0x31e kern_nanosleep(c770d280,e95d0c68,e95d0c70,3a,0,...) at 0xc0518951 =3D kern_nanosleep+0xc1 nanosleep(c770d280,e95d0cf8,c0771550,c074907b,206,...) at 0xc051a38f =3D nanosleep+0x6f syscall(e95d0d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (240, FreeBSD ELF32, nanosleep), eip =3D 0x2817bf17, esp =3D 0xbfbfec8c, ebp =3D 0xbfbfecb8 --- Tracing command sendmail pid 1735 tid 100120 td 0xc76ec780 sched_switch(c76ec780,0,104,191,f168d3ec,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,68,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c76ec780,0,c0746206,1a0,68,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_catch_signals(c0746206,160,0,100,100,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c7686da0,68,c0741e98,100,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _sleep(c7686da0,c7686dd0,168,c0741e98,0,...) at 0xc05115b4 =3D _sleep+0x354 kern_sigsuspend(c76ec780,0,0,0,0,...) at 0xc050c1de =3D kern_sigsuspend+0xa= e sigsuspend(c76ec780,e9626cf8,c0771550,c0748a9a,202,...) at 0xc050c2ed =3D sigsuspend+0x4d syscall(e9626d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (4, FreeBSD ELF32, write), eip =3D 0x2832fd9b, esp =3D 0xbfbfda9c, ebp =3D 0xbfbfdac8 --- Tracing command sendmail pid 1731 tid 100110 td 0xc7c33000 sched_switch(c7c33000,0,104,191,5be9ba34,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7c33000,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(e95eea50,c04f886a,c75cd990,0,c7c33000,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_timedwait_sig(c75cd9a4,0,e95eea80,101,0,...) at 0xc05418fa =3D sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c75cd9a4,c75cd990,1389,621,c768c9d8,...) at 0xc04c5d40 =3D _cv_timedwait_sig+0x250 seltdwait(e95eec2c,e95eec34,c8094d00,c7c33000,c159b014,...) at 0xc054c14a =3D seltdwait+0x8a kern_select(c7c33000,8,bfbfd8bc,0,0,e95eec74,20,5,0) at 0xc054d414 =3D kern_select+0x504 select(c7c33000,e95eecf8,c0771550,c0748fda,202,...) at 0xc054d766 =3D selec= t+0x66 syscall(e95eed38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip =3D 0x283d6513, esp =3D 0xbfbfd02c, ebp =3D 0xbfbfdac8 --- Tracing command sshd pid 1725 tid 100073 td 0xc768b500 sched_switch(c768b500,0,104,191,8c5518a6,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c768b500,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(c04f886a,c75b7210,0,c074025d,c768b500,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c75b7224,0,e952da80,101,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _cv_wait_sig(c75b7224,c75b7210,c074859d,621,c7693540,...) at 0xc04c6390 =3D _cv_wait_sig+0x240 seltdwait(c7693540,58,c75c2780,c768b500,e952daec,...) at 0xc054c162 =3D seltdwait+0xa2 kern_select(c768b500,5,286090ac,0,0,0,20,bfbfe2a0,c7686000) at 0xc054d414 =3D kern_select+0x504 select(c768b500,e952dcf8,c,c,206,...) at 0xc054d766 =3D select+0x66 syscall(e952dd38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip =3D 0x284a9513, esp =3D 0xbfbfe29c, ebp =3D 0xbfbfee38 --- Tracing command dbus-daemon pid 1639 tid 100097 td 0xc770dc80 sched_switch(c770dc80,0,104,191,6dae4e7a,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c770dc80,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(c04f886a,c73a0090,0,c074025d,c770dc80,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c73a00a4,0,e959aadc,101,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _cv_wait_sig(c73a00a4,c73a0090,c074859d,621,e959ab78,...) at 0xc04c6390 =3D _cv_wait_sig+0x240 seltdwait(c75d442c,c074859d,52f,c770dc80,e959ab60,...) at 0xc054c162 =3D seltdwait+0xa2 poll(c770dc80,e959acf8,c0771550,c074907b,282,...) at 0xc054c610 =3D poll+0x= 300 syscall(e959ad38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip =3D 0x281a3b0f, esp =3D 0xbfbfe95c, ebp =3D 0xbfbfe978 --- Tracing command ntpd pid 1434 tid 100085 td 0xc76efc80 sched_switch(c76efc80,0,104,191,1d845150,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c76efc80,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(c04f886a,c75cd590,0,c074025d,c76efc80,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c75cd5a4,0,e9571a80,101,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _cv_wait_sig(c75cd5a4,c75cd590,c074859d,621,c7693a48,...) at 0xc04c6390 =3D _cv_wait_sig+0x240 seltdwait(c7693a48,58,c7041400,c76efc80,0,...) at 0xc054c162 =3D seltdwait+= 0xa2 kern_select(c76efc80,20,bfbfed18,0,0,0,20,bfbfecf0,c76eb2a8) at 0xc054d414 =3D kern_select+0x504 select(c76efc80,e9571cf8,16,16,212,...) at 0xc054d766 =3D select+0x66 syscall(e9571d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip =3D 0x28354513, esp =3D 0xbfbfecec, ebp =3D 0xbfbfedb8 --- Tracing command logger pid 1400 tid 100147 td 0xc7f09a00 sched_switch(c7f09a00,0,104,191,a3d7d63d,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,4c,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7f09a00,0,c0746206,1a0,4c,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_catch_signals(c0746206,160,0,100,100,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c7f02930,4c,c07487a6,100,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _sleep(c7f02930,c7f02aa0,14c,c07487a6,0,...) at 0xc05115b4 =3D _sleep+0x354 pipe_read(c76bfce8,e967bc5c,c7041400,0,c7f09a00,...) at 0xc054fb37 =3D pipe_read+0x3f7 dofileread(e967bc5c,ffffffff,ffffffff,0,c76bfce8,...) at 0xc054cab6 =3D dofileread+0x96 kern_readv(c7f09a00,0,e967bc5c,e967bc7c,1,...) at 0xc054cde8 =3D kern_readv= +0x58 read(c7f09a00,e967bcf8,c0771550,c0748d90,216,...) at 0xc054ceff =3D read+0x= 4f syscall(e967bd38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip =3D 0x2817d593, esp =3D 0xbfbfe93c, ebp =3D 0xbfbfe958 --- Tracing command ssh pid 1399 tid 100146 td 0xc7f09c80 sched_switch(c7f09c80,0,104,191,a5716db0,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7f09c80,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(c04f886a,c75ba750,0,c074025d,c7f09c80,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c75ba764,0,e9677a80,101,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _cv_wait_sig(c75ba764,c75ba750,c074859d,621,c768dbd0,...) at 0xc04c6390 =3D _cv_wait_sig+0x240 seltdwait(c768dbd0,58,c75ce380,c7f09c80,246,...) at 0xc054c162 =3D seltdwai= t+0xa2 kern_select(c7f09c80,5,285010b0,285010b4,0,0,20,bfbfbf30,c7f042a8) at 0xc054d414 =3D kern_select+0x504 select(c7f09c80,e9677cf8,c0771550,c073e299,206,...) at 0xc054d766 =3D selec= t+0x66 syscall(e9677d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip =3D 0x28381513, esp =3D 0xbfbfbf2c, ebp =3D 0xbfbfbfd8 --- Tracing command sh pid 1390 tid 100106 td 0xc76f0500 sched_switch(c76f0500,0,104,191,ca8fe4e7,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,5c,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c76f0500,0,c0746206,1a0,5c,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_catch_signals(c0746206,160,0,100,100,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c70357f8,5c,c0748bc0,100,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _sleep(c70357f8,c7035880,15c,c0748bc0,0,...) at 0xc05115b4 =3D _sleep+0x354 kern_wait(c76f0500,ffffffff,e95e0c78,2,0,...) at 0xc04dc516 =3D kern_wait+0= xb46 wait4(c76f0500,e95e0cf8,c0771550,c0748b9f,216,...) at 0xc04dc58b =3D wait4+= 0x3b syscall(e95e0d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip =3D 0x2815dbdb, esp =3D 0xbfbfea4c, ebp =3D 0xbfbfea68 --- Tracing command named pid 1216 tid 100144 td 0xc76d2000 sched_switch(c76d2000,0,104,191,d726bc14,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,58,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c76d2000,0,c0746206,1a0,58,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_catch_signals(c0746206,160,0,100,100,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c8096400,58,c073c8b9,100,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _sleep(c8096400,c8096400,158,c073c8b9,0,...) at 0xc05115b4 =3D _sleep+0x354 kern_kevent(c76d2000,9,0,40,e9670c5c,...) at 0xc04d8c44 =3D kern_kevent+0x3= 64 kevent(c76d2000,e9670cf8,c0771550,c072e876,246,...) at 0xc04d957b =3D keven= t+0x19b syscall(e9670d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (363, FreeBSD ELF32, kevent), eip =3D 0x2849f07b, esp =3D 0xbf4f9e0c, ebp =3D 0xbf4f9f98 --- Tracing command named pid 1216 tid 100143 td 0xc76d2280 sched_switch(c76d2280,0,104,191,c3d270b1,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c76d2280,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(119cfb,c0540880,c76d2280,0,100,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_timedwait_sig(c7599ac0,0,c07434ae,100,0,...) at 0xc05418fa =3D sleepq_timedwait_sig+0x1a _sleep(c7599ac0,c07d3050,100,c07434ae,119cfb,...) at 0xc051157e =3D _sleep+= 0x31e __umtx_op_cv_wait(c76d2280,e966dcf8,e966dd2c,c06f51e5,c76d2280,...) at 0xc05209a8 =3D __umtx_op_cv_wait+0x558 _umtx_op(c76d2280,e966dcf8,c0771550,c074907b,202,...) at 0xc051baf7 =3D _umtx_op+0x27 syscall(e966dd38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (454, FreeBSD ELF32, _umtx_op), eip =3D 0x283c8227, esp =3D 0xbf5faddc, ebp =3D 0xbf5fae08 --- Tracing command named pid 1216 tid 100142 td 0xc76d2500 sched_switch(c76d2500,0,104,191,d705ebc6,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c76d2500,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(c0746206,160,0,100,100,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c7599a80,0,c07434ae,100,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _sleep(c7599a80,c07d31f0,100,c07434ae,0,...) at 0xc05115b4 =3D _sleep+0x354 __umtx_op_cv_wait(c76d2500,e966acf8,e966ad2c,c06f51e5,c76d2500,...) at 0xc05208c6 =3D __umtx_op_cv_wait+0x476 _umtx_op(c76d2500,e966acf8,c0771550,c0748ce2,202,...) at 0xc051baf7 =3D _umtx_op+0x27 syscall(e966ad38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (454, FreeBSD ELF32, _umtx_op), eip =3D 0x283c8227, esp =3D 0xbf6fbe9c, ebp =3D 0xbf6fbec8 --- Tracing command named pid 1216 tid 100141 td 0xc76d2780 sched_switch(c76d2780,0,104,191,2051c1c7,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c76d2780,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(c0746206,160,0,100,100,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c7599a40,0,c07434ae,100,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _sleep(c7599a40,c07d31f0,100,c07434ae,0,...) at 0xc05115b4 =3D _sleep+0x354 __umtx_op_cv_wait(c76d2780,e9667cf8,e9667d2c,c06f51e5,c76d2780,...) at 0xc05208c6 =3D __umtx_op_cv_wait+0x476 _umtx_op(c76d2780,e9667cf8,c0771550,c0748ce2,202,...) at 0xc051baf7 =3D _umtx_op+0x27 syscall(e9667d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (454, FreeBSD ELF32, _umtx_op), eip =3D 0x283c8227, esp =3D 0xbf7fce9c, ebp =3D 0xbf7fcec8 --- Tracing command named pid 1216 tid 100140 td 0xc76d2a00 sched_switch(c76d2a00,0,104,191,202c8ca6,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c76d2a00,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(c0746206,160,0,100,100,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c7599a00,0,c07434ae,100,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _sleep(c7599a00,c07d31f0,100,c07434ae,0,...) at 0xc05115b4 =3D _sleep+0x354 __umtx_op_cv_wait(c76d2a00,e9664cf8,e9664d2c,c06f51e5,c76d2a00,...) at 0xc05208c6 =3D __umtx_op_cv_wait+0x476 _umtx_op(c76d2a00,e9664cf8,c0771550,c0748ce2,202,...) at 0xc051baf7 =3D _umtx_op+0x27 syscall(e9664d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (454, FreeBSD ELF32, _umtx_op), eip =3D 0x283c8227, esp =3D 0xbf8fde9c, ebp =3D 0xbf8fdec8 --- Tracing command named pid 1216 tid 100139 td 0xc76d2c80 sched_switch(c76d2c80,0,104,191,d7285e1f,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c76d2c80,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(c0746206,160,0,100,100,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c75999c0,0,c07434ae,100,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _sleep(c75999c0,c07d31f0,100,c07434ae,0,...) at 0xc05115b4 =3D _sleep+0x354 __umtx_op_cv_wait(c76d2c80,e9661cf8,e9661d2c,c06f51e5,c76d2c80,...) at 0xc05208c6 =3D __umtx_op_cv_wait+0x476 _umtx_op(c76d2c80,e9661cf8,c0771550,c0748ce2,202,...) at 0xc051baf7 =3D _umtx_op+0x27 syscall(e9661d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (454, FreeBSD ELF32, _umtx_op), eip =3D 0x283c8227, esp =3D 0xbf9fee9c, ebp =3D 0xbf9feec8 --- Tracing command named pid 1216 tid 100125 td 0xc7c31280 sched_switch(c7c31280,0,104,191,371ee94d,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,68,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7c31280,0,c0746206,1a0,68,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_catch_signals(c0746206,160,0,100,100,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(e9637be4,68,c0741e9e,100,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _sleep(e9637be4,c805e330,168,c0741e9e,0,...) at 0xc05115b4 =3D _sleep+0x354 kern_sigtimedwait(c7c31280,4003,0,0,0,...) at 0xc050c57c =3D kern_sigtimedwait+0x28c sigwait(c7c31280,e9637cf8,c0771550,c0749890,200282,...) at 0xc050c9e7 =3D sigwait+0x77 syscall(e9637d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (429, FreeBSD ELF32, sigwait), eip =3D 0x284146db, esp =3D 0xbfbfec9c, ebp =3D 0xbfbfecc8 --- Tracing command syslogd pid 1117 tid 100129 td 0xc7c30780 sched_switch(c7c30780,0,104,191,d04cfff,...) at 0xc0529db0 =3D sched_switch= +0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7c30780,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(c04f886a,c75ba890,0,c074025d,c7c30780,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c75ba8a4,0,e9643a80,101,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _cv_wait_sig(c75ba8a4,c75ba890,c074859d,621,c768daf0,...) at 0xc04c6390 =3D _cv_wait_sig+0x240 seltdwait(c768daf0,58,c7041400,c7c30780,0,...) at 0xc054c162 =3D seltdwait+= 0xa2 kern_select(c7c30780,b,282310ac,0,0,0,20,bfbfe260,c805d7f8) at 0xc054d414 =3D kern_select+0x504 select(c7c30780,e9643cf8,c0771550,c0748baf,246,...) at 0xc054d766 =3D selec= t+0x66 syscall(e9643d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip =3D 0x28192513, esp =3D 0xbfbfe25c, ebp =3D 0xbfbfedc8 --- Tracing command devd pid 716 tid 100108 td 0xc7c33500 sched_switch(c7c33500,0,104,191,8a6c96bf,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7c33500,0,c0746206,1a0,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_catch_signals(c04f886a,c7599390,0,c074025d,c7c33500,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c75993a4,0,e95e6a80,101,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _cv_wait_sig(c75993a4,c7599390,c074859d,621,c768ca48,...) at 0xc04c6390 =3D _cv_wait_sig+0x240 seltdwait(c768ca48,58,c7041400,c7c33500,c054a06c,...) at 0xc054c162 =3D seltdwait+0xa2 kern_select(c7c33500,6,bfbfe9a0,0,0,0,20,bfbfe970,c70352a8) at 0xc054d414 =3D kern_select+0x504 select(c7c33500,e95e6cf8,c0771550,c0749641,206,...) at 0xc054d766 =3D selec= t+0x66 syscall(e95e6d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip =3D 0x8088343, esp =3D 0xbfbfe96c, ebp =3D 0xbfbfee48 --- Tracing command adjkerntz pid 116 tid 100093 td 0xc770e780 sched_switch(c770e780,0,104,191,55a778d1,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,68,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c770e780,0,c0746206,1a0,68,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_catch_signals(c0746206,160,0,100,100,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c7709850,68,c0741e98,100,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _sleep(c7709850,c7709880,168,c0741e98,0,...) at 0xc05115b4 =3D _sleep+0x354 kern_sigsuspend(c770e780,0,0,0,0,...) at 0xc050c1de =3D kern_sigsuspend+0xa= e sigsuspend(c770e780,e958ecf8,c0771550,c0748fda,202,...) at 0xc050c2ed =3D sigsuspend+0x4d syscall(e958ed38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (4, FreeBSD ELF32, write), eip =3D 0x280d6d9b, esp =3D 0xbfbfed6c, ebp =3D 0xbfbfee38 --- Tracing command softdepflush pid 20 tid 100069 td 0xc727da00 sched_switch(c727da00,0,104,191,e30647fa,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,44,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c727da00,0,c0746206,283,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_timedwait(c094aca0,44,c07629c5,0,0,...) at 0xc05419bb =3D sleepq_timedwait+0x6b _sleep(c094aca0,c094ac44,44,c07629c5,3e8,...) at 0xc0511599 =3D _sleep+0x33= 9 softdep_flush(0,e7332d38,c073cd0d,344,c759f2a8,...) at 0xc0675844 =3D softdep_flush+0x244 fork_exit(c0675600,0,e7332d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xe7332d70, ebp =3D 0 --- Tracing command vnlru pid 19 tid 100068 td 0xc727dc80 sched_switch(c727dc80,0,104,191,6f3462af,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,50,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c727dc80,0,c0746206,283,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_timedwait(c759f550,50,c0750849,0,0,...) at 0xc05419bb =3D sleepq_timedwait+0x6b _sleep(c759f550,c093ca98,250,c0750849,3e8,...) at 0xc0511599 =3D _sleep+0x3= 39 vnlru_proc(0,e732fd38,c073cd0d,344,c759f550,...) at 0xc0599dd7 =3D vnlru_pr= oc+0xe7 fork_exit(c0599cf0,0,e732fd38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xe732fd70, ebp =3D 0 --- Tracing command syncer pid 18 tid 100067 td 0xc7383000 sched_switch(c7383000,0,104,191,f69f2b57,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,50,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7383000,0,c0746206,260,50,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_wait(cb642308,50,c073a476,4,0,...) at 0xc0541ab3 =3D sleepq_wait+0x6= 3 __lockmgr_args(cb642308,80100,cb642370,0,0,...) at 0xc04f0fcf =3D __lockmgr_args+0xb1f ffs_lock(e732cb34,c0549fcb,c07644b5,80100,cb6422b0,...) at 0xc067f711 =3D ffs_lock+0xa1 VOP_LOCK1_APV(c07a5c40,e732cb34,c73830a4,c07b8ea0,cb6422b0,...) at 0xc070d945 =3D VOP_LOCK1_APV+0xb5 _vn_lock(cb6422b0,80100,c074f848,837,4,...) at 0xc05a3a78 =3D _vn_lock+0x78 vget(cb6422b0,80100,c7383000,39a,c763e870,...) at 0xc05975fb =3D vget+0xbb qsync(c763e870,0,c07630cd,55b,4,...) at 0xc0688af7 =3D qsync+0x197 ffs_sync(c763e870,3,c074f848,d77,c763e870,...) at 0xc067ae39 =3D ffs_sync+0= x349 sync_fsync(e732cc7c,c0773e54,c76e2cd8,e732cc7c,c76e2c18,...) at 0xc0598d4f =3D sync_fsync+0x18f VOP_FSYNC_APV(c079aea0,e732cc7c,c074f848,6b1,c7383000,...) at 0xc070b545 =3D VOP_FSYNC_APV+0xc5 sync_vnode(c093cad8,c093cac4,3e8,6e0,4e20,...) at 0xc059902b =3D sync_vnode= +0x16b sched_sync(0,e732cd38,c073cd0d,344,c759f7f8,...) at 0xc0599373 =3D sched_sync+0x273 fork_exit(c0599100,0,e732cd38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xe732cd70, ebp =3D 0 --- Tracing command bufdaemon pid 17 tid 100066 td 0xc7383280 sched_switch(c7383280,0,104,191,5a70671a,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,44,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7383280,0,c0746206,283,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_timedwait(c093c808,44,c074dc70,0,0,...) at 0xc05419bb =3D sleepq_timedwait+0x6b _sleep(c093c808,c093c80c,44,c074dc70,3e8,...) at 0xc0511599 =3D _sleep+0x33= 9 buf_daemon(0,e7329d38,c073cd0d,344,c759faa0,...) at 0xc0581368 =3D buf_daemon+0x138 fork_exit(c0581230,0,e7329d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xe7329d70, ebp =3D 0 --- Tracing command pagezero pid 16 tid 100065 td 0xc7383500 sched_switch(c7383500,0,104,191,7a36692c,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7383500,0,c0746206,283,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_timedwait(c094b974,0,c0768a7b,0,0,...) at 0xc05419bb =3D sleepq_timedwait+0x6b _sleep(c094b974,c094b530,0,c0768a7b,493e0,...) at 0xc0511599 =3D _sleep+0x3= 39 vm_pagezero(0,e7326d38,c073cd0d,344,c759fd48,...) at 0xc06b506c =3D vm_pagezero+0xdc fork_exit(c06b4f90,0,e7326d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xe7326d70, ebp =3D 0 --- Tracing command vmdaemon pid 15 tid 100064 td 0xc7383780 sched_switch(c7383780,0,104,191,acdd7263,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,68,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7383780,0,c0746206,260,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_wait(c094b59c,68,c074dc70,0,0,...) at 0xc0541ab3 =3D sleepq_wait+0x6= 3 _sleep(c094b59c,c094b5a0,68,c074dc70,0,...) at 0xc05115cb =3D _sleep+0x36b vm_daemon(0,e7323d38,c073cd0d,344,c70432a8,...) at 0xc06af449 =3D vm_daemon= +0x59 fork_exit(c06af3f0,0,e7323d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xe7323d70, ebp =3D 0 --- Tracing command pagedaemon pid 14 tid 100063 td 0xc7383a00 sched_switch(c7383a00,0,104,191,7c17d233,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,44,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7383a00,0,c0746206,283,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_timedwait(c094b564,44,c074dc70,0,0,...) at 0xc05419bb =3D sleepq_timedwait+0x6b _sleep(c094b564,c094b530,44,c074dc70,1388,...) at 0xc0511599 =3D _sleep+0x3= 39 vm_pageout(0,e7320d38,c073cd0d,344,c7043550,...) at 0xc06b052b =3D vm_pageout+0x2bb fork_exit(c06b0270,0,e7320d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xe7320d70, ebp =3D 0 --- Tracing command xpt_thrd pid 9 tid 100062 td 0xc7383c80 sched_switch(c7383c80,0,104,191,a7f69d51,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,4c,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7383c80,0,c0746206,260,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_wait(c07ba254,4c,c07298fc,0,0,...) at 0xc0541ab3 =3D sleepq_wait+0x6= 3 _sleep(c07ba254,c07ba298,4c,c07298fc,0,...) at 0xc05115cb =3D _sleep+0x36b xpt_scanner_thread(0,c6bced38,c073cd0d,344,c70437f8,...) at 0xc043fb67 =3D xpt_scanner_thread+0x47 fork_exit(c043fb20,0,c6bced38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6bced70, ebp =3D 0 --- Tracing command fdc0 pid 8 tid 100057 td 0xc7244780 sched_switch(c7244780,0,104,191,31612219,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,4c,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7244780,0,c0746206,283,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_timedwait(c737243c,4c,c0c8c2f4,0,0,...) at 0xc05419bb =3D sleepq_timedwait+0x6b _sleep(c737243c,c73724f0,4c,c0c8c2f4,3e8,...) at 0xc0511599 =3D _sleep+0x33= 9 fdc_thread(c7372400,c6bbed38,c073cd0d,344,c7043aa0,...) at 0xc0c8a10d =3D fdc_thread+0x89d fork_exit(c0c89870,c7372400,c6bbed38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6bbed70, ebp =3D 0 --- Tracing command fw0_probe pid 7 tid 100055 td 0xc7244c80 sched_switch(c7244c80,0,104,191,caaa4cfe,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,5c,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7244c80,0,c0746206,1a0,5c,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_catch_signals(c0746206,160,0,100,100,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c72a0000,5c,c0d3fa7e,100,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _sleep(c72a0000,c72a4488,15c,c0d3fa7e,0,...) at 0xc05115b4 =3D _sleep+0x354 fw_bus_probe_thread(c72a0000,c6b9ed38,c073cd0d,344,c7043d48,...) at 0xc0d32aab =3D fw_bus_probe_thread+0x9b fork_exit(c0d32a10,c72a0000,c6b9ed38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6b9ed70, ebp =3D 0 --- Tracing command usb pid 13 tid 100053 td 0xc727c280 sched_switch(c727c280,0,104,191,a6e9ca02,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c727c280,0,c0746206,260,c727c280,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_wait(c7284d0c,0,c6b94cbc,1,0,...) at 0xc0541ab3 =3D sleepq_wait+0x63 _cv_wait(c7284d0c,c7284dac,c0b67bc2,6c,c7284d14,...) at 0xc04c6970 =3D _cv_wait+0x240 usb_process(c7284d04,c6b94d38,c073cd0d,344,c714b000,...) at 0xc0b60ed9 =3D usb_process+0x159 fork_exit(c0b60d80,c7284d04,c6b94d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6b94d70, ebp =3D 0 --- Tracing command usb pid 13 tid 100052 td 0xc727c500 sched_switch(c727c500,0,104,191,49f03334,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c727c500,0,c0746206,260,c727c500,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_wait(c7284cdc,0,c6b91cbc,1,0,...) at 0xc0541ab3 =3D sleepq_wait+0x63 _cv_wait(c7284cdc,c7284dac,c0b67bc2,6c,c7284ce4,...) at 0xc04c6970 =3D _cv_wait+0x240 usb_process(c7284cd4,c6b91d38,c073cd0d,344,c714b000,...) at 0xc0b60ed9 =3D usb_process+0x159 fork_exit(c0b60d80,c7284cd4,c6b91d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6b91d70, ebp =3D 0 --- Tracing command usb pid 13 tid 100051 td 0xc727c780 sched_switch(c727c780,0,104,191,a6e48b7f,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c727c780,0,c0746206,260,c727c780,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_wait(c7284cac,0,c6b8ecbc,1,0,...) at 0xc0541ab3 =3D sleepq_wait+0x63 _cv_wait(c7284cac,c7284dac,c0b67bc2,6c,c7284cb4,...) at 0xc04c6970 =3D _cv_wait+0x240 usb_process(c7284ca4,c6b8ed38,c073cd0d,344,c714b000,...) at 0xc0b60ed9 =3D usb_process+0x159 fork_exit(c0b60d80,c7284ca4,c6b8ed38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6b8ed70, ebp =3D 0 --- Tracing command usb pid 13 tid 100050 td 0xc727ca00 sched_switch(c727ca00,0,104,191,a6e47508,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c727ca00,0,c0746206,260,c727ca00,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_wait(c7284c7c,0,c6b8bcbc,1,0,...) at 0xc0541ab3 =3D sleepq_wait+0x63 _cv_wait(c7284c7c,c7284dac,c0b67bc2,6c,c7284c84,...) at 0xc04c6970 =3D _cv_wait+0x240 usb_process(c7284c74,c6b8bd38,c073cd0d,344,c714b000,...) at 0xc0b60ed9 =3D usb_process+0x159 fork_exit(c0b60d80,c7284c74,c6b8bd38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6b8bd70, ebp =3D 0 --- Tracing command usb pid 13 tid 100049 td 0xc727cc80 sched_switch(c727cc80,0,104,191,a6e45d3b,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c727cc80,0,c0746206,260,c727cc80,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_wait(c726ddac,0,c6b86cbc,1,0,...) at 0xc0541ab3 =3D sleepq_wait+0x63 _cv_wait(c726ddac,c726de4c,c0b67bc2,6c,c726ddb4,...) at 0xc04c6970 =3D _cv_wait+0x240 usb_process(c726dda4,c6b86d38,c073cd0d,344,c714b000,...) at 0xc0b60ed9 =3D usb_process+0x159 fork_exit(c0b60d80,c726dda4,c6b86d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6b86d70, ebp =3D 0 --- Tracing command usb pid 13 tid 100048 td 0xc727d000 sched_switch(c727d000,0,104,191,d2d006a6,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c727d000,0,c0746206,260,c727d000,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_wait(c726dd7c,0,c6b83cbc,1,0,...) at 0xc0541ab3 =3D sleepq_wait+0x63 _cv_wait(c726dd7c,c726de4c,c0b67bc2,6c,c726dd84,...) at 0xc04c6970 =3D _cv_wait+0x240 usb_process(c726dd74,c6b83d38,c073cd0d,344,c714b000,...) at 0xc0b60ed9 =3D usb_process+0x159 fork_exit(c0b60d80,c726dd74,c6b83d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6b83d70, ebp =3D 0 --- Tracing command usb pid 13 tid 100047 td 0xc727d280 sched_switch(c727d280,0,104,191,a6df352f,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c727d280,0,c0746206,260,c727d280,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_wait(c726dd4c,0,c6b80cbc,1,0,...) at 0xc0541ab3 =3D sleepq_wait+0x63 _cv_wait(c726dd4c,c726de4c,c0b67bc2,6c,c726dd54,...) at 0xc04c6970 =3D _cv_wait+0x240 usb_process(c726dd44,c6b80d38,c073cd0d,344,c714b000,...) at 0xc0b60ed9 =3D usb_process+0x159 fork_exit(c0b60d80,c726dd44,c6b80d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6b80d70, ebp =3D 0 --- Tracing command usb pid 13 tid 100046 td 0xc727d500 sched_switch(c727d500,0,104,191,a6df1d7d,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c727d500,0,c0746206,260,c727d500,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_wait(c726dd1c,0,c6b7dcbc,1,0,...) at 0xc0541ab3 =3D sleepq_wait+0x63 _cv_wait(c726dd1c,c726de4c,c0b67bc2,6c,c726dd24,...) at 0xc04c6970 =3D _cv_wait+0x240 usb_process(c726dd14,c6b7dd38,c073cd0d,344,c714b000,...) at 0xc0b60ed9 =3D usb_process+0x159 fork_exit(c0b60d80,c726dd14,c6b7dd38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6b7dd70, ebp =3D 0 --- Tracing command usb pid 13 tid 100045 td 0xc71bda00 sched_switch(c71bda00,0,104,191,a6df0517,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c71bda00,0,c0746206,260,c71bda00,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_wait(c725cdac,0,c6b79cbc,1,0,...) at 0xc0541ab3 =3D sleepq_wait+0x63 _cv_wait(c725cdac,c725ce4c,c0b67bc2,6c,c725cdb4,...) at 0xc04c6970 =3D _cv_wait+0x240 usb_process(c725cda4,c6b79d38,c073cd0d,344,c714b000,...) at 0xc0b60ed9 =3D usb_process+0x159 fork_exit(c0b60d80,c725cda4,c6b79d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6b79d70, ebp =3D 0 --- Tracing command usb pid 13 tid 100044 td 0xc71bdc80 sched_switch(c71bdc80,0,104,191,d3f91ca8,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c71bdc80,0,c0746206,260,c71bdc80,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_wait(c725cd7c,0,c6b76cbc,1,0,...) at 0xc0541ab3 =3D sleepq_wait+0x63 _cv_wait(c725cd7c,c725ce4c,c0b67bc2,6c,c725cd84,...) at 0xc04c6970 =3D _cv_wait+0x240 usb_process(c725cd74,c6b76d38,c073cd0d,344,c714b000,...) at 0xc0b60ed9 =3D usb_process+0x159 fork_exit(c0b60d80,c725cd74,c6b76d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6b76d70, ebp =3D 0 --- Tracing command usb pid 13 tid 100043 td 0xc7243000 sched_switch(c7243000,0,104,191,a6d988ab,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7243000,0,c0746206,260,c7243000,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_wait(c725cd4c,0,c6b73cbc,1,0,...) at 0xc0541ab3 =3D sleepq_wait+0x63 _cv_wait(c725cd4c,c725ce4c,c0b67bc2,6c,c725cd54,...) at 0xc04c6970 =3D _cv_wait+0x240 usb_process(c725cd44,c6b73d38,c073cd0d,344,c714b000,...) at 0xc0b60ed9 =3D usb_process+0x159 fork_exit(c0b60d80,c725cd44,c6b73d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6b73d70, ebp =3D 0 --- Tracing command usb pid 13 tid 100042 td 0xc7243280 sched_switch(c7243280,0,104,191,a6d9709f,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7243280,0,c0746206,260,c7243280,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_wait(c725cd1c,0,c6b70cbc,1,0,...) at 0xc0541ab3 =3D sleepq_wait+0x63 _cv_wait(c725cd1c,c725ce4c,c0b67bc2,6c,c725cd24,...) at 0xc04c6970 =3D _cv_wait+0x240 usb_process(c725cd14,c6b70d38,c073cd0d,344,c714b000,...) at 0xc0b60ed9 =3D usb_process+0x159 fork_exit(c0b60d80,c725cd14,c6b70d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6b70d70, ebp =3D 0 --- Tracing command usb pid 13 tid 100040 td 0xc7243780 sched_switch(c7243780,0,104,191,a6d95803,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7243780,0,c0746206,260,c7243780,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_wait(c7248dac,0,c6b69cbc,1,0,...) at 0xc0541ab3 =3D sleepq_wait+0x63 _cv_wait(c7248dac,c7248e4c,c0b67bc2,6c,c7248db4,...) at 0xc04c6970 =3D _cv_wait+0x240 usb_process(c7248da4,c6b69d38,c073cd0d,344,c714b000,...) at 0xc0b60ed9 =3D usb_process+0x159 fork_exit(c0b60d80,c7248da4,c6b69d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6b69d70, ebp =3D 0 --- Tracing command usb pid 13 tid 100039 td 0xc7243a00 sched_switch(c7243a00,0,104,191,d3d3ac9b,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7243a00,0,c0746206,260,c7243a00,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_wait(c7248d7c,0,c6b66cbc,1,0,...) at 0xc0541ab3 =3D sleepq_wait+0x63 _cv_wait(c7248d7c,c7248e4c,c0b67bc2,6c,c7248d84,...) at 0xc04c6970 =3D _cv_wait+0x240 usb_process(c7248d74,c6b66d38,c073cd0d,344,c714b000,...) at 0xc0b60ed9 =3D usb_process+0x159 fork_exit(c0b60d80,c7248d74,c6b66d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6b66d70, ebp =3D 0 --- Tracing command usb pid 13 tid 100038 td 0xc7243c80 sched_switch(c7243c80,0,104,191,a6d42976,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7243c80,0,c0746206,260,c7243c80,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_wait(c7248d4c,0,c6b63cbc,1,0,...) at 0xc0541ab3 =3D sleepq_wait+0x63 _cv_wait(c7248d4c,c7248e4c,c0b67bc2,6c,c7248d54,...) at 0xc04c6970 =3D _cv_wait+0x240 usb_process(c7248d44,c6b63d38,c073cd0d,344,c714b000,...) at 0xc0b60ed9 =3D usb_process+0x159 fork_exit(c0b60d80,c7248d44,c6b63d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6b63d70, ebp =3D 0 --- Tracing command usb pid 13 tid 100037 td 0xc7244000 sched_switch(c7244000,0,104,191,a6d41350,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7244000,0,c0746206,260,c7244000,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_wait(c7248d1c,0,c6b60cbc,1,0,...) at 0xc0541ab3 =3D sleepq_wait+0x63 _cv_wait(c7248d1c,c7248e4c,c0b67bc2,6c,c7248d24,...) at 0xc04c6970 =3D _cv_wait+0x240 usb_process(c7248d14,c6b60d38,c073cd0d,344,c714b000,...) at 0xc0b60ed9 =3D usb_process+0x159 fork_exit(c0b60d80,c7248d14,c6b60d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6b60d70, ebp =3D 0 --- Tracing command usb pid 13 tid 100035 td 0xc7244500 sched_switch(c7244500,0,104,191,a6d3f7ff,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7244500,0,c0746206,260,c7244500,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_wait(c721ddac,0,c6b59cbc,1,0,...) at 0xc0541ab3 =3D sleepq_wait+0x63 _cv_wait(c721ddac,c721de4c,c0b67bc2,6c,c721ddb4,...) at 0xc04c6970 =3D _cv_wait+0x240 usb_process(c721dda4,c6b59d38,c073cd0d,344,c714b000,...) at 0xc0b60ed9 =3D usb_process+0x159 fork_exit(c0b60d80,c721dda4,c6b59d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6b59d70, ebp =3D 0 --- Tracing command usb pid 13 tid 100034 td 0xc71bc000 sched_switch(c71bc000,0,104,191,d1ceb6d7,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c71bc000,0,c0746206,260,c71bc000,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_wait(c721dd7c,0,c6b56cbc,1,0,...) at 0xc0541ab3 =3D sleepq_wait+0x63 _cv_wait(c721dd7c,c721de4c,c0b67bc2,6c,c721dd84,...) at 0xc04c6970 =3D _cv_wait+0x240 usb_process(c721dd74,c6b56d38,c073cd0d,344,c714b000,...) at 0xc0b60ed9 =3D usb_process+0x159 fork_exit(c0b60d80,c721dd74,c6b56d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6b56d70, ebp =3D 0 --- Tracing command usb pid 13 tid 100033 td 0xc71bc280 sched_switch(c71bc280,0,104,191,a6ce48e1,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c71bc280,0,c0746206,260,c71bc280,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_wait(c721dd4c,0,c6b53cbc,1,0,...) at 0xc0541ab3 =3D sleepq_wait+0x63 _cv_wait(c721dd4c,c721de4c,c0b67bc2,6c,c721dd54,...) at 0xc04c6970 =3D _cv_wait+0x240 usb_process(c721dd44,c6b53d38,c073cd0d,344,c714b000,...) at 0xc0b60ed9 =3D usb_process+0x159 fork_exit(c0b60d80,c721dd44,c6b53d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6b53d70, ebp =3D 0 --- Tracing command usb pid 13 tid 100032 td 0xc71bc500 sched_switch(c71bc500,0,104,191,a6ce3303,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c71bc500,0,c0746206,260,c71bc500,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_wait(c721dd1c,0,c6b50cbc,1,0,...) at 0xc0541ab3 =3D sleepq_wait+0x63 _cv_wait(c721dd1c,c721de4c,c0b67bc2,6c,c721dd24,...) at 0xc04c6970 =3D _cv_wait+0x240 usb_process(c721dd14,c6b50d38,c073cd0d,344,c714b000,...) at 0xc0b60ed9 =3D usb_process+0x159 fork_exit(c0b60d80,c721dd14,c6b50d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6b50d70, ebp =3D 0 --- Tracing command yarrow pid 12 tid 100017 td 0xc714ca00 sched_switch(c714ca00,0,104,191,68eca2cc,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c714ca00,0,c0746206,283,2,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_timedwait(c07d0564,0,c0bb2e24,2,0,...) at 0xc05419bb =3D sleepq_timedwait+0x6b _sleep(c07d0564,0,0,c0bb2e24,64,...) at 0xc0511599 =3D _sleep+0x339 pause(c0bb2e24,64,c0bb2e26,111,0,...) at 0xc0511727 =3D pause+0x47 random_kthread(0,c68a8d38,c073cd0d,344,c714b2a8,...) at 0xc0bac0a6 =3D random_kthread+0x1b6 fork_exit(c0babef0,0,c68a8d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc68a8d70, ebp =3D 0 --- Tracing command crypto returns pid 6 tid 100016 td 0xc714cc80 sched_switch(c714cc80,0,104,191,a6cdf2c8,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,5c,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c714cc80,0,c0746206,260,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_wait(c094a9ac,5c,c07604e8,0,0,...) at 0xc0541ab3 =3D sleepq_wait+0x6= 3 _sleep(c094a9ac,c094a9bc,5c,c07604e8,0,...) at 0xc05115cb =3D _sleep+0x36b crypto_ret_proc(0,c68a5d38,c073cd0d,344,c714b550,...) at 0xc06514bc =3D crypto_ret_proc+0x1cc fork_exit(c06512f0,0,c68a5d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc68a5d70, ebp =3D 0 --- Tracing command crypto pid 5 tid 100015 td 0xc7046280 sched_switch(c7046280,0,104,191,a6cdd885,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,5c,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7046280,0,c0746206,260,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_wait(c094a988,5c,c07606ec,0,0,...) at 0xc0541ab3 =3D sleepq_wait+0x6= 3 _sleep(c094a988,c094a998,5c,c07606ec,0,...) at 0xc05115cb =3D _sleep+0x36b crypto_proc(0,c68a2d38,c073cd0d,344,c7042000,...) at 0xc0652991 =3D crypto_proc+0x311 fork_exit(c0652680,0,c68a2d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc68a2d70, ebp =3D 0 --- Tracing command g_down pid 4 tid 100014 td 0xc7046500 sched_switch(c7046500,0,104,191,c79cdd85,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,4c,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7046500,0,c0746206,260,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_wait(c07ce2e4,4c,c0759c99,0,0,...) at 0xc0541ab3 =3D sleepq_wait+0x6= 3 _sleep(c07ce2e4,c07ce248,24c,c0759c99,0,...) at 0xc05115cb =3D _sleep+0x36b g_io_schedule_down(c7046500,0,c073913e,74,0,...) at 0xc04af5d6 =3D g_io_schedule_down+0x56 g_down_procbody(0,c689fd38,c073cd0d,344,c70422a8,...) at 0xc04afc4d =3D g_down_procbody+0x8d fork_exit(c04afbc0,0,c689fd38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc689fd70, ebp =3D 0 --- Tracing command g_up pid 3 tid 100013 td 0xc7046780 sched_switch(c7046780,0,104,191,c84f4dd4,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,4c,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7046780,0,c0746206,260,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_wait(c07ce2e0,4c,c0759c99,0,0,...) at 0xc0541ab3 =3D sleepq_wait+0x6= 3 _sleep(c07ce2e0,c07ce268,24c,c0759c99,0,...) at 0xc05115cb =3D _sleep+0x36b g_io_schedule_up(c7046780,0,c073913e,5d,0,...) at 0xc04af56e =3D g_io_schedule_up+0x11e g_up_procbody(0,c689cd38,c073cd0d,344,c7042550,...) at 0xc04afcdd =3D g_up_procbody+0x8d fork_exit(c04afc50,0,c689cd38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc689cd70, ebp =3D 0 --- Tracing command g_event pid 2 tid 100012 td 0xc7046a00 sched_switch(c7046a00,0,104,191,679d5366,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,4c,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7046a00,0,c0746206,283,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_timedwait(c07ce2d8,4c,c0759c99,0,0,...) at 0xc05419bb =3D sleepq_timedwait+0x6b _sleep(c07ce2d8,0,4c,c0759c99,64,...) at 0xc0511599 =3D _sleep+0x339 g_event_procbody(0,c6899d38,c073cd0d,344,c70427f8,...) at 0xc04afdab =3D g_event_procbody+0xcb fork_exit(c04afce0,0,c6899d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6899d70, ebp =3D 0 --- Tracing command intr pid 11 tid 100153 td 0xc8061c80 sched_switch(c8061c80,0,109,191,6644ded3,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(109,0,c073cfc9,52d,c71eb570,...) at 0xc05110b0 =3D mi_switch+0x20= 0 ithread_loop(c75b6a50,eb6afd38,c073cd0d,344,c7042aa0,...) at 0xc04e16d6 =3D ithread_loop+0x1f6 fork_exit(c04e14e0,c75b6a50,eb6afd38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xeb6afd70, ebp =3D 0 --- Tracing command intr pid 11 tid 100060 td 0xc7385280 fork_trampoline() at 0xc06d7b48 =3D fork_trampoline Tracing command intr pid 11 tid 100059 td 0xc7385500 sched_switch(c7385500,0,109,191,ca9dc676,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(109,0,c073cfc9,52d,c7040cf0,...) at 0xc05110b0 =3D mi_switch+0x20= 0 ithread_loop(c737e810,c6bc4d38,c073cd0d,344,c7042aa0,...) at 0xc04e16d6 =3D ithread_loop+0x1f6 fork_exit(c04e14e0,c737e810,c6bc4d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6bc4d70, ebp =3D 0 --- Tracing command intr pid 11 tid 100058 td 0xc7385780 sched_switch(c7385780,0,109,191,d139277c,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(109,0,c073cfc9,52d,c7041370,...) at 0xc05110b0 =3D mi_switch+0x20= 0 ithread_loop(c737e9a0,c6bc1d38,c073cd0d,344,c7042aa0,...) at 0xc04e16d6 =3D ithread_loop+0x1f6 fork_exit(c04e14e0,c737e9a0,c6bc1d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6bc1d70, ebp =3D 0 --- Tracing command intr pid 11 tid 100056 td 0xc7244a00 sched_switch(c7244a00,0,109,191,c84e9863,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(109,0,c073cfc9,52d,c7040bf0,...) at 0xc05110b0 =3D mi_switch+0x20= 0 ithread_loop(c736e120,c6ba4d38,c073cd0d,344,c7042aa0,...) at 0xc04e16d6 =3D ithread_loop+0x1f6 fork_exit(c04e14e0,c736e120,c6ba4d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6ba4d70, ebp =3D 0 --- Tracing command intr pid 11 tid 100041 td 0xc7243500 sched_switch(c7243500,0,109,191,a6a2bedf,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(109,0,c073cfc9,52d,c70409f0,...) at 0xc05110b0 =3D mi_switch+0x20= 0 ithread_loop(c71f5b80,c6b6dd38,c073cd0d,344,c7042aa0,...) at 0xc04e16d6 =3D ithread_loop+0x1f6 fork_exit(c04e14e0,c71f5b80,c6b6dd38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6b6dd70, ebp =3D 0 --- Tracing command intr pid 11 tid 100036 td 0xc7244280 sched_switch(c7244280,0,109,191,9f7398bf,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(109,0,c073cfc9,52d,c7040970,...) at 0xc05110b0 =3D mi_switch+0x20= 0 ithread_loop(c71f5970,c6b5dd38,c073cd0d,344,c7042aa0,...) at 0xc04e16d6 =3D ithread_loop+0x1f6 fork_exit(c04e14e0,c71f5970,c6b5dd38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6b5dd70, ebp =3D 0 --- Tracing command intr pid 11 tid 100031 td 0xc71bc780 fork_trampoline() at 0xc06d7b48 =3D fork_trampoline Tracing command intr pid 11 tid 100029 td 0xc71bcc80 fork_trampoline() at 0xc06d7b48 =3D fork_trampoline Tracing command intr pid 11 tid 100028 td 0xc71bd000 sched_switch(c71bd000,0,109,191,dce5eee5,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(109,0,c073cfc9,52d,c71dadf0,...) at 0xc05110b0 =3D mi_switch+0x20= 0 ithread_loop(c71ed000,c690bd38,c073cd0d,344,c7042aa0,...) at 0xc04e16d6 =3D ithread_loop+0x1f6 fork_exit(c04e14e0,c71ed000,c690bd38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc690bd70, ebp =3D 0 --- Tracing command intr pid 11 tid 100027 td 0xc71bd280 fork_trampoline() at 0xc06d7b48 =3D fork_trampoline Tracing command intr pid 11 tid 100026 td 0xc71bd500 sched_switch(c71bd500,0,109,191,78d08802,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(109,0,c073cfc9,52d,c715a870,...) at 0xc05110b0 =3D mi_switch+0x20= 0 ithread_loop(c70832e0,c68c3d38,c073cd0d,344,c7042aa0,...) at 0xc04e16d6 =3D ithread_loop+0x1f6 fork_exit(c04e14e0,c70832e0,c68c3d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc68c3d70, ebp =3D 0 --- Tracing command intr pid 11 tid 100025 td 0xc71bd780 sched_switch(c71bd780,0,109,191,8a662702,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(109,0,c073cfc9,52d,c715aa70,...) at 0xc05110b0 =3D mi_switch+0x20= 0 ithread_loop(c7083390,c68c0d38,c073cd0d,344,c7042aa0,...) at 0xc04e16d6 =3D ithread_loop+0x1f6 fork_exit(c04e14e0,c7083390,c68c0d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc68c0d70, ebp =3D 0 --- Tracing command intr pid 11 tid 100024 td 0xc7084780 fork_trampoline() at 0xc06d7b48 =3D fork_trampoline Tracing command intr pid 11 tid 100019 td 0xc714c500 sched_switch(c714c500,0,109,191,8efb5128,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(109,0,c073cfc9,52d,c715ab70,...) at 0xc05110b0 =3D mi_switch+0x20= 0 ithread_loop(c70833d0,c68aed38,c073cd0d,344,c7042aa0,...) at 0xc04e16d6 =3D ithread_loop+0x1f6 fork_exit(c04e14e0,c70833d0,c68aed38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc68aed70, ebp =3D 0 --- Tracing command intr pid 11 tid 100011 td 0xc7046c80 sched_switch(c7046c80,0,109,191,d7283908,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(109,0,c073cfc9,52d,c7040470,...) at 0xc05110b0 =3D mi_switch+0x20= 0 ithread_loop(c7083790,c6896d38,c073cd0d,344,c7042aa0,...) at 0xc04e16d6 =3D ithread_loop+0x1f6 fork_exit(c04e14e0,c7083790,c6896d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6896d70, ebp =3D 0 --- Tracing command intr pid 11 tid 100010 td 0xc7084000 sched_switch(c7084000,0,109,191,1e5b8ce5,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(109,0,c073cfc9,52d,c70404f0,...) at 0xc05110b0 =3D mi_switch+0x20= 0 ithread_loop(c70837a0,c6893d38,c073cd0d,344,c7042aa0,...) at 0xc04e16d6 =3D ithread_loop+0x1f6 fork_exit(c04e14e0,c70837a0,c6893d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6893d70, ebp =3D 0 --- Tracing command intr pid 11 tid 100009 td 0xc7084280 sched_switch(c7084280,0,109,191,5c9afc95,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(109,0,c073cfc9,52d,c7040570,...) at 0xc05110b0 =3D mi_switch+0x20= 0 ithread_loop(c70837b0,c6890d38,c073cd0d,344,c7042aa0,...) at 0xc04e16d6 =3D ithread_loop+0x1f6 fork_exit(c04e14e0,c70837b0,c6890d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6890d70, ebp =3D 0 --- Tracing command intr pid 11 tid 100008 td 0xc7084500 sched_switch(c7084500,0,109,191,a61ca00c,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(109,0,c073cfc9,52d,c70405f0,...) at 0xc05110b0 =3D mi_switch+0x20= 0 ithread_loop(c70837c0,c688dd38,c073cd0d,344,c7042aa0,...) at 0xc04e16d6 =3D ithread_loop+0x1f6 fork_exit(c04e14e0,c70837c0,c688dd38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc688dd70, ebp =3D 0 --- Tracing command intr pid 11 tid 100007 td 0xc7044000 sched_switch(c7044000,0,109,191,71f87d19,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(109,0,c073cfc9,52d,c7040670,...) at 0xc05110b0 =3D mi_switch+0x20= 0 ithread_loop(c70837d0,c688ad38,c073cd0d,344,c7042aa0,...) at 0xc04e16d6 =3D ithread_loop+0x1f6 fork_exit(c04e14e0,c70837d0,c688ad38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc688ad70, ebp =3D 0 --- Tracing command intr pid 11 tid 100006 td 0xc7044280 fork_trampoline() at 0xc06d7b48 =3D fork_trampoline Tracing command idle pid 10 tid 100005 td 0xc7044500 sched_switch(c7044500,0,60c,18c,713fd4d8,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(60c,0,c0743974,806,0,...) at 0xc05110b0 =3D mi_switch+0x200 sched_preempt(c7044500,0,1f4,c6883c74,c06d81fe,...) at 0xc0528f9f =3D sched_preempt+0x9f ipi_bitmap_handler(8,c7040028,c0740028,c7081200,c7081218,...) at 0xc06e73b4 =3D ipi_bitmap_handler+0x34 Xipi_intr_bitmap_handler() at 0xc06d81fe =3D Xipi_intr_bitmap_handler+0x2e --- interrupt, eip =3D 0xc0c11425, esp =3D 0xc6883c74, ebp =3D 0xc6883c74 -= -- acpi_cpu_c1(0,ffffffff,0,c052a9a1,c07e2950,...) at 0xc0c11425 =3D acpi_cpu_= c1+0x5 acpi_cpu_idle(c6883cb4,c06e354b,0,c6883cf8,c052ac9e,...) at 0xc0c0a1f1 =3D acpi_cpu_idle+0x1c1 cpu_idle_acpi(0,c6883cf8,c052ac9e,0,c6883cd4,...) at 0xc06e46fb =3D cpu_idle_acpi+0x1b cpu_idle(0,c6883cd4,c0743974,9fa,c7044500,...) at 0xc06e354b =3D cpu_idle+0= x1b sched_idletd(0,c6883d38,c073cd0d,344,c7042d48,...) at 0xc052ac9e =3D sched_idletd+0x23e fork_exit(c052aa60,0,c6883d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6883d70, ebp =3D 0 --- Tracing command idle pid 10 tid 100004 td 0xc7044780 cpustop_handler(2,c6880c28,c06f5456,c07d077c,c6880bc0,...) at 0xc06e8712 =3D cpustop_handler+0x32 ipi_nmi_handler(c07d077c,c6880bc0,c04f8444,c07d077c,c7042d48,...) at 0xc06e879f =3D ipi_nmi_handler+0x2f trap(c6880c34) at 0xc06f5456 =3D trap+0x36 calltrap() at 0xc06d7adb =3D calltrap+0x6 --- trap 0x13, eip =3D 0xc0c11425, esp =3D 0xc6880c74, ebp =3D 0xc6880c74 -= -- acpi_cpu_c1(0,ffffffff,1,c052a9a1,c07e2950,...) at 0xc0c11425 =3D acpi_cpu_= c1+0x5 acpi_cpu_idle(c6880cb4,c06e354b,1,c6880cf8,c052ac9e,...) at 0xc0c0a1f1 =3D acpi_cpu_idle+0x1c1 cpu_idle_acpi(1,c6880cf8,c052ac9e,1,c6880cd4,...) at 0xc06e46fb =3D cpu_idle_acpi+0x1b cpu_idle(1,c6880cd4,c0743974,9fa,c7044780,...) at 0xc06e354b =3D cpu_idle+0= x1b sched_idletd(0,c6880d38,c073cd0d,344,c7042d48,...) at 0xc052ac9e =3D sched_idletd+0x23e fork_exit(c052aa60,0,c6880d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6880d70, ebp =3D 0 --- Tracing command idle pid 10 tid 100003 td 0xc7044a00 cpustop_handler(4,c687dc70,c06f5456,c04c591b,c07d51c0,...) at 0xc06e8712 =3D cpustop_handler+0x32 ipi_nmi_handler(c04c591b,c07d51c0,0,c073b0ba,c7042d48,...) at 0xc06e879f =3D ipi_nmi_handler+0x2f trap(c687dc7c) at 0xc06f5456 =3D trap+0x36 calltrap() at 0xc06d7adb =3D calltrap+0x6 --- trap 0x13, eip =3D 0xc052ac7c, esp =3D 0xc687dcbc, ebp =3D 0xc687dcf8 -= -- sched_idletd(0,c687dd38,c073cd0d,344,c7042d48,...) at 0xc052ac7c =3D sched_idletd+0x21c fork_exit(c052aa60,0,c687dd38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc687dd70, ebp =3D 0 --- Tracing command idle pid 10 tid 100002 td 0xc7044c80 cpustop_handler(8,c687ac28,c06f5456,c07d07f4,c687abc0,...) at 0xc06e8712 =3D cpustop_handler+0x32 ipi_nmi_handler(c07d07f4,c687abc0,c04f8444,c07d07f4,c7042d48,...) at 0xc06e879f =3D ipi_nmi_handler+0x2f trap(c687ac34) at 0xc06f5456 =3D trap+0x36 calltrap() at 0xc06d7adb =3D calltrap+0x6 --- trap 0x13, eip =3D 0xc0c11425, esp =3D 0xc687ac74, ebp =3D 0xc687ac74 -= -- acpi_cpu_c1(0,ffffffff,3,c052a9a1,c07e2950,...) at 0xc0c11425 =3D acpi_cpu_= c1+0x5 acpi_cpu_idle(c687acb4,c06e354b,0,c687acf8,c052ac9e,...) at 0xc0c0a1f1 =3D acpi_cpu_idle+0x1c1 cpu_idle_acpi(0,c687acf8,c052ac9e,0,c687acd4,...) at 0xc06e46fb =3D cpu_idle_acpi+0x1b cpu_idle(0,c687acd4,c0743974,3af,c7044c80,...) at 0xc06e354b =3D cpu_idle+0= x1b sched_idletd(0,c687ad38,c073cd0d,344,c7042d48,...) at 0xc052ac9e =3D sched_idletd+0x23e fork_exit(c052aa60,0,c687ad38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc687ad70, ebp =3D 0 --- Tracing command init pid 1 tid 100001 td 0xc7046000 sched_switch(c7046000,0,104,191,74f1a766,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,5c,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7046000,0,c0746206,1a0,5c,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_catch_signals(c0746206,160,0,100,100,...) at 0xc0540ec7 =3D sleepq_catch_signals+0xb7 sleepq_wait_sig(c7043000,5c,c0748bc0,100,0,...) at 0xc0541a07 =3D sleepq_wait_sig+0x17 _sleep(c7043000,c7043088,15c,c0748bc0,0,...) at 0xc05115b4 =3D _sleep+0x354 kern_wait(c7046000,ffffffff,c6876c78,0,0,...) at 0xc04dc516 =3D kern_wait+0= xb46 wait4(c7046000,c6876cf8,c0771550,c07489d7,282,...) at 0xc04dc58b =3D wait4+= 0x3b syscall(c6876d38) at 0xc06f51e5 =3D syscall+0x1f5 Xint0x80_syscall() at 0xc06d7b40 =3D Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip =3D 0x8054b2b, esp =3D 0xbfbfe90c, ebp =3D 0xbfbfe928 --- Tracing command kernel pid 0 tid 100116 td 0xc7c31780 sched_switch(c7c31780,0,104,191,71f8975c,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7c31780,0,c0746206,260,c7c31780,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_wait(c75c5d80,0,c07428c0,c0759c99,0,...) at 0xc0541ab3 =3D sleepq_wa= it+0x63 msleep_spin(c75c5d80,c75c5d98,c0759c99,0,c074025d,...) at 0xc051197d =3D msleep_spin+0x21d taskqueue_thread_loop(c7f9fd20,e9618d38,c073cd0d,344,c07ce380,...) at 0xc05433c4 =3D taskqueue_thread_loop+0x94 fork_exit(c0543330,c7f9fd20,e9618d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xe9618d70, ebp =3D 0 --- Tracing command kernel pid 0 tid 100061 td 0xc7385000 kdb_enter(c0741c4a,c0741c4a,c073b16c,c6bcbcd0,0,...) at 0xc0536cea =3D kdb_enter+0x3a panic(c073b16c,c071770c,c76f0a00,3cea7,c7035dd0,...) at 0xc0508226 =3D pani= c+0x136 deadlkres(0,c6bcbd38,c073cd0d,344,c07ce380,...) at 0xc04c5171 =3D deadlkres= +0x361 fork_exit(c04c4e10,0,c6bcbd38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6bcbd70, ebp =3D 0 --- Tracing command kernel pid 0 tid 100054 td 0xc727c000 sched_switch(c727c000,0,104,191,6fa4f311,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c727c000,0,c0746206,260,c727c000,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_wait(c727ecc0,0,c07428c0,c0759c99,0,...) at 0xc0541ab3 =3D sleepq_wa= it+0x63 msleep_spin(c727ecc0,c727ecd8,c0759c99,0,c074025d,...) at 0xc051197d =3D msleep_spin+0x21d taskqueue_thread_loop(c72a449c,c6b9bd38,c073cd0d,344,c07ce380,...) at 0xc05433c4 =3D taskqueue_thread_loop+0x94 fork_exit(c0543330,c72a449c,c6b9bd38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6b9bd70, ebp =3D 0 --- Tracing command kernel pid 0 tid 100030 td 0xc71bca00 sched_switch(c71bca00,0,104,191,d3e324f5,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c71bca00,0,c0746206,260,c71bca00,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_wait(c7204300,0,c07428c0,c0759c99,0,...) at 0xc0541ab3 =3D sleepq_wa= it+0x63 msleep_spin(c7204300,c7204318,c0759c99,0,c074025d,...) at 0xc051197d =3D msleep_spin+0x21d taskqueue_thread_loop(c7209520,c6b49d38,c073cd0d,344,c07ce380,...) at 0xc05433c4 =3D taskqueue_thread_loop+0x94 fork_exit(c0543330,c7209520,c6b49d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc6b49d70, ebp =3D 0 --- Tracing command kernel pid 0 tid 100023 td 0xc7084a00 sched_switch(c7084a00,0,104,191,ccf1f8fb,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7084a00,0,c0746206,260,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_wait(c71ab100,0,c0759c99,0,0,...) at 0xc0541ab3 =3D sleepq_wait+0x63 _sleep(c71ab100,c71ab118,0,c0759c99,0,...) at 0xc05115cb =3D _sleep+0x36b taskqueue_thread_loop(c07e2b48,c68bad38,c073cd0d,344,c07ce380,...) at 0xc05433ea =3D taskqueue_thread_loop+0xba fork_exit(c0543330,c07e2b48,c68bad38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc68bad70, ebp =3D 0 --- Tracing command kernel pid 0 tid 100022 td 0xc7084c80 sched_switch(c7084c80,0,104,191,a6f2cbf1,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c7084c80,0,c0746206,260,c7084c80,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_wait(c71ab140,0,c07428c0,c0759c99,0,...) at 0xc0541ab3 =3D sleepq_wa= it+0x63 msleep_spin(c71ab140,c71ab158,c0759c99,0,c074025d,...) at 0xc051197d =3D msleep_spin+0x21d taskqueue_thread_loop(c0c230fc,c68b7d38,c073cd0d,344,c07ce380,...) at 0xc05433c4 =3D taskqueue_thread_loop+0x94 fork_exit(c0543330,c0c230fc,c68b7d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc68b7d70, ebp =3D 0 --- Tracing command kernel pid 0 tid 100021 td 0xc714c000 sched_switch(c714c000,0,104,191,a6f2b79f,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c714c000,0,c0746206,260,c714c000,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_wait(c71ab140,0,c07428c0,c0759c99,0,...) at 0xc0541ab3 =3D sleepq_wa= it+0x63 msleep_spin(c71ab140,c71ab158,c0759c99,0,c074025d,...) at 0xc051197d =3D msleep_spin+0x21d taskqueue_thread_loop(c0c230fc,c68b4d38,c073cd0d,344,c07ce380,...) at 0xc05433c4 =3D taskqueue_thread_loop+0x94 fork_exit(c0543330,c0c230fc,c68b4d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc68b4d70, ebp =3D 0 --- Tracing command kernel pid 0 tid 100020 td 0xc714c280 sched_switch(c714c280,0,104,191,a6f2a1e5,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c714c280,0,c0746206,260,c714c280,...) at 0xc0540ddf =3D sleepq_switch+0x15f sleepq_wait(c71ab140,0,c07428c0,c0759c99,0,...) at 0xc0541ab3 =3D sleepq_wa= it+0x63 msleep_spin(c71ab140,c71ab158,c0759c99,0,c074025d,...) at 0xc051197d =3D msleep_spin+0x21d taskqueue_thread_loop(c0c230fc,c68b1d38,c073cd0d,344,c07ce380,...) at 0xc05433c4 =3D taskqueue_thread_loop+0x94 fork_exit(c0543330,c0c230fc,c68b1d38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc68b1d70, ebp =3D 0 --- Tracing command kernel pid 0 tid 100018 td 0xc714c780 sched_switch(c714c780,0,104,191,a6f12434,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,0,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c714c780,0,c0746206,260,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_wait(c71ab200,0,c0759c99,0,0,...) at 0xc0541ab3 =3D sleepq_wait+0x63 _sleep(c71ab200,c71ab218,0,c0759c99,0,...) at 0xc05115cb =3D _sleep+0x36b taskqueue_thread_loop(c07cec84,c68abd38,c073cd0d,344,c07ce380,...) at 0xc05433ea =3D taskqueue_thread_loop+0xba fork_exit(c0543330,c07cec84,c68abd38) at 0xc04de528 =3D fork_exit+0xb8 fork_trampoline() at 0xc06d7b50 =3D fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xc68abd70, ebp =3D 0 --- Tracing command kernel pid 0 tid 100000 td 0xc07ce630 sched_switch(c07ce630,0,104,191,6a913f43,...) at 0xc0529db0 =3D sched_switc= h+0x330 mi_switch(104,0,c0746206,1eb,44,...) at 0xc05110b0 =3D mi_switch+0x200 sleepq_switch(c07ce630,0,c0746206,283,0,...) at 0xc0540ddf =3D sleepq_switc= h+0x15f sleepq_timedwait(c07ce380,44,c0744205,0,0,...) at 0xc05419bb =3D sleepq_timedwait+0x6b _sleep(c07ce380,0,44,c0744205,2710,...) at 0xc0511599 =3D _sleep+0x339 scheduler(0,101ec00,101ec00,101e000,1025000,...) at 0xc069f77e =3D schedule= r+0x23e mi_startup() at 0xc04c3116 =3D mi_startup+0x96 begin() at 0xc043ada5 =3D begin+0x2c db> capture off -- Sincerely, Dmitry nic-hdl: LYNX-RIPE From owner-freebsd-fs@FreeBSD.ORG Fri Jun 18 07:18:08 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EB6C3106564A; Fri, 18 Jun 2010 07:18:07 +0000 (UTC) (envelope-from lynx.ripe@gmail.com) Received: from mail-iw0-f182.google.com (mail-iw0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 666708FC1E; Fri, 18 Jun 2010 07:18:07 +0000 (UTC) Received: by iwn7 with SMTP id 7so985285iwn.13 for ; Fri, 18 Jun 2010 00:18:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=LVDv93pnNJT9lcqpw99Fb3tOWLyxtEfBpB/0XSTQjMQ=; b=Z/1BowuV7AKZHNT/ZJa/N9T7itQL+oFKqS08ybprtVDaZEIaHZRetH0o13Z+sdaKt9 OZcw362vClCwJxOmNK+69Ks00gozcYZxUC6NDy+Oh10s7v4lFJNKnw2qKm+RPPYMBPz7 EJVM5PeXc/SykeRxmBsFoqODjCgTWhqhUwXcI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=idelFqNtv38yc1Q+ZDlnrPC6j79nlQPNI5buMWPjqkjzfoPy8AK5Kk/EfIlXnxliik lJMyRgrnkim2hiO7oel28lxYTriJnbS+dE7sYA5XWM4itqhX4Fcy2HH/yHajl+9uBWnT i2M+9Lh55aJzBEPblSLSCBoogHfEWb7lpGN5I= MIME-Version: 1.0 Received: by 10.231.85.147 with SMTP id o19mr760350ibl.82.1276845486894; Fri, 18 Jun 2010 00:18:06 -0700 (PDT) Received: by 10.231.146.199 with HTTP; Fri, 18 Jun 2010 00:18:06 -0700 (PDT) In-Reply-To: <20100617083239.GZ13238@deviant.kiev.zoral.com.ua> References: <201006162220.o5GMK3uM002900@freefall.freebsd.org> <20100617072841.GY13238@deviant.kiev.zoral.com.ua> <20100617083239.GZ13238@deviant.kiev.zoral.com.ua> Date: Fri, 18 Jun 2010 10:18:06 +0300 Message-ID: From: Dmitry Pryanishnikov To: Kostik Belousov Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org, trasz@freebsd.org Subject: Re: kern/147890: [ufs] [regression] ufs-related lock problem in RELENG_8 (18.04.2010 -> 20.04.2010) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Jun 2010 07:18:08 -0000 Hello! 2010/6/17 Kostik Belousov : > > What _could_ happen in your case is the relock in ufs_accessx for QUOTA. > ufs_delete_denied() calls VOP_ACCESSX for both parent directory > and target vnode while parent directory is locked. If source directory > is not exclusively locked, LOR can happen, since ufs_accessx has to > unlock and lock directory vnode. I am not sure how many similar cases > of VOP_ACCESS[X] is places in the problematic locations. > > Try the (untested) patch below. Thank you! The kernel with this patch applied (RELENG_8 date =3D 14.06.2010) has been running for 9 hours, the problem does not manifest itself (with unpatched kernel the problem typically appear within 1st hour of uptime). > > diff --git a/sys/ufs/ufs/ufs_lookup.c b/sys/ufs/ufs/ufs_lookup.c > index 0030c52..c779491 100644 > --- a/sys/ufs/ufs/ufs_lookup.c > +++ b/sys/ufs/ufs/ufs_lookup.c > @@ -77,6 +77,32 @@ SYSCTL_INT(_debug, OID_AUTO, dircheck, CTLFLAG_RW, &di= rchk, 0, ""); > =A0/* true if old FS format...*/ > =A0#define OFSFMT(vp) =A0 =A0 ((vp)->v_mount->mnt_maxsymlinklen <=3D 0) > > +#ifdef QUOTA > +static int > +ufs_lookup_upgrade_lock(struct vnode *vp) > +{ > + =A0 =A0 =A0 int error; > + > + =A0 =A0 =A0 ASSERT_VOP_LOCKED(vp, __FUNCTION__); > + =A0 =A0 =A0 if (VOP_ISLOCKED(vp) =3D=3D LK_EXCLUSIVE) > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 return (0); > + > + =A0 =A0 =A0 error =3D 0; > + > + =A0 =A0 =A0 /* > + =A0 =A0 =A0 =A0* Upgrade vnode lock, since getinoquota() > + =A0 =A0 =A0 =A0* requires exclusive lock to modify inode. > + =A0 =A0 =A0 =A0*/ > + =A0 =A0 =A0 vhold(vp); > + =A0 =A0 =A0 vn_lock(vp, LK_UPGRADE | LK_RETRY); > + =A0 =A0 =A0 VI_LOCK(vp); > + =A0 =A0 =A0 if (vp->v_iflag & VI_DOOMED) > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 error =3D ENOENT; > + =A0 =A0 =A0 vdropl(vp); > + =A0 =A0 =A0 return (error); > +} > +#endif > + > =A0static int > =A0ufs_delete_denied(struct vnode *vdp, struct vnode *tdp, struct ucred *= cred, > =A0 =A0 struct thread *td) > @@ -232,6 +258,13 @@ ufs_lookup_ino(struct vnode *vdp, struct vnode **vpp= , struct componentname *cnp, > =A0 =A0 =A0 =A0vnode_create_vobject(vdp, DIP(dp, i_size), cnp->cn_thread)= ; > > =A0 =A0 =A0 =A0bmask =3D VFSTOUFS(vdp->v_mount)->um_mountp->mnt_stat.f_io= size - 1; > +#ifdef QUOTA > + =A0 =A0 =A0 if ((nameiop =3D=3D DELETE || nameiop =3D=3D RENAME) && (fl= ags & ISLASTCN)) { > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 error =3D ufs_lookup_upgrade_lock(vdp); > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (error !=3D 0) > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 return (error); > + =A0 =A0 =A0 } > +#endif > > =A0restart: > =A0 =A0 =A0 =A0bp =3D NULL; > --=20 Sincerely, Dmitry nic-hdl: LYNX-RIPE From owner-freebsd-fs@FreeBSD.ORG Fri Jun 18 09:45:17 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 51F11106566C; Fri, 18 Jun 2010 09:45:17 +0000 (UTC) (envelope-from hiroshi@soupacific.com) Received: from mail.soupacific.com (mail.soupacific.com [211.19.53.201]) by mx1.freebsd.org (Postfix) with ESMTP id 1D0138FC16; Fri, 18 Jun 2010 09:45:16 +0000 (UTC) Received: from [127.0.0.1] (unknown [192.168.1.239]) by mail.soupacific.com (Postfix) with ESMTP id 73AE76C9EA; Fri, 18 Jun 2010 09:36:48 +0000 (UTC) Message-ID: <4C1B4027.6040506@soupacific.com> Date: Fri, 18 Jun 2010 18:45:11 +0900 From: "hiroshi@soupacific.com" User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4 MIME-Version: 1.0 To: Pawel Jakub Dawidek References: <20100613074921.GB1320@garage.freebsd.pl> <4C149A5C.3070401@soupacific.com> <20100613102401.GE1320@garage.freebsd.pl> <86eigavzsg.fsf@kopusha.home.net> <20100614095044.GH1721@garage.freebsd.pl> <868w6hwt2w.fsf@kopusha.home.net> <20100614153746.GN1721@garage.freebsd.pl> <4C165871.6060609@soupacific.com> <20100614214851.GB2498@garage.freebsd.pl> <4C199A70.2060207@soupacific.com> <20100617143309.GE1672@garage.freebsd.pl> In-Reply-To: <20100617143309.GE1672@garage.freebsd.pl> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: FreeBSD 8.1 and HAST X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Jun 2010 09:45:17 -0000 Thanks Trouble was caused by 8-STABLE and CURRENT. > > You probably created pool on FreeBSD-CURRENT and now trying to use it on > 8-STABLE. There was a version bump from v13 to v14 at some point. > You can see your pool version by running: > > # zpool get version > I will play around some more days. Thanks Hiroshi