From owner-freebsd-current@FreeBSD.ORG Mon Oct 11 20:45:41 2010 Return-Path: Delivered-To: Current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 884B41065670 for ; Mon, 11 Oct 2010 20:45:41 +0000 (UTC) (envelope-from sfourman@gmail.com) Received: from mail-qy0-f182.google.com (mail-qy0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id 3AE838FC1F for ; Mon, 11 Oct 2010 20:45:40 +0000 (UTC) Received: by qyk30 with SMTP id 30so467211qyk.13 for ; Mon, 11 Oct 2010 13:45:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:date:message-id :subject:from:to:content-type; bh=dlFmYF+aFCv1jd5wMfG/CsQZW8JvErpwr87KLkJ1kSI=; b=wfsmLa/FhV7O0lFEvMRpluMtXCY8QZYuAK6MduwXvllQELI4T9fzVWiYwzezpwZvQ0 qsidQ9PGVHaMaBrLpcxRrqta5Dx1stp0ds2zxKwi6tLoHV7mlYwNvv+cK1RiZ+KYfnvc T5QgyUCltu9bpPPOt9ixSH1/tMlek/80eymxY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=r5+4fUfdtlv1DSjrt5mGjMUwXua2QNwctdwstmlUVY0f/moSX56r3pR/qte2nL+PR5 HKtzEHsmGXLSDnzM031g8CX45l4BkoIC54xg1UZ1m7B8hjqrjX6TmcRR4XLfuW4PGvWQ EtOdO6YotmJlng78/DGLZBhA2GPzgH0OIcJ+c= MIME-Version: 1.0 Received: by 10.224.184.65 with SMTP id cj1mr4923074qab.95.1286829940016; Mon, 11 Oct 2010 13:45:40 -0700 (PDT) Received: by 10.229.227.131 with HTTP; Mon, 11 Oct 2010 13:45:39 -0700 (PDT) Date: Mon, 11 Oct 2010 15:45:39 -0500 Message-ID: From: "Sam Fourman Jr." To: FreeBSD Current , Rick Macklem Content-Type: text/plain; charset=ISO-8859-1 Cc: Subject: nfs zfs lockup X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Oct 2010 20:45:41 -0000 I believe NFS is upsetting ZFS v15 on FreeBSD current (kernel sources from today) this happened while trying to sftp a 4gb file here is a back trace # procstat -k 2675 2436 1081 18 5 0 PID TID COMM TDNAME KSTACK 2675 100292 sftp-server - mi_switch sleepq_wait _cv_wait txg_wait_open zfs_freebsd_write VOP_WRITE_APV vn_write dofilewrite kern_writev write syscallenter syscall Xfast_syscall 2436 100337 cvsup - mi_switch sleepq_wait _cv_wait zio_wait dbuf_read dmu_buf_hold zap_lockdir zap_lookup_norm zap_lookup zfs_dirent_lock zfs_dirlook zfs_lookup zfs_freebsd_lookup vfs_cache_lookup VOP_LOOKUP_APV lookup namei kern_statat_vnhook 1081 100299 nfsd nfsd: master mi_switch sleepq_wait _cv_wait zio_wait dbuf_read dbuf_findbp dbuf_hold_impl dbuf_hold dmu_buf_hold zap_lockdir zap_lookup_norm zap_lookup zfs_dirent_lock zfs_dirlook zfs_lookup zfs_freebsd_lookup vfs_cache_lookup VOP_LOOKUP_APV 1081 100318 nfsd nfsd: service mi_switch sleepq_wait __lockmgr_args vop_stdlock VOP_LOCK1_APV _vn_lock zfs_fhtovp nfsvno_fhtovp nfsd_fhtovp nfsrvd_dorpc nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 1081 100319 nfsd nfsd: service mi_switch sleepq_wait __lockmgr_args vop_stdlock VOP_LOCK1_APV _vn_lock zfs_fhtovp nfsvno_fhtovp nfsd_fhtovp nfsrvd_dorpc nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 1081 100320 nfsd nfsd: service mi_switch sleepq_wait __lockmgr_args vop_stdlock VOP_LOCK1_APV _vn_lock zfs_fhtovp nfsvno_fhtovp nfsd_fhtovp nfsrvd_dorpc nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 18 100081 syncer - mi_switch sleepq_wait _cv_wait zio_wait zil_commit zfs_sync sync_fsync sync_vnode sched_sync fork_exit fork_trampoline 5 100071 zfskern arc_reclaim_thre mi_switch sleepq_timedwait _cv_timedwait arc_reclaim_thread fork_exit fork_trampoline 5 100072 zfskern l2arc_feed_threa mi_switch sleepq_timedwait _cv_timedwait l2arc_feed_thread fork_exit fork_trampoline 5 100125 zfskern txg_thread_enter mi_switch sleepq_wait _cv_wait txg_thread_wait txg_quiesce_thread fork_exit fork_trampoline 5 100126 zfskern txg_thread_enter mi_switch sleepq_timedwait _cv_timedwait txg_thread_wait txg_sync_thread fork_exit fork_trampoline 5 100182 zfskern txg_thread_enter mi_switch sleepq_wait _cv_wait txg_quiesce_thread fork_exit fork_trampoline 5 100183 zfskern txg_thread_enter mi_switch sleepq_wait _cv_wait zio_wait dsl_pool_sync spa_sync txg_sync_thread fork_exit fork_trampoline 0 100000 kernel swapper mi_switch sleepq_timedwait _sleep scheduler mi_startup btext 0 100016 kernel firmware taskq mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100021 kernel acpi_task_0 mi_switch sleepq_wait msleep_spin taskqueue_thread_loop fork_exit fork_trampoline 0 100022 kernel acpi_task_1 mi_switch sleepq_wait msleep_spin taskqueue_thread_loop fork_exit fork_trampoline 0 100023 kernel acpi_task_2 mi_switch sleepq_wait msleep_spin taskqueue_thread_loop fork_exit fork_trampoline 0 100024 kernel kqueue taskq mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100026 kernel thread taskq mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100059 kernel em0 taskq mi_switch sleepq_wait msleep_spin taskqueue_thread_loop fork_exit fork_trampoline 0 100067 kernel system_taskq_0 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100068 kernel system_taskq_1 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100069 kernel system_taskq_2 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100070 kernel system_taskq_3 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100073 kernel deadlkres mi_switch sleepq_timedwait _sleep deadlkres fork_exit fork_trampoline 0 100084 kernel zio_null_issue mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100085 kernel zio_null_intr mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100086 kernel zio_read_issue_0 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100087 kernel zio_read_issue_1 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100088 kernel zio_read_issue_2 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100089 kernel zio_read_issue_3 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100090 kernel zio_read_issue_4 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100091 kernel zio_read_issue_5 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100092 kernel zio_read_issue_6 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100093 kernel zio_read_issue_7 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100094 kernel zio_read_intr_0 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100095 kernel zio_read_intr_1 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100096 kernel zio_read_intr_2 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100097 kernel zio_write_issue_ mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100098 kernel zio_write_issue_ mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100099 kernel zio_write_issue_ mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100100 kernel zio_write_issue_ mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100101 kernel zio_write_issue_ mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100102 kernel zio_write_issue_ mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100103 kernel zio_write_issue_ mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100104 kernel zio_write_issue_ mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100105 kernel zio_write_intr_0 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100106 kernel zio_write_intr_1 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100107 kernel zio_write_intr_2 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100108 kernel zio_write_intr_3 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100109 kernel zio_write_intr_4 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100110 kernel zio_write_intr_5 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100111 kernel zio_write_intr_6 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100112 kernel zio_write_intr_7 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100113 kernel zio_write_intr_h mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100114 kernel zio_write_intr_h mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100115 kernel zio_write_intr_h mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100116 kernel zio_write_intr_h mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100117 kernel zio_write_intr_h mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100118 kernel zio_free_issue mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100119 kernel zio_free_intr mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100120 kernel zio_claim_issue mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100121 kernel zio_claim_intr mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100122 kernel zio_ioctl_issue mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100123 kernel zio_ioctl_intr mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100124 kernel zfs_vn_rele_task mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100127 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100141 kernel zio_null_issue mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100142 kernel zio_null_intr mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100143 kernel zio_read_issue_0 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100144 kernel zio_read_issue_1 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100145 kernel zio_read_issue_2 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100146 kernel zio_read_issue_3 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100147 kernel zio_read_issue_4 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100148 kernel zio_read_issue_5 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100149 kernel zio_read_issue_6 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100150 kernel zio_read_issue_7 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100151 kernel zio_read_intr_0 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100152 kernel zio_read_intr_1 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100153 kernel zio_read_intr_2 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100154 kernel zio_write_issue_ mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100155 kernel zio_write_issue_ mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100156 kernel zio_write_issue_ mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100157 kernel zio_write_issue_ mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100158 kernel zio_write_issue_ mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100159 kernel zio_write_issue_ mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100160 kernel zio_write_issue_ mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100161 kernel zio_write_issue_ mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100162 kernel zio_write_intr_0 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100163 kernel zio_write_intr_1 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100164 kernel zio_write_intr_2 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100165 kernel zio_write_intr_3 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100166 kernel zio_write_intr_4 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100167 kernel zio_write_intr_5 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100168 kernel zio_write_intr_6 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100169 kernel zio_write_intr_7 mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100170 kernel zio_write_intr_h mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100171 kernel zio_write_intr_h mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100172 kernel zio_write_intr_h mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100173 kernel zio_write_intr_h mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100174 kernel zio_write_intr_h mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100175 kernel zio_free_issue mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100176 kernel zio_free_intr mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100177 kernel zio_claim_issue mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100178 kernel zio_claim_intr mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100179 kernel zio_ioctl_issue mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100180 kernel zio_ioctl_intr mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100181 kernel zfs_vn_rele_task mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100195 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100196 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100197 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100198 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100199 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100200 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100201 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100202 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100203 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100204 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100205 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100206 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100207 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100208 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100209 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100210 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100211 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100212 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100213 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100214 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100215 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100216 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100217 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100218 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100219 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100220 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100221 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100222 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100223 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100224 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100225 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100226 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100227 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100228 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100229 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100230 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100231 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100232 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100233 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100234 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100235 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100236 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100237 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100238 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100239 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100240 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100241 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100242 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100243 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100244 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100245 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100246 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100247 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100248 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100249 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100250 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100251 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100252 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100253 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100254 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100255 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100256 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100257 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100258 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100259 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100260 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100261 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline 0 100262 kernel zil_clean mi_switch sleepq_wait _sleep taskqueue_thread_loop fork_exit fork_trampoline -- Sam Fourman Jr. Fourman Networks http://www.fourmannetworks.com