From owner-freebsd-stable@FreeBSD.ORG Wed Aug 17 12:26:10 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 752571065678; Wed, 17 Aug 2011 12:26:10 +0000 (UTC) (envelope-from prvs=1210f20b9f=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 8D7208FC2A; Wed, 17 Aug 2011 12:26:08 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Wed, 17 Aug 2011 13:14:29 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Wed, 17 Aug 2011 13:14:29 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50014633673.msg; Wed, 17 Aug 2011 13:14:27 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1210f20b9f=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: <581C95046B0948FC82D6F2E86948F87B@multiplay.co.uk> From: "Steven Hartland" To: "Andriy Gapon" References: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk><4E4380C0.7070908@FreeBSD.org> <4E43E272.1060204@FreeBSD.org> <62BF25D0ED914876BEE75E2ADF28DDF7@multiplay.co.uk> <4E440865.1040500@FreeBSD.org> <6F08A8DE780545ADB9FA93B0A8AA4DA1@multiplay.co.uk> <4E441314.6060606@FreeBSD.org> <2C4B0D05C8924F24A73B56EA652FA4B0@multiplay.co.uk> <4E48D967.9060804@FreeBSD.org> <9D034F992B064E8092E5D1D249B3E959@multiplay.co.uk> <4E490DAF.1080009@FreeBSD.org> <796FD5A096DE4558B57338A8FA1E125B@multiplay.co.uk> <4E491D01.1090902@FreeBSD.org> <570C5495A5E242F7946E806CA7AC5D68@multiplay.co.uk> <4E4AD35C.7020504@FreeBSD.org> <6A7238AED44542A880B082A40304D940@multiplay.co.uk> <4E4BA21F.6010805@FreeBSD.org> Date: Wed, 17 Aug 2011 13:15:04 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6109 Cc: freebsd-stable@FreeBSD.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Aug 2011 12:26:10 -0000 ----- Original Message ----- From: "Andriy Gapon" To: "Steven Hartland" Cc: Sent: Wednesday, August 17, 2011 12:12 PM Subject: Re: debugging frequent kernel panics on 8.2-RELEASE > on 16/08/2011 23:43 Steven Hartland said the following: >> >> ----- Original Message ----- From: "Andriy Gapon" >> To: "Steven Hartland" >> Cc: >> Sent: Tuesday, August 16, 2011 9:30 PM >> Subject: Re: debugging frequent kernel panics on 8.2-RELEASE >> >> >>> on 15/08/2011 17:56 Steven Hartland said the following: >>>> (kgdb) x/512a 0xffffff8d8f357210 >>> [snip] >>> >>> Can you please also provide the following for this core? >>> list *vm_map_growstack+93 >>> list *lim_cur+17 >>> list *lim_rlimit+18 >>> >>> Also, it would be interesting to get panic output with DDB option. >> >> Here's the info:- >> >> (kgdb) list *vm_map_growstack+93 >> 0xffffffff80543ffd is in vm_map_growstack (/usr/src/sys/vm/vm_map.c:3305). >> 3300 struct uidinfo *uip; >> 3301 >> 3302 Retry: >> 3303 PROC_LOCK(p); >> 3304 stacklim = lim_cur(p, RLIMIT_STACK); >> 3305 vmemlim = lim_cur(p, RLIMIT_VMEM); >> 3306 PROC_UNLOCK(p); >> 3307 >> 3308 vm_map_lock_read(map); >> 3309 >> (kgdb) list *lim_cur+17 >> 0xffffffff80384681 is in lim_cur (/usr/src/sys/kern/kern_resource.c:1150). >> 1145 rlim_t >> 1146 lim_cur(struct proc *p, int which) >> 1147 { >> 1148 struct rlimit rl; >> 1149 >> 1150 lim_rlimit(p, which, &rl); >> 1151 return (rl.rlim_cur); >> 1152 } >> 1153 >> 1154 /* >> (kgdb) list *lim_rlimit+18 >> 0xffffffff80384632 is in lim_rlimit (/usr/src/sys/kern/kern_resource.c:1165). >> 1160 { >> 1161 >> 1162 PROC_LOCK_ASSERT(p, MA_OWNED); >> 1163 KASSERT(which >= 0 && which < RLIM_NLIMITS, >> 1164 ("request for invalid resource limit")); >> 1165 *rlp = p->p_limit->pl_rlimit[which]; >> 1166 if (p->p_sysent->sv_fixlimit != NULL) >> 1167 p->p_sysent->sv_fixlimit(rlp, which); >> 1168 } >> 1169 >> >> I've yet to have the machine with DDB + expanded stack panic. >> >> I plan to leave it a day or so more then try a reboot to see if that >> triggers it. If not I'll drop the stack back down to 4 and see if that >> enables us to get another panic. > > OK, thank you for continuing to debug this! No thank you for the help :) > Another request: could you please execute the following commands in kgdb on the > above core file? > > define allpcpu > set $i = 0 > while ($i <= mp_maxid) > p *cpuid_to_pcpu[$i] > set $i = $i + 1 > end > end > allpcpu Here's the output. $1 = {pc_curthread = 0xffffff0012d708c0, pc_idlethread = 0xffffff0012d838c0, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb = 0xffffff8000149d00, pc_switchtime = 564139965450231, pc_switchticks = 247796551, pc_cpuid = 0, pc_cpumask = 1, pc_other_cpus = 16777214, pc_allcpu = {sle_next = 0x0}, pc_spinlocks = 0x0, pc_cnt = {v_swtch = 1246344506, v_trap = 121031682, v_syscall = 2590785278, v_intr = 866415, v_soft = 174249227, v_vm_faults = 24640099, v_cow_faults = 2606934, v_cow_optim = 678, v_zfod = 19177479, v_ozfod = 0, v_swapin = 0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 24007, v_vnodeout = 41, v_vnodepgsin = 24007, v_vnodepgsout = 322, v_intrans = 7300, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree = 0, v_tfree = 25056637, v_page_size = 0, v_page_count = 0, v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 35906, v_vforks = 21218, v_rforks = 0, v_kthreads = 20, v_forkpages = 9357854, v_vforkpages = 4445028, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {9035196, 1438, 426481, 1091491, 22402335}, pc_device = 0xffffff0012da2700, pc_netisr = 0xffffff0012cfe500, pc_rm_queue = {rmq_next = 0xffffffff808af550, rmq_prev = 0xffffffff808af550}, pc_dynamic = 3737856, pc_monitorbuf = '\0' , pc_prvspace = 0xffffffff808af400, pc_curpmap = 0xffffff0012d74ef8, pc_tssp = 0xffffffff808ae700, pc_commontssp = 0xffffffff808ae700, pc_rsp0 = -549754462976, pc_scratch_rsp = 140737488348968, pc_apic_id = 0, pc_acpi_id = 1, pc_fs32p = 0xffffffff808ad530, pc_gs32p = 0xffffffff808ad538, pc_ldt = 0xffffffff808ad578, pc_tss = 0xffffffff808ad568, pc_cmci_mask = 364} $2 = {pc_curthread = 0xffffff0012d85000, pc_idlethread = 0xffffff0012d85000, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb = 0xffffff80001bcd00, pc_switchtime = 564139964769035, pc_switchticks = 247796551, pc_cpuid = 1, pc_cpumask = 2, pc_other_cpus = 16777213, pc_allcpu = {sle_next = 0xffffffff808af400}, pc_spinlocks = 0x0, pc_cnt = {v_swtch = 457697994, v_trap = 61700571, v_syscall = 670428238, v_intr = 298981, v_soft = 58852682, v_vm_faults = 7228810, v_cow_faults = 442573, v_cow_optim = 116, v_zfod = 6082240, v_ozfod = 0, v_swapin = 0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 5151, v_vnodeout = 50, v_vnodepgsin = 5151, v_vnodepgsout = 397, v_intrans = 5575, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree = 0, v_tfree = 8282005, v_page_size = 0, v_page_count = 0, v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 10015, v_vforks = 11459, v_rforks = 0, v_kthreads = 0, v_forkpages = 2626771, v_vforkpages = 2444076, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {8641747, 395, 532547, 157762, 23624411}, pc_device = 0xffffff0012da2600, pc_netisr = 0x0, pc_rm_queue = {rmq_next = 0xffffffff808af7d0, rmq_prev = 0xffffffff808af7d0}, pc_dynamic = 18446743526093297920, pc_monitorbuf = '\0' , pc_prvspace = 0xffffffff808af680, pc_curpmap = 0xffffffff8083ea50, pc_tssp = 0xffffffff808ae768, pc_commontssp = 0xffffffff808ae768, pc_rsp0 = -549753991936, pc_scratch_rsp = 140737488339432, pc_apic_id = 1, pc_acpi_id = 13, pc_fs32p = 0xffffffff808ad598, pc_gs32p = 0xffffffff808ad5a0, pc_ldt = 0xffffffff808ad5e0, pc_tss = 0xffffffff808ad5d0, pc_cmci_mask = 0} $3 = {pc_curthread = 0xffffff06b7f9c000, pc_idlethread = 0xffffff0012d85460, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb = 0xffffff8d8f35ad00, pc_switchtime = 564139963042291, pc_switchticks = 247796550, pc_cpuid = 2, pc_cpumask = 4, pc_other_cpus = 16777211, pc_allcpu = {sle_next = 0xffffffff808af680}, pc_spinlocks = 0x0, pc_cnt = {v_swtch = 1005391948, v_trap = 95927887, v_syscall = 2033274537, v_intr = 137253, v_soft = 151981308, v_vm_faults = 14199910, v_cow_faults = 1468132, v_cow_optim = 533, v_zfod = 11032593, v_ozfod = 0, v_swapin = 0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 17238, v_vnodeout = 48, v_vnodepgsin = 17238, v_vnodepgsout = 378, v_intrans = 6753, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree = 0, v_tfree = 15435380, v_page_size = 0, v_page_count = 0, v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 24041, v_vforks = 16857, v_rforks = 0, v_kthreads = 0, v_forkpages = 6281292, v_vforkpages = 3606842, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {8629094, 693, 594838, 24425, 23707811}, pc_device = 0xffffff0012da2500, pc_netisr = 0x0, pc_rm_queue = {rmq_next = 0xffffffff808afa50, rmq_prev = 0xffffffff808afa50}, pc_dynamic = 18446743526093326592, pc_monitorbuf = '\0' , pc_prvspace = 0xffffffff808af900, pc_curpmap = 0xffffffff8083ea50, pc_tssp = 0xffffffff808ae7d0, pc_commontssp = 0xffffffff808ae7d0, pc_rsp0 = -491518579456, pc_scratch_rsp = 140737488347240, pc_apic_id = 2, pc_acpi_id = 2, pc_fs32p = 0xffffffff808ad600, pc_gs32p = 0xffffffff808ad608, pc_ldt = 0xffffffff808ad648, pc_tss = 0xffffffff808ad638, pc_cmci_mask = 8} $4 = {pc_curthread = 0xffffff0012d858c0, pc_idlethread = 0xffffff0012d858c0, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb = 0xffffff80001b2d00, pc_switchtime = 564139960864579, pc_switchticks = 247796549, pc_cpuid = 3, pc_cpumask = 8, pc_other_cpus = 16777207, pc_allcpu = {sle_next = 0xffffffff808af900}, pc_spinlocks = 0x0, pc_cnt = {v_swtch = 375825838, v_trap = 57311463, v_syscall = 571437816, v_intr = 126334, v_soft = 46300913, v_vm_faults = 6398769, v_cow_faults = 365115, v_cow_optim = 101, v_zfod = 5434860, v_ozfod = 0, v_swapin = 0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 6044, v_vnodeout = 16, v_vnodepgsin = 6044, v_vnodepgsout = 128, v_intrans = 5456, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree = 0, v_tfree = 7824928, v_page_size = 0, v_page_count = 0, v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 8975, v_vforks = 11796, v_rforks = 0, v_kthreads = 0, v_forkpages = 2359166, v_vforkpages = 2604538, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {8374580, 378, 189751, 113208, 24278945}, pc_device = 0xffffff0012eee600, pc_netisr = 0x0, pc_rm_queue = {rmq_next = 0xffffffff808afcd0, rmq_prev = 0xffffffff808afcd0}, pc_dynamic = 18446743526093355264, pc_monitorbuf = '\0' , pc_prvspace = 0xffffffff808afb80, pc_curpmap = 0xffffffff8083ea50, pc_tssp = 0xffffffff808ae838, pc_commontssp = 0xffffffff808ae838, pc_rsp0 = -549754032896, pc_scratch_rsp = 140737488341768, pc_apic_id = 3, pc_acpi_id = 14, pc_fs32p = 0xffffffff808ad668, pc_gs32p = 0xffffffff808ad670, pc_ldt = 0xffffffff808ad6b0, pc_tss = 0xffffffff808ad6a0, pc_cmci_mask = 36} $5 = {pc_curthread = 0xffffff0016ef7460, pc_idlethread = 0xffffff0012d7e000, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb = 0xffffff8d8d249d00, pc_switchtime = 564139958831726, pc_switchticks = 247796548, pc_cpuid = 4, pc_cpumask = 16, pc_other_cpus = 16777199, pc_allcpu = {sle_next = 0xffffffff808afb80}, pc_spinlocks = 0x0, pc_cnt = {v_swtch = 806444301, v_trap = 81626382, v_syscall = 1826349511, v_intr = 123653, v_soft = 144961951, v_vm_faults = 9705936, v_cow_faults = 966760, v_cow_optim = 329, v_zfod = 7605338, v_ozfod = 0, v_swapin = 0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 7070, v_vnodeout = 38, v_vnodepgsin = 7070, v_vnodepgsout = 298, v_intrans = 6176, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree = 0, v_tfree = 10505534, v_page_size = 0, v_page_count = 0, v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 16806, v_vforks = 12551, v_rforks = 0, v_kthreads = 0, v_forkpages = 4380008, v_vforkpages = 2702450, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {8548739, 486, 66166, 155291, 24186180}, pc_device = 0xffffff0012eee500, pc_netisr = 0x0, pc_rm_queue = {rmq_next = 0xffffffff808aff50, rmq_prev = 0xffffffff808aff50}, pc_dynamic = 18446743526093383936, pc_monitorbuf = '\0' , pc_prvspace = 0xffffffff808afe00, pc_curpmap = 0xffffffff8083ea50, pc_tssp = 0xffffffff808ae8a0, pc_commontssp = 0xffffffff808ae8a0, pc_rsp0 = -491553252096, pc_scratch_rsp = 140737488347240, pc_apic_id = 4, pc_acpi_id = 3, pc_fs32p = 0xffffffff808ad6d0, pc_gs32p = 0xffffffff808ad6d8, pc_ldt = 0xffffffff808ad718, pc_tss = 0xffffffff808ad708, pc_cmci_mask = 44} $6 = {pc_curthread = 0xffffff0016d40460, pc_idlethread = 0xffffff0012d7e460, pc_fpcurthread = 0xffffff0016d40460, pc_deadthread = 0x0, pc_curpcb = 0xffffff8d8d47ed00, pc_switchtime = 564139958865046, pc_switchticks = 247796548, pc_cpuid = 5, pc_cpumask = 32, pc_other_cpus = 16777183, pc_allcpu = {sle_next = 0xffffffff808afe00}, pc_spinlocks = 0x0, pc_cnt = {v_swtch = 328647871, v_trap = 51585678, v_syscall = 541729242, v_intr = 195389, v_soft = 45565082, v_vm_faults = 5629366, v_cow_faults = 317486, v_cow_optim = 82, v_zfod = 4813949, v_ozfod = 0, v_swapin = 0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 4750, v_vnodeout = 16, v_vnodepgsin = 4750, v_vnodepgsout = 125, v_intrans = 4461, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree = 0, v_tfree = 6982766, v_page_size = 0, v_page_count = 0, v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 7103, v_vforks = 10517, v_rforks = 0, v_kthreads = 0, v_forkpages = 1858863, v_vforkpages = 2375468, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {8156742, 242, 143793, 754, 24655331}, pc_device = 0xffffff0012eee400, pc_netisr = 0x0, pc_rm_queue = {rmq_next = 0xffffffff808b01d0, rmq_prev = 0xffffffff808b01d0}, pc_dynamic = 18446743526093412608, pc_monitorbuf = '\0' , pc_prvspace = 0xffffffff808b0080, pc_curpmap = 0xffffff00285acd70, pc_tssp = 0xffffffff808ae908, pc_commontssp = 0xffffffff808ae908, pc_rsp0 = -491550937856, pc_scratch_rsp = 140737488338216, pc_apic_id = 5, pc_acpi_id = 15, pc_fs32p = 0xffffffff808ad738, pc_gs32p = 0xffffffff808ad740, pc_ldt = 0xffffffff808ad780, pc_tss = 0xffffffff808ad770, pc_cmci_mask = 44} $7 = {pc_curthread = 0xffffff0012d7e8c0, pc_idlethread = 0xffffff0012d7e8c0, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb = 0xffffff80001a3d00, pc_switchtime = 564139963916274, pc_switchticks = 247796550, pc_cpuid = 6, pc_cpumask = 64, pc_other_cpus = 16777151, pc_allcpu = {sle_next = 0xffffffff808b0080}, pc_spinlocks = 0x0, pc_cnt = {v_swtch = 571134015, v_trap = 71997786, v_syscall = 1463142320, v_intr = 279742, v_soft = 132911942, v_vm_faults = 7791389, v_cow_faults = 708630, v_cow_optim = 253, v_zfod = 6277796, v_ozfod = 0, v_swapin = 0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 6392, v_vnodeout = 39, v_vnodepgsin = 6392, v_vnodepgsout = 312, v_intrans = 5737, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree = 0, v_tfree = 9292572, v_page_size = 0, v_page_count = 0, v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 11420, v_vforks = 9776, v_rforks = 0, v_kthreads = 0, v_forkpages = 2973042, v_vforkpages = 2103188, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {8387371, 350, 74084, 53153, 24441904}, pc_device = 0xffffff0012eee300, pc_netisr = 0x0, pc_rm_queue = {rmq_next = 0xffffffff808b0450, rmq_prev = 0xffffffff808b0450}, pc_dynamic = 18446743526093441280, pc_monitorbuf = '\0' , pc_prvspace = 0xffffffff808b0300, pc_curpmap = 0xffffffff8083ea50, pc_tssp = 0xffffffff808ae970, pc_commontssp = 0xffffffff808ae970, pc_rsp0 = -549754094336, ---Type to continue, or q to quit--- pc_scratch_rsp = 140737488347240, pc_apic_id = 16, pc_acpi_id = 4, pc_fs32p = 0xffffffff808ad7a0, pc_gs32p = 0xffffffff808ad7a8, pc_ldt = 0xffffffff808ad7e8, pc_tss = 0xffffffff808ad7d8, pc_cmci_mask = 44} $8 = {pc_curthread = 0xffffff0012d7f000, pc_idlethread = 0xffffff0012d7f000, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb = 0xffffff800019ed00, pc_switchtime = 564139961406818, pc_switchticks = 247796549, pc_cpuid = 7, pc_cpumask = 128, pc_other_cpus = 16777087, pc_allcpu = {sle_next = 0xffffffff808b0300}, pc_spinlocks = 0x0, pc_cnt = {v_swtch = 249485946, v_trap = 42612704, v_syscall = 513323841, v_intr = 158985, v_soft = 49793772, v_vm_faults = 4953550, v_cow_faults = 288574, v_cow_optim = 66, v_zfod = 4279446, v_ozfod = 0, v_swapin = 0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 5127, v_vnodeout = 18, v_vnodepgsin = 5127, v_vnodepgsout = 144, v_intrans = 4430, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree = 0, v_tfree = 6781637, v_page_size = 0, v_page_count = 0, v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 7191, v_vforks = 8106, v_rforks = 0, v_kthreads = 0, v_forkpages = 1911571, v_vforkpages = 1791118, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {7834102, 189, 67132, 7190, 25048249}, pc_device = 0xffffff0012eee200, pc_netisr = 0x0, pc_rm_queue = {rmq_next = 0xffffffff808b06d0, rmq_prev = 0xffffffff808b06d0}, pc_dynamic = 18446743526093469952, pc_monitorbuf = '\0' , pc_prvspace = 0xffffffff808b0580, pc_curpmap = 0xffffffff8083ea50, pc_tssp = 0xffffffff808ae9d8, pc_commontssp = 0xffffffff808ae9d8, pc_rsp0 = -549754114816, pc_scratch_rsp = 140737488341768, pc_apic_id = 17, pc_acpi_id = 16, pc_fs32p = 0xffffffff808ad808, pc_gs32p = 0xffffffff808ad810, pc_ldt = 0xffffffff808ad850, pc_tss = 0xffffffff808ad840, pc_cmci_mask = 44} $9 = {pc_curthread = 0xffffff0012d7f460, pc_idlethread = 0xffffff0012d7f460, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb = 0xffffff8000199d00, pc_switchtime = 564139961215887, pc_switchticks = 247796549, pc_cpuid = 8, pc_cpumask = 256, pc_other_cpus = 16776959, pc_allcpu = {sle_next = 0xffffffff808b0580}, pc_spinlocks = 0x0, pc_cnt = {v_swtch = 464956409, v_trap = 64334946, v_syscall = 1027059020, v_intr = 0, v_soft = 93052690, v_vm_faults = 6917455, v_cow_faults = 567595, v_cow_optim = 160, v_zfod = 5697686, v_ozfod = 0, v_swapin = 0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 4817, v_vnodeout = 36, v_vnodepgsin = 4817, v_vnodepgsout = 285, v_intrans = 5954, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree = 0, v_tfree = 7425675, v_page_size = 0, v_page_count = 0, v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 9992, v_vforks = 8433, v_rforks = 0, v_kthreads = 0, v_forkpages = 2600081, v_vforkpages = 1793522, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {7987187, 281, 218255, 10560, 24740579}, pc_device = 0xffffff0012eee100, pc_netisr = 0x0, pc_rm_queue = {rmq_next = 0xffffffff808b0950, rmq_prev = 0xffffffff808b0950}, pc_dynamic = 18446743526093498624, pc_monitorbuf = '\0' , pc_prvspace = 0xffffffff808b0800, pc_curpmap = 0xffffffff8083ea50, pc_tssp = 0xffffffff808aea40, pc_commontssp = 0xffffffff808aea40, pc_rsp0 = -549754135296, pc_scratch_rsp = 140737488349480, pc_apic_id = 18, pc_acpi_id = 5, pc_fs32p = 0xffffffff808ad870, pc_gs32p = 0xffffffff808ad878, pc_ldt = 0xffffffff808ad8b8, pc_tss = 0xffffffff808ad8a8, pc_cmci_mask = 44} $10 = {pc_curthread = 0xffffff0012d7f8c0, pc_idlethread = 0xffffff0012d7f8c0, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb = 0xffffff8000194d00, pc_switchtime = 564139962352563, pc_switchticks = 247796549, pc_cpuid = 9, pc_cpumask = 512, pc_other_cpus = 16776703, pc_allcpu = {sle_next = 0xffffffff808b0800}, pc_spinlocks = 0x0, pc_cnt = {v_swtch = 274982887, v_trap = 31399978, v_syscall = 444601703, v_intr = 3206219, v_soft = 40294508, v_vm_faults = 4841563, v_cow_faults = 271228, v_cow_optim = 62, v_zfod = 4257729, v_ozfod = 0, v_swapin = 0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 3551, v_vnodeout = 18, v_vnodepgsin = 3551, v_vnodepgsout = 144, v_intrans = 4426, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree = 0, v_tfree = 5832072, v_page_size = 0, v_page_count = 0, v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 6130, v_vforks = 6921, v_rforks = 0, v_kthreads = 0, v_forkpages = 1593321, v_vforkpages = 1503680, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {7446553, 129, 81200, 80535, 25348445}, pc_device = 0xffffff0012eee000, pc_netisr = 0x0, pc_rm_queue = {rmq_next = 0xffffffff808b0bd0, rmq_prev = 0xffffffff808b0bd0}, pc_dynamic = 18446743526093527296, pc_monitorbuf = '\0' , pc_prvspace = 0xffffffff808b0a80, pc_curpmap = 0xffffffff8083ea50, pc_tssp = 0xffffffff808aeaa8, pc_commontssp = 0xffffffff808aeaa8, pc_rsp0 = -549754155776, pc_scratch_rsp = 140737488350536, pc_apic_id = 19, pc_acpi_id = 17, pc_fs32p = 0xffffffff808ad8d8, pc_gs32p = 0xffffffff808ad8e0, pc_ldt = 0xffffffff808ad920, pc_tss = 0xffffffff808ad910, pc_cmci_mask = 44} $11 = {pc_curthread = 0xffffff0012d81000, pc_idlethread = 0xffffff0012d81000, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb = 0xffffff800018fd00, pc_switchtime = 564139960703995, pc_switchticks = 247796549, pc_cpuid = 10, pc_cpumask = 1024, pc_other_cpus = 16776191, pc_allcpu = {sle_next = 0xffffffff808b0a80}, pc_spinlocks = 0x0, pc_cnt = {v_swtch = 409239430, v_trap = 63190665, v_syscall = 916205775, v_intr = 0, v_soft = 84751808, v_vm_faults = 5931079, v_cow_faults = 475456, v_cow_optim = 101, v_zfod = 4924752, v_ozfod = 0, v_swapin = 0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 3365, v_vnodeout = 31, v_vnodepgsin = 3365, v_vnodepgsout = 248, v_intrans = 5616, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree = 0, v_tfree = 6799603, v_page_size = 0, v_page_count = 0, v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 7789, v_vforks = 7721, v_rforks = 0, v_kthreads = 0, v_forkpages = 2032846, v_vforkpages = 1672147, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {7664964, 263, 457601, 10909, 24823125}, pc_device = 0xffffff0012e73e00, pc_netisr = 0x0, pc_rm_queue = {rmq_next = 0xffffffff808b0e50, rmq_prev = 0xffffffff808b0e50}, pc_dynamic = 18446743526093555968, pc_monitorbuf = '\0' , pc_prvspace = 0xffffffff808b0d00, pc_curpmap = 0xffffffff8083ea50, pc_tssp = 0xffffffff808aeb10, pc_commontssp = 0xffffffff808aeb10, pc_rsp0 = -549754176256, pc_scratch_rsp = 140737488349208, pc_apic_id = 20, pc_acpi_id = 6, pc_fs32p = 0xffffffff808ad940, pc_gs32p = 0xffffffff808ad948, pc_ldt = 0xffffffff808ad988, pc_tss = 0xffffffff808ad978, pc_cmci_mask = 12} $12 = {pc_curthread = 0xffffff0012d7c000, pc_idlethread = 0xffffff0012d7c000, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb = 0xffffff800018ad00, pc_switchtime = 564139964659179, pc_switchticks = 247796550, pc_cpuid = 11, pc_cpumask = 2048, pc_other_cpus = 16775167, pc_allcpu = {sle_next = 0xffffffff808b0d00}, pc_spinlocks = 0x0, pc_cnt = {v_swtch = 286462305, v_trap = 43327710, v_syscall = 585895149, v_intr = 0, v_soft = 58132961, v_vm_faults = 4529044, v_cow_faults = 253158, v_cow_optim = 51, v_zfod = 3997600, v_ozfod = 0, v_swapin = 0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 2889, v_vnodeout = 14, v_vnodepgsin = 2889, v_vnodepgsout = 112, v_intrans = 4397, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree = 0, v_tfree = 5501089, v_page_size = 0, v_page_count = 0, v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 5059, v_vforks = 7478, v_rforks = 0, v_kthreads = 0, v_forkpages = 1317908, v_vforkpages = 1695236, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {6940693, 83, 34106, 26044, 25955936}, pc_device = 0xffffff0012e73d00, pc_netisr = 0x0, pc_rm_queue = {rmq_next = 0xffffffff808b10d0, rmq_prev = 0xffffffff808b10d0}, pc_dynamic = 18446743526093584640, pc_monitorbuf = '\0' , pc_prvspace = 0xffffffff808b0f80, pc_curpmap = 0xffffffff8083ea50, pc_tssp = 0xffffffff808aeb78, pc_commontssp = 0xffffffff808aeb78, pc_rsp0 = -549754196736, pc_scratch_rsp = 140737488341352, pc_apic_id = 21, pc_acpi_id = 18, pc_fs32p = 0xffffffff808ad9a8, pc_gs32p = 0xffffffff808ad9b0, pc_ldt = 0xffffffff808ad9f0, pc_tss = 0xffffffff808ad9e0, pc_cmci_mask = 32} $13 = {pc_curthread = 0xffffff0012d7c460, pc_idlethread = 0xffffff0012d7c460, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb = 0xffffff8000185d00, pc_switchtime = 564139964135679, pc_switchticks = 247796550, pc_cpuid = 12, pc_cpumask = 4096, pc_other_cpus = 16773119, pc_allcpu = {sle_next = 0xffffffff808b0f80}, pc_spinlocks = 0x0, pc_cnt = {v_swtch = 1452228911, v_trap = 138992214, v_syscall = 3481775635, v_intr = 0, v_soft = 170473572, v_vm_faults = 25297505, v_cow_faults = 2415176, v_cow_optim = 725, v_zfod = 20124416, v_ozfod = 0, v_swapin = 0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 20013, v_vnodeout = 39, v_vnodepgsin = 20013, v_vnodepgsout = 303, v_intrans = 8910, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree = 0, v_tfree = 26563945, v_page_size = 0, v_page_count = 0, v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 32999, v_vforks = 20923, v_rforks = 0, v_kthreads = 0, v_forkpages = 8640201, v_vforkpages = 4383325, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {9099202, 1744, 94283, 697, 23760936}, pc_device = 0xffffff0012e73c00, pc_netisr = 0x0, pc_rm_queue = {rmq_next = 0xffffffff808b1350, rmq_prev = 0xffffffff808b1350}, pc_dynamic = 18446743526093613312, pc_monitorbuf = '\0' , pc_prvspace = 0xffffffff808b1200, pc_curpmap = 0xffffffff8083ea50, pc_tssp = 0xffffffff808aebe0, pc_commontssp = 0xffffffff808aebe0, pc_rsp0 = -549754217216, pc_scratch_rsp = 140737488348728, pc_apic_id = 32, pc_acpi_id = 7, pc_fs32p = 0xffffffff808ada10, pc_gs32p = 0xffffffff808ada18, pc_ldt = 0xffffffff808ada58, pc_tss = 0xffffffff808ada48, pc_cmci_mask = 40} $14 = {pc_curthread = 0xffffff081149b460, pc_idlethread = 0xffffff0012d7c8c0, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb = 0xffffff8d8e5a9d00, pc_switchtime = 564139964136603, pc_switchticks = 247796550, pc_cpuid = 13, pc_cpumask = 8192, pc_other_cpus = 16769023, pc_allcpu = {sle_next = 0xffffffff808b1200}, pc_spinlocks = 0x0, pc_cnt = {v_swtch = 476914812, v_trap = 74744115, v_syscall = 936915699, v_intr = 0, v_soft = 80021803, v_vm_faults = 7523961, v_cow_faults = 390299, v_cow_optim = 133, v_zfod = 6338882, v_ozfod = 0, v_swapin = 0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 3069, v_vnodeout = 31, v_vnodepgsin = 3069, v_vnodepgsout = 242, v_intrans = 7411, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree = 0, v_tfree = 8400243, v_page_size = 0, v_page_count = 0, v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 9846, v_vforks = 12050, v_rforks = 0, v_kthreads = 0, v_forkpages = 2609276, v_vforkpages = 2565142, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {8441142, 622, 122914, 718, 24391466}, pc_device = 0xffffff0012e73b00, pc_netisr = 0x0, pc_rm_queue = {rmq_next = 0xffffffff808b15d0, rmq_prev = 0xffffffff808b15d0}, pc_dynamic = 18446743526093641984, ---Type to continue, or q to quit--- pc_monitorbuf = '\0' , pc_prvspace = 0xffffffff808b1480, pc_curpmap = 0xffffff017af0a440, pc_tssp = 0xffffffff808aec48, pc_commontssp = 0xffffffff808aec48, pc_rsp0 = -491532935936, pc_scratch_rsp = 140737488347000, pc_apic_id = 33, pc_acpi_id = 19, pc_fs32p = 0xffffffff808ada78, pc_gs32p = 0xffffffff808ada80, pc_ldt = 0xffffffff808adac0, pc_tss = 0xffffffff808adab0, pc_cmci_mask = 44} $15 = {pc_curthread = 0xffffff0012d7d000, pc_idlethread = 0xffffff0012d7d000, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb = 0xffffff800017bd00, pc_switchtime = 564139961068041, pc_switchticks = 247796549, pc_cpuid = 14, pc_cpumask = 16384, pc_other_cpus = 16760831, pc_allcpu = {sle_next = 0xffffffff808b1480}, pc_spinlocks = 0x0, pc_cnt = {v_swtch = 1106841263, v_trap = 97803361, v_syscall = 2661991003, v_intr = 0, v_soft = 167213229, v_vm_faults = 11378004, v_cow_faults = 1160991, v_cow_optim = 428, v_zfod = 8894351, v_ozfod = 0, v_swapin = 0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 12399, v_vnodeout = 42, v_vnodepgsin = 12399, v_vnodepgsout = 333, v_intrans = 8773, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree = 0, v_tfree = 11845093, v_page_size = 0, v_page_count = 0, v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 18853, v_vforks = 12152, v_rforks = 0, v_kthreads = 0, v_forkpages = 4923904, v_vforkpages = 2572782, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {8264202, 687, 502611, 825769, 23363593}, pc_device = 0xffffff0012e73a00, pc_netisr = 0x0, pc_rm_queue = {rmq_next = 0xffffffff808b1850, rmq_prev = 0xffffffff808b1850}, pc_dynamic = 18446743526093670656, pc_monitorbuf = '\0' , pc_prvspace = 0xffffffff808b1700, pc_curpmap = 0xffffffff8083ea50, pc_tssp = 0xffffffff808aecb0, pc_commontssp = 0xffffffff808aecb0, pc_rsp0 = -549754258176, pc_scratch_rsp = 140737488332648, pc_apic_id = 34, pc_acpi_id = 8, pc_fs32p = 0xffffffff808adae0, pc_gs32p = 0xffffffff808adae8, pc_ldt = 0xffffffff808adb28, pc_tss = 0xffffffff808adb18, pc_cmci_mask = 296} $16 = {pc_curthread = 0xffffff0012d7d460, pc_idlethread = 0xffffff0012d7d460, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb = 0xffffff8000176d00, pc_switchtime = 564139964473317, pc_switchticks = 247796550, pc_cpuid = 15, pc_cpumask = 32768, pc_other_cpus = 16744447, pc_allcpu = {sle_next = 0xffffffff808b1700}, pc_spinlocks = 0x0, pc_cnt = {v_swtch = 403303982, v_trap = 62431250, v_syscall = 772577344, v_intr = 0, v_soft = 66350085, v_vm_faults = 6382252, v_cow_faults = 350483, v_cow_optim = 113, v_zfod = 5469113, v_ozfod = 0, v_swapin = 0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 4512, v_vnodeout = 25, v_vnodepgsin = 4512, v_vnodepgsout = 190, v_intrans = 7276, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree = 0, v_tfree = 7254252, v_page_size = 0, v_page_count = 0, v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 9383, v_vforks = 9353, v_rforks = 0, v_kthreads = 0, v_forkpages = 2458231, v_vforkpages = 2018955, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {8065725, 424, 321250, 606, 24568857}, pc_device = 0xffffff0012e73900, pc_netisr = 0x0, pc_rm_queue = {rmq_next = 0xffffffff808b1ad0, rmq_prev = 0xffffffff808b1ad0}, pc_dynamic = 18446743526093699328, pc_monitorbuf = '\0' , pc_prvspace = 0xffffffff808b1980, pc_curpmap = 0xffffffff8083ea50, pc_tssp = 0xffffffff808aed18, pc_commontssp = 0xffffffff808aed18, pc_rsp0 = -549754278656, pc_scratch_rsp = 140737488341304, pc_apic_id = 35, pc_acpi_id = 20, pc_fs32p = 0xffffffff808adb48, pc_gs32p = 0xffffffff808adb50, pc_ldt = 0xffffffff808adb90, pc_tss = 0xffffffff808adb80, pc_cmci_mask = 68} $17 = {pc_curthread = 0xffffff0012d7d8c0, pc_idlethread = 0xffffff0012d7d8c0, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb = 0xffffff8000171d00, pc_switchtime = 564139964351635, pc_switchticks = 247796550, pc_cpuid = 16, pc_cpumask = 65536, pc_other_cpus = 16711679, pc_allcpu = {sle_next = 0xffffffff808b1980}, pc_spinlocks = 0x0, pc_cnt = {v_swtch = 877084980, v_trap = 88142770, v_syscall = 2088145984, v_intr = 0, v_soft = 177372820, v_vm_faults = 7362548, v_cow_faults = 753161, v_cow_optim = 299, v_zfod = 5798775, v_ozfod = 0, v_swapin = 0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 3826, v_vnodeout = 21, v_vnodepgsin = 3826, v_vnodepgsout = 168, v_intrans = 8678, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree = 0, v_tfree = 8584916, v_page_size = 0, v_page_count = 0, v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 11478, v_vforks = 9188, v_rforks = 0, v_kthreads = 0, v_forkpages = 2993442, v_vforkpages = 1934018, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {9852924, 500, 2875093, 71501, 20156844}, pc_device = 0xffffff0012e73800, pc_netisr = 0x0, pc_rm_queue = {rmq_next = 0xffffffff808b1d50, rmq_prev = 0xffffffff808b1d50}, pc_dynamic = 18446743526093728000, pc_monitorbuf = '\0' , pc_prvspace = 0xffffffff808b1c00, pc_curpmap = 0xffffffff8083ea50, pc_tssp = 0xffffffff808aed80, pc_commontssp = 0xffffffff808aed80, pc_rsp0 = -549754299136, pc_scratch_rsp = 140737488347240, pc_apic_id = 36, pc_acpi_id = 9, pc_fs32p = 0xffffffff808adbb0, pc_gs32p = 0xffffffff808adbb8, pc_ldt = 0xffffffff808adbf8, pc_tss = 0xffffffff808adbe8, pc_cmci_mask = 8} $18 = {pc_curthread = 0xffffff0016b94000, pc_idlethread = 0xffffff0012d71460, pc_fpcurthread = 0xffffff0016b94000, pc_deadthread = 0x0, pc_curpcb = 0xffffff8d8d389d00, pc_switchtime = 564139958856111, pc_switchticks = 247796548, pc_cpuid = 17, pc_cpumask = 131072, pc_other_cpus = 16646143, pc_allcpu = {sle_next = 0xffffffff808b1c00}, pc_spinlocks = 0x0, pc_cnt = {v_swtch = 381936054, v_trap = 50211651, v_syscall = 515466461, v_intr = 0, v_soft = 45237711, v_vm_faults = 6414094, v_cow_faults = 389134, v_cow_optim = 137, v_zfod = 5507950, v_ozfod = 0, v_swapin = 0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 5530, v_vnodeout = 53, v_vnodepgsin = 5530, v_vnodepgsout = 424, v_intrans = 6725, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree = 0, v_tfree = 7461119, v_page_size = 0, v_page_count = 0, v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 10518, v_vforks = 8957, v_rforks = 0, v_kthreads = 0, v_forkpages = 2736040, v_vforkpages = 1935285, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {7863823, 191, 438619, 69173, 24585056}, pc_device = 0xffffff0012e73700, pc_netisr = 0x0, pc_rm_queue = {rmq_next = 0xffffffff808b1fd0, rmq_prev = 0xffffffff808b1fd0}, pc_dynamic = 18446743526093756672, pc_monitorbuf = '\0' , pc_prvspace = 0xffffffff808b1e80, pc_curpmap = 0xffffffff8083ea50, pc_tssp = 0xffffffff808aede8, pc_commontssp = 0xffffffff808aede8, pc_rsp0 = -491551941376, pc_scratch_rsp = 140737488341208, pc_apic_id = 37, pc_acpi_id = 21, pc_fs32p = 0xffffffff808adc18, pc_gs32p = 0xffffffff808adc20, pc_ldt = 0xffffffff808adc60, pc_tss = 0xffffffff808adc50, pc_cmci_mask = 36} $19 = {pc_curthread = 0xffffff0012d718c0, pc_idlethread = 0xffffff0012d718c0, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb = 0xffffff8000167d00, pc_switchtime = 564139961112754, pc_switchticks = 247796549, pc_cpuid = 18, pc_cpumask = 262144, pc_other_cpus = 16515071, pc_allcpu = {sle_next = 0xffffffff808b1e80}, pc_spinlocks = 0x0, pc_cnt = {v_swtch = 766728401, v_trap = 80727218, v_syscall = 2226067386, v_intr = 0, v_soft = 164836456, v_vm_faults = 7049688, v_cow_faults = 590626, v_cow_optim = 179, v_zfod = 5751647, v_ozfod = 0, v_swapin = 0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 5189, v_vnodeout = 18, v_vnodepgsin = 5189, v_vnodepgsout = 141, v_intrans = 8491, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree = 0, v_tfree = 8391955, v_page_size = 0, v_page_count = 0, v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 9985, v_vforks = 8855, v_rforks = 0, v_kthreads = 0, v_forkpages = 2607970, v_vforkpages = 1888180, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {8073713, 395, 119748, 2812, 24760194}, pc_device = 0xffffff0012e73600, pc_netisr = 0x0, pc_rm_queue = {rmq_next = 0xffffffff808b2250, rmq_prev = 0xffffffff808b2250}, pc_dynamic = 18446743526093785344, pc_monitorbuf = '\0' , pc_prvspace = 0xffffffff808b2100, pc_curpmap = 0xffffffff8083ea50, pc_tssp = 0xffffffff808aee50, pc_commontssp = 0xffffffff808aee50, pc_rsp0 = -549754340096, pc_scratch_rsp = 140737488340200, pc_apic_id = 48, pc_acpi_id = 10, pc_fs32p = 0xffffffff808adc80, pc_gs32p = 0xffffffff808adc88, pc_ldt = 0xffffffff808adcc8, pc_tss = 0xffffffff808adcb8, pc_cmci_mask = 44} $20 = {pc_curthread = 0xffffff0016b95000, pc_idlethread = 0xffffff0012d7b000, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb = 0xffffff8d8d37ad00, pc_switchtime = 564139964352202, pc_switchticks = 247796550, pc_cpuid = 19, pc_cpumask = 524288, pc_other_cpus = 16252927, pc_allcpu = {sle_next = 0xffffffff808b2100}, pc_spinlocks = 0x0, pc_cnt = {v_swtch = 273765031, v_trap = 40409221, v_syscall = 606284542, v_intr = 0, v_soft = 60032824, v_vm_faults = 4751488, v_cow_faults = 263767, v_cow_optim = 85, v_zfod = 4135690, v_ozfod = 0, v_swapin = 0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 4928, v_vnodeout = 17, v_vnodepgsin = 4928, v_vnodepgsout = 130, v_intrans = 6643, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree = 0, v_tfree = 6264634, v_page_size = 0, v_page_count = 0, v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 7126, v_vforks = 6926, v_rforks = 0, v_kthreads = 0, v_forkpages = 1865103, v_vforkpages = 1510438, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {7556063, 190, 97828, 19877, 25282904}, pc_device = 0xffffff0012eef700, pc_netisr = 0x0, pc_rm_queue = {rmq_next = 0xffffffff808b24d0, rmq_prev = 0xffffffff808b24d0}, pc_dynamic = 18446743526093814016, pc_monitorbuf = '\0' , pc_prvspace = 0xffffffff808b2380, pc_curpmap = 0xffffffff8083ea50, pc_tssp = 0xffffffff808aeeb8, pc_commontssp = 0xffffffff808aeeb8, pc_rsp0 = -491552002816, pc_scratch_rsp = 140737488347448, pc_apic_id = 49, pc_acpi_id = 22, pc_fs32p = 0xffffffff808adce8, pc_gs32p = 0xffffffff808adcf0, pc_ldt = 0xffffffff808add30, pc_tss = 0xffffffff808add20, pc_cmci_mask = 44} $21 = {pc_curthread = 0xffffff0012d7b460, pc_idlethread = 0xffffff0012d7b460, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb = 0xffffff800015dd00, pc_switchtime = 564139966988296, pc_switchticks = 247796551, pc_cpuid = 20, pc_cpumask = 1048576, pc_other_cpus = 15728639, pc_allcpu = {sle_next = 0xffffffff808b2380}, pc_spinlocks = 0x0, pc_cnt = {v_swtch = 650673237, v_trap = 75330345, v_syscall = 1869508277, v_intr = 0, v_soft = 159703865, v_vm_faults = 6100783, v_cow_faults = 476018, v_cow_optim = 144, v_zfod = 5037626, v_ozfod = 0, v_swapin = 0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 2676, v_vnodeout = 30, v_vnodepgsin = 2676, v_vnodepgsout = 240, v_intrans = 8171, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree = 0, v_tfree = 7518937, v_page_size = 0, v_page_count = 0, v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, ---Type to continue, or q to quit--- v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 7717, v_vforks = 7894, v_rforks = 0, v_kthreads = 0, v_forkpages = 2012738, v_vforkpages = 1684979, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {7882266, 330, 49708, 171, 25024387}, pc_device = 0xffffff0012eef600, pc_netisr = 0x0, pc_rm_queue = {rmq_next = 0xffffffff808b2750, rmq_prev = 0xffffffff808b2750}, pc_dynamic = 18446743526093842688, pc_monitorbuf = '\0' , pc_prvspace = 0xffffffff808b2600, pc_curpmap = 0xffffffff8083ea50, pc_tssp = 0xffffffff808aef20, pc_commontssp = 0xffffffff808aef20, pc_rsp0 = -549754381056, pc_scratch_rsp = 140737488347000, pc_apic_id = 50, pc_acpi_id = 11, pc_fs32p = 0xffffffff808add50, pc_gs32p = 0xffffffff808add58, pc_ldt = 0xffffffff808add98, pc_tss = 0xffffffff808add88, pc_cmci_mask = 44} $22 = {pc_curthread = 0xffffff0012d7b8c0, pc_idlethread = 0xffffff0012d7b8c0, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb = 0xffffff8000158d00, pc_switchtime = 564139962266028, pc_switchticks = 247796549, pc_cpuid = 21, pc_cpumask = 2097152, pc_other_cpus = 14680063, pc_allcpu = {sle_next = 0xffffffff808b2600}, pc_spinlocks = 0x0, pc_cnt = {v_swtch = 241921442, v_trap = 27867529, v_syscall = 396426755, v_intr = 0, v_soft = 36203414, v_vm_faults = 4579069, v_cow_faults = 243971, v_cow_optim = 80, v_zfod = 4045607, v_ozfod = 0, v_swapin = 0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 2750, v_vnodeout = 13, v_vnodepgsin = 2750, v_vnodepgsout = 104, v_intrans = 6546, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree = 0, v_tfree = 6134243, v_page_size = 0, v_page_count = 0, v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 5441, v_vforks = 7475, v_rforks = 0, v_kthreads = 0, v_forkpages = 1420342, v_vforkpages = 1634961, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {7064507, 140, 130489, 35805, 25725921}, pc_device = 0xffffff0012eef500, pc_netisr = 0x0, pc_rm_queue = {rmq_next = 0xffffffff808b29d0, rmq_prev = 0xffffffff808b29d0}, pc_dynamic = 18446743526093871360, pc_monitorbuf = '\0' , pc_prvspace = 0xffffffff808b2880, pc_curpmap = 0xffffffff8083ea50, pc_tssp = 0xffffffff808aef88, pc_commontssp = 0xffffffff808aef88, pc_rsp0 = -549754401536, pc_scratch_rsp = 140737488349080, pc_apic_id = 51, pc_acpi_id = 23, pc_fs32p = 0xffffffff808addb8, pc_gs32p = 0xffffffff808addc0, pc_ldt = 0xffffffff808ade00, pc_tss = 0xffffffff808addf0, pc_cmci_mask = 44} $23 = {pc_curthread = 0xffffff0012d70000, pc_idlethread = 0xffffff0012d70000, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb = 0xffffff8000153d00, pc_switchtime = 564139963254342, pc_switchticks = 247796550, pc_cpuid = 22, pc_cpumask = 4194304, pc_other_cpus = 12582911, pc_allcpu = {sle_next = 0xffffffff808b2880}, pc_spinlocks = 0x0, pc_cnt = {v_swtch = 561077805, v_trap = 70503560, v_syscall = 1547897389, v_intr = 0, v_soft = 145263466, v_vm_faults = 5402089, v_cow_faults = 373167, v_cow_optim = 88, v_zfod = 4593818, v_ozfod = 0, v_swapin = 0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 3032, v_vnodeout = 16, v_vnodepgsin = 3032, v_vnodepgsout = 128, v_intrans = 7615, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree = 0, v_tfree = 6367193, v_page_size = 0, v_page_count = 0, v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 5531, v_vforks = 6954, v_rforks = 0, v_kthreads = 0, v_forkpages = 1440742, v_vforkpages = 1477372, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {7664569, 311, 54855, 173, 25236954}, pc_device = 0xffffff0012eef400, pc_netisr = 0x0, pc_rm_queue = {rmq_next = 0xffffffff808b2c50, rmq_prev = 0xffffffff808b2c50}, pc_dynamic = 18446743526093900032, pc_monitorbuf = '\0' , pc_prvspace = 0xffffffff808b2b00, pc_curpmap = 0xffffffff8083ea50, pc_tssp = 0xffffffff808aeff0, pc_commontssp = 0xffffffff808aeff0, pc_rsp0 = -549754422016, pc_scratch_rsp = 140737488347256, pc_apic_id = 52, pc_acpi_id = 12, pc_fs32p = 0xffffffff808ade20, pc_gs32p = 0xffffffff808ade28, pc_ldt = 0xffffffff808ade68, pc_tss = 0xffffffff808ade58, pc_cmci_mask = 44} $24 = {pc_curthread = 0xffffff0012d70460, pc_idlethread = 0xffffff0012d70460, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb = 0xffffff800014ed00, pc_switchtime = 564139962421094, pc_switchticks = 247796549, pc_cpuid = 23, pc_cpumask = 8388608, pc_other_cpus = 8388607, pc_allcpu = {sle_next = 0xffffffff808b2b00}, pc_spinlocks = 0x0, pc_cnt = {v_swtch = 206202024, v_trap = 21901993, v_syscall = 456089216, v_intr = 0, v_soft = 44500078, v_vm_faults = 4394323, v_cow_faults = 229085, v_cow_optim = 71, v_zfod = 3915765, v_ozfod = 0, v_swapin = 0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 2733, v_vnodeout = 5, v_vnodepgsin = 2733, v_vnodepgsout = 40, v_intrans = 6360, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree = 0, v_tfree = 5866068, v_page_size = 0, v_page_count = 0, v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 4299, v_vforks = 5814, v_rforks = 0, v_kthreads = 0, v_forkpages = 1116482, v_vforkpages = 1250752, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {7132190, 176, 32115, 131, 25792250}, pc_device = 0xffffff0012eef300, pc_netisr = 0x0, pc_rm_queue = {rmq_next = 0xffffffff808b2ed0, rmq_prev = 0xffffffff808b2ed0}, pc_dynamic = 18446743526093928704, pc_monitorbuf = '\0' , pc_prvspace = 0xffffffff808b2d80, pc_curpmap = 0xffffffff8083ea50, pc_tssp = 0xffffffff808af058, pc_commontssp = 0xffffffff808af058, pc_rsp0 = -549754442496, pc_scratch_rsp = 140737488347256, pc_apic_id = 53, pc_acpi_id = 24, pc_fs32p = 0xffffffff808ade88, pc_gs32p = 0xffffffff808ade90, pc_ldt = 0xffffffff808aded0, pc_tss = 0xffffffff808adec0, pc_cmci_mask = 44} > A little bit later I will send you another patch that, I hope, will produce better > diagnostics for this crash (without DDB in kernel). Kernel with the patch is now installed on the test machine. I've taken DDB, INVARIANTS and STACK changes out for now. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk.