Date: Wed, 17 Aug 2011 13:15:04 +0100 From: "Steven Hartland" <killing@multiplay.co.uk> To: "Andriy Gapon" <avg@FreeBSD.org> Cc: freebsd-stable@FreeBSD.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE Message-ID: <581C95046B0948FC82D6F2E86948F87B@multiplay.co.uk> References: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk><A71C3ACF01EC4D36871E49805C1A5321@multiplay.co.uk><4E4380C0.7070908@FreeBSD.org><EBC06A239BAB4B3293C28D793329F9CA@multiplay.co.uk> <4E43E272.1060204@FreeBSD.org> <62BF25D0ED914876BEE75E2ADF28DDF7@multiplay.co.uk> <4E440865.1040500@FreeBSD.org> <6F08A8DE780545ADB9FA93B0A8AA4DA1@multiplay.co.uk> <4E441314.6060606@FreeBSD.org> <2C4B0D05C8924F24A73B56EA652FA4B0@multiplay.co.uk> <4E48D967.9060804@FreeBSD.org> <9D034F992B064E8092E5D1D249B3E959@multiplay.co.uk> <4E490DAF.1080009@FreeBSD.org> <796FD5A096DE4558B57338A8FA1E125B@multiplay.co.uk> <4E491D01.1090902@FreeBSD.org> <570C5495A5E242F7946E806CA7AC5D68@multiplay.co.uk> <4E4AD35C.7020504@FreeBSD.org> <6A7238AED44542A880B082A40304D940@multiplay.co.uk> <4E4BA21F.6010805@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
----- Original Message -----
From: "Andriy Gapon" <avg@FreeBSD.org>
To: "Steven Hartland" <killing@multiplay.co.uk>
Cc: <freebsd-stable@FreeBSD.org>
Sent: Wednesday, August 17, 2011 12:12 PM
Subject: Re: debugging frequent kernel panics on 8.2-RELEASE
> on 16/08/2011 23:43 Steven Hartland said the following:
>>
>> ----- Original Message ----- From: "Andriy Gapon" <avg@FreeBSD.org>
>> To: "Steven Hartland" <killing@multiplay.co.uk>
>> Cc: <freebsd-stable@FreeBSD.org>
>> Sent: Tuesday, August 16, 2011 9:30 PM
>> Subject: Re: debugging frequent kernel panics on 8.2-RELEASE
>>
>>
>>> on 15/08/2011 17:56 Steven Hartland said the following:
>>>> (kgdb) x/512a 0xffffff8d8f357210
>>> [snip]
>>>
>>> Can you please also provide the following for this core?
>>> list *vm_map_growstack+93
>>> list *lim_cur+17
>>> list *lim_rlimit+18
>>>
>>> Also, it would be interesting to get panic output with DDB option.
>>
>> Here's the info:-
>>
>> (kgdb) list *vm_map_growstack+93
>> 0xffffffff80543ffd is in vm_map_growstack (/usr/src/sys/vm/vm_map.c:3305).
>> 3300 struct uidinfo *uip;
>> 3301
>> 3302 Retry:
>> 3303 PROC_LOCK(p);
>> 3304 stacklim = lim_cur(p, RLIMIT_STACK);
>> 3305 vmemlim = lim_cur(p, RLIMIT_VMEM);
>> 3306 PROC_UNLOCK(p);
>> 3307
>> 3308 vm_map_lock_read(map);
>> 3309
>> (kgdb) list *lim_cur+17
>> 0xffffffff80384681 is in lim_cur (/usr/src/sys/kern/kern_resource.c:1150).
>> 1145 rlim_t
>> 1146 lim_cur(struct proc *p, int which)
>> 1147 {
>> 1148 struct rlimit rl;
>> 1149
>> 1150 lim_rlimit(p, which, &rl);
>> 1151 return (rl.rlim_cur);
>> 1152 }
>> 1153
>> 1154 /*
>> (kgdb) list *lim_rlimit+18
>> 0xffffffff80384632 is in lim_rlimit (/usr/src/sys/kern/kern_resource.c:1165).
>> 1160 {
>> 1161
>> 1162 PROC_LOCK_ASSERT(p, MA_OWNED);
>> 1163 KASSERT(which >= 0 && which < RLIM_NLIMITS,
>> 1164 ("request for invalid resource limit"));
>> 1165 *rlp = p->p_limit->pl_rlimit[which];
>> 1166 if (p->p_sysent->sv_fixlimit != NULL)
>> 1167 p->p_sysent->sv_fixlimit(rlp, which);
>> 1168 }
>> 1169
>>
>> I've yet to have the machine with DDB + expanded stack panic.
>>
>> I plan to leave it a day or so more then try a reboot to see if that
>> triggers it. If not I'll drop the stack back down to 4 and see if that
>> enables us to get another panic.
>
> OK, thank you for continuing to debug this!
No thank you for the help :)
> Another request: could you please execute the following commands in kgdb on the
> above core file?
>
> define allpcpu
> set $i = 0
> while ($i <= mp_maxid)
> p *cpuid_to_pcpu[$i]
> set $i = $i + 1
> end
> end
> allpcpu
Here's the output.
$1 = {pc_curthread = 0xffffff0012d708c0, pc_idlethread = 0xffffff0012d838c0, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb
= 0xffffff8000149d00, pc_switchtime = 564139965450231, pc_switchticks = 247796551, pc_cpuid = 0,
pc_cpumask = 1, pc_other_cpus = 16777214, pc_allcpu = {sle_next = 0x0}, pc_spinlocks = 0x0, pc_cnt = {v_swtch = 1246344506,
v_trap = 121031682, v_syscall = 2590785278, v_intr = 866415, v_soft = 174249227,
v_vm_faults = 24640099, v_cow_faults = 2606934, v_cow_optim = 678, v_zfod = 19177479, v_ozfod = 0, v_swapin = 0, v_swapout =
0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 24007, v_vnodeout = 41, v_vnodepgsin = 24007,
v_vnodepgsout = 322, v_intrans = 7300, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree
= 0, v_tfree = 25056637, v_page_size = 0, v_page_count = 0, v_free_reserved = 0,
v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0,
v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, v_pageout_free_min = 0,
v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 35906, v_vforks = 21218, v_rforks = 0, v_kthreads = 20, v_forkpages =
9357854, v_vforkpages = 4445028, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {9035196,
1438, 426481, 1091491, 22402335}, pc_device = 0xffffff0012da2700, pc_netisr = 0xffffff0012cfe500, pc_rm_queue = {rmq_next =
0xffffffff808af550, rmq_prev = 0xffffffff808af550}, pc_dynamic = 3737856,
pc_monitorbuf = '\0' <repeats 127 times>, pc_prvspace = 0xffffffff808af400, pc_curpmap = 0xffffff0012d74ef8, pc_tssp =
0xffffffff808ae700, pc_commontssp = 0xffffffff808ae700, pc_rsp0 = -549754462976,
pc_scratch_rsp = 140737488348968, pc_apic_id = 0, pc_acpi_id = 1, pc_fs32p = 0xffffffff808ad530, pc_gs32p = 0xffffffff808ad538,
pc_ldt = 0xffffffff808ad578, pc_tss = 0xffffffff808ad568, pc_cmci_mask = 364}
$2 = {pc_curthread = 0xffffff0012d85000, pc_idlethread = 0xffffff0012d85000, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb
= 0xffffff80001bcd00, pc_switchtime = 564139964769035, pc_switchticks = 247796551, pc_cpuid = 1,
pc_cpumask = 2, pc_other_cpus = 16777213, pc_allcpu = {sle_next = 0xffffffff808af400}, pc_spinlocks = 0x0, pc_cnt = {v_swtch =
457697994, v_trap = 61700571, v_syscall = 670428238, v_intr = 298981, v_soft = 58852682,
v_vm_faults = 7228810, v_cow_faults = 442573, v_cow_optim = 116, v_zfod = 6082240, v_ozfod = 0, v_swapin = 0, v_swapout = 0,
v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 5151, v_vnodeout = 50, v_vnodepgsin = 5151,
v_vnodepgsout = 397, v_intrans = 5575, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree
= 0, v_tfree = 8282005, v_page_size = 0, v_page_count = 0, v_free_reserved = 0,
v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0,
v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, v_pageout_free_min = 0,
v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 10015, v_vforks = 11459, v_rforks = 0, v_kthreads = 0, v_forkpages =
2626771, v_vforkpages = 2444076, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {8641747,
395, 532547, 157762, 23624411}, pc_device = 0xffffff0012da2600, pc_netisr = 0x0, pc_rm_queue = {rmq_next = 0xffffffff808af7d0,
rmq_prev = 0xffffffff808af7d0}, pc_dynamic = 18446743526093297920,
pc_monitorbuf = '\0' <repeats 127 times>, pc_prvspace = 0xffffffff808af680, pc_curpmap = 0xffffffff8083ea50, pc_tssp =
0xffffffff808ae768, pc_commontssp = 0xffffffff808ae768, pc_rsp0 = -549753991936,
pc_scratch_rsp = 140737488339432, pc_apic_id = 1, pc_acpi_id = 13, pc_fs32p = 0xffffffff808ad598, pc_gs32p = 0xffffffff808ad5a0,
pc_ldt = 0xffffffff808ad5e0, pc_tss = 0xffffffff808ad5d0, pc_cmci_mask = 0}
$3 = {pc_curthread = 0xffffff06b7f9c000, pc_idlethread = 0xffffff0012d85460, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb
= 0xffffff8d8f35ad00, pc_switchtime = 564139963042291, pc_switchticks = 247796550, pc_cpuid = 2,
pc_cpumask = 4, pc_other_cpus = 16777211, pc_allcpu = {sle_next = 0xffffffff808af680}, pc_spinlocks = 0x0, pc_cnt = {v_swtch =
1005391948, v_trap = 95927887, v_syscall = 2033274537, v_intr = 137253, v_soft = 151981308,
v_vm_faults = 14199910, v_cow_faults = 1468132, v_cow_optim = 533, v_zfod = 11032593, v_ozfod = 0, v_swapin = 0, v_swapout =
0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 17238, v_vnodeout = 48, v_vnodepgsin = 17238,
v_vnodepgsout = 378, v_intrans = 6753, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree
= 0, v_tfree = 15435380, v_page_size = 0, v_page_count = 0, v_free_reserved = 0,
v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0,
v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, v_pageout_free_min = 0,
v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 24041, v_vforks = 16857, v_rforks = 0, v_kthreads = 0, v_forkpages =
6281292, v_vforkpages = 3606842, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {8629094,
693, 594838, 24425, 23707811}, pc_device = 0xffffff0012da2500, pc_netisr = 0x0, pc_rm_queue = {rmq_next = 0xffffffff808afa50,
rmq_prev = 0xffffffff808afa50}, pc_dynamic = 18446743526093326592,
pc_monitorbuf = '\0' <repeats 127 times>, pc_prvspace = 0xffffffff808af900, pc_curpmap = 0xffffffff8083ea50, pc_tssp =
0xffffffff808ae7d0, pc_commontssp = 0xffffffff808ae7d0, pc_rsp0 = -491518579456,
pc_scratch_rsp = 140737488347240, pc_apic_id = 2, pc_acpi_id = 2, pc_fs32p = 0xffffffff808ad600, pc_gs32p = 0xffffffff808ad608,
pc_ldt = 0xffffffff808ad648, pc_tss = 0xffffffff808ad638, pc_cmci_mask = 8}
$4 = {pc_curthread = 0xffffff0012d858c0, pc_idlethread = 0xffffff0012d858c0, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb
= 0xffffff80001b2d00, pc_switchtime = 564139960864579, pc_switchticks = 247796549, pc_cpuid = 3,
pc_cpumask = 8, pc_other_cpus = 16777207, pc_allcpu = {sle_next = 0xffffffff808af900}, pc_spinlocks = 0x0, pc_cnt = {v_swtch =
375825838, v_trap = 57311463, v_syscall = 571437816, v_intr = 126334, v_soft = 46300913,
v_vm_faults = 6398769, v_cow_faults = 365115, v_cow_optim = 101, v_zfod = 5434860, v_ozfod = 0, v_swapin = 0, v_swapout = 0,
v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 6044, v_vnodeout = 16, v_vnodepgsin = 6044,
v_vnodepgsout = 128, v_intrans = 5456, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree
= 0, v_tfree = 7824928, v_page_size = 0, v_page_count = 0, v_free_reserved = 0,
v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0,
v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, v_pageout_free_min = 0,
v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 8975, v_vforks = 11796, v_rforks = 0, v_kthreads = 0, v_forkpages =
2359166, v_vforkpages = 2604538, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {8374580, 378,
189751, 113208, 24278945}, pc_device = 0xffffff0012eee600, pc_netisr = 0x0, pc_rm_queue = {rmq_next = 0xffffffff808afcd0,
rmq_prev = 0xffffffff808afcd0}, pc_dynamic = 18446743526093355264,
pc_monitorbuf = '\0' <repeats 127 times>, pc_prvspace = 0xffffffff808afb80, pc_curpmap = 0xffffffff8083ea50, pc_tssp =
0xffffffff808ae838, pc_commontssp = 0xffffffff808ae838, pc_rsp0 = -549754032896,
pc_scratch_rsp = 140737488341768, pc_apic_id = 3, pc_acpi_id = 14, pc_fs32p = 0xffffffff808ad668, pc_gs32p = 0xffffffff808ad670,
pc_ldt = 0xffffffff808ad6b0, pc_tss = 0xffffffff808ad6a0, pc_cmci_mask = 36}
$5 = {pc_curthread = 0xffffff0016ef7460, pc_idlethread = 0xffffff0012d7e000, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb
= 0xffffff8d8d249d00, pc_switchtime = 564139958831726, pc_switchticks = 247796548, pc_cpuid = 4,
pc_cpumask = 16, pc_other_cpus = 16777199, pc_allcpu = {sle_next = 0xffffffff808afb80}, pc_spinlocks = 0x0, pc_cnt = {v_swtch =
806444301, v_trap = 81626382, v_syscall = 1826349511, v_intr = 123653, v_soft = 144961951,
v_vm_faults = 9705936, v_cow_faults = 966760, v_cow_optim = 329, v_zfod = 7605338, v_ozfod = 0, v_swapin = 0, v_swapout = 0,
v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 7070, v_vnodeout = 38, v_vnodepgsin = 7070,
v_vnodepgsout = 298, v_intrans = 6176, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree
= 0, v_tfree = 10505534, v_page_size = 0, v_page_count = 0, v_free_reserved = 0,
v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0,
v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, v_pageout_free_min = 0,
v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 16806, v_vforks = 12551, v_rforks = 0, v_kthreads = 0, v_forkpages =
4380008, v_vforkpages = 2702450, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {8548739,
486, 66166, 155291, 24186180}, pc_device = 0xffffff0012eee500, pc_netisr = 0x0, pc_rm_queue = {rmq_next = 0xffffffff808aff50,
rmq_prev = 0xffffffff808aff50}, pc_dynamic = 18446743526093383936,
pc_monitorbuf = '\0' <repeats 127 times>, pc_prvspace = 0xffffffff808afe00, pc_curpmap = 0xffffffff8083ea50, pc_tssp =
0xffffffff808ae8a0, pc_commontssp = 0xffffffff808ae8a0, pc_rsp0 = -491553252096,
pc_scratch_rsp = 140737488347240, pc_apic_id = 4, pc_acpi_id = 3, pc_fs32p = 0xffffffff808ad6d0, pc_gs32p = 0xffffffff808ad6d8,
pc_ldt = 0xffffffff808ad718, pc_tss = 0xffffffff808ad708, pc_cmci_mask = 44}
$6 = {pc_curthread = 0xffffff0016d40460, pc_idlethread = 0xffffff0012d7e460, pc_fpcurthread = 0xffffff0016d40460, pc_deadthread =
0x0, pc_curpcb = 0xffffff8d8d47ed00, pc_switchtime = 564139958865046,
pc_switchticks = 247796548, pc_cpuid = 5, pc_cpumask = 32, pc_other_cpus = 16777183, pc_allcpu = {sle_next =
0xffffffff808afe00}, pc_spinlocks = 0x0, pc_cnt = {v_swtch = 328647871, v_trap = 51585678, v_syscall = 541729242,
v_intr = 195389, v_soft = 45565082, v_vm_faults = 5629366, v_cow_faults = 317486, v_cow_optim = 82, v_zfod = 4813949, v_ozfod
= 0, v_swapin = 0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 4750,
v_vnodeout = 16, v_vnodepgsin = 4750, v_vnodepgsout = 125, v_intrans = 4461, v_reactivated = 0, v_pdwakeups = 0, v_pdpages =
0, v_tcached = 0, v_dfree = 0, v_pfree = 0, v_tfree = 6982766, v_page_size = 0, v_page_count = 0,
v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0,
v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0,
v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 7103, v_vforks = 10517, v_rforks = 0,
v_kthreads = 0, v_forkpages = 1858863, v_vforkpages = 2375468, v_rforkpages = 0, v_kthreadpages = 0},
pc_cp_time = {8156742, 242, 143793, 754, 24655331}, pc_device = 0xffffff0012eee400, pc_netisr = 0x0, pc_rm_queue = {rmq_next =
0xffffffff808b01d0, rmq_prev = 0xffffffff808b01d0}, pc_dynamic = 18446743526093412608,
pc_monitorbuf = '\0' <repeats 127 times>, pc_prvspace = 0xffffffff808b0080, pc_curpmap = 0xffffff00285acd70, pc_tssp =
0xffffffff808ae908, pc_commontssp = 0xffffffff808ae908, pc_rsp0 = -491550937856,
pc_scratch_rsp = 140737488338216, pc_apic_id = 5, pc_acpi_id = 15, pc_fs32p = 0xffffffff808ad738, pc_gs32p = 0xffffffff808ad740,
pc_ldt = 0xffffffff808ad780, pc_tss = 0xffffffff808ad770, pc_cmci_mask = 44}
$7 = {pc_curthread = 0xffffff0012d7e8c0, pc_idlethread = 0xffffff0012d7e8c0, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb
= 0xffffff80001a3d00, pc_switchtime = 564139963916274, pc_switchticks = 247796550, pc_cpuid = 6,
pc_cpumask = 64, pc_other_cpus = 16777151, pc_allcpu = {sle_next = 0xffffffff808b0080}, pc_spinlocks = 0x0, pc_cnt = {v_swtch =
571134015, v_trap = 71997786, v_syscall = 1463142320, v_intr = 279742, v_soft = 132911942,
v_vm_faults = 7791389, v_cow_faults = 708630, v_cow_optim = 253, v_zfod = 6277796, v_ozfod = 0, v_swapin = 0, v_swapout = 0,
v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 6392, v_vnodeout = 39, v_vnodepgsin = 6392,
v_vnodepgsout = 312, v_intrans = 5737, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree
= 0, v_tfree = 9292572, v_page_size = 0, v_page_count = 0, v_free_reserved = 0,
v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0,
v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, v_pageout_free_min = 0,
v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 11420, v_vforks = 9776, v_rforks = 0, v_kthreads = 0, v_forkpages =
2973042, v_vforkpages = 2103188, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {8387371, 350,
74084, 53153, 24441904}, pc_device = 0xffffff0012eee300, pc_netisr = 0x0, pc_rm_queue = {rmq_next = 0xffffffff808b0450,
rmq_prev = 0xffffffff808b0450}, pc_dynamic = 18446743526093441280,
pc_monitorbuf = '\0' <repeats 127 times>, pc_prvspace = 0xffffffff808b0300, pc_curpmap = 0xffffffff8083ea50, pc_tssp =
0xffffffff808ae970, pc_commontssp = 0xffffffff808ae970, pc_rsp0 = -549754094336,
---Type <return> to continue, or q <return> to quit---
pc_scratch_rsp = 140737488347240, pc_apic_id = 16, pc_acpi_id = 4, pc_fs32p = 0xffffffff808ad7a0, pc_gs32p = 0xffffffff808ad7a8,
pc_ldt = 0xffffffff808ad7e8, pc_tss = 0xffffffff808ad7d8, pc_cmci_mask = 44}
$8 = {pc_curthread = 0xffffff0012d7f000, pc_idlethread = 0xffffff0012d7f000, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb
= 0xffffff800019ed00, pc_switchtime = 564139961406818, pc_switchticks = 247796549, pc_cpuid = 7,
pc_cpumask = 128, pc_other_cpus = 16777087, pc_allcpu = {sle_next = 0xffffffff808b0300}, pc_spinlocks = 0x0, pc_cnt = {v_swtch =
249485946, v_trap = 42612704, v_syscall = 513323841, v_intr = 158985, v_soft = 49793772,
v_vm_faults = 4953550, v_cow_faults = 288574, v_cow_optim = 66, v_zfod = 4279446, v_ozfod = 0, v_swapin = 0, v_swapout = 0,
v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 5127, v_vnodeout = 18, v_vnodepgsin = 5127,
v_vnodepgsout = 144, v_intrans = 4430, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree
= 0, v_tfree = 6781637, v_page_size = 0, v_page_count = 0, v_free_reserved = 0,
v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0,
v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, v_pageout_free_min = 0,
v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 7191, v_vforks = 8106, v_rforks = 0, v_kthreads = 0, v_forkpages =
1911571, v_vforkpages = 1791118, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {7834102, 189,
67132, 7190, 25048249}, pc_device = 0xffffff0012eee200, pc_netisr = 0x0, pc_rm_queue = {rmq_next = 0xffffffff808b06d0,
rmq_prev = 0xffffffff808b06d0}, pc_dynamic = 18446743526093469952,
pc_monitorbuf = '\0' <repeats 127 times>, pc_prvspace = 0xffffffff808b0580, pc_curpmap = 0xffffffff8083ea50, pc_tssp =
0xffffffff808ae9d8, pc_commontssp = 0xffffffff808ae9d8, pc_rsp0 = -549754114816,
pc_scratch_rsp = 140737488341768, pc_apic_id = 17, pc_acpi_id = 16, pc_fs32p = 0xffffffff808ad808, pc_gs32p =
0xffffffff808ad810, pc_ldt = 0xffffffff808ad850, pc_tss = 0xffffffff808ad840, pc_cmci_mask = 44}
$9 = {pc_curthread = 0xffffff0012d7f460, pc_idlethread = 0xffffff0012d7f460, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb
= 0xffffff8000199d00, pc_switchtime = 564139961215887, pc_switchticks = 247796549, pc_cpuid = 8,
pc_cpumask = 256, pc_other_cpus = 16776959, pc_allcpu = {sle_next = 0xffffffff808b0580}, pc_spinlocks = 0x0, pc_cnt = {v_swtch =
464956409, v_trap = 64334946, v_syscall = 1027059020, v_intr = 0, v_soft = 93052690,
v_vm_faults = 6917455, v_cow_faults = 567595, v_cow_optim = 160, v_zfod = 5697686, v_ozfod = 0, v_swapin = 0, v_swapout = 0,
v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 4817, v_vnodeout = 36, v_vnodepgsin = 4817,
v_vnodepgsout = 285, v_intrans = 5954, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree
= 0, v_tfree = 7425675, v_page_size = 0, v_page_count = 0, v_free_reserved = 0,
v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0, v_inactive_target = 0,
v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0, v_pageout_free_min = 0,
v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 9992, v_vforks = 8433, v_rforks = 0, v_kthreads = 0, v_forkpages =
2600081, v_vforkpages = 1793522, v_rforkpages = 0, v_kthreadpages = 0}, pc_cp_time = {7987187, 281,
218255, 10560, 24740579}, pc_device = 0xffffff0012eee100, pc_netisr = 0x0, pc_rm_queue = {rmq_next = 0xffffffff808b0950,
rmq_prev = 0xffffffff808b0950}, pc_dynamic = 18446743526093498624,
pc_monitorbuf = '\0' <repeats 127 times>, pc_prvspace = 0xffffffff808b0800, pc_curpmap = 0xffffffff8083ea50, pc_tssp =
0xffffffff808aea40, pc_commontssp = 0xffffffff808aea40, pc_rsp0 = -549754135296,
pc_scratch_rsp = 140737488349480, pc_apic_id = 18, pc_acpi_id = 5, pc_fs32p = 0xffffffff808ad870, pc_gs32p = 0xffffffff808ad878,
pc_ldt = 0xffffffff808ad8b8, pc_tss = 0xffffffff808ad8a8, pc_cmci_mask = 44}
$10 = {pc_curthread = 0xffffff0012d7f8c0, pc_idlethread = 0xffffff0012d7f8c0, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb
= 0xffffff8000194d00, pc_switchtime = 564139962352563, pc_switchticks = 247796549,
pc_cpuid = 9, pc_cpumask = 512, pc_other_cpus = 16776703, pc_allcpu = {sle_next = 0xffffffff808b0800}, pc_spinlocks = 0x0,
pc_cnt = {v_swtch = 274982887, v_trap = 31399978, v_syscall = 444601703, v_intr = 3206219,
v_soft = 40294508, v_vm_faults = 4841563, v_cow_faults = 271228, v_cow_optim = 62, v_zfod = 4257729, v_ozfod = 0, v_swapin =
0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 3551, v_vnodeout = 18,
v_vnodepgsin = 3551, v_vnodepgsout = 144, v_intrans = 4426, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0,
v_dfree = 0, v_pfree = 0, v_tfree = 5832072, v_page_size = 0, v_page_count = 0,
v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0,
v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0,
v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 6130, v_vforks = 6921, v_rforks = 0, v_kthreads
= 0, v_forkpages = 1593321, v_vforkpages = 1503680, v_rforkpages = 0, v_kthreadpages = 0},
pc_cp_time = {7446553, 129, 81200, 80535, 25348445}, pc_device = 0xffffff0012eee000, pc_netisr = 0x0, pc_rm_queue = {rmq_next =
0xffffffff808b0bd0, rmq_prev = 0xffffffff808b0bd0}, pc_dynamic = 18446743526093527296,
pc_monitorbuf = '\0' <repeats 127 times>, pc_prvspace = 0xffffffff808b0a80, pc_curpmap = 0xffffffff8083ea50, pc_tssp =
0xffffffff808aeaa8, pc_commontssp = 0xffffffff808aeaa8, pc_rsp0 = -549754155776,
pc_scratch_rsp = 140737488350536, pc_apic_id = 19, pc_acpi_id = 17, pc_fs32p = 0xffffffff808ad8d8, pc_gs32p =
0xffffffff808ad8e0, pc_ldt = 0xffffffff808ad920, pc_tss = 0xffffffff808ad910, pc_cmci_mask = 44}
$11 = {pc_curthread = 0xffffff0012d81000, pc_idlethread = 0xffffff0012d81000, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb
= 0xffffff800018fd00, pc_switchtime = 564139960703995, pc_switchticks = 247796549,
pc_cpuid = 10, pc_cpumask = 1024, pc_other_cpus = 16776191, pc_allcpu = {sle_next = 0xffffffff808b0a80}, pc_spinlocks = 0x0,
pc_cnt = {v_swtch = 409239430, v_trap = 63190665, v_syscall = 916205775, v_intr = 0,
v_soft = 84751808, v_vm_faults = 5931079, v_cow_faults = 475456, v_cow_optim = 101, v_zfod = 4924752, v_ozfod = 0, v_swapin =
0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 3365, v_vnodeout = 31,
v_vnodepgsin = 3365, v_vnodepgsout = 248, v_intrans = 5616, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0,
v_dfree = 0, v_pfree = 0, v_tfree = 6799603, v_page_size = 0, v_page_count = 0,
v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0,
v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0,
v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 7789, v_vforks = 7721, v_rforks = 0, v_kthreads
= 0, v_forkpages = 2032846, v_vforkpages = 1672147, v_rforkpages = 0, v_kthreadpages = 0},
pc_cp_time = {7664964, 263, 457601, 10909, 24823125}, pc_device = 0xffffff0012e73e00, pc_netisr = 0x0, pc_rm_queue = {rmq_next =
0xffffffff808b0e50, rmq_prev = 0xffffffff808b0e50}, pc_dynamic = 18446743526093555968,
pc_monitorbuf = '\0' <repeats 127 times>, pc_prvspace = 0xffffffff808b0d00, pc_curpmap = 0xffffffff8083ea50, pc_tssp =
0xffffffff808aeb10, pc_commontssp = 0xffffffff808aeb10, pc_rsp0 = -549754176256,
pc_scratch_rsp = 140737488349208, pc_apic_id = 20, pc_acpi_id = 6, pc_fs32p = 0xffffffff808ad940, pc_gs32p = 0xffffffff808ad948,
pc_ldt = 0xffffffff808ad988, pc_tss = 0xffffffff808ad978, pc_cmci_mask = 12}
$12 = {pc_curthread = 0xffffff0012d7c000, pc_idlethread = 0xffffff0012d7c000, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb
= 0xffffff800018ad00, pc_switchtime = 564139964659179, pc_switchticks = 247796550,
pc_cpuid = 11, pc_cpumask = 2048, pc_other_cpus = 16775167, pc_allcpu = {sle_next = 0xffffffff808b0d00}, pc_spinlocks = 0x0,
pc_cnt = {v_swtch = 286462305, v_trap = 43327710, v_syscall = 585895149, v_intr = 0,
v_soft = 58132961, v_vm_faults = 4529044, v_cow_faults = 253158, v_cow_optim = 51, v_zfod = 3997600, v_ozfod = 0, v_swapin =
0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 2889, v_vnodeout = 14,
v_vnodepgsin = 2889, v_vnodepgsout = 112, v_intrans = 4397, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0,
v_dfree = 0, v_pfree = 0, v_tfree = 5501089, v_page_size = 0, v_page_count = 0,
v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0,
v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0,
v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 5059, v_vforks = 7478, v_rforks = 0, v_kthreads
= 0, v_forkpages = 1317908, v_vforkpages = 1695236, v_rforkpages = 0, v_kthreadpages = 0},
pc_cp_time = {6940693, 83, 34106, 26044, 25955936}, pc_device = 0xffffff0012e73d00, pc_netisr = 0x0, pc_rm_queue = {rmq_next =
0xffffffff808b10d0, rmq_prev = 0xffffffff808b10d0}, pc_dynamic = 18446743526093584640,
pc_monitorbuf = '\0' <repeats 127 times>, pc_prvspace = 0xffffffff808b0f80, pc_curpmap = 0xffffffff8083ea50, pc_tssp =
0xffffffff808aeb78, pc_commontssp = 0xffffffff808aeb78, pc_rsp0 = -549754196736,
pc_scratch_rsp = 140737488341352, pc_apic_id = 21, pc_acpi_id = 18, pc_fs32p = 0xffffffff808ad9a8, pc_gs32p =
0xffffffff808ad9b0, pc_ldt = 0xffffffff808ad9f0, pc_tss = 0xffffffff808ad9e0, pc_cmci_mask = 32}
$13 = {pc_curthread = 0xffffff0012d7c460, pc_idlethread = 0xffffff0012d7c460, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb
= 0xffffff8000185d00, pc_switchtime = 564139964135679, pc_switchticks = 247796550,
pc_cpuid = 12, pc_cpumask = 4096, pc_other_cpus = 16773119, pc_allcpu = {sle_next = 0xffffffff808b0f80}, pc_spinlocks = 0x0,
pc_cnt = {v_swtch = 1452228911, v_trap = 138992214, v_syscall = 3481775635, v_intr = 0,
v_soft = 170473572, v_vm_faults = 25297505, v_cow_faults = 2415176, v_cow_optim = 725, v_zfod = 20124416, v_ozfod = 0,
v_swapin = 0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 20013, v_vnodeout = 39,
v_vnodepgsin = 20013, v_vnodepgsout = 303, v_intrans = 8910, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0,
v_dfree = 0, v_pfree = 0, v_tfree = 26563945, v_page_size = 0, v_page_count = 0,
v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0,
v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0,
v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 32999, v_vforks = 20923, v_rforks = 0,
v_kthreads = 0, v_forkpages = 8640201, v_vforkpages = 4383325, v_rforkpages = 0, v_kthreadpages = 0},
pc_cp_time = {9099202, 1744, 94283, 697, 23760936}, pc_device = 0xffffff0012e73c00, pc_netisr = 0x0, pc_rm_queue = {rmq_next =
0xffffffff808b1350, rmq_prev = 0xffffffff808b1350}, pc_dynamic = 18446743526093613312,
pc_monitorbuf = '\0' <repeats 127 times>, pc_prvspace = 0xffffffff808b1200, pc_curpmap = 0xffffffff8083ea50, pc_tssp =
0xffffffff808aebe0, pc_commontssp = 0xffffffff808aebe0, pc_rsp0 = -549754217216,
pc_scratch_rsp = 140737488348728, pc_apic_id = 32, pc_acpi_id = 7, pc_fs32p = 0xffffffff808ada10, pc_gs32p = 0xffffffff808ada18,
pc_ldt = 0xffffffff808ada58, pc_tss = 0xffffffff808ada48, pc_cmci_mask = 40}
$14 = {pc_curthread = 0xffffff081149b460, pc_idlethread = 0xffffff0012d7c8c0, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb
= 0xffffff8d8e5a9d00, pc_switchtime = 564139964136603, pc_switchticks = 247796550,
pc_cpuid = 13, pc_cpumask = 8192, pc_other_cpus = 16769023, pc_allcpu = {sle_next = 0xffffffff808b1200}, pc_spinlocks = 0x0,
pc_cnt = {v_swtch = 476914812, v_trap = 74744115, v_syscall = 936915699, v_intr = 0,
v_soft = 80021803, v_vm_faults = 7523961, v_cow_faults = 390299, v_cow_optim = 133, v_zfod = 6338882, v_ozfod = 0, v_swapin =
0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 3069, v_vnodeout = 31,
v_vnodepgsin = 3069, v_vnodepgsout = 242, v_intrans = 7411, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0,
v_dfree = 0, v_pfree = 0, v_tfree = 8400243, v_page_size = 0, v_page_count = 0,
v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0,
v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0,
v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 9846, v_vforks = 12050, v_rforks = 0,
v_kthreads = 0, v_forkpages = 2609276, v_vforkpages = 2565142, v_rforkpages = 0, v_kthreadpages = 0},
pc_cp_time = {8441142, 622, 122914, 718, 24391466}, pc_device = 0xffffff0012e73b00, pc_netisr = 0x0, pc_rm_queue = {rmq_next =
0xffffffff808b15d0, rmq_prev = 0xffffffff808b15d0}, pc_dynamic = 18446743526093641984,
---Type <return> to continue, or q <return> to quit---
pc_monitorbuf = '\0' <repeats 127 times>, pc_prvspace = 0xffffffff808b1480, pc_curpmap = 0xffffff017af0a440, pc_tssp =
0xffffffff808aec48, pc_commontssp = 0xffffffff808aec48, pc_rsp0 = -491532935936,
pc_scratch_rsp = 140737488347000, pc_apic_id = 33, pc_acpi_id = 19, pc_fs32p = 0xffffffff808ada78, pc_gs32p =
0xffffffff808ada80, pc_ldt = 0xffffffff808adac0, pc_tss = 0xffffffff808adab0, pc_cmci_mask = 44}
$15 = {pc_curthread = 0xffffff0012d7d000, pc_idlethread = 0xffffff0012d7d000, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb
= 0xffffff800017bd00, pc_switchtime = 564139961068041, pc_switchticks = 247796549,
pc_cpuid = 14, pc_cpumask = 16384, pc_other_cpus = 16760831, pc_allcpu = {sle_next = 0xffffffff808b1480}, pc_spinlocks = 0x0,
pc_cnt = {v_swtch = 1106841263, v_trap = 97803361, v_syscall = 2661991003, v_intr = 0,
v_soft = 167213229, v_vm_faults = 11378004, v_cow_faults = 1160991, v_cow_optim = 428, v_zfod = 8894351, v_ozfod = 0, v_swapin
= 0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 12399, v_vnodeout = 42,
v_vnodepgsin = 12399, v_vnodepgsout = 333, v_intrans = 8773, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0,
v_dfree = 0, v_pfree = 0, v_tfree = 11845093, v_page_size = 0, v_page_count = 0,
v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0,
v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0,
v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 18853, v_vforks = 12152, v_rforks = 0,
v_kthreads = 0, v_forkpages = 4923904, v_vforkpages = 2572782, v_rforkpages = 0, v_kthreadpages = 0},
pc_cp_time = {8264202, 687, 502611, 825769, 23363593}, pc_device = 0xffffff0012e73a00, pc_netisr = 0x0, pc_rm_queue = {rmq_next
= 0xffffffff808b1850, rmq_prev = 0xffffffff808b1850}, pc_dynamic = 18446743526093670656,
pc_monitorbuf = '\0' <repeats 127 times>, pc_prvspace = 0xffffffff808b1700, pc_curpmap = 0xffffffff8083ea50, pc_tssp =
0xffffffff808aecb0, pc_commontssp = 0xffffffff808aecb0, pc_rsp0 = -549754258176,
pc_scratch_rsp = 140737488332648, pc_apic_id = 34, pc_acpi_id = 8, pc_fs32p = 0xffffffff808adae0, pc_gs32p = 0xffffffff808adae8,
pc_ldt = 0xffffffff808adb28, pc_tss = 0xffffffff808adb18, pc_cmci_mask = 296}
$16 = {pc_curthread = 0xffffff0012d7d460, pc_idlethread = 0xffffff0012d7d460, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb
= 0xffffff8000176d00, pc_switchtime = 564139964473317, pc_switchticks = 247796550,
pc_cpuid = 15, pc_cpumask = 32768, pc_other_cpus = 16744447, pc_allcpu = {sle_next = 0xffffffff808b1700}, pc_spinlocks = 0x0,
pc_cnt = {v_swtch = 403303982, v_trap = 62431250, v_syscall = 772577344, v_intr = 0,
v_soft = 66350085, v_vm_faults = 6382252, v_cow_faults = 350483, v_cow_optim = 113, v_zfod = 5469113, v_ozfod = 0, v_swapin =
0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 4512, v_vnodeout = 25,
v_vnodepgsin = 4512, v_vnodepgsout = 190, v_intrans = 7276, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0,
v_dfree = 0, v_pfree = 0, v_tfree = 7254252, v_page_size = 0, v_page_count = 0,
v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0,
v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0,
v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 9383, v_vforks = 9353, v_rforks = 0, v_kthreads
= 0, v_forkpages = 2458231, v_vforkpages = 2018955, v_rforkpages = 0, v_kthreadpages = 0},
pc_cp_time = {8065725, 424, 321250, 606, 24568857}, pc_device = 0xffffff0012e73900, pc_netisr = 0x0, pc_rm_queue = {rmq_next =
0xffffffff808b1ad0, rmq_prev = 0xffffffff808b1ad0}, pc_dynamic = 18446743526093699328,
pc_monitorbuf = '\0' <repeats 127 times>, pc_prvspace = 0xffffffff808b1980, pc_curpmap = 0xffffffff8083ea50, pc_tssp =
0xffffffff808aed18, pc_commontssp = 0xffffffff808aed18, pc_rsp0 = -549754278656,
pc_scratch_rsp = 140737488341304, pc_apic_id = 35, pc_acpi_id = 20, pc_fs32p = 0xffffffff808adb48, pc_gs32p =
0xffffffff808adb50, pc_ldt = 0xffffffff808adb90, pc_tss = 0xffffffff808adb80, pc_cmci_mask = 68}
$17 = {pc_curthread = 0xffffff0012d7d8c0, pc_idlethread = 0xffffff0012d7d8c0, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb
= 0xffffff8000171d00, pc_switchtime = 564139964351635, pc_switchticks = 247796550,
pc_cpuid = 16, pc_cpumask = 65536, pc_other_cpus = 16711679, pc_allcpu = {sle_next = 0xffffffff808b1980}, pc_spinlocks = 0x0,
pc_cnt = {v_swtch = 877084980, v_trap = 88142770, v_syscall = 2088145984, v_intr = 0,
v_soft = 177372820, v_vm_faults = 7362548, v_cow_faults = 753161, v_cow_optim = 299, v_zfod = 5798775, v_ozfod = 0, v_swapin =
0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 3826, v_vnodeout = 21,
v_vnodepgsin = 3826, v_vnodepgsout = 168, v_intrans = 8678, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0,
v_dfree = 0, v_pfree = 0, v_tfree = 8584916, v_page_size = 0, v_page_count = 0,
v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0,
v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0,
v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 11478, v_vforks = 9188, v_rforks = 0,
v_kthreads = 0, v_forkpages = 2993442, v_vforkpages = 1934018, v_rforkpages = 0, v_kthreadpages = 0},
pc_cp_time = {9852924, 500, 2875093, 71501, 20156844}, pc_device = 0xffffff0012e73800, pc_netisr = 0x0, pc_rm_queue = {rmq_next
= 0xffffffff808b1d50, rmq_prev = 0xffffffff808b1d50}, pc_dynamic = 18446743526093728000,
pc_monitorbuf = '\0' <repeats 127 times>, pc_prvspace = 0xffffffff808b1c00, pc_curpmap = 0xffffffff8083ea50, pc_tssp =
0xffffffff808aed80, pc_commontssp = 0xffffffff808aed80, pc_rsp0 = -549754299136,
pc_scratch_rsp = 140737488347240, pc_apic_id = 36, pc_acpi_id = 9, pc_fs32p = 0xffffffff808adbb0, pc_gs32p = 0xffffffff808adbb8,
pc_ldt = 0xffffffff808adbf8, pc_tss = 0xffffffff808adbe8, pc_cmci_mask = 8}
$18 = {pc_curthread = 0xffffff0016b94000, pc_idlethread = 0xffffff0012d71460, pc_fpcurthread = 0xffffff0016b94000, pc_deadthread =
0x0, pc_curpcb = 0xffffff8d8d389d00, pc_switchtime = 564139958856111,
pc_switchticks = 247796548, pc_cpuid = 17, pc_cpumask = 131072, pc_other_cpus = 16646143, pc_allcpu = {sle_next =
0xffffffff808b1c00}, pc_spinlocks = 0x0, pc_cnt = {v_swtch = 381936054, v_trap = 50211651,
v_syscall = 515466461, v_intr = 0, v_soft = 45237711, v_vm_faults = 6414094, v_cow_faults = 389134, v_cow_optim = 137, v_zfod
= 5507950, v_ozfod = 0, v_swapin = 0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0,
v_vnodein = 5530, v_vnodeout = 53, v_vnodepgsin = 5530, v_vnodepgsout = 424, v_intrans = 6725, v_reactivated = 0, v_pdwakeups
= 0, v_pdpages = 0, v_tcached = 0, v_dfree = 0, v_pfree = 0, v_tfree = 7461119, v_page_size = 0,
v_page_count = 0, v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count =
0, v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0,
v_cache_max = 0, v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 10518, v_vforks = 8957,
v_rforks = 0, v_kthreads = 0, v_forkpages = 2736040, v_vforkpages = 1935285, v_rforkpages = 0,
v_kthreadpages = 0}, pc_cp_time = {7863823, 191, 438619, 69173, 24585056}, pc_device = 0xffffff0012e73700, pc_netisr = 0x0,
pc_rm_queue = {rmq_next = 0xffffffff808b1fd0, rmq_prev = 0xffffffff808b1fd0},
pc_dynamic = 18446743526093756672, pc_monitorbuf = '\0' <repeats 127 times>, pc_prvspace = 0xffffffff808b1e80, pc_curpmap =
0xffffffff8083ea50, pc_tssp = 0xffffffff808aede8, pc_commontssp = 0xffffffff808aede8,
pc_rsp0 = -491551941376, pc_scratch_rsp = 140737488341208, pc_apic_id = 37, pc_acpi_id = 21, pc_fs32p = 0xffffffff808adc18,
pc_gs32p = 0xffffffff808adc20, pc_ldt = 0xffffffff808adc60, pc_tss = 0xffffffff808adc50,
pc_cmci_mask = 36}
$19 = {pc_curthread = 0xffffff0012d718c0, pc_idlethread = 0xffffff0012d718c0, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb
= 0xffffff8000167d00, pc_switchtime = 564139961112754, pc_switchticks = 247796549,
pc_cpuid = 18, pc_cpumask = 262144, pc_other_cpus = 16515071, pc_allcpu = {sle_next = 0xffffffff808b1e80}, pc_spinlocks = 0x0,
pc_cnt = {v_swtch = 766728401, v_trap = 80727218, v_syscall = 2226067386, v_intr = 0,
v_soft = 164836456, v_vm_faults = 7049688, v_cow_faults = 590626, v_cow_optim = 179, v_zfod = 5751647, v_ozfod = 0, v_swapin =
0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 5189, v_vnodeout = 18,
v_vnodepgsin = 5189, v_vnodepgsout = 141, v_intrans = 8491, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0,
v_dfree = 0, v_pfree = 0, v_tfree = 8391955, v_page_size = 0, v_page_count = 0,
v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0,
v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0,
v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 9985, v_vforks = 8855, v_rforks = 0, v_kthreads
= 0, v_forkpages = 2607970, v_vforkpages = 1888180, v_rforkpages = 0, v_kthreadpages = 0},
pc_cp_time = {8073713, 395, 119748, 2812, 24760194}, pc_device = 0xffffff0012e73600, pc_netisr = 0x0, pc_rm_queue = {rmq_next =
0xffffffff808b2250, rmq_prev = 0xffffffff808b2250}, pc_dynamic = 18446743526093785344,
pc_monitorbuf = '\0' <repeats 127 times>, pc_prvspace = 0xffffffff808b2100, pc_curpmap = 0xffffffff8083ea50, pc_tssp =
0xffffffff808aee50, pc_commontssp = 0xffffffff808aee50, pc_rsp0 = -549754340096,
pc_scratch_rsp = 140737488340200, pc_apic_id = 48, pc_acpi_id = 10, pc_fs32p = 0xffffffff808adc80, pc_gs32p =
0xffffffff808adc88, pc_ldt = 0xffffffff808adcc8, pc_tss = 0xffffffff808adcb8, pc_cmci_mask = 44}
$20 = {pc_curthread = 0xffffff0016b95000, pc_idlethread = 0xffffff0012d7b000, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb
= 0xffffff8d8d37ad00, pc_switchtime = 564139964352202, pc_switchticks = 247796550,
pc_cpuid = 19, pc_cpumask = 524288, pc_other_cpus = 16252927, pc_allcpu = {sle_next = 0xffffffff808b2100}, pc_spinlocks = 0x0,
pc_cnt = {v_swtch = 273765031, v_trap = 40409221, v_syscall = 606284542, v_intr = 0,
v_soft = 60032824, v_vm_faults = 4751488, v_cow_faults = 263767, v_cow_optim = 85, v_zfod = 4135690, v_ozfod = 0, v_swapin =
0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 4928, v_vnodeout = 17,
v_vnodepgsin = 4928, v_vnodepgsout = 130, v_intrans = 6643, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0,
v_dfree = 0, v_pfree = 0, v_tfree = 6264634, v_page_size = 0, v_page_count = 0,
v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0,
v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0,
v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 7126, v_vforks = 6926, v_rforks = 0, v_kthreads
= 0, v_forkpages = 1865103, v_vforkpages = 1510438, v_rforkpages = 0, v_kthreadpages = 0},
pc_cp_time = {7556063, 190, 97828, 19877, 25282904}, pc_device = 0xffffff0012eef700, pc_netisr = 0x0, pc_rm_queue = {rmq_next =
0xffffffff808b24d0, rmq_prev = 0xffffffff808b24d0}, pc_dynamic = 18446743526093814016,
pc_monitorbuf = '\0' <repeats 127 times>, pc_prvspace = 0xffffffff808b2380, pc_curpmap = 0xffffffff8083ea50, pc_tssp =
0xffffffff808aeeb8, pc_commontssp = 0xffffffff808aeeb8, pc_rsp0 = -491552002816,
pc_scratch_rsp = 140737488347448, pc_apic_id = 49, pc_acpi_id = 22, pc_fs32p = 0xffffffff808adce8, pc_gs32p =
0xffffffff808adcf0, pc_ldt = 0xffffffff808add30, pc_tss = 0xffffffff808add20, pc_cmci_mask = 44}
$21 = {pc_curthread = 0xffffff0012d7b460, pc_idlethread = 0xffffff0012d7b460, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb
= 0xffffff800015dd00, pc_switchtime = 564139966988296, pc_switchticks = 247796551,
pc_cpuid = 20, pc_cpumask = 1048576, pc_other_cpus = 15728639, pc_allcpu = {sle_next = 0xffffffff808b2380}, pc_spinlocks = 0x0,
pc_cnt = {v_swtch = 650673237, v_trap = 75330345, v_syscall = 1869508277, v_intr = 0,
v_soft = 159703865, v_vm_faults = 6100783, v_cow_faults = 476018, v_cow_optim = 144, v_zfod = 5037626, v_ozfod = 0, v_swapin =
0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 2676, v_vnodeout = 30,
v_vnodepgsin = 2676, v_vnodepgsout = 240, v_intrans = 8171, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0,
v_dfree = 0, v_pfree = 0, v_tfree = 7518937, v_page_size = 0, v_page_count = 0,
v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0,
v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0,
---Type <return> to continue, or q <return> to quit---
v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 7717, v_vforks = 7894, v_rforks = 0, v_kthreads
= 0, v_forkpages = 2012738, v_vforkpages = 1684979, v_rforkpages = 0, v_kthreadpages = 0},
pc_cp_time = {7882266, 330, 49708, 171, 25024387}, pc_device = 0xffffff0012eef600, pc_netisr = 0x0, pc_rm_queue = {rmq_next =
0xffffffff808b2750, rmq_prev = 0xffffffff808b2750}, pc_dynamic = 18446743526093842688,
pc_monitorbuf = '\0' <repeats 127 times>, pc_prvspace = 0xffffffff808b2600, pc_curpmap = 0xffffffff8083ea50, pc_tssp =
0xffffffff808aef20, pc_commontssp = 0xffffffff808aef20, pc_rsp0 = -549754381056,
pc_scratch_rsp = 140737488347000, pc_apic_id = 50, pc_acpi_id = 11, pc_fs32p = 0xffffffff808add50, pc_gs32p =
0xffffffff808add58, pc_ldt = 0xffffffff808add98, pc_tss = 0xffffffff808add88, pc_cmci_mask = 44}
$22 = {pc_curthread = 0xffffff0012d7b8c0, pc_idlethread = 0xffffff0012d7b8c0, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb
= 0xffffff8000158d00, pc_switchtime = 564139962266028, pc_switchticks = 247796549,
pc_cpuid = 21, pc_cpumask = 2097152, pc_other_cpus = 14680063, pc_allcpu = {sle_next = 0xffffffff808b2600}, pc_spinlocks = 0x0,
pc_cnt = {v_swtch = 241921442, v_trap = 27867529, v_syscall = 396426755, v_intr = 0,
v_soft = 36203414, v_vm_faults = 4579069, v_cow_faults = 243971, v_cow_optim = 80, v_zfod = 4045607, v_ozfod = 0, v_swapin =
0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 2750, v_vnodeout = 13,
v_vnodepgsin = 2750, v_vnodepgsout = 104, v_intrans = 6546, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0,
v_dfree = 0, v_pfree = 0, v_tfree = 6134243, v_page_size = 0, v_page_count = 0,
v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0,
v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0,
v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 5441, v_vforks = 7475, v_rforks = 0, v_kthreads
= 0, v_forkpages = 1420342, v_vforkpages = 1634961, v_rforkpages = 0, v_kthreadpages = 0},
pc_cp_time = {7064507, 140, 130489, 35805, 25725921}, pc_device = 0xffffff0012eef500, pc_netisr = 0x0, pc_rm_queue = {rmq_next =
0xffffffff808b29d0, rmq_prev = 0xffffffff808b29d0}, pc_dynamic = 18446743526093871360,
pc_monitorbuf = '\0' <repeats 127 times>, pc_prvspace = 0xffffffff808b2880, pc_curpmap = 0xffffffff8083ea50, pc_tssp =
0xffffffff808aef88, pc_commontssp = 0xffffffff808aef88, pc_rsp0 = -549754401536,
pc_scratch_rsp = 140737488349080, pc_apic_id = 51, pc_acpi_id = 23, pc_fs32p = 0xffffffff808addb8, pc_gs32p =
0xffffffff808addc0, pc_ldt = 0xffffffff808ade00, pc_tss = 0xffffffff808addf0, pc_cmci_mask = 44}
$23 = {pc_curthread = 0xffffff0012d70000, pc_idlethread = 0xffffff0012d70000, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb
= 0xffffff8000153d00, pc_switchtime = 564139963254342, pc_switchticks = 247796550,
pc_cpuid = 22, pc_cpumask = 4194304, pc_other_cpus = 12582911, pc_allcpu = {sle_next = 0xffffffff808b2880}, pc_spinlocks = 0x0,
pc_cnt = {v_swtch = 561077805, v_trap = 70503560, v_syscall = 1547897389, v_intr = 0,
v_soft = 145263466, v_vm_faults = 5402089, v_cow_faults = 373167, v_cow_optim = 88, v_zfod = 4593818, v_ozfod = 0, v_swapin =
0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 3032, v_vnodeout = 16,
v_vnodepgsin = 3032, v_vnodepgsout = 128, v_intrans = 7615, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0,
v_dfree = 0, v_pfree = 0, v_tfree = 6367193, v_page_size = 0, v_page_count = 0,
v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0,
v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0,
v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 5531, v_vforks = 6954, v_rforks = 0, v_kthreads
= 0, v_forkpages = 1440742, v_vforkpages = 1477372, v_rforkpages = 0, v_kthreadpages = 0},
pc_cp_time = {7664569, 311, 54855, 173, 25236954}, pc_device = 0xffffff0012eef400, pc_netisr = 0x0, pc_rm_queue = {rmq_next =
0xffffffff808b2c50, rmq_prev = 0xffffffff808b2c50}, pc_dynamic = 18446743526093900032,
pc_monitorbuf = '\0' <repeats 127 times>, pc_prvspace = 0xffffffff808b2b00, pc_curpmap = 0xffffffff8083ea50, pc_tssp =
0xffffffff808aeff0, pc_commontssp = 0xffffffff808aeff0, pc_rsp0 = -549754422016,
pc_scratch_rsp = 140737488347256, pc_apic_id = 52, pc_acpi_id = 12, pc_fs32p = 0xffffffff808ade20, pc_gs32p =
0xffffffff808ade28, pc_ldt = 0xffffffff808ade68, pc_tss = 0xffffffff808ade58, pc_cmci_mask = 44}
$24 = {pc_curthread = 0xffffff0012d70460, pc_idlethread = 0xffffff0012d70460, pc_fpcurthread = 0x0, pc_deadthread = 0x0, pc_curpcb
= 0xffffff800014ed00, pc_switchtime = 564139962421094, pc_switchticks = 247796549,
pc_cpuid = 23, pc_cpumask = 8388608, pc_other_cpus = 8388607, pc_allcpu = {sle_next = 0xffffffff808b2b00}, pc_spinlocks = 0x0,
pc_cnt = {v_swtch = 206202024, v_trap = 21901993, v_syscall = 456089216, v_intr = 0,
v_soft = 44500078, v_vm_faults = 4394323, v_cow_faults = 229085, v_cow_optim = 71, v_zfod = 3915765, v_ozfod = 0, v_swapin =
0, v_swapout = 0, v_swappgsin = 0, v_swappgsout = 0, v_vnodein = 2733, v_vnodeout = 5,
v_vnodepgsin = 2733, v_vnodepgsout = 40, v_intrans = 6360, v_reactivated = 0, v_pdwakeups = 0, v_pdpages = 0, v_tcached = 0,
v_dfree = 0, v_pfree = 0, v_tfree = 5866068, v_page_size = 0, v_page_count = 0,
v_free_reserved = 0, v_free_target = 0, v_free_min = 0, v_free_count = 0, v_wire_count = 0, v_active_count = 0,
v_inactive_target = 0, v_inactive_count = 0, v_cache_count = 0, v_cache_min = 0, v_cache_max = 0,
v_pageout_free_min = 0, v_interrupt_free_min = 0, v_free_severe = 0, v_forks = 4299, v_vforks = 5814, v_rforks = 0, v_kthreads
= 0, v_forkpages = 1116482, v_vforkpages = 1250752, v_rforkpages = 0, v_kthreadpages = 0},
pc_cp_time = {7132190, 176, 32115, 131, 25792250}, pc_device = 0xffffff0012eef300, pc_netisr = 0x0, pc_rm_queue = {rmq_next =
0xffffffff808b2ed0, rmq_prev = 0xffffffff808b2ed0}, pc_dynamic = 18446743526093928704,
pc_monitorbuf = '\0' <repeats 127 times>, pc_prvspace = 0xffffffff808b2d80, pc_curpmap = 0xffffffff8083ea50, pc_tssp =
0xffffffff808af058, pc_commontssp = 0xffffffff808af058, pc_rsp0 = -549754442496,
pc_scratch_rsp = 140737488347256, pc_apic_id = 53, pc_acpi_id = 24, pc_fs32p = 0xffffffff808ade88, pc_gs32p =
0xffffffff808ade90, pc_ldt = 0xffffffff808aded0, pc_tss = 0xffffffff808adec0, pc_cmci_mask = 44}
> A little bit later I will send you another patch that, I hope, will produce better
> diagnostics for this crash (without DDB in kernel).
Kernel with the patch is now installed on the test machine.
I've taken DDB, INVARIANTS and STACK changes out for now.
Regards
Steve
================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.
In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postmaster@multiplay.co.uk.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?581C95046B0948FC82D6F2E86948F87B>