Date: Mon, 26 Mar 2012 11:10:04 GMT From: Christian Esken <Christian.Esken@trivago.com> To: freebsd-bugs@FreeBSD.org Subject: Re: misc/166340: Process under FreeBSD 9.0 hangs in uninterruptable sleep with apparently no syscall (empty wchan) Message-ID: <201203261110.q2QBA481020440@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
The following reply was made to PR kern/166340; it has been noted by GNATS.
From: Christian Esken <Christian.Esken@trivago.com>
To: Kostik Belousov <kostikbel@gmail.com>
Cc: bug-followup@FreeBSD.org, avg@freebsd.org
Subject: Re: misc/166340: Process under FreeBSD 9.0 hangs in uninterruptable
sleep with apparently no syscall (empty wchan)
Date: Mon, 26 Mar 2012 11:58:12 +0200
Am Samstag, den 24.03.2012, 00:54 +0200 schrieb Kostik Belousov:
> Please, attach the kgdb to the running system when the process hang
> with the '-' wchan.
> Use the command like "kgdb /boot/kernel/kernel.symbols /dev/mem".
>
> Then, run the shell command ps -o pid,paddr | grep <pid> where pid is
> the pid of the hung
> process. Take the printed address A and, from kgdb, do:
> p *(struct proc *)A
> p/x *(((struct proc *)A)->p_threads.tqh_first)
> and show us the output.
Thanks for the quick response. Requested information can be found below.
I am also showing signal status, as signals seem to be relevant in
the kernel code section shown by Andriy.
Christian
# ps -o user,pid,ppid,pending,caught,ignored,blocked,stat,wchan 1780
USER PID PPID PENDING CAUGHT IGNORED BLOCKED STAT WCHAN
nobody 1780 1776 0 80005006 79fa8011 0 D -
# ps wwwaux -o pid,paddr | grep 1780
nobody 1780 0.0 0.0 55252 6040 ?? D 12:11PM
0:01.09 /usr/local/bin/s 1780 fffffe003eb71488
(kgdb) p *(struct proc *)0xfffffe003eb71488
$1 = {p_list = {le_next = 0xfffffe003eb71910, le_prev = 0xfffffe003e20f488}, p_threads = {
tqh_first = 0xfffffe000bcf7460, tqh_last = 0xfffffe000bcf7470}, p_slock = {lock_object = {
lo_name = 0xffffffff80ccb0fc "process slock", lo_flags = 720896, lo_data = 0,
lo_witness = 0xffffff8000689f80}, mtx_lock = 4}, p_ucred = 0xfffffe003e850200,
p_fd = 0xfffffe003ec9aa00, p_fdtol = 0x0, p_stats = 0xfffffe003eca3400,
p_limit = 0xfffffe0008372500, p_limco = {c_links = {sle = {sle_next = 0x0}, tqe = {
tqe_next = 0x0, tqe_prev = 0x0}}, c_time = 0, c_arg = 0x0, c_func = 0,
c_lock = 0xfffffe003eb71580, c_flags = 0, c_cpu = 0}, p_sigacts = 0xfffffe003e9de000,
p_flag = 268452096, p_state = PRS_NORMAL, p_pid = 1780, p_hash = {le_next = 0x0,
le_prev = 0xfffffe03889e00b8}, p_pglist = {le_next = 0xfffffe003eb71910,
le_prev = 0xfffffe01936a39d8}, p_pptr = 0xfffffe003e99b488, p_sibling = {
le_next = 0xfffffe003eb71910, le_prev = 0xfffffe01936a39f0}, p_children = {
lh_first = 0x0}, p_mtx = {lock_object = {lo_name = 0xffffffff80ccb0ef "process lock",
lo_flags = 21168128, lo_data = 0, lo_witness = 0xffffff8000688400}, mtx_lock = 4},
p_ksi = 0xfffffe000b0a6380, p_sigqueue = {sq_signals = {__bits = {0, 0, 0, 0}}, sq_kill = {
__bits = {0, 0, 0, 0}}, sq_list = {tqh_first = 0x0, tqh_last = 0xfffffe003eb715c8},
sq_proc = 0xfffffe003eb71488, sq_flags = 1}, p_oppid = 0, p_dbg_child = 0,
p_vmspace = 0xfffffe000b1e7620, p_swtick = 249340, p_realtimer = {it_interval = {
tv_sec = 0, tv_usec = 0}, it_value = {tv_sec = 0, tv_usec = 0}}, p_ru = {ru_utime = {
tv_sec = 0, tv_usec = 0}, ru_stime = {tv_sec = 0, tv_usec = 0}, ru_maxrss = 0,
ru_ixrss = 0, ru_idrss = 0, ru_isrss = 0, ru_minflt = 0, ru_majflt = 0, ru_nswap = 0,
ru_inblock = 0, ru_oublock = 0, ru_msgsnd = 0, ru_msgrcv = 0, ru_nsignals = 0,
ru_nvcsw = 0, ru_nivcsw = 0}, p_rux = {rux_runtime = 2178328093, rux_uticks = 77,
rux_sticks = 39, rux_iticks = 0, rux_uu = 722939, rux_su = 366163, rux_tu = 1089103},
p_crux = {rux_runtime = 0, rux_uticks = 0, rux_sticks = 0, rux_iticks = 0, rux_uu = 0,
rux_su = 0, rux_tu = 0}, p_profthreads = 0, p_exitthreads = 0, p_traceflag = 0,
p_tracevp = 0x0, p_tracecred = 0x0, p_textvp = 0xfffffe003ebdfb40, p_lock = 0,
p_sigiolst = {slh_first = 0x0}, p_sigparent = 20, p_sig = 0, p_code = 0, p_stops = 0,
p_stype = 0, p_step = 0 '\0', p_pfsflags = 0 '\0', p_nlminfo = 0x0, p_aioinfo = 0x0,
p_singlethread = 0x0, p_suspcount = 0, p_xthread = 0x0, p_boundary_count = 0,
p_pendingcnt = 0, p_itimers = 0x0, p_procdesc = 0x0, p_magic = 3203398350,
p_osrel = 900044, p_comm = "serelog", '\0' <repeats 12 times>, p_pgrp = 0xfffffe003e76a200,
p_sysent = 0xffffffff81066440, p_args = 0xfffffe003e77f700,
p_cpulimit = 9223372036854775807, p_nice = 0 '\0', p_fibnum = 0, p_xstat = 0, p_klist = {
kl_list = {slh_first = 0x0}, kl_lock = 0xffffffff807d4880 <knlist_mtx_lock>,
kl_unlock = 0xffffffff807d48a0 <knlist_mtx_unlock>,
kl_assert_locked = 0xffffffff807d4700 <knlist_mtx_assert_locked>,
kl_assert_unlocked = 0xffffffff807d4710 <knlist_mtx_assert_unlocked>,
kl_lockarg = 0xfffffe003eb71580}, p_numthreads = 1, p_md = {md_ldt = 0x0, md_ldt_sd = {
sd_lolimit = 0, sd_lobase = 0, sd_type = 0, sd_dpl = 0, sd_p = 0, sd_hilimit = 0,
sd_xx0 = 0, sd_gran = 0, sd_hibase = 0, sd_xx1 = 0, sd_mbz = 0, sd_xx2 = 0}},
p_itcallout = {c_links = {sle = {sle_next = 0x0}, tqe = {tqe_next = 0x0, tqe_prev = 0x0}},
c_time = 0, c_arg = 0x0, c_func = 0, c_lock = 0x0, c_flags = 16, c_cpu = 0},
p_acflag = 0, p_peers = 0x0, p_leader = 0xfffffe003eb71488, p_emuldata = 0x0,
p_label = 0x0, p_sched = 0xfffffe003eb71910, p_ktr = {stqh_first = 0x0,
stqh_last = 0xfffffe003eb718c0}, p_mqnotifier = {lh_first = 0x0}, p_dtrace = 0x0,
p_pwait = {cv_description = 0xffffffff80ccb64f "ppwait", cv_waiters = 0}, p_dbgwait = {
cv_description = 0xffffffff80ccb656 "dbgwait", cv_waiters = 0}, p_prev_runtime = 0,
p_racct = 0x0}
Current language: auto; currently asm
(kgdb) p/x *(((struct proc *)0xfffffe003eb71488)->p_threads.tqh_first)
$2 = {td_lock = 0xffffffff810d9400, td_proc = 0xfffffe003eb71488, td_plist = {tqe_next = 0x0,
tqe_prev = 0xfffffe003eb71498}, td_runq = {tqe_next = 0x0,
tqe_prev = 0xffffffff810d9628}, td_slpq = {tqe_next = 0x0,
tqe_prev = 0xfffffe000b1f0b80}, td_lockq = {tqe_next = 0x0,
tqe_prev = 0xffffff8465523920}, td_hash = {le_next = 0x0, le_prev = 0xffffff80006b3c10},
td_cpuset = 0xfffffe0008349dc8, td_sel = 0xfffffe003e77f900,
td_sleepqueue = 0xfffffe000b1f0b80, td_turnstile = 0xfffffe000bc6ec00,
td_umtxq = 0xfffffe000bce0680, td_tid = 0x18b82, td_sigqueue = {sq_signals = {__bits = {
0x0, 0x0, 0x0, 0x0}}, sq_kill = {__bits = {0x0, 0x0, 0x0, 0x0}}, sq_list = {
tqh_first = 0x0, tqh_last = 0xfffffe000bcf7510}, sq_proc = 0xfffffe003eb71488,
sq_flags = 0x1}, td_lend_user_pri = 0xff, td_flags = 0x14, td_inhibitors = 0x2,
td_pflags = 0x0, td_dupfd = 0x0, td_sqqueue = 0x0, td_wchan = 0x0, td_wmesg = 0x0,
td_lastcpu = 0x3, td_oncpu = 0xff, td_owepreempt = 0x0, td_tsqueue = 0x0, td_locks = 0x1,
td_rw_rlocks = 0x0, td_lk_slocks = 0x0, td_blocked = 0x0, td_lockname = 0x0,
td_contested = {lh_first = 0x0}, td_sleeplocks = 0xffffffff81249590,
td_intr_nesting_level = 0x0, td_pinned = 0x0, td_ucred = 0xfffffe003e850200,
td_estcpu = 0x0, td_slptick = 0x0, td_blktick = 0x0, td_swvoltick = 0x83351, td_ru = {
ru_utime = {tv_sec = 0x0, tv_usec = 0x0}, ru_stime = {tv_sec = 0x0, tv_usec = 0x0},
ru_maxrss = 0x1788, ru_ixrss = 0xd5e0, ru_idrss = 0x2c5cc, ru_isrss = 0x3a00,
ru_minflt = 0x1e4, ru_majflt = 0x0, ru_nswap = 0x0, ru_inblock = 0x0, ru_oublock = 0x0,
ru_msgsnd = 0x1e18, ru_msgrcv = 0x3c2e, ru_nsignals = 0x0, ru_nvcsw = 0x6eef,
ru_nivcsw = 0xcd7}, td_rux = {rux_runtime = 0x81d6a61d, rux_uticks = 0x4d,
rux_sticks = 0x27, rux_iticks = 0x0, rux_uu = 0xb07fb, rux_su = 0x59653,
rux_tu = 0x109e4f}, td_incruntime = 0x0, td_runtime = 0x81d6a61d, td_pticks = 0x0,
td_sticks = 0x0, td_iticks = 0x0, td_uticks = 0x0, td_intrval = 0x0, td_oldsigmask = {
__bits = {0x0, 0x0, 0x0, 0x0}}, td_sigmask = {__bits = {0x0, 0x0, 0x0, 0x0}},
td_generation = 0x7bc6, td_sigstk = {ss_sp = 0x0, ss_size = 0x0, ss_flags = 0x4},
td_xsig = 0x0, td_profil_addr = 0x0, td_profil_ticks = 0x0, td_name = {0x73, 0x65, 0x72,
0x65, 0x6c, 0x6f, 0x67, 0x0 <repeats 13 times>}, td_fpop = 0x0, td_dbgflags = 0x0,
td_dbgksi = {ksi_link = {tqe_next = 0x0, tqe_prev = 0x0}, ksi_info = {si_signo = 0x0,
si_errno = 0x0, si_code = 0x0, si_pid = 0x0, si_uid = 0x0, si_status = 0x0,
si_addr = 0x0, si_value = {sival_int = 0x0, sival_ptr = 0x0, sigval_int = 0x0,
sigval_ptr = 0x0}, _reason = {_fault = {_trapno = 0x0}, _timer = {_timerid = 0x0,
_overrun = 0x0}, _mesgq = {_mqd = 0x0}, _poll = {_band = 0x0}, __spare__ = {
__spare1__ = 0x0, __spare2__ = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}}},
ksi_flags = 0x0, ksi_sigq = 0x0}, td_ng_outbound = 0x0, td_osd = {osd_nslots = 0x0,
osd_slots = 0x0, osd_next = {le_next = 0x0, le_prev = 0x0}}, td_map_def_user = 0x0,
td_dbg_forked = 0x0, td_rqindex = 0x1e, td_base_pri = 0x7a, td_priority = 0x7a,
td_pri_class = 0x3, td_user_pri = 0x7a, td_base_user_pri = 0x7a,
td_pcb = 0xffffff8465262d10, td_state = 0x1, td_retval = {0x0, 0x4}, td_slpcallout = {
c_links = {sle = {sle_next = 0x0}, tqe = {tqe_next = 0x0,
tqe_prev = 0xffffff800083c110}}, c_time = 0x35fa11, c_arg = 0xfffffe000bcf7460,
c_func = 0xffffffff8083da40, c_lock = 0x0, c_flags = 0x16, c_cpu = 0x3},
td_frame = 0xffffff8465262c50, td_kstack_obj = 0xfffffe003ecfad80,
td_kstack = 0xffffff846525f000, td_kstack_pages = 0x4, td_critnest = 0x1, td_md = {
md_spinlock_count = 0x1, md_saved_flags = 0x246}, td_sched = 0xfffffe000bcf7888,
td_ar = 0x0, td_lprof = {{lh_first = 0x0}, {lh_first = 0x0}}, td_dtrace = 0x0,
td_errno = 0x0, td_vnet = 0x0, td_vnet_lpush = 0x0, td_intr_frame = 0x0}
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201203261110.q2QBA481020440>