From owner-freebsd-hackers@freebsd.org Thu Nov 28 20:53:03 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D6A601B7AF5 for ; Thu, 28 Nov 2019 20:53:03 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47P8wt3Ppdz453X for ; Thu, 28 Nov 2019 20:53:02 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 40262AD361; Thu, 28 Nov 2019 21:52:52 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qiGLNNNhYSwT; Thu, 28 Nov 2019 21:52:50 +0100 (CET) Received: from [192.168.10.9] (vaio [192.168.10.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id 8C3CEAD360; Thu, 28 Nov 2019 21:52:50 +0100 (CET) Subject: Re: Process in T state does not want to die..... To: Konstantin Belousov Cc: FreeBSD Hackers , Eugene Grosbein References: <966f830c-bf09-3683-90da-e70aa343cc16@digiware.nl> <3c57e51d-fa36-39a3-9691-49698e8d2124@grosbein.net> <91490c30-45e9-3c38-c55b-12534fd09e28@digiware.nl> <20191128115122.GN10580@kib.kiev.ua> <296874db-40f0-c7c9-a573-410e4c86049a@digiware.nl> <20191128195013.GU10580@kib.kiev.ua> From: Willem Jan Withagen Message-ID: <1ae7ad65-902c-8e5f-bcf1-1e98448c64bb@digiware.nl> Date: Thu, 28 Nov 2019 21:52:50 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.9.1 MIME-Version: 1.0 In-Reply-To: <20191128195013.GU10580@kib.kiev.ua> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: nl X-Rspamd-Queue-Id: 47P8wt3Ppdz453X X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of wjw@digiware.nl designates 2001:4cb8:90:ffff::3 as permitted sender) smtp.mailfrom=wjw@digiware.nl X-Spamd-Result: default: False [-5.29 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[digiware.nl]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; IP_SCORE(-2.99)[ip: (-9.52), ipnet: 2001:4cb8::/29(-4.63), asn: 28878(-0.83), country: NL(0.02)]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:28878, ipnet:2001:4cb8::/29, country:NL]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Nov 2019 20:53:03 -0000  # ps -o pid,lwp,flags,flags2,state,tracer,command -p 3532  PID    LWP        F       F2 STAT TRACER COMMAND 3532 103955 11080081 00000000 TsJ       0 ceph-osd -i 5 # procstat -kk 3532   PID    TID COMM                TDNAME              KSTACK  3532 103166 ceph-osd            log                 mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103167 ceph-osd            service             mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103168 ceph-osd            admin_socket        mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103169 ceph-osd            msgr-worker-0       mi_switch+0xe2 thread_suspend_switch+0x140 thread_single+0x47b sigexit+0x53 postsig+0x304 ast+0x327 fast_syscall_common+0x198  3532 103170 ceph-osd            msgr-worker-1       mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103171 ceph-osd            msgr-worker-2       mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103172 ceph-osd            signal_handler      mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103406 ceph-osd            OpHistorySvc        mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103407 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103418 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103419 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103421 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103427 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103663 ceph-osd            fn_anonymous        mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103675 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103677 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103678 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103679 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103680 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103681 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103682 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103683 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103684 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103685 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103955 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104621 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104826 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104827 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104828 ceph-osd            wb_throttle         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104829 ceph-osd            filestore_sync      mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104830 ceph-osd            journal_write       mi_switch+0xe2 sleepq_wait+0x2c _sleep+0x247 bwillwrite+0x97 dofilewrite+0x93 sys_writev+0x6e amd64_syscall+0x364 fast_syscall_common+0x101  3532 104831 ceph-osd            fn_jrn_objstore     mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104832 ceph-osd            tp_fstore_op        mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104833 ceph-osd            tp_fstore_op        mi_switch+0xe2 sleepq_wait+0x2c _sleep+0x247 bwillwrite+0x97 vn_open_cred+0xc8 zfs_setextattr+0x216 VOP_SETEXTATTR_APV+0x7c extattr_set_vp+0x11d sys_extattr_set_fd+0xee amd64_syscall+0x364 fast_syscall_common+0x101  3532 104834 ceph-osd            fn_odsk_fstore      mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104835 ceph-osd            fn_appl_fstore      mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104836 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104838 ceph-osd            ms_dispatch         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104839 ceph-osd            ms_local            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104841 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104843 ceph-osd            fn_anonymous        mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105398 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105399 ceph-osd            ms_dispatch         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105400 ceph-osd            ms_local            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105401 ceph-osd            ms_dispatch         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105402 ceph-osd            ms_local            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105403 ceph-osd            ms_dispatch         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105404 ceph-osd            ms_local            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105405 ceph-osd            ms_dispatch         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105406 ceph-osd            ms_local            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105407 ceph-osd            ms_dispatch         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105408 ceph-osd            ms_local            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105409 ceph-osd            ms_dispatch         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105410 ceph-osd            ms_local            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105411 ceph-osd            fn_anonymous        mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105412 ceph-osd            finisher            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105413 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105414 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105415 ceph-osd            osd_srv_agent       mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105416 ceph-osd            tp_osd_tp           mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105417 ceph-osd            tp_osd_tp           mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105418 ceph-osd            tp_osd_tp           mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105419 ceph-osd            tp_osd_tp           mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105420 ceph-osd            tp_osd_tp           mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105421 ceph-osd            tp_osd_cmd          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105422 ceph-osd            osd_srv_heartbt     mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f root@cephtest:~ # procstat -kk 3532 | less   PID    TID COMM                TDNAME              KSTACK  3532 103166 ceph-osd            log                 mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103167 ceph-osd            service             mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103168 ceph-osd            admin_socket        mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103169 ceph-osd            msgr-worker-0       mi_switch+0xe2 thread_suspend_switch+0x140 thread_single+0x47b sigexit+0x53 postsig+0x304 ast+0x327 fast_syscall_common+0x198  3532 103170 ceph-osd            msgr-worker-1       mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103171 ceph-osd            msgr-worker-2       mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103172 ceph-osd            signal_handler      mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103406 ceph-osd            OpHistorySvc        mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103407 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103418 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103419 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103421 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103427 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103663 ceph-osd            fn_anonymous        mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103675 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103677 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103678 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103679 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103680 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103681 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103682 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103683 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103684 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103685 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103955 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104621 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104826 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104827 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104828 ceph-osd            wb_throttle         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104829 ceph-osd            filestore_sync      mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104830 ceph-osd            journal_write       mi_switch+0xe2 sleepq_wait+0x2c _sleep+0x247 bwillwrite+0x97 dofilewrite+0x93 sys_writev+0x6e amd64_syscall+0x364 fast_syscall_common+0x101  3532 104831 ceph-osd            fn_jrn_objstore     mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104832 ceph-osd            tp_fstore_op        mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104833 ceph-osd            tp_fstore_op        mi_switch+0xe2 sleepq_wait+0x2c _sleep+0x247 bwillwrite+0x97 vn_open_cred+0xc8 zfs_setextattr+0x216 VOP_SETEXTATTR_APV+0x7c extattr_set_vp+0x11d sys_extattr_set_fd+0xee amd64_syscall+0x364 fast_syscall_common+0x101  3532 104834 ceph-osd            fn_odsk_fstore      mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104835 ceph-osd            fn_appl_fstore      mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104836 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104838 ceph-osd            ms_dispatch         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104839 ceph-osd            ms_local            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104841 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104843 ceph-osd            fn_anonymous        mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105398 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105399 ceph-osd            ms_dispatch         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105400 ceph-osd            ms_local            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105401 ceph-osd            ms_dispatch         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105402 ceph-osd            ms_local            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105403 ceph-osd            ms_dispatch         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105404 ceph-osd            ms_local            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105405 ceph-osd            ms_dispatch         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105406 ceph-osd            ms_local            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105407 ceph-osd            ms_dispatch         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105408 ceph-osd            ms_local            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105409 ceph-osd            ms_dispatch         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105410 ceph-osd            ms_local            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105411 ceph-osd            fn_anonymous        mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105412 ceph-osd            finisher            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105413 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105414 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105415 ceph-osd            osd_srv_agent       mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105416 ceph-osd            tp_osd_tp           mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105417 ceph-osd            tp_osd_tp           mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105418 ceph-osd            tp_osd_tp           mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105419 ceph-osd            tp_osd_tp           mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105420 ceph-osd            tp_osd_tp           mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105421 ceph-osd            tp_osd_cmd          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105422 ceph-osd            osd_srv_heartbt     mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f root@cephtest:~ #