From nobody Mon May 30 18:00:40 2022 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id B656E1B43E4E for ; Mon, 30 May 2022 18:00:43 +0000 (UTC) (envelope-from paulf2718@gmail.com) Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4LBjrB6PN7z4tnH for ; Mon, 30 May 2022 18:00:42 +0000 (UTC) (envelope-from paulf2718@gmail.com) Received: by mail-wr1-x431.google.com with SMTP id s24so8340928wrb.10 for ; Mon, 30 May 2022 11:00:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:date:mime-version:user-agent:subject:content-language:to :references:from:in-reply-to:content-transfer-encoding; bh=qyD0UWlBXJhNQCaLQJ+8ti9CK7DdukrlJrnLj+DIuCs=; b=ZxsMhw/EXS31HEZse5Jaxyar+eM/cBjbMizcO5Wu18J8wSXD+1Z8EyGbiB32jzEZcF QnKZ9TFzO9/5TKNIrCNMJIXpu4JOkVvwi/veZgL0NKWeeZH2RAualCF3PfcTAnoHhCEb aWBXhEhx6W8VH4hoKHN2jCmtpVNySKaagk4AxM3lngBBxnUafYjggEALw2H+I4KZ8NYE 6EGLwRAezYR+uefIazanMKXfj30uFHgNY+alpeDlQ/avDejfRVJvYFKpyJqXC8xGmNBY k2esAR2JHHtCYEKT+XyNv9akxuGY7tnAY45fhho8YDA1f4tvLndtuT8Y70/LbY9Lu/i5 IFCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent:subject :content-language:to:references:from:in-reply-to :content-transfer-encoding; bh=qyD0UWlBXJhNQCaLQJ+8ti9CK7DdukrlJrnLj+DIuCs=; b=tGApLbN/CVyyOXZSjCPTIsS5hfbC158dVx/TE+ifXh7DQbovGsEtv6bv8UrtwtSlAV sB6JXrMrXj+EigiHf6Zz+xy1mJ0oDo9OZ8wB7cn7qXcVJQkPr4+W6u6o2nxAkWqdbIzv 8rtEB16NuJ5DV/zQI9zt7etTOLuufDWDmJFouFWzVTOyYOzUd7XEAGvvRnq0fZJIhhvz 3U5cODVtgkWFeYXsfUinuNRdD41p/r0XCLkp2PcilgBuxMV9OlZ+Lt6/UvNl7GNksF/b +IbDzZmIpFh/lDCzURld8X0/7HCXQtdB0M99hTTT5ANpFLSVJcVjjV5aclz8cwP5HD1N LcOA== X-Gm-Message-State: AOAM530QU4Ja+GQ+jAnZiztGcEO0qjPRaMsvmQ41wSyaFmFo2+DR8C2i 3zjdYLAVJO/6vcJ/6+T8XPVGnOsxzHcnyQ== X-Google-Smtp-Source: ABdhPJyZ90/Dkrzw2i3Zsjqn9UXcqeM/C+yy8itNFYIQNzg5ONSLrSf5FCHsp61gOjsRHKVRMDSMPA== X-Received: by 2002:adf:f2cb:0:b0:20f:d291:7064 with SMTP id d11-20020adff2cb000000b0020fd2917064mr35068436wrp.319.1653933641815; Mon, 30 May 2022 11:00:41 -0700 (PDT) Received: from [192.168.1.28] (lfbn-lyo-1-398-93.w2-7.abo.wanadoo.fr. [2.7.225.93]) by smtp.gmail.com with ESMTPSA id n64-20020a1ca443000000b003973b9d0447sm12901243wme.36.2022.05.30.11.00.41 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 30 May 2022 11:00:41 -0700 (PDT) Message-ID: <0eb6bf73-475e-ace9-3df8-e96a6bb2cb96@gmail.com> Date: Mon, 30 May 2022 20:00:40 +0200 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:91.0) Gecko/20100101 Thunderbird/91.10.0 Subject: Re: Hang ast / pipelk / piperd Content-Language: en-US To: FreeBSD Hackers References: <84015bf9-8504-1c3c-0ba5-58d0d7824843@gmail.com> From: Paul Floyd In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4LBjrB6PN7z4tnH X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b="ZxsMhw/E"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of paulf2718@gmail.com designates 2a00:1450:4864:20::431 as permitted sender) smtp.mailfrom=paulf2718@gmail.com X-Spamd-Result: default: False [-3.68 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.68)[-0.675]; RECEIVED_SPAMHAUS_PBL(0.00)[2.7.225.93:received]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::431:from]; MLMMJ_DEST(0.00)[freebsd-hackers]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N > "procstat -kk " might help to reveal what's going on, > since it sounds like the hand/livelock is happening somewhere in the > kernel. Hi Here is the output paulf@freebsd:~ $ procstat -kk 864 PID TID COMM TDNAME KSTACK 864 100075 none-amd64-freebsd - mi_switch+0xc2 sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 umtxq_sleep+0x143 do_wait+0x3e5 __umtx_op_wait+0x53 sys__umtx_op+0x7e amd64_syscall+0x10c fast_syscall_common+0xf8 864 100175 none-amd64-freebsd - mi_switch+0xc2 intr_event_handle+0x167 intr_execute_handlers+0x4b Xapic_isr1+0xdc setrunnable+0x31 wakeup_one+0x1f pipe_read+0x38f dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8 864 100176 none-amd64-freebsd - mi_switch+0xc2 sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 pipe_read+0xb3 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8 864 100177 none-amd64-freebsd - mi_switch+0xc2 sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 pipe_read+0xb3 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8 864 100178 none-amd64-freebsd - mi_switch+0xc2 sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 pipe_read+0x3d6 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8 864 100179 none-amd64-freebsd - mi_switch+0xc2 sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 pipe_read+0x3d6 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8 864 100180 none-amd64-freebsd - mi_switch+0xc2 sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 pipe_read+0x3d6 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8 864 100181 none-amd64-freebsd - mi_switch+0xc2 sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 pipe_read+0x3d6 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8 864 100182 none-amd64-freebsd - mi_switch+0xc2 sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 pipe_read+0x3d6 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8 864 100183 none-amd64-freebsd - mi_switch+0xc2 sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 pipe_read+0x3d6 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8 864 100184 none-amd64-freebsd - mi_switch+0xc2 sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 pipe_read+0xb3 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8 864 100185 none-amd64-freebsd - mi_switch+0xc2 ast+0x1e6 doreti_ast+0x1f 864 100186 none-amd64-freebsd - mi_switch+0xc2 sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 pipe_read+0xb3 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8 864 100187 none-amd64-freebsd - mi_switch+0xc2 sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 pipe_read+0xb3 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8 864 100188 none-amd64-freebsd - mi_switch+0xc2 sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 pipe_read+0xb3 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8 It doesn't seem to be totally hung. If I repeatedly sample I do see activity changing between the threads other than main (in _umtx_op) and 12 (stuck in ast / doreti_ast. But it looks like none of the calls to read is completing, meaning the Valgrind scheduler is blocked. A+ Paul