From owner-freebsd-stable@freebsd.org Mon Mar 4 15:09:51 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7C4FF15146A1 for ; Mon, 4 Mar 2019 15:09:51 +0000 (UTC) (envelope-from ncrogers@gmail.com) Received: from mail-yw1-xc43.google.com (mail-yw1-xc43.google.com [IPv6:2607:f8b0:4864:20::c43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8E4E76E576; Mon, 4 Mar 2019 15:09:50 +0000 (UTC) (envelope-from ncrogers@gmail.com) Received: by mail-yw1-xc43.google.com with SMTP id k14so4299845ywe.4; Mon, 04 Mar 2019 07:09:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=4y+cqLp6SrDOfMXZVnErgIWPK7dsIzZhTE9FJqddNdw=; b=Bu/sQZmPidieuyt2Qe9J56Q6aVmje5ZZhrXHxbsObdCi4mz3177l3dBF/IA6AYIkL8 ymeUM2HQJuiLY5V/OQIUcq7FnBqP5LfJHrjFBhaNTbOmbab7BW35vTKIRhC2dCuVhpti oFvrTYg3WFAFLcguLazNuVSmB60bywMdOt4PJmvHZey8jnFh6pBeD2Fe6xSAfcpyYBF1 CvWPzWaUStLXamz44DKa2BKHcXj3GxYkALhMkz5XGoH8xlw3DwoLGsWKHXwmxwXbNR95 TMIfVMk8AmzKYBKptbhWwa1jZES/H30lFGow9K/8usj4mbjXZkduiWeBkwcwv5FrMiVJ bWjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=4y+cqLp6SrDOfMXZVnErgIWPK7dsIzZhTE9FJqddNdw=; b=dbIcysSQTOFtZqSbp00kPbixx0w39SCH8KS97vTMG4kZGwZme/zotS81mClQIEpIkz v5FQmuZgynQxxK8iLCO56BkUiin2sQlY2nYvr/lRWnKkNB90kJzv8HpCuXLHQoO+gh58 Vs5vuxVutjvCrS79/grUAq/PJ2F/pMB+NHKN+DYNnsoZ95429Xybb4F8VGgs8vLPytW1 Jwt6EOwRRCoLTnpVWDECoEBtot95JMNNwG67uea4a4GzsPh6daNDdJBrRBYPq7/lFbBI CKHVSJ3A/oylMGGq7ZNKYk7brMenJiVdGD6sD6di8pqVe4KttuhFnN0brSWOFe+OKArx 0jYA== X-Gm-Message-State: APjAAAXxgu5vmEset9NJwdPjrOmpV0Tf4dxm3wZJl+HW7hFTtw6ZQ1DB cKmCVwjnzjcA5rRNhie9zJTKZJy+OeFVePELXyq/yYfo X-Google-Smtp-Source: APXvYqy0k8w9jolitQX1l0yFH4mXw9yd1M4L924NOEnZvCz0ZTcpxdW3g6B606pXPIfKJgp2dcgDf+0ya1KbQV58YCM= X-Received: by 2002:a25:ab2f:: with SMTP id u44mr1210335ybi.377.1551712189723; Mon, 04 Mar 2019 07:09:49 -0800 (PST) MIME-Version: 1.0 References: <233f3723-a78e-91db-ebe6-9e13c1c9b50d@FreeBSD.org> In-Reply-To: <233f3723-a78e-91db-ebe6-9e13c1c9b50d@FreeBSD.org> From: Nick Rogers Date: Mon, 4 Mar 2019 10:09:38 -0500 Message-ID: Subject: Re: 12.0-RELEASE zfs/vnode deadlock issue To: Andriy Gapon Cc: FreeBSD STABLE X-Rspamd-Queue-Id: 8E4E76E576 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=Bu/sQZmP; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of ncrogers@gmail.com designates 2607:f8b0:4864:20::c43 as permitted sender) smtp.mailfrom=ncrogers@gmail.com X-Spamd-Result: default: False [-3.77 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.990,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.48)[-0.476,0]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[3.4.c.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-0.29)[ip: (3.32), ipnet: 2607:f8b0::/32(-2.69), asn: 15169(-2.03), country: US(-0.07)]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Mar 2019 15:09:51 -0000 Thanks for the insight, it does appear that in all instances of this problem there is always one thread stuck on zfs_znode_alloc. Unfortunately its always a different application (e.g., perl, sh, postgres). I will post more information in the bug. On Sat, Mar 2, 2019 at 12:48 PM Andriy Gapon wrote: > On 01/03/2019 17:00, Nick Rogers wrote: > > 36704 101146 perl - mi_switch+0xe1 > > sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_xlock_hard+0x19c > VOP_LOCK1_APV+0x7e > > _vn_lock+0x40 zfs_znode_alloc+0x434 zfs_mknode+0xa9d > > zfs_freebsd_create+0x512 VOP_CREATE_APV+0x78 vn_open_cred+0x2c9 > > kern_openat+0x20c amd64_syscall+0x369 fast_syscall_common+0x101 > > I suspect that this thread is a root cause of the problem. > In this place, the vnode should be freshly created and not visible to > anything > but the current thread. So, vn_lock() should always immediately succeed. > I > cannot understand how the vnode lock could be held by another thread. > > -- > Andriy Gapon >