From owner-freebsd-fs@freebsd.org Tue Oct 1 05:21:02 2019 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 80F2E145EA3 for ; Tue, 1 Oct 2019 05:21:02 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qk1-x729.google.com (mail-qk1-x729.google.com [IPv6:2607:f8b0:4864:20::729]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46j70F4Pt6z4BW5 for ; Tue, 1 Oct 2019 05:21:01 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qk1-x729.google.com with SMTP id f16so10009140qkl.9 for ; Mon, 30 Sep 2019 22:21:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=iXJG5kwlWgoxQKPdEp64eumT1Olg3QGkVTZlFEHYC80=; b=GFg1iVjiJetGP0+O1A3/Qxq8K3xIlSZLRhtOPjiktInyEvucbYKQXXxzy1yPyDkr3u zh6Q7+jX5/X1OyRqLFJ2mGdXMnSkJW8fM83CXscZtzHMCzesxYkFZDX+Bw5/vp/uWf3C wU16WHiLq5gVXTlCfKxVriIzE2iX/X0j28qO4b360gQv1kdxmZTQ0kr3M9rmULWwJ5Si p7fhJ1eZQarND0JhqK/jBS0cQDT1ZSiiZSRWh6Ydc2fgG6gAk7DcCOZuesV/lfiRztB/ Hu9Q75wSmhyBBXmI0Sj6TJ1yY9Xl5mqMQbG5qXzqikpiy3v++qFJgChpxrtQ6EzuRY0K FAEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=iXJG5kwlWgoxQKPdEp64eumT1Olg3QGkVTZlFEHYC80=; b=X2eaII/AV7PTlT3svSw0gQL0E8xKntDOdFF3HsjWc22BtIuYnxhdIUj9aRB8SofGmX L2vS/9jzbWNdufoOUcMmeLY9mSIgj408EYjoCQDd+Mpae8tpc9CSegI4ai7i0TAxALRX VjbTMkwLXqwruZrn6pdO91kocLSsnQU+7+wCwwYnz50k4VaUTb60T7UeAMFk8ASyQaa9 TpZadqKXTXrl1Bg8KOXFgPdJCdZ5WbZfNCCRa/es813c1NPzEn3a4p6sGVDrBpAbS6vT X2I6nJkvAkbRfT07wvWs4jtjuLJe8qisQfmFbozsofgItZJ11T5W2dltLNJNvVsiiHSG glsw== X-Gm-Message-State: APjAAAV0jZheVLmyYWVHYbJkknHPPywhWlOe/MAvDXgHi7mKklsEQAQb 0P+CltI/AEG+80EvyVg9lB8yJk/oYEXoDZFZvU51zA== X-Google-Smtp-Source: APXvYqy34o3V6RWRREMNQrIuV6eBUIQSWZzka/l9UShTAIskvIFRkYbdFIEsXOxSu7qIqd7eqhfFxyMwWy0FJ4jte/E= X-Received: by 2002:a05:620a:6af:: with SMTP id i15mr4050065qkh.380.1569907260291; Mon, 30 Sep 2019 22:21:00 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Mon, 30 Sep 2019 23:20:48 -0600 Message-ID: Subject: Re: [zfs] filesystem reads hanging To: Reshad Patuck Cc: FreeBSD FS X-Rspamd-Queue-Id: 46j70F4Pt6z4BW5 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=GFg1iVji; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::729) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-3.81 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; URI_COUNT_ODD(1.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[9.2.7.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; FREEMAIL_TO(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-2.81)[ip: (-9.26), ipnet: 2607:f8b0::/32(-2.58), asn: 15169(-2.16), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Oct 2019 05:21:02 -0000 On Mon, Sep 30, 2019, 10:56 PM Reshad Patuck wrote: > Hi, > > I have a FreeBSD 12.0-RELEASE-p9 system running ZFS. > The system runs an application that uses postgres, and python (among other > services). > > I have noticed that python suddenly is not able to connect to postgres. > When I try to investigate further, certain files on disk can not be read. > The commands `cat` and `ls -l` hang (no output and I can not ctrl-c or kill > -9 them), ps -aux shows them in a D+ state. > On killing the SSH session these processes continue running in orphans, I > am not able to kill them. > > Someone on IRC suggested running a zfs scrub to check for data corruption, > but running `zpool scrub zroot` has the same effect. > The command does not return, ctrl-c does not kill it and `zpool scrub -s > zroot` says "cannot cancel scrubbing zroot: there is no active scrub". > > This has happened in the past 1 month to two of my production servers and > since the application was critical they were rebooted and the boxes > function as normal after the reboot. > Files that were not cat-able on the production servers were working fine > and a zfs scrub worked fine to show 0 errors and 0 fixes. > One of these boxes needed a hard reboot as it got stuck in the shutting > down stage of a soft reboot. > > I am not sure where to start debugging this or if there are any ways to get > metrics on a box stuck in this state. > Please let me know if you would like me to fetch any metrics or run and > commands, etc. for you. > Any help would be much appreciated. > Step 1 should be to make sure there are no disk errors... the successful scrub suggests not, but it doesn't hurt to rule out hardware... Warner Best regards, > > Reshad > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >