From owner-freebsd-bugs@freebsd.org Sun Mar 8 00:45:01 2020 Return-Path: Delivered-To: freebsd-bugs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DA162256E89 for ; Sun, 8 Mar 2020 00:45:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 48ZjLP3tnyz4FK7 for ; Sun, 8 Mar 2020 00:45:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 4E4CF256E88; Sun, 8 Mar 2020 00:45:01 +0000 (UTC) Delivered-To: bugs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4DBE2256E87 for ; Sun, 8 Mar 2020 00:45:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48ZjLP0RZRz4FJ4 for ; Sun, 8 Mar 2020 00:45:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id A690719AED for ; Sun, 8 Mar 2020 00:45:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 0280j0Ia009263 for ; Sun, 8 Mar 2020 00:45:00 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 0280j0lr009258 for bugs@FreeBSD.org; Sun, 8 Mar 2020 00:45:00 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 244665] Very slow NFS I/O during ZFS resilver - _sleep() sleeps really long... Date: Sun, 08 Mar 2020 00:45:00 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.3-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: pen@lysator.liu.se X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 08 Mar 2020 00:45:01 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D244665 Bug ID: 244665 Summary: Very slow NFS I/O during ZFS resilver - _sleep() sleeps really long... Product: Base System Version: 11.3-RELEASE Hardware: Any OS: Any Status: New Severity: Affects Some People Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: pen@lysator.liu.se I've been investigating an issue we're having that gives our users a really lousy NFS experience during ZFS resilver on one of our servers.=20 In order to try to pinpoint exactly what's going on I've tried various thin= gs. Lately I've been dtrace the call path for "mkdir()" and it seems that it is= the _sleep() kernel call that sometimes sleeps for _really_ long times like 1-20 seconds... Sure, the kernel is busy doing the ZFS resilver and other stuff but this is silly... Any ideas? Too low priority of the _sleep (prio=3D99?) compared to= other kernel threads so it never gets scheduled? One example: 21 54842 nfsrvd_dorpc:entry Start 21 47273 nfsv4_lock:entry Start(lp->nfslock_lock=3D6, iwantlock=3D0) 21 37590 nfsmsleep:entry Start(ffffffff81e9982c, ffffffff81e998a0, 99) 21 54396 _sleep:entry Start(prio=3D99, timeo=3D0) 16 54397 _sleep:return 10942618 =C2=B5s 16 37591 nfsmsleep:return 10942626 =C2=B5s 16 37590 nfsmsleep:entry Start(ffffffff81e9982c, ffffffff81e998a0, 99) 16 54396 _sleep:entry Start(prio=3D99, timeo=3D0) 7 54397 _sleep:return 344 =C2=B5s 7 37591 nfsmsleep:return 352 =C2=B5s 7 47274 nfsv4_lock:return 10942987 =C2=B5s 7 13973 nfsrvd_statstart:entry Start 7 13974 nfsrvd_statstart:return 2 =C2=B5s 7 44369 nfsrv_mallocmget_limit:entry Start 7 44370 nfsrv_mallocmget_limit:return 2 =C2=B5s 7 13971 nfsrvd_statend:entry Start 7 13972 nfsrvd_statend:return 2 =C2=B5s 7 13973 nfsrvd_statstart:entry Start 7 13974 nfsrvd_statstart:return 1 =C2=B5s 7 51444 nfsrv_mtofh:entry Start 7 51445 nfsrv_mtofh:return 2 =C2=B5s 7 47329 nfsd_fhtovp:entry Start 7 48059 __mtx_lock_flags:entry Start 7 48060 __mtx_lock_flags:return 2 =C2=B5s 7 47330 nfsd_fhtovp:return 14 =C2=B5s 7 13971 nfsrvd_statend:entry Start 7 13972 nfsrvd_statend:return 1 =C2=B5s 7 13973 nfsrvd_statstart:entry Start 7 13974 nfsrvd_statstart:return 1 =C2=B5s 7 43887 nfsrvd_getattr:entry Start 7 38608 nfsv4root_getreferral:entry Start 7 38609 nfsv4root_getreferral:return 2 =C2=B5s 7 43888 nfsrvd_getattr:return 22 =C2=B5s 7 13971 nfsrvd_statend:entry Start 7 13972 nfsrvd_statend:return 1 =C2=B5s 7 54843 nfsrvd_dorpc:return 10943076 =C2=B5s --=20 You are receiving this mail because: You are the assignee for the bug.=