From nobody Mon Mar 4 00:28:23 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tp0113CSFz5CdX8 for ; Mon, 4 Mar 2024 00:28:37 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pf1-x436.google.com (mail-pf1-x436.google.com [IPv6:2607:f8b0:4864:20::436]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tp0103wf5z4h3K for ; Mon, 4 Mar 2024 00:28:36 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=EeT5Fmdp; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2607:f8b0:4864:20::436 as permitted sender) smtp.mailfrom=rick.macklem@gmail.com Received: by mail-pf1-x436.google.com with SMTP id d2e1a72fcca58-6e58a984ea1so2498753b3a.2 for ; Sun, 03 Mar 2024 16:28:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709512115; x=1710116915; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=2g4BMOi33bCbE0rIxXZiy0KlU2ywmAEUe6BFKFG8eEc=; b=EeT5FmdpaLjQMwT3J0APUsfoTwkbpjdd9n05nSkmmG2D/iPwf5XaN890Fdcn21PXrJ aFjWCx9v8yR2vW+xQ9aNqyJ3hU03I+ktCFKdcYFJKoug46o5o+yroDetIWEG9HTD5EKp SwMqJNUcqpoxulKOwE9cIEyl3bIpOCxM2dwilKTcuDKY3BtXXaPWiRZQ49scNqXc0K48 E5nMzUXoyaCgyR+5xWbXr5B26mAawlYZnxJvQBtrpxrpi6HaiVpNWAQEFUM61j28xDLd 3irsryF1sU1I4onDtqzyVtVzSH4ua/KtSSuyhW7PEZ0niJEHG/6fHUL0ikdrGH8HCGmF bAHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709512115; x=1710116915; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2g4BMOi33bCbE0rIxXZiy0KlU2ywmAEUe6BFKFG8eEc=; b=VHHtiD7zfZ3m2aGZk3j/VTXVlSPJxqt19sdyYk5V28s+W9kK8RIOXDVrSwBV3yuj5W ls1tWmfyWKbBri+YQVFCUN7suY+2CqxUorTqbo7WT+v94yZnGX9iKCkKjRRbcKiSx+rM boOnuJ3akD7ONRRfvw5GEQN/1o8Pnre+sJIZlcCoAOW25Yrek96DZSE5QYmKII2dxM6C EozDNuGGFbOO8JqNtQhufCi6dkETaJON/VpkzLvRU0BbCtiek/4QqgzyxeckdzL+8+mu EcKhhw/tQ9H6b6RIrbZseCIzjyfVPIksdHTl6yr7t2eT5JVnEArp865owBh23Vrm3u40 0USA== X-Gm-Message-State: AOJu0Yz7kzcTGqwhvnl/XcvdtP+uskZu6j2TGVdWzQoiwSeAU017W/37 3zNS91oi+K+SjydXvMNehpW/25porDBzsN6l0LHjqI9PvwtNQzM9kp3Ts6e+/c44MuKJZD5WwzV 0Y5LYc10XikKc91/ALzbg09krBCdJfXg= X-Google-Smtp-Source: AGHT+IF+zAKeA+nP0yZK0vlzD0gh5NG2tJ8kl/N7lZ4KiuFujP0irTkEngUJ0Kf+WhiXwqzLbwvHNR0kd7uS9lP18ro= X-Received: by 2002:a05:6a20:428a:b0:1a0:f5e6:110d with SMTP id o10-20020a056a20428a00b001a0f5e6110dmr6246412pzj.7.1709512114837; Sun, 03 Mar 2024 16:28:34 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: <26078.50375.679881.64018@hergotha.csail.mit.edu> <26083.64612.717082.366639@hergotha.csail.mit.edu> In-Reply-To: From: Rick Macklem Date: Sun, 3 Mar 2024 16:28:23 -0800 Message-ID: Subject: Re: 13-stable NFS server hang To: Garrett Wollman Cc: stable@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: --- X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TAGGED_FROM(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; MISSING_XM_UA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[stable@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MLMMJ_DEST(0.00)[stable@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::436:from] X-Rspamd-Queue-Id: 4Tp0103wf5z4h3K On Sun, Mar 3, 2024 at 3:27=E2=80=AFPM Rick Macklem wrote: > > On Sun, Mar 3, 2024 at 1:17=E2=80=AFPM Rick Macklem wrote: > > > > On Sat, Mar 2, 2024 at 8:28=E2=80=AFPM Garrett Wollman wrote: > > > > > > > > > I wrote previously: > > > > PID TID COMM TDNAME KSTACK > > > > 997 108481 nfsd nfsd: master mi_switch sleepq= _timedwait _sleep nfsv4_lock nfsrvd_dorpc nfssvc_program svc_run_internal s= vc_run nfsrvd_nfsd nfssvc_nfsd sys_nfssvc amd64_syscall fast_syscall_common > > > > 997 960918 nfsd nfsd: service mi_switch sleepq= _timedwait _sleep nfsv4_lock nfsrv_setclient nfsrvd_exchangeid nfsrvd_dorpc= nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline > > > > 997 962232 nfsd nfsd: service mi_switch _cv_wa= it txg_wait_synced_impl txg_wait_synced dmu_offset_next zfs_holey zfs_freeb= sd_ioctl vn_generic_copy_file_range vop_stdcopy_file_range VOP_COPY_FILE_RA= NGE vn_copy_file_range nfsrvd_copy_file_range nfsrvd_dorpc nfssvc_program s= vc_run_internal svc_thread_start fork_exit fork_trampoline > > > > > > I spent some time this evening looking at this last stack trace, and > > > stumbled across the following comment in > > > sys/contrib/openzfs/module/zfs/dmu.c: > > > > > > | /* > > > | * Enable/disable forcing txg sync when dirty checking for holes wi= th lseek(). > > > | * By default this is enabled to ensure accurate hole reporting, it= can result > > > | * in a significant performance penalty for lseek(SEEK_HOLE) heavy = workloads. > > > | * Disabling this option will result in holes never being reported = in dirty > > > | * files which is always safe. > > > | */ > > > | int zfs_dmu_offset_next_sync =3D 1; > > > > > > I believe this explains why vn_copy_file_range sometimes takes much > > > longer than a second: our servers often have lots of data waiting to > > > be written to disk, and if the file being copied was recently modifie= d > > > (and so is dirty), this might take several seconds. I've set > > > vfs.zfs.dmu_offset_next_sync=3D0 on the server that was hurting the m= ost > > > and am watching to see if we have more freezes. > > > > > > If this does the trick, then I can delay deploying a new kernel until > > > April, after my upcoming vacation. > > Interesting. Please let us know how it goes. > Btw, I just tried this for my trivial test and it worked very well. > A 1Gbyte file was cpied in two Copy RPCs of 1sec and slightly less than > 1sec. Oops, I spoke too soon. The Copy RPCs worked fine (as above) but the Commit RPCs took a long time, so it still looks like you may need the patches. rick > > So, your vacation may be looking better, rick > > > > > And enjoy your vacation, rick > > > > > > > > -GAWollman > > > From nobody Mon Mar 4 00:29:50 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tp02h2bl0z5CdfH for ; Mon, 4 Mar 2024 00:30:04 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pf1-x42e.google.com (mail-pf1-x42e.google.com [IPv6:2607:f8b0:4864:20::42e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tp02g4jFmz4hs4 for ; Mon, 4 Mar 2024 00:30:03 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=UXHXgs2K; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2607:f8b0:4864:20::42e as permitted sender) smtp.mailfrom=rick.macklem@gmail.com Received: by mail-pf1-x42e.google.com with SMTP id d2e1a72fcca58-6e4560664b5so3197987b3a.1 for ; Sun, 03 Mar 2024 16:30:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709512202; x=1710117002; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=HgFDXfE8ViO+mtfmLC/wmKQqeYrAf+CZRccHBwyhLYE=; b=UXHXgs2KlT1itW+CLT4fE1nSfjVQzw8Ld75wcefQuiFvzGmhPRYTboZNw6raclCMtU C/pUbnNt9f4F9qZzehmxywdYzFWRGuQl09Th7BVo1AHZPVBXoA2a9Yemvuycy3YMBRhS PU0BMrKoKoAsFg8n4ie7zH75O3cDHceIJ1TTTAuU+ETJwpGMN8HNWRpN2qK8pjctx/9R yD7oshwnv18sDBfUIrzFgmt0fHljIJWYzu8rqfGIn94/IsLMG9Vjlmda9Ouno+CpSBjS tBWnm93aHBxZYf88CLNU+7Ax2oKMkRnw9wlN0I1O5veNp3pP/r5bteNPXs5oaRGMeRK/ f8jA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709512202; x=1710117002; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HgFDXfE8ViO+mtfmLC/wmKQqeYrAf+CZRccHBwyhLYE=; b=WyGSRaMqjyqFHWS553nqEkheybsMt7U97ip6HcwMIG5NPwkBCG2eIAP6Em6wdkFIXi vhUzOdXCIN8w2g/P6oNC5ycuC5eSJr9LUdlSaRdaVtlHUQfcgwcpmjDnxpDcRrueE7kr qN6KIv3rxnS457eYjxI9avYuic+ocCCK7VvzTrdqAcSnB3Yb0PF4EYNlANQ+0kPiQiME IQc4xwgAf59C166mkZq/oSENIVa3UNmacUnQZgCOpTYkuS/CmC3/aI3ouyC2fwJrW/YM zjlGjsAq4nOjWBf9JngrlBO8YC/W7D5iz7PCbSpAUJLuCoKPB4voOfGKvbQnz8+zQlw4 P9BA== X-Gm-Message-State: AOJu0YzUdTeqUGEzqHK0IK33L3HALA2+me1FTl9b7/R19fX2y+nCPeyN 2sbNooI1lSQA4NE/Tkjow4JZEHC/KqcSdwt02EIpgcC9LO33MyKAB6lO+j561PMWfJiP2muH0Xd Pj6fbDAxHIb1vp0dQ5GTE2K5lvx/TWMw= X-Google-Smtp-Source: AGHT+IETQg4YdaMPYIcqQwGdaQmARUWQ5qf+5ljZHtCBWy06rOOJxImG9S5uFXZ3UDdRVrnqwjlim/h62nh7aJ2yRl8= X-Received: by 2002:a05:6a20:1595:b0:1a1:1817:b13 with SMTP id h21-20020a056a20159500b001a118170b13mr10209665pzj.15.1709512202122; Sun, 03 Mar 2024 16:30:02 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: <26078.50375.679881.64018@hergotha.csail.mit.edu> <26083.64612.717082.366639@hergotha.csail.mit.edu> In-Reply-To: From: Rick Macklem Date: Sun, 3 Mar 2024 16:29:50 -0800 Message-ID: Subject: Re: 13-stable NFS server hang To: Garrett Wollman Cc: stable@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: --- X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TAGGED_FROM(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; MISSING_XM_UA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[stable@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MLMMJ_DEST(0.00)[stable@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::42e:from] X-Rspamd-Queue-Id: 4Tp02g4jFmz4hs4 On Sun, Mar 3, 2024 at 4:28=E2=80=AFPM Rick Macklem wrote: > > On Sun, Mar 3, 2024 at 3:27=E2=80=AFPM Rick Macklem wrote: > > > > On Sun, Mar 3, 2024 at 1:17=E2=80=AFPM Rick Macklem wrote: > > > > > > On Sat, Mar 2, 2024 at 8:28=E2=80=AFPM Garrett Wollman wrote: > > > > > > > > > > > > I wrote previously: > > > > > PID TID COMM TDNAME KSTACK > > > > > 997 108481 nfsd nfsd: master mi_switch slee= pq_timedwait _sleep nfsv4_lock nfsrvd_dorpc nfssvc_program svc_run_internal= svc_run nfsrvd_nfsd nfssvc_nfsd sys_nfssvc amd64_syscall fast_syscall_comm= on > > > > > 997 960918 nfsd nfsd: service mi_switch slee= pq_timedwait _sleep nfsv4_lock nfsrv_setclient nfsrvd_exchangeid nfsrvd_dor= pc nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoli= ne > > > > > 997 962232 nfsd nfsd: service mi_switch _cv_= wait txg_wait_synced_impl txg_wait_synced dmu_offset_next zfs_holey zfs_fre= ebsd_ioctl vn_generic_copy_file_range vop_stdcopy_file_range VOP_COPY_FILE_= RANGE vn_copy_file_range nfsrvd_copy_file_range nfsrvd_dorpc nfssvc_program= svc_run_internal svc_thread_start fork_exit fork_trampoline > > > > > > > > I spent some time this evening looking at this last stack trace, an= d > > > > stumbled across the following comment in > > > > sys/contrib/openzfs/module/zfs/dmu.c: > > > > > > > > | /* > > > > | * Enable/disable forcing txg sync when dirty checking for holes = with lseek(). > > > > | * By default this is enabled to ensure accurate hole reporting, = it can result > > > > | * in a significant performance penalty for lseek(SEEK_HOLE) heav= y workloads. > > > > | * Disabling this option will result in holes never being reporte= d in dirty > > > > | * files which is always safe. > > > > | */ > > > > | int zfs_dmu_offset_next_sync =3D 1; > > > > > > > > I believe this explains why vn_copy_file_range sometimes takes much > > > > longer than a second: our servers often have lots of data waiting t= o > > > > be written to disk, and if the file being copied was recently modif= ied > > > > (and so is dirty), this might take several seconds. I've set > > > > vfs.zfs.dmu_offset_next_sync=3D0 on the server that was hurting the= most > > > > and am watching to see if we have more freezes. > > > > > > > > If this does the trick, then I can delay deploying a new kernel unt= il > > > > April, after my upcoming vacation. > > > Interesting. Please let us know how it goes. > > Btw, I just tried this for my trivial test and it worked very well. > > A 1Gbyte file was cpied in two Copy RPCs of 1sec and slightly less than > > 1sec. > Oops, I spoke too soon. > The Copy RPCs worked fine (as above) but the Commit RPCs took > a long time, so it still looks like you may need the patches. And I should mention that my test is done on a laptop without a ZIL, so maybe a ZIL on a separate device might generate different results. rick > > rick > > > > > So, your vacation may be looking better, rick > > > > > > > > And enjoy your vacation, rick > > > > > > > > > > > -GAWollman > > > > From nobody Mon Mar 4 02:44:20 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tp31h4zLwz5Bc26 for ; Mon, 4 Mar 2024 02:44:24 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (tunnel82308-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "garrett.wollman.name", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tp31g5Sc0z4vPY for ; Mon, 4 Mar 2024 02:44:23 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bimajority.org (policy=none); spf=pass (mx1.freebsd.org: domain of wollman@hergotha.csail.mit.edu designates 2001:470:1f06:ccb::2 as permitted sender) smtp.mailfrom=wollman@hergotha.csail.mit.edu Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.17.1/8.17.1) with ESMTPS id 4242iLYO074759 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sun, 3 Mar 2024 21:44:21 -0500 (EST) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.17.1/8.17.1/Submit) id 4242iKmA074758; Sun, 3 Mar 2024 21:44:20 -0500 (EST) (envelope-from wollman) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <26085.13700.20875.520319@hergotha.csail.mit.edu> Date: Sun, 3 Mar 2024 21:44:20 -0500 From: Garrett Wollman To: Rick Macklem Cc: Garrett Wollman , stable@freebsd.org Subject: Re: 13-stable NFS server hang In-Reply-To: References: <26078.50375.679881.64018@hergotha.csail.mit.edu> <26083.64612.717082.366639@hergotha.csail.mit.edu> X-Mailer: VM 8.2.0b under 29.1 (amd64-portbld-freebsd13.2) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.4 (hergotha.csail.mit.edu [0.0.0.0]); Sun, 03 Mar 2024 21:44:21 -0500 (EST) X-Spam-Status: No, score=-4.5 required=5.0 tests=ALL_TRUSTED, HEADER_FROM_DIFFERENT_DOMAINS,NICE_REPLY_A,T_SCC_BODY_TEXT_LINE autolearn=disabled version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on hergotha.csail.mit.edu X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.90 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; FORGED_SENDER(0.30)[wollman@bimajority.org,wollman@hergotha.csail.mit.edu]; R_SPF_ALLOW(-0.20)[+ip6:2001:470:1f06:ccb::2]; DMARC_POLICY_SOFTFAIL(0.10)[bimajority.org : SPF not aligned (relaxed), No valid DKIM,none]; MIME_GOOD(-0.10)[text/plain]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEFALL_USER(0.00)[wollman]; TO_DN_SOME(0.00)[]; TAGGED_RCPT(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; R_DKIM_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_NEQ_ENVFROM(0.00)[wollman@bimajority.org,wollman@hergotha.csail.mit.edu]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[stable@freebsd.org]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_THREE(0.00)[3] X-Rspamd-Queue-Id: 4Tp31g5Sc0z4vPY < said: >> [I wrote:] >> (and so is dirty), this might take several seconds. I've set >> vfs.zfs.dmu_offset_next_sync=0 on the server that was hurting the most >> and am watching to see if we have more freezes. >> >> If this does the trick, then I can delay deploying a new kernel until >> April, after my upcoming vacation. > Interesting. Please let us know how it goes. It's been about 22 hours since I flipped the sysctl and it hasn't happened once yet. Of course I don't know what the users are up to right now, so I'll continue to monitor. This is the script I ended up with to monitor: nfsstat -dW | awk 'BEGIN { n = 0 } (n == 0) && ($12 == 0) { n = n + 1; system("date"); next } (n > 0) && ($12 == 0) { system("date; procstat -k 1184 1198; netstat -n -p tcp"); exit(0) } { n = 0 }' This should (if I haven't botched it) trigger only if two consecutive seconds show no forward progress. -GAWollman From nobody Mon Mar 4 09:42:34 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TpDJD0ZnPz5CHB1 for ; Mon, 4 Mar 2024 09:42:36 +0000 (UTC) (envelope-from des@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TpDJC6vg6z4hLN; Mon, 4 Mar 2024 09:42:35 +0000 (UTC) (envelope-from des@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1709545356; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YWR239Fe3SUQom3+FOn5v20znw093WolKa6OMcJLr4U=; b=HDZoNI8By36UxdHRNXqhq6Zh+lfS2G5XHRdXS+UPIhp34elJMif6pXne8k6AZYYXKdITdm ZtZ0mZcPqNYdpL69m8z+11QrUi5v5sKaDU8o0sX7ezhzvHUmghMxB/8Fr5DMKQ0deKE3LH PiDHoAL5LUdVRQ/IgmRnytKMjds8xuRbomXUr0iPEShQZOnzUWL83uxKmde+KYSGonjWpu mAcLUMFuX7TE212T+7STfMyGMY7UJzPqvIDNmZgenbgzMtw+30mJHmJXw96F7up5uMClTe SN5uy095QPw6btPBWNuihvPAY9kdKTSibee5CemvTOqUfiNqlaT9JFRRexfmIQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1709545356; a=rsa-sha256; cv=none; b=LqW10ixSkCqByqGeBBOPiyqS7S6z/61QF6HnuE/b13Sp8j/YEM5ILX1kEcz8LKmP3+gP2f qgJ4YJFDQRU9oF0WnesS4PYFuavjRBtQTIoe+D1u20P5PxHGwAg88TPZbO2xIWaRWvs04W 1X5ufpCHiMtZX3K5RFG8Jc+EJVvVk4w6K0qj7JK+Y1cpnnRVdgWEr5yDA973xgXYhJPw7b kYff44UPTnPxgwBoh/8U+F5SNTY1WppqyEBwxSEzgrkoTamlnkkecxI9wymX5TT/zZsVtC B9vcE1MuK7luCLkhoQW1TR74tS3SLZMyRLQ2vbQhKXbzY7UujsNSKL++L1U+Bg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1709545356; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YWR239Fe3SUQom3+FOn5v20znw093WolKa6OMcJLr4U=; b=VUo2vjSNa+2PM/DOFn435AY4k/5lM+EEdm7SX7jYweIckTia537yutpSVllD1lrD1yQCnu lCMYwg9nkc7GjYG09UMxcOfwGfXRmEkDjTfsdlKD/A+bCy8qZ15V4WkMxtIUwHQAhYB2+0 94WOTs7fQCPHr1mO+ejr0ULRlcvNjgK4s+fhXqhcwX73EYK9bDWW3bmMLOA3/U6m/fO2Dg cPVkaz92cWCZpUsVTkbjgZm2DwI/9plfyAPW7ojOGLgBhD9JtTcnn0+kmKV5ErgHdEQBC9 AWnJ/Xk4EnvOr49JWp/6Z0UX/ki+11zZPxrDHzktmHdVjScpfZozT7LYQbUnzQ== Received: from ltc.des.dev (163.23.65.37.rev.sfr.net [37.65.23.163]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: des) by smtp.freebsd.org (Postfix) with ESMTPSA id 4TpDJC5kMGzHSD; Mon, 4 Mar 2024 09:42:35 +0000 (UTC) (envelope-from des@freebsd.org) Received: by ltc.des.dev (Postfix, from userid 1001) id 950E91082DB; Mon, 4 Mar 2024 10:42:34 +0100 (CET) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Johannes Totz Cc: FreeBSD Stable Subject: Re: Failed compiling releng/13.2, commit missing? In-Reply-To: <9d1a95e5-2972-4e52-88ef-8d7993275f14@bruelltuete.com> (Johannes Totz's message of "Sun, 3 Mar 2024 17:11:48 +0000") References: <9d1a95e5-2972-4e52-88ef-8d7993275f14@bruelltuete.com> User-Agent: Gnus/5.13 (Gnus v5.13) Date: Mon, 04 Mar 2024 10:42:34 +0100 Message-ID: <86jzmixrhx.fsf@ltc.des.dev> List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Johannes Totz writes: > Releng/13.2 has "import tzdata 2023d" and "...2024a", but no > "...2023c". Is that commit simply missing on that branch? No, releng/13.2 does not have 2023d or 2024a. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@FreeBSD.org From nobody Mon Mar 4 21:11:16 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TpWb35Qgrz5DJ8F for ; Mon, 4 Mar 2024 21:11:27 +0000 (UTC) (envelope-from jo@bruelltuete.com) Received: from email.jo-t.de (seppel.jo-t.de [45.132.244.126]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TpWb33Trpz4vfc; Mon, 4 Mar 2024 21:11:27 +0000 (UTC) (envelope-from jo@bruelltuete.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=bruelltuete.com; s=bruelltuete18a; t=1709586680; bh=12bGTNYCPsguI26udXXEl3SHQc4r4F0WPYBn4BpAYl0=; h=Message-ID:Date:MIME-Version:Subject:To:References:From:From; b=Z/2Zh/FutK6UcfURd/0QT0IxUZ9OqFKtuHEq88LvszGQDI74Z5STVdjZr2ulMBj/W BI3wohKZ0kNAgzV3W+8Ds5p8f2H+aQxwcdV5GmDHtiVS3QgT3dLhvQhcbgErPf1/Zy Ud+hI1iD71+ENHnCEdxiKVwQKE1xrre0m+O2osEP7fFYBFWYDWXvoxuco24C1Weasa 7SICBkyYflTqr63VUD5iR384uvVGW3F+m1cZ1o40HvPphYT5Auti7gT6lpzO852qT0 46PFDGKBzUApLA7pDv2TQUnY8yeb2a0C2lSdAQmJJUTDckmQeq0Y3nJ8Zi0/Q7m04y GzTaSMea1V8wQ== Message-ID: <66308856-35d2-464f-9018-2a7c0a10b8ca@bruelltuete.com> Date: Mon, 4 Mar 2024 21:11:16 +0000 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Subject: Re: Failed compiling releng/13.2, commit missing? To: =?UTF-8?Q?Dag-Erling_Sm=C3=B8rgrav?= Cc: FreeBSD Stable References: <9d1a95e5-2972-4e52-88ef-8d7993275f14@bruelltuete.com> <86jzmixrhx.fsf@ltc.des.dev> Content-Language: en-GB From: Johannes Totz In-Reply-To: <86jzmixrhx.fsf@ltc.des.dev> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:197540, ipnet:45.132.244.0/22, country:DE] X-Rspamd-Queue-Id: 4TpWb33Trpz4vfc On 04/03/2024 09:42, Dag-Erling Smørgrav wrote: > Johannes Totz writes: >> Releng/13.2 has "import tzdata 2023d" and "...2024a", but no >> "...2023c". Is that commit simply missing on that branch? > > No, releng/13.2 does not have 2023d or 2024a. I might be mis-reading https://github.com/freebsd/freebsd-src/commits/releng/13.2/ ... It shows: "contrib/tzdata: import tzdata 2023d" and "contrib/tzdata: import tzdata 2024a". Both came with FreeBSD-EN-24:01.tzdata on the 14th Feb. cheers, Johannes From nobody Mon Mar 4 21:43:16 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TpXHp4F6zz5DLJ3 for ; Mon, 4 Mar 2024 21:43:18 +0000 (UTC) (envelope-from des@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TpXHp3fn9z40kj; Mon, 4 Mar 2024 21:43:18 +0000 (UTC) (envelope-from des@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1709588598; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YLIQ7Ql0m7PrDmz/bk2l6Avm+Xrdti9zFjiD7B4RTP4=; b=yVz6dGCTa9sI3xE36PsHZgGSXtu9hG0sQwBGQz/PmxJM4wn30AOQ8eIfoFbarZyjXzNL37 pALPvNS7JmQXfgyK4MrVU2kju3JTZ7x4dxBuK9/OdPACH9Amfsr3F30RwxQsNRfwg79jt1 cAL/P8701W6uHp3Ja+GHPodiNjm02TujsCGH2pslDZEqljRPsmuHDY8ATEuAgniXyOVym1 1e5yqPblxScZOPYHpjo4eby/90QhT7cirmA6coyzLMW8AL4dZSt3AXqU/oYTWMKlIRnK7j /sESG4rPgoEaAmk4mHcr5H5WeKhUtGzmlZho41COWidc/l8sdz0EDI3DlXdjQw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1709588598; a=rsa-sha256; cv=none; b=F2veqTy5HVpTE7hj/51clou2vrGz3M2cMP4XTdvPhAjuuXlbVlt8ujM4hcgi4mVvZc7Ua7 Be+asubQMHCf4dRagx13EG/X5rorrVCwJZOB8tiQhbxcVDYDyGrvp2MAEP1p9RSquZJcuw 5cp698bWO0CsV7PsXhtM1AQgYnex/g5bAZ0/hbmG95XPKUq7+sTaEtFFZSqnIsgjIoMQcO bTwZzHQAwIY2V3w08F4OajX2YrjvERFBlCgZ718DXK5F+xyd9ZZNUzLTfN7WQwVwqWQFoT uMWADhLSFQjWLYv+bVfCwrWBGhyyQE4TPodlnQ/6aLf66l24O+Iz9Jn+sSOTtQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1709588598; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YLIQ7Ql0m7PrDmz/bk2l6Avm+Xrdti9zFjiD7B4RTP4=; b=lD4blY8cMXdQnWPZ3o6qghRVbChUVRE3nUdnSDTByW6uVb1dPJxLzMNi7AjVT2KsSL/Xq/ H7270UJpbyXuo3CHasK9ypi0IXj09r1l/wPQ8+fj73sTwEjqmhfMZfYRmc240b4FUqD3vO 523OqbdVB+Z1Z8cnxLo5RcfjEQaQrKyvpcnog1+4Mf3LxSaZPlj1OX/mx+iS4PHLDNTuVf voRj5cosr8Ay32CKTcCfWjA+hTePAKzpexZpdXwU7yUTlex23i0jXnW37nUM0X6njm6A2H FZC2ZH16P9D1EbjFSFbgs7R3Acj5d+GzxnPjT2qpFfMPq5FRj1alI/J/UID+mw== Received: from ltc.des.dev (163.23.65.37.rev.sfr.net [37.65.23.163]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: des) by smtp.freebsd.org (Postfix) with ESMTPSA id 4TpXHp2WdrzY9R; Mon, 4 Mar 2024 21:43:18 +0000 (UTC) (envelope-from des@freebsd.org) Received: by ltc.des.dev (Postfix, from userid 1001) id 28081116C84; Mon, 4 Mar 2024 22:43:16 +0100 (CET) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Johannes Totz Cc: FreeBSD Stable Subject: Re: Failed compiling releng/13.2, commit missing? In-Reply-To: <66308856-35d2-464f-9018-2a7c0a10b8ca@bruelltuete.com> (Johannes Totz's message of "Mon, 4 Mar 2024 21:11:16 +0000") References: <9d1a95e5-2972-4e52-88ef-8d7993275f14@bruelltuete.com> <86jzmixrhx.fsf@ltc.des.dev> <66308856-35d2-464f-9018-2a7c0a10b8ca@bruelltuete.com> User-Agent: Gnus/5.13 (Gnus v5.13) Date: Mon, 04 Mar 2024 22:43:16 +0100 Message-ID: <86bk7ty8p7.fsf@ltc.des.dev> List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Johannes Totz writes: > I might be mis-reading > https://github.com/freebsd/freebsd-src/commits/releng/13.2/ ... > > It shows: > "contrib/tzdata: import tzdata 2023d" and > "contrib/tzdata: import tzdata 2024a". > Both came with FreeBSD-EN-24:01.tzdata on the 14th Feb. You're confusing tzdata and tzcode. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@FreeBSD.org From nobody Tue Mar 5 00:55:24 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TpcYW3Wv0z5DcVp for ; Tue, 5 Mar 2024 00:55:27 +0000 (UTC) (envelope-from henrichhartzer@tuta.io) Received: from w1.tutanota.de (w1.tutanota.de [81.3.6.162]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.tutanota.de", Issuer "Sectigo RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TpcYV5Zgrz4Nfv for ; Tue, 5 Mar 2024 00:55:26 +0000 (UTC) (envelope-from henrichhartzer@tuta.io) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=tuta.io header.s=s1 header.b="Z/z26ro2"; dmarc=pass (policy=quarantine) header.from=tuta.io; spf=pass (mx1.freebsd.org: domain of henrichhartzer@tuta.io designates 81.3.6.162 as permitted sender) smtp.mailfrom=henrichhartzer@tuta.io Received: from tutadb.w10.tutanota.de (unknown [192.168.1.10]) by w1.tutanota.de (Postfix) with ESMTP id F009EFBFB78 for ; Tue, 5 Mar 2024 00:55:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1709600124; s=s1; d=tuta.io; h=From:From:To:To:Subject:Subject:Content-Description:Content-ID:Content-Type:Content-Type:Content-Transfer-Encoding:Content-Transfer-Encoding:Cc:Date:Date:In-Reply-To:MIME-Version:MIME-Version:Message-ID:Message-ID:Reply-To:References:Sender; bh=q8MOpZNMhZ4y9ms4Cp6YhE/Pqx/RSVQdkq5ZA2sCZ6w=; b=Z/z26ro2A89g2BeEAUWFtbAbwG2+tU35L/enJE5K0umVFEz2F18yN8QQk7TKmPze d9jjZa9jaqkTqbnuCefeQYna2lKaBS4lFz/H2KKq4FjWIle4/qObbxvNyjvSsG77ptW RQwqzkA2hG3Qqm8SQQfVrcOZTcqgnIY7nYFHm11rCILon39J+BhnbfcYSr+0mjj9p4g 9SM0Epk2Pl7+pJWuHKxjt11/7B/Azxu182y/k73Fcsq2CN3LlBMM9yFft3BxR7lboY4 tC8ee4DWhKssUfHltGZRT58nyYUWPoTY0S/9o8Xw3f2pEO7dFhRnQqKQ3DsteQ8ayIU tPcQpyZjGQ== Date: Tue, 5 Mar 2024 01:55:24 +0100 (CET) From: henrichhartzer@tuta.io To: Freebsd Stable Message-ID: Subject: Running tests for a single program in src/bin, src/usr.bin, etc List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.10 / 15.00]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[tuta.io,quarantine]; R_SPF_ALLOW(-0.20)[+ip4:81.3.6.160/28]; R_DKIM_ALLOW(-0.20)[tuta.io:s=s1]; RWL_MAILSPIKE_VERYGOOD(-0.20)[81.3.6.162:from]; MIME_GOOD(-0.10)[text/plain]; ONCE_RECEIVED(0.10)[]; FROM_NO_DN(0.00)[]; ASN(0.00)[asn:24679, ipnet:81.3.0.0/18, country:DE]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_ONE(0.00)[1]; MISSING_XM_UA(0.00)[]; ARC_NA(0.00)[]; TO_DN_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_TRACE(0.00)[tuta.io:+] X-Rspamd-Queue-Id: 4TpcYV5Zgrz4Nfv Hi everyone, Not sure if this is the best list for this. Maybe hackers@ would be better? I wanted to tinker with utilities in /usr/src/bin and /usrsrc/usr.bin. I noticed that rmdir exits 1 for usage, which is a pet peeve of mine. I updated it to exit 2 and then wanted to alter the tests to ensure it's exiting 2. At this point, I'm unsure of how to run tests for this single component. Is there a way I can be in /usr/src/bin/rmdir and just run a succesful `make test` (or equivalent) without some combination of `make install` + kyua out of /usr/tests? Seems like for self-contained tests, simple testing should be possible. I did find `make check`, but I'm having issues with checkdir sticking around. Thank you! -Henrich From nobody Tue Mar 5 10:13:22 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TprxS0J0Cz5CLyg for ; Tue, 5 Mar 2024 10:13:32 +0000 (UTC) (envelope-from SRS0=s+YQ=KL=klop.ws=ronald-lists@realworks.nl) Received: from smtp-relay-int.realworks.nl (smtp-relay-int.realworks.nl [194.109.157.24]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TprxQ70G4z4Ww9; Tue, 5 Mar 2024 10:13:30 +0000 (UTC) (envelope-from SRS0=s+YQ=KL=klop.ws=ronald-lists@realworks.nl) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=klop.ws header.s=rw2 header.b=kmqx+IIs; dmarc=pass (policy=quarantine) header.from=klop.ws; spf=pass (mx1.freebsd.org: domain of "SRS0=s+YQ=KL=klop.ws=ronald-lists@realworks.nl" designates 194.109.157.24 as permitted sender) smtp.mailfrom="SRS0=s+YQ=KL=klop.ws=ronald-lists@realworks.nl" Date: Tue, 5 Mar 2024 11:13:22 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=klop.ws; s=rw2; t=1709633603; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=/hwZMteHhWYgomWKg792O+mNit3zFd2UT5iaehjeYJU=; b=kmqx+IIsQwfptJlpgGAIOxbSVJV0DAJSEQQOkyVKqxVB2+nH36iOYtMJdLm+DmgievxZP1 4VR/ixqquHIkVbWpRNabvi01NgvRKlPw8vTQJKPnpzT2eyljJ3XENFLQrJIeFDBA5k4jpu LWn2lPC8xYY3hKm5UGBDB/x2JdT29sLUfas28emoQfYToOJTGxNbZ3wPAMw8gJwm6BWDBa KvJLMIlIwHa0Olp8Eh1Hh50/WBI00aEsrRY9e53tYxNOOoEedk0N0JWnrC4vVu8o/7NaO+ 7Ll9kbw+LOa3ogwnq+3Qkg1gUNcEOnp2fXobW2HUQWG7qbNoghSgmDGFrsTn6g== From: Ronald Klop To: Rick Macklem Cc: rmacklem@freebsd.org, Garrett Wollman , stable@freebsd.org Message-ID: <1608503215.4731.1709633602802@localhost> In-Reply-To: References: <1020651467.1592.1709280020993@localhost> Subject: Re: 13-stable NFS server hang List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_4730_243690855.1709633602791" X-Mailer: Realworks (692.43) Importance: Normal X-Priority: 3 (Normal) X-Spamd-Bar: - X-Spamd-Result: default: False [-1.70 / 15.00]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[klop.ws,quarantine]; MID_RHS_NOT_FQDN(0.50)[]; FORGED_SENDER(0.30)[ronald-lists@klop.ws,SRS0=s@realworks.nl]; R_SPF_ALLOW(-0.20)[+ip4:194.109.157.0/24]; R_DKIM_ALLOW(-0.20)[klop.ws:s=rw2]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; ARC_NA(0.00)[]; ASN(0.00)[asn:3265, ipnet:194.109.0.0/16, country:NL]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; TAGGED_RCPT(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; RCPT_COUNT_THREE(0.00)[4]; FROM_NEQ_ENVFROM(0.00)[ronald-lists@klop.ws,SRS0=s@realworks.nl]; FROM_HAS_DN(0.00)[]; RCVD_COUNT_ZERO(0.00)[0]; HAS_X_PRIO_THREE(0.00)[3]; TAGGED_FROM(0.00)[YQ=KL=klop.ws=ronald-lists]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[stable@freebsd.org]; DKIM_TRACE(0.00)[klop.ws:+] X-Rspamd-Queue-Id: 4TprxQ70G4z4Ww9 ------=_Part_4730_243690855.1709633602791 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Van: Rick Macklem Datum: vrijdag, 1 maart 2024 15:23 Aan: Ronald Klop CC: Garrett Wollman , stable@freebsd.org, rmacklem@freebsd.org Onderwerp: Re: 13-stable NFS server hang > > On Fri, Mar 1, 2024 at 12:00AM Ronald Klop wrote: > > > > Interesting read. > > > > Would it be possible to separate locking for admin actions like a client mounting an fs from traffic flowing for file operations? > Well, the NFS server does not really have any concept of a mount. > What I am referring to is the ClientID maintained for NFSv4 mounts, > which all the open/lock/session/layout state hangs off of. > > For most cases, this state information can safely be accessed/modified > via a mutex, but there are three exceptions: > - creating a new ClientID (which is done by the ExchangeID operation) > and typically happens when a NFS client does a mount. > - delegation Recall (which only happens when delegations are enabled) > One of the reasons delegations are not enabled by default on the > FreeBSD server. > - the DestroyClientID which is typically done by a NFS client during dismount. > For these cases, it is just too difficult to do them without sleeping. > As such, there is a sleep lock which the nfsd threads normally acquire shared > when doing NFSv4 operations, but for the above cases the lock is aquired > exclusive. > - I had to give the exclusive lock priority over shared lock > acquisition (it is a > custom locking mechanism with assorted weirdnesses) because without > that someone reported that new mounts took up to 1/2hr to occur. > (The exclusive locker waited for 30min before all the other nfsd threads > were not busy.) > Because of this priority, once a nfsd thread requests the exclusive lock, > all other nfsd threads executing NFSv4 RPCs block after releasing their > shared lock, until the exclusive locker releases the exclusive lock. > > In summary, NFSv4 has certain advantages over NFSv3, but it comes > with a lot of state complexity. It just is not feasible to manipulate all that > state with only mutex locking. > > rick > > > > > Like ongoing file operations could have a read only view/copy of the mount table. Only new operations will have to wait. > > But the mount never needs to wait for ongoing operations before locking the structure. > > > > Just a thought in the morning > > > > Regards, > > Ronald. > > > > Van: Rick Macklem > > Datum: 1 maart 2024 00:31 > > Aan: Garrett Wollman > > CC: stable@freebsd.org, rmacklem@freebsd.org > > Onderwerp: Re: 13-stable NFS server hang > > > > On Wed, Feb 28, 2024 at 4:04PM Rick Macklem wrote: > > > > > > On Tue, Feb 27, 2024 at 9:30PM Garrett Wollman wrote: > > > > > > > > Hi, all, > > > > > > > > We've had some complaints of NFS hanging at unpredictable intervals. > > > > Our NFS servers are running a 13-stable from last December, and > > > > tonight I sat in front of the monitor watching `nfsstat -dW`. I was > > > > able to clearly see that there were periods when NFS activity would > > > > drop *instantly* from 30,000 ops/s to flat zero, which would last > > > > for about 25 seconds before resuming exactly as it was before. > > > > > > > > I wrote a little awk script to watch for this happening and run > > > > `procstat -k` on the nfsd process, and I saw that all but two of the > > > > service threads were idle. The three nfsd threads that had non-idle > > > > kstacks were: > > > > > > > > PID TID COMM TDNAME KSTACK > > > > 997 108481 nfsd nfsd: master mi_switch sleepq_timedwait _sleep nfsv4_lock nfsrvd_dorpc nfssvc_program svc_run_internal svc_run nfsrvd_nfsd nfssvc_nfsd sys_nfssvc amd64_syscall fast_syscall_common > > > > 997 960918 nfsd nfsd: service mi_switch sleepq_timedwait _sleep nfsv4_lock nfsrv_setclient nfsrvd_exchangeid nfsrvd_dorpc nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline > > > > 997 962232 nfsd nfsd: service mi_switch _cv_wait txg_wait_synced_impl txg_wait_synced dmu_offset_next zfs_holey zfs_freebsd_ioctl vn_generic_copy_file_range vop_stdcopy_file_range VOP_COPY_FILE_RANGE vn_copy_file_range nfsrvd_copy_file_range nfsrvd_dorpc nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline > > > > > > > > I'm suspicious of two things: first, the copy_file_range RPC; second, > > > > the "master" nfsd thread is actually servicing an RPC which requires > > > > obtaining a lock. The "master" getting stuck while performing client > > > > RPCs is, I believe, the reason NFS service grinds to a halt when a > > > > client tries to write into a near-full filesystem, so this problem > > > > would be more evidence that the dispatching function should not be > > > > mixed with actual operations. I don't know what the clients are > > > > doing, but is it possible that nfsrvd_copy_file_range is holding a > > > > lock that is needed by one or both of the other two threads? > > > > > > > > Near-term I could change nfsrvd_copy_file_range to just > > > > unconditionally return NFSERR_NOTSUP and force the clients to fall > > > > back, but I figured I would ask if anyone else has seen this. > > > I have attached a little patch that should limit the server's Copy size > > > to vfs.nfsd.maxcopyrange (default of 10Mbytes). > > > Hopefully this makes sure that the Copy does not take too long. > > > > > > You could try this instead of disabling Copy. It would be nice to know if > > > this is suffciient? (If not, I'll probably add a sysctl to disable Copy.) > > I did a quick test without/with this patch,where I copied a 1Gbyte file. > > > > Without this patch, the Copy RPCs mostly replied in just under 1sec > > (which is what the flag requests), but took over 4sec for one of the Copy > > operations. This implies that one Read/Write of 1Mbyte on the server > > took over 3 seconds. > > I noticed the first Copy did over 600Mbytes, but the rest did about 100Mbytes > > each and it was one of these 100Mbyte Copy operations that took over 4sec. > > > > With the patch, there were a lot more Copy RPCs (as expected) of 10Mbytes > > each and they took a consistent 0.25-0.3sec to reply. (This is a test of a local > > mount on an old laptop, so nowhere near a server hardware config.) > > > > So, the patch might be sufficient? > > > > It would be nice to avoid disabling Copy, since it avoids reading the data > > into the client and then writing it back to the server. > > > > I will probably commit both patches (10Mbyte clip of Copy size and > > disabling Copy) to main soon, since I cannot say if clipping the size > > of the Copy will always be sufficient. > > > > Pleas let us know how trying these patches goes, rick > > > > > > > > rick > > > > > > > > > > > -GAWollman > > > > > > > > > > > > ________________________________ > > > > > > > > > > > Hi Rick, You are much more into the NFS code than I am so excuse me if what I'm speaking about does not make sense. I was reading nfsrvd_compound() which calls nfsrvd_copy_file_range() via the nfsrv4_ops2 structure. Nfsrvd_compound holds a lock or refcount on nfsv4rootfs_lock during the whole operation. Which is why nfsrv_setclient() is waiting in this specific case of "NFS server hang". But I don't see what is being modified on the nfsdstate after the IO operation ends. Or why the IO operation itself needs the lock to the nfsdstate. IMHO the in-progress IOs will happen anyway regardless of the nfsdstate. Changes to the nfsdstate during an IO operation would not affect the ongoing IO operation. Wouldn't it be possible to lock the nfsv4rootfs_lock, do checks on or modify the nfsdstate as needed, unlock and then do the IO operation? That would remove a lot of the possible lock contention during (u)mount. Otherwise, if we do modify the nfsdstate after the IO operation, isn't it possible to relock nfsv4rootfs_lock after the IO operation finishes? I hope this makes any sense and thanks for all your work on the NFS code. Regards, Ronald. ------=_Part_4730_243690855.1709633602791 Content-Type: text/html; charset=us-ascii Content-Transfer-Encoding: 7bit

Van: Rick Macklem <rick.macklem@gmail.com>
Datum: vrijdag, 1 maart 2024 15:23
Aan: Ronald Klop <ronald-lists@klop.ws>
CC: Garrett Wollman <wollman@bimajority.org>, stable@freebsd.org, rmacklem@freebsd.org
Onderwerp: Re: 13-stable NFS server hang

On Fri, Mar 1, 2024 at 12:00AM Ronald Klop <ronald-lists@klop.ws> wrote:
>
> Interesting read.
>
>  Would it be possible to separate locking for admin actions like a client mounting an fs from traffic flowing for file operations?
Well, the NFS server does not really have any concept of a mount.
What I am referring to is the ClientID maintained for NFSv4 mounts,
which all the open/lock/session/layout state hangs off of.

For most cases, this state information can safely be accessed/modified
via a mutex, but there are three exceptions:
- creating a new ClientID (which is done by the ExchangeID operation)
  and typically happens when a NFS client does a mount.
- delegation Recall (which only happens when delegations are enabled)
  One of the reasons delegations are not enabled by default on the
FreeBSD server.
- the DestroyClientID which is typically done by a NFS client during dismount.
For these cases, it is just too difficult to do them without sleeping.
As such, there is a sleep lock which the nfsd threads normally acquire shared
when doing NFSv4 operations, but for the above cases the lock is aquired
exclusive.
- I had to give the exclusive lock priority over shared lock
acquisition (it is a
  custom locking mechanism with assorted weirdnesses) because without
  that someone reported that new mounts took up to 1/2hr to occur.
  (The exclusive locker waited for 30min before all the other nfsd threads
   were not busy.)
  Because of this priority, once a nfsd thread requests the exclusive lock,
  all other nfsd threads executing NFSv4 RPCs block after releasing their
  shared lock, until the exclusive locker releases the exclusive lock.

In summary, NFSv4 has certain advantages over NFSv3, but it comes
with a lot of state complexity. It just is not feasible to manipulate all that
state with only mutex locking.

rick

>
> Like ongoing file operations could have a read only view/copy of the mount table. Only new operations will have to wait.
> But the mount never needs to wait for ongoing operations before locking the structure.
>
> Just a thought in the morning
>
> Regards,
> Ronald.
>
> Van: Rick Macklem <rick.macklem@gmail.com>
> Datum: 1 maart 2024 00:31
> Aan: Garrett Wollman <wollman@bimajority.org>
> CC: stable@freebsd.org, rmacklem@freebsd.org
> Onderwerp: Re: 13-stable NFS server hang
>
> On Wed, Feb 28, 2024 at 4:04PM Rick Macklem wrote:
> >
> > On Tue, Feb 27, 2024 at 9:30PM Garrett Wollman wrote:
> > >
> > > Hi, all,
> > >
> > > We've had some complaints of NFS hanging at unpredictable intervals.
> > > Our NFS servers are running a 13-stable from last December, and
> > > tonight I sat in front of the monitor watching `nfsstat -dW`.  I was
> > > able to clearly see that there were periods when NFS activity would
> > > drop *instantly* from 30,000 ops/s to flat zero, which would last
> > > for about 25 seconds before resuming exactly as it was before.
> > >
> > > I wrote a little awk script to watch for this happening and run
> > > `procstat -k` on the nfsd process, and I saw that all but two of the
> > > service threads were idle.  The three nfsd threads that had non-idle
> > > kstacks were:
> > >
> > >   PID    TID COMM                TDNAME              KSTACK
> > >   997 108481 nfsd                nfsd: master        mi_switch sleepq_timedwait _sleep nfsv4_lock nfsrvd_dorpc nfssvc_program svc_run_internal svc_run nfsrvd_nfsd nfssvc_nfsd sys_nfssvc amd64_syscall fast_syscall_common
> > >   997 960918 nfsd                nfsd: service       mi_switch sleepq_timedwait _sleep nfsv4_lock nfsrv_setclient nfsrvd_exchangeid nfsrvd_dorpc nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline
> > >   997 962232 nfsd                nfsd: service       mi_switch _cv_wait txg_wait_synced_impl txg_wait_synced dmu_offset_next zfs_holey zfs_freebsd_ioctl vn_generic_copy_file_range vop_stdcopy_file_range VOP_COPY_FILE_RANGE vn_copy_file_range nfsrvd_copy_file_range nfsrvd_dorpc nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline
> > >
> > > I'm suspicious of two things: first, the copy_file_range RPC; second,
> > > the "master" nfsd thread is actually servicing an RPC which requires
> > > obtaining a lock.  The "master" getting stuck while performing client
> > > RPCs is, I believe, the reason NFS service grinds to a halt when a
> > > client tries to write into a near-full filesystem, so this problem
> > > would be more evidence that the dispatching function should not be
> > > mixed with actual operations.  I don't know what the clients are
> > > doing, but is it possible that nfsrvd_copy_file_range is holding a
> > > lock that is needed by one or both of the other two threads?
> > >
> > > Near-term I could change nfsrvd_copy_file_range to just
> > > unconditionally return NFSERR_NOTSUP and force the clients to fall
> > > back, but I figured I would ask if anyone else has seen this.
> > I have attached a little patch that should limit the server's Copy size
> > to vfs.nfsd.maxcopyrange (default of 10Mbytes).
> > Hopefully this makes sure that the Copy does not take too long.
> >
> > You could try this instead of disabling Copy. It would be nice to know if
> > this is suffciient? (If not, I'll probably add a sysctl to disable Copy.)
> I did a quick test without/with this patch,where I copied a 1Gbyte file.
>
> Without this patch, the Copy RPCs mostly replied in just under 1sec
> (which is what the flag requests), but took over 4sec for one of the Copy
> operations. This implies that one Read/Write of 1Mbyte on the server
> took over 3 seconds.
> I noticed the first Copy did over 600Mbytes, but the rest did about 100Mbytes
> each and it was one of these 100Mbyte Copy operations that took over 4sec.
>
> With the patch, there were a lot more Copy RPCs (as expected) of 10Mbytes
> each and they took a consistent 0.25-0.3sec to reply. (This is a test of a local
> mount on an old laptop, so nowhere near a server hardware config.)
>
> So, the patch might be sufficient?
>
> It would be nice to avoid disabling Copy, since it avoids reading the data
> into the client and then writing it back to the server.
>
> I will probably commit both patches (10Mbyte clip of Copy size and
> disabling Copy) to main soon, since I cannot say if clipping the size
> of the Copy will always be sufficient.
>
> Pleas let us know how trying these patches goes, rick
>
> >
> > rick
> >
> > >
> > > -GAWollman
> > >
> > >
>
> ________________________________
>
>
>
>


Hi Rick,

You are much more into the NFS code than I am so excuse me if what I'm speaking about does not make sense.

I was reading nfsrvd_compound() which calls nfsrvd_copy_file_range() via the nfsrv4_ops2 structure.
Nfsrvd_compound holds a lock or refcount on nfsv4rootfs_lock during the whole operation. Which is why nfsrv_setclient() is waiting in this specific case of "NFS server hang".

But I don't see what is being modified on the nfsdstate after the IO operation ends. Or why the IO operation itself needs the lock to the nfsdstate. IMHO the in-progress IOs will happen anyway regardless of the nfsdstate. Changes to the nfsdstate during an IO operation would not affect the ongoing IO operation.
Wouldn't it be possible to lock the nfsv4rootfs_lock, do checks on or modify the nfsdstate as needed, unlock and then do the IO operation? That would remove a lot of the possible lock contention during (u)mount.
Otherwise, if we do modify the nfsdstate after the IO operation, isn't it possible to relock nfsv4rootfs_lock after the IO operation finishes?

I hope this makes any sense and thanks for all your work on the NFS code.

Regards,
Ronald.


  ------=_Part_4730_243690855.1709633602791-- From nobody Tue Mar 5 14:06:06 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tpy641n12z5Cldl for ; Tue, 5 Mar 2024 14:06:20 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-ot1-f52.google.com (mail-ot1-f52.google.com [209.85.210.52]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tpy636WH7z4v92 for ; Tue, 5 Mar 2024 14:06:19 +0000 (UTC) (envelope-from asomers@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ot1-f52.google.com with SMTP id 46e09a7af769-6e4e952467fso947577a34.2 for ; Tue, 05 Mar 2024 06:06:19 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709647578; x=1710252378; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Uuc0JLyECrMtQjrCBTQzC7TCi04291YrRdN9LIY9wPQ=; b=Hhnd3AxdFPTXjibKTv2O/ETWnp9nH30yJMeDAyMKKR2zKCdIKhslq37Z/C6siad0bi vmlZMQl2Qey5aiSvmICwzYnPIOn5F/DXhBrNcjqWZAbfj6L9ABNu8+SjwFxHCz0HSZ8g TsNoAZj0+KKxmC5+4LTtN5queBa69oQ0JD1P0SAZJWaaPL8gubcNmKIlOgkqqfo2EX6l QTiJxJ7ZC80njg4916T+iqGIw003Q2Q2NsXz/37Abh2GQLbT98d43ydup7kHj9g3y5RM LVY73gc0Mwy0ZxKmL3nrYDpilP1riXvqJA+7XgM/bFulFo1sM+XaHONaMi8aAnBm7xxh igqw== X-Gm-Message-State: AOJu0YyzY5ZRcpg1TNvHFrecylf/2DOo5mY0X3K2Vgi42EMYW7c6e3XM lXqDr2ItmMURkrBNsp+Kk2/39dgtaT/y3LwAoemJAPVekbORrXpHqiu1yAu6Bc3sSA9+AuLvDlX 3xdJBggKvf+Ej7FH7iVMzAH9/yJU= X-Google-Smtp-Source: AGHT+IE4PI48bXG+DYZCOezxxVGIXO4lu9/2vqKnPpRVeWivkgU+/Rpor0aGnrv1dZ5NLnL4e0YAjH88MmRoWPGWqk4= X-Received: by 2002:a05:6808:200d:b0:3c1:d903:fd40 with SMTP id q13-20020a056808200d00b003c1d903fd40mr2343599oiw.55.1709647578561; Tue, 05 Mar 2024 06:06:18 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Alan Somers Date: Tue, 5 Mar 2024 07:06:06 -0700 Message-ID: Subject: Re: Running tests for a single program in src/bin, src/usr.bin, etc To: henrichhartzer@tuta.io Cc: Freebsd Stable Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US] X-Rspamd-Queue-Id: 4Tpy636WH7z4v92 On Mon, Mar 4, 2024 at 5:55=E2=80=AFPM wrote: > > Hi everyone, > > Not sure if this is the best list for this. Maybe hackers@ would be bette= r? > > I wanted to tinker with utilities in /usr/src/bin and /usrsrc/usr.bin. I = noticed that rmdir exits 1 for usage, which is a pet peeve of mine. I updat= ed it to exit 2 and then wanted to alter the tests to ensure it's exiting 2= . > > At this point, I'm unsure of how to run tests for this single component. = Is there a way I can be in /usr/src/bin/rmdir and just run a succesful `mak= e test` (or equivalent) without some combination of `make install` + kyua o= ut of /usr/tests? Seems like for self-contained tests, simple testing shoul= d be possible. > > I did find `make check`, but I'm having issues with checkdir sticking aro= und. > > Thank you! > > -Henrich If you don't like the fact that checkdir disappears, then you'll have to do "make install". But you don't need to run Kyua on the entire test suite. You can do it just on a subdir, like this: sudo kyua test --kyuafile /usr/tests/bin/rmdir/Kyuafile From nobody Tue Mar 5 14:34:32 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tpykv4Zqzz5CnYQ for ; Tue, 5 Mar 2024 14:34:47 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pj1-x102a.google.com (mail-pj1-x102a.google.com [IPv6:2607:f8b0:4864:20::102a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tpykv1YkNz3xrt; Tue, 5 Mar 2024 14:34:47 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pj1-x102a.google.com with SMTP id 98e67ed59e1d1-29a7951c6faso3743866a91.1; Tue, 05 Mar 2024 06:34:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709649286; x=1710254086; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=51K7ZTHw+dT/p6Uu34XAUNjlUpoIOdqoeUSmijwbapw=; b=VcptUoMRhbeFh6W5jEtByzmLqFJmyYtnZdllNfC6sMQUfAAB9aR9MH7Zgl0iG5JEuI wTdq/j5xBo4y6JMX0JCzqlC5nmjI5NWKXIs3MzUIIjUB/JCXcfPhCxOovmYMk0IPD6L/ UL8JPT+02jtP6Es9usGFy8UGl7nzvaMAfpeiRXSQv1FV0pgFII1IMm0AaUvx7rb5yaY+ 0QlcPSU6XSxlEnnSTGyJdR8WKnQnQrwAayb91AKUFurROd3HWlm42b9NRRrXqBf7uPEp 4rWWMLjoGxmR5nmULa2BbA0lvl4Jo2DgUeIzKSpsva50DO57sfftirVyQYwI52F8iQfT +p0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709649286; x=1710254086; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=51K7ZTHw+dT/p6Uu34XAUNjlUpoIOdqoeUSmijwbapw=; b=JUjsI4J7CK32tgddPnMIK8N5ALYdieMYidLWs1G36F0uVxMOY+OOp6n3r9DCx27CkZ Q58sPRz0/yZxXPHfH1hYsPXJNeC69USfijY1CuQBNaL5Az2xO5p3u/5x3csulGFhOXds +6BSKL5n4qIdk0vnXL1v0U2CITNfyYB02dRrQtjsaKpYUPO2WvVDFBdaoYtTYFPW0LCk i/Hs4gJ3fbIk+Q9OnV2WtHkm+O8WoZezMjx+7LRLet5lhf+Wwvu2ob9dOzrjL4z4j2Mt SUhsdSXPq1Cv9ljvkOnlI6fKjPcToi1moKlYnFyO4u3rVWZRoIyY+/LuODFgQSkon3SU 3RQw== X-Forwarded-Encrypted: i=1; AJvYcCWGenFX1ZWyFeINnaLnKpM0aZAl48v90xSOgRZW4gIT9pcG1/SLjxypoOpGdgFC8OGSRp8TaJIJxv+csGIyA5aetJo= X-Gm-Message-State: AOJu0Yxy+ZxTmD7KY43xoLa2CO3VeCBuuZLU56Nj6cmDdpJ5vMl9pT6C nWzLypop7X//a13fQvrEIUjhCHawhsgRLVJZKQQXkMWt5f7iADE3j9nm2x94WZYVGKik194ja3p gdioCNZasyaukaW9IAT6UgDXRvhlrx6g= X-Google-Smtp-Source: AGHT+IF98HUXpWNlkuUlOCQ0xY4iQrEAm0dG+hjQ57ZSW1kHuK/AEHMNgPnDtDtjdXzROvCxMBkkb14DaWy+D0phax8= X-Received: by 2002:a17:90a:394a:b0:29a:729a:be2c with SMTP id n10-20020a17090a394a00b0029a729abe2cmr9281174pjf.21.1709649285577; Tue, 05 Mar 2024 06:34:45 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: <1020651467.1592.1709280020993@localhost> <1608503215.4731.1709633602802@localhost> In-Reply-To: <1608503215.4731.1709633602802@localhost> From: Rick Macklem Date: Tue, 5 Mar 2024 06:34:32 -0800 Message-ID: Subject: Re: 13-stable NFS server hang To: Ronald Klop Cc: rmacklem@freebsd.org, Garrett Wollman , stable@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4Tpykv1YkNz3xrt On Tue, Mar 5, 2024 at 2:13=E2=80=AFAM Ronald Klop w= rote: > > > Van: Rick Macklem > Datum: vrijdag, 1 maart 2024 15:23 > Aan: Ronald Klop > CC: Garrett Wollman , stable@freebsd.org, rmackle= m@freebsd.org > Onderwerp: Re: 13-stable NFS server hang > > On Fri, Mar 1, 2024 at 12:00AM Ronald Klop wrote: > > > > Interesting read. > > > > Would it be possible to separate locking for admin actions like a clie= nt mounting an fs from traffic flowing for file operations? > Well, the NFS server does not really have any concept of a mount. > What I am referring to is the ClientID maintained for NFSv4 mounts, > which all the open/lock/session/layout state hangs off of. > > For most cases, this state information can safely be accessed/modified > via a mutex, but there are three exceptions: > - creating a new ClientID (which is done by the ExchangeID operation) > and typically happens when a NFS client does a mount. > - delegation Recall (which only happens when delegations are enabled) > One of the reasons delegations are not enabled by default on the > FreeBSD server. > - the DestroyClientID which is typically done by a NFS client during dism= ount. > For these cases, it is just too difficult to do them without sleeping. > As such, there is a sleep lock which the nfsd threads normally acquire sh= ared > when doing NFSv4 operations, but for the above cases the lock is aquired > exclusive. > - I had to give the exclusive lock priority over shared lock > acquisition (it is a > custom locking mechanism with assorted weirdnesses) because without > that someone reported that new mounts took up to 1/2hr to occur. > (The exclusive locker waited for 30min before all the other nfsd thread= s > were not busy.) > Because of this priority, once a nfsd thread requests the exclusive loc= k, > all other nfsd threads executing NFSv4 RPCs block after releasing their > shared lock, until the exclusive locker releases the exclusive lock. > > In summary, NFSv4 has certain advantages over NFSv3, but it comes > with a lot of state complexity. It just is not feasible to manipulate all= that > state with only mutex locking. > > rick > > > > > Like ongoing file operations could have a read only view/copy of the mo= unt table. Only new operations will have to wait. > > But the mount never needs to wait for ongoing operations before locking= the structure. > > > > Just a thought in the morning > > > > Regards, > > Ronald. > > > > Van: Rick Macklem > > Datum: 1 maart 2024 00:31 > > Aan: Garrett Wollman > > CC: stable@freebsd.org, rmacklem@freebsd.org > > Onderwerp: Re: 13-stable NFS server hang > > > > On Wed, Feb 28, 2024 at 4:04PM Rick Macklem wrote: > > > > > > On Tue, Feb 27, 2024 at 9:30PM Garrett Wollman wrote: > > > > > > > > Hi, all, > > > > > > > > We've had some complaints of NFS hanging at unpredictable intervals= . > > > > Our NFS servers are running a 13-stable from last December, and > > > > tonight I sat in front of the monitor watching `nfsstat -dW`. I wa= s > > > > able to clearly see that there were periods when NFS activity would > > > > drop *instantly* from 30,000 ops/s to flat zero, which would last > > > > for about 25 seconds before resuming exactly as it was before. > > > > > > > > I wrote a little awk script to watch for this happening and run > > > > `procstat -k` on the nfsd process, and I saw that all but two of th= e > > > > service threads were idle. The three nfsd threads that had non-idl= e > > > > kstacks were: > > > > > > > > PID TID COMM TDNAME KSTACK > > > > 997 108481 nfsd nfsd: master mi_switch slee= pq_timedwait _sleep nfsv4_lock nfsrvd_dorpc nfssvc_program svc_run_internal= svc_run nfsrvd_nfsd nfssvc_nfsd sys_nfssvc amd64_syscall fast_syscall_comm= on > > > > 997 960918 nfsd nfsd: service mi_switch slee= pq_timedwait _sleep nfsv4_lock nfsrv_setclient nfsrvd_exchangeid nfsrvd_dor= pc nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoli= ne > > > > 997 962232 nfsd nfsd: service mi_switch _cv_= wait txg_wait_synced_impl txg_wait_synced dmu_offset_next zfs_holey zfs_fre= ebsd_ioctl vn_generic_copy_file_range vop_stdcopy_file_range VOP_COPY_FILE_= RANGE vn_copy_file_range nfsrvd_copy_file_range nfsrvd_dorpc nfssvc_program= svc_run_internal svc_thread_start fork_exit fork_trampoline > > > > > > > > I'm suspicious of two things: first, the copy_file_range RPC; secon= d, > > > > the "master" nfsd thread is actually servicing an RPC which require= s > > > > obtaining a lock. The "master" getting stuck while performing clie= nt > > > > RPCs is, I believe, the reason NFS service grinds to a halt when a > > > > client tries to write into a near-full filesystem, so this problem > > > > would be more evidence that the dispatching function should not be > > > > mixed with actual operations. I don't know what the clients are > > > > doing, but is it possible that nfsrvd_copy_file_range is holding a > > > > lock that is needed by one or both of the other two threads? > > > > > > > > Near-term I could change nfsrvd_copy_file_range to just > > > > unconditionally return NFSERR_NOTSUP and force the clients to fall > > > > back, but I figured I would ask if anyone else has seen this. > > > I have attached a little patch that should limit the server's Copy si= ze > > > to vfs.nfsd.maxcopyrange (default of 10Mbytes). > > > Hopefully this makes sure that the Copy does not take too long. > > > > > > You could try this instead of disabling Copy. It would be nice to kno= w if > > > this is suffciient? (If not, I'll probably add a sysctl to disable Co= py.) > > I did a quick test without/with this patch,where I copied a 1Gbyte file= . > > > > Without this patch, the Copy RPCs mostly replied in just under 1sec > > (which is what the flag requests), but took over 4sec for one of the Co= py > > operations. This implies that one Read/Write of 1Mbyte on the server > > took over 3 seconds. > > I noticed the first Copy did over 600Mbytes, but the rest did about 100= Mbytes > > each and it was one of these 100Mbyte Copy operations that took over 4s= ec. > > > > With the patch, there were a lot more Copy RPCs (as expected) of 10Mbyt= es > > each and they took a consistent 0.25-0.3sec to reply. (This is a test o= f a local > > mount on an old laptop, so nowhere near a server hardware config.) > > > > So, the patch might be sufficient? > > > > It would be nice to avoid disabling Copy, since it avoids reading the d= ata > > into the client and then writing it back to the server. > > > > I will probably commit both patches (10Mbyte clip of Copy size and > > disabling Copy) to main soon, since I cannot say if clipping the size > > of the Copy will always be sufficient. > > > > Pleas let us know how trying these patches goes, rick > > > > > > > > rick > > > > > > > > > > > -GAWollman > > > > > > > > > > > > ________________________________ > > > > > > > > > ________________________________ > > > > Hi Rick, > > You are much more into the NFS code than I am so excuse me if what I'm sp= eaking about does not make sense. > > I was reading nfsrvd_compound() which calls nfsrvd_copy_file_range() via = the nfsrv4_ops2 structure. > Nfsrvd_compound holds a lock or refcount on nfsv4rootfs_lock during the w= hole operation. Which is why nfsrv_setclient() is waiting in this specific = case of "NFS server hang". > > But I don't see what is being modified on the nfsdstate after the IO oper= ation ends. Or why the IO operation itself needs the lock to the nfsdstate.= IMHO the in-progress IOs will happen anyway regardless of the nfsdstate. C= hanges to the nfsdstate during an IO operation would not affect the ongoing= IO operation. > Wouldn't it be possible to lock the nfsv4rootfs_lock, do checks on or mod= ify the nfsdstate as needed, unlock and then do the IO operation? That woul= d remove a lot of the possible lock contention during (u)mount. > Otherwise, if we do modify the nfsdstate after the IO operation, isn't it= possible to relock nfsv4rootfs_lock after the IO operation finishes? Well, there are a couple of reasons. Every implementation has design tradeo= ffs: 1 - A NFSv4 RPC is a compound, which can be a pretty arbitrary list of operations. As such, the NFSv4 server does not know if an open/byte range lock is coming after the operation it is currently performing, since the implementation does not pre-parse the entire compound. (I had a discussion w.r.t. pre-parsing with one of the main Linux knfsd server maintainers and he noted that he was not aware of any extant server that did pre-parse the compound. Although it would be useful for the server to have the ordered list of operations before commencing the RPC, we both agreed it was too hard to implement. --> It could possibly unlock/relock later, but see #2. Also, if relocking took a long time, it would result in the compound RPC taking too long (see below). 2 - For NFSv4.1/4.2 almost all RPCs are handled by a session. One non-arbit= rary part of almost all NFSv4.1/4.2 RPCs is that the Sequence operation (the one that handles the session) must come first. Session(s) are associated with the ClientID, which means the ClientID and the session must not go away while the compound RPC is in progress. - This is ensured by the shared lock on the ClientID (that nfsv4rootfh_lock). Since 99.99% of operations can be done with the shared lock, I do not think there is a lot of contention. Although there is nothing wired down in the RFCs, there is an understanding in the NFSv4 community that a server should reply to an RPC in a reasonable time. Typically assumed to be 1-2sec. If the server does this, then a delay= for the rare case of a new ClientID shouldn't be a big problem? (The is also delegation recall, which is one reason why delegations are not enabled by default.) Btw, the RFC does define an asynchronous Copy, where the operation replies as soon as the copy is started and the server notifies the client of comple= tion later. I have not implemented this, because it introduces complexities that I do not want to deal with. For example, what happens when the server crashes/reboots while the copy is in progress? The file is left in a non-deterministic state, depending on= what the client does when it does not receive the completion notify. rick > > I hope this makes any sense and thanks for all your work on the NFS code. > > Regards, > Ronald. > > > From nobody Tue Mar 5 14:43:05 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tpywn2z98z5CpZs for ; Tue, 5 Mar 2024 14:43:21 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pg1-x535.google.com (mail-pg1-x535.google.com [IPv6:2607:f8b0:4864:20::535]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tpywm0dbZz41KF; Tue, 5 Mar 2024 14:43:20 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b="Sk/EbXI5"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2607:f8b0:4864:20::535 as permitted sender) smtp.mailfrom=rick.macklem@gmail.com Received: by mail-pg1-x535.google.com with SMTP id 41be03b00d2f7-5ce9555d42eso4935268a12.2; Tue, 05 Mar 2024 06:43:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709649798; x=1710254598; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=icQszXU0mgP1EtQS7+lcBBVHXyOvRohV/GwQmFYjnSI=; b=Sk/EbXI5nVbmcgpHrEl5lEB22k7RWkohzHBMnPydtXvKTSDj44pdaQjnZh9sl/iqFN 3veooS6Zo9v7yljxx2I7yB0We3BsEffuIPxiCk+2RvLjlAiPr+XcCAazz8oZf6Hp8j3q udQNeUUvPXz+/76nf9cNpbuuxk3xEBOZojTf1N6ARH1nvf6kWYKa1JzFsH7GP/u98L5n m0rgNmTvGSu7OrdAWNpDj6DZR07qRvqc+4KaUhDACvsMEzMoYBZja2oWo66y6+tWbRuG HZa/TTW4yjOAXfzWHwCRPoVKDDTME0AWmnob6FlvPMKWlDI5YkOSghTcCIz900WqRZTF TTjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709649798; x=1710254598; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=icQszXU0mgP1EtQS7+lcBBVHXyOvRohV/GwQmFYjnSI=; b=u+Mm0nyOZlCbp2cu22uk8wrkmjFB5VWXFpYtAlGTAYKonfueIBuIIXp+kcMtyA9bL1 9RcrwR0h4MdI5zJ7C/0BE2orCO6xsBzQ+r959wJnRWs8jaQ+ghGAyqRetW97qlurF1aw qE8HUJhOFbKe0GIz2o+A3imxz9LRORFtCiheT4LB47BqnHQxnMV7XNw1k45/JxzNavF0 lj6avX8JUnFEju1ks4AnCvHhLgtbykYk01OXisl3ysKw0DdBUFE3QMYInOlAEnDewOzX 7ubc+BBA8hdXclepNE/QQaOSWajGqNG/wdvvzJ9oPX3dRpbuv7hXjON8E2bXpQOwugl4 bS0g== X-Forwarded-Encrypted: i=1; AJvYcCWMdUMMUgtB0e+aIsZrwlCAC5FSlQVBVQqRZvkmPQsdASGp2h/qVTGaOTiNWaGtYpsPegUUuzDH6y4J3fTbSERrJNY= X-Gm-Message-State: AOJu0Yx7BwonwQkBCMJUDWkSh4L1uomwQSFQb3Ya7H85vGC//49RF+K8 irJf0l+lSty9DzJ5NYdAOy7Kv+VKVHBX8+ol9MhAspAdBdRwS0upWMp3oE7/SevIq25JakBWnbl LBFu2PRgL0O1VBdi2paLyRAEJ8w== X-Google-Smtp-Source: AGHT+IHW8S2CAo2TndR/iqOuwsGJy05sjML8HcokGwseUaHjbxPgQ7FYq+aZiqHo1GHJ9skFHL8hj1tq5Hbul36VvMo= X-Received: by 2002:a17:90b:20c:b0:29a:9f04:fbcd with SMTP id fy12-20020a17090b020c00b0029a9f04fbcdmr11089171pjb.20.1709649797898; Tue, 05 Mar 2024 06:43:17 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: <1020651467.1592.1709280020993@localhost> <1608503215.4731.1709633602802@localhost> In-Reply-To: From: Rick Macklem Date: Tue, 5 Mar 2024 06:43:05 -0800 Message-ID: Subject: Re: 13-stable NFS server hang To: Ronald Klop Cc: rmacklem@freebsd.org, Garrett Wollman , stable@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.38 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.38)[-0.378]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; MISSING_XM_UA(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MLMMJ_DEST(0.00)[stable@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::535:from]; FREEMAIL_ENVFROM(0.00)[gmail.com] X-Rspamd-Queue-Id: 4Tpywm0dbZz41KF On Tue, Mar 5, 2024 at 6:34=E2=80=AFAM Rick Macklem wrote: > > On Tue, Mar 5, 2024 at 2:13=E2=80=AFAM Ronald Klop = wrote: > > > > > > Van: Rick Macklem > > Datum: vrijdag, 1 maart 2024 15:23 > > Aan: Ronald Klop > > CC: Garrett Wollman , stable@freebsd.org, rmack= lem@freebsd.org > > Onderwerp: Re: 13-stable NFS server hang > > > > On Fri, Mar 1, 2024 at 12:00AM Ronald Klop wrote= : > > > > > > Interesting read. > > > > > > Would it be possible to separate locking for admin actions like a cl= ient mounting an fs from traffic flowing for file operations? > > Well, the NFS server does not really have any concept of a mount. > > What I am referring to is the ClientID maintained for NFSv4 mounts, > > which all the open/lock/session/layout state hangs off of. > > > > For most cases, this state information can safely be accessed/modified > > via a mutex, but there are three exceptions: > > - creating a new ClientID (which is done by the ExchangeID operation) > > and typically happens when a NFS client does a mount. > > - delegation Recall (which only happens when delegations are enabled) > > One of the reasons delegations are not enabled by default on the > > FreeBSD server. > > - the DestroyClientID which is typically done by a NFS client during di= smount. > > For these cases, it is just too difficult to do them without sleeping. > > As such, there is a sleep lock which the nfsd threads normally acquire = shared > > when doing NFSv4 operations, but for the above cases the lock is aquire= d > > exclusive. > > - I had to give the exclusive lock priority over shared lock > > acquisition (it is a > > custom locking mechanism with assorted weirdnesses) because without > > that someone reported that new mounts took up to 1/2hr to occur. > > (The exclusive locker waited for 30min before all the other nfsd thre= ads > > were not busy.) > > Because of this priority, once a nfsd thread requests the exclusive l= ock, > > all other nfsd threads executing NFSv4 RPCs block after releasing the= ir > > shared lock, until the exclusive locker releases the exclusive lock. > > > > In summary, NFSv4 has certain advantages over NFSv3, but it comes > > with a lot of state complexity. It just is not feasible to manipulate a= ll that > > state with only mutex locking. > > > > rick > > > > > > > > Like ongoing file operations could have a read only view/copy of the = mount table. Only new operations will have to wait. > > > But the mount never needs to wait for ongoing operations before locki= ng the structure. > > > > > > Just a thought in the morning > > > > > > Regards, > > > Ronald. > > > > > > Van: Rick Macklem > > > Datum: 1 maart 2024 00:31 > > > Aan: Garrett Wollman > > > CC: stable@freebsd.org, rmacklem@freebsd.org > > > Onderwerp: Re: 13-stable NFS server hang > > > > > > On Wed, Feb 28, 2024 at 4:04PM Rick Macklem wrote: > > > > > > > > On Tue, Feb 27, 2024 at 9:30PM Garrett Wollman wrote: > > > > > > > > > > Hi, all, > > > > > > > > > > We've had some complaints of NFS hanging at unpredictable interva= ls. > > > > > Our NFS servers are running a 13-stable from last December, and > > > > > tonight I sat in front of the monitor watching `nfsstat -dW`. I = was > > > > > able to clearly see that there were periods when NFS activity wou= ld > > > > > drop *instantly* from 30,000 ops/s to flat zero, which would last > > > > > for about 25 seconds before resuming exactly as it was before. > > > > > > > > > > I wrote a little awk script to watch for this happening and run > > > > > `procstat -k` on the nfsd process, and I saw that all but two of = the > > > > > service threads were idle. The three nfsd threads that had non-i= dle > > > > > kstacks were: > > > > > > > > > > PID TID COMM TDNAME KSTACK > > > > > 997 108481 nfsd nfsd: master mi_switch sl= eepq_timedwait _sleep nfsv4_lock nfsrvd_dorpc nfssvc_program svc_run_intern= al svc_run nfsrvd_nfsd nfssvc_nfsd sys_nfssvc amd64_syscall fast_syscall_co= mmon > > > > > 997 960918 nfsd nfsd: service mi_switch sl= eepq_timedwait _sleep nfsv4_lock nfsrv_setclient nfsrvd_exchangeid nfsrvd_d= orpc nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampo= line > > > > > 997 962232 nfsd nfsd: service mi_switch _c= v_wait txg_wait_synced_impl txg_wait_synced dmu_offset_next zfs_holey zfs_f= reebsd_ioctl vn_generic_copy_file_range vop_stdcopy_file_range VOP_COPY_FIL= E_RANGE vn_copy_file_range nfsrvd_copy_file_range nfsrvd_dorpc nfssvc_progr= am svc_run_internal svc_thread_start fork_exit fork_trampoline > > > > > > > > > > I'm suspicious of two things: first, the copy_file_range RPC; sec= ond, > > > > > the "master" nfsd thread is actually servicing an RPC which requi= res > > > > > obtaining a lock. The "master" getting stuck while performing cl= ient > > > > > RPCs is, I believe, the reason NFS service grinds to a halt when = a > > > > > client tries to write into a near-full filesystem, so this proble= m > > > > > would be more evidence that the dispatching function should not b= e > > > > > mixed with actual operations. I don't know what the clients are > > > > > doing, but is it possible that nfsrvd_copy_file_range is holding = a > > > > > lock that is needed by one or both of the other two threads? > > > > > > > > > > Near-term I could change nfsrvd_copy_file_range to just > > > > > unconditionally return NFSERR_NOTSUP and force the clients to fal= l > > > > > back, but I figured I would ask if anyone else has seen this. > > > > I have attached a little patch that should limit the server's Copy = size > > > > to vfs.nfsd.maxcopyrange (default of 10Mbytes). > > > > Hopefully this makes sure that the Copy does not take too long. > > > > > > > > You could try this instead of disabling Copy. It would be nice to k= now if > > > > this is suffciient? (If not, I'll probably add a sysctl to disable = Copy.) > > > I did a quick test without/with this patch,where I copied a 1Gbyte fi= le. > > > > > > Without this patch, the Copy RPCs mostly replied in just under 1sec > > > (which is what the flag requests), but took over 4sec for one of the = Copy > > > operations. This implies that one Read/Write of 1Mbyte on the server > > > took over 3 seconds. > > > I noticed the first Copy did over 600Mbytes, but the rest did about 1= 00Mbytes > > > each and it was one of these 100Mbyte Copy operations that took over = 4sec. > > > > > > With the patch, there were a lot more Copy RPCs (as expected) of 10Mb= ytes > > > each and they took a consistent 0.25-0.3sec to reply. (This is a test= of a local > > > mount on an old laptop, so nowhere near a server hardware config.) > > > > > > So, the patch might be sufficient? > > > > > > It would be nice to avoid disabling Copy, since it avoids reading the= data > > > into the client and then writing it back to the server. > > > > > > I will probably commit both patches (10Mbyte clip of Copy size and > > > disabling Copy) to main soon, since I cannot say if clipping the size > > > of the Copy will always be sufficient. > > > > > > Pleas let us know how trying these patches goes, rick > > > > > > > > > > > rick > > > > > > > > > > > > > > -GAWollman > > > > > > > > > > > > > > > > ________________________________ > > > > > > > > > > > > > > ________________________________ > > > > > > > > Hi Rick, > > > > You are much more into the NFS code than I am so excuse me if what I'm = speaking about does not make sense. > > > > I was reading nfsrvd_compound() which calls nfsrvd_copy_file_range() vi= a the nfsrv4_ops2 structure. > > Nfsrvd_compound holds a lock or refcount on nfsv4rootfs_lock during the= whole operation. Which is why nfsrv_setclient() is waiting in this specifi= c case of "NFS server hang". > > > > But I don't see what is being modified on the nfsdstate after the IO op= eration ends. Or why the IO operation itself needs the lock to the nfsdstat= e. IMHO the in-progress IOs will happen anyway regardless of the nfsdstate.= Changes to the nfsdstate during an IO operation would not affect the ongoi= ng IO operation. > > Wouldn't it be possible to lock the nfsv4rootfs_lock, do checks on or m= odify the nfsdstate as needed, unlock and then do the IO operation? That wo= uld remove a lot of the possible lock contention during (u)mount. > > Otherwise, if we do modify the nfsdstate after the IO operation, isn't = it possible to relock nfsv4rootfs_lock after the IO operation finishes? > Well, there are a couple of reasons. Every implementation has design trad= eoffs: > 1 - A NFSv4 RPC is a compound, which can be a pretty arbitrary list of > operations. > As such, the NFSv4 server does not know if an open/byte range > lock is coming > after the operation it is currently performing, since the > implementation does not > pre-parse the entire compound. (I had a discussion w.r.t. > pre-parsing with one of > the main Linux knfsd server maintainers and he noted that he was > not aware of > any extant server that did pre-parse the compound. Although it > would be useful > for the server to have the ordered list of operations before > commencing the RPC, > we both agreed it was too hard to implement. > --> It could possibly unlock/relock later, but see #2. Also, if > relocking took a long time, > it would result in the compound RPC taking too long (see below)= . > 2 - For NFSv4.1/4.2 almost all RPCs are handled by a session. One non-arb= itrary > part of almost all NFSv4.1/4.2 RPCs is that the Sequence > operation (the one that > handles the session) must come first. > Session(s) are associated with the ClientID, which means the > ClientID and the > session must not go away while the compound RPC is in progress. > - This is ensured by the shared lock on the ClientID (that > nfsv4rootfh_lock). > Since 99.99% of operations can be done with the shared lock, I do not thi= nk > there is a lot of contention. > > Although there is nothing wired down in the RFCs, there is an understandi= ng > in the NFSv4 community that a server should reply to an RPC in a reasonab= le > time. Typically assumed to be 1-2sec. If the server does this, then a del= ay for > the rare case of a new ClientID shouldn't be a big problem? > (The is also delegation recall, which is one reason why delegations > are not enabled > by default.) > > Btw, the RFC does define an asynchronous Copy, where the operation replie= s > as soon as the copy is started and the server notifies the client of comp= letion > later. I have not implemented this, because it introduces complexities th= at > I do not want to deal with. > For example, what happens when the server crashes/reboots while the copy > is in progress? The file is left in a non-deterministic state, depending = on what > the client does when it does not receive the completion notify. > Oh, I should also note that the "shared lock" is actually called a reference count in the code and is there to ensure that the ClientID/Session does not go aw= ay during execution of the compound. The problem in this case (which I should revisit) was that I could not figu= re out how to safely add a new ClientID while other nfsd threads were in progr= ess performing other RPCs. Due to retries etc, there might be another RPC in progress using the ClientID. One thing to note here is that the identity of the ClientID is not known until the Sequence operation has been performed. (And there is cruft for NFSv4.0, since it does not have a Sequence operation.) As such, the RPC must be in progress before it is known. > rick > > > > I hope this makes any sense and thanks for all your work on the NFS cod= e. > > > > Regards, > > Ronald. > > > > > > From nobody Wed Mar 6 11:46:16 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TqVyB6L9rz5CQCR for ; Wed, 6 Mar 2024 11:46:26 +0000 (UTC) (envelope-from SRS0=MB1c=KM=klop.ws=ronald-lists@realworks.nl) Received: from smtp-relay-int-backup.realworks.nl (smtp-relay-int-backup.realworks.nl [87.255.56.188]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TqVy90mW2z49Vp; Wed, 6 Mar 2024 11:46:24 +0000 (UTC) (envelope-from SRS0=MB1c=KM=klop.ws=ronald-lists@realworks.nl) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=klop.ws header.s=rw2 header.b=yNdYzskk; dmarc=pass (policy=quarantine) header.from=klop.ws; spf=pass (mx1.freebsd.org: domain of "SRS0=MB1c=KM=klop.ws=ronald-lists@realworks.nl" designates 87.255.56.188 as permitted sender) smtp.mailfrom="SRS0=MB1c=KM=klop.ws=ronald-lists@realworks.nl" Received: from smtp-relay-int.realworks.nl (rwvirtual374.colo.realworks.nl [10.0.10.74]) by mailrelayint1.colo2.realworks.nl (Postfix) with ESMTP id 4TqVy06XcPz9W; Wed, 6 Mar 2024 12:46:16 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=klop.ws; s=rw2; t=1709725576; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=rXQnYE35H/LkFoF3EryjtCYcLX93bpFBfL+qxY3nmUE=; b=yNdYzskke+P7DCbnC+m2+CZ512DD79q+rACdyDIazWLkaUNkXlicUMx8WR9+WOXmnI+KfW EzwAX14Sf8r+e6CVfqEPx4D5O9joVwwvZ1x9MarxSBDZ5CmYZVucZIsD+FXKerGobzwFea KmJUDgc45K70k+KUYWZaocYsjLwB1DsOZfDHx7Q8arXHmwPqwwaiy823o2QvR3xOxGpIsB ydj2T4rrmbARWd4d8gq6NDtK4Ts9fxhHMBwNFTyuXSJv+u5aeyLAfpiEyuhOb+eVb+mJSe 3MDt/gXjx1HARbTOlqc20tgcpolCviJXIzFfEd/8S/8TzrJp8EACGY9HtnQL+A== Received: from rwvirtual374.colo.realworks.nl (localhost [127.0.0.1]) by rwvirtual374.colo.realworks.nl (Postfix) with ESMTP id AD230C0B6E; Wed, 6 Mar 2024 12:46:16 +0100 (CET) Date: Wed, 6 Mar 2024 12:46:16 +0100 (CET) From: Ronald Klop To: Rick Macklem Cc: rmacklem@freebsd.org, Garrett Wollman , stable@freebsd.org Message-ID: <1277770972.4770.1709725576695@localhost> In-Reply-To: References: <1020651467.1592.1709280020993@localhost> <1608503215.4731.1709633602802@localhost> Subject: Re: 13-stable NFS server hang List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_4769_153177038.1709725576674" X-Mailer: Realworks (692.49) X-Originating-Host: from (89-20-164-210.static.ef-service.nl [89.20.164.210]) by rwvirtual374 [10.0.10.74] with HTTP; Wed, 06 Mar 2024 12:46:16 +0100 Importance: Normal X-Priority: 3 (Normal) X-Originating-User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:123.0) Gecko/20100101 Firefox/123.0 X-Spamd-Bar: - X-Spamd-Result: default: False [-1.69 / 15.00]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.987]; DMARC_POLICY_ALLOW(-0.50)[klop.ws,quarantine]; MID_RHS_NOT_FQDN(0.50)[]; FORGED_SENDER(0.30)[ronald-lists@klop.ws,SRS0=MB1c=KM=klop.ws=ronald-lists@realworks.nl]; R_SPF_ALLOW(-0.20)[+ip4:87.255.56.128/26]; R_DKIM_ALLOW(-0.20)[klop.ws:s=rw2]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; ARC_NA(0.00)[]; ASN(0.00)[asn:38930, ipnet:87.255.32.0/19, country:NL]; TO_DN_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; HAS_X_PRIO_THREE(0.00)[3]; TAGGED_RCPT(0.00)[]; FROM_NEQ_ENVFROM(0.00)[ronald-lists@klop.ws,SRS0=MB1c=KM=klop.ws=ronald-lists@realworks.nl]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[stable@freebsd.org]; FREEMAIL_TO(0.00)[gmail.com]; RCPT_COUNT_THREE(0.00)[4]; DKIM_TRACE(0.00)[klop.ws:+] X-Rspamd-Queue-Id: 4TqVy90mW2z49Vp ------=_Part_4769_153177038.1709725576674 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Van: Rick Macklem Datum: dinsdag, 5 maart 2024 15:43 Aan: Ronald Klop CC: rmacklem@freebsd.org, Garrett Wollman , stable@freebsd.org Onderwerp: Re: 13-stable NFS server hang > > On Tue, Mar 5, 2024 at 6:34AM Rick Macklem wrote: > > > > On Tue, Mar 5, 2024 at 2:13AM Ronald Klop wrote: > > > > > > > > > Van: Rick Macklem > > > Datum: vrijdag, 1 maart 2024 15:23 > > > Aan: Ronald Klop > > > CC: Garrett Wollman , stable@freebsd.org, rmacklem@freebsd.org > > > Onderwerp: Re: 13-stable NFS server hang > > > > > > On Fri, Mar 1, 2024 at 12:00AM Ronald Klop wrote: > > > > > > > > Interesting read. > > > > > > > > Would it be possible to separate locking for admin actions like a client mounting an fs from traffic flowing for file operations? > > > Well, the NFS server does not really have any concept of a mount. > > > What I am referring to is the ClientID maintained for NFSv4 mounts, > > > which all the open/lock/session/layout state hangs off of. > > > > > > For most cases, this state information can safely be accessed/modified > > > via a mutex, but there are three exceptions: > > > - creating a new ClientID (which is done by the ExchangeID operation) > > > and typically happens when a NFS client does a mount. > > > - delegation Recall (which only happens when delegations are enabled) > > > One of the reasons delegations are not enabled by default on the > > > FreeBSD server. > > > - the DestroyClientID which is typically done by a NFS client during dismount. > > > For these cases, it is just too difficult to do them without sleeping. > > > As such, there is a sleep lock which the nfsd threads normally acquire shared > > > when doing NFSv4 operations, but for the above cases the lock is aquired > > > exclusive. > > > - I had to give the exclusive lock priority over shared lock > > > acquisition (it is a > > > custom locking mechanism with assorted weirdnesses) because without > > > that someone reported that new mounts took up to 1/2hr to occur. > > > (The exclusive locker waited for 30min before all the other nfsd threads > > > were not busy.) > > > Because of this priority, once a nfsd thread requests the exclusive lock, > > > all other nfsd threads executing NFSv4 RPCs block after releasing their > > > shared lock, until the exclusive locker releases the exclusive lock. > > > > > > In summary, NFSv4 has certain advantages over NFSv3, but it comes > > > with a lot of state complexity. It just is not feasible to manipulate all that > > > state with only mutex locking. > > > > > > rick > > > > > > > > > > > Like ongoing file operations could have a read only view/copy of the mount table. Only new operations will have to wait. > > > > But the mount never needs to wait for ongoing operations before locking the structure. > > > > > > > > Just a thought in the morning > > > > > > > > Regards, > > > > Ronald. > > > > > > > > Van: Rick Macklem > > > > Datum: 1 maart 2024 00:31 > > > > Aan: Garrett Wollman > > > > CC: stable@freebsd.org, rmacklem@freebsd.org > > > > Onderwerp: Re: 13-stable NFS server hang > > > > > > > > On Wed, Feb 28, 2024 at 4:04PM Rick Macklem wrote: > > > > > > > > > > On Tue, Feb 27, 2024 at 9:30PM Garrett Wollman wrote: > > > > > > > > > > > > Hi, all, > > > > > > > > > > > > We've had some complaints of NFS hanging at unpredictable intervals. > > > > > > Our NFS servers are running a 13-stable from last December, and > > > > > > tonight I sat in front of the monitor watching `nfsstat -dW`. I was > > > > > > able to clearly see that there were periods when NFS activity would > > > > > > drop *instantly* from 30,000 ops/s to flat zero, which would last > > > > > > for about 25 seconds before resuming exactly as it was before. > > > > > > > > > > > > I wrote a little awk script to watch for this happening and run > > > > > > `procstat -k` on the nfsd process, and I saw that all but two of the > > > > > > service threads were idle. The three nfsd threads that had non-idle > > > > > > kstacks were: > > > > > > > > > > > > PID TID COMM TDNAME KSTACK > > > > > > 997 108481 nfsd nfsd: master mi_switch sleepq_timedwait _sleep nfsv4_lock nfsrvd_dorpc nfssvc_program svc_run_internal svc_run nfsrvd_nfsd nfssvc_nfsd sys_nfssvc amd64_syscall fast_syscall_common > > > > > > 997 960918 nfsd nfsd: service mi_switch sleepq_timedwait _sleep nfsv4_lock nfsrv_setclient nfsrvd_exchangeid nfsrvd_dorpc nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline > > > > > > 997 962232 nfsd nfsd: service mi_switch _cv_wait txg_wait_synced_impl txg_wait_synced dmu_offset_next zfs_holey zfs_freebsd_ioctl vn_generic_copy_file_range vop_stdcopy_file_range VOP_COPY_FILE_RANGE vn_copy_file_range nfsrvd_copy_file_range nfsrvd_dorpc nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline > > > > > > > > > > > > I'm suspicious of two things: first, the copy_file_range RPC; second, > > > > > > the "master" nfsd thread is actually servicing an RPC which requires > > > > > > obtaining a lock. The "master" getting stuck while performing client > > > > > > RPCs is, I believe, the reason NFS service grinds to a halt when a > > > > > > client tries to write into a near-full filesystem, so this problem > > > > > > would be more evidence that the dispatching function should not be > > > > > > mixed with actual operations. I don't know what the clients are > > > > > > doing, but is it possible that nfsrvd_copy_file_range is holding a > > > > > > lock that is needed by one or both of the other two threads? > > > > > > > > > > > > Near-term I could change nfsrvd_copy_file_range to just > > > > > > unconditionally return NFSERR_NOTSUP and force the clients to fall > > > > > > back, but I figured I would ask if anyone else has seen this. > > > > > I have attached a little patch that should limit the server's Copy size > > > > > to vfs.nfsd.maxcopyrange (default of 10Mbytes). > > > > > Hopefully this makes sure that the Copy does not take too long. > > > > > > > > > > You could try this instead of disabling Copy. It would be nice to know if > > > > > this is suffciient? (If not, I'll probably add a sysctl to disable Copy.) > > > > I did a quick test without/with this patch,where I copied a 1Gbyte file. > > > > > > > > Without this patch, the Copy RPCs mostly replied in just under 1sec > > > > (which is what the flag requests), but took over 4sec for one of the Copy > > > > operations. This implies that one Read/Write of 1Mbyte on the server > > > > took over 3 seconds. > > > > I noticed the first Copy did over 600Mbytes, but the rest did about 100Mbytes > > > > each and it was one of these 100Mbyte Copy operations that took over 4sec. > > > > > > > > With the patch, there were a lot more Copy RPCs (as expected) of 10Mbytes > > > > each and they took a consistent 0.25-0.3sec to reply. (This is a test of a local > > > > mount on an old laptop, so nowhere near a server hardware config.) > > > > > > > > So, the patch might be sufficient? > > > > > > > > It would be nice to avoid disabling Copy, since it avoids reading the data > > > > into the client and then writing it back to the server. > > > > > > > > I will probably commit both patches (10Mbyte clip of Copy size and > > > > disabling Copy) to main soon, since I cannot say if clipping the size > > > > of the Copy will always be sufficient. > > > > > > > > Pleas let us know how trying these patches goes, rick > > > > > > > > > > > > > > rick > > > > > > > > > > > > > > > > > -GAWollman > > > > > > > > > > > > > > > > > > > > ________________________________ > > > > > > > > > > > > > > > > > > > ________________________________ > > > > > > > > > > > > Hi Rick, > > > > > > You are much more into the NFS code than I am so excuse me if what I'm speaking about does not make sense. > > > > > > I was reading nfsrvd_compound() which calls nfsrvd_copy_file_range() via the nfsrv4_ops2 structure. > > > Nfsrvd_compound holds a lock or refcount on nfsv4rootfs_lock during the whole operation. Which is why nfsrv_setclient() is waiting in this specific case of "NFS server hang". > > > > > > But I don't see what is being modified on the nfsdstate after the IO operation ends. Or why the IO operation itself needs the lock to the nfsdstate. IMHO the in-progress IOs will happen anyway regardless of the nfsdstate. Changes to the nfsdstate during an IO operation would not affect the ongoing IO operation. > > > Wouldn't it be possible to lock the nfsv4rootfs_lock, do checks on or modify the nfsdstate as needed, unlock and then do the IO operation? That would remove a lot of the possible lock contention during (u)mount. > > > Otherwise, if we do modify the nfsdstate after the IO operation, isn't it possible to relock nfsv4rootfs_lock after the IO operation finishes? > > Well, there are a couple of reasons. Every implementation has design tradeoffs: > > 1 - A NFSv4 RPC is a compound, which can be a pretty arbitrary list of > > operations. > > As such, the NFSv4 server does not know if an open/byte range > > lock is coming > > after the operation it is currently performing, since the > > implementation does not > > pre-parse the entire compound. (I had a discussion w.r.t. > > pre-parsing with one of > > the main Linux knfsd server maintainers and he noted that he was > > not aware of > > any extant server that did pre-parse the compound. Although it > > would be useful > > for the server to have the ordered list of operations before > > commencing the RPC, > > we both agreed it was too hard to implement. > > --> It could possibly unlock/relock later, but see #2. Also, if > > relocking took a long time, > > it would result in the compound RPC taking too long (see below). > > 2 - For NFSv4.1/4.2 almost all RPCs are handled by a session. One non-arbitrary > > part of almost all NFSv4.1/4.2 RPCs is that the Sequence > > operation (the one that > > handles the session) must come first. > > Session(s) are associated with the ClientID, which means the > > ClientID and the > > session must not go away while the compound RPC is in progress. > > - This is ensured by the shared lock on the ClientID (that > > nfsv4rootfh_lock). > > Since 99.99% of operations can be done with the shared lock, I do not think > > there is a lot of contention. > > > > Although there is nothing wired down in the RFCs, there is an understanding > > in the NFSv4 community that a server should reply to an RPC in a reasonable > > time. Typically assumed to be 1-2sec. If the server does this, then a delay for > > the rare case of a new ClientID shouldn't be a big problem? > > (The is also delegation recall, which is one reason why delegations > > are not enabled > > by default.) > > > > Btw, the RFC does define an asynchronous Copy, where the operation replies > > as soon as the copy is started and the server notifies the client of completion > > later. I have not implemented this, because it introduces complexities that > > I do not want to deal with. > > For example, what happens when the server crashes/reboots while the copy > > is in progress? The file is left in a non-deterministic state, depending on what > > the client does when it does not receive the completion notify. > > > Oh, I should also note that the "shared lock" is actually called a > reference count > in the code and is there to ensure that the ClientID/Session does not go away > during execution of the compound. > > The problem in this case (which I should revisit) was that I could not figure > out how to safely add a new ClientID while other nfsd threads were in progress > performing other RPCs. Due to retries etc, there might be another RPC > in progress > using the ClientID. > > One thing to note here is that the identity of the ClientID > is not known until the Sequence operation has been performed. (And there is > cruft for NFSv4.0, since it does not have a Sequence operation.) > As such, the RPC must be in progress before it is known. > > > rick > > > > > > I hope this makes any sense and thanks for all your work on the NFS code. > > > > > > Regards, > > > Ronald. > > > > > > > > > > > > Hi Rick, Thanks for the elaborate answer. Would it make sense to have the current RPC/compound have a lock on its ClientID/session, but not on the whole nfsdstate (nfsv4rootfs_lock)? So concurrent requests like a new mount creating a new ClientID can go on in parallel, but removing or modifying the locked ClientID will wait for the lock. Or am I thinking too simple still? Regards, Ronald. ------=_Part_4769_153177038.1709725576674 Content-Type: text/html; charset=us-ascii Content-Transfer-Encoding: 7bit

Van: Rick Macklem <rick.macklem@gmail.com>
Datum: dinsdag, 5 maart 2024 15:43
Aan: Ronald Klop <ronald-lists@klop.ws>
CC: rmacklem@freebsd.org, Garrett Wollman <wollman@bimajority.org>, stable@freebsd.org
Onderwerp: Re: 13-stable NFS server hang

On Tue, Mar 5, 2024 at 6:34AM Rick Macklem <rick.macklem@gmail.com> wrote:
>
> On Tue, Mar 5, 2024 at 2:13AM Ronald Klop <ronald-lists@klop.ws> wrote:
> >
> >
> > Van: Rick Macklem <rick.macklem@gmail.com>
> > Datum: vrijdag, 1 maart 2024 15:23
> > Aan: Ronald Klop <ronald-lists@klop.ws>
> > CC: Garrett Wollman <wollman@bimajority.org>, stable@freebsd.org, rmacklem@freebsd.org
> > Onderwerp: Re: 13-stable NFS server hang
> >
> > On Fri, Mar 1, 2024 at 12:00AM Ronald Klop <ronald-lists@klop.ws> wrote:
> > >
> > > Interesting read.
> > >
> > >  Would it be possible to separate locking for admin actions like a client mounting an fs from traffic flowing for file operations?
> > Well, the NFS server does not really have any concept of a mount.
> > What I am referring to is the ClientID maintained for NFSv4 mounts,
> > which all the open/lock/session/layout state hangs off of.
> >
> > For most cases, this state information can safely be accessed/modified
> > via a mutex, but there are three exceptions:
> > - creating a new ClientID (which is done by the ExchangeID operation)
> >   and typically happens when a NFS client does a mount.
> > - delegation Recall (which only happens when delegations are enabled)
> >   One of the reasons delegations are not enabled by default on the
> > FreeBSD server.
> > - the DestroyClientID which is typically done by a NFS client during dismount.
> > For these cases, it is just too difficult to do them without sleeping.
> > As such, there is a sleep lock which the nfsd threads normally acquire shared
> > when doing NFSv4 operations, but for the above cases the lock is aquired
> > exclusive.
> > - I had to give the exclusive lock priority over shared lock
> > acquisition (it is a
> >   custom locking mechanism with assorted weirdnesses) because without
> >   that someone reported that new mounts took up to 1/2hr to occur.
> >   (The exclusive locker waited for 30min before all the other nfsd threads
> >    were not busy.)
> >   Because of this priority, once a nfsd thread requests the exclusive lock,
> >   all other nfsd threads executing NFSv4 RPCs block after releasing their
> >   shared lock, until the exclusive locker releases the exclusive lock.
> >
> > In summary, NFSv4 has certain advantages over NFSv3, but it comes
> > with a lot of state complexity. It just is not feasible to manipulate all that
> > state with only mutex locking.
> >
> > rick
> >
> > >
> > > Like ongoing file operations could have a read only view/copy of the mount table. Only new operations will have to wait.
> > > But the mount never needs to wait for ongoing operations before locking the structure.
> > >
> > > Just a thought in the morning
> > >
> > > Regards,
> > > Ronald.
> > >
> > > Van: Rick Macklem <rick.macklem@gmail.com>
> > > Datum: 1 maart 2024 00:31
> > > Aan: Garrett Wollman <wollman@bimajority.org>
> > > CC: stable@freebsd.org, rmacklem@freebsd.org
> > > Onderwerp: Re: 13-stable NFS server hang
> > >
> > > On Wed, Feb 28, 2024 at 4:04PM Rick Macklem wrote:
> > > >
> > > > On Tue, Feb 27, 2024 at 9:30PM Garrett Wollman wrote:
> > > > >
> > > > > Hi, all,
> > > > >
> > > > > We've had some complaints of NFS hanging at unpredictable intervals.
> > > > > Our NFS servers are running a 13-stable from last December, and
> > > > > tonight I sat in front of the monitor watching `nfsstat -dW`.  I was
> > > > > able to clearly see that there were periods when NFS activity would
> > > > > drop *instantly* from 30,000 ops/s to flat zero, which would last
> > > > > for about 25 seconds before resuming exactly as it was before.
> > > > >
> > > > > I wrote a little awk script to watch for this happening and run
> > > > > `procstat -k` on the nfsd process, and I saw that all but two of the
> > > > > service threads were idle.  The three nfsd threads that had non-idle
> > > > > kstacks were:
> > > > >
> > > > >   PID    TID COMM                TDNAME              KSTACK
> > > > >   997 108481 nfsd                nfsd: master        mi_switch sleepq_timedwait _sleep nfsv4_lock nfsrvd_dorpc nfssvc_program svc_run_internal svc_run nfsrvd_nfsd nfssvc_nfsd sys_nfssvc amd64_syscall fast_syscall_common
> > > > >   997 960918 nfsd                nfsd: service       mi_switch sleepq_timedwait _sleep nfsv4_lock nfsrv_setclient nfsrvd_exchangeid nfsrvd_dorpc nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline
> > > > >   997 962232 nfsd                nfsd: service       mi_switch _cv_wait txg_wait_synced_impl txg_wait_synced dmu_offset_next zfs_holey zfs_freebsd_ioctl vn_generic_copy_file_range vop_stdcopy_file_range VOP_COPY_FILE_RANGE vn_copy_file_range nfsrvd_copy_file_range nfsrvd_dorpc nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline
> > > > >
> > > > > I'm suspicious of two things: first, the copy_file_range RPC; second,
> > > > > the "master" nfsd thread is actually servicing an RPC which requires
> > > > > obtaining a lock.  The "master" getting stuck while performing client
> > > > > RPCs is, I believe, the reason NFS service grinds to a halt when a
> > > > > client tries to write into a near-full filesystem, so this problem
> > > > > would be more evidence that the dispatching function should not be
> > > > > mixed with actual operations.  I don't know what the clients are
> > > > > doing, but is it possible that nfsrvd_copy_file_range is holding a
> > > > > lock that is needed by one or both of the other two threads?
> > > > >
> > > > > Near-term I could change nfsrvd_copy_file_range to just
> > > > > unconditionally return NFSERR_NOTSUP and force the clients to fall
> > > > > back, but I figured I would ask if anyone else has seen this.
> > > > I have attached a little patch that should limit the server's Copy size
> > > > to vfs.nfsd.maxcopyrange (default of 10Mbytes).
> > > > Hopefully this makes sure that the Copy does not take too long.
> > > >
> > > > You could try this instead of disabling Copy. It would be nice to know if
> > > > this is suffciient? (If not, I'll probably add a sysctl to disable Copy.)
> > > I did a quick test without/with this patch,where I copied a 1Gbyte file.
> > >
> > > Without this patch, the Copy RPCs mostly replied in just under 1sec
> > > (which is what the flag requests), but took over 4sec for one of the Copy
> > > operations. This implies that one Read/Write of 1Mbyte on the server
> > > took over 3 seconds.
> > > I noticed the first Copy did over 600Mbytes, but the rest did about 100Mbytes
> > > each and it was one of these 100Mbyte Copy operations that took over 4sec.
> > >
> > > With the patch, there were a lot more Copy RPCs (as expected) of 10Mbytes
> > > each and they took a consistent 0.25-0.3sec to reply. (This is a test of a local
> > > mount on an old laptop, so nowhere near a server hardware config.)
> > >
> > > So, the patch might be sufficient?
> > >
> > > It would be nice to avoid disabling Copy, since it avoids reading the data
> > > into the client and then writing it back to the server.
> > >
> > > I will probably commit both patches (10Mbyte clip of Copy size and
> > > disabling Copy) to main soon, since I cannot say if clipping the size
> > > of the Copy will always be sufficient.
> > >
> > > Pleas let us know how trying these patches goes, rick
> > >
> > > >
> > > > rick
> > > >
> > > > >
> > > > > -GAWollman
> > > > >
> > > > >
> > >
> > > ________________________________
> > >
> > >
> > >
> > >
> > ________________________________
> >
> >
> >
> > Hi Rick,
> >
> > You are much more into the NFS code than I am so excuse me if what I'm speaking about does not make sense.
> >
> > I was reading nfsrvd_compound() which calls nfsrvd_copy_file_range() via the nfsrv4_ops2 structure.
> > Nfsrvd_compound holds a lock or refcount on nfsv4rootfs_lock during the whole operation. Which is why nfsrv_setclient() is waiting in this specific case of "NFS server hang".
> >
> > But I don't see what is being modified on the nfsdstate after the IO operation ends. Or why the IO operation itself needs the lock to the nfsdstate. IMHO the in-progress IOs will happen anyway regardless of the nfsdstate. Changes to the nfsdstate during an IO operation would not affect the ongoing IO operation.
> > Wouldn't it be possible to lock the nfsv4rootfs_lock, do checks on or modify the nfsdstate as needed, unlock and then do the IO operation? That would remove a lot of the possible lock contention during (u)mount.
> > Otherwise, if we do modify the nfsdstate after the IO operation, isn't it possible to relock nfsv4rootfs_lock after the IO operation finishes?
> Well, there are a couple of reasons. Every implementation has design tradeoffs:
> 1 - A NFSv4 RPC is a compound, which can be a pretty arbitrary list of
> operations.
>      As such, the NFSv4 server does not know if an open/byte range
> lock is coming
>      after the operation it is currently performing, since the
> implementation does not
>      pre-parse the entire compound. (I had a discussion w.r.t.
> pre-parsing with one of
>      the main Linux knfsd server maintainers and he noted that he was
> not aware of
>      any extant server that did pre-parse the compound. Although it
> would be useful
>      for the server to have the ordered list of operations before
> commencing the RPC,
>      we both agreed it was too hard to implement.
>      --> It could possibly unlock/relock later, but see #2. Also, if
> relocking took a long time,
>           it would result in the compound RPC taking too long (see below).
> 2 - For NFSv4.1/4.2 almost all RPCs are handled by a session. One non-arbitrary
>      part of almost all NFSv4.1/4.2 RPCs is that the Sequence
> operation (the one that
>      handles the session) must come first.
>      Session(s) are associated with the ClientID, which means the
> ClientID and the
>      session must not go away while the compound RPC is in progress.
>      - This is ensured by the shared lock on the ClientID (that
> nfsv4rootfh_lock).
> Since 99.99% of operations can be done with the shared lock, I do not think
> there is a lot of contention.
>
> Although there is nothing wired down in the RFCs, there is an understanding
> in the NFSv4 community that a server should reply to an RPC in a reasonable
> time. Typically assumed to be 1-2sec. If the server does this, then a delay for
> the rare case of a new ClientID shouldn't be a big problem?
> (The is also delegation recall, which is one reason why delegations
> are not enabled
> by default.)
>
> Btw, the RFC does define an asynchronous Copy, where the operation replies
> as soon as the copy is started and the server notifies the client of completion
> later. I have not implemented this, because it introduces complexities that
> I do not want to deal with.
> For example, what happens when the server crashes/reboots while the copy
> is in progress? The file is left in a non-deterministic state, depending on what
> the client does when it does not receive the completion notify.
>
Oh, I should also note that the "shared lock" is actually called a
reference count
in the code and is there to ensure that the ClientID/Session does not go away
during execution of the compound.

The problem in this case (which I should revisit) was that I could not figure
out how to safely add a new ClientID while other nfsd threads were in progress
performing other RPCs. Due to retries etc, there might be another RPC
in progress
using the ClientID.

One thing to note here is that the identity of the ClientID
is not known until the Sequence operation has been performed. (And there is
cruft for NFSv4.0, since it does not have a Sequence operation.)
As such, the RPC must be in progress before it is known.

> rick
> >
> > I hope this makes any sense and thanks for all your work on the NFS code.
> >
> > Regards,
> > Ronald.
> >
> >
> >


Hi Rick,

Thanks for the elaborate answer.

Would it make sense to have the current RPC/compound have a lock on its ClientID/session, but not on the whole nfsdstate (nfsv4rootfs_lock)?

So concurrent requests like a new mount creating a new ClientID can go on in parallel, but removing or modifying the locked ClientID will wait for the lock.

Or am I thinking too simple still?

Regards,
Ronald.
  ------=_Part_4769_153177038.1709725576674-- From nobody Thu Mar 7 15:01:39 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TrCF51mRJz5D9yQ for ; Thu, 7 Mar 2024 15:01:45 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from DEU01-BE0-obe.outbound.protection.outlook.com (mail-be0deu01on20701.outbound.protection.outlook.com [IPv6:2a01:111:f403:2622::701]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TrCF43qqmz43M7 for ; Thu, 7 Mar 2024 15:01:44 +0000 (UTC) (envelope-from hausen@punkt.de) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of hausen@punkt.de designates 2a01:111:f403:2622::701 as permitted sender) smtp.mailfrom=hausen@punkt.de; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kxD/MblKl78P8y+byv+oeLmyaI/SkMLjDgUl1aR8LcTDsuEv6J8o7MSYzDzUSkZyttF4M29yQodFIjQ0dzLBhGKKqztU5+DmLLQx/Bfx9743TT5mpzTugm2zdjZDjYmx7CxPDwCqw3DGp22/sFmRpcIRYNCki+kbS9UZuX2A/ttMq5uo/+ORsYhnQnbgUDJotH9ngaVAoOnjg0Zy4Q6UUZvqPHlSZoGZS0aX9sUvy5qVCwfkUkL1gFP5PsqOs74oxdemHJCedzW4sLSqN78bQPYexaEgJytej40pl4tLmXMNWZnsm6QTD9lwMTiGy/K/Vm+r0IfS5wEluHOSgPDyfg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=LghswUlqisWL35D/14TU7VPonjyhQNDBJnV3aNo1UEA=; b=AD0/JwvpTGzzcLoLOlGys56Loh03RZaB1VtR2oECt+qhvmPRAffztn0BF8Yu5xGevHihooWXD+DWa7z1wm5CFbTLy5F9QNzBnm849p+IXPyh88MSv/4KDAkxyOQ0vNHhF77znSnO7++J6lBaZ6dJEzMdWVEXw0a25PhXoV4acUomobt6ozJMqNdU+siZI84pDCsGrKngY8JKTzblLBQU3CEqzxnF3iQx595YK7auJC7EyXpMWpTaGxVytw4Kb56u6qhBVy9+YiXs9va6tYgAHKAnZEV8ehMrgbvNaoBU+N9+tNH+nQ5h9n93nhOPXqMPG16GqbZxB23yScM/Lk9BaA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=punkt.de; dmarc=pass action=none header.from=punkt.de; dkim=pass header.d=punkt.de; arc=none Received: from FRYP281MB3306.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:73::11) by BEZP281MB2867.DEUP281.PROD.OUTLOOK.COM (2603:10a6:b10:72::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7362.26; Thu, 7 Mar 2024 15:01:39 +0000 Received: from FRYP281MB3306.DEUP281.PROD.OUTLOOK.COM ([fe80::3e0e:ddc9:b987:28e0]) by FRYP281MB3306.DEUP281.PROD.OUTLOOK.COM ([fe80::3e0e:ddc9:b987:28e0%7]) with mapi id 15.20.7362.019; Thu, 7 Mar 2024 15:01:39 +0000 From: "Patrick M. Hausen" To: Freebsd Stable Subject: kqueuex() system call was MFC'ed to releng/13 and releng/13.3? Thread-Topic: kqueuex() system call was MFC'ed to releng/13 and releng/13.3? Thread-Index: AQHacKBbI10FYIJro0iYV0ImAlGIfQ== Date: Thu, 7 Mar 2024 15:01:39 +0000 Message-ID: Accept-Language: de-DE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: FRYP281MB3306:EE_|BEZP281MB2867:EE_ x-ms-office365-filtering-correlation-id: 69f8851c-837f-4a1a-c501-08dc3eb77e23 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: +v72qZ6ZuXl9AuioiHdklVq+TnQUNFZ+gAhxjCakHo+0tqRH3fmUACJahG5e/Xv52QMDmpeaKikIUJKwENb5frAlnlBFhjxYvk3snj1h/vF+KYQGqVV/phQKoKG0amiscsHlL5bDxItsEkkhMV3P0QoJ2QyjkHPf77D5tMEoql328xTqZlsesWZhZZkE6Hv3kiyR6G3PetOIfiKaBMX1jr6P0CFEcp3NgjuTHUyJ16uUs+xHLx+Fvgk4DwiDYxtHiZ6NUomX0UK7hXmcqKmQDulw0ST6/k2jHrxzvzq4VYnx0k7fH5Li+KQAsnefZU7LzNnYqihL/x3tPUxP29JSc/jwx+9m3UG0o9Qyqy1Mdp6GcgV9+RcZV+q1b+jvvm14I1+E5jdSpOw2tzWyDPcERFvw25iMkd5C/6J6FAm+bYfl85hJkcGVf3M2DohSlTtyWgQRnBfM563Q3dbtFCA2A/1RDWNJVWO5G6EdUDJotODLaPJpiwPVfjWqpuAw6wsg3JbZbbxHfgkohVmmsu6CJCXUWgrgAqvbdulmpN4zPQxH2tM9VT6mtnZmPKt92S0WO8igYqS8dmIx/vfjvUcgSnM1wyZHCWGhMQjyb8VmQiG+cu3oMcm2BNcRsIdARTjq49OwB2M9h7ptxOvUTG6VO/Ew9TMzBNPZfBLdMnEzKW0= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:FRYP281MB3306.DEUP281.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230031)(376005)(38070700009);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?utf-8?B?dnBHTnArMmtRM0g1WFU4clhDc08zWXpFWG9teGtLbzcwR1pXem5xNlcrTW1r?= =?utf-8?B?WVUyNU9NQzFwcHdUM0l0TkFNU1ZBTXo3RjF4SzVYZ1owRWMxYlhtQlU1VkFP?= =?utf-8?B?QnpOdDhlbVRDd3NiblNKeTFjSXJwOG0zd0dXaEtjdFNQbWNXTlZ0bDlaUHdE?= =?utf-8?B?TDI4aXNDRHE1VTQwVWRzWnA3a1ZKQzJVUmlQS2JNK0RaL1R4b2N1VDNMQk5h?= =?utf-8?B?blFqZWQwNXZTV2lKclRGSTA0RndRbVFxSmJWc3lmWkNRMmNUSjIrVUhrZzBW?= =?utf-8?B?dXU5MWFUU3ZwemtVVXFHVW9zVmZkV0hRYklwYVhhR3dHbXZ5dTk0WnVINnR0?= =?utf-8?B?aVc5dzhTdlUzV3VpZmFYK1QzYm1uVWwyREEwU1k3cGdraWswa1BlQmZrcTF5?= =?utf-8?B?NlJ2bkhNWWc4ajV3c3JEMGY1WTg4MFQwdlBsL1VVd3U5bkNBcnVsYXNlOTdJ?= =?utf-8?B?ZUF4RXp6ZDVocVdla2ZDTGtGZWpucFl1QnlrNmxucmtOVFVGVWpUTHhiYWZU?= =?utf-8?B?UzFCUkFNVDhtdGFaOEVEbXpHZTVtQnh6T1RJWU1QYk1ESk5BemFMK0NQbmJw?= =?utf-8?B?dTN2NUFRNjRTS3pReHBCUU9EcUFqQnFvYi9DYkpzU0p5bmJXa21Vbm8zNFRM?= =?utf-8?B?Mk1DSEJVamc5U0RaSTdmTVNvRFFleVVwcmQ4N3FyQWphTmN0VERhend6L3pB?= =?utf-8?B?R1ZrcGxRYitvSE1PUnBPNW0yb0hnQnJudUVmUWVmQnVvVnB5ZFZNeE9ZL3J1?= =?utf-8?B?SWxRN1c3eGdaQUxoWk8rOE1xZHRZYWlOUWwvQThwNlRlYlJsZDcxOEo4Z3NR?= =?utf-8?B?R0JKSEU4WUhncExNY3Q5aGNBKzJiakJYdldnTm9ZcmVHMk84U0syS2E2L3NY?= =?utf-8?B?b1VYc2hKOUw5c1UrNjV6Q0lMZlQvN0gzNFp5SHBFWmk3NVBhaVYwU3ZQRUw5?= =?utf-8?B?Qm5LT3hjOW04L2JDTm9kTnpOOUJWTDF0TnArSkFJb1pIVHJTTDRpUThuSzBG?= =?utf-8?B?ZFEyYS96UEJsZjRvVitSeVQ1c0ZIdHh5WEZ5N1FESzFwZlhxN2doK0orbkhP?= =?utf-8?B?WXZJM1haV2RWVWJHeUFrR1FGOUY3TFhzNzlEWldoRkFHMFBLUk9uL1ZlZzRD?= =?utf-8?B?V1loUkdtUFE0ZFN3ZnhudldiUGVtdytMRFdkbFFxUFhLcnVpclViUjlRa2lB?= =?utf-8?B?bjlDLzhpWWlWT0ZKWW9pSHlHQVVRbEIwMFAwUERBYVBkTnBJSTBqSkhhUnk1?= =?utf-8?B?eE51SFE4U2w1cURGMnc4RGJQcWxhdlM3VHkxbEZLcFJhQXBxajlOQkwyVDBh?= =?utf-8?B?d01Da281bVhPZS9LL2J6SGZCOFZDQVg0SklIZ0tMQVEwYkE5VnJuOW5IcG5D?= =?utf-8?B?WDJoditPc3ZScHZocUhBZVpMd3hKTzEvL0RMK3duMU8xYjZQeW9aenlOZC9u?= =?utf-8?B?dmRkV3YxQzN5SStaRk1ocFpHdWR2bk41SVUvNll3S09xZUh1VzM1SUZqOHhN?= =?utf-8?B?YWRqcXVmNVVHeVZUTzVkMXJoTVBOVDA3eDNwSEZ1L0FFZHZCRUQwRG91Q1Jh?= =?utf-8?B?eWZOdlBsMlR6emRESjc3d3hLbU1RZWlJdWtSKy9YWXZlZjVWVTlndmdYREx4?= =?utf-8?B?dTM4QmFrVS84d0RQNmtzalc2Q2dMVjZ4bk1VK2FSZ2xFTlVYQ3I2dEFuT0dz?= =?utf-8?B?MDM0MnhBVTd4MkM1Vjc0Ym91dUh1blBPTGx4b2pBTHVEdThpN2hndUxja3M5?= =?utf-8?B?S0ozTTE5QkJ5V1Z0a29nTzhOSzNwKzVxUnJKbVBzWm9NMUF6b2lscm1WTFox?= =?utf-8?B?ekFmRVNNRThDVzVJa3Y4MWVUbW5zSTM2SmZaOTN4OHlpS2Zsb3FydHNBY0w4?= =?utf-8?B?NTZRc0N4bUdmeUNONWdkTmJLWTUvVlVmVmVBejMwQ3BGdW9Tb0I0aFFnUlV0?= =?utf-8?B?VzRNY3Y0S2kyTXd6VXhKc3AvV2Z1ZzVjTTRLOUlhTTA1REVKV2VDakJZWHpY?= =?utf-8?B?bjJhSFV3NUxBUWxCQktGdFNkazFUc2lTMU5lcm5FUHhaMTI1UEc5VGRUOVFW?= =?utf-8?B?d1QwSVBaNUp2UjZPdWZHMFVLVkdDTjZZYjJtV2N2SnlsYlp0ajI5clJnVGlJ?= =?utf-8?B?WVpIaGwrdW1yS2lmakdkLzhkLzdXTnBiYUxRMDB4R3hBUG1DakpnemNWUnds?= =?utf-8?Q?seHACJNBLJ1gp/dLjCS/lDWYAC5qTHgfuv45pF76Dzsk?= Content-Type: text/plain; charset="utf-8" Content-ID: <3CFAE33705AD4C40801451AF7325C507@DEUP281.PROD.OUTLOOK.COM> Content-Transfer-Encoding: base64 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: punkt.de X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: FRYP281MB3306.DEUP281.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 69f8851c-837f-4a1a-c501-08dc3eb77e23 X-MS-Exchange-CrossTenant-originalarrivaltime: 07 Mar 2024 15:01:39.5670 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: d1aa1808-3734-45fc-a490-f8ba49028756 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: fLkwNXn/mv5KD9oZYTLlswYugkqxErefA3oZDZRFeAW05eoSfdjp77O4fOzz7Hjc X-MS-Exchange-Transport-CrossTenantHeadersStamped: BEZP281MB2867 X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.55 / 15.00]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.997]; NEURAL_HAM_SHORT(-0.36)[-0.357]; R_SPF_ALLOW(-0.20)[+ip6:2a01:111:f403::/49]; MIME_GOOD(-0.10)[text/plain]; MIME_BASE64_TEXT(0.10)[]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:8075, ipnet:2a01:111:f000::/36, country:US]; MIME_TRACE(0.00)[0:+]; MISSING_XM_UA(0.00)[]; DMARC_NA(0.00)[punkt.de]; RCVD_COUNT_TWO(0.00)[2]; R_DKIM_NA(0.00)[]; TO_DN_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; FROM_HAS_DN(0.00)[] X-Rspamd-Queue-Id: 4TrCF43qqmz43M7 SGkgYWxsLA0KDQpJIGp1c3QgZm91bmQgdGhhdCBkYWVtb24oOCkgc3RvcHBlZCB3b3JraW5nIGFm dGVyIEkgdXBncmFkZWQgYSBqYWlsIHJ1bm5pbmcgb24gYSAxMy4xIGhvc3QgZnJvbSAxMy4yIHRv IDEzLjMuDQoNCglIb3N0OiAxMy4xIC0gamFpbDogMTMuMiAtIHdvcmtpbmcNCglIb3N0OiAxMy4x IC0gamFpbDogMTMuMyAtIGRhZW1vbig4KSBmYWlscyB0byBydW4NCg0KVGhlIGNhdXNlIHNlZW1z IHRvIGJlIHRoZSBpbnRyb2R1Y3Rpb24gb2YgdGhlIGtxdWV1ZXgoKSBzeXN0ZW0gY2FsbCBpbiAx My4zIHdoaWNoIGRvZXMgbm90IGV4aXN0IGluIDEzLjEgb3IgMTMuMi4NCg0KQW0gSSBjb3JyZWN0 IGluIHRoaXMgYW5hbHlzaXM/IElmIHllcywgSSBmZWFyIHRoZXJlIGlzIG5vIHdheSBhcm91bmQg dGhhdD8NCg0KSXNuJ3QgLXN0YWJsZSBpbXBseWluZyB0aGVyZSB3b24ndCBiZSBpbmNvbXBhdGli bGUgQUJJIGNoYW5nZXM/DQoNClRoaXMgY3JlYXRlcyBhIGh1Z2UgcHJvYmxlbSBmb3IgZXZlcnlv bmUgcnVubmluZyBqYWlscyBvbiBUcnVlTkFTIENPUkUgd2hlbiB0aGUgcGFja2FnZXMgZm9yIHJl bGVuZy8xMw0Kd2lsbCBzd2l0Y2ggZnJvbSAxMy4yIHRvIDEzLjMgLi4uDQoNCk9mIGNvdXJzZSB0 aGlzIGlzIG5vdCB0aGUgRnJlZUJTRCBwcm9qZWN0cyBmYXVsdCBidXQgdGhhdCBkaWQgY29tZSBh cyBhbiB1bnBsZWFzYW50IHN1cnByaXNlLCBub25ldGhlbGVzcy4NCg0KS2luZCByZWdhcmRzLA0K UGF0cmljaw0KLS0gDQpwdW5rdC5kZSBHbWJIDQpQYXRyaWNrIE0uIEhhdXNlbg0KLmluZnJhc3Ry dWN0dXJlDQoNClNvcGhpZW5zdHIuIDE4Nw0KNzYxODUgS2FybHNydWhlDQoNClRlbC4gKzQ5IDcy MSA5MTA5NTAwDQoNCmh0dHBzOi8vaW5mcmFzdHJ1Y3R1cmUucHVua3QuZGUNCmluZm9AcHVua3Qu ZGUNCg0KQUcgTWFubmhlaW0gMTA4Mjg1DQpHZXNjaMOkZnRzZsO8aHJlcjogRGFuaWVsIExpZW5l cnQsIEZhYmlhbiBTdGVpbg0KDQo= From nobody Thu Mar 7 15:44:39 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TrDBt0T5Lz5DFKY for ; Thu, 7 Mar 2024 15:44:54 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-ua1-f51.google.com (mail-ua1-f51.google.com [209.85.222.51]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TrDBs5DFCz46t0 for ; Thu, 7 Mar 2024 15:44:53 +0000 (UTC) (envelope-from asomers@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ua1-f51.google.com with SMTP id a1e0cc1a2514c-7dba5564dddso117252241.1 for ; Thu, 07 Mar 2024 07:44:53 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709826292; x=1710431092; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pPS3Wk+ADt/oV6SIKSWSrrXanrkdY1oI02tY5oP1eRg=; b=tskoFt0j2iAHgEQ3jVSUQQJ20JOdZWU/aNFGOErqrHEIUrTXyD0VEoWz3J5tS/i3Ar izMo9uvMgSIZT/CfngK5bPZpeYAPPH7P1OkTy440pbQI30hGWhK1nXXz4dY+/sPPE/Rg haGSSFw3hdVB9JvCOmANeg4SyRsPZdqLYVYzMLicFPaYUruleC5rUArzySmuemHNRVUg LC/6qXMLnle5fTio1DNtgofelk76hCG/88/rxx7V+XeyjRPXJDivNjL4QWI7jgPOE0YO piRUqm+F2La1Odu9FQUvuIMAflALmtwvpvhMxrVHRTmFc25+4jMGsIkumPGDR59MHm4x gGpg== X-Gm-Message-State: AOJu0Yxep4cXbNLb7iK0ABqcLCSoiE3gF789JSS0josn+W6/ThBf7m4q kXZLGcYpq71F4XMtdnzR7pVvIJ0Bod7k5VjFBVI/sViAKMmzjVB/stDEfEe/zMAP8XDaPuOSqxq SWP+JiQpUX6eNYTql76fka2yKpY8t61Gq X-Google-Smtp-Source: AGHT+IGL1sd3Mjr9iLKEnaMH3DnGovs/Z+wehISuCs5uzH5Zx8nCINz1dqafI9uSL9hJ0USHPgNPL6jlqltPLAu+Shc= X-Received: by 2002:a05:6122:2522:b0:4ce:96b7:c2f6 with SMTP id cl34-20020a056122252200b004ce96b7c2f6mr1405866vkb.5.1709826291633; Thu, 07 Mar 2024 07:44:51 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Alan Somers Date: Thu, 7 Mar 2024 08:44:39 -0700 Message-ID: Subject: Re: kqueuex() system call was MFC'ed to releng/13 and releng/13.3? To: "Patrick M. Hausen" Cc: Freebsd Stable Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US] X-Rspamd-Queue-Id: 4TrDBs5DFCz46t0 On Thu, Mar 7, 2024 at 8:02=E2=80=AFAM Patrick M. Hausen = wrote: > > Hi all, > > I just found that daemon(8) stopped working after I upgraded a jail runni= ng on a 13.1 host from 13.2 to 13.3. > > Host: 13.1 - jail: 13.2 - working > Host: 13.1 - jail: 13.3 - daemon(8) fails to run > > The cause seems to be the introduction of the kqueuex() system call in 13= .3 which does not exist in 13.1 or 13.2. > > Am I correct in this analysis? If yes, I fear there is no way around that= ? This sounds like a reasonable explanation. > Isn't -stable implying there won't be incompatible ABI changes? Yes, there won't be any _incompatible_ ABI changes. But the addition of a new syscall is a _compatible_ change. Similar changes have happened before. > > This creates a huge problem for everyone running jails on TrueNAS CORE wh= en the packages for releng/13 > will switch from 13.2 to 13.3 ... > > Of course this is not the FreeBSD projects fault but that did come as an = unpleasant surprise, nonetheless. > Running jails that are newer than the host has never been supported. It was just luck that your 13.2 jail worked on a 13.1 host. The host must always run a kernel at least as new as what the jail was built for. -Alan From nobody Thu Mar 7 16:29:39 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TrFBd2HDFz5DJjj for ; Thu, 7 Mar 2024 16:29:45 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from DEU01-BE0-obe.outbound.protection.outlook.com (mail-be0deu01on2124.outbound.protection.outlook.com [40.107.127.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TrFBc2VPRz4Dmw; Thu, 7 Mar 2024 16:29:44 +0000 (UTC) (envelope-from hausen@punkt.de) Authentication-Results: mx1.freebsd.org; none ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=X8Ysz7zOcQBUTbfRQE00MPN7+rz57+Wa4xMPL6aQ++uYKRbW0qG+Y93h6NvXtI42UIAQPHaCucvj8m6fN/b0LVW68KwM5x3aCOdhgmTm5AFe3h5Snr+rgwvTl3ddILx8vu8Mt/M1SA9PrRpPT4dmdk1GGOW9+ElyaJmXe91UWZyi5Zl8QPu9vfWBgsWlENbnlN4Qeq+SScEdwvMmNZMo1h1ZvsEck/ep56t8jHvpBUqNIJH03MgPl8ueE8o5T4/WNUckt5g/QTI/AvPO5053Melgcn/0WFSE9hpJYJAPiGRaZHpb+8+lVXYfVaVY+F9OikSBgj4qqfRRkLxs65atGQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=eVxzw+cuyK6+Lvm21h5VOXz3s6R/tlC8z9gR5kbaOVI=; b=KZ6ENFlosHCk4TLYkXk9SCI+2QN60FfNR8jb8ceNOqkKWdODjbg796DjWkzV9I2DPHn+ovdIGxnUjPlLWR0Eo0ZA1BdL2T1Rq3wg3ctmYCKSVJkLT8XVLk1hdODyP3/3nl3LkAol/bAZQOrBepbxuxNr4WXgOvqDrBEiBDWottFYl1wNwwSPNYleCOXBcQt0pr8fKTWFJtXT1vp7KhYkb+9sdEFOzXeqFOtqdErL43tYZFNsCMzi87DPQvuqMSV330FmdG9Bw/V6jsHgQyHKjqxWxMArvaTX2NN+Bwf1WZvVTrJ67TT7iUougRIVZ9+VmUh7yVKIshUiK87T+laKsw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=punkt.de; dmarc=pass action=none header.from=punkt.de; dkim=pass header.d=punkt.de; arc=none Received: from FRYP281MB3306.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:73::11) by BEZP281MB3301.DEUP281.PROD.OUTLOOK.COM (2603:10a6:b10:25::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7362.27; Thu, 7 Mar 2024 16:29:40 +0000 Received: from FRYP281MB3306.DEUP281.PROD.OUTLOOK.COM ([fe80::3e0e:ddc9:b987:28e0]) by FRYP281MB3306.DEUP281.PROD.OUTLOOK.COM ([fe80::3e0e:ddc9:b987:28e0%7]) with mapi id 15.20.7362.024; Thu, 7 Mar 2024 16:29:40 +0000 From: "Patrick M. Hausen" To: Alan Somers CC: Freebsd Stable Subject: Re: kqueuex() system call was MFC'ed to releng/13 and releng/13.3? Thread-Topic: kqueuex() system call was MFC'ed to releng/13 and releng/13.3? Thread-Index: AQHacKBb2+tu7402r0ajTnGcYH2pgLEsa3mAgAAMhgA= Date: Thu, 7 Mar 2024 16:29:39 +0000 Message-ID: References: In-Reply-To: Accept-Language: de-DE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: FRYP281MB3306:EE_|BEZP281MB3301:EE_ x-ms-office365-filtering-correlation-id: 4388d7ac-3d33-449e-5de4-08dc3ec3c957 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: wc0al6M2E6pVAFO0wa4URwCgiv2RC/3PiV/UO+lH5ZdPh0r8IS81nD7xVDcab5/DBRBsjrKVg71t8Nw9wQENYKrWUu2k4lSs47jiruzR0o8d1EasVqLHjitZbzUMQjcgCLcNFsgddW/WptP/aR2N31XGyhsjZTl1X4zyViPQt2y654WIKVkeuSFoT87cc7CZKG6WRigtoz6ymiZEHbVJ8brLARjh1IlMK8IvtL2yOp0dJjPXpAs/6Rn5qYGD+zMz51zbDj/e3N+ajdsAG5NWn0q8GN10/AETVX7dIWucuoYnOmgX8rtSkifRO6Px0Ye7fowaP9qva7lArj41wlO6AYpfoRPMzw8qOrJj85JP0PTwUCW3PJBY+8H61Tsolth4K8E9+E+56f+pgo8aKD2wKXZqXiFA5LsUP5GAbPl/FbMCW/TlreUwWQC1fgmAwYNvhReMhuH0tW9emkNViog3UzbtBK7UrAjEFYSnXJbhcaEZqyQZlW0zGoK5f46LexUFg2Rgm3CYfIvOvptSF362WpKy8ia4Qv6QrS5Z/01BsOw9Yr44vRwxfH/M37R5p7kqCPwWQ7wmCq4273BuT5cvMylYyxHotp3w2tP/niQOKrNni3wcjpFvOZgg2Q8TExWy0A2eixRPFwQAwCVsWNrFCZWYC/ifSw5WkItLZJt7QS8= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:FRYP281MB3306.DEUP281.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230031)(376005)(38070700009);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?utf-8?B?Wk5oK1JsbnN1ZGxrdFBxSERkWU5HejFaQ3Yva2YrQXdMMHU3SHB4NWttQXdX?= =?utf-8?B?aHh3ZmptV1E0N0NFdTJFWTVqUFpEZlBrbys2OTdYRWRUOUY4V0pETE92WVdl?= =?utf-8?B?Mnl0MHpqbnVBTzZ2UGhjK3BOS3BQREVhNUFCWnRLVzNFemMvaHVkbm9hdXpa?= =?utf-8?B?cDZPQlU2blBYazBkbHR2K2plY3RtdUdmSTFyUGV0N3lRUzNhWUFvOW1uNnVx?= =?utf-8?B?VlNDazNncVJqMUNmMlZxUWc0eTE0VDk5T1dtOUZxTnZ2QytxS2p5UDFURGtO?= =?utf-8?B?MEhidG0zYmYyTXdCY2ZodjFRMTU0QkZvK3VXY0N0aHluUVJWSmdseHBJdVFo?= =?utf-8?B?R0RBaXRUbVp5SU9GZDFQNzFQVHpBVGpXWUVnejVrRVJqQm43U2c1RVNBYWxP?= =?utf-8?B?eFhPQnJlKzhsalJibjF1Yy9LRDhBK0JWUGF2TGpVbzRHamxLZWZEcGhpMFkx?= =?utf-8?B?OU1xV2J0UjBKKzZoT2NaNnYxbDQ3bm1OSUhzYUVjRnNiNVUwaW42ekZmOWRN?= =?utf-8?B?K3R1U2ZPTFF6eGpSUFFGN1dyQVpBd2dkZTdscHRqNnFZekNBSm91cEsxWWxQ?= =?utf-8?B?UFJZa0VEU3dXT3hMelNndmtjdDVqUDU0bWFnZ1NaUXNGZFZxYzFYbGNBbUkv?= =?utf-8?B?VjdwWDY4bWZyWVhzb3lVVk55TnRlOWpQdjN1d1dZQmhkaGZEd3RYSzljQ01S?= =?utf-8?B?UlR3azMvQ21Cek1ZclJyb002SFpzcWY0cjlWNGpXc3lnRXdJYUlHUVU0L296?= =?utf-8?B?NWJVbFAxMWpLOVpseEJtVWRSWkN5ZklPd01mekxxazFLODlqUkdVRUlsZzZs?= =?utf-8?B?dnhVa2ltVWtjOG9LeEhJQW5SakFOWHNVbkhock95N0NzbGpXN1Zrak5CYUJK?= =?utf-8?B?QitsK2RrSWh0a0Zib0U3NVhIRGdWQjNUTmxKbW0vSVJFaGRtYlhoU1hnR1Az?= =?utf-8?B?ZHl4RmZTM1lrZ1ZITHJEcUk4Z3NFMUZzY05DNzdwVmJHUXZQOFR0bE9EY1ZR?= =?utf-8?B?Tk5mVXp0ZFVmQkp3MFVocWRYa25PN2hkUVRQMzczQkhHVWNicU56NVlTbTcr?= =?utf-8?B?VlhxYXowT0hjM0l4VTk3Vlpyd0t3NG5UMlRNWmdHYjY0bTBUT2sxdUNrdkdO?= =?utf-8?B?TWtpYS9iaUk3KzJ3dzhwbThsNHBkVnlVdEtLcTEyczhNZ2J4eW9uWER0K21t?= =?utf-8?B?WkF2ZnczS0Nzb2JyNUh3TG9PdytNQ2pLYnplaFkwREVqa0oycGswVWpEY29T?= =?utf-8?B?U0hVUFBMRHplVXB0RktFeWMzbTdpZzl4c1VSUTVXVXY2VzdoZ2lEelQvRUhQ?= =?utf-8?B?T1JOZ3FMZGZXNHQ3bU0ybm9UZThuTVAzRWRUZDRCN09wSE82UmdKdUljRk4w?= =?utf-8?B?a2RYNWhNcjVWNXNxYTVOV2Z2MTFxaXRabzVsSkNIbXc1N1p5UzdKdStKV1dQ?= =?utf-8?B?aFJ4YjRTSm1JRG5OZDJMSEMwUHZOL0NHQ0FyRWRLbmMyYnptcThTUTNubGRh?= =?utf-8?B?RzdXVXRMbzdxU3ZEbktDWlVPV1oyWVhnKzB6MzRPNnVLOUVMVkJ6SUMwOUJ6?= =?utf-8?B?VVVVVjJMQjVidkJkWU0rUFJaUGtnMHU2MkdpM2p1UVkydEI1K1FUQ2p4Qmly?= =?utf-8?B?M29QY1NaRWxuckgyemxpWDl4TzEvM2NqUkt5TzNmelZRaHcxV2dSZDdRNitJ?= =?utf-8?B?a1didVJUV3dScElaMFlldCsvNmRybGFITEZkckFTWExhRkhBbVBHcHptQW1p?= =?utf-8?B?K2tqVll1dyswcStWQmhicUNQL20vYkMrUkxsb0R0N0VwT2l0K0w0Qjc2UzRn?= =?utf-8?B?U0hpSUU2SEk4UHlxcTJZdW5vZldsMHNMSFFIbzV6aUdmS3pyS21OaU1rVGg0?= =?utf-8?B?UlJ0VDkzN1NsVlIzcGRTd1lxMVhLRHVBK1ZHbVBRQ280RmltZEJjdkcyMG95?= =?utf-8?B?eXA2NVJPa0dYWjg2eWc0eXdyamxtVnM3TndnTUZzMWFLN3dZblBya3ZNc0pH?= =?utf-8?B?V2RDOHBIVVVvN09NdUxVT3Z1a0ZLRHhucjd0TXBHT2dsQkRYWVlwS1dKd3J6?= =?utf-8?B?Zk1DSk5seUZvNXRISWwwckpkaVZkT2h3STQrYytQeXZQRWNINTVLNWdxOVAr?= =?utf-8?B?RkUydFZBd2tXMEtTM1l4WUZPSC83L2F5ZEw1eDFVUkZDZ012RXI0OGttY0Vy?= =?utf-8?Q?52m5ENTExQKt3roJD9Qju/FsEQ3N/Pb7YQFmUAVrCS3g?= Content-Type: text/plain; charset="utf-8" Content-ID: Content-Transfer-Encoding: base64 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: punkt.de X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: FRYP281MB3306.DEUP281.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 4388d7ac-3d33-449e-5de4-08dc3ec3c957 X-MS-Exchange-CrossTenant-originalarrivaltime: 07 Mar 2024 16:29:39.6822 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: d1aa1808-3734-45fc-a490-f8ba49028756 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: KaJzntOFrMGTIebovjk7ApiGvp9bu4+9WHO0HOwIu+sr3t2pvdFcN3n6o5tqLYmU X-MS-Exchange-Transport-CrossTenantHeadersStamped: BEZP281MB3301 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US] X-Rspamd-Queue-Id: 4TrFBc2VPRz4Dmw SGkhDQoNCj4gQW0gMDcuMDMuMjAyNCB1bSAxNjo0NCBzY2hyaWViIEFsYW4gU29tZXJzIDxhc29t ZXJzQGZyZWVic2Qub3JnPjoNCj4+IElzbid0IC1zdGFibGUgaW1wbHlpbmcgdGhlcmUgd29uJ3Qg YmUgaW5jb21wYXRpYmxlIEFCSSBjaGFuZ2VzPw0KPiANCj4gWWVzLCB0aGVyZSB3b24ndCBiZSBh bnkgX2luY29tcGF0aWJsZV8gQUJJIGNoYW5nZXMuICBCdXQgdGhlIGFkZGl0aW9uDQo+IG9mIGEg bmV3IHN5c2NhbGwgaXMgYSBfY29tcGF0aWJsZV8gY2hhbmdlLiAgU2ltaWxhciBjaGFuZ2VzIGhh dmUNCj4gaGFwcGVuZWQgYmVmb3JlLg0KPiANCj4+IA0KPj4gVGhpcyBjcmVhdGVzIGEgaHVnZSBw cm9ibGVtIGZvciBldmVyeW9uZSBydW5uaW5nIGphaWxzIG9uIFRydWVOQVMgQ09SRSB3aGVuIHRo ZSBwYWNrYWdlcyBmb3IgcmVsZW5nLzEzDQo+PiB3aWxsIHN3aXRjaCBmcm9tIDEzLjIgdG8gMTMu MyAuLi4NCj4+IA0KPj4gT2YgY291cnNlIHRoaXMgaXMgbm90IHRoZSBGcmVlQlNEIHByb2plY3Rz IGZhdWx0IGJ1dCB0aGF0IGRpZCBjb21lIGFzIGFuIHVucGxlYXNhbnQgc3VycHJpc2UsIG5vbmV0 aGVsZXNzLg0KPj4gDQo+IA0KPiBSdW5uaW5nIGphaWxzIHRoYXQgYXJlIG5ld2VyIHRoYW4gdGhl IGhvc3QgaGFzIG5ldmVyIGJlZW4gc3VwcG9ydGVkLg0KPiBJdCB3YXMganVzdCBsdWNrIHRoYXQg eW91ciAxMy4yIGphaWwgd29ya2VkIG9uIGEgMTMuMSBob3N0LiAgVGhlIGhvc3QNCj4gbXVzdCBh bHdheXMgcnVuIGEga2VybmVsIGF0IGxlYXN0IGFzIG5ldyBhcyB3aGF0IHRoZSBqYWlsIHdhcyBi dWlsdA0KPiBmb3IuDQoNClRoYW5rcyBmb3IgdGhlIGV4cGxhbmF0aW9uLg0KDQpLaW5kIHJlZ2Fy ZHMsDQpQYXRyaWNrDQotLSANCnB1bmt0LmRlIEdtYkgNClBhdHJpY2sgTS4gSGF1c2VuDQouaW5m cmFzdHJ1Y3R1cmUNCg0KU29waGllbnN0ci4gMTg3DQo3NjE4NSBLYXJsc3J1aGUNCg0KVGVsLiAr NDkgNzIxIDkxMDk1MDANCg0KaHR0cHM6Ly9pbmZyYXN0cnVjdHVyZS5wdW5rdC5kZQ0KaW5mb0Bw dW5rdC5kZQ0KDQpBRyBNYW5uaGVpbSAxMDgyODUNCkdlc2Now6RmdHNmw7xocmVyOiBEYW5pZWwg TGllbmVydCwgRmFiaWFuIFN0ZWluDQoNCg== From nobody Thu Mar 7 16:35:16 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TrFKK1DS0z5DKLx for ; Thu, 7 Mar 2024 16:35:33 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ej1-x630.google.com (mail-ej1-x630.google.com [IPv6:2a00:1450:4864:20::630]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TrFKJ5BWkz4GFL for ; Thu, 7 Mar 2024 16:35:32 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x630.google.com with SMTP id a640c23a62f3a-a36126ee41eso161963866b.2 for ; Thu, 07 Mar 2024 08:35:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1709829328; x=1710434128; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=pMfEokmJs5R+UUBustMBw+LBe0hgs1Cj2kzud0YcsFU=; b=wZoVs6i/7ugN0WOymdsPHtmhoR3JclkTFeTGtp9N3ptN1k/ji4qnyHMJzIU6/6t4SW ZtOXo/gAo7nQuolHEkdQev+B0okHPYQbWuP1mhfyIKuHUvkKF8DHemvHxVOTLqM6AQCN latM4SZ4YVyFaqZvzOjMcF7BF7GVUbraaPxgqDVVc5J570swJzOdZIqH2+ySHhOWE7n1 3y6HYajvh9HpK29iiRyNM/DdIa//5R30VoEdFQTAqfU3Exj+rQRcMHNUOX4IxPGrqovy YTNnVlCU5l26nAyAwzucniEkqRM+44IXC8inaG68C1LJf9IDrE/JkroiTHTgpn4rLKbX fo2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709829328; x=1710434128; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=pMfEokmJs5R+UUBustMBw+LBe0hgs1Cj2kzud0YcsFU=; b=OBmW5OGGvMoYrjJBnZTEkORK+TUQNHfoB9ZpwPWZ8Kg/jseki4es8oq+onJiw5efxL 2nAAG/DHwfNWvlyYBz8VnaLdagz7aIP9HizF2KyCHYFkorgC30KuWz15rk3LlRrTBU31 zk48Fy5TaH2W3vaNr3EkKcWRw5QWRJnG7vvZ2kDpIr3OE0xPedmGnkiOTqGJO1G9/57Q 4qWvLJcQOgCSRAM27MuZzfdXwQFzmEi4aGtK/V48ISnXGQoOuN0/KKBhjsOSaFiS2VV0 +Hi0EiE+I/Vtb8rymAM0GgHEvr7VWIljIq/lcxcKPI0dg0knXV3Fq3GDo+MRwYiakNyD 8Xug== X-Forwarded-Encrypted: i=1; AJvYcCUu2DZ1wjWPmb9G1qFdhUZk/azGhAWfWfVnaLWpTsGL8hHMtZ+jR6rx1QnFfRQlTAzNKmn0APkaJtUY1yS9qSVnhSE3BS6Lw/Br7w== X-Gm-Message-State: AOJu0YzW9PzoZ8ra9FVvp/w/yYldtkmpO3HTVLMEGfSn4oaIu6zWR53i A545uuVFb+EKY7f+hzTgh1BJ8yLFbyymix6GBdAw1It7hmAZJOk76YpxpfdoMdvnozN6XxqzUF4 SqneDSKgQGY5yp90WpoWR76TMsectf2m+uPiAcA== X-Google-Smtp-Source: AGHT+IGSUz4x5hiXscI45OhzhR/K8MlmVdp9i+jjANhefOhD09tg3F008b3aYFoJVr8+C+24kjznQsR1wuAo5TG2if4= X-Received: by 2002:a17:906:184e:b0:a44:b90a:8459 with SMTP id w14-20020a170906184e00b00a44b90a8459mr12083673eje.53.1709829327640; Thu, 07 Mar 2024 08:35:27 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Thu, 7 Mar 2024 08:35:16 -0800 Message-ID: Subject: Re: kqueuex() system call was MFC'ed to releng/13 and releng/13.3? To: "Patrick M. Hausen" Cc: Alan Somers , Freebsd Stable Content-Type: multipart/alternative; boundary="000000000000b3732a061314a968" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4TrFKJ5BWkz4GFL --000000000000b3732a061314a968 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Mar 7, 2024, 8:30=E2=80=AFAM Patrick M. Hausen wr= ote: > Hi! > > > Am 07.03.2024 um 16:44 schrieb Alan Somers : > >> Isn't -stable implying there won't be incompatible ABI changes? > > > > Yes, there won't be any _incompatible_ ABI changes. But the addition > > of a new syscall is a _compatible_ change. Similar changes have > > happened before. > > > >> > >> This creates a huge problem for everyone running jails on TrueNAS CORE > when the packages for releng/13 > >> will switch from 13.2 to 13.3 ... > >> > >> Of course this is not the FreeBSD projects fault but that did come as > an unpleasant surprise, nonetheless. > >> > > > > Running jails that are newer than the host has never been supported. > > It was just luck that your 13.2 jail worked on a 13.1 host. The host > > must always run a kernel at least as new as what the jail was built > > for. > > Thanks for the explanation. > Yea. It's hard to know the future so old kernels can't know about new system calls. Our support for forwards compatibility has typically been confined to short periods in current when the issue affected bootstrapping. New features in the kernel are often impossible to emulate on older kernels which is why we generally don't support new binaries with old kernels. Warner > > Kind regards, > Patrick > -- > punkt.de GmbH > Patrick M. Hausen > .infrastructure > > Sophienstr. 187 > 76185 Karlsruhe > > Tel. +49 721 9109500 > > https://infrastructure.punkt.de > info@punkt.de > > AG Mannheim 108285 > Gesch=C3=A4ftsf=C3=BChrer: Daniel Lienert, Fabian Stein > > --000000000000b3732a061314a968 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Thu, Mar 7, 2024, 8:30=E2=80=AFAM Patrick M. Hausen= <hausen@punkt.de> wrote:
<= /div>
Hi!

> Am 07.03.2024 um 16:44 schrieb Alan Somers <asomers@freebsd.org>:
>> Isn't -stable implying there won't be incompatible ABI cha= nges?
>
> Yes, there won't be any _incompatible_ ABI changes.=C2=A0 But the = addition
> of a new syscall is a _compatible_ change.=C2=A0 Similar changes have<= br> > happened before.
>
>>
>> This creates a huge problem for everyone running jails on TrueNAS = CORE when the packages for releng/13
>> will switch from 13.2 to 13.3 ...
>>
>> Of course this is not the FreeBSD projects fault but that did come= as an unpleasant surprise, nonetheless.
>>
>
> Running jails that are newer than the host has never been supported. > It was just luck that your 13.2 jail worked on a 13.1 host.=C2=A0 The = host
> must always run a kernel at least as new as what the jail was built > for.

Thanks for the explanation.


Kind regards,
Patrick
--
punkt.de GmbH
Patrick M. Hausen
.infrastructure

Sophienstr. 187
76185 Karlsruhe

Tel. +49 721 9109500

https://infrastructure.punkt.de
info@= punkt.de

AG Mannheim 108285
Gesch=C3=A4ftsf=C3=BChrer: Daniel Lienert, Fabian Stein

--000000000000b3732a061314a968-- From nobody Fri Mar 8 03:59:02 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TrXV62fwkz5Cl7M for ; Fri, 8 Mar 2024 03:59:10 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (tunnel82308-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "garrett.wollman.name", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TrXV53zh0z4SdX for ; Fri, 8 Mar 2024 03:59:09 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bimajority.org (policy=none); spf=pass (mx1.freebsd.org: domain of wollman@hergotha.csail.mit.edu designates 2001:470:1f06:ccb::2 as permitted sender) smtp.mailfrom=wollman@hergotha.csail.mit.edu Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.17.1/8.17.1) with ESMTPS id 4283x2vi010094 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 7 Mar 2024 22:59:02 -0500 (EST) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.17.1/8.17.1/Submit) id 4283x2VA010093; Thu, 7 Mar 2024 22:59:02 -0500 (EST) (envelope-from wollman) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <26090.36102.379531.160926@hergotha.csail.mit.edu> Date: Thu, 7 Mar 2024 22:59:02 -0500 From: Garrett Wollman To: Rick Macklem Cc: stable@freebsd.org Subject: Re: 13-stable NFS server hang In-Reply-To: <26083.64612.717082.366639@hergotha.csail.mit.edu> References: <26078.50375.679881.64018@hergotha.csail.mit.edu> <26083.64612.717082.366639@hergotha.csail.mit.edu> X-Mailer: VM 8.2.0b under 29.1 (amd64-portbld-freebsd13.2) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.4 (hergotha.csail.mit.edu [0.0.0.0]); Thu, 07 Mar 2024 22:59:02 -0500 (EST) X-Spam-Status: No, score=-4.5 required=5.0 tests=ALL_TRUSTED, HEADER_FROM_DIFFERENT_DOMAINS,NICE_REPLY_A,T_SCC_BODY_TEXT_LINE autolearn=disabled version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on hergotha.csail.mit.edu X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.82 / 15.00]; NEURAL_HAM_SHORT(-1.00)[-0.999]; NEURAL_HAM_LONG(-1.00)[-0.999]; NEURAL_HAM_MEDIUM(-0.92)[-0.920]; FORGED_SENDER(0.30)[wollman@bimajority.org,wollman@hergotha.csail.mit.edu]; R_SPF_ALLOW(-0.20)[+ip6:2001:470:1f06:ccb::2]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[bimajority.org : SPF not aligned (relaxed), No valid DKIM,none]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; TO_DN_SOME(0.00)[]; TAGGED_RCPT(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEFALL_USER(0.00)[wollman]; FROM_HAS_DN(0.00)[]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[stable@freebsd.org]; RCVD_COUNT_TWO(0.00)[2]; FROM_NEQ_ENVFROM(0.00)[wollman@bimajority.org,wollman@hergotha.csail.mit.edu]; FREEMAIL_TO(0.00)[gmail.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[] X-Rspamd-Queue-Id: 4TrXV53zh0z4SdX < I believe this explains why vn_copy_file_range sometimes takes much > longer than a second: our servers often have lots of data waiting to > be written to disk, and if the file being copied was recently modified > (and so is dirty), this might take several seconds. I've set > vfs.zfs.dmu_offset_next_sync=0 on the server that was hurting the most > and am watching to see if we have more freezes. > If this does the trick, then I can delay deploying a new kernel until > April, after my upcoming vacation. Since zeroing dmu_offset_next_sync, I've seen about 8000 copy operations on the problematic server and no NFS work stoppages due to the copy. I have observed a few others in a similar posture, where one client wants to ExchangeID and is waiting for other requests to drain, but nothing long enough to cause a service problem.[1] I think in general this choice to prefer "accurate" but very slow hole detection is a poor choice on the part of the OpenZFS developers, but so long as we can disable it, I don't think we need to change anything in the NFS server itself. It would be a good idea longer term to figure out a lock-free or synchronization-free way of handling these client session accept/teardown operations, because it is still a performance degradation, just not disruptive enough for users to notice. -GAWollman [1] Saw one with a slow nfsrv_readdirplus and another with a bunch of threads blocked on an upcall to nfsuserd. From nobody Fri Mar 8 09:35:45 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TrgyZ636Pz5DGBW for ; Fri, 8 Mar 2024 09:35:50 +0000 (UTC) (envelope-from void@f-m.fm) Received: from wfhigh4-smtp.messagingengine.com (wfhigh4-smtp.messagingengine.com [64.147.123.155]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TrgyZ0qjVz437Y for ; Fri, 8 Mar 2024 09:35:50 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm1 header.b=WxzAMjUp; dkim=pass header.d=messagingengine.com header.s=fm1 header.b=gjo0jsct; dmarc=pass (policy=none) header.from=f-m.fm; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 64.147.123.155 as permitted sender) smtp.mailfrom=void@f-m.fm Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailfhigh.west.internal (Postfix) with ESMTP id 803FF18000D2 for ; Fri, 8 Mar 2024 04:35:48 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Fri, 08 Mar 2024 04:35:48 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :message-id:mime-version:reply-to:subject:subject:to:to; s=fm1; t=1709890548; x=1709976948; bh=CWk8IFinuytDiAkYdPln5e+G+OCxcfRR raQwe192OWE=; b=WxzAMjUp6vFc8LXlW2oWyJ5tQYOZCLYKecjIuP9Ad9FWHUQU +1tmg4DUOz41dMOsqGHnVnFCVSpc94GbX4jS1XqSAU4dFnhfVmGd31Qh+bM0yHld NTEb7fCHRFeLMMllx7Zw6Ng7/Cazc+4vkd4s9ynkQY6I9OuviGyuETYm1CKs2sPR VOdt/410QW+wTmo9a5otGi5LTdsRf6WKC8jf35yFBxu6Q+42dWM7D7SfeEbCwfQn Z/3Qr5Tsy3pNUkInCOtewmCWfef9o1kNthEh3WByqIbRttGB5EeiTTedRornSghR qEb9hVYQ9zAOXCZ/VvApwpYP+OoE3HNktKyxmw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:message-id :mime-version:reply-to:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1709890548; x=1709976948; bh=CWk8IFinuytDiAkYdPln5e+G+OCxcfRRraQ we192OWE=; b=gjo0jsctURgSG9/rtsaUZiBCLy5C3iesjPe0F5dJA7SSlwgo/e+ h/9DV1QSBhsfc2rgGftXFRORIyxblTGoSUMFaZ45JzU3pk+yhUrbVEVOKKu6v8vu qFCB5cPmZ/GYVe/me1IcAL64t5/nSdq7pZDCxoKdYltVgQJ0MCdxkmi4M9FSZUx7 6deM+hLCZzcTS5nrwQNTqXXP+KeR5fCTgpGzsvk+od+Mu4+SzLj1bTOQ+rO/DPf2 gjo39RwObhNlfy+LRfOyDsvCslZG9FIJBSGRxaSqML2dfxzHfEBHY2GkS9+PeB5L nyb03kgyZ3rgzvnDSViUyXg3K7CmvC1K7og== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrieehgddtiecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfggtggusehttdertddttd dvnecuhfhrohhmpehvohhiugcuoehvohhiugesfhdqmhdrfhhmqeenucggtffrrghtthgv rhhnpeevudffiedvffffgffhgeefjeefffdtieetheetkeefhfdvfefgtedtueehgeffue enucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehvohhi ugesfhdqmhdrfhhm X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Fri, 8 Mar 2024 04:35:47 -0500 (EST) Date: Fri, 8 Mar 2024 09:35:45 +0000 From: void To: freebsd-stable@freebsd.org Subject: KVA_PAGES Message-ID: Mail-Followup-To: freebsd-stable@freebsd.org List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.83 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.90)[-0.903]; SUBJ_ALL_CAPS(0.68)[9]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm1,messagingengine.com:s=fm1]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.128/27]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[64.147.123.155:from]; RCPT_COUNT_ONE(0.00)[1]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; FREEMAIL_FROM(0.00)[f-m.fm]; FROM_HAS_DN(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; ASN(0.00)[asn:29838, ipnet:64.147.123.0/24, country:US]; RCVD_VIA_SMTP_AUTH(0.00)[]; MISSING_XM_UA(0.00)[]; DWL_DNSWL_NONE(0.00)[messagingengine.com:dkim] X-Rspamd-Queue-Id: 4TrgyZ0qjVz437Y Hi, In 14-stable on amd64, what's the default value for KVA_PAGES ? I'm asking because WINE says something about it: "Some ZFS tuning guides recommend setting KVA_PAGES=512 in your kernel configuration. This is incompatible with Wine. The maximum possible is KVA_PAGES=500, which should still be enough for ZFS." I can't find a KVA_PAGES setting in /usr/src/sys/amd64/conf/* -- From nobody Fri Mar 8 10:25:24 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Trj405dHDz5CrL6 for ; Fri, 8 Mar 2024 10:25:36 +0000 (UTC) (envelope-from SRS0=WhEN=KO=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Trj3z5NGmz47Sn for ; Fri, 8 Mar 2024 10:25:35 +0000 (UTC) (envelope-from SRS0=WhEN=KO=quip.cz=000.fbsd@elsa.codelab.cz) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=quip.cz header.s=private header.b=w5eUdSEg; dkim=pass header.d=quip.cz header.s=private header.b=OhgXAWp5; dmarc=none; spf=none (mx1.freebsd.org: domain of "SRS0=WhEN=KO=quip.cz=000.fbsd@elsa.codelab.cz" has no SPF policy when checking 94.124.105.4) smtp.mailfrom="SRS0=WhEN=KO=quip.cz=000.fbsd@elsa.codelab.cz" Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 97D19D78C1 for ; Fri, 8 Mar 2024 11:25:26 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quip.cz; s=private; t=1709893526; bh=3g6grXBGtOzCNr++/hMrr4oSpzUjhpgOzUbcHZ8cu10=; h=Date:Subject:To:References:From:In-Reply-To; b=w5eUdSEgxVLP0Kay5KmgUVVravAn0Tv91N0CRz+E+La6E8OerDjaeQmq/OKcfYLef ZmqVcwgbLskcmtX1Oq+q8jpW4Uo7JJktfB72ZMu7pwsTyLKfnPSyGPuYOvxsXZbWeB +uzxIZPf3gizEkVAj42+AzEYviuhO5oEdq3yO/sM= Received: from [192.168.145.49] (ip-89-177-27-225.bb.vodafone.cz [89.177.27.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 80BB5D7893 for ; Fri, 8 Mar 2024 11:25:25 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quip.cz; s=private; t=1709893525; bh=3g6grXBGtOzCNr++/hMrr4oSpzUjhpgOzUbcHZ8cu10=; h=Date:Subject:To:References:From:In-Reply-To; b=OhgXAWp5D5bjT+xxSl/7s9pyEjZTSlEwe6X5mdmDU5dHHyz12P74HdB/6MOhMV7wu T2QKPvf/RWWs5HPT8AKPSiQ3yHjPL0GIsjUXiO5os3CJRciN6TR27PPUHYpxit2yby sDILGICzFU1Nca5A/UnwkPlFXBh55TELlD+VUwKk= Message-ID: <615f6011-c7df-4be2-bdad-f0c5dd67ee16@quip.cz> Date: Fri, 8 Mar 2024 11:25:24 +0100 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: KVA_PAGES To: freebsd-stable@freebsd.org References: Content-Language: en-US From: Miroslav Lachman <000.fbsd@quip.cz> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.31 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; SUBJ_ALL_CAPS(0.68)[9]; FORGED_SENDER(0.30)[000.fbsd@quip.cz,SRS0=WhEN=KO=quip.cz=000.fbsd@elsa.codelab.cz]; R_DKIM_ALLOW(-0.20)[quip.cz:s=private]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; ARC_NA(0.00)[]; R_SPF_NA(0.00)[no SPF record]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[quip.cz]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; RCVD_TLS_LAST(0.00)[]; FROM_NEQ_ENVFROM(0.00)[000.fbsd@quip.cz,SRS0=WhEN=KO=quip.cz=000.fbsd@elsa.codelab.cz]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; DKIM_TRACE(0.00)[quip.cz:+] X-Rspamd-Queue-Id: 4Trj3z5NGmz47Sn On 08/03/2024 10:35, void wrote: > In 14-stable on amd64, what's the default value for KVA_PAGES ? > I'm asking because WINE says something about it: > > "Some ZFS tuning guides recommend setting KVA_PAGES=512 in your kernel > configuration.  This is incompatible with Wine.  The maximum possible > is KVA_PAGES=500, which should still be enough for ZFS." > > I can't find a KVA_PAGES setting in /usr/src/sys/amd64/conf/* According to this page https://wiki.freebsd.org/KVA_PAGES it was used only on i386 platform. You can try to search actual kernel config by "config" command # config -x /boot/kernel/kernel | grep -i kva_pages (it is empty on amd64) And if you see this message only as pkg message when yoy install Wine, you can ignore it. https://forums.freebsd.org/threads/wine-kva_pages-512-in-your-kernel-configuration-this-is-incompatible-with-wine.71416/ Kind regards Miroslav Lachman From nobody Fri Mar 8 14:41:09 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TrplD2Qm7z5DGcr for ; Fri, 8 Mar 2024 14:41:28 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pf1-x42b.google.com (mail-pf1-x42b.google.com [IPv6:2607:f8b0:4864:20::42b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TrplC290Gz4cgv; Fri, 8 Mar 2024 14:41:27 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=BQdPYASX; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2607:f8b0:4864:20::42b as permitted sender) smtp.mailfrom=rick.macklem@gmail.com Received: by mail-pf1-x42b.google.com with SMTP id d2e1a72fcca58-6e5eaf5bb3eso1856938b3a.3; Fri, 08 Mar 2024 06:41:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709908886; x=1710513686; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=E8jRn0uKOQyNzsTA0nVQDmDrQKptTeElLqkv6Nj1W0w=; b=BQdPYASXz3UHfjTElZjd3Oo1dNKTV8bXJ1r89QtPQwppesX2vKHLgMvPE/VF8Gvm0z I/AQsOynGpuISmGwlqgFoI4v6mJJWXL9lBuG8jgwa8ChrAdBCrBSagOmCpu1SPm6aLCt aUSHR5YMs37EkiWjuW+kNyj3p7T6/if14LXpET4zkV5q45ctZVoI78zmuyEI3I4jtQKv J7VYPUZKLlLP5LipZ+j5CRIe8NgyXV59VhF8zLttz70SLVHoP2LY8WOkc8Nrqf0UK5Wv dkAAESDudx8B8ep1yKVBzLqgEkvQiDnLhTVnqKDbx6ZYvV7uuGbTUgrODfnYlWIfOJqm TG3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709908886; x=1710513686; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=E8jRn0uKOQyNzsTA0nVQDmDrQKptTeElLqkv6Nj1W0w=; b=aegNoi6qMy700D5y4kg153LcFEhBDD6Bx+q//y5dgYqY1RqPFS+0D0gjCZF5gLZ7uc q10EUxa9aaHhEjGO1NYKub0o7O65WyIkXA6920p6s65A0xOZfqVe5zHpfgCRCf5IdQZf FNe+ZVh/baDWYFQe82FCL00kGw1JsEj59xbfkUx8f1fHSw0KY+Yx3pjJrghAyxPLscIV Qzg5e8dQhyLqt5Rk4x9y9cE4KxnyDOslOJVVflR6BjvKoC6pkRSNuWxWwOWTVCSiv6IA h0YcBIcPLxbHTkyELnoV0CdYXTpehfCovQi4n6jb9STQiK9eR0Ac1+9IgAqWFH05Oh5w eagg== X-Forwarded-Encrypted: i=1; AJvYcCW2xNFtZ6skeQXu6xdHbkIYMcL2Ytu6aK/FVcP8xd56nAQ2uDXKheWa9ADgI6WZDm85V9aLKD3jThAEaiI4ZtIFE7s= X-Gm-Message-State: AOJu0Yw0srU15IBuIlT8juduhEIJQ2vKqJMe4eT189xTNcmXtb7Dxyyd aKheyIcrGRHZDGG5Y8HA+KOrLCN8mzJ0N+VS/ZOXjFd+u3enBQoyiM0GaasXK+86PVuAM8Y+dm0 B/+DCloNz1EXNqJrjli24yJMT+W36yMg= X-Google-Smtp-Source: AGHT+IHsFtDkLW7g7NAElHV4tU8oQzgx+D10sen3aVeMsxhLUYyGDbHm/l57L5/2HnpyRQ9DK3NbgUu2OV6Mrqd/02I= X-Received: by 2002:a05:6a20:918b:b0:1a1:86e8:317e with SMTP id v11-20020a056a20918b00b001a186e8317emr1336380pzd.33.1709908885523; Fri, 08 Mar 2024 06:41:25 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: <1020651467.1592.1709280020993@localhost> <1608503215.4731.1709633602802@localhost> <1277770972.4770.1709725576695@localhost> In-Reply-To: <1277770972.4770.1709725576695@localhost> From: Rick Macklem Date: Fri, 8 Mar 2024 06:41:09 -0800 Message-ID: Subject: Re: 13-stable NFS server hang To: Ronald Klop Cc: rmacklem@freebsd.org, Garrett Wollman , stable@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: --- X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; TAGGED_FROM(0.00)[]; TO_DN_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; MISSING_XM_UA(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MLMMJ_DEST(0.00)[stable@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::42b:from]; FREEMAIL_ENVFROM(0.00)[gmail.com] X-Rspamd-Queue-Id: 4TrplC290Gz4cgv On Wed, Mar 6, 2024 at 3:46=E2=80=AFAM Ronald Klop w= rote: > > > Van: Rick Macklem > Datum: dinsdag, 5 maart 2024 15:43 > Aan: Ronald Klop > CC: rmacklem@freebsd.org, Garrett Wollman , stabl= e@freebsd.org > Onderwerp: Re: 13-stable NFS server hang > > On Tue, Mar 5, 2024 at 6:34AM Rick Macklem wrote= : > > > > On Tue, Mar 5, 2024 at 2:13AM Ronald Klop wrote: > > > > > > > > > Van: Rick Macklem > > > Datum: vrijdag, 1 maart 2024 15:23 > > > Aan: Ronald Klop > > > CC: Garrett Wollman , stable@freebsd.org, rma= cklem@freebsd.org > > > Onderwerp: Re: 13-stable NFS server hang > > > > > > On Fri, Mar 1, 2024 at 12:00AM Ronald Klop wro= te: > > > > > > > > Interesting read. > > > > > > > > Would it be possible to separate locking for admin actions like a = client mounting an fs from traffic flowing for file operations? > > > Well, the NFS server does not really have any concept of a mount. > > > What I am referring to is the ClientID maintained for NFSv4 mounts, > > > which all the open/lock/session/layout state hangs off of. > > > > > > For most cases, this state information can safely be accessed/modifie= d > > > via a mutex, but there are three exceptions: > > > - creating a new ClientID (which is done by the ExchangeID operation) > > > and typically happens when a NFS client does a mount. > > > - delegation Recall (which only happens when delegations are enabled) > > > One of the reasons delegations are not enabled by default on the > > > FreeBSD server. > > > - the DestroyClientID which is typically done by a NFS client during = dismount. > > > For these cases, it is just too difficult to do them without sleeping= . > > > As such, there is a sleep lock which the nfsd threads normally acquir= e shared > > > when doing NFSv4 operations, but for the above cases the lock is aqui= red > > > exclusive. > > > - I had to give the exclusive lock priority over shared lock > > > acquisition (it is a > > > custom locking mechanism with assorted weirdnesses) because without > > > that someone reported that new mounts took up to 1/2hr to occur. > > > (The exclusive locker waited for 30min before all the other nfsd th= reads > > > were not busy.) > > > Because of this priority, once a nfsd thread requests the exclusive= lock, > > > all other nfsd threads executing NFSv4 RPCs block after releasing t= heir > > > shared lock, until the exclusive locker releases the exclusive lock= . > > > > > > In summary, NFSv4 has certain advantages over NFSv3, but it comes > > > with a lot of state complexity. It just is not feasible to manipulate= all that > > > state with only mutex locking. > > > > > > rick > > > > > > > > > > > Like ongoing file operations could have a read only view/copy of th= e mount table. Only new operations will have to wait. > > > > But the mount never needs to wait for ongoing operations before loc= king the structure. > > > > > > > > Just a thought in the morning > > > > > > > > Regards, > > > > Ronald. > > > > > > > > Van: Rick Macklem > > > > Datum: 1 maart 2024 00:31 > > > > Aan: Garrett Wollman > > > > CC: stable@freebsd.org, rmacklem@freebsd.org > > > > Onderwerp: Re: 13-stable NFS server hang > > > > > > > > On Wed, Feb 28, 2024 at 4:04PM Rick Macklem wrote: > > > > > > > > > > On Tue, Feb 27, 2024 at 9:30PM Garrett Wollman wrote: > > > > > > > > > > > > Hi, all, > > > > > > > > > > > > We've had some complaints of NFS hanging at unpredictable inter= vals. > > > > > > Our NFS servers are running a 13-stable from last December, and > > > > > > tonight I sat in front of the monitor watching `nfsstat -dW`. = I was > > > > > > able to clearly see that there were periods when NFS activity w= ould > > > > > > drop *instantly* from 30,000 ops/s to flat zero, which would la= st > > > > > > for about 25 seconds before resuming exactly as it was before. > > > > > > > > > > > > I wrote a little awk script to watch for this happening and run > > > > > > `procstat -k` on the nfsd process, and I saw that all but two o= f the > > > > > > service threads were idle. The three nfsd threads that had non= -idle > > > > > > kstacks were: > > > > > > > > > > > > PID TID COMM TDNAME KSTACK > > > > > > 997 108481 nfsd nfsd: master mi_switch = sleepq_timedwait _sleep nfsv4_lock nfsrvd_dorpc nfssvc_program svc_run_inte= rnal svc_run nfsrvd_nfsd nfssvc_nfsd sys_nfssvc amd64_syscall fast_syscall_= common > > > > > > 997 960918 nfsd nfsd: service mi_switch = sleepq_timedwait _sleep nfsv4_lock nfsrv_setclient nfsrvd_exchangeid nfsrvd= _dorpc nfssvc_program svc_run_internal svc_thread_start fork_exit fork_tram= poline > > > > > > 997 962232 nfsd nfsd: service mi_switch = _cv_wait txg_wait_synced_impl txg_wait_synced dmu_offset_next zfs_holey zfs= _freebsd_ioctl vn_generic_copy_file_range vop_stdcopy_file_range VOP_COPY_F= ILE_RANGE vn_copy_file_range nfsrvd_copy_file_range nfsrvd_dorpc nfssvc_pro= gram svc_run_internal svc_thread_start fork_exit fork_trampoline > > > > > > > > > > > > I'm suspicious of two things: first, the copy_file_range RPC; s= econd, > > > > > > the "master" nfsd thread is actually servicing an RPC which req= uires > > > > > > obtaining a lock. The "master" getting stuck while performing = client > > > > > > RPCs is, I believe, the reason NFS service grinds to a halt whe= n a > > > > > > client tries to write into a near-full filesystem, so this prob= lem > > > > > > would be more evidence that the dispatching function should not= be > > > > > > mixed with actual operations. I don't know what the clients ar= e > > > > > > doing, but is it possible that nfsrvd_copy_file_range is holdin= g a > > > > > > lock that is needed by one or both of the other two threads? > > > > > > > > > > > > Near-term I could change nfsrvd_copy_file_range to just > > > > > > unconditionally return NFSERR_NOTSUP and force the clients to f= all > > > > > > back, but I figured I would ask if anyone else has seen this. > > > > > I have attached a little patch that should limit the server's Cop= y size > > > > > to vfs.nfsd.maxcopyrange (default of 10Mbytes). > > > > > Hopefully this makes sure that the Copy does not take too long. > > > > > > > > > > You could try this instead of disabling Copy. It would be nice to= know if > > > > > this is suffciient? (If not, I'll probably add a sysctl to disabl= e Copy.) > > > > I did a quick test without/with this patch,where I copied a 1Gbyte = file. > > > > > > > > Without this patch, the Copy RPCs mostly replied in just under 1sec > > > > (which is what the flag requests), but took over 4sec for one of th= e Copy > > > > operations. This implies that one Read/Write of 1Mbyte on the serve= r > > > > took over 3 seconds. > > > > I noticed the first Copy did over 600Mbytes, but the rest did about= 100Mbytes > > > > each and it was one of these 100Mbyte Copy operations that took ove= r 4sec. > > > > > > > > With the patch, there were a lot more Copy RPCs (as expected) of 10= Mbytes > > > > each and they took a consistent 0.25-0.3sec to reply. (This is a te= st of a local > > > > mount on an old laptop, so nowhere near a server hardware config.) > > > > > > > > So, the patch might be sufficient? > > > > > > > > It would be nice to avoid disabling Copy, since it avoids reading t= he data > > > > into the client and then writing it back to the server. > > > > > > > > I will probably commit both patches (10Mbyte clip of Copy size and > > > > disabling Copy) to main soon, since I cannot say if clipping the si= ze > > > > of the Copy will always be sufficient. > > > > > > > > Pleas let us know how trying these patches goes, rick > > > > > > > > > > > > > > rick > > > > > > > > > > > > > > > > > -GAWollman > > > > > > > > > > > > > > > > > > > > ________________________________ > > > > > > > > > > > > > > > > > > > ________________________________ > > > > > > > > > > > > Hi Rick, > > > > > > You are much more into the NFS code than I am so excuse me if what I'= m speaking about does not make sense. > > > > > > I was reading nfsrvd_compound() which calls nfsrvd_copy_file_range() = via the nfsrv4_ops2 structure. > > > Nfsrvd_compound holds a lock or refcount on nfsv4rootfs_lock during t= he whole operation. Which is why nfsrv_setclient() is waiting in this speci= fic case of "NFS server hang". > > > > > > But I don't see what is being modified on the nfsdstate after the IO = operation ends. Or why the IO operation itself needs the lock to the nfsdst= ate. IMHO the in-progress IOs will happen anyway regardless of the nfsdstat= e. Changes to the nfsdstate during an IO operation would not affect the ong= oing IO operation. > > > Wouldn't it be possible to lock the nfsv4rootfs_lock, do checks on or= modify the nfsdstate as needed, unlock and then do the IO operation? That = would remove a lot of the possible lock contention during (u)mount. > > > Otherwise, if we do modify the nfsdstate after the IO operation, isn'= t it possible to relock nfsv4rootfs_lock after the IO operation finishes? > > Well, there are a couple of reasons. Every implementation has design tr= adeoffs: > > 1 - A NFSv4 RPC is a compound, which can be a pretty arbitrary list of > > operations. > > As such, the NFSv4 server does not know if an open/byte range > > lock is coming > > after the operation it is currently performing, since the > > implementation does not > > pre-parse the entire compound. (I had a discussion w.r.t. > > pre-parsing with one of > > the main Linux knfsd server maintainers and he noted that he was > > not aware of > > any extant server that did pre-parse the compound. Although it > > would be useful > > for the server to have the ordered list of operations before > > commencing the RPC, > > we both agreed it was too hard to implement. > > --> It could possibly unlock/relock later, but see #2. Also, if > > relocking took a long time, > > it would result in the compound RPC taking too long (see belo= w). > > 2 - For NFSv4.1/4.2 almost all RPCs are handled by a session. One non-a= rbitrary > > part of almost all NFSv4.1/4.2 RPCs is that the Sequence > > operation (the one that > > handles the session) must come first. > > Session(s) are associated with the ClientID, which means the > > ClientID and the > > session must not go away while the compound RPC is in progress. > > - This is ensured by the shared lock on the ClientID (that > > nfsv4rootfh_lock). > > Since 99.99% of operations can be done with the shared lock, I do not t= hink > > there is a lot of contention. > > > > Although there is nothing wired down in the RFCs, there is an understan= ding > > in the NFSv4 community that a server should reply to an RPC in a reason= able > > time. Typically assumed to be 1-2sec. If the server does this, then a d= elay for > > the rare case of a new ClientID shouldn't be a big problem? > > (The is also delegation recall, which is one reason why delegations > > are not enabled > > by default.) > > > > Btw, the RFC does define an asynchronous Copy, where the operation repl= ies > > as soon as the copy is started and the server notifies the client of co= mpletion > > later. I have not implemented this, because it introduces complexities = that > > I do not want to deal with. > > For example, what happens when the server crashes/reboots while the cop= y > > is in progress? The file is left in a non-deterministic state, dependin= g on what > > the client does when it does not receive the completion notify. > > > Oh, I should also note that the "shared lock" is actually called a > reference count > in the code and is there to ensure that the ClientID/Session does not go = away > during execution of the compound. > > The problem in this case (which I should revisit) was that I could not fi= gure > out how to safely add a new ClientID while other nfsd threads were in pro= gress > performing other RPCs. Due to retries etc, there might be another RPC > in progress > using the ClientID. > > One thing to note here is that the identity of the ClientID > is not known until the Sequence operation has been performed. (And there = is > cruft for NFSv4.0, since it does not have a Sequence operation.) > As such, the RPC must be in progress before it is known. > > > rick > > > > > > I hope this makes any sense and thanks for all your work on the NFS c= ode. > > > > > > Regards, > > > Ronald. > > > > > > > > > > ________________________________ > > > > Hi Rick, > > Thanks for the elaborate answer. > > Would it make sense to have the current RPC/compound have a lock on its C= lientID/session, but not on the whole nfsdstate (nfsv4rootfs_lock)? Nope. It is the structure of the linked lists (an open is in three of them) that defines the state relationship for open_owners/opens/lock_owners/locks. The sessions are the exception. Since the code mostly updates contents of t= hem, each session structure has its own mutex and a refcnt to avoid use after fr= ee, Then there is a mutex for each hash list that is used to find the session. The code for the clientID was first written over 20years ago (NFSv4.0 calls= the operation SetClientID, but it does the same thing.) There is a confirmation= step done by a CreateSession with a correct seq#. As I've said, I'll look and see if I can figure out how o do it without the exclusive lock. > > So concurrent requests like a new mount creating a new ClientID can go on= in parallel, but removing or modifying the locked ClientID will wait for t= he lock. > > Or am I thinking too simple still? > > Regards, > Ronald. > From nobody Fri Mar 8 15:54:34 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TrrMs64kCz5DNDj for ; Fri, 8 Mar 2024 15:54:49 +0000 (UTC) (envelope-from bsd-lists@bsdforge.com) Received: from udns.ultimatedns.net (udns.ultimatedns.net [24.113.41.81]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "ultimatedns.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TrrMr6FVZz4kmT for ; Fri, 8 Mar 2024 15:54:48 +0000 (UTC) (envelope-from bsd-lists@bsdforge.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of bsd-lists@bsdforge.com has no SPF policy when checking 24.113.41.81) smtp.mailfrom=bsd-lists@bsdforge.com Received: from ultimatedns.net (localhost [127.0.0.1]) by udns.ultimatedns.net (8.16.1/8.16.1) with ESMTP id 428FsZMc076074 for ; Fri, 8 Mar 2024 07:54:41 -0800 (PST) (envelope-from bsd-lists@bsdforge.com) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Date: Fri, 08 Mar 2024 07:54:34 -0800 From: Chris To: freebsd-stable@freebsd.org Subject: Re: KVA_PAGES In-Reply-To: References: User-Agent: UDNSMS/17.0 Message-ID: <49139dd3c63580ad8cb85f2f3ce63b48@bsdforge.com> X-Sender: bsd-lists@bsdforge.com Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Bar: / X-Rspamd-Pre-Result: action=no action; module=multimap; Matched map: local_wl_ip X-Spamd-Result: default: False [0.68 / 15.00]; SUBJ_ALL_CAPS(0.68)[9]; MIME_GOOD(-0.10)[text/plain]; ONCE_RECEIVED(0.10)[]; local_wl_ip(0.00)[24.113.41.81]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; R_SPF_NA(0.00)[no SPF record]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; ASN(0.00)[asn:11404, ipnet:24.113.0.0/16, country:US]; FROM_HAS_DN(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+] X-Rspamd-Queue-Id: 4TrrMr6FVZz4kmT On 2024-03-08 01:35, void wrote: > Hi, > > In 14-stable on amd64, what's the default value for KVA_PAGES ? > I'm asking because WINE says something about it: > > "Some ZFS tuning guides recommend setting KVA_PAGES=512 in your kernel > configuration. This is incompatible with Wine. The maximum possible > is KVA_PAGES=500, which should still be enough for ZFS." > > I can't find a KVA_PAGES setting in /usr/src/sys/amd64/conf/* Apologies if this has already been answered. But a trip to /usr/src/sys followed by a # grep -F KVA_PAGES -RH . ./conf/options.i386:KVA_PAGES opt_global.h ./i386/i386/pmap.c: * KPTmap is created that can support KVA_PAGES page table pages. ./i386/i386/pmap.c: SYSMAP(pt_entry_t *, KPTD, KPTmap, KVA_PAGES) ./i386/include/pmap.h:#define NKPDE (KVA_PAGES) /* number of page tables/pde's */ ./i386/include/pmap_nopae.h:#define KVA_PAGES (256*4) ./i386/include/pmap_pae.h:#define KVA_PAGES (512*4) HTH --Chris From nobody Sat Mar 9 00:01:00 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Ts38s6P7hz5CMBH for ; Sat, 9 Mar 2024 00:01:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Ts38s5DZSz4Kf8 for ; Sat, 9 Mar 2024 00:01:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1709942461; a=rsa-sha256; cv=none; b=k9FBXjsXhbVD+c/aYCZyRrc7RMLuVgAamr8h3wM6QdaMIKX4LaUOENjPuIcl7PNv38iAhH +j1F/qgC11TyyVUp9lUyTMrxth9luwjORAZO0wFGEg7Oj8vlgwLaNnMbNsCzZ8/9wKyYWW 4HJVBsEYYT/2cda3bUgYWjpDRgxDGisVObhNLSaM/qYhoxrdWbj3nwJ0b1sZSLpN+sRDq+ 3zPB5CsCWp163I0sTwq/ZSpk2iNjbvJctkTBem5mJ14iMmRu0GaH6qiSYWx5DfNs6aP42I wkRvC9F6rXsKqTQuYPQTfEmPkqnTrup3rnydpb5reKH9zM/J1FYJf0d2hcVumw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1709942461; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZR8aDw+lehpRUOzwQ7lxIb5oPxhtAhSSddROViW0naI=; b=gKG7xRwTNG3IgHhHpfPMG+4uhnPxNpNcybXMAthmdNFfgeBoYlLOJY8XLSRj+GtkeCE/ld BemPyzGoqEU8GDb6A2chvrjTduJk0ZI25KETWQyoFAWXyzom4JSIEzxZHL3wJEjD6Wsz6L jb58hSRdHGDKpphQSNbS5wEhIJEcoB3xQrNeukhP1AYW1veqLFHBXqM2hP0Noh5jsluNnq 1L3b3MjlwKQY5CfqDBd2QEX/YXGm+QXQUcMeSkYs5Hddu1Zc0puOkQcw82A+EeYlVXikP/ ojiMn+U3szfProjBw2BO+7FVWB99mrcZclt86nl/PkB9uuNaX13Kcou0N1j4UA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Ts38s4rVjzfl6 for ; Sat, 9 Mar 2024 00:01:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 429011Ll063963 for ; Sat, 9 Mar 2024 00:01:01 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 429011N5063958 for stable@FreeBSD.org; Sat, 9 Mar 2024 00:01:01 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: stable@FreeBSD.org Subject: [Bug 271238] mpr (LSI SAS3816) driver not finding ses devices in HP D6020 enclosures Date: Sat, 09 Mar 2024 00:01:00 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.4-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mgrooms@shrew.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D271238 mgrooms@shrew.net changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mgrooms@shrew.net --- Comment #4 from mgrooms@shrew.net --- I'm seeing strange behavior with mpr and an LSI SAS3816 16i adapter where e= very 8th drive in not seen of my 16 drive array. After swapping out for an older SAS3224 16i adapter, the drives are identified without issue. I just wanted= to add this to the open ticket since the problem sounded similar. I have screenshots and hardware to test with if that would be helpful. --=20 You are receiving this mail because: You are on the CC list for the bug.= From nobody Sat Mar 9 00:46:48 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Ts4B419pbz5CR8t for ; Sat, 9 Mar 2024 00:47:08 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pg1-x52b.google.com (mail-pg1-x52b.google.com [IPv6:2607:f8b0:4864:20::52b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Ts4B31HjPz4QMD for ; Sat, 9 Mar 2024 00:47:07 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pg1-x52b.google.com with SMTP id 41be03b00d2f7-5e152c757a5so1109335a12.2 for ; Fri, 08 Mar 2024 16:47:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709945225; x=1710550025; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=2+ffu+lzzwULPn//NO10EgXT94vD2C7bOYMybCXh1ow=; b=i6yTpWr+w9lPg6Z7dcwW/LNaHT5pBCSPfVqxCsWJaqxC6akPed2y0L9wLVU3sTCxAr rdqyTXtWNgqPAbCV0pGjPfOghbA5OzKvxNeRUtgx+Ub4N9Z3dWU8/gA9NSjE7O6U+bQY Ta0hGpwHobQ2hCgZixC94LxdMbVcnlbSpK9gT/oCrsRYbk+EXuOhtMq4QSGgam2PfYcb DAf6vyu1ktNeSIHES+YYge8NpDouNpQhDgQ+kGfKJ4XS1JdyakDlzuRI8E36NMh9+6gW HfChgFZuRaJoPG7cUR2rP1e6fphuFy3VAruIfOQXoRk2gdq8iShnyIoKKw5fYs2blJrD 7khA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709945225; x=1710550025; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2+ffu+lzzwULPn//NO10EgXT94vD2C7bOYMybCXh1ow=; b=KQVhrI3NZ27W6iVXDHzJ/eTFDBBChne8ntU6PKaiQq+O0I1ubQZBHPiOtJ56//ye1w 9SxjQ9S3LrGcKF+7GcSX8+PdhKgez1zgOqDk/ycTJW1geMFo76sBqtd0NDDS0lVDgsH/ aNIgJYf4NBlGAMcEBzEsZ3Eq1HDIC1M92aYBJxbQotcDdQwWc1QGtyjBTb0LuhEKrlNw 0kK4dxuOokMOOl3W3ejQS/xyFSp33liiRWOctJ1FY4pyr3SuUUNUDotzfX6lr1vpKQTM voz3wJ/GOXTEvqb76+Uk2NPtM4CKE48FurKbLZwjrS/gMo/lBh8A0KQ6AV9sSLOzW/7k bvzA== X-Gm-Message-State: AOJu0YylsrGNh3GIcaLeAaRqncH2Xoz/iXgiua+ZxDuFEY86U854+Tj3 ehZG71uMJuNT/TJiU0oSmbJLL2ku58keC1oTP2wcN8PJfOm4vpueJtMth8/DeFdnr0BgC6hR4VB cYcb+R8PUU+c08D0i5za5GCVtmdtrCGg= X-Google-Smtp-Source: AGHT+IHcCWVoGphdIj1DaT/Yua9T0uv4JMOxGi7us0az0Q0jAPFwbL9WJPzgLQyoyzqSpMaU3paAFIRANSbS2uQdFtY= X-Received: by 2002:a05:6a20:5485:b0:1a1:4d8b:6f2c with SMTP id i5-20020a056a20548500b001a14d8b6f2cmr367529pzk.2.1709945225262; Fri, 08 Mar 2024 16:47:05 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: <26078.50375.679881.64018@hergotha.csail.mit.edu> <26083.64612.717082.366639@hergotha.csail.mit.edu> <26090.36102.379531.160926@hergotha.csail.mit.edu> In-Reply-To: <26090.36102.379531.160926@hergotha.csail.mit.edu> From: Rick Macklem Date: Fri, 8 Mar 2024 16:46:48 -0800 Message-ID: Subject: Re: 13-stable NFS server hang To: Garrett Wollman Cc: stable@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4Ts4B31HjPz4QMD On Thu, Mar 7, 2024 at 7:59=E2=80=AFPM Garrett Wollman wrote: > > < > > I believe this explains why vn_copy_file_range sometimes takes much > > longer than a second: our servers often have lots of data waiting to > > be written to disk, and if the file being copied was recently modified > > (and so is dirty), this might take several seconds. I've set > > vfs.zfs.dmu_offset_next_sync=3D0 on the server that was hurting the mos= t > > and am watching to see if we have more freezes. > > > If this does the trick, then I can delay deploying a new kernel until > > April, after my upcoming vacation. > > Since zeroing dmu_offset_next_sync, I've seen about 8000 copy > operations on the problematic server and no NFS work stoppages due to > the copy. I have observed a few others in a similar posture, where > one client wants to ExchangeID and is waiting for other requests to > drain, but nothing long enough to cause a service problem.[1] > > I think in general this choice to prefer "accurate" but very slow hole > detection is a poor choice on the part of the OpenZFS developers, but > so long as we can disable it, I don't think we need to change anything > in the NFS server itself. So the question is... How can this be documented? In the BUGS section of "man nfsd" maybe. What do others think? > It would be a good idea longer term to > figure out a lock-free or synchronization-free way of handling these > client session accept/teardown operations, because it is still a > performance degradation, just not disruptive enough for users to > notice. Yes, as I've noted, it is on my todo list to take a look at it. Good sleuthing, rick > > -GAWollman > > [1] Saw one with a slow nfsrv_readdirplus and another with a bunch of > threads blocked on an upcall to nfsuserd. From nobody Sat Mar 9 12:14:00 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TsMR55DW0z5D3W8 for ; Sat, 9 Mar 2024 12:14:25 +0000 (UTC) (envelope-from void@f-m.fm) Received: from wfout4-smtp.messagingengine.com (wfout4-smtp.messagingengine.com [64.147.123.147]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TsMR45BBSz4bDk for ; Sat, 9 Mar 2024 12:14:24 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm1 header.b="n9x/LUR+"; dkim=pass header.d=messagingengine.com header.s=fm1 header.b=XkkLLeS8; dmarc=pass (policy=none) header.from=f-m.fm; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 64.147.123.147 as permitted sender) smtp.mailfrom=void@f-m.fm Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfout.west.internal (Postfix) with ESMTP id B79FD1C00096 for ; Sat, 9 Mar 2024 07:14:21 -0500 (EST) Received: from imap46 ([10.202.2.96]) by compute6.internal (MEProxy); Sat, 09 Mar 2024 07:14:21 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm1; t=1709986461; x=1710072861; bh=Vua3mRSMlk /A33kycE7+lJep23De3/Ubf7gY5i0FUzQ=; b=n9x/LUR+51wUIUeg4nXwbosu+m L+IAmmz6RM9wDoyJVJhDLNvHEpZJ2rTQc27yDWtp4my/dmwziIBztGzhYh8h3nCa k03Lqhf1WmJaum7tyKHNTJWVi0AMp85sLbmx417bRQSqp8o+Y9jfUol+CZZxdmO4 ogZYU8FeC/wGbebzuMWGhagYayGvx5EoMB73wOQsC7XoTJpsUCea+k9VYhe7AnVH BhG2TBolcZWxr99h8ijWfAY2V7sw5Pl5pQpLKal9Y3Nd0xKK9+EbyloKylH/hoGY dH018QktHOeHPSJUI1kFLshTpHHLj6EBSo7rs35/FQ4wlKDxVx9HVdIOenDg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; t=1709986461; x=1710072861; bh=Vua3mRSMlk/A33kycE7+lJep23De 3/Ubf7gY5i0FUzQ=; b=XkkLLeS8VWrl8VAI//VEt4T3kXBzrGgNPxyKbiR6jj5W JOqm0rp6W7W0v6kX2DHlcPayb03PbujqITV9UFQJj6L0bhsNDhSphpuTAZr0Qynm ++VP+w068hfXKgmShj6IZ9pfCwetv/ZH6igXx4wtWw8fEEPVxOFY34xm9YOu9yKF /Qb22j1rW9aIfOqxbKF9B7R32vLh/zEM4/Qr5pq0aKZNj2RaaXbbNPaZC1bOxwQC eElVmoiingGIZsF2T4LMoThhUybG1DlR/zvjZXxH51mrIN1ZEHMcsuJzAAoW2wdT 2gtTJJS1IgO0UdzHDStajBMkalkRjfxvggsqsn70+Q== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrieejgdefgecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepofgfggfkjghffffhvffutgesthdtre dtreertdenucfhrhhomhepvhhoihguuceovhhoihgusehfqdhmrdhfmheqnecuggftrfgr thhtvghrnhepieetvdeuhedthedtvdfhuefhveehvdeiledvieffheevleehgeefudelje dukedvnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhep vhhoihgusehfqdhmrdhfmh X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id 042C82A2008B; Sat, 9 Mar 2024 07:14:20 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.11.0-alpha0-251-g8332da0bf6-fm-20240305.001-g8332da0b List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Message-Id: In-Reply-To: <615f6011-c7df-4be2-bdad-f0c5dd67ee16@quip.cz> References: <615f6011-c7df-4be2-bdad-f0c5dd67ee16@quip.cz> Date: Sat, 09 Mar 2024 12:14:00 +0000 From: void To: stable@freebsd.org Subject: Re: KVA_PAGES Content-Type: text/plain X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.41 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.994]; SUBJ_ALL_CAPS(0.68)[9]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm1,messagingengine.com:s=fm1]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.128/27]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[64.147.123.147:from]; XM_UA_NO_VERSION(0.01)[]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; FREEMAIL_FROM(0.00)[f-m.fm]; RCVD_COUNT_THREE(0.00)[3]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_ONE(0.00)[1]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[stable@freebsd.org]; ASN(0.00)[asn:29838, ipnet:64.147.123.0/24, country:US]; MLMMJ_DEST(0.00)[stable@freebsd.org]; DWL_DNSWL_NONE(0.00)[messagingengine.com:dkim] X-Rspamd-Queue-Id: 4TsMR45BBSz4bDk On Fri, 8 Mar 2024, at 10:25, Miroslav Lachman wrote: > # config -x /boot/kernel/kernel | grep -i kva_pages > > (it is empty on amd64) aha, makes sense. It's 64-bit only here. thanks all, From nobody Sat Mar 9 15:05:13 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TsRDJ260Pz5DMtv for ; Sat, 9 Mar 2024 15:05:20 +0000 (UTC) (envelope-from filippomore@yahoo.com) Received: from sonic301-31.consmr.mail.ne1.yahoo.com (sonic301-31.consmr.mail.ne1.yahoo.com [66.163.184.200]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TsRDH0lNBz4jfG for ; Sat, 9 Mar 2024 15:05:19 +0000 (UTC) (envelope-from filippomore@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=ajEE1r9Y; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of filippomore@yahoo.com designates 66.163.184.200 as permitted sender) smtp.mailfrom=filippomore@yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1709996716; bh=RVLWuoOrFOkwPtlbqZD3Ik/lh/YkOo8CsW/pMHAd/2U=; h=Date:From:To:Subject:References:From:Subject:Reply-To; b=ajEE1r9Y7yyufNftq5kI0zGn+WatBCO0btcrEahbEyPez4At19Vlyu2gdnnTgyRyvuWZfItTt3uRHSoU+AMEU2Zb/QKJMEQCZ6H8qPIdiWXEEM4Aq3g8FjmCJZbVziEfzqspaOadutlDw97M7uTZv1lWzw12Jl6PDMeAxWs6JWTjurnz32cDPp6VIz2NNFB7O2357wbKTE1FWwkS/N6s9s/q63RsP7FIn5UGKGNObhCx6Ddbryb9WTtesn5e2zcRiAs6T8/qIBUWZC31C2PTrZfgMa+EiHOZWQSyh6eRx+y9JgKfCB6h0lK6IdaHyIPv3u51X8vAkLC5YLl5RCSdGA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1709996716; bh=fA7zEDKyMp8JrWI35MdY7tDpoZF6QK25UuoKV5Oa9iw=; h=X-Sonic-MF:Date:From:To:Subject:From:Subject; b=f6GUtFLvWoCSuYCzQBLMzJnhQJRSYHOlXLG2NynRbD/bz6pZiTtoqpXyLvXFqi1V2PqHs+0BOV7iIxcwWF7YB2xiiGAUiiogIoeD9F3poIwEdDgEnddciUtfS/xrT3ssAb27t+1Qc0kp/s+Ckqpns1hNaJkoAGISpZuwXQIZAJmrPYKTNtLzHYCwFz3SGq53HloFtbRDWB4FSV89qBjqVM9jY03bkfxwc/ILAYzfIpsZ8oZw9fvXrQhXSWpFbD9pI7AlOMdgZQynwyQdWIfirnJecitv9lZXHi3j4/eZJRWYeOwTjl+vpCpEjoYivJgEW3p8sbFmPRxwxqx7QoKJxg== X-YMail-OSG: SAMY3zAVM1lwaMARaj3.I0tw_MPD.JXVq3v1AzayMPANIbz7OevNfV7uvVL899O ZqTvPVBZYgn_dd6tj7Ofsz6rHwEyCmaiEga6xiUQ4kIotwgVJp7CQeDgjLdQe.xFMonkONx5dImq z4xKbgHEmHuvtFwRHm21Iz0ujJoV1SoYGkr6W6BNjJMtDa5euic016IFmCNR4iXe3OUvgXLgt7h1 gzOBwMHTFhvK2G7hoXsxwgphik8w1x8WIYEyEyfDSjIqH1u6i55_UHgOMvFVHILi1d5DNZtnUFCT nLgGLjskqlKunvY.K6LnUQp7H.PQzLkXzHRiDiwsdNu6PO_KONnD2t_dmHDkRt6T7irEvMUU4WJ9 WK9xBB5bk5DXjyYILTPRbJSLeFn2c2oDbLS8QDXDQFC68IducCcN5haUE1iYbHRUtpQoNkysdb.a 2JHz04nY1EPEflMQrIY9gE9Qtk_2eoCYs8HDEYLd6em464LzSpGLTyhqJqN6qKzqqMKQINeLuNKZ 9vj8buD0fgZ.Y..wX4crktTsqCcUqriXmZLv95AlYXMaPrp8pYw3PcxPJmdFbGnLAeFHsA7q2ZxA aTfq4gTPJGM4D4X7Tn4wqaRaYh99jYsbowg0pfU1i0THjgrdbRgzqKRrNOQ6DPpH2IP3HLUscJOO tSECMPuNFdGmp1sSmnS1mA6QiF4M92BlwU1jEp2qk3otbekTNF0rwPpwkedyBOct6lpY3E3SM.0F TQULDWuEdjf3TgHlkoQ5KYTb4hhB5S3NFIJWRlD0jZRIw3FjayM4CESzXzJInS8o_mYMbM23b1ZI JmxK9MZC5LHe345KISs1z7_q8.Ct8IzdJMdxXWQftvdcWRmyJhu_pdjqqQYGLtvDDkuwHkIfxX9L AaOD_7VqyFhx1iqJny04oivQDH7qB.4OP2a8Atplv4fadJhP.YebbZXz78bLamzRrlzlnBV46SmU PDUsF_6LqAL3R4__28JzeZlE6yxTbFBk3AOq_II0B6CFo5WJreIS4Daf_bMdou.UGPxn0Gh4vIW6 ToM.pIl2HiEy6gfLaMLy2WHeAl2hvFYb0dfHwV.yeFPKISmZvQDRLnGFVH3fHOr1Y6ez6tMCGNeN UPGTmXlgkO64NvzV9gnWGftJaOhlRzlBlXodbNEfNhzRxWg5ADYiFzV4mI01_QOsycak68PkZ_fK F2k5heGIfScT5mEE0XTIeWHNh68.HgFnVG93m0Wb2PEiS.GTQNv5tUwhWKQ8OgpdmUw3wmdUPtQJ nEYlI4Bc9IXtPwjS8J647lyH_tO9sxX.N5IuPR7KTZwiSUv00ep3qOuZT9mAWs50XWMH15vhppZx Ukl6swXDNGKQHQa0j6qURVQofhX5fhA2cKjDusUWO.kHZm5L763_XARB_dUx8uXEAe.pXH1AczjO 3f0eIKNGuEYcxEINFHk8Cfyk9DCdXq4qJebN0PlkqvF_65MMLekQiwBi5aB.aTT9vuL_j_ULApDI utv.JTd1c4k5TP09OqkWGlIg27h8I_JMjhNwIHgco6aqTxFsPquY7ES8C5OIGjoxYYGe_mDVRKMS KAwAWYf3fudM8agUt6M_GA2WbA9ttM4CrshA2fPrGSoMHAa7R260Tq_RW2IJbcH8RsQdsgg8ECro wiEWqk_WCSsiT9rudBolgMDH1Rnb5cDMBxuzCIHTPsoisR8IIhvcsVLf2RNq4u0LT6TEArvPQKWN ZokdsazZiF548cYAfTRg35obSksF9yx.GeJAwOh6giMowCbIusf1_ABZNyTNHHSufPYhDU_rwZsw 3g4LdOKXHZ97YO5m3mn4IN7fyZU9nIAFkM3wENT9euS7ebKllg9AEWWC55CXDwg0t0aRi7sdWJbU ZhDVtl8OS.ViT3gbpTdL1Tm9k7O_MF7V2H6aKv7vc510kw6tUuwdR8vAtZIUw6pFHwfc_Jd2jP3e moYhc9fAnaw0g3vc0tw_JJbQJi79k5aVCf4ehd7i96gLcgdjT3bKxbAF89YqQDfixTvZcs91h5yd NYlr732AL5ejxiZDFnK7hJI7_sTZnqxV7T8tDsV8x7OOgWdZxZvnn1CYoCvzAGVWS0zj5IDjdgTU 5WzYtOsO8fnjs2dCSm8ja54V6js8AuK34ONQU5NPviEhPjtlU0A_qauvqRlJbc5JIHw5S17Wx8km a5p.6f5Ka5atp3uQAMDACwg-- X-Sonic-MF: X-Sonic-ID: d7a8ea73-894b-4e02-a0fd-de143c5823cb Received: from sonic.gate.mail.ne1.yahoo.com by sonic301.consmr.mail.ne1.yahoo.com with HTTP; Sat, 9 Mar 2024 15:05:16 +0000 Date: Sat, 9 Mar 2024 15:05:13 +0000 (UTC) From: Filippo Moretti To: Freebsd-stable List Message-ID: <843286264.2438054.1709996713562@mail.yahoo.com> Subject: Error building py-setuptools-scm in STABLE.10 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_2438053_798683948.1709996713561" References: <843286264.2438054.1709996713562.ref@mail.yahoo.com> X-Mailer: WebService/1.1.22129 YMailNorrin X-Spamd-Bar: --- X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-0.997]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:36646, ipnet:66.163.184.0/21, country:US]; RCVD_COUNT_ONE(0.00)[1]; FREEMAIL_FROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_IN_DNSWL_NONE(0.00)[66.163.184.200:from]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[stable@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[66.163.184.200:from]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+] X-Rspamd-Queue-Id: 4TsRDH0lNBz4jfG ------=_Part_2438053_798683948.1709996713561 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable The following is the error while attempting the build os this portFilippo =3D=3D=3D>=C2=A0 Building for py39-setuptools-scm-8.0.4 * Getting build dependencies for wheel... Traceback (most recent call last): =C2=A0 File "/usr/local/lib/python3.9/site-packages/pyproject_hooks/_in_pro= cess/_in_process.py", line 353, in =C2=A0=C2=A0=C2=A0 main() =C2=A0 File "/usr/local/lib/python3.9/site-packages/pyproject_hooks/_in_pro= cess/_in_process.py", line 335, in main =C2=A0=C2=A0=C2=A0 json_out['return_val'] =3D hook(**hook_input['kwargs']) =C2=A0 File "/usr/local/lib/python3.9/site-packages/pyproject_hooks/_in_pro= cess/_in_process.py", line 118, in get_requires_for_build_wheel =C2=A0=C2=A0=C2=A0 return hook(config_settings) =C2=A0 File "/usr/local/lib/python3.9/site-packages/setuptools/build_meta.p= y", line 177, in get_requires_for_build_wheel =C2=A0=C2=A0=C2=A0 return self._get_build_requires( =C2=A0 File "/usr/local/lib/python3.9/site-packages/setuptools/build_meta.p= y", line 159, in _get_build_requires =C2=A0=C2=A0=C2=A0 self.run_setup() =C2=A0 File "/usr/local/lib/python3.9/site-packages/setuptools/build_meta.p= y", line 174, in run_setup =C2=A0=C2=A0=C2=A0 exec(compile(code, __file__, 'exec'), locals()) =C2=A0 File "setup.py", line 1, in =C2=A0 File "/usr/local/lib/python3.9/site-packages/setuptools/__init__.py"= , line 87, in setup =C2=A0=C2=A0=C2=A0 return distutils.core.setup(**attrs) =C2=A0 File "/usr/local/lib/python3.9/site-packages/setuptools/_distutils/c= ore.py", line 139, in setup =C2=A0=C2=A0=C2=A0 _setup_distribution =3D dist =3D klass(attrs) =C2=A0 File "/usr/local/lib/python3.9/site-packages/setuptools/dist.py", li= ne 476, in __init__ =C2=A0=C2=A0=C2=A0 _Distribution.__init__( =C2=A0 File "/usr/local/lib/python3.9/site-packages/setuptools/_distutils/d= ist.py", line 275, in __init__ =C2=A0=C2=A0=C2=A0 self.finalize_options() =C2=A0 File "/usr/local/lib/python3.9/site-packages/setuptools/dist.py", li= ne 899, in finalize_options =C2=A0=C2=A0=C2=A0 for ep in sorted(loaded, key=3Dby_order): =C2=A0 File "/usr/local/lib/python3.9/site-packages/setuptools/dist.py", li= ne 898, in =C2=A0=C2=A0=C2=A0 loaded =3D map(lambda e: e.load(), filtered) =C2=A0 File "/usr/local/lib/python3.9/site-packages/setuptools/_vendor/impo= rtlib_metadata/__init__.py", line 196, in load =C2=A0=C2=A0=C2=A0 return functools.reduce(getattr, attrs, module) AttributeError: module 'setuptools_scm.integration' has no attribute 'infer= _version' ERROR Backend subprocess exited when trying to invoke get_requires_for_buil= d_wheel *** Error code 1 Stop. make: stopped in /usr/ports/devel/py-setuptools-scm [root@ROXY /usr/ports/devel/py-setuptools-scm]#=20 ------=_Part_2438053_798683948.1709996713561 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
The f= ollowing is the error while attempting the build os this port
Filippo




=3D=3D=3D>  Building for py39= -setuptools-scm-8.0.4
* Getting build dependencies for wheel...
Trace= back (most recent call last):
  File "/usr/local/lib/python3.9/site= -packages/pyproject_hooks/_in_process/_in_process.py", line 353, in <mod= ule>
    main()
  File "/usr/local/lib/python3= .9/site-packages/pyproject_hooks/_in_process/_in_process.py", line 335, in = main
    json_out['return_val'] =3D hook(**hook_input['kw= args'])
  File "/usr/local/lib/python3.9/site-packages/pyproject_ho= oks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel<= br>    return hook(config_settings)
  File "/usr/loc= al/lib/python3.9/site-packages/setuptools/build_meta.py", line 177, in get_= requires_for_build_wheel
    return self._get_build_requi= res(
  File "/usr/local/lib/python3.9/site-packages/setuptools/buil= d_meta.py", line 159, in _get_build_requires
    self.run= _setup()
  File "/usr/local/lib/python3.9/site-packages/setuptools/= build_meta.py", line 174, in run_setup
    exec(compile(c= ode, __file__, 'exec'), locals())
  File "setup.py", line 1, in <= ;module>
  File "/usr/local/lib/python3.9/site-packages/setuptoo= ls/__init__.py", line 87, in setup
    return distutils.c= ore.setup(**attrs)
  File "/usr/local/lib/python3.9/site-packages/s= etuptools/_distutils/core.py", line 139, in setup
    _se= tup_distribution =3D dist =3D klass(attrs)
  File "/usr/local/lib/p= ython3.9/site-packages/setuptools/dist.py", line 476, in __init__
 =    _Distribution.__init__(
  File "/usr/local/lib/python3= .9/site-packages/setuptools/_distutils/dist.py", line 275, in __init__
&= nbsp;   self.finalize_options()
  File "/usr/local/lib/py= thon3.9/site-packages/setuptools/dist.py", line 899, in finalize_options    for ep in sorted(loaded, key=3Dby_order):
  Fil= e "/usr/local/lib/python3.9/site-packages/setuptools/dist.py", line 898, in= <lambda>
    loaded =3D map(lambda e: e.load(), fi= ltered)
  File "/usr/local/lib/python3.9/site-packages/setuptools/_= vendor/importlib_metadata/__init__.py", line 196, in load
  &n= bsp; return functools.reduce(getattr, attrs, module)
AttributeError: mod= ule 'setuptools_scm.integration' has no attribute 'infer_version'

ER= ROR Backend subprocess exited when trying to invoke get_requires_for_build_= wheel
*** Error code 1

Stop.
make: stopped in /usr/ports/devel= /py-setuptools-scm
[root@ROXY /usr/ports/devel/py-setuptools-scm]#
<= br>

------=_Part_2438053_798683948.1709996713561-- From nobody Sat Mar 9 23:17:12 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tsf8354sMz5CgdP for ; Sat, 9 Mar 2024 23:17:23 +0000 (UTC) (envelope-from jo@bruelltuete.com) Received: from email.jo-t.de (seppel.jo-t.de [45.132.244.126]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tsf823qfDz4kV4; Sat, 9 Mar 2024 23:17:22 +0000 (UTC) (envelope-from jo@bruelltuete.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bruelltuete.com header.s=bruelltuete18a header.b=E8s37Vhx; dmarc=pass (policy=none) header.from=bruelltuete.com; spf=pass (mx1.freebsd.org: domain of jo@bruelltuete.com designates 45.132.244.126 as permitted sender) smtp.mailfrom=jo@bruelltuete.com DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=bruelltuete.com; s=bruelltuete18a; t=1710026236; bh=E5OWXgwHD78Y5QsLiiUcQgrZBigHizMSBD36H9ePcN4=; h=Message-ID:Date:MIME-Version:Subject:To:References:From:From; b=E8s37VhxIrNm56frdx63uqo7LPsh3eFmlbIvgdbvRj8XVIwmlwrwSpwwxSN8gfBNU BZj4lsu8DuMdAVHUeGDgo7O+6lMkHcg2nTdmUB3e0D8FCVPy2Cyd3vtDLECtcnjJEJ obNx+xlH8h6FVCMLxxIkxbzmOCKcK8Tlx7PU/BLI8ZVN1Hyl0cB+y+raqk2uXrxncF zxvmJfvMRu4FtPrRJ714BlEBC65Zo8M2K7uysfOBHifIMokCnVlpW/qzam/hhucEKw kyjtsRzfGX126U/yqEnuEZn/5qMb4xjGwQ/PKqjpwLMadY7VCPaCxA8VS3lzB068mT /UqEoyk5r003A== Message-ID: <996ed8b7-9d37-43cb-95fa-add1cd31336a@bruelltuete.com> Date: Sat, 9 Mar 2024 23:17:12 +0000 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Subject: Re: Failed compiling releng/13.2, commit missing? Content-Language: en-GB To: =?UTF-8?Q?Dag-Erling_Sm=C3=B8rgrav?= Cc: FreeBSD Stable References: <9d1a95e5-2972-4e52-88ef-8d7993275f14@bruelltuete.com> <86jzmixrhx.fsf@ltc.des.dev> <66308856-35d2-464f-9018-2a7c0a10b8ca@bruelltuete.com> <86bk7ty8p7.fsf@ltc.des.dev> From: Johannes Totz In-Reply-To: <86bk7ty8p7.fsf@ltc.des.dev> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Bar: -- X-Spamd-Result: default: False [-3.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[bruelltuete.com,none]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[bruelltuete.com:s=bruelltuete18a]; MIME_GOOD(-0.10)[text/plain]; ASN(0.00)[asn:197540, ipnet:45.132.244.0/22, country:DE]; MIME_TRACE(0.00)[0:+]; MISSING_XM_UA(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_ZERO(0.00)[0]; RCPT_COUNT_TWO(0.00)[2]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; TO_DN_ALL(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[bruelltuete.com:+] X-Rspamd-Queue-Id: 4Tsf823qfDz4kV4 On 04/03/2024 21:43, Dag-Erling Smørgrav wrote: > Johannes Totz writes: >> I might be mis-reading >> https://github.com/freebsd/freebsd-src/commits/releng/13.2/ ... >> >> It shows: >> "contrib/tzdata: import tzdata 2023d" and >> "contrib/tzdata: import tzdata 2024a". >> Both came with FreeBSD-EN-24:01.tzdata on the 14th Feb. > > You're confusing tzdata and tzcode. Ha, yes! Looks very similar... I've applied tzcode 2023c to my releng/13.2 branch and it builds now. I was just curious if the build works out of the box for other folks. Anyway, it's all temporary given 13.3. thanks, Johannes From nobody Sun Mar 10 11:26:35 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TsyKq1hQCz5Cf7B for ; Sun, 10 Mar 2024 11:26:55 +0000 (UTC) (envelope-from haramrae@gmail.com) Received: from mail-ej1-x636.google.com (mail-ej1-x636.google.com [IPv6:2a00:1450:4864:20::636]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TsyKp3vdhz4blg for ; Sun, 10 Mar 2024 11:26:54 +0000 (UTC) (envelope-from haramrae@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=Hzz0dehZ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of haramrae@gmail.com designates 2a00:1450:4864:20::636 as permitted sender) smtp.mailfrom=haramrae@gmail.com Received: by mail-ej1-x636.google.com with SMTP id a640c23a62f3a-a450615d1c4so603558566b.0 for ; Sun, 10 Mar 2024 04:26:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710070009; x=1710674809; darn=freebsd.org; h=to:date:message-id:subject:mime-version:content-transfer-encoding :from:from:to:cc:subject:date:message-id:reply-to; bh=pwsv8Fm3bEKj3MOQztkaAdX9bqUwNFUy/CfChQXTBLg=; b=Hzz0dehZLfjjPWURePtdTHprjFTDsZukgZssxlNMIHlM38z5Yfkqf/aQLbFLrGGJS4 NIgvuWRNvnnn7GjxXaT82DFxla+3QGV2vEcw1+dzOe437n45kt2+Gjs9tltH0mFF92X0 2Zr9RWseG0BBbiZAWEO5n2eiqUoh+CxsNtfeUUIw5TKWjtXvlVFUwbzdMzT0//+cXW1K nST3vKQg8ygP+J+t5PFbmc3HAyml44/7PGYfSheHO7LQGgW0lIwGmomTyzcpapWDFK4C 8/g9yWipEkgEStUpnz/L2FoDSXeXKnrNTvbq67qrGXUpEveeXBRnPIi9kSY86/TweIJ1 y7cw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710070009; x=1710674809; h=to:date:message-id:subject:mime-version:content-transfer-encoding :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=pwsv8Fm3bEKj3MOQztkaAdX9bqUwNFUy/CfChQXTBLg=; b=tSKBh9/5xAYGYn0o4pviniHoMl7yQmEhw0Qh/gyg+buo+VyoVI32lXbONKbFVK60om 0y0xKC32ctfOqHRUHjGRjVBT/iCs60V+R9iMjmQHfo+iq4C0gimAdDKzDdnzsBfMGP9N /IZ6KjrjWFdD9ggJf0j4sMvQXs85vEBHUfGrz0fvPt0Ol8WtUnH09976y43XwjFxRHUc FFSJiVPc/Y/TLLUx4uT6pPTOYowxer0DXVqtdu1s5rkNQPT6u4eU+ydI+Mxpo1qeRpcS ZTEnGUnoeD9fdNT3qFWi7G4N8NW+GoKytjvQs9DcDXXLLokSFKPcyYuNtTKrqQXIV204 Jl2g== X-Gm-Message-State: AOJu0Yw7G3hsOsfA66gazStv5ZAK5T6ESzWYNwAUxAcP5vteKOSv2Fd7 CFzyUKcU5qFbmqhiQVvN/Ae4DrXcP0nC8V0Bos5odq5rRazbDc7LfatfXherTBs= X-Google-Smtp-Source: AGHT+IFUCgAm92YuIJpvSELxEuQfzqlWYcVaSyWumoxbAuKkBMJ0qqPKiJA51vRgGnXhQnIZxkw2ng== X-Received: by 2002:a17:906:7c82:b0:a45:40e4:8c8 with SMTP id w2-20020a1709067c8200b00a4540e408c8mr3158334ejo.16.1710070008767; Sun, 10 Mar 2024 04:26:48 -0700 (PDT) Received: from smtpclient.apple ([109.37.142.163]) by smtp.gmail.com with ESMTPSA id bk2-20020a170906b0c200b00a44ef54b6b6sm1819902ejb.58.2024.03.10.04.26.47 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 10 Mar 2024 04:26:48 -0700 (PDT) From: Alban Hertroys Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.300.61.1.2\)) Subject: No DHCP lease for ipheth, no bpf attached Message-Id: Date: Sun, 10 Mar 2024 12:26:35 +0100 To: freebsd-stable@freebsd.org X-Mailer: Apple Mail (2.3774.300.61.1.2) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.36 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.86)[-0.860]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_FROM(0.00)[gmail.com]; FROM_HAS_DN(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TO_DN_NONE(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::636:from] X-Rspamd-Queue-Id: 4TsyKp3vdhz4blg Hi, I=E2=80=99m trying to get ipheth to work as a 5G hotspot on my FreeBSD = home router/server, using the method described in man 4 ipheth. The = final step fails to get a DHCP lease. The configuration is on FreeBSD 14.0-RELEASE-p5/amd64 and an iPhone 13 = mini. Is this configuration supported? And if so, where did I go wrong? Details below. # kldstat | grep ipheth 23 1 0xffffffff82997000 21e0 if_ipheth.ko # usbconfig | grep Apple ugen2.2: at usbus2, cfg=3D3 md=3DHOST spd=3DHIGH = (480Mbps) pwr=3DON (500mA) # usbconfig -d 2.2 dump_all_config_desc | grep -E '(^ Conf|iConf)' Configuration index 0 iConfiguration =3D 0x0005 Configuration index 1 iConfiguration =3D 0x0006 Configuration index 2 iConfiguration =3D 0x0007 Configuration index 3 iConfiguration =3D 0x0008 # usbconfig -d 2.2 set_config 3 # usbconfig | grep 'Apple.*cfg=3D3' ugen2.2: at usbus2, cfg=3D3 md=3DHOST spd=3DHIGH = (480Mbps) pwr=3DON (500mA) # dmesg | grep 'ue[0-9]' ue0: on ipheth0 ue0: Ethernet address: b2:67:b5:ce:d4:2d At this point the man page expects to see a line: ue0: bpf attached, = that I don=E2=80=99t have. Without that, it is no surprise that the next = steps fail, DHCP requires bpf according to man bpf. The iPhone did indeed ask whether the machine could be trusted (granted, = of course), and from the green background on the time display it does = appear to believe that it is indeed functioning as a hotspot for the = machine (or whatever that's supposed to mean). # sysrc ifconfig_ue0 ifconfig_ue0: SYNCDHCP # service netif restart ue0 Stopping dhclient. Waiting for PIDS: 51707. Stopping Network: ue0. ue0: flags=3D8842 metric 0 mtu 1500 options=3D0 ether b2:67:b5:ce:d4:2d nd6 options=3D29 Starting dhclient. DHCPDISCOVER on ue0 to 255.255.255.255 port 67 interval 5 DHCPDISCOVER on ue0 to 255.255.255.255 port 67 interval 12 DHCPDISCOVER on ue0 to 255.255.255.255 port 67 interval 21 DHCPDISCOVER on ue0 to 255.255.255.255 port 67 interval 21 DHCPDISCOVER on ue0 to 255.255.255.255 port 67 interval 2 No DHCPOFFERS received. No working leases in persistent database - sleeping. Starting Network: ue0. ue0: flags=3D8843 metric 0 mtu = 1500 options=3D0 ether b2:67:b5:ce:d4:2d nd6 options=3D29 For completeness sake, I did have usbmuxd running during all this: > doas usbmuxd --enable-exit --foreground --verbose --user root Password: [12:07:06.047][3] usbmuxd v1.1.1 starting up [12:07:06.048][4] Creating socket [12:07:06.048][4] Not dropping privileges to root [12:07:06.048][4] Initializing USB [12:07:06.048][3] Using libusb 1.0.0 [12:07:06.048][4] Registering for libusb hotplug events [12:07:06.048][4] Found new device with v/p 05ac:12a8 at 2-2 [12:07:06.048][4] Found interface 1 with endpoints 04/85 for device 2-2 [12:07:06.049][4] Using wMaxPacketSize=3D512 for device 2-2 [12:07:06.049][4] USB Speed is 480 MBit/s for device 2-2 [12:07:06.049][4] 1 device detected [12:07:06.049][3] Initialization complete [12:07:06.049][3] Enabled exit on SIGUSR1 if no devices are attached. = Start a new instance with "--exit" to trigger. [12:07:06.049][4] Got lang ID 1033 for device 2-2 [12:07:06.050][4] Got serial '000081100014302A0201401E' for device 2-2 [12:07:06.050][3] Connecting to new device on location 0x20002 as ID 1 [12:07:06.050][3] Connected to v2.0 device 1 on location 0x20002 with = serial number 00008110-0014302A0201401E [12:07:06.051][4] preflight_worker_handle_device_add: Starting preflight = on device 00008110-0014302A0201401E... [12:07:06.051][4] Client 10 accepted [12:07:06.052][4] Client connected to device 1 (1->62078) [12:07:06.053][4] Client 12 accepted [12:07:06.053][4] Client 12 connection closed [12:07:06.053][4] Client 12 is going to be disconnected [12:07:06.062][4] Client 12 accepted [12:07:06.063][4] Client 12 connection closed [12:07:06.063][4] Client 12 is going to be disconnected [12:07:06.287][4] preflight_worker_handle_device_add: StartSession = success for device 00008110-0014302A0201401E [12:07:06.287][4] preflight_worker_handle_device_add: Finished preflight = on device 00008110-0014302A0201401E [12:07:06.289][4] Client 10 is going to be disconnected While this is a custom kernel, it includes GENERIC and doesn=E2=80=99t = remove device bpf. I used it successfully prior to these attempts with = DHCP on an Intel ethernet adapter: # sysrc ifconfig_em1 ifconfig_em1: DHCP defaultif -tso4 -lro -vlanhwtso So I know that DHCP works (in fact, this machine also runs isc_dhcp to = provide my home network with address information). Regards, Alban Hertroys -- There is always an exception to always.