From owner-freebsd-stable@freebsd.org Tue Feb 23 02:55:19 2021 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8000A55A051 for ; Tue, 23 Feb 2021 02:55:19 +0000 (UTC) (envelope-from dweimer@dweimer.net) Received: from webmail.dweimer.net (024-240-198-186.biz.spectrum.com [24.240.198.186]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "dweimer.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Dl3ZG5PjHz3mZC for ; Tue, 23 Feb 2021 02:55:18 +0000 (UTC) (envelope-from dweimer@dweimer.net) Received-SPF: pass (webmail.dweimer.net: authenticated connection) receiver=webmail.dweimer.net; client-ip=10.9.5.1; helo=www.dweimer.net; envelope-from=dweimer@dweimer.net; x-software=spfmilter 2.001 http://www.acme.com/software/spfmilter/ with libspf2-1.2.10; Received: from www.dweimer.net (pfsense.dweimer.me [10.9.5.1]) (authenticated bits=0) by webmail.dweimer.net (8.16.1/8.16.1) with ESMTPSA id 11N2tE78047326 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Mon, 22 Feb 2021 20:55:15 -0600 (CST) (envelope-from dweimer@dweimer.net) MIME-Version: 1.0 Date: Mon, 22 Feb 2021 20:55:09 -0600 From: "Dean E. Weimer" To: Warner Losh Cc: FreeBSD Stable Subject: Re: 13-BETA3 installation from source problems. Reply-To: dweimer@dweimer.net In-Reply-To: References: <909bf509b35ec1cda7b70c749edc6b75@dweimer.net> <0b5141137f69e2f86dd49edd4ffd1e78@dweimer.net> User-Agent: Roundcube Webmail/1.4.11 Message-ID: <71f2608341cdea1e4a36551b61d7e4a0@dweimer.net> X-Sender: dweimer@dweimer.net Organization: dweimer.net X-Rspamd-Queue-Id: 4Dl3ZG5PjHz3mZC X-Spamd-Bar: --- X-Spamd-Result: default: False [-4.00 / 15.00]; HAS_REPLYTO(0.00)[dweimer@dweimer.net]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:24.240.198.184/29]; REPLYTO_ADDR_EQ_FROM(0.00)[]; HAS_ORG_HEADER(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[dweimer.net:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[dweimer.net,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[24.240.198.186:from]; ASN(0.00)[asn:20115, ipnet:24.240.196.0/22, country:US]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[dweimer.net:s=2017.01.31]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; SPAMHAUS_ZRD(0.00)[24.240.198.186:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-stable] Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Feb 2021 02:55:19 -0000 On 2021-02-22 5:52 pm, Warner Losh wrote: > What does this patch do for you? > > diff --git a/stand/i386/loader/Makefile b/stand/i386/loader/Makefile > index ad95948ec50a..cbbe15bd1fc0 100644 > --- a/stand/i386/loader/Makefile > +++ b/stand/i386/loader/Makefile > @@ -90,7 +90,8 @@ FILES+= ${LOADER} > FILESMODE_${LOADER}= ${BINMODE} -b > > # XXX crt0.o needs to be first for pxeboot(8) to work > -OBJS= ${BTXCRT} > +# Can't add it to OBJS w/o pain and suffering > +LDFLAGS+= ${BTXCRT} > > DPADD= ${LDR_INTERP32} ${LIBFIREWIRE} ${LIBI386} ${LIBSA32} > LDADD= ${LDR_INTERP32} ${LIBFIREWIRE} ${LIBI386} ${LIBSA32} > > Anything? > > Warner Success, mostly anyways. I tried copying and pasting above into a file (/tmp/patch) and using patch < /tmp/patch from usr/src. It failed to apply, so I just manually edited file, removed OBJS= line and added the comment and LDFLAGS+= line. ran make clean in usr/src/stand and then from usr/src ran make -DNO_CLEAN buildworld, then buildinstall completed. Don't know if email client added a special character that screwed up patch or I just did it wrong, but the fix appears to have worked. -- Thanks, Dean E. Weimer http://www.dweimer.net/ From owner-freebsd-stable@freebsd.org Tue Feb 23 03:22:18 2021 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 85B4055A960 for ; Tue, 23 Feb 2021 03:22:18 +0000 (UTC) (envelope-from xxjack12xx@gmail.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4Dl49Q2VHDz3pQW for ; Tue, 23 Feb 2021 03:22:18 +0000 (UTC) (envelope-from xxjack12xx@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 5585055AC9F; Tue, 23 Feb 2021 03:22:18 +0000 (UTC) Delivered-To: stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 542B555ABF9 for ; Tue, 23 Feb 2021 03:22:18 +0000 (UTC) (envelope-from xxjack12xx@gmail.com) Received: from mail-qt1-x830.google.com (mail-qt1-x830.google.com [IPv6:2607:f8b0:4864:20::830]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Dl49Q1lbpz3pW2 for ; Tue, 23 Feb 2021 03:22:17 +0000 (UTC) (envelope-from xxjack12xx@gmail.com) Received: by mail-qt1-x830.google.com with SMTP id b24so10963349qtp.13 for ; Mon, 22 Feb 2021 19:22:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=QrO33EM3dWtsf5WbqWWOjnVT1PZ8MWFEovWljpdGfXY=; b=dGwcFBoAdGsXNZmCapf3D34w0X9X0djs/zSPJHdV6sq84KblxPrBKnzAr/mXq5H6Co YELyiauVnBjP+BbUMel6BDT8p+tcJFItlnNKyFqd3oCjrdtVLcXFozXAqHSZbzbyD6Kb s7oaVf6DZzy9p6V55f3K4jEx2m5bSVAXU30ovirVltYtG5MYXw/sJ8l8hBlax+xfnaev sxKapMUoGm1aQsrQYS3J3fvNR0JndftBvmKzqX7gV1x9vocfRd8eYwyScAnSrFWeMufn GXJAWtvbuAv6YSKG0QgCFpVUTwIHdYSamhSFt4SEaHPRL93AY0RfNr6YJtO/ib/bNrN0 4W7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=QrO33EM3dWtsf5WbqWWOjnVT1PZ8MWFEovWljpdGfXY=; b=J3JBNA5eO3kSIYQIbPiv78VXBKxu6faXY6rcaWQ+4KF3Qt/LrGycGwv9OW0n3dEKVv CZBUHz1QujITISZ/Pp6FShy7khizeJalB+7fBiWlA7HoB+1osJsAUi6qKodNqIZCdJIr wyN1QVcVhXsGtKRJx2rhRFuiXZdYtnQ4hG4dyaNtOU3A2DY3OZmThWvpEX4GgqcWkQ9s Qql3oBrXViBOcYyfAO0dyamQ3Mk9N4qiCz4RuNARQz1+f9MebABvK5WABm0iCM26yXbx qPK/t+rXG7YWkUUqfMPBcFsIisxOJRdBsGbVLqS6Ys3LgusXIuxc6PIqLHOjYFD1Qqgv VVhw== X-Gm-Message-State: AOAM5309hDAy4e8gIR9lAAfrJHd+rIrvNKnGWW3jHyV2su4HHOtuAO1z 8WWRGVV0fwgZoiZNBOvxbAgP79rjXofXmzYHTNbH1hOV X-Google-Smtp-Source: ABdhPJxKMSvY7+dLtEZHSNK/OvxNY05ttZSuJxWXLXFjSQ8DfICUW+FvvxjvS80OrwUwU/Exi4xwv1aTKN3Lo0FZtpU= X-Received: by 2002:ac8:110d:: with SMTP id c13mr22455350qtj.361.1614050536854; Mon, 22 Feb 2021 19:22:16 -0800 (PST) MIME-Version: 1.0 References: <01190f48-1071-c2ee-0012-9a9ea00530dd@smeets.xyz> In-Reply-To: <01190f48-1071-c2ee-0012-9a9ea00530dd@smeets.xyz> From: "Jack L." Date: Mon, 22 Feb 2021 19:21:40 -0800 Message-ID: Subject: Re: 13.0-BETA3 frequent file system (UFS) hangs To: Florian Smeets Cc: stable@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4Dl49Q1lbpz3pW2 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Feb 2021 03:22:18 -0000 I'm also having the same exact issues across several systems on 13-BETA3 and 14-CURRENT as of last week. I recently upgraded everything to the latest 13-STABLE and 14-CURRENT but uptime is only a day or so and not sure if it's still a problem. On Mon, Feb 22, 2021 at 12:07 PM Florian Smeets via freebsd-stable wrote: > > Hi, > > on Sunday I upgraded a lightly loaded web server (Nginx+PHP-FPM+MariaDB) > from 12.2-RELEASE-p3 to 13.0-BETA3. The server has been hanging every > few hours (between 2-12). Until now the only remedy I found was a reboot. > > When the issue appears I can still read from the file system, I was able > to touch a (new) file, but not delete it. > > load: 0.49 cmd: rm 78200 [biowr] 7.31r 0.00u 0.28s 1% 2192k > mi_switch+0xc1 _sleep+0x1cb bwait+0x6e bufwrite+0x206 ffs_update+0x2d0 > ffs_syncvnode+0x552 softdep_prelink+0x14b ufs_remove+0x85 > VOP_REMOVE_APV+0x27 kern_funlinkat+0x2d5 sys_unlink+0x28 > amd64_syscall+0x10c fast_syscall_common+0xf8 > > The system is UFS only. > > /dev/da0s1a on / (ufs, local, noatime, journaled soft-updates, writes: > sync 19483 async 3361, reads: sync 24501 async 4191, fsid 374666591e63cded) > > For some of the reboots I see the following in the log before rebooting: > > Feb 21 15:21:57 web01 kernel: Waiting (max 60 seconds) for system > process `vnlru' to stop... done > Feb 21 15:21:57 web01 kernel: Waiting (max 60 seconds) for system > process `syncer' to stop... > Feb 21 15:21:57 web01 kernel: Syncing disks, vnodes remaining... 20 > fsync: giving up on dirty (error = 35) 0xfffff80003815b70: type VCHR > Feb 21 15:21:57 web01 kernel: usecount 1, writecount 0, refcount > 1074 seqc users 0 rdev 0xfffff80004c3f000 > Feb 21 15:21:57 web01 kernel: hold count flags () > Feb 21 15:21:57 web01 kernel: flags () > Feb 21 15:21:57 web01 kernel: v_object 0xfffff800030d0c60 ref 0 > pages 45500 cleanbuf 1071 dirtybuf 1 > Feb 21 15:21:57 web01 kernel: lock type mntfs: EXCL by thread > 0xfffffe00c33fc000 (pid 24, syncer, tid 100097) > Feb 21 15:21:57 web01 kernel: 9 5 0 0 done > > Is this related to the thread "FreeBSD 13.0-BETA2 and slow IO" ? > > I have a stable/13 kernel on there now, but that also hung once already. > > Do I need to disable SU/SUJ or put a main kernel on there to get it stable? > > Thanks, > Florian