From owner-svn-src-head@freebsd.org Sat Sep 26 23:20:57 2020 Return-Path: Delivered-To: svn-src-head@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B18E23E3B85; Sat, 26 Sep 2020 23:20:57 +0000 (UTC) (envelope-from freebsd@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4BzPsh02Sgz4D8Q; Sat, 26 Sep 2020 23:20:55 +0000 (UTC) (envelope-from freebsd@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (localhost [127.0.0.1]) by gndrsh.dnsmgr.net (8.13.3/8.13.3) with ESMTP id 08QNKrjs056872; Sat, 26 Sep 2020 16:20:53 -0700 (PDT) (envelope-from freebsd@gndrsh.dnsmgr.net) Received: (from freebsd@localhost) by gndrsh.dnsmgr.net (8.13.3/8.13.3/Submit) id 08QNKr09056871; Sat, 26 Sep 2020 16:20:53 -0700 (PDT) (envelope-from freebsd) From: "Rodney W. Grimes" Message-Id: <202009262320.08QNKr09056871@gndrsh.dnsmgr.net> Subject: Re: svn commit: r365643 - head/bin/cp In-Reply-To: To: Warner Losh Date: Sat, 26 Sep 2020 16:20:53 -0700 (PDT) CC: Benjamin Kaduk , "Rodney W. Grimes" , Ian Lepore , src-committers , svn-src-all , svn-src-head Reply-To: rgrimes@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-Rspamd-Queue-Id: 4BzPsh02Sgz4D8Q X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of freebsd@gndrsh.dnsmgr.net has no SPF policy when checking 69.59.192.140) smtp.mailfrom=freebsd@gndrsh.dnsmgr.net X-Spamd-Result: default: False [0.57 / 15.00]; HAS_REPLYTO(0.00)[rgrimes@freebsd.org]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_MEDIUM(-0.23)[-0.234]; NEURAL_HAM_LONG(-0.07)[-0.072]; MIME_GOOD(-0.10)[text/plain]; REPLYTO_DOM_NEQ_FROM_DOM(0.00)[]; DMARC_NA(0.00)[dnsmgr.net]; AUTH_NA(1.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-0.02)[-0.019]; RCPT_COUNT_SEVEN(0.00)[7]; RCVD_TLS_LAST(0.00)[]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:13868, ipnet:69.59.192.0/19, country:US]; MAILMAN_DEST(0.00)[svn-src-all,svn-src-head]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org] X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Sep 2020 23:20:57 -0000 > On Sat, Sep 26, 2020 at 12:58 PM Benjamin Kaduk wrote: > > > On Sat, Sep 26, 2020 at 11:55 AM Warner Losh wrote: > > > >> And there's the rub: how did this file come to exist? I'm certain it isn't > >> booting or shutting down the system based on when devfs is mounted (before > >> init) and unmounted (it's not done by the shutdown scripts). Now, it's > >> always possible there's a hole in my understanding of the sequence of > >> events. But given the examination of code, it's crazy to think this could > >> be created by anything but some weird bug. That's why a step-by-step from > >> scratch guide is needed. Im happy to look into it, but I need a bit more > >> to > >> go on. Some others have run dumps, nfs exports, so far I see 4 file system report, I see these infrequently, so I would not consider these 4 file systems to be a very large data set. > >> > >> > > I don't think it's terribly complicated; either set up a multi-boot system > > or > > pull the physical drive with / from one machine, and mount it while booted > > into a different environment. Then, chroot into it and do ... just about > > anything. > > If you didn't mount devfs before chrooting, then you get a file /dev/null > > (and some > > really confused errors from, e.g., buildworld!). This is not the situation that I am reporting. > > I think there's two different things that are being talked about here. > Let's not confuse the two. > > The first is making the build system not depend on /dev/null. You'll find > that's hard to do if you and try to do it since /dev/null is used about 200 > times in the current build system in about a dozen different ways. Some are > easy, most are a bit tricky since you can't just close stdout/stderr > because then any files opened by the program will get those FDs and > printf/fprint(stderr, will collide. This dependency won't be fixed any > time soon, though I could add a seatbelt to buildworld/buildkernel that > ensures that /dev/null is a character device. > > The second is a report that /dev/null is created all the time through Correction, I never stated "all the time". I stated I have seen this on SOME analysis of disk pulled from running systems. Let me add to that these driver are only ever mounted Readonly, as what is going on is a forensics state post mortem of the system. These are not jails, some may be VM images, but even in the VM cases these are raw disk images that are never mounted read/write. It is possible that a cdrom or nfs boot was done during the life time of the node(s) with a chroot into a root file system mounted some place else. > normal means before devfs can be mounted. However, several people have > looked and found no evidence on their system. This means there's something > special / unique to Rod's setup that's generating them (assuming it isn't a > simple chroot without devfs). What that is, and how they come to be, hasn't > been explained in enough detail to reproduce. That's what people are asking > Rod about: how do we get there? How did it happen? Once we know those > answers, we can fix it. Problem is these are being found in after the fact analysis, so "getting there" is going to be hard. I'll try to collect better data such as inode contents and dates and see if I can correlate that to system install time, or some time during the systems life time. Given what kib, and ian have said I am starting to suspect that this may be occuring during the install process, the dates on the null inode and a find of the oldest inode on the disk should correlate that next time I see one of these. If I could easily reroduce it we would not be having this conversation, it would of been fixed. > Warner -- Rod Grimes rgrimes@freebsd.org