From owner-freebsd-current@freebsd.org Thu Apr 23 16:49:22 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9AFE02BCC20 for ; Thu, 23 Apr 2020 16:49:22 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qk1-x734.google.com (mail-qk1-x734.google.com [IPv6:2607:f8b0:4864:20::734]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 497NYs684Nz42CP for ; Thu, 23 Apr 2020 16:49:21 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qk1-x734.google.com with SMTP id o19so7097436qkk.5 for ; Thu, 23 Apr 2020 09:49:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=oA26VB8Y1+LrCt2JPwfqZNfoEm/iZtbPNW4HhjvFy/A=; b=eC28yOymplmYth0ya4iNDfijSoFCz7HIH/4Ytb1VaKPGXpMk/GiRd/SaDysva4CUXz Ju3/WgQo5/aYEJZxcJ1fj2Na3tUNuOXYGDMCbWbjZO+hBYhJ+xI/aWOLXSQUBqLqNZ4j SApTc06hMNhe/DvTLBDWmJJZdOUVkZ2Msp7S+jpiWHoXeSUUHwwUAe/INnTZYfZJ2sN2 ZIODE0CTiOCtiNUOgzz1EXKqrOsBvwN/fX5razqrlhl2gI/UJjawGDg1Oz/Mf8E606tx urH0cx6PDYLPA4S9BKO2bi59v06JcJOJ4gfgKjnmp4SQMeWM1CFya2YpvXrqX97vm0eF nNIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=oA26VB8Y1+LrCt2JPwfqZNfoEm/iZtbPNW4HhjvFy/A=; b=E2ZjPPGP9MwaC5dqBh5zAPEVd3PjVujsSNat5qTr94FlYneNXHUT4WsR2j95Pb6VPP kAB13iKiBWpcNeoYM/6ssT24YaC4jCv2XjVmaKPtDK+jBotmlCaofFTJCzwWABWvFDFs GZx2EN6prFcjbD//rD60a7Vmo6zCxp4t1P50cjodiOrTsKDmm4en2R6wEqIXnKH9aDPp e+6yelbJzhedkIeNd2AtCJxJSwYeZLAZlmv6GJfzYlUK4QC783FU1sof70bk9O4NX2J5 DmGWYp0l4AgpNOa1jJODOgc1A6MFlO2M/SFN+TxF6U4joZL0W4OtFj/9Yc5dex2B7of3 lbOg== X-Gm-Message-State: AGi0PuaYX1U+abyrgESQ4V5hYpHM8yOkpUox/Xtr8hnWADl8nET1mMSQ 9yAL2tIJdJbVIgD7GmdegF7ebvkHozBcuv/zGFem6A== X-Google-Smtp-Source: APiQypLZIXo/aXCvKA3e+X3Aqx7u6VhBcykF7F0fbBddX3Scb0ds52EDSgluxRYwfxB7UrWgaorCU6v8ippF3C/of3A= X-Received: by 2002:a37:9a57:: with SMTP id c84mr4460791qke.380.1587660560112; Thu, 23 Apr 2020 09:49:20 -0700 (PDT) MIME-Version: 1.0 References: <202004171047.03HAlFk6050161@gndrsh.dnsmgr.net> In-Reply-To: From: Warner Losh Date: Thu, 23 Apr 2020 10:49:09 -0600 Message-ID: Subject: Re: Ordering of files in zoneinfo [Was Re: sort.core error doing installworld on Current.] To: Xin LI Cc: Johan Hendriks , FreeBSD Current , Brooks Davis , "M. Warner Losh" , Glen Barber , Bryan Drewery , Ed Maste , fk@fabiankeil.de, Brad Davis X-Rspamd-Queue-Id: 497NYs684Nz42CP X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=eC28yOym; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::734) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-2.99 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TAGGED_RCPT(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; URI_COUNT_ODD(1.00)[21]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; RCVD_IN_DNSWL_NONE(0.00)[4.3.7.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; RCPT_COUNT_SEVEN(0.00)[10]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; FREEMAIL_TO(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-1.99)[ip: (-9.13), ipnet: 2607:f8b0::/32(-0.33), asn: 15169(-0.43), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; FREEMAIL_CC(0.00)[gmail.com] X-Mailman-Approved-At: Thu, 23 Apr 2020 17:28:50 +0000 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Apr 2020 16:49:22 -0000 On Thu, Apr 23, 2020 at 10:45 AM Xin LI wrote: > Hi, > > Thanks for raising this. > > I have took a look at the change history, it seems that the find operation > was introduced in r245265 > (brooks@, > to support packaged base) and sort was initially implemented as find -s in > r289451 > (ngie@, to make METALOG reproducible) then as sort in r328958 > (imp@, > for portability). > > I wonder if we could drop the sort and replace ${TZS} in line 100 with > ${TZS:O} instead? > I haven't thought carefully about that, but a quick look suggests that it's OK. It happens in a bootstrapped make, so that will work everywhere which addresses the issues around r328958. Using TZS:O should give us the same built-to-build stability we need. I don't recall the issue with a lot of clarity, so there's small chance I'm missing something. Warner > By the way, looking at > https://github.com/freebsd/pkg/blob/master/libpkg/metalog.c , I wonder if > the sort should really happen in pkg(8) instead? > > On Fri, Apr 17, 2020 at 7:28 AM Johan Hendriks > wrote: > >> Op 17-04-2020 om 13:30 schreef Johan Hendriks: >> > >> > Op 17-04-2020 om 12:47 schreef Rodney W. Grimes: >> >>>> Op 17-04-2020 om 03:31 schreef Rodney W. Grimes: >> >>>>>> On Thu, Apr 16, 2020 at 12:39 PM Kevin Oberman >> >>>>>> wrote: >> >>>>>> >> >>>>>>> So you some how had a sort core dump sitting in >> >>>>>>> /usr/obj/usr/src/amd64.amd64/share/zoneinfo/builddir. The >> >>>>>>> questions, how >> >>>>>>> did get there? I'd take a look at the date on the file and, it >> >>>>>>> it is older >> >>>>>>> than the buildworld, just assume that it was left-over garbage. >> >>>>>>> In either >> >>>>>>> case, you can delete it and do another installworld. >> >>>>>>> >> >>>>>>> That should most likely fix things, but, if the buildworld or >> >>>>>>> installworld >> >>>>>>> had a crash of sort(1) that left the file, further investigation >> >>>>>>> might be >> >>>>>>> needed. Re-making the zoneinfo would help track it down should >> >>>>>>> this be a re >> >>>>>>> al bug, but it's my uneducated guess that it's not. >> >>>>>>> -- >> >>>>>>> Kevin Oberman, Part time kid herder and retired Network Engineer >> >>>>>>> E-mail: rkoberman@gmail.com >> >>>>>>> PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 >> >>>>>>> >> >>>>>> Please forgive that awful post! I missed a part of your message >> >>>>>> by laziness. >> >>>>>> >> >>>>>> It's odd that the error of sort(1) crashing was not caught by the >> >>>>>> script. >> >>>>> Yes, that is a Makefile flaw someplace. >> >>>>> Further there must be a wildcard being used to decide which files to >> >>>>> install, that is a further Makefile flaw. Wildcards should NOT be >> >>>>> used >> >>>>> in the source of an install list, exactly because of this type of >> >>>>> cruft >> >>>>> that can be dropped in an obj dir. >> >> From src/share/zoneinfo/Makefile at about line 93: >> >> 92 if make(*install*) >> >> 93 TZS!= cd ${TZBUILDDIR} && find * -type f | LC_ALL=C sort >> >> ^^^^ this is a very bad thing to do >> >> in a Makefile. >> >> >> >> 94 .endif >> >> >> >> Now I still don't know why sort cored, but I am sure this is the line >> >> that did it. >> >> >> >>>>>> Clearly, sort should NOT crash! Again, a re-build of zoneinfo >> >>>>>> might catch >> >>>>>> something. Looking at the core might tell you which "sort" was >> >>>>>> involved... >> >>>>>> the one you just built or the one in the base system. This could >> >>>>>> be just a >> >>>>>> FOTU, but I would not bet on it. >> >>>>> I suspect a recent zoneinfo commit as the root cause. >> >>>>> >> >>>> I have no idea how to bypass this issue. >> >>>> I have used sort from the latest snapshot and placed that file on the >> >>>> system and in the build dir, but i keep getting the core >> >>>> >> >>>> How can i test an build and install part for zoneinfo >> >>>> >> >>>> If i go into the dir /usr/src/share/zoneinfo and do make install it >> >>>> does >> >>>> not work, do i need to add something? >> >>> Can you show us the output from >> >>> cd /usr/src/share/zoneinfo >> >>> make clean && make depend && make all && make install >> >>> Someplace in that we should get to see sort crashing... >> >>> >> > On both machines my src.conf file is the same. >> > >> > I will start over from a clean world by doing a make cleanworld and >> > see if it then still gives the errors >> > Maybe some old artifacts are hanging around. >> > >> > >> > >> >>> >> >>>> Thank you both for your time >> >>>> >> >>>>>> -- >> >>>>>> Kevin Oberman, Part time kid herder and retired Network Engineer >> >>>>>> E-mail: rkoberman@gmail.com >> >>>>>> PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 >> >>>>>> >> >>>>>> >> >>>>>>> On Thu, Apr 16, 2020 at 11:49 AM Johan Hendriks >> >>>>>>> >> >>>>>>> wrote: >> >>>>>>> >> >>>>>>>> I have a machine running FreeBSD head. >> >>>>>>>> rev 13.0-CURRENT #11 r360008 >> >>>>>>>> >> >>>>>>>> This is a quite powerful machine, so i thought it was a good >> >>>>>>>> idea to let >> >>>>>>>> that server do the build and for my virtualbox machine i can >> >>>>>>>> use the >> >>>>>>>> powerful machine to do a installword over NFS. >> >>>>>>>> >> >>>>>>>> But when i did the make installworld step the client so to say >> >>>>>>>> gives an >> >>>>>>>> error. >> >>>>>>>> >> >>>>>>>> install -o root -g wheel -m 444 >> >>>>>>>> /usr/obj/usr/src/amd64.amd64/share/zoneinfo/builddir/Zulu >> >>>>>>>> /usr/share/zoneinfo/Zulu >> >>>>>>>> install -o root -g wheel -m 444 >> >>>>>>>> /usr/obj/usr/src/amd64.amd64/share/zoneinfo/builddir/posixrules >> >>>>>>>> /usr/share/zoneinfo/posixrules >> >>>>>>>> install -o root -g wheel -m 444 >> >>>>>>>> /usr/obj/usr/src/amd64.amd64/share/zoneinfo/builddir/sort.core >> >>>>>>>> /usr/share/zoneinfo/sort.core >> >>>>>>>> install: >> >>>>>>>> /usr/obj/usr/src/amd64.amd64/share/zoneinfo/builddir/sort.core: >> >>>>>>>> Permission denied >> >>>>>>>> *** Error code 71 >> >>>>>>>> >> >>>>>>>> Stop. >> >>>>>>>> bmake[5]: stopped in /usr/src/share/zoneinfo >> >>>>>>>> *** Error code 1 >> >>>>>>>> >> >>>>>>>> Stop. >> >>>>>>>> bmake[4]: stopped in /usr/src/share >> >>>>>>>> *** Error code 1 >> >>>>>>>> >> >>>>>>>> Stop. >> >>>>>>>> bmake[3]: stopped in /usr/src >> >>>>>>>> *** Error code 1 >> >>>>>>>> >> >>>>>>>> Stop. >> >>>>>>>> bmake[2]: stopped in /usr/src >> >>>>>>>> *** Error code 1 >> >>>>>>>> >> >>>>>>>> Stop. >> >>>>>>>> bmake[1]: stopped in /usr/src >> >>>>>>>> *** Error code 1 >> >>>>>>>> >> >>>>>>>> Stop. >> >>>>>>>> make: stopped in /usr/src >> >>>>>>>> .ERROR_TARGET='installworld' >> >>>>>>>> .ERROR_META_FILE='' >> >>>>>>>> .MAKE.LEVEL='0' >> >>>>>>>> MAKEFILE='' >> >>>>>>>> .MAKE.MODE='normal' >> >>>>>>>> _ERROR_CMD='.PHONY' >> >>>>>>>> .CURDIR='/usr/src' >> >>>>>>>> .MAKE='make' >> >>>>>>>> .OBJDIR='/usr/obj/usr/src/amd64.amd64' >> >>>>>>>> .TARGETS='installworld' >> >>>>>>>> DESTDIR='' >> >>>>>>>> LD_LIBRARY_PATH='' >> >>>>>>>> MACHINE='amd64' >> >>>>>>>> MACHINE_ARCH='amd64' >> >>>>>>>> MAKEOBJDIRPREFIX='/usr/obj' >> >>>>>>>> MAKESYSPATH='/usr/src/share/mk' >> >>>>>>>> MAKE_VERSION='20181221' >> >>>>>>>> PATH='/sbin:/bin:/usr/sbin:/usr/bin' >> >>>>>>>> SRCTOP='/usr/src' >> >>>>>>>> OBJTOP='/usr/obj/usr/src/amd64.amd64' >> >>>>>>>> >> >>>>>>>> It looks likes sort coredumps in the usr/share/zoneinfo part of >> >>>>>>>> the base. >> >>>>>>>> As it has no permission on the NFS share it errors out. >> >>>>>>>> On the server itself, the installworld goes well, but it leaves a >> >>>>>>>> sort.core file behind in /usr/share/zoneinfo >> >>>>>>>> >> >>>>>>>> cd /usr/share/zoneinfo >> >>>>>>>> ls -al >> >>>>>>>> >> >>>>>>>> >> >>>>>> _______________________________________________ >> >>>>>> freebsd-current@freebsd.org mailing list >> >>>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-current >> >>>>>> To unsubscribe, send any mail to >> >>>>>> "freebsd-current-unsubscribe@freebsd.org" >> >>>>>> >> >>>> _______________________________________________ >> >>>> freebsd-current@freebsd.org mailing list >> >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-current >> >>>> To unsubscribe, send any mail to >> >>>> "freebsd-current-unsubscribe@freebsd.org" >> >>>> >> >>> -- >> >>> Rod Grimes rgrimes@freebsd.org >> >>> _______________________________________________ >> >>> freebsd-current@freebsd.org mailing list >> >>> https://lists.freebsd.org/mailman/listinfo/freebsd-current >> >>> To unsubscribe, send any mail to >> >>> "freebsd-current-unsubscribe@freebsd.org" >> >>> >> I have rebuild everything on the host and did a make cleanworld. >> Al is fine now. >> I should have done that before i asked here. >> Sorry to have wasted your time. >> But we did find a Makefile that should be doing things differently. >> >> regards >> Johan >> >> >> _______________________________________________ >> freebsd-current@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-current >> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org >> " >> >