Date: Wed, 09 Oct 2019 09:52:19 +0000 From: bugzilla-noreply@freebsd.org To: ports-bugs@FreeBSD.org Subject: [Bug 241148] Mk/Scripts/do-fetch.sh: script made by fetch-list fetches the same file several times Message-ID: <bug-241148-7788@https.bugs.freebsd.org/bugzilla/>
next in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D241148 Bug ID: 241148 Summary: Mk/Scripts/do-fetch.sh: script made by fetch-list fetches the same file several times Product: Ports & Packages Version: Latest Hardware: Any OS: Any Status: New Severity: Affects Only Me Priority: --- Component: Individual Port(s) Assignee: rust@FreeBSD.org Reporter: brigadir15@gmail.com CC: portmgr@FreeBSD.org Flags: maintainer-feedback?(rust@FreeBSD.org) Assignee: rust@FreeBSD.org After r507705[1] fetch-list for ports with DISTFILES having entries with subdirectories, produces shell script that causes downloading of the same f= ile several time. For example: $ cd /usr/ports/lang/rust $ make fetch-list mkdir -p /usr/ports/distfiles/rust && cd /usr/ports/distfiles/rust && { env /usr/bin/fetch -Fpr -S 96163304 https://static.rust-lang.org/dist/rustc-1.38.0-src.tar.xz || env /usr/bin/fetch -Fpr -S 96163304 http://distcache.FreeBSD.org/ports-distfiles/rust/rustc-1.38.0-src.tar.xz = || echo "rustc-1.38.0-src.tar.xz" not fetched; } mkdir -p /usr/ports/distfiles/rust && cd /usr/ports/distfiles/rust && { mkd= ir -p "2019-08-15" && env /usr/bin/fetch -Fpr -S 70318740 -o 2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz https://static.rust-lang.org/dist/2019-08-15/rustc-1.37.0-x86_64-unknown-fr= eebsd.tar.gz || mkdir -p "2019-08-15" && env /usr/bin/fetch -Fpr -S 70318740 -o 2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz http://distcache.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.= 37.0-x86_64-unknown-freebsd.tar.gz || mkdir -p "2019-08-15" && env /usr/bin/fetch -Fpr -S 70318740 -o 2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz http://distcache.us-east.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/= rustc-1.37.0-x86_64-unknown-freebsd.tar.gz || mkdir -p "2019-08-15" && env /usr/bin/fetch -Fpr -S 70318740 -o 2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz http://distcache.eu.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc= -1.37.0-x86_64-unknown-freebsd.tar.gz || mkdir -p "2019-08-15" && env /usr/bin/fetch -Fpr -S 70318740 -o 2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz http://distcache.us-west.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/= rustc-1.37.0-x86_64-unknown-freebsd.tar.gz || mkdir -p "2019-08-15" && env /usr/bin/fetch -Fpr -S 70318740 -o 2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz http://distcache.FreeBSD.org/ports-distfiles/rust/2019-08-15/rustc-1.37.0-x= 86_64-unknown-freebsd.tar.gz || echo "2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz" not fetche= d; } mkdir -p /usr/ports/distfiles/rust && cd /usr/ports/distfiles/rust && { mkd= ir -p "2019-08-15" && env /usr/bin/fetch -Fpr -S 87956779 -o 2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz https://static.rust-lang.org/dist/2019-08-15/rust-std-1.37.0-x86_64-unknown= -freebsd.tar.gz || mkdir -p "2019-08-15" && env /usr/bin/fetch -Fpr -S 87956779 -o 2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz http://distcache.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rust-std= -1.37.0-x86_64-unknown-freebsd.tar.gz || mkdir -p "2019-08-15" && env /usr/bin/fetch -Fpr -S 87956779 -o 2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz http://distcache.us-east.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/= rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz || mkdir -p "2019-08-15" && env /usr/bin/fetch -Fpr -S 87956779 -o 2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz http://distcache.eu.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rust-= std-1.37.0-x86_64-unknown-freebsd.tar.gz || mkdir -p "2019-08-15" && env /usr/bin/fetch -Fpr -S 87956779 -o 2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz http://distcache.us-west.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/= rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz || mkdir -p "2019-08-15" && env /usr/bin/fetch -Fpr -S 87956779 -o 2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz http://distcache.FreeBSD.org/ports-distfiles/rust/2019-08-15/rust-std-1.37.= 0-x86_64-unknown-freebsd.tar.gz || echo "2019-08-15/rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz" not fetched; } mkdir -p /usr/ports/distfiles/rust && cd /usr/ports/distfiles/rust && { mkd= ir -p "2019-08-15" && env /usr/bin/fetch -Fpr -S 6712004 -o 2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz https://static.rust-lang.org/dist/2019-08-15/cargo-0.38.0-x86_64-unknown-fr= eebsd.tar.gz || mkdir -p "2019-08-15" && env /usr/bin/fetch -Fpr -S 6712004 -o 2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz http://distcache.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/cargo-0.= 38.0-x86_64-unknown-freebsd.tar.gz || mkdir -p "2019-08-15" && env /usr/bin/fetch -Fpr -S 6712004 -o 2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz http://distcache.us-east.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/= cargo-0.38.0-x86_64-unknown-freebsd.tar.gz || mkdir -p "2019-08-15" && env /usr/bin/fetch -Fpr -S 6712004 -o 2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz http://distcache.eu.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/cargo= -0.38.0-x86_64-unknown-freebsd.tar.gz || mkdir -p "2019-08-15" && env /usr/bin/fetch -Fpr -S 6712004 -o 2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz http://distcache.us-west.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/= cargo-0.38.0-x86_64-unknown-freebsd.tar.gz || mkdir -p "2019-08-15" && env /usr/bin/fetch -Fpr -S 6712004 -o 2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz http://distcache.FreeBSD.org/ports-distfiles/rust/2019-08-15/cargo-0.38.0-x= 86_64-unknown-freebsd.tar.gz || echo "2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd.tar.gz" not fetche= d; } Execution of that script ends up with this result: $ su root -c "/usr/home/user/fetch_rust.sh" Password: fetch: https://static.rust-lang.org/dist/rustc-1.38.0-src.tar.xz: size unkn= own fetch: https://static.rust-lang.org/dist/rustc-1.38.0-src.tar.xz: size of remote file is not known rustc-1.38.0-src.tar.xz 91 MB 4241 kBps = 22s 2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd 67 MB 3724 kBps = 08s fetch: http://distcache.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.= 37.0-x86_64-unknown-freebsd.tar.gz: Not Found fetch: http://distcache.us-east.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/= rustc-1.37.0-x86_64-unknown-freebsd.tar.gz: Not Found fetch: http://distcache.eu.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc= -1.37.0-x86_64-unknown-freebsd.tar.gz: Not Found fetch: http://distcache.us-west.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/= rustc-1.37.0-x86_64-unknown-freebsd.tar.gz: Not Found 2019-08-15/rust-std-1.37.0-x86_64-unknown-free 83 MB 4212 kBps = 21s fetch: http://distcache.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rust-std= -1.37.0-x86_64-unknown-freebsd.tar.gz: Not Found fetch: http://distcache.us-east.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/= rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz: Not Found fetch: http://distcache.eu.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rust-= std-1.37.0-x86_64-unknown-freebsd.tar.gz: Not Found fetch: http://distcache.us-west.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/= rust-std-1.37.0-x86_64-unknown-freebsd.tar.gz: Not Found 2019-08-15/cargo-0.38.0-x86_64-unknown-freebsd 6554 kB 3455 kBps = 02s fetch: http://distcache.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/cargo-0.= 38.0-x86_64-unknown-freebsd.tar.gz: Not Found fetch: http://distcache.us-east.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/= cargo-0.38.0-x86_64-unknown-freebsd.tar.gz: Not Found fetch: http://distcache.eu.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/cargo= -0.38.0-x86_64-unknown-freebsd.tar.gz: Not Found fetch: http://distcache.us-west.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/= cargo-0.38.0-x86_64-unknown-freebsd.tar.gz: Not Found As you can see, the script tries to download, for example, rustc-1.37.0-x86_64-unknown-freebsd.tar.gz five times. Even after the scri= pt has downloaded it successfully on the first try. That happens because of the following lines in fetch-list's script: mkdir -p /usr/ports/distfiles/rust && cd /usr/ports/distfiles/rust && { mkd= ir -p "2019-08-15" && env /usr/bin/fetch -Fpr -S 70318740 -o 2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz https://static.rust-lang.org/dist/2019-08-15/rustc-1.37.0-x86_64-unknown-fr= eebsd.tar.gz || mkdir -p "2019-08-15" && env /usr/bin/fetch -Fpr -S 70318740 -o 2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz http://distcache.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc-1.= 37.0-x86_64-unknown-freebsd.tar.gz || mkdir -p "2019-08-15" && env /usr/bin/fetch -Fpr -S 70318740 -o 2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz http://distcache.us-east.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/= rustc-1.37.0-x86_64-unknown-freebsd.tar.gz || mkdir -p "2019-08-15" && env /usr/bin/fetch -Fpr -S 70318740 -o 2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz http://distcache.eu.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/rustc= -1.37.0-x86_64-unknown-freebsd.tar.gz || mkdir -p "2019-08-15" && env /usr/bin/fetch -Fpr -S 70318740 -o 2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz http://distcache.us-west.FreeBSD.org/local-distfiles/tobik/rust/2019-08-15/= rustc-1.37.0-x86_64-unknown-freebsd.tar.gz || mkdir -p "2019-08-15" && env /usr/bin/fetch -Fpr -S 70318740 -o 2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz http://distcache.FreeBSD.org/ports-distfiles/rust/2019-08-15/rustc-1.37.0-x= 86_64-unknown-freebsd.tar.gz || echo "2019-08-15/rustc-1.37.0-x86_64-unknown-freebsd.tar.gz" not fetche= d; } It makes "2019-08-15", downloads (successfully) rustc-1.37.0-x86_64-unknown-freebsd.tar.gz from https://static.rust-lang.org/dist/2019-08-15/rustc-1.37.0-x86_64-unknown-fr= eebsd.tar.gz (having zero exit status), skips `mkdir -p "2019-08-15"` (because of "no errors" exit status), and fetches the file again after `&&` (once again -- because of zero exit status). This `fetch` is subsequently failing, but `mkdir` after that `fetch` succeeds (because it had `-p` specified). The script `fetch`es again. And so on. I believe `mkdir` and `fetch` must be grouped together between all those `|= |` operators. See also: bug 239293[2] P.S. If someone will fix this, please also take bug 239761[3] into account, where my patch allows to combine several fetching lines into single one, as usual. [1] https://svnweb.freebsd.org/ports?view=3Drevision&revision=3D507705 [2] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D239293 [3] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D239761 --=20 You are receiving this mail because: You are the assignee for the bug.=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-241148-7788>