From owner-svn-src-releng@FreeBSD.ORG Wed May 13 22:52:38 2015 Return-Path: Delivered-To: svn-src-releng@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0FA927C4; Wed, 13 May 2015 22:52:38 +0000 (UTC) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F143C1251; Wed, 13 May 2015 22:52:37 +0000 (UTC) Received: from svn.freebsd.org ([127.0.1.70]) by svn.freebsd.org (8.14.9/8.14.9) with ESMTP id t4DMqbAR068468; Wed, 13 May 2015 22:52:37 GMT (envelope-from delphij@FreeBSD.org) Received: (from delphij@localhost) by svn.freebsd.org (8.14.9/8.14.9/Submit) id t4DMqaZq068459; Wed, 13 May 2015 22:52:36 GMT (envelope-from delphij@FreeBSD.org) Message-Id: <201505132252.t4DMqaZq068459@svn.freebsd.org> X-Authentication-Warning: svn.freebsd.org: delphij set sender to delphij@FreeBSD.org using -f From: Xin LI Date: Wed, 13 May 2015 22:52:36 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-releng@freebsd.org Subject: svn commit: r282873 - in releng/10.1: . sys/conf sys/ufs/ffs usr.sbin/freebsd-update X-SVN-Group: releng MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-releng@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for the release engineering / security commits to the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 May 2015 22:52:38 -0000 Author: delphij Date: Wed May 13 22:52:35 2015 New Revision: 282873 URL: https://svnweb.freebsd.org/changeset/base/282873 Log: Fix bug with freebsd-update(8) that does not ensure the previous upgrade was completed. [EN-15:04] Fix deadlock on reboot with UFS tuned with SU+J. [EN-15:05] Approved by: so Modified: releng/10.1/UPDATING releng/10.1/sys/conf/newvers.sh releng/10.1/sys/ufs/ffs/ffs_softdep.c releng/10.1/sys/ufs/ffs/ffs_vfsops.c releng/10.1/sys/ufs/ffs/softdep.h releng/10.1/usr.sbin/freebsd-update/freebsd-update.8 releng/10.1/usr.sbin/freebsd-update/freebsd-update.sh Modified: releng/10.1/UPDATING ============================================================================== --- releng/10.1/UPDATING Wed May 13 22:39:29 2015 (r282872) +++ releng/10.1/UPDATING Wed May 13 22:52:35 2015 (r282873) @@ -16,6 +16,14 @@ from older versions of FreeBSD, try WITH stable/10, and then rebuild without this option. The bootstrap process from older version of current is a bit fragile. +20150513: p10 FreeBSD-EN-15:04.freebsd-update + FreeBSD-EN-15:05.ufs + + Fix bug with freebsd-update(8) that does not ensure the previous + upgrade was completed. [EN-15:04] + + Fix deadlock on reboot with UFS tuned with SU+J. [EN-15:05] + 20150407: p9 FreeBSD-SA-15:04.igmp [revised] FreeBSD-SA-15:07.ntp FreeBSD-SA-15:08.bsdinstall Modified: releng/10.1/sys/conf/newvers.sh ============================================================================== --- releng/10.1/sys/conf/newvers.sh Wed May 13 22:39:29 2015 (r282872) +++ releng/10.1/sys/conf/newvers.sh Wed May 13 22:52:35 2015 (r282873) @@ -32,7 +32,7 @@ TYPE="FreeBSD" REVISION="10.1" -BRANCH="RELEASE-p9" +BRANCH="RELEASE-p10" if [ "X${BRANCH_OVERRIDE}" != "X" ]; then BRANCH=${BRANCH_OVERRIDE} fi Modified: releng/10.1/sys/ufs/ffs/ffs_softdep.c ============================================================================== --- releng/10.1/sys/ufs/ffs/ffs_softdep.c Wed May 13 22:39:29 2015 (r282872) +++ releng/10.1/sys/ufs/ffs/ffs_softdep.c Wed May 13 22:52:35 2015 (r282873) @@ -735,9 +735,10 @@ static struct malloc_type *memtype[] = { static void check_clear_deps(struct mount *); static void softdep_error(char *, int); static int softdep_process_worklist(struct mount *, int); -static int softdep_waitidle(struct mount *); +static int softdep_waitidle(struct mount *, int); static void drain_output(struct vnode *); static struct buf *getdirtybuf(struct buf *, struct rwlock *, int); +static int check_inodedep_free(struct inodedep *); static void clear_remove(struct mount *); static void clear_inodedeps(struct mount *); static void unlinked_inodedep(struct mount *, struct inodedep *); @@ -1377,6 +1378,10 @@ softdep_flush(addr) mp = (struct mount *)addr; ump = VFSTOUFS(mp); atomic_add_int(&stat_flush_threads, 1); + ACQUIRE_LOCK(ump); + ump->softdep_flags &= ~FLUSH_STARTING; + wakeup(&ump->softdep_flushtd); + FREE_LOCK(ump); if (print_threads) { if (stat_flush_threads == 1) printf("Running %s at pid %d\n", bufdaemonproc->p_comm, @@ -1389,7 +1394,7 @@ softdep_flush(addr) VFSTOUFS(mp)->softdep_jblocks->jb_suspended)) kthread_suspend_check(); ACQUIRE_LOCK(ump); - if ((ump->softdep_flags & FLUSH_CLEANUP) == 0) + if ((ump->softdep_flags & (FLUSH_CLEANUP | FLUSH_EXIT)) == 0) msleep(&ump->softdep_flushtd, LOCK_PTR(ump), PVM, "sdflush", hz / 2); ump->softdep_flags &= ~FLUSH_CLEANUP; @@ -1419,11 +1424,9 @@ worklist_speedup(mp) ump = VFSTOUFS(mp); LOCK_OWNED(ump); - if ((ump->softdep_flags & (FLUSH_CLEANUP | FLUSH_EXIT)) == 0) { + if ((ump->softdep_flags & (FLUSH_CLEANUP | FLUSH_EXIT)) == 0) ump->softdep_flags |= FLUSH_CLEANUP; - if (ump->softdep_flushtd->td_wchan == &ump->softdep_flushtd) - wakeup(&ump->softdep_flushtd); - } + wakeup(&ump->softdep_flushtd); } static int @@ -1468,14 +1471,10 @@ softdep_speedup(ump) TAILQ_INSERT_TAIL(&softdepmounts, sdp, sd_next); FREE_GBLLOCK(&lk); if ((altump->softdep_flags & - (FLUSH_CLEANUP | FLUSH_EXIT)) == 0) { + (FLUSH_CLEANUP | FLUSH_EXIT)) == 0) altump->softdep_flags |= FLUSH_CLEANUP; - altump->um_softdep->sd_cleanups++; - if (altump->softdep_flushtd->td_wchan == - &altump->softdep_flushtd) { - wakeup(&altump->softdep_flushtd); - } - } + altump->um_softdep->sd_cleanups++; + wakeup(&altump->softdep_flushtd); FREE_LOCK(altump); } } @@ -1887,8 +1886,8 @@ softdep_flushworklist(oldmnt, countp, td struct thread *td; { struct vnode *devvp; - int count, error = 0; struct ufsmount *ump; + int count, error; /* * Alternately flush the block device associated with the mount @@ -1897,6 +1896,7 @@ softdep_flushworklist(oldmnt, countp, td * are found. */ *countp = 0; + error = 0; ump = VFSTOUFS(oldmnt); devvp = ump->um_devvp; while ((count = softdep_process_worklist(oldmnt, 1)) > 0) { @@ -1904,36 +1904,47 @@ softdep_flushworklist(oldmnt, countp, td vn_lock(devvp, LK_EXCLUSIVE | LK_RETRY); error = VOP_FSYNC(devvp, MNT_WAIT, td); VOP_UNLOCK(devvp, 0); - if (error) + if (error != 0) break; } return (error); } +#define SU_WAITIDLE_RETRIES 20 static int -softdep_waitidle(struct mount *mp) +softdep_waitidle(struct mount *mp, int flags __unused) { struct ufsmount *ump; - int error; - int i; + struct vnode *devvp; + struct thread *td; + int error, i; ump = VFSTOUFS(mp); + devvp = ump->um_devvp; + td = curthread; + error = 0; ACQUIRE_LOCK(ump); - for (i = 0; i < 10 && ump->softdep_deps; i++) { + for (i = 0; i < SU_WAITIDLE_RETRIES && ump->softdep_deps != 0; i++) { ump->softdep_req = 1; - if (ump->softdep_on_worklist) - panic("softdep_waitidle: work added after flush."); - msleep(&ump->softdep_deps, LOCK_PTR(ump), PVM, "softdeps", 1); + KASSERT((flags & FORCECLOSE) == 0 || + ump->softdep_on_worklist == 0, + ("softdep_waitidle: work added after flush")); + msleep(&ump->softdep_deps, LOCK_PTR(ump), PVM | PDROP, + "softdeps", 10 * hz); + vn_lock(devvp, LK_EXCLUSIVE | LK_RETRY); + error = VOP_FSYNC(devvp, MNT_WAIT, td); + VOP_UNLOCK(devvp, 0); + if (error != 0) + break; + ACQUIRE_LOCK(ump); } ump->softdep_req = 0; - FREE_LOCK(ump); - error = 0; - if (i == 10) { + if (i == SU_WAITIDLE_RETRIES && error == 0 && ump->softdep_deps != 0) { error = EBUSY; printf("softdep_waitidle: Failed to flush worklist for %p\n", mp); } - + FREE_LOCK(ump); return (error); } @@ -1990,7 +2001,7 @@ retry_flush: error = EBUSY; } if (!error) - error = softdep_waitidle(oldmnt); + error = softdep_waitidle(oldmnt, flags); if (!error) { if (oldmnt->mnt_kern_flag & MNTK_UNMOUNT) { retry = 0; @@ -2490,9 +2501,18 @@ softdep_mount(devvp, mp, fs, cred) /* * Start our flushing thread in the bufdaemon process. */ + ACQUIRE_LOCK(ump); + ump->softdep_flags |= FLUSH_STARTING; + FREE_LOCK(ump); kproc_kthread_add(&softdep_flush, mp, &bufdaemonproc, &ump->softdep_flushtd, 0, 0, "softdepflush", "%s worker", mp->mnt_stat.f_mntonname); + ACQUIRE_LOCK(ump); + while ((ump->softdep_flags & FLUSH_STARTING) != 0) { + msleep(&ump->softdep_flushtd, LOCK_PTR(ump), PVM, "sdstart", + hz / 2); + } + FREE_LOCK(ump); /* * When doing soft updates, the counters in the * superblock may have gotten out of sync. Recomputation @@ -7629,17 +7649,13 @@ check_inode_unwritten(inodedep) return (1); } -/* - * Try to free an inodedep structure. Return 1 if it could be freed. - */ static int -free_inodedep(inodedep) +check_inodedep_free(inodedep) struct inodedep *inodedep; { LOCK_OWNED(VFSTOUFS(inodedep->id_list.wk_mp)); - if ((inodedep->id_state & (ONWORKLIST | UNLINKED)) != 0 || - (inodedep->id_state & ALLCOMPLETE) != ALLCOMPLETE || + if ((inodedep->id_state & ALLCOMPLETE) != ALLCOMPLETE || !LIST_EMPTY(&inodedep->id_dirremhd) || !LIST_EMPTY(&inodedep->id_pendinghd) || !LIST_EMPTY(&inodedep->id_bufwait) || @@ -7654,6 +7670,21 @@ free_inodedep(inodedep) inodedep->id_nlinkdelta != 0 || inodedep->id_savedino1 != NULL) return (0); + return (1); +} + +/* + * Try to free an inodedep structure. Return 1 if it could be freed. + */ +static int +free_inodedep(inodedep) + struct inodedep *inodedep; +{ + + LOCK_OWNED(VFSTOUFS(inodedep->id_list.wk_mp)); + if ((inodedep->id_state & (ONWORKLIST | UNLINKED)) != 0 || + !check_inodedep_free(inodedep)) + return (0); if (inodedep->id_state & ONDEPLIST) LIST_REMOVE(inodedep, id_deps); LIST_REMOVE(inodedep, id_hash); @@ -13838,7 +13869,8 @@ softdep_check_suspend(struct mount *mp, { struct bufobj *bo; struct ufsmount *ump; - int error; + struct inodedep *inodedep; + int error, unlinked; bo = &devvp->v_bufobj; ASSERT_BO_WLOCKED(bo); @@ -13899,6 +13931,20 @@ softdep_check_suspend(struct mount *mp, break; } + unlinked = 0; + if (MOUNTEDSUJ(mp)) { + for (inodedep = TAILQ_FIRST(&ump->softdep_unlinked); + inodedep != NULL; + inodedep = TAILQ_NEXT(inodedep, id_unlinked)) { + if ((inodedep->id_state & (UNLINKED | UNLINKLINKS | + UNLINKONLIST)) != (UNLINKED | UNLINKLINKS | + UNLINKONLIST) || + !check_inodedep_free(inodedep)) + continue; + unlinked++; + } + } + /* * Reasons for needing more work before suspend: * - Dirty buffers on devvp. @@ -13908,8 +13954,8 @@ softdep_check_suspend(struct mount *mp, error = 0; if (bo->bo_numoutput > 0 || bo->bo_dirty.bv_cnt > 0 || - softdep_depcnt != 0 || - ump->softdep_deps != 0 || + softdep_depcnt != unlinked || + ump->softdep_deps != unlinked || softdep_accdepcnt != ump->softdep_accdeps || secondary_writes != 0 || mp->mnt_secondary_writes != 0 || Modified: releng/10.1/sys/ufs/ffs/ffs_vfsops.c ============================================================================== --- releng/10.1/sys/ufs/ffs/ffs_vfsops.c Wed May 13 22:39:29 2015 (r282872) +++ releng/10.1/sys/ufs/ffs/ffs_vfsops.c Wed May 13 22:52:35 2015 (r282873) @@ -1502,8 +1502,11 @@ ffs_sync(mp, waitfor) if (fs->fs_fmod != 0 && fs->fs_ronly != 0 && ump->um_fsckpid == 0) panic("%s: ffs_sync: modification on read-only filesystem", fs->fs_fsmnt); - if (waitfor == MNT_LAZY) - return (ffs_sync_lazy(mp)); + if (waitfor == MNT_LAZY) { + if (!rebooting) + return (ffs_sync_lazy(mp)); + waitfor = MNT_NOWAIT; + } /* * Write back each (modified) inode. @@ -1560,7 +1563,7 @@ loop: /* * Force stale filesystem control information to be flushed. */ - if (waitfor == MNT_WAIT) { + if (waitfor == MNT_WAIT || rebooting) { if ((error = softdep_flushworklist(ump->um_mountp, &count, td))) allerror = error; /* Flushed work items may create new vnodes to clean */ @@ -1577,9 +1580,12 @@ loop: if (bo->bo_numoutput > 0 || bo->bo_dirty.bv_cnt > 0) { BO_UNLOCK(bo); vn_lock(devvp, LK_EXCLUSIVE | LK_RETRY); - if ((error = VOP_FSYNC(devvp, waitfor, td)) != 0) - allerror = error; + error = VOP_FSYNC(devvp, waitfor, td); VOP_UNLOCK(devvp, 0); + if (MOUNTEDSOFTDEP(mp) && (error == 0 || error == EAGAIN)) + error = ffs_sbupdate(ump, waitfor, 0); + if (error != 0) + allerror = error; if (allerror == 0 && waitfor == MNT_WAIT) goto loop; } else if (suspend != 0) { Modified: releng/10.1/sys/ufs/ffs/softdep.h ============================================================================== --- releng/10.1/sys/ufs/ffs/softdep.h Wed May 13 22:39:29 2015 (r282872) +++ releng/10.1/sys/ufs/ffs/softdep.h Wed May 13 22:52:35 2015 (r282873) @@ -1063,6 +1063,8 @@ struct mount_softdeps { */ #define FLUSH_EXIT 0x0001 /* time to exit */ #define FLUSH_CLEANUP 0x0002 /* need to clear out softdep structures */ +#define FLUSH_STARTING 0x0004 /* flush thread not yet started */ + /* * Keep the old names from when these were in the ufsmount structure. */ Modified: releng/10.1/usr.sbin/freebsd-update/freebsd-update.8 ============================================================================== --- releng/10.1/usr.sbin/freebsd-update/freebsd-update.8 Wed May 13 22:39:29 2015 (r282872) +++ releng/10.1/usr.sbin/freebsd-update/freebsd-update.8 Wed May 13 22:52:35 2015 (r282873) @@ -25,7 +25,7 @@ .\" .\" $FreeBSD$ .\" -.Dd July 14, 2010 +.Dd March 2, 2015 .Dt FREEBSD-UPDATE 8 .Os FreeBSD .Sh NAME @@ -36,10 +36,12 @@ .Op Fl b Ar basedir .Op Fl d Ar workdir .Op Fl f Ar conffile +.Op Fl F .Op Fl k Ar KEY .Op Fl r Ar newrelease .Op Fl s Ar server .Op Fl t Ar address +.Op Fl -not-running-from-cron .Cm command ... .Sh DESCRIPTION The @@ -54,16 +56,16 @@ by the .Fx Release Engineering Team, e.g., .Fx -7.3-RELEASE and +9.3-RELEASE and .Fx -8.0-RELEASE, but not +10.1-RELEASE, but not .Fx -6.3-STABLE or +9.3-STABLE or .Fx -9.0-CURRENT. +11-CURRENT. .Sh OPTIONS The following options are supported: -.Bl -tag -width "-f conffile" +.Bl -tag -width "-r newrelease" .It Fl b Ar basedir Operate on a system mounted at .Ar basedir . @@ -81,6 +83,10 @@ Read configuration options from .Ar conffile . (default: .Pa /etc/freebsd-update.conf ) +.It Fl F +Force +.Nm Cm fetch +to proceed where it normally would not, such as an unfinished upgrade .It Fl k Ar KEY Trust an RSA key with SHA256 of .Ar KEY . @@ -98,12 +104,21 @@ Mail output of command, if any, to .Ar address . (default: root, or as given in the configuration file.) +.It Fl -not-running-from-cron +Force +.Nm Cm fetch +to proceed when there is no controlling tty. +This is for use by automated scripts and orchestration tools. +Please do not run +.Nm Cm fetch +from crontab or similar using this flag, see: +.Nm Cm cron .El .Sh COMMANDS The .Cm command can be any one of the following: -.Bl -tag -width "-f conffile" +.Bl -tag -width "rollback" .It Cm fetch Based on the currently installed world and the configuration options set, fetch all available binary updates. Modified: releng/10.1/usr.sbin/freebsd-update/freebsd-update.sh ============================================================================== --- releng/10.1/usr.sbin/freebsd-update/freebsd-update.sh Wed May 13 22:39:29 2015 (r282872) +++ releng/10.1/usr.sbin/freebsd-update/freebsd-update.sh Wed May 13 22:52:35 2015 (r282873) @@ -43,12 +43,15 @@ Options: (default: /var/db/freebsd-update/) -f conffile -- Read configuration options from conffile (default: /etc/freebsd-update.conf) + -F -- Force a fetch operation to proceed -k KEY -- Trust an RSA key with SHA256 hash of KEY -r release -- Target for upgrade (e.g., 6.2-RELEASE) -s server -- Server from which to fetch updates (default: update.FreeBSD.org) -t address -- Mail output of cron command, if any, to address (default: root) + --not-running-from-cron + -- Run without a tty, for use by automated tools Commands: fetch -- Fetch updates from server cron -- Sleep rand(3600) seconds, fetch updates, and send an @@ -399,6 +402,12 @@ init_params () { # No commands specified yet COMMANDS="" + + # Force fetch to proceed + FORCEFETCH=0 + + # Run without a TTY + NOTTYOK=0 } # Parse the command line @@ -411,6 +420,12 @@ parse_cmdline () { if [ ! -z "${CONFFILE}" ]; then usage; fi shift; CONFFILE="$1" ;; + -F) + FORCEFETCH=1 + ;; + --not-running-from-cron) + NOTTYOK=1 + ;; # Configuration file equivalents -b) @@ -665,6 +680,14 @@ fetch_check_params () { echo "(Did you mean 'upgrade' instead?)" exit 1 fi + + # Check that we have updates ready to install + if [ -f ${BDHASH}-install/kerneldone -a $FORCEFETCH -eq 0 ]; then + echo "You have a partially completed upgrade pending" + echo "Run '$0 install' first." + echo "Run '$0 fetch -F' to proceed anyway." + exit 1 + fi } # Perform sanity checks etc. before fetching upgrades. @@ -3202,7 +3225,7 @@ get_params () { # Fetch command. Make sure that we're being called # interactively, then run fetch_check_params and fetch_run cmd_fetch () { - if [ ! -t 0 ]; then + if [ ! -t 0 -a $NOTTYOK -eq 0 ]; then echo -n "`basename $0` fetch should not " echo "be run non-interactively." echo "Run `basename $0` cron instead." From owner-svn-src-releng@FreeBSD.ORG Wed May 13 22:52:53 2015 Return-Path: Delivered-To: svn-src-releng@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EDDE88EF; Wed, 13 May 2015 22:52:53 +0000 (UTC) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DA5DE1256; Wed, 13 May 2015 22:52:53 +0000 (UTC) Received: from svn.freebsd.org ([127.0.1.70]) by svn.freebsd.org (8.14.9/8.14.9) with ESMTP id t4DMqrE9068555; Wed, 13 May 2015 22:52:53 GMT (envelope-from delphij@FreeBSD.org) Received: (from delphij@localhost) by svn.freebsd.org (8.14.9/8.14.9/Submit) id t4DMqpdI068543; Wed, 13 May 2015 22:52:51 GMT (envelope-from delphij@FreeBSD.org) Message-Id: <201505132252.t4DMqpdI068543@svn.freebsd.org> X-Authentication-Warning: svn.freebsd.org: delphij set sender to delphij@FreeBSD.org using -f From: Xin LI Date: Wed, 13 May 2015 22:52:51 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-releng@freebsd.org Subject: svn commit: r282874 - in releng: 8.4 8.4/sys/conf 8.4/usr.sbin/freebsd-update 9.3 9.3/sys/conf 9.3/usr.sbin/freebsd-update X-SVN-Group: releng MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-releng@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for the release engineering / security commits to the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 May 2015 22:52:54 -0000 Author: delphij Date: Wed May 13 22:52:51 2015 New Revision: 282874 URL: https://svnweb.freebsd.org/changeset/base/282874 Log: Fix bug with freebsd-update(8) that does not ensure the previous upgrade was completed. [EN-15:04] Approved by: so Modified: releng/8.4/UPDATING releng/8.4/sys/conf/newvers.sh releng/8.4/usr.sbin/freebsd-update/freebsd-update.8 releng/8.4/usr.sbin/freebsd-update/freebsd-update.sh releng/9.3/UPDATING releng/9.3/sys/conf/newvers.sh releng/9.3/usr.sbin/freebsd-update/freebsd-update.8 releng/9.3/usr.sbin/freebsd-update/freebsd-update.sh Modified: releng/8.4/UPDATING ============================================================================== --- releng/8.4/UPDATING Wed May 13 22:52:35 2015 (r282873) +++ releng/8.4/UPDATING Wed May 13 22:52:51 2015 (r282874) @@ -15,6 +15,11 @@ NOTE TO PEOPLE WHO THINK THAT FreeBSD 8. debugging tools present in HEAD were left in place because sun4v support still needs work to become production ready. +20150513: p28 FreeBSD-EN-15:04.freebsd-update + + Fix bug with freebsd-update(8) that does not ensure the previous + upgrade was completed. [EN-15:04] + 20150407: p27 FreeBSD-SA-15:04.igmp [revised] FreeBSD-SA-15:07.ntp FreeBSD-SA-15:09.ipv6 Modified: releng/8.4/sys/conf/newvers.sh ============================================================================== --- releng/8.4/sys/conf/newvers.sh Wed May 13 22:52:35 2015 (r282873) +++ releng/8.4/sys/conf/newvers.sh Wed May 13 22:52:51 2015 (r282874) @@ -32,7 +32,7 @@ TYPE="FreeBSD" REVISION="8.4" -BRANCH="RELEASE-p27" +BRANCH="RELEASE-p28" if [ "X${BRANCH_OVERRIDE}" != "X" ]; then BRANCH=${BRANCH_OVERRIDE} fi Modified: releng/8.4/usr.sbin/freebsd-update/freebsd-update.8 ============================================================================== --- releng/8.4/usr.sbin/freebsd-update/freebsd-update.8 Wed May 13 22:52:35 2015 (r282873) +++ releng/8.4/usr.sbin/freebsd-update/freebsd-update.8 Wed May 13 22:52:51 2015 (r282874) @@ -25,7 +25,7 @@ .\" .\" $FreeBSD$ .\" -.Dd July 14, 2010 +.Dd March 2, 2015 .Dt FREEBSD-UPDATE 8 .Os FreeBSD .Sh NAME @@ -36,10 +36,12 @@ .Op Fl b Ar basedir .Op Fl d Ar workdir .Op Fl f Ar conffile +.Op Fl F .Op Fl k Ar KEY .Op Fl r Ar newrelease .Op Fl s Ar server .Op Fl t Ar address +.Op Fl -not-running-from-cron .Cm command ... .Sh DESCRIPTION The @@ -49,21 +51,21 @@ updates to the FreeBSD base system. Note that updates are only available if they are being built for the FreeBSD release and architecture being used; in particular, the .Fx -Security Team only builds updates for releases shipped in binary form +Security Team only builds updates for releases shipped in binary form by the .Fx Release Engineering Team, e.g., .Fx -7.3-RELEASE and +9.3-RELEASE and .Fx -8.0, but not +10.1-RELEASE, but not .Fx -6.3-STABLE or +9.3-STABLE or .Fx -9.0-CURRENT. +11-CURRENT. .Sh OPTIONS The following options are supported: -.Bl -tag -width "-f conffile" +.Bl -tag -width "-r newrelease" .It Fl b Ar basedir Operate on a system mounted at .Ar basedir . @@ -81,6 +83,10 @@ Read configuration options from .Ar conffile . (default: .Pa /etc/freebsd-update.conf ) +.It Fl F +Force +.Nm Cm fetch +to proceed where it normally would not, such as an unfinished upgrade .It Fl k Ar KEY Trust an RSA key with SHA256 of .Ar KEY . @@ -98,13 +104,21 @@ Mail output of command, if any, to .Ar address . (default: root, or as given in the configuration file.) +.It Fl -not-running-from-cron +Force +.Nm Cm fetch +to proceed when there is no controlling tty. +This is for use by automated scripts and orchestration tools. +Please do not run +.Nm Cm fetch +from crontab or similar using this flag, see: +.Nm Cm cron .El .Sh COMMANDS The .Cm command can be any one of the following: -.Pp -.Bl -tag -width "-f conffile" +.Bl -tag -width "rollback" .It Cm fetch Based on the currently installed world and the configuration options set, fetch all available binary updates. @@ -128,6 +142,11 @@ Fetch files necessary for upgrading to a Before using this command, make sure that you read the announcement and release notes for the new release in case there are any special steps needed for upgrading. +Note that this command may require up to 500 MB of space in +.Ar workdir +depending on which components of the +.Fx +base system are installed. .It Cm install Install the most recently fetched updates or upgrade. .It Cm rollback @@ -149,7 +168,7 @@ other than 3AM, to avoid overly imposing on the server(s) hosting the updates. .It In spite of its name, -.Cm +.Nm IDS should not be relied upon as an "Intrusion Detection System", since if the system has been tampered with it cannot be trusted to operate correctly. @@ -158,11 +177,11 @@ purposes, make sure you boot from a secu .El .Sh FILES .Bl -tag -width "/etc/freebsd-update.conf" -.It /etc/freebsd-update.conf +.It Pa /etc/freebsd-update.conf Default location of the .Nm configuration file. -.It /var/db/freebsd-update/ +.It Pa /var/db/freebsd-update/ Default location where .Nm stores temporary files and downloaded updates. @@ -170,4 +189,4 @@ stores temporary files and downloaded up .Sh SEE ALSO .Xr freebsd-update.conf 5 .Sh AUTHORS -.An Colin Percival Aq cperciva@FreeBSD.org +.An Colin Percival Aq Mt cperciva@FreeBSD.org Modified: releng/8.4/usr.sbin/freebsd-update/freebsd-update.sh ============================================================================== --- releng/8.4/usr.sbin/freebsd-update/freebsd-update.sh Wed May 13 22:52:35 2015 (r282873) +++ releng/8.4/usr.sbin/freebsd-update/freebsd-update.sh Wed May 13 22:52:51 2015 (r282874) @@ -43,12 +43,15 @@ Options: (default: /var/db/freebsd-update/) -f conffile -- Read configuration options from conffile (default: /etc/freebsd-update.conf) + -F -- Force a fetch operation to proceed -k KEY -- Trust an RSA key with SHA256 hash of KEY -r release -- Target for upgrade (e.g., 6.2-RELEASE) -s server -- Server from which to fetch updates (default: update.FreeBSD.org) -t address -- Mail output of cron command, if any, to address (default: root) + --not-running-from-cron + -- Run without a tty, for use by automated tools Commands: fetch -- Fetch updates from server cron -- Sleep rand(3600) seconds, fetch updates, and send an @@ -284,6 +287,9 @@ config_TargetRelease () { else return 1 fi + if echo ${TARGETRELEASE} | grep -qE '^[0-9.]+$'; then + TARGETRELEASE="${TARGETRELEASE}-RELEASE" + fi } # Define what happens to output of utilities @@ -396,6 +402,12 @@ init_params () { # No commands specified yet COMMANDS="" + + # Force fetch to proceed + FORCEFETCH=0 + + # Run without a TTY + NOTTYOK=0 } # Parse the command line @@ -408,6 +420,12 @@ parse_cmdline () { if [ ! -z "${CONFFILE}" ]; then usage; fi shift; CONFFILE="$1" ;; + -F) + FORCEFETCH=1 + ;; + --not-running-from-cron) + NOTTYOK=1 + ;; # Configuration file equivalents -b) @@ -569,7 +587,7 @@ fetch_setup_verboselevel () { # running *-p[0-9]+, strip off the last part; if the # user is running -SECURITY, call it -RELEASE. Chdir # into the working directory. -fetch_check_params () { +fetchupgrade_check_params () { export HTTP_USER_AGENT="freebsd-update (${COMMAND}, `uname -r`)" _SERVERNAME_z=\ @@ -577,6 +595,7 @@ fetch_check_params () { _KEYPRINT_z="Key must be given via -k option or configuration file." _KEYPRINT_bad="Invalid key fingerprint: " _WORKDIR_bad="Directory does not exist or is not writable: " + _WORKDIR_bad2="Directory is not on a persistent filesystem: " if [ -z "${SERVERNAME}" ]; then echo -n "`basename $0`: " @@ -600,6 +619,13 @@ fetch_check_params () { echo ${WORKDIR} exit 1 fi + case `df -T ${WORKDIR}` in */dev/md[0-9]* | *tmpfs*) + echo -n "`basename $0`: " + echo -n "${_WORKDIR_bad2}" + echo ${WORKDIR} + exit 1 + ;; + esac chmod 700 ${WORKDIR} cd ${WORKDIR} || exit 1 @@ -652,9 +678,29 @@ fetch_check_params () { BDHASH=`echo ${BASEDIR} | sha256 -q` } +# Perform sanity checks etc. before fetching updates. +fetch_check_params () { + fetchupgrade_check_params + + if ! [ -z "${TARGETRELEASE}" ]; then + echo -n "`basename $0`: " + echo -n "-r option is meaningless with 'fetch' command. " + echo "(Did you mean 'upgrade' instead?)" + exit 1 + fi + + # Check that we have updates ready to install + if [ -f ${BDHASH}-install/kerneldone -a $FORCEFETCH -eq 0 ]; then + echo "You have a partially completed upgrade pending" + echo "Run '$0 install' first." + echo "Run '$0 fetch -F' to proceed anyway." + exit 1 + fi +} + # Perform sanity checks etc. before fetching upgrades. upgrade_check_params () { - fetch_check_params + fetchupgrade_check_params # Unless set otherwise, we're upgrading to the same kernel config. NKERNCONF=${KERNCONF} @@ -1185,7 +1231,7 @@ fetch_metadata_sanity () { # Some aliases to save space later: ${P} is a character which can # appear in a path; ${M} is the four numeric metadata fields; and # ${H} is a sha256 hash. - P="[-+./:=%@_[~[:alnum:]]" + P="[-+./:=,%@_[~[:alnum:]]" M="[0-9]+\|[0-9]+\|[0-9]+\|[0-9]+" H="[0-9a-f]{64}" @@ -1456,7 +1502,7 @@ fetch_inspect_system () { sort -k 3,3 -t '|' > $2.tmp rm filelist - # Check if an error occured during system inspection + # Check if an error occurred during system inspection if [ -f .err ]; then return 1 fi @@ -2240,6 +2286,19 @@ upgrade_oldall_to_oldnew () { mv $2 $3 } +# Helper for upgrade_merge: Return zero true iff the two files differ only +# in the contents of their RCS tags. +samef () { + X=`sed -E 's/\\$FreeBSD.*\\$/\$FreeBSD\$/' < $1 | ${SHA256}` + Y=`sed -E 's/\\$FreeBSD.*\\$/\$FreeBSD\$/' < $2 | ${SHA256}` + + if [ $X = $Y ]; then + return 0; + else + return 1; + fi +} + # From the list of "old" files in $1, merge changes in $2 with those in $3, # and update $3 to reflect the hashes of merged files. upgrade_merge () { @@ -2323,6 +2382,14 @@ upgrade_merge () { # Ask the user to handle any files which didn't merge. while read F; do + # If the installed file differs from the version in + # the old release only due to RCS tag expansion + # then just use the version in the new release. + if samef merge/old/${F} merge/${OLDRELNUM}/${F}; then + cp merge/${RELNUM}/${F} merge/new/${F} + continue + fi + cat <<-EOF The following file could not be merged automatically: ${F} @@ -2337,9 +2404,18 @@ manually... # Ask the user to confirm that he likes how the result # of merging files. while read F; do - # Skip files which haven't changed. - if [ -f merge/new/${F} ] && - cmp -s merge/old/${F} merge/new/${F}; then + # Skip files which haven't changed except possibly + # in their RCS tags. + if [ -f merge/old/${F} ] && [ -f merge/new/${F} ] && + samef merge/old/${F} merge/new/${F}; then + continue + fi + + # Skip files where the installed file differs from + # the old file only due to RCS tags. + if [ -f merge/old/${F} ] && + [ -f merge/${OLDRELNUM}/${F} ] && + samef merge/old/${F} merge/${OLDRELNUM}/${F}; then continue fi @@ -2526,6 +2602,10 @@ upgrade_run () { # Leave a note behind to tell the "install" command that the kernel # needs to be installed before the world. touch ${BDHASH}-install/kernelfirst + + # Remind the user that they need to run "freebsd-update install" + # to install the downloaded bits, in case they didn't RTFM. + echo "To install the downloaded upgrades, run \"$0 install\"." } # Make sure that all the file hashes mentioned in $@ have corresponding @@ -2577,14 +2657,14 @@ backup_kernel_finddir () { while true ; do # Pathname does not exist, so it is OK use that name # for backup directory. - if [ ! -e $BACKUPKERNELDIR ]; then + if [ ! -e $BASEDIR/$BACKUPKERNELDIR ]; then return 0 fi # If directory do exist, we only use if it has our # marker file. - if [ -d $BACKUPKERNELDIR -a \ - -e $BACKUPKERNELDIR/.freebsd-update ]; then + if [ -d $BASEDIR/$BACKUPKERNELDIR -a \ + -e $BASEDIR/$BACKUPKERNELDIR/.freebsd-update ]; then return 0 fi @@ -2592,7 +2672,7 @@ backup_kernel_finddir () { # the end and try again. CNT=$((CNT + 1)) if [ $CNT -gt 9 ]; then - echo "Could not find valid backup dir ($BACKUPKERNELDIR)" + echo "Could not find valid backup dir ($BASEDIR/$BACKUPKERNELDIR)" exit 1 fi BACKUPKERNELDIR="`echo $BACKUPKERNELDIR | sed -Ee 's/[0-9]\$//'`" @@ -2619,17 +2699,17 @@ backup_kernel () { # Remove old kernel backup files. If $BACKUPKERNELDIR was # "not ours", backup_kernel_finddir would have exited, so # deleting the directory content is as safe as we can make it. - if [ -d $BACKUPKERNELDIR ]; then - rm -fr $BACKUPKERNELDIR + if [ -d $BASEDIR/$BACKUPKERNELDIR ]; then + rm -fr $BASEDIR/$BACKUPKERNELDIR fi # Create directories for backup. - mkdir -p $BACKUPKERNELDIR - mtree -cdn -p "${KERNELDIR}" | \ - mtree -Ue -p "${BACKUPKERNELDIR}" > /dev/null + mkdir -p $BASEDIR/$BACKUPKERNELDIR + mtree -cdn -p "${BASEDIR}/${KERNELDIR}" | \ + mtree -Ue -p "${BASEDIR}/${BACKUPKERNELDIR}" > /dev/null # Mark the directory as having been created by freebsd-update. - touch $BACKUPKERNELDIR/.freebsd-update + touch $BASEDIR/$BACKUPKERNELDIR/.freebsd-update if [ $? -ne 0 ]; then echo "Could not create kernel backup directory" exit 1 @@ -2647,8 +2727,8 @@ backup_kernel () { fi # Backup all the kernel files using hardlinks. - (cd $KERNELDIR && find . -type f $FINDFILTER -exec \ - cp -pl '{}' ${BACKUPKERNELDIR}/'{}' \;) + (cd ${BASEDIR}/${KERNELDIR} && find . -type f $FINDFILTER -exec \ + cp -pl '{}' ${BASEDIR}/${BACKUPKERNELDIR}/'{}' \;) # Re-enable patchname expansion. set +f @@ -2746,7 +2826,7 @@ install_files () { # Update linker.hints if necessary if [ -s INDEX-OLD -o -s INDEX-NEW ]; then - kldxref -R /boot/ 2>/dev/null + kldxref -R ${BASEDIR}/boot/ 2>/dev/null fi # We've finished updating the kernel. @@ -2797,14 +2877,14 @@ Kernel updates have been installed. Ple install_delete INDEX-OLD INDEX-NEW || return 1 # Rebuild /etc/spwd.db and /etc/pwd.db if necessary. - if [ /etc/master.passwd -nt /etc/spwd.db ] || - [ /etc/master.passwd -nt /etc/pwd.db ]; then - pwd_mkdb /etc/master.passwd + if [ ${BASEDIR}/etc/master.passwd -nt ${BASEDIR}/etc/spwd.db ] || + [ ${BASEDIR}/etc/master.passwd -nt ${BASEDIR}/etc/pwd.db ]; then + pwd_mkdb -d ${BASEDIR}/etc ${BASEDIR}/etc/master.passwd fi # Rebuild /etc/login.conf.db if necessary. - if [ /etc/login.conf -nt /etc/login.conf.db ]; then - cap_mkdb /etc/login.conf + if [ ${BASEDIR}/etc/login.conf -nt ${BASEDIR}/etc/login.conf.db ]; then + cap_mkdb ${BASEDIR}/etc/login.conf fi # We've finished installing the world and deleting old files @@ -3011,21 +3091,8 @@ IDS_compare () { mv INDEX-NOTMATCHING.tmp INDEX-NOTMATCHING # Go through the lines and print warnings. - while read LINE; do - FPATH=`echo "${LINE}" | cut -f 1 -d '|'` - TYPE=`echo "${LINE}" | cut -f 2 -d '|'` - OWNER=`echo "${LINE}" | cut -f 3 -d '|'` - GROUP=`echo "${LINE}" | cut -f 4 -d '|'` - PERM=`echo "${LINE}" | cut -f 5 -d '|'` - HASH=`echo "${LINE}" | cut -f 6 -d '|'` - LINK=`echo "${LINE}" | cut -f 7 -d '|'` - P_TYPE=`echo "${LINE}" | cut -f 8 -d '|'` - P_OWNER=`echo "${LINE}" | cut -f 9 -d '|'` - P_GROUP=`echo "${LINE}" | cut -f 10 -d '|'` - P_PERM=`echo "${LINE}" | cut -f 11 -d '|'` - P_HASH=`echo "${LINE}" | cut -f 12 -d '|'` - P_LINK=`echo "${LINE}" | cut -f 13 -d '|'` - + local IFS='|' + while read FPATH TYPE OWNER GROUP PERM HASH LINK P_TYPE P_OWNER P_GROUP P_PERM P_HASH P_LINK; do # Warn about different object types. if ! [ "${TYPE}" = "${P_TYPE}" ]; then echo -n "${FPATH} is a " @@ -3153,7 +3220,7 @@ get_params () { # Fetch command. Make sure that we're being called # interactively, then run fetch_check_params and fetch_run cmd_fetch () { - if [ ! -t 0 ]; then + if [ ! -t 0 -a $NOTTYOK -eq 0 ]; then echo -n "`basename $0` fetch should not " echo "be run non-interactively." echo "Run `basename $0` cron instead." Modified: releng/9.3/UPDATING ============================================================================== --- releng/9.3/UPDATING Wed May 13 22:52:35 2015 (r282873) +++ releng/9.3/UPDATING Wed May 13 22:52:51 2015 (r282874) @@ -11,6 +11,11 @@ handbook: Items affecting the ports and packages system can be found in /usr/ports/UPDATING. Please read that file before running portupgrade. +20150513: p14 FreeBSD-EN-15:04.freebsd-update + + Fix bug with freebsd-update(8) that does not ensure the previous + upgrade was completed. [EN-15:04] + 20150407: p13 FreeBSD-SA-15:04.igmp [revised] FreeBSD-SA-15:07.ntp FreeBSD-SA-15:09.ipv6 Modified: releng/9.3/sys/conf/newvers.sh ============================================================================== --- releng/9.3/sys/conf/newvers.sh Wed May 13 22:52:35 2015 (r282873) +++ releng/9.3/sys/conf/newvers.sh Wed May 13 22:52:51 2015 (r282874) @@ -32,7 +32,7 @@ TYPE="FreeBSD" REVISION="9.3" -BRANCH="RELEASE-p13" +BRANCH="RELEASE-p14" if [ "X${BRANCH_OVERRIDE}" != "X" ]; then BRANCH=${BRANCH_OVERRIDE} fi Modified: releng/9.3/usr.sbin/freebsd-update/freebsd-update.8 ============================================================================== --- releng/9.3/usr.sbin/freebsd-update/freebsd-update.8 Wed May 13 22:52:35 2015 (r282873) +++ releng/9.3/usr.sbin/freebsd-update/freebsd-update.8 Wed May 13 22:52:51 2015 (r282874) @@ -25,7 +25,7 @@ .\" .\" $FreeBSD$ .\" -.Dd July 14, 2010 +.Dd March 2, 2015 .Dt FREEBSD-UPDATE 8 .Os FreeBSD .Sh NAME @@ -36,10 +36,12 @@ .Op Fl b Ar basedir .Op Fl d Ar workdir .Op Fl f Ar conffile +.Op Fl F .Op Fl k Ar KEY .Op Fl r Ar newrelease .Op Fl s Ar server .Op Fl t Ar address +.Op Fl -not-running-from-cron .Cm command ... .Sh DESCRIPTION The @@ -54,16 +56,16 @@ by the .Fx Release Engineering Team, e.g., .Fx -7.3-RELEASE and +9.3-RELEASE and .Fx -8.0-RELEASE, but not +10.1-RELEASE, but not .Fx -6.3-STABLE or +9.3-STABLE or .Fx -9.0-CURRENT. +11-CURRENT. .Sh OPTIONS The following options are supported: -.Bl -tag -width "-f conffile" +.Bl -tag -width "-r newrelease" .It Fl b Ar basedir Operate on a system mounted at .Ar basedir . @@ -81,6 +83,10 @@ Read configuration options from .Ar conffile . (default: .Pa /etc/freebsd-update.conf ) +.It Fl F +Force +.Nm Cm fetch +to proceed where it normally would not, such as an unfinished upgrade .It Fl k Ar KEY Trust an RSA key with SHA256 of .Ar KEY . @@ -98,12 +104,21 @@ Mail output of command, if any, to .Ar address . (default: root, or as given in the configuration file.) +.It Fl -not-running-from-cron +Force +.Nm Cm fetch +to proceed when there is no controlling tty. +This is for use by automated scripts and orchestration tools. +Please do not run +.Nm Cm fetch +from crontab or similar using this flag, see: +.Nm Cm cron .El .Sh COMMANDS The .Cm command can be any one of the following: -.Bl -tag -width "-f conffile" +.Bl -tag -width "rollback" .It Cm fetch Based on the currently installed world and the configuration options set, fetch all available binary updates. Modified: releng/9.3/usr.sbin/freebsd-update/freebsd-update.sh ============================================================================== --- releng/9.3/usr.sbin/freebsd-update/freebsd-update.sh Wed May 13 22:52:35 2015 (r282873) +++ releng/9.3/usr.sbin/freebsd-update/freebsd-update.sh Wed May 13 22:52:51 2015 (r282874) @@ -43,12 +43,15 @@ Options: (default: /var/db/freebsd-update/) -f conffile -- Read configuration options from conffile (default: /etc/freebsd-update.conf) + -F -- Force a fetch operation to proceed -k KEY -- Trust an RSA key with SHA256 hash of KEY -r release -- Target for upgrade (e.g., 6.2-RELEASE) -s server -- Server from which to fetch updates (default: update.FreeBSD.org) -t address -- Mail output of cron command, if any, to address (default: root) + --not-running-from-cron + -- Run without a tty, for use by automated tools Commands: fetch -- Fetch updates from server cron -- Sleep rand(3600) seconds, fetch updates, and send an @@ -399,6 +402,12 @@ init_params () { # No commands specified yet COMMANDS="" + + # Force fetch to proceed + FORCEFETCH=0 + + # Run without a TTY + NOTTYOK=0 } # Parse the command line @@ -411,6 +420,12 @@ parse_cmdline () { if [ ! -z "${CONFFILE}" ]; then usage; fi shift; CONFFILE="$1" ;; + -F) + FORCEFETCH=1 + ;; + --not-running-from-cron) + NOTTYOK=1 + ;; # Configuration file equivalents -b) @@ -665,6 +680,14 @@ fetch_check_params () { echo "(Did you mean 'upgrade' instead?)" exit 1 fi + + # Check that we have updates ready to install + if [ -f ${BDHASH}-install/kerneldone -a $FORCEFETCH -eq 0 ]; then + echo "You have a partially completed upgrade pending" + echo "Run '$0 install' first." + echo "Run '$0 fetch -F' to proceed anyway." + exit 1 + fi } # Perform sanity checks etc. before fetching upgrades. @@ -3202,7 +3225,7 @@ get_params () { # Fetch command. Make sure that we're being called # interactively, then run fetch_check_params and fetch_run cmd_fetch () { - if [ ! -t 0 ]; then + if [ ! -t 0 -a $NOTTYOK -eq 0 ]; then echo -n "`basename $0` fetch should not " echo "be run non-interactively." echo "Run `basename $0` cron instead."