From owner-freebsd-stable@FreeBSD.ORG Fri Jul 19 08:09:21 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 42B5EB93; Fri, 19 Jul 2013 08:09:21 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.16.84]) by mx1.freebsd.org (Postfix) with ESMTP id BDEEF8A9; Fri, 19 Jul 2013 08:09:20 +0000 (UTC) Received: from pampa.cs.huji.ac.il ([132.65.80.32]) by kabab.cs.huji.ac.il with esmtp id 1V05bF-000InB-3t; Fri, 19 Jul 2013 10:59:37 +0300 X-Mailer: exmh version 2.7.2 01/07/2005 with nmh-1.3 To: Jeremy Chadwick Subject: Re: make buildworld is now 50% slower In-reply-to: <20130707101714.GA51445@icarus.home.lan> References: <20130705145839.GB5449@icarus.home.lan> <20130707101714.GA51445@icarus.home.lan> Comments: In-reply-to Jeremy Chadwick message dated "Sun, 07 Jul 2013 03:17:14 -0700." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Fri, 19 Jul 2013 10:59:37 +0300 From: Daniel Braniss Message-ID: Cc: "freebsd-stable@freebsd.org Stable" , Dimitry Andric X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Jul 2013 08:09:21 -0000 > On Sun, Jul 07, 2013 at 11:50:29AM +0300, Daniel Braniss wrote: > > > On Fri, Jul 05, 2013 at 02:39:00PM +0200, Dimitry Andric wrote: > > > > [redirecting to the correct mailing list, freebsd-stable@ ...] > > > > > > > > On Jul 5, 2013, at 10:53, Daniel Braniss wrote: > > > > > after today's update of 9.1-STABLE I noticed that make build[world|kernel] are > > > > > taking conciderable more time, is it because the upgrade of clang? > > > > > and if so, is the code produced any better? > > > > > > > > > > before: > > > > > buildwordl: 26m4.52s real 2h28m32.12s user 36m6.27s sys > > > > > buildkernel: 7m29.42s real 23m22.22s user 4m26.26s sys > > > > > > > > > > today: > > > > > buildwordl: 34m29.80s real 2h38m9.37s user 37m7.61s sys > > > > > buildkernel: 15m31.52s real 22m59.40s user 4m33.06s sys > > > > > > > > Ehm, your user and sys times are not that much different at all, they > > > > add up to about 5% slower for buildworld, and 1% faster for build kernel. > > > > Are you sure nothing else is running on that machine, eating up CPU time > > > > while you are building? :) > > > > > > > > But yes, clang 3.3 is of course somewhat larger than 3.2. You might > > > > especially notice that, if you are using gcc, which is very slow at > > > > compiling C++. > > > > > > > > In any case, if you do not care about clang, just set WITHOUT_CLANG= in > > > > your /etc/src.conf, and you can shave off some build time. > > > > > > I just built world/kernel (stable/9 r252769) 5 hours ago. Results: > > > > > > time make -j4 buildworld = roughly 21 minutes on my hardware > > > time make -j4 buildkernel = roughly 8 minutes on my hardware > > > > > > > It's been a long time since I saw such numbers, maybe it's time > > to see where time is being spent, I will run it without clang to compare with > > your numbers. > > > > > These numbers are about the norm for me, meaning I do not see a > > > substantial increase in build times. > > > > > > Key point: I do not use/build/grok clang, i.e. WITHOUT_CLANG=true is in > > > my src.conf. But I am aware of the big clang change in r252723. > > > > > > If hardware details are wanted, ask, but I don't think it's relevant to > > > what the root cause is. > > > > > > > from what you are saying, I guess clang is not responsible. > > looking for my Sherlock Holmes hat. > > Some points to those numbers I stated above: > > - System is an Intel Q9550 with 8GB of RAM > > - Single SSD (UFS2+SU+TRIM) is used for root, /usr, /var, /tmp, and swap > > - /usr/src is on ZFS (raidz1 + 3 disks) -- however I got equally small > numbers when it was on the SSD > > - /usr/src is using compression=lz4 (to folks from -fs: yeah, I'm > trying it out to see how much of an impact it has on interactivity. I > can still tell when it kicks in, but it's way, way better than lzjb. > Rather not get into that here) > > - Contents of /etc/src.conf (to give you some idea of what I disable): > > WITHOUT_ATM=true > WITHOUT_BLUETOOTH=true > WITHOUT_CLANG=true > WITHOUT_FLOPPY=true > WITHOUT_FREEBSD_UPDATE=true > WITHOUT_INET6=true > WITHOUT_IPFILTER=true > WITHOUT_IPX=true > WITHOUT_KERBEROS=true > WITHOUT_LIB32=true > WITHOUT_LPR=true > WITHOUT_NDIS=true > WITHOUT_NETGRAPH=true > WITHOUT_PAM_SUPPORT=true > WITHOUT_PPP=true > WITHOUT_SENDMAIL=true > WITHOUT_WIRELESS=true > WITH_OPENSSH_NONE_CIPHER=true > > It's WITHOUT_CLANG that cuts down the buildworld time by a *huge* amount > (I remember when it got introduced, my buildworld jumped up to something > like 40 minutes); the rest probably save a minute or two at most. > > - /etc/make.conf doesn't contain much that's relevant, other than: > > CPUTYPE?=core2 > > # For DTrace; also affects ports > STRIP= > CFLAGS+= -fno-omit-frame-pointer > > - I do some tweaks in /etc/sysctl.conf (mainly vfs.read_min and > vfs.read_max), but I will admit I am not completely sure what those > do quite yet (I just saw the commit from scottl@ a while back talking > about how an increased vfs.read_min helps them at Netflix quite a > lot). I also adjust kern.maxvnodes. > > - Some ZFS ARC settings are adjusted in /boot/loader.conf (I'm playing > with some stuff I read in Andriy Gapon's ZFS PDF), but they definitely > do not have a major impact on the numbers I listed off. > > - I do increase kern.maxdsiz, kern.dfldsiz, and kern.maxssiz in > /boot/loader.conf to 2560M/2560M/256M respectively, but that was mainly > from the days when I ran MySQL and needed a huge userland processes. > > All in all my numbers are low/small because of two things: the SSD, and > WITHOUT_CLANG. > > Hope this gives you somewhere to start/stuff to ponder. > indeed! on my pretty much standard dev machine, PowerEdge R710, X5550 @ 2.67GHz, 16384 MB, root on ufs, the rest is via ZFS, this is what I'm getting for buildworld: with clang 27m32.91s real 2h42m52.82s user 36m20.69s sys without clang 13m24.19s real 1h23m26.52s user 29m5.18s sys on a similar machine but with root via NFS, and /var on ZFS, the numbers are similar: with clang 23m30.92s real 2h9m8.85s user 29m27.84s sys without clang 12m7.53s real 1h7m54.24s user 23m9.78s sys (this host is newer, PowerEdge C6220, E5-2643 0 @ 3.30GHz, 32768 MB) it's still not clear what happened last week, but probably my machine had some runaway process. cheers, danny