From owner-freebsd-stable@FreeBSD.ORG Sun Sep 18 11:27:32 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 532D3106564A for ; Sun, 18 Sep 2011 11:27:32 +0000 (UTC) (envelope-from to.my.trociny@gmail.com) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id C77658FC0C for ; Sun, 18 Sep 2011 11:27:31 +0000 (UTC) Received: by fxg9 with SMTP id 9so4127360fxg.13 for ; Sun, 18 Sep 2011 04:27:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=from:to:cc:subject:references:x-comment-to:sender:date:in-reply-to :message-id:user-agent:mime-version:content-type; bh=DCb9nWRhgkFYt2UKVDOIV6H0YAjf4OBP6nsOn7FfUjQ=; b=bJR1gfrp5uBl8AiLOje0XDTm2/+QivXBRi6B+57d24VcF6he6O4wwVgxs/5hvYH+XK X64B/HMDFIWmvlc53xyXnuEyTA+U1/WR1GQ3/97T92bPXWpoVUDFPTL+t3GiIN29I6s2 OuWX4jQ3ryPELDndx6v8vT+FJ3QcL6QJPJIXc= Received: by 10.223.66.88 with SMTP id m24mr2948286fai.68.1316343517504; Sun, 18 Sep 2011 03:58:37 -0700 (PDT) Received: from localhost ([95.69.173.122]) by mx.google.com with ESMTPS id i5sm8919624fai.15.2011.09.18.03.58.34 (version=TLSv1/SSLv3 cipher=OTHER); Sun, 18 Sep 2011 03:58:35 -0700 (PDT) From: Mikolaj Golub To: "Ronald Klop" References: <20110918045413.GA63773@DataIX.net> <20110918053901.GA31617@icarus.home.lan> X-Comment-To: Ronald Klop Sender: Mikolaj Golub Date: Sun, 18 Sep 2011 13:58:32 +0300 In-Reply-To: (Ronald Klop's message of "Sun, 18 Sep 2011 08:47:13 +0200") Message-ID: <86d3eydsmf.fsf@kopusha.home.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.2 (berkeley-unix) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Cc: "freebsd-stable@freebsd.org" , Jeremy Chadwick , cperciva@freebsd.org Subject: Re: /usr/bin/script eating 100% cpu with portupgrade and xargs X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Sep 2011 11:27:32 -0000 --=-=-= On Sun, 18 Sep 2011 08:47:13 +0200 Ronald Klop wrote: RK> On Sun, 18 Sep 2011 07:39:01 +0200, Jeremy Chadwick RK> wrote: >> On Sun, Sep 18, 2011 at 12:54:13AM -0400, Jason Hellenthal wrote: >>> On Sun, Sep 18, 2011 at 01:49:15AM +0200, Ronald Klop wrote: >>> > Hi, >>> > >>> > I'm running portupgrade in screen to update all the ports for >>> > 9-BETA2/9-CURRENT on amd64. While doing this script eats 100% cpu. >>> > Because portupgrade -fa crashed I'm running this command to update the >>> > remaining non-updates ports. >>> > find /var/db/pkg -name +DESC -mtime +2 |cut -d / -f 5 | xargs >>> time nice -n >>> > 20 portupgrade -f >>> > >>> > The output of truss -p `pgrep script` is this: >>> > clock_gettime(13,{1316301104.000000000 }) = 0 (0x0) >>> > select(5,{0 4},0x0,0x0,{30.000000 }) = 1 (0x1) >>> > read(0,0x7fffffffcdf0,1024) = 0 (0x0) >>> > write(4,0x7fffffffcdf0,0) = 0 (0x0) >>> > clock_gettime(13,{1316301104.000000000 }) = 0 (0x0) >>> > select(5,{0 4},0x0,0x0,{30.000000 }) = 1 (0x1) >>> > read(0,0x7fffffffcdf0,1024) = 0 (0x0) >>> > write(4,0x7fffffffcdf0,0) = 0 (0x0) >>> > clock_gettime(13,{1316301104.000000000 }) = 0 (0x0) >>> > select(5,{0 4},0x0,0x0,{30.000000 }) = 1 (0x1) >>> > read(0,0x7fffffffcdf0,1024) = 0 (0x0) >>> > write(4,0x7fffffffcdf0,0) = 0 (0x0) >>> > clock_gettime(13,{1316301104.000000000 }) = 0 (0x0) >>> > select(5,{0 4},0x0,0x0,{30.000000 }) = 1 (0x1) >>> > read(0,0x7fffffffcdf0,1024) = 0 (0x0) >>> > write(4,0x7fffffffcdf0,0) = 0 (0x0) >>> > >>> > So it is really fast in reading and writing 0 bytes most of the time. >>> > >>> > I also found >>> http://web.archiveorange.com/archive/v/6ETvLvjo60Gj9geAUAb6 >>> > and I think I am better of by rewriting my command so stdin/stdout is >>> > still the terminal. Although the link is a couple of years old. >>> > >>> > Is this known? Can somebody explain me why my xargs command is >>> not working >>> > well? >>> > >>> >>> Are you absolutely sure that its script(1) causing this ? 100% CPU usage >>> has been a known side effect of screen(1) for quite some time. Rebuild >>> it and try again. >> >> Jason's referring to this, I believe: >> http://www.freebsd.org/cgi/cvsweb.cgi/ports/sysutils/screen/Makefile#rev1.55 >> >> To clarify the what the commit message means: it does not mean "when the >> package is installed the installation takes up 100% CPU". It means >> "once the package is installed and screen is used, screen takes up 100% >> CPU". I know because I've seen this behaviour in the past (one of the >> many, many reasons I build ports from source). >> >> However: >> http://www.freebsd.org/cgi/cvsweb.cgi/ports/sysutils/screen/Makefile#rev1.78 >> >> So: If a binary package is being installed through your above >> portupgrade command, and you're seeing this problem, then it sounds to >> me like commit revision 1.78 is a regression and NO_PACKAGE should be >> put back into place + packages removed from all mirrors. >> >> There are many reasons to not use GNU screen at all, or if you must have >> something like it, use tmux. I recently had to provide an analysis of >> how GNU screen destroys one's terminal[1]; so if the above problem turns >> out to be caused by GNU screen as well, I'll just add it to my >> ever-growing list of reasons the software should be nuked from orbit. >> >> Otherwise, if this turns out to be a problem with portupgrade (which you >> found some evidence supporting such), then the solution is simple: stop >> using portupgrade, use portmaster (if it lacks things you need ask Doug >> Barton, he's incredibly receptive to adding new features/fixing things). >> Two databases that aren't compatible, ruby shims, and other crap = not >> worth it. Think the database ordeal is long over with/fixed/whatever? >> It isn't[2]. >> >> [1]: >> http://lists.freebsd.org/pipermail/freebsd-stable/2011-June/063052.html >> [2]: >> http://www.dslreports.com/forum/r26304856-FreeBSD-defining-portmaster-alias >> RK> I have a repeatable test. Run top in a window and this command in another. RK> $ echo test | script /tmp/script-test sleep 1000 RK> Script started, output file is /tmp/script-test RK> test RK> PID USERNAME THR PRI NICE SIZE RES STATE C TIME RK> CPU COMMAND RK> 29656 ronald 1 103 0 12324K 1244K CPU4 4 1:03 RK> 100.00% script RK> So it has nothing to do with portupgrade or screen. The output of RK> truss -p29656 is the same as posted previously. I believe the behaviour is after this commit: http://svnweb.freebsd.org/base?view=revision&revision=125848 I think we should skip select on STDIN after reading EOF from it, like in the patch below. -- Mikolaj Golub --=-=-= Content-Type: text/x-patch Content-Disposition: inline; filename=script.c.patch Index: usr.bin/script/script.c =================================================================== --- usr.bin/script/script.c (revision 225648) +++ usr.bin/script/script.c (working copy) @@ -53,6 +53,7 @@ static const char sccsid[] = "@(#)script.c 8.1 (Be #include #include #include +#include #include #include #include @@ -86,6 +87,7 @@ main(int argc, char *argv[]) char ibuf[BUFSIZ]; fd_set rfd; int flushtime = 30; + bool stdin_closed; aflg = kflg = 0; while ((ch = getopt(argc, argv, "aqkt:")) != -1) @@ -161,10 +163,12 @@ main(int argc, char *argv[]) tvp = NULL; start = time(0); - FD_ZERO(&rfd); + stdin_closed = false; for (;;) { + FD_ZERO(&rfd); FD_SET(master, &rfd); - FD_SET(STDIN_FILENO, &rfd); + if (!stdin_closed) + FD_SET(STDIN_FILENO, &rfd); if (flushtime > 0) { tv.tv_sec = flushtime; tv.tv_usec = 0; @@ -176,8 +180,13 @@ main(int argc, char *argv[]) cc = read(STDIN_FILENO, ibuf, BUFSIZ); if (cc < 0) break; - if (cc == 0) + if (cc == 0) { + /* + * Send EOF to the program being scripted. + */ (void)write(master, ibuf, 0); + stdin_closed = true; + } if (cc > 0) { (void)write(master, ibuf, cc); if (kflg && tcgetattr(master, &stt) >= 0 && --=-=-=--