From owner-freebsd-ports@freebsd.org Fri Oct 30 23:36:49 2015 Return-Path: Delivered-To: freebsd-ports@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 71BFCA221CC for ; Fri, 30 Oct 2015 23:36:49 +0000 (UTC) (envelope-from rcarter@pinyon.org) Received: from quine.pinyon.org (quine.pinyon.org [65.101.5.249]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 27ADC19B8; Fri, 30 Oct 2015 23:36:48 +0000 (UTC) (envelope-from rcarter@pinyon.org) Received: by quine.pinyon.org (Postfix, from userid 122) id 528651601D0; Fri, 30 Oct 2015 16:36:40 -0700 (MST) X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on quine.pinyon.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham autolearn_force=no version=3.4.1 Received: from feyerabend.n1.pinyon.org (acipenser.esturion.net [65.101.5.252]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by quine.pinyon.org (Postfix) with ESMTPSA id CC2D01601D0; Fri, 30 Oct 2015 16:36:35 -0700 (MST) Subject: Re: hung poudriere bulk recovery To: Bryan Drewery , FreeBSD Ports ML References: <562A6185.5000305@pinyon.org> <563147BE.2070604@FreeBSD.org> From: "Russell L. Carter" Message-ID: <5633FF03.1090204@pinyon.org> Date: Fri, 30 Oct 2015 16:36:35 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <563147BE.2070604@FreeBSD.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Oct 2015 23:36:49 -0000 Got a hung poudriere run last night: On 10/28/15 15:10, Bryan Drewery wrote: > On 10/23/2015 9:34 AM, Russell L. Carter wrote: >> >> Greetings, >> >> Recently my nightly cron poudriere builds have been occasionally >> hanging. For instance, here's last night's, with apparently no >> progress for over 10 hours: >> >> root@terpsichore> poudriere status >> SET PORTS JAIL BUILD STATUS QUEUE >> BUILT FAIL SKIP IGNORE REMAIN TIME LOGS >> - default 10-stable-amd64 2015-10-22_22h30m08s parallel_build 488 >> 34 0 0 0 454 10:45:56 >> /ssd1/poudriere/data/logs/bulk/10-stable-amd64-default/2015-10-22_22h30m08s >> root@terpsichore> >> > > Also check 'poudriere status -b' to see per-builder status. Something > may be actually doing something. Poudriere will timeout builds after a > long time. I forget the default but it may be up to 24 hours. > root@terpsichore> date Fri Oct 30 15:19:55 MST 2015 root@terpsichore> poudriere status -b [10-stable-amd64-default] [2015-10-29_22h30m07s] [parallel_build:] Queued: 129 Built: 34 Failed: 0 Skipped: 0 Ignored: 0 Tobuild: 95 Time: 16:49:55 [01]: x11-toolkits/gtk30 build_port_done (16:40:58) [02]: graphics/ImageMagick build_port_done (16:43:16) [03]: www/webkit-gtk2 build_port_done (16:43:58) ====>> Logs: /ssd1/poudriere/data/logs/bulk/10-stable-amd64-default/2015-10-29_22h30m07s root@terpsichore> > Please record 'procstat -kka' before rebooting in case this is some kind > of deadlock. invoked right after the poudriere status -b: http://rcarter.esturion.net/procstat-kka.txt >> >> I'm not sure how to debug this, but in the interim, I'm very curious >> how I can stop the hung bulk run, and either restart it, or clean up >> the various mounted zfs filesystems and manually restart from the >> beginning w/o rebooting. Studying the man page, it's not clear at all >> the Right Way to do this, so any pointers here would be appreciated. > > Kill -TERM the main poudriere process. It will clean up children. > Beyond that you can 'poudriere jail -j NAME -p TREE -z SET -k' to clean > up any mounts leftover from a previous build. A bit of trial and error led me to this solution, without either '-p' or '-z', so I'm good. In the above, I ran the exact same bulk script manually and the poudriere bulk build ran to completion. Thanks! Russell