From owner-freebsd-hackers@FreeBSD.ORG Tue Jan 7 19:33:28 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3D8023D9; Tue, 7 Jan 2014 19:33:28 +0000 (UTC) Received: from mail-wg0-x231.google.com (mail-wg0-x231.google.com [IPv6:2a00:1450:400c:c00::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A45671653; Tue, 7 Jan 2014 19:33:27 +0000 (UTC) Received: by mail-wg0-f49.google.com with SMTP id x12so566854wgg.4 for ; Tue, 07 Jan 2014 11:33:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=E8jtLqdjkfn2mU4zDqNdg0qZp1+KkBmpsupiNvJCAfQ=; b=0pOHZkxBWqSH8wj4zruHXu5zDzUV9Vey1Elz5WWhLaUs0mCIAlh5c+xmlf/Iq+cSaG px6XZP/HPClWovRhg/XP9RFf5jJtQ3q/BJineUEXtBxA8A2TqSeSbaDAKXZ1hgVKp4Ug yubkQjNW3+UfaMPEvuS54/pnbCWl9bw9npjlRqTW01CU1ECOc9qHhfYX2rLgOP5gU2fe cFR4GHLnahqFdGgdSpSiQvC1fC7d9tR2Jflw5u3ImE8DfVDQa6vEOJf9X/iW2ThnClR0 gIbZsmOghFBZw/5njUeLiVm/jDqA7k7XXPP+/0REIR2rToknBoDTkxlsBjq2Zg3PPMh5 te0w== MIME-Version: 1.0 X-Received: by 10.180.37.69 with SMTP id w5mr18116617wij.53.1389123206030; Tue, 07 Jan 2014 11:33:26 -0800 (PST) Sender: asomers@gmail.com Received: by 10.194.22.35 with HTTP; Tue, 7 Jan 2014 11:33:25 -0800 (PST) In-Reply-To: References: Date: Tue, 7 Jan 2014 12:33:25 -0700 X-Google-Sender-Auth: N4Pw2HhpsA3KVCsyTthw5Xw59e8 Message-ID: Subject: Re: Continual benchmarking / regression testing? From: Alan Somers To: Ivan Voras Content-Type: text/plain; charset=ISO-8859-1 Cc: Julio Merino , "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Jan 2014 19:33:28 -0000 On Tue, Jan 7, 2014 at 10:49 AM, Ivan Voras wrote: > On 7 January 2014 18:01, Alan Somers wrote: >> Unfortunately, my framework is extremely incomplete. It's not even >> good enough for internal use, much less a wider audience. And I fear >> that my bosses won't give me any more time to work on it. It's also >> written in Ruby and uses STAF to command slave nodes, which the >> FreeBSD community might not be excited about. However, if there is >> any interest, I can ask for permission to share my design as a >> starting point for a more general framework. > > I don't think I have the time+spare brainpower to learn Ruby (I'm a > Pythonista) but it would be interested to see the design you've > created. > > I've looked at STAF before and I'm curious why you picked it? It > seemed a combination of being antiquated and an overkill to me... I picked Ruby and STAF for the same reason: we're already using them at $JOB. In the case of STAF, we were already using it as part of our continuous install and test system. We aren't using very many of its capabilities, though. In particular, we've long since switched from STAX to ATF/Kyua. Basically, we only use STAF to execute commands on a remote machine. It offers a little more control than using SSH. We also use the STAF semaphore service to ensure exclusive access to both test machines and network slaves.