From owner-freebsd-testing@FreeBSD.ORG Mon May 11 14:46:16 2015 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 69290EBE for ; Mon, 11 May 2015 14:46:16 +0000 (UTC) Received: from mail-ob0-x234.google.com (mail-ob0-x234.google.com [IPv6:2607:f8b0:4003:c01::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 300DB1C3F for ; Mon, 11 May 2015 14:46:16 +0000 (UTC) Received: by obcus9 with SMTP id us9so71742749obc.2 for ; Mon, 11 May 2015 07:46:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=2xqdNYt8IOdd/ZbMuN4agRe0SKqCuQYVrf5z8beWHJo=; b=ALYJa0aMmPvmUWGUc0fHvadhT91ujH865AmuFh8QBhqwTdZTT8a86ZatoRlwGp++0O eAKCHZ2/BM6FkunJclWjdf/A7F0BtGBEHye9/WKn08NeiP5449f+bGeF2LyABdVbY+LC lckbCUslSUH+7i9lGwSFTcGl1fO0REyvaxEBeHQMAXy07NoRPUt41r/mJV5wL1LwQnD6 0Bz4LYetBEMMJ6OWABLWA0T/lWWztiKBRb4LcBwzaZtJkT5igHvgd8jkA1E8ZLmHvYuK 5vgJyWHKOljfR/rPRLfo5NG1eVJ3p7MeC7j220U7JNDVx/OWYM/2k6vufR5pwcVhSHcU Nxgw== MIME-Version: 1.0 X-Received: by 10.202.172.196 with SMTP id v187mr7667129oie.51.1431355574916; Mon, 11 May 2015 07:46:14 -0700 (PDT) Sender: asomers@gmail.com Received: by 10.202.173.204 with HTTP; Mon, 11 May 2015 07:46:14 -0700 (PDT) In-Reply-To: References: Date: Mon, 11 May 2015 08:46:14 -0600 X-Google-Sender-Auth: ntIqpL2x_BEvr4SYxXfYI-drGFw Message-ID: Subject: Re: Freebsd testing methodology From: Alan Somers To: Siddharth Tuli Cc: "freebsd-testing@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 May 2015 14:46:16 -0000 On Fri, May 8, 2015 at 2:56 PM, Siddharth Tuli wrote: > I am looking out for all kinds testing that is being carried out for > FreeBSD. > I know of, > -stress2 tests, that mostly exercise the kernel components > -AFT tests > -POSIX tests. > > Are there other test suites that are being used for testing FreeBSD? > > Regards > Siddharth There's also the ZFS test suite, which uses ATF. https://svnweb.freebsd.org/base/projects/zfsd/head/tests/sys/cddl/ -Alan From owner-freebsd-testing@FreeBSD.ORG Tue May 12 08:31:54 2015 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BD778D61 for ; Tue, 12 May 2015 08:31:54 +0000 (UTC) Received: from mail-ig0-x22e.google.com (mail-ig0-x22e.google.com [IPv6:2607:f8b0:4001:c05::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8A9A41AFE for ; Tue, 12 May 2015 08:31:54 +0000 (UTC) Received: by igbsb11 with SMTP id sb11so2923488igb.0 for ; Tue, 12 May 2015 01:31:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=bo1AJPnl+rvGx8ySZNCVqM7KiIPJmoN3cOMH9GwOKiA=; b=nvOLVNYtoSMgnDhtwFviVyfvgUNLaSaiU/REOgdO93KrqoXbZAkE/PzU2vQr+Z0IM2 Xq0QPYKR7SRDFWjOs1AtKKjBicsUCRbRwN+pBe8qKPl6RmMRBZEscCurgPr+UM5vjlxB zDHiNNtTO4340o1n6tRQ51br5QJ5QQn1/BzE7GJjbUYN+1bx7T33ABRonDIxT64HYr/M OSO8HRQT+NHX3wVS1V8pX2Iq8jKtCF7D9a+SqDQfgl6vVMIyYO2qcW1HyqeEVad5L8x1 p4yfdNrJSGIAVec/PGIMpFF6sMvx4EkXzN54+pE1ZM5uUB/cv3TYMuagGUv1+QOnn5AF 9vcA== MIME-Version: 1.0 X-Received: by 10.50.142.67 with SMTP id ru3mr2219077igb.40.1431419514009; Tue, 12 May 2015 01:31:54 -0700 (PDT) Received: by 10.36.30.137 with HTTP; Tue, 12 May 2015 01:31:53 -0700 (PDT) Date: Tue, 12 May 2015 01:31:53 -0700 Message-ID: Subject: Getting timestamp data with kyua report, etc (ticket/stopgap solution) From: Garrett Cooper To: "freebsd-testing@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 May 2015 08:31:54 -0000 Hi all, With the increased adoption of kyua at $work as more and more subsystems and external components have been thrown into mix, one of the pain points (for us) has been correlating data retrieved by syslogd with testcase results. I filed an kyua ticket for this issue ( https://github.com/jmmv/kyua/issues/139 ), but in the interim have also thrown together a python script for this issue : https://github.com/yaneurabeya/scratch/blob/master/bayonetta/scratch/scripts/freebsd/annotate_kyua_runs_with_date.py . It's not useful with parallel execution, but it works for serial execution quite nicely -- all you need is the starting date. I've included some sample output below. Any and all feedback would be appreciated. Thanks, -NGie $ curl -s XXXX | ./annotate_kyua_runs_with_date.py --start-time-format '%a %b %d %H:%M:%S %Y' --start-time 'Wed Apr 22 01:44:51 2015' 2015-04-22 01:44:51: bin/sh/legacy_test:main -> passed [12.28] 2015-04-22 01:45:03: bin/mv/legacy_test:main -> passed [3.43] 2015-04-22 01:45:06: bin/ls/t_ls:ls_without_option -> passed [0.113] 2015-04-22 01:45:06: bin/ls/t_ls:ls_inode_hexadecimal -> passed [0.085] 2015-04-22 01:45:06: bin/ls/t_ls:ls_inode_validate -> passed [0.051] 2015-04-22 01:45:06: bin/ls/t_ls:ls_with_option_J -> passed [0.071] 2015-04-22 01:45:07: bin/ls/t_ls:ls_without_option_J -> passed [0.085] 2015-04-22 01:45:07: bin/ls/t_ls:ls_acl -> passed [0.228] 2015-04-22 01:45:07: bin/chmod/t_chmod:chmod_nonexistent_file -> passed [0.054] 2015-04-22 01:45:07: bin/chmod/t_chmod:chmod_file -> passed [0.1] 2015-04-22 01:45:07: bin/chmod/t_chmod:chmod_symlink -> passed [0.151] 2015-04-22 01:45:07: bin/chmod/t_chmod:chmod_follow_symlink -> passed [0.127] 2015-04-22 01:45:07: bin/chmod/t_chmod:chmod_directory -> passed [0.096] 2015-04-22 01:45:07: bin/chmod/t_chmod:chmod_directory_recursive -> passed [0.235] 2015-04-22 01:45:08: bin/chmod/t_chmod:chmod_file_op_symbols -> passed [0.118] 2015-04-22 01:45:08: bin/dd/t_dd:io -> passed [0.136] 2015-04-22 01:45:08: bin/dd/t_dd:seek -> passed [0.175] 2015-04-22 01:45:08: bin/dd/t_dd:irseek -> passed [0.121] 2015-04-22 01:45:08: bin/dd/t_dd:orseek -> passed [0.129] 2015-04-22 01:45:08: bin/pax/legacy_test:main -> passed [0.34] 2015-04-22 01:45:09: bin/date/legacy_test:main -> passed [0.21] 2015-04-22 01:45:09: bin/pkill/pgrep-F_test:main -> passed [0.354] 2015-04-22 01:45:09: bin/pkill/pgrep-LF_test:main -> passed [0.659] 2015-04-22 01:45:10: bin/pkill/pgrep-P_test:main -> passed [0.36] 2015-04-22 01:45:10: bin/pkill/pgrep-U_test:main -> passed [0.717] 2015-04-22 01:45:11: bin/pkill/pgrep-_g_test:main -> passed [0.705] 2015-04-22 01:45:12: bin/pkill/pgrep-_s_test:main -> passed [0.055] 2015-04-22 01:45:12: bin/pkill/pgrep-g_test:main -> passed [0.667] 2015-04-22 01:45:12: bin/pkill/pgrep-i_test:main -> passed [0.368] 2015-04-22 01:45:13: bin/pkill/pgrep-l_test:main -> passed [0.369] 2015-04-22 01:45:13: bin/pkill/pgrep-n_test:main -> passed [0.35] 2015-04-22 01:45:13: bin/pkill/pgrep-o_test:main -> passed [0.364] 2015-04-22 01:45:14: bin/pkill/pgrep-q_test:main -> passed [0.361] 2015-04-22 01:45:14: bin/pkill/pgrep-s_test:main -> passed [0.674] 2015-04-22 01:45:15: bin/pkill/pgrep-t_test:main -> passed [0.36] 2015-04-22 01:45:15: bin/pkill/pgrep-v_test:main -> passed [0.368] 2015-04-22 01:45:16: bin/pkill/pgrep-x_test:main -> passed [0.384] 2015-04-22 01:45:16: bin/pkill/pkill-F_test:main -> passed [0.368] 2015-04-22 01:45:16: bin/pkill/pkill-LF_test:main -> passed [0.702] 2015-04-22 01:45:17: bin/pkill/pkill-P_test:main -> passed [0.363] 2015-04-22 01:45:17: bin/pkill/pkill-U_test:main -> passed [0.721] 2015-04-22 01:45:18: bin/pkill/pkill-_g_test:main -> passed [0.688] 2015-04-22 01:45:19: bin/pkill/pkill-g_test:main -> passed [0.671] 2015-04-22 01:45:19: bin/pkill/pkill-i_test:main -> passed [0.361] 2015-04-22 01:45:20: bin/pkill/pkill-s_test:main -> passed [0.678] 2015-04-22 01:45:21: bin/pkill/pkill-t_test:main -> passed [0.669] 2015-04-22 01:45:21: bin/pkill/pkill-x_test:main -> passed [0.704] ...