From owner-freebsd-hackers@freebsd.org Sat Aug 31 14:54:27 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0788ED3D4D; Sat, 31 Aug 2019 14:54:27 +0000 (UTC) (envelope-from sjg@juniper.net) Received: from mx0a-00273201.pphosted.com (mx0a-00273201.pphosted.com [208.84.65.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.pphosted.com", Issuer "Thawte RSA CA 2018" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46LKBB5rXTz4YlV; Sat, 31 Aug 2019 14:54:26 +0000 (UTC) (envelope-from sjg@juniper.net) Received: from pps.filterd (m0108158.ppops.net [127.0.0.1]) by mx0a-00273201.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id x7VEsO0V018357; Sat, 31 Aug 2019 07:54:24 -0700 Received: from nam02-bl2-obe.outbound.protection.outlook.com (mail-bl2nam02lp2051.outbound.protection.outlook.com [104.47.38.51]) by mx0a-00273201.pphosted.com with ESMTP id 2uqmkq8dvg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sat, 31 Aug 2019 07:54:24 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Do0TjvybFu5WsE+DMB8AHNsYtFYiASK+sToCaFvFTGr1snY9WqgpmKOT1jX0L2DFW7L+7tkjZxR8k2F83VrG1W+Kp24nnV128oqzYeW81K8H6LSawLCdf8CbzCP8sy19mwyPH3RBAojJVrSUrOcFMnC4YcdPY1IwkXxgnVGvshTEFi5IvOqHgSEhq1u4dMoTG333rONOuJaZWJbSxLHyyarVvXxKL6lP409w/OYEPyk/8M16Eb96Nn+eS7tacYPn6FLVil5tMYnFFiKFYk9ylfBVO2726qsRVtl1NE+XdQjHERCmLKZsv+QBk52cY3tt0UZ//BBQhJW+/hX/EW3v9A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JOI47K+2zg0zrn1pD9EAZcRd+qqbszg8o1IKtPLoWXY=; b=SxE4qQzGgZnRvKc1TWYGBylfLTmq4NeGPty38pfDF07BC+VP+eDOXiInBucnqFF/2bODIPDQ9G/uRSSsqDDC+e8Sn/asfchErGUiJ10WiC2YPjaNTxqpivz5/ZsJVd8We+LoYnrh0W4q30DILETDhuOndktHpW8otxlUvfuNOeb6GWHSMj/GZQiyKddbsyN2GilZaUb43Qd5urfe4iQ8lEJXMWT4TeykHOwTz39n4VxQa453IBYxrh7fvoiPoW8+wEGWmU0dkCcedU1j61mPrf4bG5W9Ua9ZYOgztQMQDVn1LiGRI12MVfk1hqij/xdX5MMg2d/7iBoXQK1RPAuM/w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=softfail (sender ip is 66.129.239.13) smtp.rcpttodomain=gmail.com smtp.mailfrom=juniper.net; dmarc=fail (p=reject sp=reject pct=100) action=oreject header.from=juniper.net; dkim=none (message not signed); arc=none Received: from BN3PR05CA0028.namprd05.prod.outlook.com (2603:10b6:400::38) by BL0PR05MB4692.namprd05.prod.outlook.com (2603:10b6:208:29::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2241.4; Sat, 31 Aug 2019 14:54:21 +0000 Received: from DM3NAM05FT013.eop-nam05.prod.protection.outlook.com (2a01:111:f400:7e51::203) by BN3PR05CA0028.outlook.office365.com (2603:10b6:400::38) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2241.7 via Frontend Transport; Sat, 31 Aug 2019 14:54:21 +0000 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.13 as permitted sender) Received: from P-EXFEND-EQX-02.jnpr.net (66.129.239.13) by DM3NAM05FT013.mail.protection.outlook.com (10.152.98.122) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.2241.7 via Frontend Transport; Sat, 31 Aug 2019 14:54:19 +0000 Received: from P-EXBEND-EQX-01.jnpr.net (10.104.8.52) by P-EXFEND-EQX-02.jnpr.net (10.104.8.55) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Sat, 31 Aug 2019 07:54:19 -0700 Received: from P-EXBEND-EQX-02.jnpr.net (10.104.8.53) by P-EXBEND-EQX-01.jnpr.net (10.104.8.52) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Sat, 31 Aug 2019 07:54:18 -0700 Received: from p-mailhub01.juniper.net (10.104.20.6) by P-EXBEND-EQX-02.jnpr.net (10.104.8.53) with Microsoft SMTP Server (TLS) id 15.0.1367.3 via Frontend Transport; Sat, 31 Aug 2019 07:54:18 -0700 Received: from kaos.jnpr.net (kaos.jnpr.net [172.23.50.162]) by p-mailhub01.juniper.net (8.14.4/8.11.3) with ESMTP id x7VEsIiH018044; Sat, 31 Aug 2019 07:54:18 -0700 (envelope-from sjg@juniper.net) Received: by kaos.jnpr.net (Postfix, from userid 1377) id 5CA113F79D; Sat, 31 Aug 2019 07:54:18 -0700 (PDT) Received: from kaos.jnpr.net (localhost [127.0.0.1]) by kaos.jnpr.net (Postfix) with ESMTP id 5AE593F79C; Sat, 31 Aug 2019 07:54:18 -0700 (PDT) To: Ed Maste CC: Konstantin Belousov , FreeBSD Hackers , Li-Wen Hsu , , Subject: Re: FCP 20190401-ci_policy: CI policy In-Reply-To: References: <20190829114057.GZ71821@kib.kiev.ua> <20190830065534.GC71821@kib.kiev.ua> Comments: In-reply-to: Ed Maste message dated "Fri, 30 Aug 2019 21:54:41 -0400." From: "Simon J. Gerraty" X-Mailer: MH-E 8.6+git; nmh 1.7.1; GNU Emacs 26.1 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <70179.1567263258.1@kaos.jnpr.net> Date: Sat, 31 Aug 2019 07:54:18 -0700 Message-ID: <73111.1567263258@kaos.jnpr.net> X-EXCLAIMER-MD-CONFIG: e3cb0ff2-54e7-4646-8a04-0dae4ac7b136 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:66.129.239.13; IPV:NLI; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10019020)(4636009)(376002)(136003)(396003)(346002)(39860400002)(2980300002)(189003)(199004)(7696005)(81156014)(8676002)(336012)(117636001)(23726003)(50226002)(53936002)(50466002)(97756001)(76506006)(70586007)(55016002)(9686003)(229853002)(446003)(11346002)(126002)(6246003)(7126003)(8936002)(476003)(6916009)(2906002)(305945005)(4326008)(6266002)(478600001)(486006)(186003)(356004)(26005)(86362001)(81166006)(70206006)(76176011)(4744005)(54906003)(47776003)(5660300002)(97876018)(46406003)(16586007)(316002)(53416004)(107886003); DIR:OUT; SFP:1102; SCL:1; SRVR:BL0PR05MB4692; H:P-EXFEND-EQX-02.jnpr.net; FPR:; SPF:SoftFail; LANG:en; PTR:InfoDomainNonexistent; MX:1; A:1; X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f3fe614f-ff4a-4e92-3156-08d72e231aa8 X-Microsoft-Antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600166)(711020)(4605104)(4710121)(4711137)(1401327)(4618075)(2017052603328); SRVR:BL0PR05MB4692; X-MS-TrafficTypeDiagnostic: BL0PR05MB4692: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-Forefront-PRVS: 014617085B X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Message-Info: fIj1RjtuLpmbCmXCFtzuC/d+H8Zlr2oFxNLQ6aNHHwL8KWdDbfh2ew13713fQ2wox71TuqnBbSZPFAmGVbKM7RmkDakbge6ShXTSNwXWgRwF+OQ6XprbT8gwXrLAXRJz3S4tAj93Nls+3y4RHn20Vtgxzt0uK6Nq9cWx7wGdpj27i8HsAPRAR0aIfchYyazhlb4+w70y4ILoUgEfVP3+L3IJdUFpE35uxrWxmQgv6Fk8N8/aB7AMMtmjG9/IBhCSzz/Pql/LVQ6+UHYp7GXqLw4QWJklq2RpIQ4uo6beMiCLc0+niCs6TTdoJrr+UAFUZe9RFuMWkVpw/1fjNIUvxkisFwnM3j648oA9/UaVcU+LhKmpNRkwkc22AK2QnFpvUtP1NgMCQP1tff7GOslRcoe9rjoB3zPSI1mkQz2fqbE= X-OriginatorOrg: juniper.net X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 Aug 2019 14:54:19.9765 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f3fe614f-ff4a-4e92-3156-08d72e231aa8 X-MS-Exchange-CrossTenant-Id: bea78b3c-4cdb-4130-854a-1d193232e5f4 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=bea78b3c-4cdb-4130-854a-1d193232e5f4; Ip=[66.129.239.13]; Helo=[P-EXFEND-EQX-02.jnpr.net] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR05MB4692 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.70,1.0.8 definitions=2019-08-31_05:2019-08-29,2019-08-31 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_spam_notspam policy=outbound_spam score=0 mlxlogscore=650 spamscore=0 mlxscore=0 phishscore=0 lowpriorityscore=0 priorityscore=1501 suspectscore=0 bulkscore=0 malwarescore=0 impostorscore=0 clxscore=1011 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-1906280000 definitions=main-1908310173 X-Rspamd-Queue-Id: 46LKBB5rXTz4YlV X-Spamd-Bar: ------ X-Spamd-Result: default: False [-6.95 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[]; NEURAL_HAM_SHORT(-0.95)[-0.953,0] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Aug 2019 14:54:27 -0000 Ed Maste wrote: > > > - The test is difficult to maintain > > This is too. My main complain is that to debug a test case, I must strip > > all atf* to be able to examine it under a debugger. IMO this is a bug in the state of ATF in FreeBSD. The ability to run atf tests on the host in-line with the build is one of the reasons I picked ATF for Junos - but has been discarded in FreeBSD. With atf tests runnable on build host in situ they are not hard to debug there too. I recently added a test suite for my package system (for Junos) and had to use the bmake framework, because FreeBSD made it impossible to use ATF/Kyua > > Yes, this is my biggest complaint about our current test setup. But > this impacts the tests' friendliness to developers, but not their > efficacy in reporting regressions.