From owner-freebsd-testing@freebsd.org Wed May 1 16:41:00 2019 Return-Path: Delivered-To: freebsd-testing@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 57FF3159B350 for ; Wed, 1 May 2019 16:41:00 +0000 (UTC) (envelope-from olivier@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F07AD848A0 for ; Wed, 1 May 2019 16:40:59 +0000 (UTC) (envelope-from olivier@freebsd.org) Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) (Authenticated sender: olivier/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id A08FA3423 for ; Wed, 1 May 2019 16:40:59 +0000 (UTC) (envelope-from olivier@freebsd.org) Received: by mail-pf1-f174.google.com with SMTP id e67so8822118pfe.10 for ; Wed, 01 May 2019 09:40:59 -0700 (PDT) X-Gm-Message-State: APjAAAVE5D2D6jIHPACwAbIwuaNoDzluXF/loehsdr/ExZhbwqw8asMv PcN5yBEpjZeRr1TFv11RF09ih9y4VOa2I8rCFpc= X-Google-Smtp-Source: APXvYqxjG9CHj4nXFDkY82JGqXUpKSlIOZD5iJIYUoZyx7rCaQj3wevbpUhvk80g6VMMJmLWe+H86Gozj0F4lVjhvy4= X-Received: by 2002:a63:e048:: with SMTP id n8mr74873893pgj.41.1556728858684; Wed, 01 May 2019 09:40:58 -0700 (PDT) MIME-Version: 1.0 From: =?UTF-8?Q?Olivier_Cochard=2DLabb=C3=A9?= Date: Wed, 1 May 2019 18:40:46 +0200 X-Gmail-Original-Message-ID: Message-ID: Subject: Non-stable tests To: freebsd-testing@freebsd.org X-Rspamd-Queue-Id: F07AD848A0 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.99 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.99)[-0.991,0]; ASN(0.00)[asn:11403, ipnet:96.47.64.0/20, country:US] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 May 2019 16:41:00 -0000 Hi, On our CI (Jenkins running the full kyua test suite on a bhyve VM), some tests are sometimes failing without a valid cause. So I had to run a second time the kyua test suite to confirm it's a "real" regression, and the second time I didn't meet any problem. The last 2 tests that meet this problem are: - sbin.devd.client_test.seqpacket, failed with "Test case body timed out" - lib.libc.sys.getitimer_test.setitimer_old, failed with "setitimer(2) did not return remaining time, failed with" I've tried to rerun these tests a thousand times, but I didn't reach to reproduce the problem: they've succeeded each time after. So I wonder if there is a problem about "non stable" tests ? I've done a rapid search on ci.freebsd.org and I've found an example. In this series: https://ci.freebsd.org/job/FreeBSD-head-amd64-test/10391/ (r344839) https://ci.freebsd.org/job/FreeBSD-head-amd64-test/10392/ (r344840) https://ci.freebsd.org/job/FreeBSD-head-amd64-test/10393/ (r344843) => with job 10392, the test "lib.libarchive.functional_test.test_fuzz_zip" failed, but there is no change in the code regarding libarchive during these 3 commits revision period ( https://svnweb.freebsd.org/base/head/?view=log&pathrev=344843). So why it failed once here ? This is the same symptom I meet on my platform too. Regards, Olivier