Date: Sat, 25 Oct 2014 09:49:46 -0700 From: Craig Rodrigues <rodrigc@FreeBSD.org> To: Alfred Perlstein <alfred@freebsd.org> Cc: "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org>, FreeBSD stable <freebsd-stable@freebsd.org>, "freebsd-virtualization@freebsd.org" <freebsd-virtualization@freebsd.org> Subject: Re: Automatically running /usr/tests on stable/10 branch under Jenkins Message-ID: <CAG=rPVexyG_ZW8%2BXPWUu6Dt7--ZjwKG7Wtga3esR0JqpJPJtPw@mail.gmail.com> In-Reply-To: <544B46BA.4000008@freebsd.org> References: <CAG=rPVe-hCYiH5YuC%2BrzrucJbHJvEFmik0RAA%2Brq%2BXQ5K_A0Ww@mail.gmail.com> <20141024053636.GH11222@dft-labs.eu> <CAHM0Q_MOLoYGVhVOwAHfxKmMdX8bBK0Y=OoiR0TR=t3kQyYtVQ@mail.gmail.com> <CAG=rPVcRkCtwjNdzO2p6PuMVTLTFh7qKN=pxPVDrE0DM=R_a9w@mail.gmail.com> <81030948-E60F-4AAD-AAF1-16349607917D@gmail.com> <544B46BA.4000008@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Oct 24, 2014 at 11:44 PM, Alfred Perlstein <alfred@freebsd.org> wrote: > > On 10/24/14 9:45 PM, Garrett Cooper wrote: > >> I think getting tools/regression/zfs working first would be a better idea >> (which means that ZFS developers will need to go debug/fix the issue noted >> in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191574 ). I'll go >> ahead and commit my fixes to head from my github fork so it runs. >> > If tools/regression/zfs has Kyuafiles that makes it easier to run under the kyua tool, that would greatly facilitate running under Jenkins automation, and would be useful. I notice some of the fixes you are applying to the regression/zfs tests is to make certain tests not run on FreeBSD because they cause known kernel panics such as this: https://lists.freebsd.org/pipermail/svn-src-all/2014-October/093671.html I'm not entirely convinced that this is a good way to "fix" a test. If a test causes a panic, then that's what it does, it it should not be swept under the rug, as Alfred has pointed out. Printing out a warning with a pointer to the PR just before running this type of test is OK, though. > >> Alan also suggested against integrating the test suite as-is, because as >> he said, "Remember, don't run these tests on a production system. They >> WILL cause panics and deadlocks, and they may cause data loss too." >> >> Cheers, >> -Garrett >> > > Wait, we want to sweep those bugs under the rug? What exactly is wrong > with making a test harness that can very easily reproduce a known problem? > The chances are that anyone will dive into it once the bug is easily > reproducible. > I agree with Alfred on this. Even though Alan's test suite may kernel panic or cause problems, there is still value in running it and making the results visible on jenkins.freebsd.org. Running these tests inside a VM which is generated during the build will allow these types of test to run, but still keep the test machine usable, even if the VM gets corrupted while running the tests. If we have test suites for ZFS, but no one runs them, then no one will bother to investigate and fix the bugs. Running the test suites under automation that is visible on jenkins.freebsd.org is going to force developers to see problems much sooner than they do now. Just having the tests in the tree and hoping that people run them and care to look into the problems is not enough. -- Craig
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAG=rPVexyG_ZW8%2BXPWUu6Dt7--ZjwKG7Wtga3esR0JqpJPJtPw>