Date: Tue, 17 Jan 2017 15:01:25 -0800 From: "Ngie Cooper (yaneurabeya)" <yaneurabeya@gmail.com> To: "freebsd-testing@freebsd.org" <testing@freebsd.org> Subject: Looking at replacing ATF/Kyua (in a limited fashion) with Google Test/shunit2 Message-ID: <45D23581-C780-4C55-80CF-19A81813D672@gmail.com>
index | next in thread | raw e-mail
[-- Attachment #1 --] Hello all, I had an initial discussion with a handful of other test stakeholders and due to the number of caveats with ATF/Kyua and a variety of issues in contributing back to the ATF/Kyua projects (time on maintainer’s end, legal issues, technical issues, etc), I'm seriously considering replacing parts of ATF/Kyua with GoogleTest and shunit2. In particular, I want to do these things [better]: - Detecting kernels features and modules. It is ad hoc and done on a per-testcase basis or in ad hoc common routines, which bloats tests unnecessarily. - Detecting network interfaces. My employer desires that — we had something in use internally for detecting interfaces in tests, but it was really messy; it would be nice if there was a clean, generalized way to do this. - Adding test fixtures to ease testing on a per-suite basis. For example, setting up a service like bsnmpd for testing, running pjdfstest in different configurations, (a temporarily setup UFS or ZFS filesystem, etc), collecting core files after tests for analysis purposes, etc. Why these frameworks? GoogleTest: - It is easier to extend upon. - It is better documented than ATF/Kyua. - It has more brevity in terms of writing testcases. - It has features that I (and others who write tests) really want. - It is very similar to atf-c-api in core ways, in terms of macros. - It has been accepted by multiple open source projects, like clang/llvm, so including them in the FreeBSD Test Suite is more trivial after the porting and integrating work. - It is lightweight and a good candidate for inclusion in the base system. - Google and other groups are actively maintaining the project. shunit2: - It has more brevity in terms of writing testcases. - It has features that I (and others who write tests) really want. - It has syntactic sugar/API construction patterned after JUnit/XUnit in other ways, so it’s familiar to testers who have written JUnit/XUnit tests. Some concerns that have been brought up: Q: Why those frameworks? Have you considered other options? A: Yes. A number of users have provided a matrix on Wikipedia and I’ve looked at the list to try and determine : a. C: https://en.wikipedia.org/wiki/List_of_unit_testing_frameworks#C b. C++: https://en.wikipedia.org/wiki/List_of_unit_testing_frameworks#C.2B.2B c. sh: https://en.wikipedia.org/wiki/List_of_unit_testing_frameworks#Shell Q: Google sometimes tinkers with projects, then abandons them later. Is GoogleTest one of those projects? A: As far as I can tell, no. This is widely used internally and developed on by their engineers. It is used external to Google by larger projects like LLVM, so even if Google disappeared tomorrow I believe there’s enough inertia and interest to keep things going here. Q: What about C test integration? A: I’m trying to see how far I can get with GoogleTest integration on the C API side, and if replacing atf-c-api is possible/doable using C integration in libc++ (the syntax/API construction for atf-c-api and GoogleTest is similar in some regards). I’m really early on in the evaluation phase, so if anyone has any input to provide with the direction I’m going, I would really love the input. Writing tests for bsnmp will be my first proof of concept (I hope to use both GoogleTest and shunit2). Thank you, -Ngie [-- Attachment #2 --] -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJYfqJFAAoJEPWDqSZpMIYV7osP/j9ymv75+SXLwsIs+K2wXAuj TTR64vU0LtW30FLT3Y2iNOsXPLxoUlTeG1IQY/yTjGiVYTdtLsRmfYW1DbMmpMF1 OOoe2yUHZrx4NcbR8dU9CHSqhFyDI+WBmAxoZrvF+VylhukqlrZJtfwKGhH4quK9 pdHQM8b8pX/kh9I8NZcZHfYeKZf3qbO/e7Nid2dIvx0D7v2ws7Y+8Zjq/PP0beuf Sw3CM4YZkKtiOif6/gQ1JTBh7sSwcE4soWeCuuEOEmdGJqd1yeEb/MO5fGbMAD6c dcRkyDcmm03P63JKbW0qjlnFED2DPGuhmKjBue4vaHtjTCoFhWE8yPti3lBwHqBU Yby6KI0hlh0skECSRYm1njc8oNviyMpv5wpSIOtMqSVqn5lcSBnK4a8XK70K1oPC aiuV7eFkUQNU1jfnebm6q4pIX4NSWuhiSchvPrIxpommGftriNXwovf2BohkNSxT hfq4oLe/SDZ9+kN2HRdhoM1xUKC35m0jIrXZDg30MlCiR9+wI1yAIvfJukfUxHld kj4zrk5XeznSHgO2aaSozTNdwkgH/1Cplhj6RGtN1B7FO+xUTqkT8UKeJ/8RCd00 gtBnventU9as9Deb9mjS4kS13lh9eQrqrXQlebVn3ZgNeTOaBu27ONdNaJqA5KSj 59btJvmEZ7uMu+P4ifnk =1ThW -----END PGP SIGNATURE-----help
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?45D23581-C780-4C55-80CF-19A81813D672>