Date: Wed, 5 Jul 2000 18:34:04 +0930 From: Greg Lehey <grog@lemis.com> To: FreeBSD advocacy list <FreeBSD-advocacy@FreeBSD.org> Subject: Re: LSB-VSX test suite V1.0-1 Message-ID: <20000705183404.M97425@wantadilla.lemis.com>
next in thread | raw e-mail | index | archive | help
Is this something FreeBSD people should be looking at? Greg ----- Forwarded message from Christopher Yeoh <cyeoh@linuxcare.com.au> ----- > For those who are interested, the LSB-VSX test suite has just been > released. This tests for POSIX.1-1990 (which is part of the LSB > specification) compliance. > > Just a warning - if you run the test suite on a 2.4test kernel (at > least up to test3pre2) it can/will cause a lockup of the kernel and > perhaps some filesystem corruption as well. It works fine on 2.2 > kernels. > >> From the systems (2.2.x kernels) I've run it on there are about 30 > failures out of 6000 tests. There seems to be little difference > between distributions. > > Chris. > >> All >> The first snapshot release of the LSB-VSX test suite for the >> POSIX.1 aspects of the Linux Standard Base is now available for download from: >> >> ftp://lsb.sourceforge.net/pub/lsb/LSB-VSX1.0-1/ >> >> >> This version of the VSX-PCTS has been setup to autoconfigure >> on Linux systems. A front end script install_wrapper.sh is used >> to auto install, setup and run the test suite. In theory this should >> allow running of the test suite by those unfamiliar with POSIX.1 >> and its myriad of options and thus the associated test suite >> configurables. >> >> >> I include the release notes below. >> >> If you encounter a problem in the suite, please send in a bug report >> directly to me at ajosey@opengroup.org, cc lsb-test@linuxbase.org, please >> include the test output together with any suggested fixes. Please include >> the following subject line in your message >> >> Subject: BUG in LSB-VSX (testcase-name-here) >> >> If possible include the assertion reference number. >> >> Acknowledgements are due to : >> Chris Yeoh >> Nick Petreley >> Dale Scheetz >> George Kraft >> plus many others involved in this packaging of the POSIX test suite >> for Linux. >> >> Our next steps are now to expand our test coverage to other >> areas outside the core POSIX. >> >> ---------------------------------------------------------------- >> >> >> LSB-VSX Release 1.0-1 >> Release Notes >> >> >> This document provides Release notes for the Verification suite for the >> POSIX.1 coverage of the Linux Standard Base. >> >> 1. Release Overview >> >> LSB-VSX 1.0-1 is the first release of this test suite for the POSIX.1 >> coverage for the Linux Standard Base. LSB-VSX is built using the VSXgen >> (the generic VSX test framework), with the VSX-PCTS 4.4.4 test set >> merged into it. The test coverage corresponds to the core "classic" >> POSIX.1 (including ISO C). >> >> >> This test suite has not yet been formally approved by the Linux Standard >> Base team and no claims should be made of LSB conformance to this version >> of the test suite. The test assertions being tested are included with >> the distribution in the MAN/tset directory. >> >> 1.1 Changes since the last release >> >> The following changes have been made since the last >> release: >> >> This version of the VSX-PCTS has been setup to autoconfigure >> on Linux systems. A front end script install_wrapper.sh is used >> to auto install, setup and run the test suite. >> >> 1.2 Status >> >> A few bugs were found in the suite during the port which were verified >> with the official VSX-PCTS support team and fixed; for example the >> namespace test tool did not understand GNU header files correctly >> (#include_next and other features). Also a bug shown up by fclose() >> behavior in glibc was fixed. >> >> Pseudottys versus Real Ttys: >> >> The tests for the tty subsystem run best when using real tty ports, >> however since many test configuration may not have two tty ports and a loopback >> cable available we have enabled the tests to run using pseudottys. >> This does mean that users will encounter more failures when running using >> pseudottys than real tty ports, and this is because pseudottys are >> unable to emulate all the hardware characteristics of real ttys. The >> list of expected failures is given with the release in the FAILURES.VSX4 >> file. >> >> >> The remaining failures have been categorised as Linux non-compliance >> issues, specification issues and test suite issues. >> >> 1.2.1. Linux compliance Issues: >> >> Kernel level problems : Incorrect errno values being set (eg dup2, unlink) >> >> Header file inconsistencies between the limits in limits.h >> and the kernel returns from sysconf() (eg LINK_MAX). >> >> For the tests related to mount points, the system is setting errno to >> the wrong value when an attempt is made to remove a directory that is >> a mount point. Since the directory exists, ENOENT is not appropriate. >> The calls should be setting errno to EBUSY (if they fail). >> >> Kernel inconsistency? >> >> For directories in use by another process, it seems that rename() gives >> EBUSY when it attempts to remove this type of busy directory, but remove() >> and rmdir() can successfully remove them. (Otherwise rmdir 7 and remove 7 >> would show two failures, the same as in rename 13). This is a strangely >> inconsistent implementation. Every implementation we have encountered so >> far has treated all attempts to remove a busy directory in the same way, >> regardless of which system call is used. That is why VSX4 only has the >> one VSX_REMOVE_DIR_EBUSY parameter for all the calls. One opinion of >> this difference between the behavior is a bug in the implementation, >> however, it isn't a non-compliance. >> >> >> glibc related problems >> >> Mostly header file problems (visiblity of function prototypes, etc) - >> expected to be fixed in later releases >> >> locale related failures need to be investigated further , experiments >> on glibc2.2 appear to have resolved these, they could be a test locale >> setup problem >> >> Some of the above problems may be fixed in 2.4.0 and many are reportedly >> fixed in glibc 2.2 to be released soon. >> >> 1.2.2. Specification Issues >> >> tmpfile() ignores umask. There is some debate at POSIX whether this >> is a good idea or not, at the moment the specification requires tmpfile() >> to honour umask. >> >> time.h not explicitly allowed to be included in signal.h . This is >> expected to be resolved in favour of the current implementation. >> >> 1.2.3. Issues Identified with VSX-PCTS >> >> These are mainly setup related in areas that are allowed to change >> for each operating system . >> >> Two tests fail because of the way test suite is setup (for ease of use reasons, >> allowing us to switch between suites). >> (a) getlogin() does not match vsx0 user >> (b) Home directory of vsx0 user is not the same as $HOME >> These would pass once test suite setup correctly >> >> Possible test locale setup problem (VSX4L2) under investigation (see >> glibc problems listed before). At the moment we are unable to build >> the locale, with localedef on glibc2.1 systems that has passed the >> tests under glibc2.2. >> >> >> 2. Release Contents >> >> The release consists of the following files. >> >> LSB-VSX-1.0-1.tar.gz The source code to the suite in gzip'd tar format >> vsxgde.ps The Users Guide in postscript >> vsxgde.pdf The Users Guide in PDF format >> relnote.txt These release notes >> >> 3. TET >> >> LSB-VSX runs under TET3 (The Test Environment Toolkit test framework). >> A suitable version of TET3 is bundled with the release. The latest >> release can also be obtained at: >> >> ftp://ftp.rdg.opengroup.org/pub/TET3/ >> >> TET3 must be configured to build as the Lite version . Do not use >> the Distributed version. For more information on TET , see >> http://tetworks.opengroup.org . >> >> 4. Documentation >> >> Two files are included with the release to ease installation and >> test setup: INSTALL.SETUP and INSTALL.VSX4. Please follow the instructions >> within these to install, configure and execute the tests. >> >> For more in depth information consult the User's Guide which provides >> comprehensive information on the installation, configuration and use >> of this test suite. >> >> 5. Support >> >> Support related questions and problem reports should be directed to by >> email to ajosey@rdg.opengroup.org. >> >> 6. Size >> >> The size of LSB-VSX when the suite is built is approximately 40MB. The >> size of the compressed distribution is roughly 3.5 MB. When running the >> complete test suite, journal files will be produced that are typically >> less than 1MB in length, so its recommended that at least 20MB of disk >> space be allowed for a test campaign. >> >> 7. Time >> >> The test suite takes approximately six to eight hours to run depending on the >> speed of the platform under test. The signals tests are known to take >> several hours to run. >> >> >> Licensing information. >> >> See the Licence file included in the distribution for licensing >> information. At the moment this is the Artistic License. The Open >> Group intends to change the license to that recommended by the Linux >> Standard Base team. >> >> >> ----- >> Andrew Josey The Open Group >> Director, Server Platforms Apex Plaza,Forbury Road, >> Email: a.josey@opengroup.org Reading,Berks.RG1 1AX,England >> Tel: +44 118 9508311 ext 2250 Fax: +44 118 9500110 >> >> >> -- >> >> To UNSUBSCRIBE, email to lsb-test-request@lists.linuxbase.org >> with subject of "unsubscribe". Trouble? Email listmaster@lists.linuxbase.org ----- End forwarded message ----- -- Finger grog@lemis.com for PGP public key See complete headers for address and phone numbers To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-advocacy" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20000705183404.M97425>