Date: Fri, 16 Dec 2016 11:08:53 +0100 From: Torsten Zuehlsdorff <mailinglists@toco-domains.de> To: David Demelier <demelier.david@gmail.com>, Matthew Seaman <matthew@freebsd.org> Cc: freebsd-ports@freebsd.org Subject: Re: (In)Stability of the Quarterly Branch Message-ID: <7e071aa9-4fd8-9e6e-07f1-d0fa6632087a@toco-domains.de> In-Reply-To: <CAO%2BPfDd0BWuUnY62EeOVzv21pZ3estu=Vgz4chvbqWcaD6adaQ@mail.gmail.com> References: <3e7f94efc6428181a289742d7dd627df@acheronmedia.com> <CAO%2BPfDcYDy=w9Xaf02zWiYNO38Yex0ioX6z4a-5KL8k7e9qgQA@mail.gmail.com> <20161215170154.0ca2017914c0bb032516b413@gmail.com> <d806b1f6-9f9a-6546-d44f-5b04f3c422a9@FreeBSD.org> <CAO%2BPfDd0BWuUnY62EeOVzv21pZ3estu=Vgz4chvbqWcaD6adaQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 16.12.2016 08:24, David Demelier wrote: > 2016-12-15 17:25 GMT+01:00 Matthew Seaman <matthew@freebsd.org>: >> On 2016/12/15 16:01, Olivier Duchateau wrote: >>>> The problem is that there are no tests in FreeBSD ports. All source >>>> based systems I've tested: pkgsrc, FreeBSD ports, OpenBSD, Gentoo; >>>> FreeBSD is the one that have the most instability. Not to mention >>>> committers that commit without testing the port, just look at >>>> www/redmine to get your point of view on that issue. >> >>> Are your serious when you said, there're no tests on FreeBSD ports. I >>> can tell you Xfce ports are tested with FreeBSD i386 9.3 and amd64 >>> 11.0 machines (on real hardware, no virtualization), and on poudriere >>> with Gtk+ 3.20 (port version is not not in ports tree, it's defaut >>> toolkits for the next stable release 4.14). >>> >>> For the LXQt desktop is the same thing (tested with official ports >>> tree Qt5 and which one in plasma5 branch (on KDE repository). >>> >>> I'm also working on the Pantheon desktop (desktop environment of >>> Elementary OS, I use Vala 0.30.2 and Vala 0.34.4, in order to test >>> stability of applications. >>> >>> I use also OpenBSD macppc, it's piece of shit. WebKit browers are >>> broken, Xfce components crash often, stable branch is outdated, fix >>> are not propagated in stable branch. Personally I prefer the FreeBSD >>> scheme, because I'm sure it's quite stable. >> >> Most port committers will run compile tests any time they update a port: >> the better ones will test compilation on all supported FreeBSD versions >> and all hardware architectures they have access to (ie. generally i386 >> and amd64). > > I'm not talking about being sure that the port builds, but that the > software works. This is a next step that is too often forgotten. For > example I remember several years ago having a problem with > audio/mumble. The port was building fine, the window opened fine but > it was impossible to speak because there was a problem regarding the > CELT libraries IIRC. I really support this, but from a committer perspective its quite impossible. Often enough we just have no idea how the port needs to work - so we must trust the maintainer. But too often there is none. >> Additionally the package build cluster will rebuild any modified ports >> within a few days for all of the OS versions and architectures the >> project tries to provide ports for: that's yet another level of >> validating the coding of the port itself. >> >> However, I believe the OP's point is that *we do not routinely run the >> software's own built-in regression tests for the packages we succeed in >> building*. This is something that is slowly coming. For instance, you >> can run 'make test' for many python, ruby or perl packages and see those >> tests being run. TEST_DEPENDS is pretty much standardized as the way to >> install dependencies required for testing nowadays. >> > > Yes, I fully understand the requirements of such tests. I just would > like that maintainers test the port by building it and *by running* > them. This is time consuming for sure, but if the maintainers have no > time, then just keep an old but fully working version :-) There are two problems with this: 1) Many ports have no maintainer 2) Even more ports have no tests at all I'm a big fan for testing. I test every software i wrote and in every firm i worked i had the lowest bug count. But if you try to teach other people to write tests too you hit a hard wall... So we need to address multiple problems. I'm currently working on an project to improve the management of the ports-tree. Hopefully i can free some time for all so we can test the runtime much better. Greetings, Torsten
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7e071aa9-4fd8-9e6e-07f1-d0fa6632087a>