Date: Sat, 28 Oct 2017 13:09:42 -0400 From: Johannes M Dieterich <jmd@freebsd.org> To: Tijl Coosemans <tijl@FreeBSD.org> Cc: Joseph Mingrone <jrm@FreeBSD.org>, ports-committers@freebsd.org, svn-ports-all@freebsd.org, svn-ports-head@freebsd.org, portmgr@FreeBSD.org Subject: Re: svn commit: r452962 - head/math/libRmath Message-ID: <20171028130942.68845cf0@manray.ogolem.org> In-Reply-To: <20171028182237.1f83708c@kalimero.tijl.coosemans.org> References: <201710270052.v9R0qf7A058644@repo.freebsd.org> <20171027125130.39e98c9c@kalimero.tijl.coosemans.org> <86d158vcve.fsf@phe.ftfl.ca> <20171028124843.56f8e8d3@kalimero.tijl.coosemans.org> <86y3nvtjlt.fsf@phe.ftfl.ca> <20171028182237.1f83708c@kalimero.tijl.coosemans.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Dear Tijl, dear Joe, On Sat, 28 Oct 2017 18:22:48 +0200 Tijl Coosemans <tijl@FreeBSD.org> wrote: > On Sat, 28 Oct 2017 11:47:26 -0300 Joseph Mingrone <jrm@FreeBSD.org> > wrote: > > Tijl Coosemans <tijl@FreeBSD.org> writes: > >> On Fri, 27 Oct 2017 12:17:41 -0300 Joseph Mingrone > >> <jrm@FreeBSD.org> wrote: > >>> Tijl Coosemans <tijl@FreeBSD.org> writes: > >>>> I don't see this approach to support flang work. Will this have > >>>> to be added to all Fortran ports? Will you modify all Fortran > >>>> ports whenever flang supports a new architecture? Will users > >>>> run into problems when they select gfortran in some ports and > >>>> flang in others? Will you allow port maintainers to set > >>>> different defaults in their ports? It all looks like high > >>>> maintenance and highly error prone. Too many modifiable > >>>> variables in too many places. > >>>> > >>>> USES arguments are the wrong mechanism for this in my opinion. > >>>> DEFAULT_VERSIONS seems much better to me. Users can then add > >>>> fortran=gfortran or fortran=flang to DEFAULT_VERSIONS. > >>>> fortran.mk would look at FORTRAN_DEFAULT to determine the > >>>> fortran compiler instead of fortran_ARGS. All ports would > >>>> simply have USES=fortran and no options. That's two variables in > >>>> two locations: the user's DEFAULT_VERSIONS in make.conf and the > >>>> ports tree FORTRAN_DEFAULT in bsd.default-versions.mk (which > >>>> could eventually be set to flang on amd64 if that turns out to > >>>> be a better default). Advanced users that want to build some > >>>> ports with flang and some with gfortran and think they know what > >>>> they're doing can figure out the if-elseif-else logic needed in > >>>> their make.conf. Port maintainers/committers should not have to > >>>> deal with the support requests resulting from such mixed > >>>> configurations which is very likely if you add per port options > >>>> like you do in this commit. > >>> > >>> The DEFAULT_VERSIONS solution is cleaner, but switching all > >>> Fortran ports to build with flang would cause breakage and, as > >>> you know, flang is currently only available for amd64. The > >>> current, more complicated, fine-tuned approach is less drastic. > >>> If port maintainers do not make any changes, nothing with their > >>> port changes. They have time to test with flang and make a > >>> choice to opt in. For example, math/R defaults to flang (on > >>> amd64) because we have problems with gfortran [1][2] and a > >>> work-in-progress port for Rstudio [3] only works when math/R uses > >>> flang. Hopefully flang will mature and become supported on more > >>> architectures, port maintainers will put in some work to support > >>> flang, and it will be an obvious choice to move to the > >>> DEFAULT_VERSIONS approach. This is not a precedent for choosing > >>> the compiler as a port option [4]. > >>> > >>> [1] https://wiki.freebsd.org/libgcc%20problem > >>> [2] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=220359 > >>> [3] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221127 > >>> [4] E.G. cad/ghdl games/eduke32 lang/erlang-runtime16 > >>> lang/gambit-c math/opensolaris-libm multimedia/x264 > >>> net-p2p/cpuminer net/asterisk11 www/mod_spdy > >> > >> The download problem you have in math/R looks like the problem > >> reported at > >> https://lists.freebsd.org/pipermail/freebsd-current/2017-August/066855.html > >> https://lists.freebsd.org/pipermail/freebsd-current/2017-October/067254.html > >> and is unrelated to Fortran. > > > > Thanks for this. The problem is with gfortran (gcc). It is good to > > know it is simply with gcc/gfortran and not a result of mixing > > libgcc libraries. The main point for users of math/R is that > > problems are only triggered when it is built with gfortran and not > > with flang. If we want R to work as it should, we need flang, but > > if we have to switch all Fortran ports to flang we will break > > (many) other ports. > > It's not a gfortran problem. It's a kernel or libthr problem. You > can work around it by avoiding the combination of recycled thread > stacks and exception handling done by ports libgcc_s. When you > switched from curl to wget you avoided threads and that fixed the > problem. Switching to flang avoids ports libgcc_s and that also > fixes the problem, but it's not necessary to go this way. You could > probably go back to curl if you build it with the CARES option > instead of THREADED_RESOLVER. > > >> You could then ask portmgr for an exp-run with the default set to > >> flang to see where the problems are. Point upstream flang > >> developers and anyone else who wants to push flang to the error > >> logs and prioritise fixing ports that cause the biggest fallout. > >> You'll get better results faster that way than when you expect > >> individual port maintainers to experiment with flang or fix flang > >> related issues. > > > > I did this with a private poudriere run and there was a lot of > > breakage with flang. It was a few months ago though, so I will try > > again. If a high proportion break with flang, then I still feel it > > might be best (in terms of breakage for users) to ease in with the > > current approach. I will report back soon(ish). > > If it's really that bad then flang is not ready and it's somewhat > irresponsible to use it for such an important package as R. Flang has > no proven track record. Even if it builds without errors in case of R > you can't be reasonably sure it produces correct code. With compilers > you have to be a bit more conservative and cautious than usual because > everything else depends on it. As the maintainer of devel/flang and as somebody that unfortunately is required to work with Fortran code and compilers on a daily basis, I want to put this a bit into perspective. Yes, flang is relatively newly released and indeed they are actively developing it. The people behind it are NVIDIA and (more importantly) PGI though, which obviously have a proven track record with Fortran compilers. Indeed, most of the front end is PGI's, as one can see from error messages etc. The back end is llvm, again something that definitely has a proven track record - especially in FreeBSD. flang slots in with the rest of our LLVM infrastructure. Most of the breakage I've observed in the codes I maintain for work were parsing errors. Now, to argue that's solely flang's fault would be to ignore the average quality of Fortran codes (again, from daily experience) which is abysmal - Intel's ifort is known to digest illegal codes, gfortran is (much) better but also more aware of "weird" code constructs, flang is learning. Developing Fortran for most "developers" (even commercial ones, I am afraid) boils down to "it compiles and seems to run for me and my configuration". Additional breakage has happened for me with configure/make systems just "assuming" there is only gfortran or ifort (or maybe pgf). I have not observed a single problem (and these are quantum chemical codes, so not known to be very forgiving) with results coming out wrong yet. So I suggest we wait to see what the actual problems were in the poudriere run. That being said, I absolutely do not advocate to think about making flang default yet or even mid-term. Reasons: no stable release yet, no architecture beyond amd64 supported, the port is still not ideal (e.g., I have a wrapper script supplying link flags), we have no performance numbers yet, and I haven't been able to upstream patches yet. However, lang/gcc is not a good option either long-term. -Wl,-rpath= is most certainly not something the ubiquitous homebrew Fortran build environments have by default (and it changes with every lang/gcc upgrade). I cannot mix and match OpenMP with clang/clang++ with OpenMP in gfortran (at least easily), ... Effectively, once a library has USES=fortran, everything downstream needs to be carefully handled in my experience. Especially with mixed language codes. Concerning R: I have no opinion as I have never used it. I do however appreciate that Joe worked on using flang to give it more exposure and that we now have fortran:flang to actually test with flang thanks to him! Johannes
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20171028130942.68845cf0>