Date: Mon, 15 Oct 2018 04:06:40 -0400 From: Viktor Dukhovni <ietf-dane@dukhovni.org> To: freebsd-haskell@freebsd.org Subject: Updating ports to GHC 8.4.4? Message-ID: <20181015080640.GA983@straasha.imrryr.org>
next in thread | raw e-mail | index | archive | help
--8t9RHnE3ZwKMSgU+
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
The Haskell project has just released GHC 8.4.4, which is a bugfix
release for 8.4.3 fixing non-trivial bugs:
https://mail.haskell.org/pipermail/haskell-cafe/2018-October/130090.html
Is anyone planning to update the FreeBSD port? If so, please
consider the attached additional patches to the port "files"
directory.
I contributed the improved OSMem.c upstream, but this has only been
merged into GHC 8.6.x, and yet this also win for 8.4, and I've been
using it intensively in 8.4.3 (my code runs for 4.5 hours allocating
a cumulative 4.6TB of memory). In addition I'm adding a missing
LLVM target for amd64.
The attached files are patching patches, I hope that's OK. If
there's some other form in which to make these available, please
let me know.
--
Viktor.
--8t9RHnE3ZwKMSgU+
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment; filename="0001-Backport-memory-allocation-improvements-from-8.6.patch"
>From 1490ba37a2d5862a3b33c168394e45a7f7a0687a Mon Sep 17 00:00:00 2001
From: Viktor Dukhovni <ietf-dane@dukhovni.org>
Date: Mon, 15 Oct 2018 03:54:12 -0400
Subject: [PATCH 1/3] Backport memory allocation improvements from 8.6
1. Enable two-step allocator on FreeBSD
Simplify #ifdef nesting and use MAP_GUARD on FreeBSD and similar
systems. This allows the two-step allocator to be used on FreeBSD,
fixing #15348.
2. rts: Query system rlimit for maximum address-space size
When we attempt to reserve the heap, we query the system's rlimit to
establish the starting point for our search over sizes.
3. rts/posix: Use less aggressive backoff schedule for heap reservation sizing
When we allocate the heap on POSIX platforms we generally just ask for a
1TB chunk of address space and call it a day. However, if the user has
set a ulimit then this request will fail. In this case we would
previously try successively smaller allocation requests, reducing the
request size by a factor of two each time.
However, this means that GHC will significantly allocate a significantly
smaller heap than the available physical memory size in some
circumstances. Imagine, for instance, a machine with 512 GB of physical
memory but a ulimit of 511 GB: we would be limited to a 256 GB heap.
We now use a less aggressive back-off policy, reducing by one-eighth the
last allocation size each try.
---
patch-rts__posix__OSMem.c | 251 ++++++++++++++++++++++++++++++++++++++
1 file changed, 251 insertions(+)
create mode 100644 patch-rts__posix__OSMem.c
diff --git a/patch-rts__posix__OSMem.c b/patch-rts__posix__OSMem.c
new file mode 100644
index 0000000..e6aacfb
--- /dev/null
+++ b/patch-rts__posix__OSMem.c
@@ -0,0 +1,251 @@
+--- rts/posix/OSMem.c 2017-11-28 11:39:14.000000000 -0500
++++ rts/posix/OSMem.c
+@@ -36,6 +36,10 @@
+ #if defined(HAVE_NUMAIF_H)
+ #include <numaif.h>
+ #endif
++#if defined(HAVE_SYS_RESOURCE_H) && defined(HAVE_SYS_TIME_H)
++#include <sys/time.h>
++#include <sys/resource.h>
++#endif
+
+ #include <errno.h>
+
+@@ -45,6 +49,29 @@
+ #include <sys/sysctl.h>
+ #endif
+
++#ifndef MAP_FAILED
++# define MAP_FAILED ((void *)-1)
++#endif
++
++#if defined(hpux_HOST_OS)
++# ifndef MAP_ANON
++# define MAP_ANON MAP_ANONYMOUS
++# endif
++#endif
++
++#ifndef darwin_HOST_OS
++# undef RESERVE_FLAGS
++# if defined(MAP_GUARD)
++# define RESERVE_FLAGS MAP_GUARD /* FreeBSD */
++# elif defined(MAP_NORESERVE)
++# define RESERVE_FLAGS MAP_NORESERVE | MAP_ANON | MAP_PRIVATE;
++# else
++# if defined(USE_LARGE_ADDRESS_SPACE)
++# error USE_LARGE_ADDRESS_SPACE needs MAP_NORESERVE or MAP_GUARD
++# endif
++# endif
++#endif
++
+ static void *next_request = 0;
+
+ void osMemInit(void)
+@@ -98,8 +125,10 @@ void osMemInit(void)
+ The naming is chosen from the Win32 API (VirtualAlloc) which does the
+ same thing and has done so forever, while support for this in Unix systems
+ has only been added recently and is hidden in the posix portability mess.
+- It is confusing because to get the reserve behavior we need MAP_NORESERVE
+- (which tells the kernel not to allocate backing space), but heh...
++ The Linux manpage suggests that mmap must be passed MAP_NORESERVE in order
++ to get reservation-only behavior. It is confusing because to get the reserve
++ behavior we need MAP_NORESERVE (which tells the kernel not to allocate backing
++ space), but heh...
+ */
+ enum
+ {
+@@ -108,6 +137,44 @@ enum
+ MEM_RESERVE_AND_COMMIT = MEM_RESERVE | MEM_COMMIT
+ };
+
++#if defined(linux_HOST_OS)
++static void *
++linux_retry_mmap(int operation, W_ size, void *ret, void *addr, int prot, int flags)
++{
++ if (addr != 0 && (operation & MEM_RESERVE)) {
++ // Try again with no hint address.
++ // It's not clear that this can ever actually help,
++ // but since our alternative is to abort, we may as well try.
++ ret = mmap(0, size, prot, flags, -1, 0);
++ }
++ if (ret == MAP_FAILED && errno == EPERM) {
++ // Linux is not willing to give us any mapping,
++ // so treat this as an out-of-memory condition
++ // (really out of virtual address space).
++ errno = ENOMEM;
++ }
++ return ret;
++}
++#endif /* defined(linux_HOST_OS) */
++
++static void
++post_mmap_madvise(int operation, W_ size, void *ret)
++{
++#if defined(MADV_WILLNEED)
++ if (operation & MEM_COMMIT) {
++ madvise(ret, size, MADV_WILLNEED);
++# if defined(MADV_DODUMP)
++ madvise(ret, size, MADV_DODUMP);
++# endif
++ } else {
++ madvise(ret, size, MADV_DONTNEED);
++# if defined(MADV_DONTDUMP)
++ madvise(ret, size, MADV_DONTDUMP);
++# endif
++ }
++#endif
++}
++
+ /* Returns NULL on failure; errno set */
+ static void *
+ my_mmap (void *addr, W_ size, int operation)
+@@ -149,69 +216,44 @@ my_mmap (void *addr, W_ size, int operation)
+ VM_PROT_READ|VM_PROT_WRITE);
+ }
+
+-#else
++#else /* defined(darwin_HOST_OS) */
+
+ int prot, flags;
+- if (operation & MEM_COMMIT)
++ if (operation & MEM_COMMIT) {
+ prot = PROT_READ | PROT_WRITE;
+- else
++ } else {
+ prot = PROT_NONE;
+- if (operation == MEM_RESERVE)
+-# if defined(MAP_NORESERVE)
+- flags = MAP_NORESERVE;
++ }
++
++ if (operation == MEM_RESERVE) {
++# if defined(RESERVE_FLAGS)
++ flags = RESERVE_FLAGS;
+ # else
+-# if defined(USE_LARGE_ADDRESS_SPACE)
+-# error USE_LARGE_ADDRESS_SPACE needs MAP_NORESERVE
+-# endif
+ errorBelch("my_mmap(,,MEM_RESERVE) not supported on this platform");
+ # endif
+- else if (operation == MEM_COMMIT)
+- flags = MAP_FIXED;
+- else
+- flags = 0;
++ } else if (operation == MEM_COMMIT) {
++ flags = MAP_FIXED | MAP_ANON | MAP_PRIVATE;
++ } else {
++ flags = MAP_ANON | MAP_PRIVATE;
++ }
+
+-#if defined(hpux_HOST_OS)
+- ret = mmap(addr, size, prot, flags | MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
+-#elif defined(linux_HOST_OS)
+- ret = mmap(addr, size, prot, flags | MAP_ANON | MAP_PRIVATE, -1, 0);
+- if (ret == (void *)-1 && errno == EPERM) {
++ ret = mmap(addr, size, prot, flags, -1, 0);
++# if defined(linux_HOST_OS)
++ if (ret == MAP_FAILED && errno == EPERM) {
+ // Linux may return EPERM if it tried to give us
+ // a chunk of address space below mmap_min_addr,
+ // See Trac #7500.
+- if (addr != 0 && (operation & MEM_RESERVE)) {
+- // Try again with no hint address.
+- // It's not clear that this can ever actually help,
+- // but since our alternative is to abort, we may as well try.
+- ret = mmap(0, size, prot, flags | MAP_ANON | MAP_PRIVATE, -1, 0);
+- }
+- if (ret == (void *)-1 && errno == EPERM) {
+- // Linux is not willing to give us any mapping,
+- // so treat this as an out-of-memory condition
+- // (really out of virtual address space).
+- errno = ENOMEM;
+- }
++ ret = linux_retry_mmap(operation, size, ret, addr, prot, flags);
+ }
+-
+- if (operation & MEM_COMMIT) {
+- madvise(ret, size, MADV_WILLNEED);
+-#if defined(MADV_DODUMP)
+- madvise(ret, size, MADV_DODUMP);
+-#endif
+- } else {
+- madvise(ret, size, MADV_DONTNEED);
+-#if defined(MADV_DONTDUMP)
+- madvise(ret, size, MADV_DONTDUMP);
+-#endif
+- }
+-
+-#else
+- ret = mmap(addr, size, prot, flags | MAP_ANON | MAP_PRIVATE, -1, 0);
+-#endif
+-#endif
+-
+- if (ret == (void *)-1) {
++# endif
++ if (ret == MAP_FAILED) {
+ return NULL;
+ }
++#endif /* defined(darwin_HOST_OS) */
++
++ // Map in committed pages rather than take a fault for each chunk.
++ // Also arrange to include them in core-dump files.
++ post_mmap_madvise(operation, size, ret);
+
+ return ret;
+ }
+@@ -435,6 +477,8 @@ osTryReserveHeapMemory (W_ len, void *hint)
+ void *base, *top;
+ void *start, *end;
+
++ ASSERT((len & ~MBLOCK_MASK) == len);
++
+ /* We try to allocate len + MBLOCK_SIZE,
+ because we need memory which is MBLOCK_SIZE aligned,
+ and then we discard what we don't need */
+@@ -500,8 +544,19 @@ void *osReserveHeapMemory(void *startAddressPtr, W_ *len)
+ (void*)startAddress, (void*)minimumAddress);
+ }
+
++#if defined(HAVE_SYS_RESOURCE_H) && defined(HAVE_SYS_TIME_H)
++ struct rlimit limit;
++ if (!getrlimit(RLIMIT_AS, &limit)
++ && limit.rlim_cur > 0
++ && *len > limit.rlim_cur) {
++ *len = limit.rlim_cur;
++ }
++#endif
++
+ attempt = 0;
+ while (1) {
++ *len &= ~MBLOCK_MASK;
++
+ if (*len < MBLOCK_SIZE) {
+ // Give up if the system won't even give us 16 blocks worth of heap
+ barf("osReserveHeapMemory: Failed to allocate heap storage");
+@@ -512,9 +567,14 @@ void *osReserveHeapMemory(void *startAddressPtr, W_ *len)
+ if (at == NULL) {
+ // This means that mmap failed which we take to mean that we asked
+ // for too much memory. This can happen due to POSIX resource
+- // limits. In this case we reduce our allocation request by a factor
+- // of two and try again.
+- *len /= 2;
++ // limits. In this case we reduce our allocation request by a
++ // fraction of the current size and try again.
++ //
++ // Note that the previously would instead decrease the request size
++ // by a factor of two; however, this meant that significant amounts
++ // of memory will be wasted (e.g. imagine a machine with 512GB of
++ // physical memory but a 511GB ulimit). See #14492.
++ *len -= *len / 8;
+ } else if ((W_)at >= minimumAddress) {
+ // Success! We were given a block of memory starting above the 8 GB
+ // mark, which is what we were looking for.
+@@ -536,7 +596,7 @@ void osCommitMemory(void *at, W_ size)
+ {
+ void *r = my_mmap(at, size, MEM_COMMIT);
+ if (r == NULL) {
+- barf("Unable to commit %d bytes of memory", size);
++ barf("Unable to commit %" FMT_Word " bytes of memory", size);
+ }
+ }
+
--
2.19.1
--8t9RHnE3ZwKMSgU+
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment; filename="0002-use_large_address_space-no-no-longer-needed.patch"
>From 89fabf4d28d312ac20dbce910450e02511dba26c Mon Sep 17 00:00:00 2001
From: Viktor Dukhovni <ietf-dane@dukhovni.org>
Date: Mon, 15 Oct 2018 03:37:54 -0400
Subject: [PATCH 2/3] use_large_address_space=no no longer needed
---
patch-configure.ac | 12 ------------
1 file changed, 12 deletions(-)
diff --git a/patch-configure.ac b/patch-configure.ac
index 4d19057..92938eb 100644
--- a/patch-configure.ac
+++ b/patch-configure.ac
@@ -10,15 +10,3 @@
# If 'host' and 'target' differ, then this means we are building a cross-compiler.
if test "$TargetPlatform" != "$HostPlatform" ; then
CrossCompiling=YES
-@@ -1163,6 +1158,11 @@ if test "$ac_cv_sizeof_void_p" -eq 8 ; t
- # The flag MAP_NORESERVE is supported for source compatibility reasons,
- # but is completely ignored by OS mmap
- use_large_address_space=no
-+ elif test "$ghc_host_os" = "freebsd" ; then
-+ # FreeBSD does not support mmap with MAP_NORESERVE, removed in r273250.
-+ # The flag MAP_NORESERVE is supported for source compatibility reasons,
-+ # but is completely ignored by OS mmap
-+ use_large_address_space=no
- else
- AC_CHECK_DECLS([MAP_NORESERVE, MADV_FREE, MADV_DONTNEED],[],[],
- [
--
2.19.1
--8t9RHnE3ZwKMSgU+
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment; filename="0003-FreeBSD-amd64-LLVM-target.patch"
>From ac6d71b685eb07067b24cd1573a685f7c0a6fbe9 Mon Sep 17 00:00:00 2001
From: Viktor Dukhovni <ietf-dane@dukhovni.org>
Date: Mon, 15 Oct 2018 03:54:40 -0400
Subject: [PATCH 3/3] FreeBSD amd64 LLVM target
---
patch-llvm-targets | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/patch-llvm-targets b/patch-llvm-targets
index 8ea0574..f6220ae 100644
--- a/patch-llvm-targets
+++ b/patch-llvm-targets
@@ -1,6 +1,14 @@
--- llvm-targets.orig 2018-03-17 14:04:41 UTC
+++ llvm-targets
-@@ -20,4 +20,7 @@
+@@ -15,6 +15,7 @@
+ ,("x86_64-unknown-linux", ("e-m:e-i64:64-f80:128-n8:16:32:64-S128", "x86-64", ""))
+ ,("armv7-unknown-linux-androideabi", ("e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64", "generic", ""))
+ ,("aarch64-unknown-linux-android", ("e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128", "generic", "+neon"))
++,("amd64-portbld-freebsd", ("e-m:e-i64:64-f80:128-n8:16:32:64-S128", "x86-64", ""))
+ ,("arm-unknown-nto-qnx-eabi", ("e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64", "arm7tdmi", "+strict-align"))
+ ,("i386-apple-darwin", ("e-m:o-p:32:32-f64:32:64-f80:128-n8:16:32-S128", "yonah", ""))
+ ,("x86_64-apple-darwin", ("e-m:o-i64:64-f80:128-n8:16:32:64-S128", "core2", ""))
+@@ -22,4 +23,7 @@
,("aarch64-apple-ios", ("e-m:o-i64:64-i128:128-n32:64-S128", "generic", "+neon"))
,("i386-apple-ios", ("e-m:o-p:32:32-f64:32:64-f80:128-n8:16:32-S128", "yonah", ""))
,("x86_64-apple-ios", ("e-m:o-i64:64-f80:128-n8:16:32:64-S128", "core2", ""))
--
2.19.1
--8t9RHnE3ZwKMSgU+--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20181015080640.GA983>