From owner-svn-doc-all@freebsd.org Fri Jul 22 05:19:46 2016
Return-Path: The &os; Release Engineering Team completed the 10.3-RELEASE
cycle late April, led by &a.marius;. The release was one week
- behind the original schedule, to accommodate for a few last
- minute critical issues that were essential to include in the
+ behind the original schedule, to accommodate for a few
+ last-minute critical issues that were essential to include in the
final release. The &os; 11.0-RELEASE cycle started late May, one month
@@ -106,10 +106,10 @@
to accommodate for packaging the &os; base system with the
pkg(8) utility. However, as work on this progressed,
it became apparent that there were too many outstanding
- issues. As a result, packaged base will be a "beta" feature
+ issues. As a result, packaged base will be a "beta" feature
for 11.0-RELEASE, with the goal of promoting it to a
- first-class feature in 11.1-RELEASE, with additional
- provisions to ensure a seamless transition for earlier
+ first-class feature in 11.1-RELEASE. It is expected that
+ provisions will be made to ensure a seamless transition from older
supported releases. Despite the fact that packaged base is not going to be a
@@ -120,7 +120,7 @@
Allwinner SoCs are used in multiple hobbyist devboards and - single board computers. Recently, support for these SoCs + single-board computers. Recently, support for these SoCs received many updates.
Theses tasks were completed during the second quarter of 2016:
Now that the process-shared locks are implemented for our - POSIX threads implementation library, libthr, - the only major lacking feature for POSIX compliance is robust - mutexes. Robust mutexes allow the application to detect, and +
Now that process-shared locks are implemented for our + POSIX threads implementation, libthr, + the only major feature lacking for POSIX compliance is robust + mutexes. Robust mutexes allow applications to detect, and theoretically, recover from crashes which occur while modifying the shared state. The supported model is to protect shared state by a pthread_mutex, and the crash is @@ -284,7 +284,7 @@ recover from failures. The pthread_mutex_lock() function may return new error EOWNERDEAD, which indicates that the previous owner of the lock terminated while - still owning the lock. Despite returning the non-zero value, + still owning the lock. Despite returning this non-zero value, the lock is granted to the caller. In the simplest form, an application may detect the error and refuse to operate until the persistent shared data is recovered, such as by manual @@ -299,22 +299,22 @@
It is curious but not unexpected that this interface is not used widely. The only real-life application which utilizes it - is Samba. Using Samba with an updated FreeBSD base uncovered - minor bugs both in the FreeBSD robustness implementation, and + is Samba. Using Samba with an updated &os; base uncovered + minor bugs both in the &os; robust mutex implementation, and in Samba itself.
-It is believed that libthr in FreeBSD 11 is - POSIX-compliant for large features. Further work is planned - to look at the lock structures inlining to remove overhead - and improve performance of the library.
+It is believed that libthr in &os; 11 is + POSIX-compliant for major features. Further work is planned + to look at inlining the lock structures to remove overhead + and improve the performance of the library.
Most of the implementation of the robustness feature consisted of making small changes in the lock and unlock paths, both in libthr and in kern_umtx.c. This literally required reading all of the code dealing with - mutexes and conditional variables, which was something I + mutexes and condition variables, which was something I wanted to help future developers with. In the end, with the - help of Ed Maste, the man pages for umtx(2) and all + help of Ed Maste, man pages for umtx(2) and all thr*(2) syscalls were written and added to the base system's documentation set.
@@ -329,7 +329,7 @@Both boot1 and loader have been refactored - to talk through the EFI_SIMPLE_FILE_SYSTEM interface. - In loader, this is accomplished with a dummy + to utilize the EFI_SIMPLE_FILE_SYSTEM interface. + In the loader, this is accomplished with a dummy filesystem driver that is just a translation layer between the - loader filesystem interface and + loader filesystem interface and EFI_SIMPLE_FILE_SYSTEM. A reverse translation layer allows the existing filesystem drivers to function as EFI drivers.
-The EFI refactoring by itself exists in - this branch.
+The EFI refactoring by itself exists in a + branch + on github.
Additionally, GELI support has been added using the EFI refactoring. This allows booting from a GELI-encrypted @@ -376,7 +377,7 @@ framework, which allows injection of keys directly into a loaded kernel, without the need to pass them through arguments or environment variables. This patch only uses the - intake buffer for EFI GELI support as legacy BIOS GELI support + intake buffer for EFI GELI support, as legacy BIOS GELI support still uses environment variables.
EFI GELI support depends on the @@ -404,7 +405,7 @@
Install an EFI shell on the ESP.
+Install an EFI shell on the EFI System Partition (ESP).
Try switching over to an encrypted main partition once - everything else has worked.
+ everything else is working.The port has been updated to GDB 7.11.1.
+The devel/gdb port has been updated to GDB 7.11.1.
Support for system call catchpoints has been committed
upstream. Support for examining ELF auxiliary vector data via
@@ -484,12 +484,12 @@
VIMAGE is a virtualization framework on top of FreeBSD jails
+ VIMAGE is a virtualization framework on top of &os; jails
that was introduced to the kernel about eight years ago with
the vnet virtualized network stack. The vnet teardown has been changed to be from top to
bottom, trying to tear down layer by layer. This is
preferable to removing interfaces first and then cleaning
- everything up, as no more packets could flow. Along with this
- work, various parts with potential memory leaks were plugged.
+ everything up, as no more packets could flow once the
+ interfaces are gone. Along with this
+ work, various paths with potential memory leaks were plugged.
Lastly, vnet support was added to formerly
unvirtualized components, such as the pf and
ipfilter firewalls and some virtual interfaces. Half a year ago, I started a promotion campaign to improve
support for fetching ports via IPv6. Research performed in
- December, 2015 showed that 10,308 of 25,522 ports are not
- fetchable when using IPv6-only as these ports ignore the FreeBSD.org pkg mirror.
As a result of the campaign, the following servers now successfully support IPv6:
@@ -590,7 +591,7 @@This enables 711 more ports to be fetched via IPv6.
-I would like to thank Wolfgang Zenker who is very active in +
I would like to thank Wolfgang Zenker, who is very active in
supporting the adoption of IPv6. During the latest RIPE
meeting, he brought up the topic of non-support of IPv6
being a hindrance to business. I am hopeful that his talk
@@ -637,15 +638,15 @@
During BSDCan 2016, Microsoft announced - the global availability of FreeBSD 10.3 images in Azure. - There are many FreeBSD-based Azure virtual appliances in the + the global availability of &os; 10.3 images in Azure. + There are many &os;-based Azure virtual appliances in the Azure Marketplace, including Citrix Systems' NetScaler and Netgate's pfSense. Microsoft also made an in-depth technical presentation to introduce how the performance of the Hyper-V @@ -659,11 +660,11 @@ performance of Hyper-V network and storage device drivers. Work is ongoing to replace the internal data structure in the LRO kernel API from a singly-linked list to a double-linked - list, to speed up the LRO lookup by hash table, and to compare + list, to speed up the LRO lookup by hash table, and to evaluate the performance with tcp_lro_queue_mbuf().
The handling of SCSI inquiry in the Hyper-V storage driver is
- enhanced to make sure disk hotplug and smartctl(8)
+ enhanced to make sure that disk hotplug and smartctl(8)
work reliably. Refer to PR 210425
and Ceph main site
Ceph is a distributed object store and file system designed
- to provide excellent performance, reliability, and
+ Ceph is a distributed object store and filesystem designed
+ to provide excellent performance, reliability, and
scalability. It provides the following features: I started looking into Ceph as using HAST with CARP and
+ I started looking into Ceph because using HAST with CARP and
ggate did not meet my requirements. My primary goal
with Ceph is to run a storage cluster of ZFS storage nodes
- where the clients run bhyve on RBD disks stored in Ceph.
@@ -716,38 +717,38 @@
access to block device images that are striped and
replicated across the entire storage cluster.
-
-
The &os; build process can build most of the tools in - Ceph. However, the RBD-dependent items do not work since + Ceph. However, the RBD-dependent items do not work, since &os; does not yet provide RBD support.
Since the last quarterly report, the following progress was made:
11-CURRENT is used to compile and build test Ceph. The - Clang toolset needs to be at least version 3.7 as Clang 3.4 + Clang toolset needs to be at least version 3.7, as Clang 3.4 does not have all of the capabilities required to compile everything.
@@ -755,7 +756,7 @@Install bash and link it in /bin +
Install bash and link to it in /bin (requires root privileges):
sudo pkg install bash
@@ -787,33 +788,33 @@Tests Not Yet Included:
Current work aims to fix the open problems, get the latest - major version into the port, and create the documentation for + major version into the port, and create documentation for the update progress.
@@ -931,107 +932,117 @@proccontrol -m (trace|aslr) [-q] [-s (enable|disable)] [-p pid | command]
--m (specifies trace mode to control debugger +
with possible arguments
+ +-m (specifies the trace mode to control debugger attachments)
-q (queries the state of the specified mode for the - process with the PID specified by -p option)
+ process with the PID specified by the -p option)-e (toggles the feature on or off for the given process or itself)
-If the command is specified, it inherits the applied +
If a command is specified, it inherits the applied settings from proccontrol. For instance, to start a build of a program with ASLR disabled, use proccontrol -m aslr -s disable make.
-The ports exp run was done with ASLR tuned up to the most +
A ports exp-run was done with ASLR tuned up to the most aggressive settings. The results can be found in PR 208580.
+Case study: Lisp
+SBCL is an interesting case which illustrates several points. It is much smaller than JDK, and its build system is easier to - work with. The code provides a very non C-ish language - runtime which utilizes a lot of corner cases and non-standard - uses of VM, at least from the point of view of a typical C + work with. The code provides a very non C-like language + runtime which utilizes a lot of corner cases and makes non-standard + uses of the VM system, at least from the point of view of a typical C programmer.
SBCL compiles Lisp forms into the machine native code and manages its own arena for objects. The precompiled Lisp - runtime is mapped from the core file. SBCL relies on + runtime is mapped from a core file. SBCL relies on the operating system's C runtime for the initial load of Lisp, and needs a functional libc to issue many system calls, including syscalls, as well as the dynamic loader. The end result is that there are unfixed mmap(2) calls - during both startup and runtime, interfering with the - MAP_FIXED mmaps. The core file loading and arenas - are hard-coded to exist at fixed addresses.
+ during both startup and runtime, interfering with other + MAP_FIXED mmaps. The loading of the core file + and the private arenas are hard-coded to exist at fixed + addresses.This happens to work on the default address map, which is not changed often, so the SBCL choices of the base addresses evolved to work. But any significant distortion of the standard map results in SBCL mmap(MAP_FIXED) requests - to override memory from other allocators.
+ attempting to override memory from other allocators.&os; uses the MAP_EXCL flag to mmap(2), - which must be used in MAP_FIXED|MAP_EXCL form to - cause mmap(2) failure if the requested range is + which must be used in the MAP_FIXED|MAP_EXCL form to + cause mmap(2) to fail if the requested range is already used. I tried to force MAP_FIXED requests - from SBCL to implicitely set MAP_EXCL, but this did + from SBCL to implicitly set MAP_EXCL, but this did not go well since SBCL sometimes pre-allocates regions for later use with MAP_FIXED. So, MAP_EXCL mappings failed, dumping the process into ldb.
-On Linux, if a kernel is detected in AS-randomization mode, - the initial SBCL runtime sets personality to non-random and +
On Linux, if it is detected that the kernel is in AS-randomization mode, + the initial SBCL runtime sets its personality to non-random and re-execs. This might be a solution for &os; as well, after the ASLR patch is committed, so that the procctl(2) knob is officially available.
SBCL still has issues on Linux, even with re-exec, when - more aggressive randomization from PaX patch is applied, as + more aggressive randomization from the PaX patch is applied, as seen in bug 1523213.
+Case study: Emacs
+The Emacs build procedure involves loading the temacs image with the compiled Emacs Lisp files and - then dumping the memory to recreate the image with the - preloaded content, in order to shrink the start time.
+ then dumping its memory to create an image with the content + preloaded, in order to reduce startup time.Recent Emacs sources seem to generally avoid MAP_FIXED, except in some situations. When Emacs does use the flag, it carefully checks that the selected - region is not busy. In fact, Emacs would benefit from + region is not busy. In fact, Emacs would benefit from using MAP_EXCL.
I tried several runs of building Emacs and running the dumped - binary, but was not able to reproduce the issue. It seems + binary, but was not able to reproduce any issues. It seems that the code improved enough to tolerate ASLR both in Linux and NetBSD without turning it off.
In my opinion, it is not reasonable to fight the issues in the kernel as most of it is not fixable from the kernel side. The procctl(2) interface and proccontrol(1) - utilities provide the override when needed, but are not + utilities provide an override when needed, but are not automated.
+Conclusions
+The set of ports which cannot be built with ASLR turned on should be limited but fluid. However, exp-runs may not reliably uncover all problems due to randomization, as seen in - the Emacs example. In the route to enable ASLR by default in - non-aggressive settings, the ports framework should provide an + the Emacs example. In the route to enable ASLR by default (with + non-aggressive settings), the ports framework should provide an option like ASLR_UNSAFE=yes which spawns proccontrol -m aslr -s disable make for the build stages of the unsafe port. Users would still need to be aware of proccontrol(1) in order to run the resulting - binary.
+ binary or wrapper scripts provided to do so.A recommended approach is a flag in the ELF binary to mark it as not compatible with non-standard AS layouts. This frees users from having to use proccontrol(1), but still - requires patching and upstreaming. This approach is also - useful outside the context of ASLR. However, the + requires patching the application's build process and + upstreaming the changes. This approach is also + useful outside the context of ASLR. However, that mechanism is not yet ready, and developing it is a larger work than ASLR itself.
@@ -1062,7 +1073,7 @@The GSoC project about being able to connect a /dev entry to sysctl nodes is making progress. After some fruitful discussons on the freebsd-arch@ mailing-list, + href="https://lists.freebsd.org/mailman/listinfo/freebsd-arch">freebsd-arch@ + mailing-list, Kiloreux finished the design and is now implementing the solution. The GSoC project on libdevq was abandoned.
@@ -1090,7 +1102,7 @@The Radeon AMD/ATI has been updated to GCN 1.0. This has +
The Radeon AMD/ATI driver has been updated to GCN 1.0. This has only been tested on an R7 240. 2D-accelerated X works. Due to apparent issues with user library support, X does not recognize the KMS driver as being 3D-capable and reports it as - "not DRI2 capable". The OpenCL benchmark clpeak + "not DRI2 capable". The OpenCL benchmark clpeak fails in drm/ttm, so there may in fact be issues in the underlying 3D support.
@@ -1131,7 +1143,7 @@ -Another of Core's important functions is to ensure good relations amongst developers. To that end, members of Core provided oversight over the backing-out of disputed - blacklistd related patches to OpenSSH, and acted to + blacklistd-related patches to OpenSSH, and acted to smooth over ruffled tempers.
This quarter saw the usual quota of gentle reminders to avoid @@ -1503,7 +1515,7 @@
-Since the Triage Team was introduced in the October-December +
Since the Triage Team was introduced in the October–December 2015 report, it has been working on the following three major aspects of issue triage:
@@ -1520,13 +1532,13 @@Our efforts have almost exclusively focused on issues in the - "Ports & Packages" component as that is the easiest - starting point. Other categories like "Base System" require + "Ports & Packages" component as that is the easiest + starting point. Other categories like "Base System" require more knowledge and experience with problem content and workflow.
During this time, Rodrigo was inactive due to lack of - available time and Vladimir was unable to commit enough time + available time, and Vladimir was unable to commit enough time during the first quarter of the year, but provided active contribution during the second. It became obvious that the Issue Triage Team must concentrate on additional recruitment @@ -1537,17 +1549,17 @@ Wiki page. A summary of those issues is given here:
The Diffoscope tool performs in-depth comparison of files, @@ -1795,11 +1808,11 @@
A lot of work was done on modernizing the infrastructure of - the Ports Tree, by introducing 6 new USES, one new + the Ports Tree, by introducing 6 new USES knobs, one new keyword, and splitting out the larger targets of bsd.port.mk into separate scripts. There were a total of 42 exp-runs to validate these and other @@ -1889,7 +1901,7 @@ the package building infrastructure.
During BSDCan, mat worked on various items, - including updating the Porter's Handbook, and portmgr held a + including updating the Porter's Handbook, and portmgr held a meeting to discuss various items.