Introduction

From owner-svn-doc-all@freebsd.org Sun Jan 31 22:49:42 2016 Return-Path: Delivered-To: svn-doc-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7DA14A74AB9; Sun, 31 Jan 2016 22:49:42 +0000 (UTC) (envelope-from wblock@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2D7DC1DDF; Sun, 31 Jan 2016 22:49:42 +0000 (UTC) (envelope-from wblock@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id u0VMnfQS028257; Sun, 31 Jan 2016 22:49:41 GMT (envelope-from wblock@FreeBSD.org) Received: (from wblock@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id u0VMnf0B028256; Sun, 31 Jan 2016 22:49:41 GMT (envelope-from wblock@FreeBSD.org) Message-Id: <201601312249.u0VMnf0B028256@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: wblock set sender to wblock@FreeBSD.org using -f From: Warren Block Date: Sun, 31 Jan 2016 22:49:41 +0000 (UTC) To: doc-committers@freebsd.org, svn-doc-all@freebsd.org, svn-doc-head@freebsd.org Subject: svn commit: r48133 - head/en_US.ISO8859-1/htdocs/news/status X-SVN-Group: doc-head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-doc-all@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "SVN commit messages for the entire doc trees \(except for " user" , " projects" , and " translations" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 31 Jan 2016 22:49:42 -0000 Author: wblock Date: Sun Jan 31 22:49:41 2016 New Revision: 48133 URL: https://svnweb.freebsd.org/changeset/doc/48133 Log: Whitespace-only fixes, translators please ignore. Modified: head/en_US.ISO8859-1/htdocs/news/status/report-2015-10-2015-12.xml Modified: head/en_US.ISO8859-1/htdocs/news/status/report-2015-10-2015-12.xml ============================================================================== --- head/en_US.ISO8859-1/htdocs/news/status/report-2015-10-2015-12.xml Sun Jan 31 21:51:40 2016 (r48132) +++ head/en_US.ISO8859-1/htdocs/news/status/report-2015-10-2015-12.xml Sun Jan 31 22:49:41 2016 (r48133) @@ -16,9 +16,9 @@ Introduction

This is a draft of the October–December 2015 - status report. Please check back after it is finalized, and - an announcement email is sent to the &os;-Announce mailing - list.

+ status report. Please check back after it is finalized, and + an announcement email is sent to the &os;-Announce mailing + list.

This report covers &os;-related projects between October and @@ -26,13 +26,13 @@ 2015.

The fourth quarter of 2015 was another productive quarter for - the &os; project and community. [...]

+ the &os; project and community. [...]

Thanks to all the reporters for the excellent work!

The deadline for submissions covering the period from January to March 2016 is April 7, 2016.

- ?> + ?> @@ -84,7 +84,8 @@ - Linux Kernel as a Library Added to the Ports Collection + Linux Kernel as a Library Added to the Ports + Collection @@ -102,10 +103,10 @@

LKL ("Linux Kernel as a Library") is a special - "architecture" of the full Linux kernel that builds as a - userspace library on various platforms, including &os;. One - application of such a library is using Linux filesystem drivers - to implement a FUSE backend.

+ "architecture" of the full Linux kernel that builds + as a userspace library on various platforms, including &os;. + One application of such a library is using Linux filesystem + drivers to implement a FUSE backend.

fusefs-lkl's lklfuse binary is such a FUSE filesystem. It can mount ext4/3/2, XFS, and @@ -119,7 +120,8 @@ - <tt>style(9)</tt> Enhanced to Allow C99 <tt>bool</tt> + <tt>style(9)</tt> Enhanced to Allow C99 + <tt>bool</tt> @@ -146,8 +148,8 @@

Use of bool is now allowed. It was allowed - previously, as well, but now it is really allowed. Party - like it's 1999!

+ previously, as well, but now it is really allowed. + Party like it's 1999!

@@ -156,7 +158,8 @@ -

Specify style(9)'s opinion on iso646.h.

Specify style(9)'s opinion on + iso646.h.

@@ -201,15 +204,16 @@ -

Support was added for fixed-width sysctls - (signed and unsigned 8-bit, 16-bit, 32-bit, and 64-bit integers). - The new KPIs are documented in the sysctl(9) manual page. - The sysctl(8) command line tool supports all of the new - types.

- -

sysctl(8) gained the -t flag, which prints sysctl type - information (the original patch was submitted by Yoshihiro Ota). - This support includes the newly added fixed-width types.

Support was added for fixed-width sysctls (signed and + unsigned 8-bit, 16-bit, 32-bit, and 64-bit integers). The new + KPIs are documented in the sysctl(9) manual page. + The sysctl(8) command line tool supports all of the + new types.

+ +

sysctl(8) gained the -t flag, which prints + sysctl type information (the original patch was submitted by + Yoshihiro Ota). This support includes the newly added + fixed-width types.

@@ -236,17 +240,16 @@ -

I/OAT DMA engines are bulk memory operation offload - engines built into some Intel Server/Storage platform - CPUs.

- -

Several enhancements were made to the driver. - It now avoids memory allocation in locked paths, which - should avoid deadlocking in memory pressure scenarios. Support - for Broadwell-EP devices has been added. The +

I/OAT DMA engines are bulk memory operation offload engines + built into some Intel Server/Storage platform CPUs.

+ +

Several enhancements were made to the driver. It now avoids + memory allocation in locked paths, which should avoid + deadlocking in memory pressure scenarios. Support for + Broadwell-EP devices has been added. The "blockfill" operation and a non-contiguous 8 KB copy - operation have been added to the API. The driver can recover from - various programming errors by resetting the hardware.

+ operation have been added to the API. The driver can recover + from various programming errors by resetting the hardware.

@@ -255,13 +258,15 @@ -

XOR and other advanced ("RAID") operation support.

XOR and other advanced ("RAID") operation + support.

- <tt>ntb_hw(4)</tt>/<tt>if_ntb(4)</tt> Driver Synced up to Linux + <tt>ntb_hw(4)</tt>/<tt>if_ntb(4)</tt> Driver Synced up to + Linux @@ -279,16 +284,17 @@ -

ntb_hw(4) is now up-to-date with the Linux NTB driver as - of the work-in-progress 4.4 kernel (and actually, contains some - fixes that haven't landed in the mainline Linux tree yet but will - land in 4.5). Only Back-to-back ("B2B") configurations - are supported at this time. Going forward, newer hardware may - only support the B2B configuration.

- -

if_ntb(4) is mostly up-to-date with the Linux NTB netdevice - driver. Notably absent is support for changing the MTU at - runtime.

ntb_hw(4) is now up-to-date with the Linux NTB + driver as of the work-in-progress 4.4 kernel (and actually, + contains some fixes that haven't landed in the mainline Linux + tree yet but will land in 4.5). Only Back-to-back + ("B2B") configurations are supported at this time. + Going forward, newer hardware may only support the B2B + configuration.

+ +

if_ntb(4) is mostly up-to-date with the Linux NTB + netdevice driver. Notably absent is support for changing the + MTU at runtime.

@@ -298,8 +304,8 @@

Improving if_ntb(4) to avoid using the entire Base - Address Register (BAR) when very large BAR sizes are configured - (e.g., 512 GB).

+ Address Register (BAR) when very large BAR sizes are + configured (e.g., 512 GB).

@@ -337,13 +343,12 @@ -

We made the changes required to support the - Amlogic Meson Ethernet controller on the Hardkernel ODROID-C1 - board, which has an Amlogic aml8726-m8b SoC. The main effort - needed was to write a glue driver for the Ethernet controller - — the Amlogic Meson Ethernet controller is compatible with - Synopsys DesignWare 10/100/1000 Ethernet MAC - (if_dwc).

We made the changes required to support the Amlogic Meson + Ethernet controller on the Hardkernel ODROID-C1 board, which + has an Amlogic aml8726-m8b SoC. The main effort needed was to + write a glue driver for the Ethernet controller — the + Amlogic Meson Ethernet controller is compatible with Synopsys + DesignWare 10/100/1000 Ethernet MAC (if_dwc).

@@ -369,16 +374,16 @@

The Mellanox &os; team is proud to announce support for the ConnectX-4 series of network cards in &os; 11-current and &os; 10-stable. These devices deliver top performance, with up to - 100GBit/s of raw transfer capacity, and support both Ethernet and - Infiniband. Currently, the Ethernet driver is ready for use and - the Infiniband support for ConnectX-4 is making good progress. We - hope that it will be complete before &os; 11.0 is released. For - more technical information, refer to the mlx5en(4) manual - page in 11-current. The new driver for ConnectX-4 cards is called - mlx5 and is put under /sys/dev and not under - /sys/ofed as was done for the previous - mlx4 driver. The mlx5en(4) kernel module is - compiled by default in GENERIC kernels.

+ 100GBit/s of raw transfer capacity, and support both Ethernet + and Infiniband. Currently, the Ethernet driver is ready for + use and the Infiniband support for ConnectX-4 is making good + progress. We hope that it will be complete before &os; 11.0 + is released. For more technical information, refer to the + mlx5en(4) manual page in 11-current. The new driver + for ConnectX-4 cards is called mlx5 and is put under + /sys/dev and not under /sys/ofed as was done + for the previous mlx4 driver. The mlx5en(4) + kernel module is compiled by default in GENERIC kernels.

@@ -406,16 +411,16 @@

FreeBSD Mastery: Specialty Filesystems is now in - copyediting. The ebook should be available by the end of January - at all major vendors, and the print in February.

+ copyediting. The ebook should be available by the end of + January at all major vendors, and the print in February.

The book covers everything from removable media, to FUSE, NFSv4 ACLs, iSCSI, CIFS, and more.

If you act really quickly, you can get the electronic early - access version at a 10% discount. You will get the final ebook when - it comes out as well. (This offer evaporates when the final - version comes out.)

+ access version at a 10% discount. You will get the final + ebook when it comes out as well. (This offer evaporates when + the final version comes out.)

@@ -437,20 +442,21 @@ -

The KGDB option is now on by default in the devel/gdb - port.

The KGDB option is now on by default in the + devel/gdb port.

Changes to support cross-debugging of crashdumps in - libkvm were committed to HEAD in r291406.

Changes to support cross-debugging of crashdumps in libkvm + were committed to HEAD in r291406.

A new thread target for &os; that is suitable for merging - upstream has been written and lightly tested. However, it is not - yet available as an option in the port. This thread target uses - ptrace(2) directly rather than libthread_db and - as such supports threads on all ABIs (such as &os;/i386 binaries - on &os;/amd64 and possibly Linux binaries, though that is not yet - tested). It also requires less-invasive changes in the MD targets - in GDB compared to the libthread_db-based target.

+ upstream has been written and lightly tested. However, it is + not yet available as an option in the port. This thread + target uses ptrace(2) directly rather than + libthread_db and as such supports threads on all ABIs + (such as &os;/i386 binaries on &os;/amd64 and possibly Linux + binaries, though that is not yet tested). It also requires + less-invasive changes in the MD targets in GDB compared to the + libthread_db-based target.

@@ -469,7 +475,8 @@

Add support for more platforms (arm, mips, aarch64) to - upstream gdb for both userland and kgdb.

+ upstream gdb for both userland and + kgdb.

@@ -496,9 +503,9 @@ -

iMX.6 is a family of SoC used in multiple hobbyist ARM - boards such as the Hummingboard, RIoTboard, and Cubox. Most of - these products have HDMI output, but until recently, &os; did not +

iMX.6 is a family of SoC used in multiple hobbyist ARM boards + such as the Hummingboard, RIoTboard, and Cubox. Most of these + products have HDMI output, but until recently, &os; did not benefit from it. As of r292574, there is basic video output support so you can use the console on iMX6-based boards and probably run Xorg (not yet tested).

@@ -521,7 +528,8 @@ - Touchscreen Support for Raspberry Pi and Beaglebone Black + Touchscreen Support for Raspberry Pi and Beaglebone + Black @@ -541,17 +549,17 @@

There are two working proof-of-concept drivers for the - AM335x touchscreen and for the official Raspberry Pi's touchscreen - LCD.

- + AM335x touchscreen and for the official Raspberry Pi's + touchscreen LCD.

Proper touchscreen support would consist of a userland event reading API, a kernel event reporting API, and kernel hardware - drivers for specific devices. There is an ongoing effort to port - the Linux evdev API to &os; so applications that use libraries like - libinput or tslib could be used without any major changes. Since - it is not yet complete, I created a naive evdev-like API for both - kernel and tslib and was able to run a demo on a Beaglebone Black - with 4DCAPE-43T.

+ drivers for specific devices. There is an ongoing effort to + port the Linux evdev API to &os; so applications that use + libraries like libinput or tslib could be used without any + major changes. Since it is not yet complete, I created a + naive evdev-like API for both kernel and tslib and was able to + run a demo on a Beaglebone Black with 4DCAPE-43T.

Once evdev makes it into the tree, both hardware drivers can be modified to include "report events" @@ -610,59 +618,61 @@

This completed project includes changes to better manage - the vnode freelist and to streamline the allocation and freeing of - vnodes.

+ the vnode freelist and to streamline the allocation and + freeing of vnodes.

Vnode cache recycling was reworked to meet free and unused - vnode targets. Free vnodes are rarely completely free; rather, - they are just ones that are cheap to recycle. Usually they are - for files which have been stat'd but not read; these usually have - inode and namecache data attached to them. The free vnode target - is the preferred minimum size of a sub-cache consisting mostly of - such files. The system balances the size of this sub-cache with - its complement to try to prevent either from thrashing while the - other is relatively inactive. The targets express a preference - for the best balance.

+ vnode targets. Free vnodes are rarely completely free; + rather, they are just ones that are cheap to recycle. Usually + they are for files which have been stat'd but not read; these + usually have inode and namecache data attached to them. The + free vnode target is the preferred minimum size of a sub-cache + consisting mostly of such files. The system balances the size + of this sub-cache with its complement to try to prevent either + from thrashing while the other is relatively inactive. The + targets express a preference for the best balance.

"Above" this target there are 2 further targets (watermarks) related to the recyling of free vnodes. In the - best-operating case, the cache is exactly full, the free list has - size between vlowat and vhiwat above the free target, and - recycling from the free list and normal use maintains this state. - Sometimes the free list is below vlowat or even empty, but this - state is even better for immediate use, provided the cache is not - full. Otherwise, vnlru_proc() runs to reclaim enough vnodes - (usually non-free ones) to reach one of these states. The - watermarks are currently hard-coded as 4% and 9% of the available - space. These, and the default of 25% for wantfreevnodes, are too - large if the memory size is large. For example, 9% of 75% of MAXVNODES - is more than 566000 vnodes to reclaim whenever vnlru_proc() - becomes active.

- -

The vfs.vlru_alloc_cache_src sysctl is removed. - New code frees namecache sources as the last chance to satisfy the - highest watermark, instead of selecting source vnodes randomly. - This provides good enough behavior to keep vn_fullpath() working - in most situations. Filesystem layouts with deep trees, where the - removed knob was required, is thus handled automatically.

+ best-operating case, the cache is exactly full, the free list + has size between vlowat and vhiwat above the free target, and + recycling from the free list and normal use maintains this + state. Sometimes the free list is below vlowat or even empty, + but this state is even better for immediate use, provided the + cache is not full. Otherwise, vnlru_proc() runs to + reclaim enough vnodes (usually non-free ones) to reach one of + these states. The watermarks are currently hard-coded as 4% + and 9% of the available space. These, and the default of 25% + for wantfreevnodes, are too large if the memory size is large. + For example, 9% of 75% of MAXVNODES is more than 566000 vnodes + to reclaim whenever vnlru_proc() becomes active.

+ +

The vfs.vlru_alloc_cache_src sysctl is removed. New + code frees namecache sources as the last chance to satisfy the + highest watermark, instead of selecting source vnodes + randomly. This provides good enough behavior to keep + vn_fullpath() working in most situations. Filesystem + layouts with deep trees, where the removed knob was required, + is thus handled automatically.

As the kernel allocates and frees vnodes, it fully - initializes them on every allocation and fully releases them on - every free. These are not trivial costs: it starts by zeroing a - large structure, then initializes a mutex, a lock manager lock, an - rw lock, four lists, and six pointers. Looking at - vfs.vnodes_created, these operations are being done - millions of times an hour on a busy machine.

+ initializes them on every allocation and fully releases them + on every free. These are not trivial costs: it starts by + zeroing a large structure, then initializes a mutex, a lock + manager lock, an rw lock, four lists, and six pointers. + Looking at vfs.vnodes_created, these operations are + being done millions of times an hour on a busy machine.

As a performance optimization, this code update uses the uma_init and uma_fini routines to do these initializations and - cleanups only as the vnodes enter and leave the vnode zone. With - this change, the initializations are done kern.maxvnodes - times at system startup, and then only rarely again. The frees - are done only if the vnode zone shrinks, which never happens in - practice. For those curious about the avoided work, look at the - vnode_init() and vnode_fini() functions in sys/kern/vfs_subr.c to - see the code that has been removed from the main vnode + cleanups only as the vnodes enter and leave the vnode zone. + With this change, the initializations are done + kern.maxvnodes times at system startup, and then only + rarely again. The frees are done only if the vnode zone + shrinks, which never happens in practice. For those curious + about the avoided work, look at the vnode_init() and + vnode_fini() functions in sys/kern/vfs_subr.c to see + the code that has been removed from the main vnode allocation/free path.

@@ -682,9 +692,9 @@

The QLogic HBA driver, isp(4), received a - substantial set of changes. The primary goal was to make the Fibre - Channel target role work well with CTL, but many other things were - also fixed/improved:

+ substantial set of changes. The primary goal was to make the + Fibre Channel target role work well with CTL, but many other + things were also fixed/improved:

Added support for modern 16Gbps 26xx FC cards.
The firmware in ispfw(4) were updated to the latest versions.
Target role support was fixed and tested for all FC cards from - ancient 1Gbps 22xx to modern 16Gbps 26xx.
Target role support was fixed and tested for all FC cards + from ancient 1Gbps 22xx to modern 16Gbps 26xx.
Port database handling was unified for target and initiator - roles, allowing HBA port to play both roles at the same time.
Port database handling was unified for target and + initiator roles, allowing HBA port to play both roles at the + same time.
The maximal number of ports was increased from 256 to 1024.
Multi-ID (NPIV) functionality was fixed/implemented, allowing - 24xx and above cards to provide up to 255 virtual FC ports per - physical port.
Multi-ID (NPIV) functionality was fixed/implemented, + allowing 24xx and above cards to provide up to 255 virtual + FC ports per physical port.
Added support for 8-byte LUNs for 24xx and above cards.
Added support for 8-byte LUNs for 24xx and above + cards.

The code is committed to &os; head and stable/10 branches.

The code is committed to &os; head and stable/10 + branches.

@@ -727,7 +740,8 @@ - Raspberry Pi: VideoCore Userland Application Packaging + Raspberry Pi: VideoCore Userland Application + Packaging @@ -750,15 +764,15 @@

The Raspberry Pi SoC consists of two parts: ARM and GPU (VideoCore). Many interesting features like OpenGL, video - playback, and HDMI controls are implemented on the VideoCore side - and can be accessed from the OS through libraries provided by - Broadcom (userland repo). These libraries were ported to &os; - some time ago, so Mikaël created the port - misc/raspberrypi-userland for them. He also created a - port for omxplayer (a low-level video player that + playback, and HDMI controls are implemented on the VideoCore + side and can be accessed from the OS through libraries + provided by Broadcom (userland repo). These libraries were + ported to &os; some time ago, so Mikaël created the port + misc/raspberrypi-userland for them. He also created + a port for omxplayer (a low-level video player that utilizes VideoCore APIs) and is working on a port for Kodi - (formerly XBMC), a more user-firendly media player software with - Raspberry Pi support.

+ (formerly XBMC), a more user-firendly media player software + with Raspberry Pi support.

@@ -782,16 +796,16 @@

LXQt is the Qt port of and - the upcoming version of LXDE, the Lightweight Desktop Environment. - It is the product of the merge between the LXDE-Qt and the - Razor-qt projects.

+ the upcoming version of LXDE, the Lightweight Desktop + Environment. It is the product of the merge between the + LXDE-Qt and the Razor-qt projects.

The porting effort remains very much a work in progress: it needs some components of Plasma 5, the new major KDE workspace.

Currently, only the 0.10 branch is functional. See our - wiki page for a complete list of applications.

Currently, only the 0.10 branch is functional. See our wiki + page for a complete list of applications.

We also sent updates for some components of LXDE, required for the LXQt desktop:

@@ -802,9 +816,8 @@

x11/lxmenu-data 0.1.4

Binary packages are available (only for test purposes) - which are regularly tested with the KDE development - repository.

Binary packages are available (only for test purposes) which + are regularly tested with the KDE development repository.

@@ -838,14 +851,15 @@

Node.js is a platform built on Chrome's JavaScript runtime - for easily building fast, scalable network applications. It uses - an event-driven, non-blocking I/O model that makes it lightweight - and efficient — perfect for data-intensive real-time applications - that run across distributed devices.

+ for easily building fast, scalable network applications. It + uses an event-driven, non-blocking I/O model that makes it + lightweight and efficient — perfect for data-intensive + real-time applications that run across distributed + devices.

The goal of this project is to make it easy to install the - modules available in the npm package - registry.

+ modules available in the + npm package registry.

Currently, the repository contains slightly fewer than 300 new ports, in particular:

@@ -878,7 +892,8 @@ -

Bring in grunt.js (and modules), the JavaScript task runner.

Bring in grunt.js (and modules), the JavaScript task + runner.

@@ -949,17 +964,18 @@

I recently became involved with &os; (as in, the last 2-3 - weeks), and found myself quickly involved with Ports development. - What struck me immediately was the difficulty in providing a Python - package that was depended upon by multiple versions of Python. As - it turns out, poudriere can currently only generate one package - per port, meaning that a Python version-neutral (compatible with - 2.x and 3.x) port cannot simultaneously be packaged for each - variant at the same time.

+ weeks), and found myself quickly involved with Ports + development. What struck me immediately was the difficulty in + providing a Python package that was depended upon by multiple + versions of Python. As it turns out, poudriere can currently + only generate one package per port, meaning that a Python + version-neutral (compatible with 2.x and 3.x) port cannot + simultaneously be packaged for each variant at the same + time.

I discussed the issue with &a.koobs;, who suggested that I - look into implementing a "variants protocol" within the - Ports framework and the necessary changes to poudriere to + look into implementing a "variants protocol" within + the Ports framework and the necessary changes to poudriere to allow a port to generate more than one package.

Support for variants is strongly needed in Ports and @@ -967,16 +983,16 @@

It would allow Python and other languages to provide - packages for dependencies for multiple language versions from the - same port.
It alleviates the need for so-called "slave - ports", as a single port could now have multiple generated - packages from a single port.
It would have a very small impact on the greater Ports - ecosystem: adding only two new variables, VARIANT and - VARIANTS.

VARIANT

VARIANTS

It would provide a more consistent approach between different packaging teams for handling variations.

Fortunately, this is not a wishful thinking piece. I dug - in my heels and have implemented a proof-of-concept implementation - of variants in the Ports framework, including the necessary - modifications to poudriere in order to support it. It was mildly - upsettling to find that poudriere is mostly written in Bourne shell - scripts, but I pressed on nonetheless.

I started with the - prototype made by &a.bapt; as a base, and built from there. - The poudriere PoC aims to limit changes as much as possible to - merely adding support for the new variants flags, while also at - the request of &a.koobs; making the logging output more - package-centric (as opposed to port-centric) as a result of these - changes.

I started with + the prototype made by &a.bapt; + as a base, and built from there. The poudriere PoC aims to + limit changes as much as possible to merely adding support for + the new variants flags, while also at the request of &a.koobs; + making the logging output more package-centric (as opposed to + port-centric) as a result of these changes.

This is a work in progress, and I would love to hear your feedback. I have enjoyed my first few weeks - working on &os;, and I hope to stay here for quite some time.

@@ -1018,8 +1035,8 @@ -

Hopefully the code will be of sufficient quality to be considered - for formal review in the coming months.

Hopefully the code will be of sufficient quality to be + considered for formal review in the coming months.

When I starting working on ports for &os; in the last - couple of weeks, I found that my workflow was not as efficient as - it could be using just the available tools, so I made a few that +

When I starting working on ports for &os; in the last couple + of weeks, I found that my workflow was not as efficient as it + could be using just the available tools, so I made a few that could be useful to the development community at large. All of - these have been or will soon be added to the Ports tree, - so you can play with them today!

pytoport is a command-line application that generates a skeleton port for a given PyPI package name. It attempts to generate the correct dependencies, makes a good - attempt at guessing the license using spdx-lookup, and - generates a pkg-descr. This made generating the fifteen - or so ports I was working on a complete breeze.

spdx-lookup

pkg-descr

While doing this, however, I noticed that some ports were - bringing in dependencies that I did not expect, and I needed some - way to visualise this. skog builds a dependency tree - from the depends lists output by the Ports framework, and displays - it on the command line (with extra shiny output if you are using - UTF-8). No more pesky example and documentation dependencies - being dragged in when you clearly toggled that - OPTION as far off as it would go.

While doing all of this, I found it cumbersome to be - copying ports back and forth between my small development tree - living in git and the larger upstream SVN tree I was using in + bringing in dependencies that I did not expect, and I needed + some way to visualise this. skog builds a dependency + tree from the depends lists output by the Ports framework, and + displays it on the command line (with extra shiny output if + you are using UTF-8). No more pesky example and documentation + dependencies being dragged in when you clearly + toggled that OPTION as far off as it would go.

While doing all of this, I found it cumbersome to be copying + ports back and forth between my small development tree living + in git and the larger upstream SVN tree I was using in poudriere. I built a tool called bandar that takes - advantage of the FUSE version of unionfs to easily overlay my dev - tree on the upstream tree, run linting, poudriere, and generate - archives with ease.

I am very impressed with how easy it was to build more - tooling for &os;. I hope some of these tools will be of some use - to you, and as always, I'd love to hear your feedback!

Improve skog to support searching a tree for a certain - port.

Improve skog to support searching a tree for a + certain port.

@@ -1092,8 +1109,8 @@ -

Continue to improve pytoport, adding trove support and better - dependency handling.

Continue to improve pytoport, adding + trove support and better dependency handling.

@@ -1118,68 +1135,70 @@

The Out of Memory (OOM) code is intended to handle the situation where the system needs free memory to make progress, - but no memory can be reused. Most often, the situation is that - to free memory, the system needs more free memory. Consider a - case where the system needs to page-out dirty pages, but needs to - allocate structures to track the writes. OOM "solves" - the problem by killing some selection of user processes. In other - words, it trades away system deadlock by suffering a partial loss - of user data. The assumption is that it is better to kill a - process and recover data in other processes than to lose - everything.

+ but no memory can be reused. Most often, the situation is + that to free memory, the system needs more free memory. + Consider a case where the system needs to page-out dirty + pages, but needs to allocate structures to track the writes. + OOM "solves" the problem by killing some selection + of user processes. In other words, it trades away system + deadlock by suffering a partial loss of user data. The + assumption is that it is better to kill a process and recover + data in other processes than to lose everything.

Free memory in the &os; Virtual Memory (VM) system appears - from two sources. One is the voluntary reclamation of pages used - by a process, for example unmapping private anonymous regions, or - the last unlink of an otherwise unreferenced file with cached - pages. Another source is the pagedaemon, which forcefully frees - pages which carry data, of course after the data is moved to some - other storage, like swap or file blocks. OOM is triggered when - the pagedaemon definitely cannot free memory to satisfy the - requests.

- -

The old criteria to trigger the OOM action was a combination of - low free swap space and a low count of free pages (the latter is - expressed precisely with the paging targets constants, but this is - not relevant to the discussion). That test is mostly incorrect. - For example, a low free page state might be caused by a greedy consumer - allocating all pages freed by the page daemon in the current pass, - but this does not preclude the page daemon from producing more - pages. Also, since page-outs are asynchronous, the previous page - daemon pass might not immmediately produce any free pages, but - they would appear some short time later.

+ from two sources. One is the voluntary reclamation of pages + used by a process, for example unmapping private anonymous + regions, or the last unlink of an otherwise unreferenced file + with cached pages. Another source is the pagedaemon, which + forcefully frees pages which carry data, of course after the + data is moved to some other storage, like swap or file blocks. + OOM is triggered when the pagedaemon definitely cannot free + memory to satisfy the requests.

+ +

The old criteria to trigger the OOM action was a combination + of low free swap space and a low count of free pages (the + latter is expressed precisely with the paging targets + constants, but this is not relevant to the discussion). That + test is mostly incorrect. For example, a low free page state + might be caused by a greedy consumer allocating all pages + freed by the page daemon in the current pass, but this does + not preclude the page daemon from producing more pages. Also, + since page-outs are asynchronous, the previous page daemon + pass might not immmediately produce any free pages, but they + would appear some short time later.

More seriously, low swap space does not necessarily indicate that we are in trouble: lots of pages might not require swap - allocations to be freed, like clean pages or pages backed by files. - The last notion is serious, since swap-less systems were - considered as having full swap.

- -

Instead of trying to deduce the deadlock from looking at - the current VM state, the new OOM handler tracks the history of - page daemon passes. Only when several consequtive passes failed to - meet the paging target is an OOM kill considered necessary. The - count of consequent failed passes was selected empirically, by - testing on small (32M) and large (512G) machines. Auto-tuning of - the counter is possible, but requires some more architectural - changes to the I/O subsystem.

+ allocations to be freed, like clean pages or pages backed by + files. The last notion is serious, since swap-less systems + were considered as having full swap.

+ +

Instead of trying to deduce the deadlock from looking at the + current VM state, the new OOM handler tracks the history of + page daemon passes. Only when several consequtive passes + failed to meet the paging target is an OOM kill considered + necessary. The count of consequent failed passes was selected + empirically, by testing on small (32M) and large (512G) + machines. Auto-tuning of the counter is possible, but + requires some more architectural changes to the I/O + subsystem.

Another issue was identified with the algorithm which - selects a victim process for OOM kill. It compared the counts of +

Another issue was identified with the algorithm which selects + a victim process for OOM kill. It compared the counts of pages mapping entries (PTEs) installed into the machine paging structures. For different reasons, machine-dependent VM code - (pmap) may remove the pte for a memory-resident page. Under some - circumstances related to other measures to prevent low memory - deadlock, very large processes which consume all system memory - could have few or no ptes. The old OOM selector ignored the - process which caused the deadlock, killing unrelated - processes.

- -

A new function, vm_pageout_oom_pagecount(), was written which - applies a reasonable heuristic to estimate the number of pages - freed by killing the given process. This - eliminates the effect of selecting small unrelated processes for - OOM kill.

+ (pmap) may remove the pte for a memory-resident page. Under + some circumstances related to other measures to prevent low + memory deadlock, very large processes which consume all system + memory could have few or no ptes. The old OOM selector + ignored the process which caused the deadlock, killing + unrelated processes.

+ +

A new function, vm_pageout_oom_pagecount(), was + written which applies a reasonable heuristic to estimate the + number of pages freed by killing the given process. This + eliminates the effect of selecting small unrelated processes + for OOM kill.

The rewrite was committed to HEAD in r290917 and r290920.

@@ -1205,15 +1224,16 @@ -

A new driver, cxgbei, enabling hardware - accelerated iSCSI with Chelsio's T5- and T4-based offload-capable - cards, has been committed to HEAD. Both Initiator and Target are - supported. The wire traffic is standard iSCSI (SCSI over TCP as - per RFC 3720, etc.) so an Initiator/Target using this driver will - interoperate with all other standards-compliant +

A new driver, cxgbei, enabling hardware accelerated + iSCSI with Chelsio's T5- and T4-based offload-capable cards, + has been committed to HEAD. Both Initiator and Target are + supported. The wire traffic is standard iSCSI (SCSI over TCP + as per RFC 3720, etc.) so an Initiator/Target using this + driver will interoperate with all other standards-compliant implementations.

Hardware assistance provided by the T5 and T4 ASICs includes:

Hardware assistance provided by the T5 and T4 ASICs + includes:

Complete TCP processing.

The driver is in advanced stage QA and will see some - bugfixes and performance enhancements in the very near future. - MFC is possible as soon as the QA cycle completes.

+ bugfixes and performance enhancements in the very near + future. MFC is possible as soon as the QA cycle + completes.

OpenBSM is a BSD-licensed implementation of Sun's Basic - Security Module (BSM) API and file format. It is the user-space - side of the CAPP Audit implementations in &os; and Mac OS X. - Additionally, the audit trail processing tools are expected to - work on Linux.

Progress has been slow but steady this quarter, culminating in OpenBSM 1.2 alpha 4, the first release in three years. It features various bug fixes and documentation improvements; the - complete list of changes is documented in the NEWS + complete list of changes is documented in the + NEWS file on GitHub. The release was imported into &os; HEAD and merged to &os; 10-STABLE. As such it will be part of &os; 10.3-RELEASE.

Test the new release on different versions of &os;, Mac - OS X, and Linux. In particular, testing on Mac OS X 10.9 (Mavericks) - and newer would be greatly appreciated.

Test the new release on different versions of &os;, Mac OS + X, and Linux. In particular, testing on Mac OS X 10.9 + (Mavericks) and newer would be greatly appreciated.

@@ -1354,12 +1375,12 @@ *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***