From owner-freebsd-stable@freebsd.org Mon Dec 25 23:56:25 2017 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9CCBDE82C7B; Mon, 25 Dec 2017 23:56:25 +0000 (UTC) (envelope-from kaduk@mit.edu) Received: from dmz-mailsec-scanner-4.mit.edu (dmz-mailsec-scanner-4.mit.edu [18.9.25.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EE51079001; Mon, 25 Dec 2017 23:56:21 +0000 (UTC) (envelope-from kaduk@mit.edu) X-AuditID: 1209190f-c8dff70000003990-47-5a41901c401c Received: from mailhub-auth-1.mit.edu ( [18.9.21.35]) (using TLS with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by dmz-mailsec-scanner-4.mit.edu (Symantec Messaging Gateway) with SMTP id A1.1E.14736.C10914A5; Mon, 25 Dec 2017 18:56:12 -0500 (EST) Received: from outgoing.mit.edu (OUTGOING-AUTH-1.MIT.EDU [18.9.28.11]) by mailhub-auth-1.mit.edu (8.13.8/8.9.2) with ESMTP id vBPNuAGT021437; Mon, 25 Dec 2017 18:56:11 -0500 Received: from kduck.kaduk.org (24-107-191-124.dhcp.stls.mo.charter.com [24.107.191.124]) (authenticated bits=56) (User authenticated as kaduk@ATHENA.MIT.EDU) by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id vBPNu6hN017279 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Mon, 25 Dec 2017 18:56:09 -0500 Date: Mon, 25 Dec 2017 17:56:07 -0600 From: Benjamin Kaduk To: freebsd-hackers@FreeBSD.org Cc: freebsd-current@FreeBSD.org, freebsd-stable@FreeBSD.org Subject: FreeBSD Quarterly Status Report - Third Quarter 2017 Message-ID: <20171225235606.GE59505@kduck.kaduk.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="DSayHWYpDlRfCAAQ" Content-Disposition: inline User-Agent: Mutt/1.9.1 (2017-09-22) X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrFKsWRmVeSWpSXmKPExsUixCmqrCszwTHKYEe7gsWua6fZLea8+cBk sX3zP0aLw81CDiweMz7NZwlgjOKySUnNySxLLdK3S+DK+Nw3naXg9T6mip0n/zM3MD79ydjF yMkhIWAiceXka7YuRi4OIYHFTBIrf/xihXA2Mko8+j2fHcK5yiRxZs1sNpAWFgFViWs/v4HZ bAJqEutXXGMGsUUE5CX2Nb1nB7GZBawl/j1oBIsLC9hK9H9dClbPC7Su6/lcKFtQ4uTMJywQ 9WUSL243AtkcQLa0xPJ/HCBhUQFlib19h9gnMPLNQtIxC0nHLIQOiLCOxM6td9gwhLUlli18 zQxh20qsW/eeZQEj+ypG2ZTcKt3cxMyc4tRk3eLkxLy81CJdE73czBK91JTSTYzgAJfk38E4 p8H7EKMAB6MSD++BCMcoIdbEsuLK3EOMkhxMSqK8ZdMcooT4kvJTKjMSizPii0pzUosPMaoA 7Xq0YfUFRimWvPy8VCURXmeQOt6UxMqq1KJ8mDJpDhYlcV53E+0oIYH0xJLU7NTUgtQimKwM B4eSBK9rP9BOwaLU9NSKtMycEoQ0EwfnIUYJDh6g4fNBaniLCxJzizPTIfKnGMM5jm26/IeJ 48CEK0Dy1MPbQPLRjbtAcsX/e0Dy2czXDcwc845/a2LmOPT7UCuzENitUuK8e0DGCYCMyyjN g9sISoIS2ftrXjGKAwNDmLcapIoHmEDhdr4COocJ6Jx0T3uQc0oSEVJSDYw7L+lxTmlL/9p/ POBp35/AsDXidwJNYstW/tSO+Frc0rvo1M3J52YvOnunkPfTqe5FSXov2yI6ziyO1l5ioMrc 9FPgQ+u2Rbm+1VW5Z5z3nVx+YmOHdbCL962FvfNa1otMTFjn80j43t6///tbf9b/VFY0LjRd VzRN7czE1becllQ3CdlsPMyuxFKckWioxVxUnAgAhkbwcV0DAAA= X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Dec 2017 23:56:25 -0000 --DSayHWYpDlRfCAAQ Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable FreeBSD Project Quarterly Status Report - 3rd Quarter 2017 This quarter's FreeBSD developments continue to provide excitement and promise for further developments. I myself have a soft spot for manual pages, so it is especially good to see that we have gained some documentation for writing them (and I hope that this will translate to more and improved manual pages in the future!). The core@ entry is also of particular note, with the introduction of the FCP process and the recognition of the first non-committer FreeBSD Project Member (and more). Read on to find out more about these, as well as improved support for the AMD Zen family of processors (e.g., Ryzen), and a whole lot more! --Benjamin Kaduk __________________________________________________________________ The deadline for submissions covering the period from October to December 2017 is January 14, 2017. __________________________________________________________________ FreeBSD Team Reports * FreeBSD Release Engineering Team * Ports Collection * The FreeBSD Core Team * The FreeBSD Foundation Projects * FreeBSD CI Kernel * Intel 10G iflib Driver Update * Intel iWARP Support * pNFS Server Plan B Architectures * AMD Zen (family 17h) support Userland Programs * Updates to GDB Ports * FreeBSDDesktop * OpenJFX 8 * Puppet Documentation * Absolute FreeBSD, 3rd Edition * Manual Pages Third-Party Projects * The nosh Project __________________________________________________________________ FreeBSD Team Reports Entries from the various official and semi-official teams, as found in the Administration Page. FreeBSD Release Engineering Team Links FreeBSD 11.1-RELEASE Announcement URL: https://www.FreeBSD.org/releases/11.1R/announce.html FreeBSD 10.4-RELEASE Schedule URL: https://www.FreeBSD.org/releases/10.4R/schedule.html FreeBSD Development Snapshots URL: https://download.FreeBSD.org/ftp/snapshots/ISO-IMAGES/ Contact: FreeBSD Release Engineering Team The FreeBSD Release Engineering Team is responsible for setting and publishing release schedules for official project releases of FreeBSD, announcing code freezes, and maintaining the respective branches, among other things. The FreeBSD Release Engineering Team continued finalizing the 11.1-RELEASE cycle, with the final release builds starting on July 21 and the official release announcement email sent on July 26. Thank you to everyone who helped test 11.1-RELEASE, ensuring its quality and stability. [1] FreeBSD 11.1-RELEASE is the second release from the stable/11 branch. Additionally, the FreeBSD Release Engineering Team started the 10.4-RELEASE cycle, with the code slush starting on July 28. With the final release build expected to start on September 29 and the official announcement overlapping the end of the quarter, everything is on schedule as of this writing. [2] FreeBSD 10.4-RELEASE will be the fifth release from the stable/10 branch, and is planned to be the final release of the 10.x series. This project was sponsored by The FreeBSD Foundation [1]. This project was sponsored in part by The FreeBSD Foundation [2]. __________________________________________________________________ Ports Collection Links About FreeBSD Ports URL: https://www.FreeBSD.org/ports/ Contributing to Ports URL: https://www.freebsd.org/doc/en_US.ISO8859-1/articles/contributing/= ports-contributing.html FreeBSD Ports Monitoring URL: http://portsmon.freebsd.org/index.html Ports Management Team Website URL: https://www.freebsd.org/portmgr/index.html FreeBSD portmgr on Twitter (@freebsd_portmgr) URL: https://twitter.com/freebsd_portmgr/ FreeBSD Ports Management Team on Facebook URL: https://www.facebook.com/portmgr FreeBSD Ports Management Team on Google+ URL: https://plus.google.com/communities/108335846196454338383 Contact: Ren=E9 Ladan Contact: FreeBSD Ports Management Team The Ports Collection now features over 31,600 ports. There are currently 2671 problem reports, of which 718 are unassigned. This quarter saw almost 5,900 commits from 175 committers. The number of open PRs grew compared to last quarter, and outpaced the number of changes. This quarter, we welcomed Zach Leslie (zleslie@), Luca Pizzamiglio (pizzamig@), Craig Leres (leres@), Adriaan de Groot (adridg@), and Dave Cottlehuber (dch@) as new committers. The commit bits of the following committers were taken in for safekeeping: alonso@ after 19 months of inactivity, rpaulo@ per his request, and ache@ after he passed away. Despite several tries and changing mentors, kami@ lacked interest in completing his mentorship, so his commit bit was also taken in for safekeeping. On the infrastructure side, two USES values were removed because they outlived their usefulness: * execinfo: libexecinfo is now available in the base system of all supported FreeBSD versions * twisted: there is only one Twisted port left The default version of GCC was bumped from 5 to 6. Firefox was updated to version 56.0 and Chromium to version 61.0.3163.100. The version of pkg itself was updated to 1.10.1. During this quarter, antoine@ performed 28 exp-runs to test version updates of major ports, improving USE_GITHUB and SHEBANG_FILES, and API changes to the base system. This quarter, the foundation for ports "flavors" was committed, though more development and testing will be performed in the coming quarter before it goes live. Open tasks: 1. The PR load needs more attention, as the number of open PRs has started to increase again. __________________________________________________________________ The FreeBSD Core Team Contact: FreeBSD Core Team The new "FreeBSD Community Process" was drafted during BSDCan earlier this year. The first such document, FCP 0, defines how the whole process works. After some time for discussion and revision, FCP 0 was voted on and accepted by core, following the procedure laid down within that document. Currently the use of FCPs is entirely optional; we shall see how the community begins to adopt their usage and evolve the process based on experience. A draft update to the Code of Conduct has been prepared by the advisory committee. Core is currently reviewing the text, and will soon vote on accepting it. Core is keen to avoid the trap of "rules lawyering". At the moment, the feeling is that we need to add a preamble to the CoC to articulate the goals of the project and to act as a general guide to the exercise of the code. This quarter has been quite a busy one concerning changes to the roster of committers and project members. We have elected our first new Project Member: John Hixson, who will be familiar from many conferences where he has given presentations and ably represented iXsystems. A second proposed Project Member was not accepted by core, but only because core felt that Fedor Uporov really deserved a commit bit instead. In addition to Fedor Uporov, please also welcome (in no particular order) Matt Joras, Marcin Wojtas, Chuck Tuffli, Ilya Bakulin and Alex Richardson as brand-new committers. We have also awarded Steven Hurd and Eugene Grosbein src commit bits to go with their existing ports bits. Welcome back Gordon Tetlow as a src committer, essential for his new role within secteam. Eric Davis and Rui Paulo have both decided to hang up their commit bits: we wish them well in their future endeavours. Finally, we must report the sad death of Andrey Chernov, who will be sorely missed by his colleagues and collaborators. Andrey's death has highlighted another question which is only going to become more complex over time. Keeping track of copyrights is already hard enough within a mature source tree with many contributors, such as the FreeBSD sources. Now we need to consider trying to keep track of the heirs and beneficiaries of contributors who have sadly passed away. Core will consult with the Foundation legal team to discuss possible approaches to alleviate this. There have been complaints that the workings of Core are being kept overly confidential, and that consequently the majority of the project has too little idea of what is going on. This is certainly not intentional by Core, and we are keen to open up Core's business to more general community scrutiny as far as seems reasonable. Core dealt with a number of licensing questions: * When upstreaming patches and other original works to VirtualBox or other Oracle properties, pragmatically it works best to provide them under the terms of the MIT license (one of two opensource licenses accepted by Oracle). Of course, this only applies to work upstreamed by or with the permission of the original author. * The Viking software license is sufficiently BSD-like that magic constants from their drivers can be used in FreeBSD code. * There is no separate register of deviations from the allowed BSD-like licenses in the source tree: any code in the tree under other than BSD-like license terms can be assumed to have been approved by core. * At the moment the FreeBSD copyright requirement to include the copyright notice in redistributions in binary form is satisfied by making the FreeBSD sources, with all of the detailed copyright information included in the different source code files, available alongside pre-compiled system images. However, this does not necessarily meet the needs of downstream projects based on FreeBSD, and given the new "packaged base", adding per-package licensing metadata in a way similar to how the Ports Collection works is under consideration as an alternative mechanism. Concerns were raised regarding the pending HardenedBSD entry in the previous quarterly report prior to publication. The FreeBSD project welcomes reports from separate (but derived) projects in quarterly reports and has included similar reports in the past from other projects (such as TrueOS and pfSense). The HardenedBSD report was edited for length and to concentrate on activities during the quarter in question. Amazon is proposing to set up mirrors of the freebsd-update and pkg servers within AWS in order to provide faster access for EC2 users. These mirrors will be publicly accessible, but the expectation is that use will primarily be from within EC2. FreeBSD AMIs will have a preset configuration that references the Amazon servers. The old, long-deprecated, and insecure "r-commands" (rsh, rlogin, rcp) are being removed from the base system for 12.0-RELEASE. Notice of this was added to the man pages and release notes in time for 11.1-RELEASE and 10.4-RELEASE. Anyone requiring these commands for backwards compatibility can use the new net/bsdrcmds port. Work to replace Heimdal Kerberos in base with the more widely compatible MIT Kerberos has begun in a new projects/krb5 branch. This should not fall afoul of any US cryptography export regulations: the project is required to notify the US government that cryptographic software can be downloaded from FreeBSD servers, and this already covers MIT Kerberos, already available within ports. A number of Bay Area FreeBSD User Group-related domain names are being given up by their original owner. The current BAFUG organisers have been made aware. Core has voted on a change to the Doceng voting rules to provide for a "did not vote" status during doceng voting similar to how portmgr and core voting operates. The current requirement for all five members of doceng to register a vote on issues was proving to be a significant bottleneck. __________________________________________________________________ The FreeBSD Foundation Links FreeBSD Foundation Website URL: https://www.FreeBSDFoundation.org/ FreeBSD Foundation Quarterly Newsletter URL: https://www.FreeBSDfoundation.org/wp-content/uploads/2017/08/FreeB= SD-Foundation-July-August-2017-Update.pdf Contact: Deb Goodkin The FreeBSD Foundation is a 501(c)(3) non-profit organization dedicated to supporting and promoting the FreeBSD Project and community worldwide. Funding comes from individual and corporate donations and is used to fund and manage software development projects, conferences and developer summits, and provide travel grants to FreeBSD contributors. The Foundation purchases and supports hardware to improve and maintain FreeBSD infrastructure and provides full-time Release Engineering support; publishes marketing material to promote, educate, and advocate for the FreeBSD Project; facilitates collaboration between commercial vendors and FreeBSD developers; and finally, represents the FreeBSD Project in executing contracts, license agreements, and other legal arrangements that require a recognized legal entity. Here are some highlights of what we did to help FreeBSD last quarter: Fundraising Efforts Our work is 100% funded by your donations. This year we have raised over $860,000 from over 500 donors. Our 2017 fundraising goal is $1,250,00 and we are continuing to work hard to meet and exceed this goal! Please consider making a donation to help us continue and increase our support for FreeBSD: https://www.FreeBSDfoundation.org/donate/. We also have a new Partnership Program, to provide more benefits for our larger commercial donors. Find out more information at https://www.FreeBSDfoundation.org/FreeBSD-foundation-partnership-program/ and share with your companies! OS Improvements The Foundation improves the FreeBSD operating system by employing our technical staff to maintain and improve critical kernel subsystems, add features and functionality, and fix problems. This also includes funding separate project grants like the arm64 port, blacklistd access control daemon, and the integration of VIMAGE support, to make sure that FreeBSD remains a viable solution for research, education, computing, products and more. We kicked off or continued the following projects last quarter: * OpenZFS RAID-Z Expansion project * Broadcom Wi-Fi infrastructural improvements (bhnd(4) driver) * Headless mode out-of-the-box for the Beaglebone Black * Extending bhyve/ARMv7 features * Porting bhyve/ARM to an ARMv8 platform Having software developers on staff has allowed us to jump in and work directly on projects to improve FreeBSD like: * ZFS improvements * New Intel server support * kqueue(2) updates * 64-bit inode support * Stack guard * Kernel Undefined Behavior Sanitizer * Toolchain projects * i915 driver investigation * NVDIMM support in acpiconf(8) * Continuous integration dashboard (web page and physical hardware) * FAT filesystem support in makefs(8) Staff and board members continued hosting bi-weekly conference calls to facilitate efforts for individuals to collaborate on different technologies. Release Engineering The Foundation provides a full-time staff member to lead the release engineering efforts. This has provided timely and reliable releases over the last few years. Last quarter, our full-time staff member worked with the FreeBSD Release Engineering and Security Teams to finalize 11.1-RELEASE. He also supported the 10.4 release effort, and has continued producing 10-STABLE, 11-STABLE, and 12-CURRENT development snapshot builds throughout the quarter. At the vBSDCon Developer Summit, he gave a presentation on the state of the release engineering team. You can find out more about the support we provided to the Release Engineering Team by reading their status update in this report. Supporting FreeBSD Infrastructure The Foundation provides hardware and support to improve the FreeBSD infrastructure. Last quarter, we continued supporting FreeBSD hardware located around the world. FreeBSD Advocacy and Education A large part of our efforts are dedicated to advocating for the Project. This includes promoting work being done by others with FreeBSD; producing advocacy literature to teach people about FreeBSD and help make the path to starting using FreeBSD or contributing to the Project easier; and attending and getting other FreeBSD contributors to volunteer to run FreeBSD events, staff FreeBSD tables, and give FreeBSD presentations. The FreeBSD Foundation sponsors many conferences, events, and summits around the globe. These events can be BSD-related, open source, or technology events geared towards underrepresented groups. We support the FreeBSD-focused events to help provide a venue for sharing knowledge, to work together on projects, and to facilitate collaboration between developers and commercial users. This all helps provide a healthy ecosystem. We support the non-FreeBSD events to promote and raise awareness of FreeBSD, to increase the use of FreeBSD in different applications, and to recruit more contributors to the Project. Here is a list highlighting some of the advocacy and education work we did last quarter: * Organized and ran the Essen FreeBSD Hackathon in Essen Germany * Sponsored and participated in the FreeBSD Developer Summit BSDCam, in Cambridge, England * Represented FreeBSD at the ARM Partner Meeting * Presented and taught about FreeBSD at SdNOG 4 in Khartoum, Sudan * Sponsored and gave presentations and tutorials at EuroBSDCon in Paris, France * Organized and ran the Paris FreeBSD Developer Summit * Organized and ran the FreeBSD Developer Summit at vBSDCon * Sponsored and attended vBSDCon * Proved travel grants to FreeBSD contributors to attend the above events. * Sponsored the 2017 USENIX Security Symposium in Vancouver BC as an Industry Partner * Provided FreeBSD advocacy material * Sponsored the 2017 USENIX Annual Technical Conference in Santa Clara, CA as an Industry Partner We continued producing FreeBSD advocacy material to help people promote FreeBSD around the world. We help educate the world about FreeBSD by publishing the professionally produced FreeBSD Journal. Last quarter we published the July/August issue that you can find at https://www.FreeBSDfoundation.org/journal/. You can find out more about events we attended and upcoming events at https://www.FreeBSDfoundation.org/news-and-events/. Legal/FreeBSD IP The Foundation owns the FreeBSD trademarks, and it is our responsibility to protect them. We also provide legal support for the core team to investigate questions that arise. Go to http://www.FreeBSDfoundation.org to find out how we support FreeBSD and how we can help you! __________________________________________________________________ Projects Projects that span multiple categories, from the kernel and userspace to the Ports Collection or external projects. FreeBSD CI Links freebsd-ci Repository URL: https://github.com/freebsd/freebsd-ci freebsd-testing Mailing List URL: https://lists.FreeBSD.org/mailman/listinfo/freebsd-testing FreeBSD Jenkins Instance URL: http://ci.FreeBSD.org Contact: Jenkins Admins The FreeBSD CI team runs various continuous integration solutions for FreeBSD, regularly checking that the current state of the Subversion repository can successfully build, and performing various tests and analysis upon the build results. We have introduced a DTrace test pipeline, with the results and artifacts available at: * https://ci.FreeBSD.org/job/FreeBSD-head-amd64-dtrace_test/ * https://artifact.ci.FreeBSD.org/dtrace-test/ We had team meetings at two developer summits during Q3: * BSDcam * EuroBSDCon Open tasks: 1. Fix the failing test cases and builds. 2. Create builds for additional architectures. 3. Add more tests. 4. The additional TODO items listed at https://wiki.FreeBSD.org/Jenkins/TODO . __________________________________________________________________ Kernel Updates to kernel subsystems/features, driver support, filesystems, and more. Intel 10G iflib Driver Update Links ixgbe iflib Conversion URL: https://reviews.FreeBSD.org/D11727 Contact: Chris Galazka Contact: Piotr Pietruszewski The ix and ixv network interface drivers support a variety of Intel network interfaces, with line speeds at 10 Gbit/second. This quarter, with the help of Matt Macy and Sean Bruno (among others), we have submitted a review in Phabricator for the conversion of the ixgbe driver to use the new (and evolving) iflib framework. Stay tuned for the conversion of the 40G driver (ixl), as it is currently being ported to use iflib. Open tasks: 1. Additional testing. __________________________________________________________________ Intel iWARP Support Links iWARP for ixl URL: https://reviews.FreeBSD.org/D11378 Contact: Bartosz Sobczak iWARP is a protocol suite that enables efficient movement of data across the network, building on Remote Direct Memory Access, Direct Data Placement, and Marker PDU Aligned Framing. It endeavors to avoid unnecessary (local) data copies and to offload work from the main CPU to dedicated hardware. An initial commit adding iWARP support for the Intel X722 family of network adapters is under review. This is an important step towards introducing full iWARP support on systems equipped with Intel C620 Series Chipsets. Currently, with the iw_ixl driver, only the kVerbs API is supported. Open tasks: 1. Additional testing. __________________________________________________________________ pNFS Server Plan B Links Instructions for Testing URL: http://people.FreeBSD.org/~rmacklem/pnfs-planb-setup.txt Contact: Rick Macklem A pNFS server allows an NFS service to be spread over multiple servers, separating the MetaData operations from the Data operations (Read and Write). This project will add the ability to use FreeBSD systems to create a pNFS service consisting of a single MetaData Server plus a set of Data Servers. The Data Servers can be mirrored, so that redundant copies of the file data are maintained. The support for non-mirrored Data Servers is now believed to be complete. Support for mirrored Data Servers using the Flexible File Layout, which will soon be published as an RFC, is implemented. However, there is still significant work to be done, since the current implementation of mirrored Data Servers does not handle failed Data Servers or their resilvering/recovery. It is hoped that support for failure/recovery of Data Servers will be implemented in the next six months. The patched FreeBSD sources may now be accessed for testing via either Subversion or downloading a gzipped tarball. They consist of a patched kernel and nfsd and can be used on any FreeBSD 11 or later system. The installation procedure is covered in the linked document. Open tasks: 1. Testing by others will be needed, now that the implementation is available. 2. Implementation and testing of mirror failure/recovery. __________________________________________________________________ Architectures Updating platform-specific features and bringing in support for new hardware platforms. AMD Zen (family 17h) support Contact: Conrad Meyer cem@FreeBSD.org <> This quarter, a bit of work was done to enhance platform support for AMD Zen (Ryzen, Threadripper, Epyc) processors: * The CPU topology detection code was enhanced to properly detect Zen dies and CPU Complexes. This gives the scheduler more locality information to use when making scheduling decisions. * The x86 topology analysis was enhanced to report dies and CPU Complexes, in addition to the existing reporting on packages, cores, and threads. An example of the new output is FreeBSD/SMP: 1 package(s) x 2 groups x 2 cache groups x 4 core(s) x 2 hardware threads. * The amdsmn(4) driver for accessing SMN (System Management Network) registers was added. * CPU temperature monitoring support for Zen was added to amdtemp(4). * In cpufreq(4): + Added support for decoding Zen P-state information from Machine State Registers (which is usually not necessary, since it is largely redundant with ACPI P-state information, but is potentially useful) + Work around the apparent Ryzen inability to achieve the P1 state by not busying cores waiting to transition to it * The intpm(4) smbus driver was fixed to attach to the AMD FCH (Fusion Controller Hub). * All MCA banks are now enabled and monitored on Zen CPUs. * Feature-bit decoding was added for: CLZERO, SVM features, and RAS capabilities. * SHA intrinsic support was added to the aesni(4) driver. Ryzen is currently the only desktop processor to feature these intrinsics. Support for these intrinsics is also present in Intel's Goldmont line of low-end SoCs. Overall, Zen is now a very usable platform for x86 workstations and servers. This project was sponsored by Dell EMC Isilon. Open tasks: 1. Add HWPMC support for the new performance counters avilable on the Zen architecture. 2. Add support for the CCP (Crypto Co-Processor). __________________________________________________________________ Userland Programs Changes affecting the base system and programs in it. Updates to GDB Contact: John Baldwin Contact: Luca Pizzamiglio The devel/gdb port has been updated to GDB 8.0.1. Support for FreeBSD/aarch64 userland binaries has been committed upstream. These patches, along with support for debugging FreeBSD/aarch64 kernels, have been committed to the port. Upstream patches adding improved support for FreeBSD/arm userland binaries are currently in review. FreeBSD 12 has recently grown support for debugging VFP registers via ptrace() and core dumps as part of this work. Support for FreeBSD/arm kernels will be added to the port after the upstream patches are added to the port. Support for $_siginfo has been committed upstream. This uses the recently added NT_LWPINFO note to extract signal information from process cores. Hangs that occured when GDB's kill command was used were fixed in FreeBSD in r313992. Open tasks: 1. Figure out why the powerpc kgdb targets are not able to unwind the stack past the initial frame. 2. Test support for sparc64 binaries and kernels. 3. Add support for debugging powerpc vector registers. 4. Implement info proc commands. 5. Implement info os commands. __________________________________________________________________ Ports Changes affecting the Ports Collection, whether sweeping changes that touch most of the tree, or individual ports themselves. FreeBSDDesktop Links FreeBSDDesktop on GitHub URL: https://github.com/FreeBSDDesktop/ Contact: Johannes Dieterich Contact: Mark Johnston Contact: Hans Petter Selasky Contact: Matthew Macy The FreeBSDDesktop team is happy to announce the availability of graphics/drm-next-kmod. This port for FreeBSD-CURRENT (amd64) provides support for the amdgpu, i915, and radeon DRM modules using the linuxkpi compatibility framework. The port currently corresponds to the DRM from Linux 4.9 and is in an experimental state. It works reliably for many testers with modern GPU hardware (AMD HD7000 series/Tahiti to Polaris and Intel HD3000/Sandy Bridge to Skylake). Broader testing and reporting/fixing of bugs is appreciated. Open tasks: 1. Resolve issues that cause radeonkms and amdgpu to fail with EFI boot (though there is a workaround available). 2. Upgrade to Linux 4.10 and higher DRM versions. 3. Get feedback from broader testing. __________________________________________________________________ OpenJFX 8 Links OpenJFX Wiki URL: https://wiki.openjdk.java.net/display/OpenJFX/Main java/openjfx8-devel URL: https://www.freshports.org/java/openjfx8-devel java/openjfx8-scenebuilder URL: https://www.freshports.org/java/openjfx8-scenebuilder AsciidocFX URL: https://github.com/asciidocfx/AsciidocFX Contact: Tobias Kortkamp OpenJFX is an open source, next generation, client application platform for desktop and embedded systems, based on JavaSE. This quarter, the OpenJFX port was reworked and has received some significant improvements. More modules are being built. With the new web module we gain support for applications that have their own builtin web browser such as AsciidocFX. The new media module allows JavaFX applications to play audio and video files. A port of the JavaFX scenebuilder, a RAD tool for building JavaFX scenes, was added to the ports tree. The OpenGL Prism backend for GPU acceleration was enabled by default. From a mainainer's and contributor's perspective, the port was simplified by moving all FreeBSD-local patches to the ports tree and fetching the upstream sources directly, instead of using a separate repository for them. Open tasks: 1. Upstream some of the patches in the Ports Collection. __________________________________________________________________ Puppet Links Puppetlab's FreeBSD Slack Channel URL: https://puppetcommunity.slack.com/messages/C6CK0UGB1/ Contact: Puppet Team This summer has seen the creation of a puppet@ team to help maintain the approximately 30 Puppet-related ports in the FreeBSD Ports Collection. These ports were previously maintained by various committers, and from time to time the distributed maintainership introduced some delays when updating a port, due to the need to wait for a maintainer's approval for a related change to a different port. Puppet 5 is now in the ports tree (as sysutils/puppet5). The C++ version of Facter (sysutils/facter) got a lot of attention and is now a drop-in replacement for the previous Ruby version (sysutils/rubygem-facter); it is the default facts source for the Puppet 5 port. Work continues on bringing in Puppetserver 5 to the ports tree, and on keeping all the ports up-to-date. Open tasks: 1. The pkg package provider has some minor issues (it breaks things when no repos are configured, and is not working properly from the context of the MCollective package agent). 2. The databases/puppetdb[345] and sysutils/puppetserver[45] ports rely on Clojure and Java, and download compiled jar files instead of building them from source. __________________________________________________________________ Documentation Noteworthy changes in the documentation tree or new external books/documents. Absolute FreeBSD, 3rd Edition Links Official Announcement URL: https://blather.michaelwlucas.com/archives/3020 Contact: Michael Lucas The first draft of the third edition of Absolute FreeBSD is finished. It is 220,200 words, or roughly enough to stun a medium-sized ox. It's on target to be in print before BSDCan 2018. Open tasks: 1. Stare at the wall blankly for a few days. 2. Fix all the problems pointed out by dozens of community reviewers. 3. Fix all the problems pointed out by John Baldwin, tech reviewer extraordinaire. 4. Editing. Copyediting. Page layout. Page editing. Re-editing. Indexing. Edits discovered by indexer. 5. Pre-orders should open some time next year. 6. Restrain myself from strangling people who ask when the fourth edition is coming. __________________________________________________________________ Manual Pages Links FreeBSD Documentation Project Primer URL: https://www.freebsd.org/doc/en_US.ISO8859-1/books/fdp-primer/ Contact: Warren Block Over the last year, interest has increased in manual pages, in large part due to excellent infrastructure work by Baptiste Daroussin and others, and promotion by George Neville-Neil and others. This increased interest has been both gratifying and problematic. Our man pages are underappreciated gems, but we have sadly lacked any substantial documentation on how to write new ones. In September, I added a new chapter to the FreeBSD Documentation Project Primer describing the basics of creating a man page. It includes descriptions of the markup, section structure, recommended optional material such as examples, and sample templates for the most common types of man pages. The Resources section includes links to several external resources, including the excellent Practical UNIX Manuals: mdoc. While this chapter is not a full tutorial, it does begin to fill in a large gap in our documentation resources and provide a starting point from which to grow. Open tasks: 1. Add more explanation and examples of markup usage. 2. Expand the sample templates with additional desired standard features, like an EXAMPLES section. 3. Add more sample templates. __________________________________________________________________ Third-Party Projects Many projects build upon FreeBSD or incorporate components of FreeBSD into their project. As these projects may be of interest to the broader FreeBSD community, we sometimes include brief updates submitted by these projects in our quarterly report. The FreeBSD project makes no representation as to the accuracy or veracity of any claims in these submissions. The nosh Project Links Introduction URL: http://jdebp.eu./Softwares/nosh/ FreeBSD Binary Packages URL: http://jdebp.eu./Softwares/nosh/freebsd-binary-packages.html Installation How-To URL: http://jdebp.eu./Softwares/nosh/timorous-admin-installation-how-to= =2Ehtml Roadmap URL: http://jdebp.eu./Softwares/nosh/roadmap.html A Slightly Outdated User Guide URL: http://jdebp.eu./Softwares/nosh/guide/index.html Archnosh URL: http://framagit.org/taca/archnosh Contact: Jonathan de Boyne Pollard The nosh project is a suite of system-level utilities for initializing, running, and shutting down BSD systems; and for managing daemons, terminals, and logging. It attempts to supersede BSD init, the Mewburn rc.d system, and OpenRC as used on FreeBSD and TrueOS, drawing inspiration from Solaris SMF for named milestones, daemontools-encore for service control/status mechanisms, UCSPI, and IBM AIX for separated service and system management. It includes a range of compatibility mechanisms, including shims for familiar commands from other systems, and an automatic import mechanism that takes existing configuration data from /etc/fstab, /etc/rc.conf{,.local}, /etc/ttys, and elsewhere, applying them to its native service definitions and creating additional native services. It is portable (including to Linux) and composable, it provides a migration path from the world of systemd Linux, and it does not require new kernel APIs. It provides clean service environments, orderings and dependencies between services, parallelized startup and shutdown (including fsck), strictly size-capped and autorotated logging, the service manager as a "subreaper", and uses kevent(2) for event-driven parallelism. Since the last status report, in December 2015, the project has seen: restructured and finer-grained packaging that has fewer conflicts with other toolsets; the addition of zsh completion files; improvements to the virtual terminal subsystem, keyboard map, mouse support, and ugen and DECSCUSR support; RFC 5424/5426 remote logging support; replacement of libkqueue and the C library's environment handling functions; several new helper commands; support for Java VM autolocation; improved socket-passing code; an extended status API and "one-shot" service support; additional pre-supplied service bundles; support for service aliases; improved handling of per-user D-Bus services; improved importing of MySQL, MariaDB, Percona, and OpenVPN services; improved configuration import support; and extensive additions to the nosh Guide. On the recently updated roadmap you can see plans for even more documentation, continuing the work to extend the capabilities of the networking subsystem, and the scant handful of rc.d-related items remaining. There are also some ideas still in the speculative or planning phases, including work that may depend on incorporating nosh support into other software. Open tasks: 1. Improve Ansible and SaltStack integration (the maintainer of the Arch Linux nosh integration has some ideas). 2. Command-line completions are still needed for bash, csh, and fish. 3. Document convert-systemd-units for use by port maintainers in making packaged service bundles from systemd unit files. 4. nosh could take advantage of several proposed features for the base system: + the boot loader signaling "emergency" and "rescue" modes of operation + adding machine-readable status output to fsck + adding runtime support for more clang-compilable languages in the early bootstrap stage + adding hooks for invoking external configuration import mechanisms __________________________________________________________________ --DSayHWYpDlRfCAAQ Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQG3BAABCgAdFiEE2WGV4E2ARf9BYP0XKNmm82TrdRIFAlpBkBAACgkQKNmm82Tr dRJOgwwg5WCG7cNAQURqjmTzoc7Uc9JwYnksJ6dJzrod9rV0ToAqPJzMD+G2TEKX L7IX2yB2IMGFKb6pLRZ9UF6Z3EasSXDwq4H2G8qxe12mK9Oc/yV8rY1YZu6DYLoV czpgzhZbpvDOkjXl7/z4x7NI097Rrh9gj9SO8jZ5g/ufAzfo/qPtt9tNlcxarNI3 TGjN4jFadgbqohI12NO9AnsJO2sKfgDOO5UKbk8r2pcBoQBns2GkftjEiX0xKMc6 ZWxht72aVUtI6gaWLVq2yDSAg8hR86NB+0mib28QXBPL7jGg6cYYwXHRVKYTYMS7 BTqdQMQVva2xDg5KLZA20D+OWb+G4HEAjvMmB7TZeINbFJuLl6NWw/cDJAAR7gxF Z6pyAPSmadkXRlJ4TPzCm8LGJflEOozafv/4bsN6s3v+Fq1SpYVF/v9VwtqL/JxT NsfTzHnE3ZSujXmPM9bAD87VPO5vsrK2G4lYBx8MoVsLpD5asNvPbvJ/Z8nip+Ov dtbK8avd2FqQFQ== =5EQJ -----END PGP SIGNATURE----- --DSayHWYpDlRfCAAQ-- From owner-freebsd-stable@freebsd.org Tue Dec 26 09:11:04 2017 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 99925EABD7C; Tue, 26 Dec 2017 09:11:04 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [78.47.246.247]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 2EE9B6B97B; Tue, 26 Dec 2017 09:11:03 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221] (may be forged)) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id vBQ9AouN058874 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 26 Dec 2017 10:10:51 +0100 (CET) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: freebsd-stable@FreeBSD.org Received: from eg.sd.rdtc.ru (eugen@localhost [127.0.0.1]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTP id vBQ9Ag3p048301; Tue, 26 Dec 2017 16:10:43 +0700 (+07) (envelope-from eugen@grosbein.net) To: FreeBSD Stable From: Eugene Grosbein Subject: idprio(1) broken X-Enigmail-Draft-Status: N1110 Cc: FreeBSD Hackers Message-ID: <5A421212.4040703@grosbein.net> Date: Tue, 26 Dec 2017 16:10:42 +0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=2.2 required=5.0 tests=BAYES_00, LOCAL_FROM, RDNS_NONE autolearn=no autolearn_force=no version=3.4.1 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 1.9 RDNS_NONE Delivered to internal network by a host with no rDNS * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on hz.grosbein.net X-Spam-Level: ** X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Dec 2017 09:11:04 -0000 Hi! Is idprio(1) broken in stable/11? As root, start one bzip2 instance with idprio and one additional bzip2 intance per CPU core: # idprio 5 bzip2 -9 /dev/null & # n=$(sysctl -n kern.smp.cpus) # i=1; while [ $i -le $n ]; do bzip2 -9 /dev/null & i=$(($i+1)); done # top For dual core system, I see that idprio'd bzip2 takes all cycles of first core and two "normal" bzip2's share cycles of second core each taking ~50% of CPU time. It is expected that idprio'd bzip2 get no CPU time at all and each of "normal" bzip2's get ~100% of single CPU core for such setup. Eugene Grosbein From owner-freebsd-stable@freebsd.org Tue Dec 26 09:18:32 2017 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 314F5EAC4C1; Tue, 26 Dec 2017 09:18:32 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [78.47.246.247]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id A9ECB6BF75; Tue, 26 Dec 2017 09:18:31 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221] (may be forged)) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id vBQ9IND2058935 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 26 Dec 2017 10:18:24 +0100 (CET) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: freebsd-stable@FreeBSD.org Received: from eg.sd.rdtc.ru (eugen@localhost [127.0.0.1]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTP id vBQ9IKLl050625; Tue, 26 Dec 2017 16:18:20 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: idprio(1) broken To: FreeBSD Stable References: <5A421212.4040703@grosbein.net> Cc: FreeBSD Hackers From: Eugene Grosbein Message-ID: <5A4213DC.20508@grosbein.net> Date: Tue, 26 Dec 2017 16:18:20 +0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 In-Reply-To: <5A421212.4040703@grosbein.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=2.2 required=5.0 tests=BAYES_00, LOCAL_FROM, RDNS_NONE, T_DATE_IN_FUTURE_Q_PLUS autolearn=no autolearn_force=no version=3.4.1 X-Spam-Report: * 0.0 T_DATE_IN_FUTURE_Q_PLUS Date: is over 4 months after Received: date * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 1.9 RDNS_NONE Delivered to internal network by a host with no rDNS * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on hz.grosbein.net X-Spam-Level: ** X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Dec 2017 09:18:32 -0000 On 26.12.2017 16:10, Eugene Grosbein wrote: > Is idprio(1) broken in stable/11? > > As root, start one bzip2 instance with idprio and one additional bzip2 intance per CPU core: > > # idprio 5 bzip2 -9 /dev/null & > # n=$(sysctl -n kern.smp.cpus) > # i=1; while [ $i -le $n ]; do bzip2 -9 /dev/null & i=$(($i+1)); done > # top > > For dual core system, I see that idprio'd bzip2 takes all cycles of first core > and two "normal" bzip2's share cycles of second core each taking ~50% of CPU time. > > It is expected that idprio'd bzip2 get no CPU time at all and each of "normal" bzip2's > get ~100% of single CPU core for such setup. This works as expected for stable/10. From owner-freebsd-stable@freebsd.org Tue Dec 26 12:01:35 2017 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 87B8DE891D9; Tue, 26 Dec 2017 12:01:35 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf0-f54.google.com (mail-lf0-f54.google.com [209.85.215.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D55BA722D0; Tue, 26 Dec 2017 12:01:34 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf0-f54.google.com with SMTP id f13so39419506lff.12; Tue, 26 Dec 2017 04:01:34 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=CrhIZ/Q9EvJosyG+gLCU2iiFg13+DyriWf3cYt1lSAg=; b=Mq0yQRR6JIz+nOtORXfk3qmdh3CsAesiaHcd5Zqr5Fpukq+NU/l0wI9l9n/6/jytWI 7zHtFLvu02NsNinP7P/Kckk9jVrrOIZht9dyRdu3UiHX8Oge4eYLkAHAuqUYrbjGlq3w mgI+8qM6fZOR7+39kaKLVaWcvoWXAME4SFIwn7ayfkXd4V/R3g00t9kjGm4zcBGiTjGG xQiUWkFJHTwysWnNUc9MBEbBvzPreL6WmZBy4MUbsv+HDqra2XTVum1avi4VwNZafFT0 75+QzIITv81DO/9dHPMfH9RReRoLyq7hGq2wIYv5JjzLllBrkTYpk/mDWPm0xM4WsPyA mzdg== X-Gm-Message-State: AKGB3mICn93Ms5qSVsBqYmudCOWFdzx52BJjmCSCVtIU0JEJsLg+iPU5 otnoQU2JUL2OlS11reU9xFrt92qM X-Google-Smtp-Source: ACJfBovQeQ7JCmsuO5TVessS2UqUSr/aJftKJz/yjXBFJo6HNscB4PwFbz3wlADAerhV7Rerbij9QA== X-Received: by 10.25.145.9 with SMTP id t9mr4179775lfd.88.1514288247340; Tue, 26 Dec 2017 03:37:27 -0800 (PST) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id v207sm1860299lfa.3.2017.12.26.03.37.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 26 Dec 2017 03:37:26 -0800 (PST) Subject: Re: idprio(1) broken To: Eugene Grosbein , FreeBSD Stable Cc: FreeBSD Hackers References: <5A421212.4040703@grosbein.net> <5A4213DC.20508@grosbein.net> From: Andriy Gapon Message-ID: <92a254fb-8066-4930-e4ad-f4fedb0e84d0@FreeBSD.org> Date: Tue, 26 Dec 2017 13:37:25 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.5.0 MIME-Version: 1.0 In-Reply-To: <5A4213DC.20508@grosbein.net> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Dec 2017 12:01:35 -0000 On 26/12/2017 11:18, Eugene Grosbein wrote: > On 26.12.2017 16:10, Eugene Grosbein wrote: > >> Is idprio(1) broken in stable/11? >> >> As root, start one bzip2 instance with idprio and one additional bzip2 intance per CPU core: >> >> # idprio 5 bzip2 -9 /dev/null & >> # n=$(sysctl -n kern.smp.cpus) >> # i=1; while [ $i -le $n ]; do bzip2 -9 /dev/null & i=$(($i+1)); done >> # top >> >> For dual core system, I see that idprio'd bzip2 takes all cycles of first core >> and two "normal" bzip2's share cycles of second core each taking ~50% of CPU time. >> >> It is expected that idprio'd bzip2 get no CPU time at all and each of "normal" bzip2's >> get ~100% of single CPU core for such setup. > > This works as expected for stable/10. Seems to work as expected on head as well. Tested on a 6-core physical system and a 2-core bhyve VM. # ps axwwl | fgrep bzip2 0 943 935 0 129 5 10564 2196 - RN 1 0:00.00 idprio 5 bzip2 -9 0 945 935 0 108 5 18332 9676 - RN 1 1:16.15 bzip2 -9 0 946 935 0 108 5 18332 9676 - RN 1 1:16.34 bzip2 -9 idprio isn't even able to exec bzip2. # ps axwwl | fgrep bzip2 0 28986 86816 0 129 5 18348 2324 - RN 17 0:00.02 bzip2 -9 0 28988 86816 0 106 5 18348 9680 - RN 17 1:27.25 bzip2 -9 0 28989 86816 0 106 5 18348 9680 - RN 17 1:27.86 bzip2 -9 0 28990 86816 0 108 5 18348 9680 - RN 17 1:35.50 bzip2 -9 0 28991 86816 0 106 5 18348 9680 - RN 17 1:27.00 bzip2 -9 0 28992 86816 0 106 5 18348 9680 - RN 17 1:25.83 bzip2 -9 0 28993 86816 0 106 5 18348 9680 - RN 17 1:26.76 bzip2 -9 -- Andriy Gapon From owner-freebsd-stable@freebsd.org Tue Dec 26 12:41:31 2017 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C71F9E8B9D4; Tue, 26 Dec 2017 12:41:31 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [78.47.246.247]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 59DA073D17; Tue, 26 Dec 2017 12:41:30 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221] (may be forged)) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id vBQCfMYO060437 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 26 Dec 2017 13:41:22 +0100 (CET) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: avg@FreeBSD.org Received: from eg.sd.rdtc.ru (eugen@localhost [127.0.0.1]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTP id vBQCfI47018041; Tue, 26 Dec 2017 19:41:18 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: idprio(1) broken To: Andriy Gapon , FreeBSD Stable References: <5A421212.4040703@grosbein.net> <5A4213DC.20508@grosbein.net> <92a254fb-8066-4930-e4ad-f4fedb0e84d0@FreeBSD.org> Cc: FreeBSD Hackers From: Eugene Grosbein Message-ID: <5A42436E.9060502@grosbein.net> Date: Tue, 26 Dec 2017 19:41:18 +0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 In-Reply-To: <92a254fb-8066-4930-e4ad-f4fedb0e84d0@FreeBSD.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=2.2 required=5.0 tests=BAYES_00, LOCAL_FROM, RDNS_NONE, T_DATE_IN_FUTURE_Q_PLUS autolearn=no autolearn_force=no version=3.4.1 X-Spam-Report: * 0.0 T_DATE_IN_FUTURE_Q_PLUS Date: is over 4 months after Received: date * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 1.9 RDNS_NONE Delivered to internal network by a host with no rDNS * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on hz.grosbein.net X-Spam-Level: ** X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Dec 2017 12:41:31 -0000 On 26.12.2017 18:37, Andriy Gapon wrote: >>> Is idprio(1) broken in stable/11? >>> >>> As root, start one bzip2 instance with idprio and one additional bzip2 intance per CPU core: >>> >>> # idprio 5 bzip2 -9 /dev/null & >>> # n=$(sysctl -n kern.smp.cpus) >>> # i=1; while [ $i -le $n ]; do bzip2 -9 /dev/null & i=$(($i+1)); done >>> # top >>> >>> For dual core system, I see that idprio'd bzip2 takes all cycles of first core >>> and two "normal" bzip2's share cycles of second core each taking ~50% of CPU time. >>> >>> It is expected that idprio'd bzip2 get no CPU time at all and each of "normal" bzip2's >>> get ~100% of single CPU core for such setup. >> >> This works as expected for stable/10. > > Seems to work as expected on head as well. I have a report that head r323607 has the problem and head r326729 has not. I can't check it myself as I have do not run head yet. From owner-freebsd-stable@freebsd.org Wed Dec 27 03:10:21 2017 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 159D2E9D304 for ; Wed, 27 Dec 2017 03:10:21 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: from asp.reflexion.net (outbound-mail-210-101.reflexion.net [208.70.210.101]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C685973C61 for ; Wed, 27 Dec 2017 03:10:19 +0000 (UTC) (envelope-from markmi@dsl-only.net) Received: (qmail 22177 invoked from network); 27 Dec 2017 03:03:33 -0000 Received: from unknown (HELO rtc-sm-01.app.dca.reflexion.local) (10.81.150.1) by 0 (rfx-qmail) with SMTP; 27 Dec 2017 03:03:33 -0000 Received: by rtc-sm-01.app.dca.reflexion.local (Reflexion email security v8.40.3) with SMTP; Tue, 26 Dec 2017 22:03:33 -0500 (EST) Received: (qmail 11430 invoked from network); 27 Dec 2017 03:03:33 -0000 Received: from unknown (HELO iron2.pdx.net) (69.64.224.71) by 0 (rfx-qmail) with (AES256-SHA encrypted) SMTP; 27 Dec 2017 03:03:33 -0000 Received: from [192.168.1.25] (c-76-115-7-162.hsd1.or.comcast.net [76.115.7.162]) by iron2.pdx.net (Postfix) with ESMTPSA id 5A087EC7B31; Tue, 26 Dec 2017 19:03:32 -0800 (PST) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: 11.1-STABLE for amd64: jumping from -r326142 to -r327228: all_subdir_cxgbe/t4_firmware failed to build Message-Id: <05A0A079-EF5A-4BAE-B2E9-97142D50A1D1@dsl-only.net> Date: Tue, 26 Dec 2017 19:03:31 -0800 To: FreeBSD Toolchain , FreeBSD-STABLE Mailing List X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Dec 2017 03:10:21 -0000 This update spans the clang upgrade to 5.0.1 and ld is listed in _ERROR_CMD. But I've no direct evidence that these contributed. Cleaning out /usr/obj/amd64_clang/amd64.amd64/ and rebuilding instead of having an incremental build did not reproduce the problem. I provide the information anyway, in case others sometimes see similar examples. --- all_subdir_cxgbe/t4_firmware --- *** [t4fw_cfg.txt.fwo] Error code 1 make[5]: stopped in /usr/src/sys/modules/cxgbe/t4_firmware .ERROR_TARGET=3D't4fw_cfg.txt.fwo' = .ERROR_META_FILE=3D'/usr/obj/amd64_clang/amd64.amd64/usr/src/sys/GENERIC/m= odules/usr/src/sys/modules/cxgbe/t4_firmware/t4fw_cfg.txt.fwo.meta' .MAKE.LEVEL=3D'5' MAKEFILE=3D'' .MAKE.MODE=3D'meta missing-filemon=3Dyes missing-meta=3Dyes silent=3Dyes = verbose' _ERROR_CMD=3D'@echo t4fw_cfg.txt = /usr/src/sys/dev/cxgbe/firmware/t4fw_cfg.txt; @if [ -e t4fw_cfg.txt ]; = then ld -b binary --no-warn-mismatch -d -warn-common = -m elf_x86_64_fbsd -r -d -o t4fw_cfg.txt.fwo = t4fw_cfg.txt; else = ln -s /usr/src/sys/dev/cxgbe/firmware/t4fw_cfg.txt t4fw_cfg.txt; ld -b = binary --no-warn-mismatch -d -warn-common -m elf_x86_64_fbsd -r = -d -o t4fw_cfg.txt.fwo t4fw_cfg.txt; = rm t4fw_cfg.txt; fi;' .CURDIR=3D'/usr/src/sys/modules/cxgbe/t4_firmware' .MAKE=3D'make' = .OBJDIR=3D'/usr/obj/amd64_clang/amd64.amd64/usr/src/sys/GENERIC/modules/us= r/src/sys/modules/cxgbe/t4_firmware' .TARGETS=3D'all' DESTDIR=3D'' LD_LIBRARY_PATH=3D'' MACHINE=3D'amd64' MACHINE_ARCH=3D'amd64' = MAKEOBJDIRPREFIX=3D'/usr/obj/amd64_clang/amd64.amd64/usr/src/sys/GENERIC/m= odules' MAKESYSPATH=3D'/usr/src/share/mk' MAKE_VERSION=3D'20170720' = PATH=3D'/usr/obj/amd64_clang/amd64.amd64/usr/src/tmp/legacy/usr/sbin:/usr/= obj/amd64_clang/amd64.amd64/usr/src/tmp/legacy/usr/bin:/usr/obj/amd64_clan= g/amd64.amd64/usr/src/tmp/legacy/bin:/usr/obj/amd64_clang/amd64.amd64/usr/= src/tmp/usr/sbin:/usr/obj/amd64_clang/amd64.amd64/usr/src/tmp/usr/bin:/sbi= n:/bin:/usr/sbin:/usr/bin' SRCTOP=3D'/usr/src' = OBJTOP=3D'/usr/obj/amd64_clang/amd64.amd64/usr/src/sys/GENERIC/modules/usr= /src' .MAKE.MAKEFILES=3D'/usr/src/share/mk/sys.mk = /usr/src/share/mk/local.sys.env.mk /usr/src/share/mk/src.sys.env.mk = /root/src.configs/src.conf.amd64-clang.amd64-host = /usr/src/share/mk/bsd.mkopt.mk /usr/src/share/mk/local.sys.mk = /usr/src/share/mk/src.sys.mk /dev/null = /usr/src/sys/modules/cxgbe/t4_firmware/Makefile = /usr/src/share/mk/bsd.kmod.mk /usr/src/sys/conf/kmod.mk = /usr/src/share/mk/bsd.init.mk /usr/src/share/mk/bsd.opts.mk = /usr/src/share/mk/bsd.cpu.mk /usr/src/share/mk/local.init.mk = /usr/src/share/mk/src.init.mk /usr/src/share/mk/bsd.own.mk = /usr/src/share/mk/bsd.compiler.mk /usr/src/sys/conf/kern.opts.mk = /usr/src/sys/conf/config.mk /usr/src/share/mk/bsd.links.mk = /usr/src/share/mk/bsd.dep.mk /usr/src/share/mk/bsd.clang-analyze.mk = /usr/src/share/mk/bsd.obj.mk /usr/src/share/mk/bsd.subdir.mk = /usr/src/sys/conf/kern.mk' .PATH=3D'. /usr/src/sys/modules/cxgbe/t4_firmware = /usr/src/sys/dev/cxgbe/firmware = /usr/obj/amd64_clang/amd64.amd64/usr/src/sys/GENERIC' # less = /usr/obj/amd64_clang/amd64.amd64/usr/src/sys/GENERIC/modules/usr/src/sys/m= odules/cxgbe/t4_firmware/t4fw_cfg.txt.fwo.meta # Meta data file = /usr/obj/amd64_clang/amd64.amd64/usr/src/sys/GENERIC/modules/usr/src/sys/m= odules/cxgbe/t4_firmware/t4fw_cfg.txt.fwo.meta CMD @echo t4fw_cfg.txt /usr/src/sys/dev/cxgbe/firmware/t4fw_cfg.txt CMD @if [ -e t4fw_cfg.txt ]; then ld -b binary = --no-warn-mismatch -d -warn-common -m elf_x86_64_fbsd -r -d = -o t4fw_cfg.txt.fwo t4fw_cfg.txt; else = ln -s = /usr/src/sys/dev/cxgbe/firmware/t4fw_cfg.txt t4fw_cfg.txt; ld -b binary = --no-warn-mismatch -d -warn-common -m elf_x86_64_fbsd -r -d = -o t4fw_cfg.txt.fwo t4fw_cfg.txt; rm = t4fw_cfg.txt; fi CWD = /usr/obj/amd64_clang/amd64.amd64/usr/src/sys/GENERIC/modules/usr/src/sys/m= odules/cxgbe/t4_firmware TARGET t4fw_cfg.txt.fwo -- command output -- t4fw_cfg.txt /usr/src/sys/dev/cxgbe/firmware/t4fw_cfg.txt *** Error code 1 -- filemon acquired metadata -- # filemon version 5 # Target pid 99801 # Start 1514338319.353476 V 5 E 99829 /bin/sh R 99829 /etc/libmap.conf R 99829 /var/run/ld-elf.so.hints R 99829 /lib/libedit.so.7 R 99829 /lib/libc.so.7 R 99829 /lib/libncursesw.so.8 F 99829 99831 E 99831 /bin/ln R 99831 /etc/libmap.conf R 99831 /var/run/ld-elf.so.hints R 99831 /lib/libc.so.7 L 99831 '/usr/src/sys/dev/cxgbe/firmware/t4fw_cfg.txt' 't4fw_cfg.txt' X 99831 0 0 F 99829 99835 E 99835 /usr/obj/amd64_clang/amd64.amd64/usr/src/tmp/usr/bin/ld D 99835 t4fw_cfg.txt.fwo R 99835 t4fw_cfg.txt.fwo W 99835 t4fw_cfg.txt.fwo R 99835 t4fw_cfg.txt X 99835 1 0 X 99829 1 0 # Stop 1514338319.363473 # Bye bye The "L 99831 '/usr/src/sys/dev/cxgbe/firmware/t4fw_cfg.txt' = 't4fw_cfg.txt'" indicates execution of the (whitespace changed below): else=20 ln -s /usr/src/sys/dev/cxgbe/firmware/t4fw_cfg.txt t4fw_cfg.txt; ld -b binary --no-warn-mismatch -d -warn-common -m elf_x86_64_fbsd -r = -d -o t4fw_cfg.txt.fwo t4fw_cfg.txt; rm t4fw_cfg.txt; fi The "E 99835 /usr/obj/amd64_clang/amd64.amd64/usr/src/tmp/usr/bin/ld" indicates which ld was executed. "X 99835 1 0" indicates a non-zero status return if I understand right. There is no "D t4fw_cfg.txt" line to match up with the "rm t4fw_cfg.txt", nor an "E" to match up with rm. =20 # uname -apKU FreeBSD FBSDFS 11.1-STABLE FreeBSD 11.1-STABLE r326142 amd64 amd64 = 1101506 1101506 # svnlite info /usr/src/ | grep "Re[plv]" Relative URL: ^/stable/11 Repository Root: svn://svn.freebsd.org/base Repository UUID: ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f Revision: 327228 Last Changed Rev: 327228 # more = ~/sys_build_scripts.amd64-host/make_amd64_nodebug_clang-amd64-host.sh=20 kldload -n filemon && \ script = /typescripts/sys_typescripts/typescript_make_amd64_nodebug_clang-amd64-hos= t-$(date +%Y-%m-%d:%H:%M:%S) \ env __MAKE_CONF=3D"/root/src.configs/make.conf" SRCCONF=3D"/dev/null" = SRC_ENV_CONF=3D"/root/src.configs/src.conf.amd64-clang.amd64-host" \ WITH_META_MODE=3Dyes \ MAKEOBJDIRPREFIX=3D"/usr/obj/amd64_clang/amd64.amd64" \ make $* # more /root/src.configs/src.conf.amd64-clang.amd64-host=20 TO_TYPE=3Damd64 # KERNCONF=3DGENERIC TARGET=3D${TO_TYPE} .if ${.MAKE.LEVEL} =3D=3D 0 TARGET_ARCH=3D${TO_TYPE} .export TARGET_ARCH .endif # WITH_META_MODE=3D #WITH_CROSS_COMPILER=3D WITH_SYSTEM_COMPILER=3D # WITH_LIBCPLUSPLUS=3D WITH_BINUTILS_BOOTSTRAP=3D WITH_ELFTOOLCHAIN_BOOTSTRAP=3D #WITH_CLANG_BOOTSTRAP=3D WITH_CLANG=3D WITH_CLANG_IS_CC=3D WITH_CLANG_FULL=3D WITH_CLANG_EXTRAS=3D #WITH_LLD=3D #WITHOUT_LLD_IS_LD=3D #WITH_LLVM_LIBUNWIND=3D #WITH_LLDB=3D #PORTS_MODULES=3Demulators/virtualbox-ose-additions # WITH_BOOT=3D WITH_LIB32=3D # WITHOUT_GCC_BOOTSTRAP=3D WITHOUT_GCC=3D WITHOUT_GCC_IS_CC=3D WITHOUT_GNUCXX=3D # NO_WERROR=3D #WERROR=3D MALLOC_PRODUCTION=3D # WITH_REPRODUCIBLE_BUILD=3D WITH_DEBUG_FILES=3D Ryzen Threadripper 1950X HW but FreeBSD -r327142 running under a Windows 10 Pro Hyper-V virtual machine. 110592 MB of RAM assigned. 29 virtual processors assigned. Physical hard disk used, not a virtual one. =3D=3D=3D Mark Millard markmi at dsl-only.net From owner-freebsd-stable@freebsd.org Wed Dec 27 10:50:36 2017 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A1524E84A42; Wed, 27 Dec 2017 10:50:36 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [78.47.246.247]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 2F49E1446; Wed, 27 Dec 2017 10:50:35 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221] (may be forged)) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id vBRAoLre070082 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 27 Dec 2017 11:50:22 +0100 (CET) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: avg@FreeBSD.org Received: from eg.sd.rdtc.ru (eugen@localhost [127.0.0.1]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTP id vBRAoDWO010683; Wed, 27 Dec 2017 17:50:13 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: idprio(1) broken To: Andriy Gapon , FreeBSD Stable References: <5A421212.4040703@grosbein.net> <5A4213DC.20508@grosbein.net> <92a254fb-8066-4930-e4ad-f4fedb0e84d0@FreeBSD.org> Cc: FreeBSD Hackers From: Eugene Grosbein Message-ID: <5A437AE5.4080104@grosbein.net> Date: Wed, 27 Dec 2017 17:50:13 +0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 In-Reply-To: <92a254fb-8066-4930-e4ad-f4fedb0e84d0@FreeBSD.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=2.2 required=5.0 tests=BAYES_00, LOCAL_FROM, RDNS_NONE, T_DATE_IN_FUTURE_Q_PLUS autolearn=no autolearn_force=no version=3.4.1 X-Spam-Report: * 0.0 T_DATE_IN_FUTURE_Q_PLUS Date: is over 4 months after Received: date * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 1.9 RDNS_NONE Delivered to internal network by a host with no rDNS * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on hz.grosbein.net X-Spam-Level: ** X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Dec 2017 10:50:36 -0000 On 26.12.2017 18:37, Andriy Gapon wrote: >>> It is expected that idprio'd bzip2 get no CPU time at all and each of "normal" bzip2's >>> get ~100% of single CPU core for such setup. >> >> This works as expected for stable/10. > > Seems to work as expected on head as well. I've updated my old stable/11 to recent stable/11 r327192 and problem's gone. Sorry for noise.